{"id":5714,"date":"2020-01-21T13:00:10","date_gmt":"2020-01-21T13:00:10","guid":{"rendered":"http:\/\/writeasync.net\/?p=5714"},"modified":"2020-01-19T23:32:18","modified_gmt":"2020-01-19T23:32:18","slug":"lets-do-dhcp-fuzzing","status":"publish","type":"post","link":"https:\/\/writeasync.net\/?p=5714","title":{"rendered":"Let&#8217;s do DHCP: fuzzing"},"content":{"rendered":"<p>The <a href=\"http:\/\/writeasync.net\/?p=5681\">DHCP server project<\/a> is still alive and well. It&#8217;s definitely sample code, for demonstration purposes only, and other disclaimers. Even so, it is a technology that processes arbitrary network data, and we ought to worry about <a href=\"https:\/\/owasp-aasvs.readthedocs.io\/en\/latest\/v5.html\">malicious input<\/a>. To quote from <a href=\"https:\/\/tools.ietf.org\/html\/rfc1122\">RFC 1122<\/a>, &#8220;assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect.&#8221;<\/p>\n<p>If we want this DHCP server to be <a href=\"https:\/\/en.wikipedia.org\/wiki\/Robustness_principle\">robust<\/a>, we cannot allow read operations to result in any undesirable behavior (hangs, crashes, etc.). Note that we will not attempt to protect <strong>write<\/strong> operations in the same way; these do not stem from untrusted input, but rather programmer errors (&#8220;boneheaded exceptions&#8221; in <a href=\"https:\/\/docs.microsoft.com\/en-us\/archive\/blogs\/ericlippert\/vexing-exceptions\">Eric Lippert&#8217;s classification<\/a>).<\/p>\n<p>Let&#8217;s start by devising a strategy to detect bad read behavior. Ideally, we want a set of operations which are simple and <em>exhaustive<\/em> &#8212; guaranteed to touch all relevant parts of the read buffer. String formatting is a good choice to meet both criteria. As of now, there are four general areas where this would be relevant:<\/p>\n<ol>\n<li>The entire DHCP message header<\/li>\n<li>The entire DHCP options buffer<\/li>\n<li>DHCP sub-options contained in a DHCP option (e.g. <a href=\"https:\/\/tools.ietf.org\/html\/rfc3046\">relay agent information<\/a>)<\/li>\n<li>Data contained within a DHCP sub-option (e.g. <a href=\"https:\/\/tools.ietf.org\/html\/rfc4014\">RADIUS attributes<\/a>)<\/li>\n<\/ol>\n<p>In practice, this results in <a href=\"http:\/\/writeasync.net\/?p=5707\"><code>TryFormat<\/code> methods<\/a> on these data types:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpMessageBuffer.cs#L359\">DhcpMessageBuffer<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpMessageBuffer.cs#L550\">DhcpMessageBuffer+OptionsSequence<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpOption.cs#L46\">DhcpOption<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpOption.cs#L81\">DhcpOption+SubOptionsSequence<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpSubOption.cs#L43\">DhcpSubOption<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpRelayAgentInformationSubOption.cs#L56\">DhcpRelayAgentInformationSubOption<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/DhcpRelayAgentInformationSubOption.cs#L160\">DhcpRelayAgentInformationSubOption+Sequence<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/RadiusAttribute.cs#L41\">RadiusAttribute<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/blob\/e9659a0ae792e44ad5c1945c9afe89647ec7c6b1\/DhcpServer.Core\/RadiusAttribute.cs#L76\">RadiusAttribute+Sequence<\/a><\/li>\n<\/ul>\n<p>Whew&#8230; hard work, but it&#8217;s for a good cause. And now we&#8217;re going to get to the fun part: fuzzing! We will use <a href=\"https:\/\/en.wikipedia.org\/wiki\/Fuzzing\">fuzz testing<\/a> (&#8220;providing invalid, unexpected, or random data as inputs&#8221;) to make sure that we caught all the possible read errors we might see in the <a href=\"https:\/\/archive.org\/details\/WildWorld\">wild world of the Internet<\/a>.<\/p>\n<p>Our fuzzing tool of choice is <a href=\"https:\/\/github.com\/Metalnem\/sharpfuzz\">SharpFuzz<\/a>. It works on .NET Core and has a good track record of <a href=\"https:\/\/github.com\/Metalnem\/sharpfuzz#trophies\">finding serious bugs<\/a> (even in <a href=\"https:\/\/github.com\/dotnet\/corefx\/issues\/35780\">.NET Core itself<\/a>). The one catch is that it doesn&#8217;t work natively on Windows, but <a href=\"https:\/\/docs.microsoft.com\/en-us\/windows\/wsl\/wsl2-about\">WSL 2<\/a> will do fine.<\/p>\n<p>I wanted this to be a fair fight, so to speak, so before running SharpFuzz I attempted to find and fix any obvious buffer handling issues. There were quite a few (<a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/cdef9c9b0890aa1d799e15f6a67e4a4e8d2f344a\">[1]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/c9cf03f58da23bf2140ea0f0d4d7c45531ca7c47\">[2]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/d912bc3cb1d81502df9db57ceffb104519bc42e8\">[3]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/878fad50d1fb25aa4fc230d12ccd327326e8b737\">[4]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/675bdca4b2b43a63f483b6864ef8d8806444805e\">[5]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/ba0b7121e3b049c0fbc7169c2fbc41f53ca36646\">[6]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/1f9cc340aaceefb5e429c3941138e04618c43b27\">[7]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/10ff37d8fa4932295e05a8decc5e702407ff9797\">[8]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/90270250a60dd0d4014929a35ee00a3e94d6425d\">[9]<\/a>) &#8212; so many, that I thought, &#8220;Surely, there won&#8217;t be anything left for SharpFuzz to do.&#8221; (Famous last words&#8230;)<\/p>\n<p>SharpFuzz needs a simple test and at least one input file to do its magic (I chose the basic DHCP request binary file I had been using for other unit tests). The test program just exercises all possible formatting scenarios, assuming that no exceptions should ever be thrown:<\/p>\n<pre class=\"brush: csharp; title: ; notranslate\" title=\"\">\r\n    internal sealed class Program\r\n    {\r\n        private static Memory&lt;byte&gt; bytes;\r\n        private static Memory&lt;char&gt; chars;\r\n        private static Lazy&lt;DhcpMessageBuffer&gt; lazyBuffer;\r\n\r\n        private static void Main(string&#x5B;] args)\r\n        {\r\n            ushort size = 300;\r\n            if (args.Length &gt; 0)\r\n            {\r\n                size = ushort.Parse(args&#x5B;0]);\r\n            }\r\n\r\n            bytes = new Memory&lt;byte&gt;(new byte&#x5B;size]);\r\n            chars = new Memory&lt;char&gt;(new char&#x5B;65536]);\r\n            lazyBuffer = new Lazy&lt;DhcpMessageBuffer&gt;(() =&gt; new DhcpMessageBuffer(bytes));\r\n            Fuzzer.Run(Run);\r\n        }\r\n\r\n        private static void Run(Stream stream)\r\n        {\r\n            DhcpMessageBuffer buffer = lazyBuffer.Value;\r\n            Span&lt;char&gt; destination = chars.Span;\r\n            ushort length = (ushort)stream.Read(buffer.Span);\r\n            buffer.Load(length);\r\n            buffer.TryFormat(destination, out _);\r\n            buffer.Options.TryFormat(destination, out _);\r\n            foreach (DhcpOption option in buffer.Options)\r\n            {\r\n                option.RelayAgentInformation().TryFormat(destination, out _);\r\n                option.SubOptions.TryFormat(destination, out _);\r\n                foreach (DhcpSubOption subOption in option.SubOptions)\r\n                {\r\n                    subOption.RadiusAttributes().TryFormat(destination, out _);\r\n                }\r\n            }\r\n        }\r\n    }\r\n<\/pre>\n<p>Note that since <a href=\"https:\/\/mijailovic.net\/2019\/01\/03\/sharpfuzz\/#technical-details\">SharpFuzz uses special binary instrumentation<\/a>, <a href=\"https:\/\/github.com\/Metalnem\/sharpfuzz\/issues\/8#issuecomment-522594667\"><strong>you cannot invoke the instrumented code outside of a Fuzzer.Run call<\/strong><\/a>. (I found this out the hard way.)<\/p>\n<p>Once I got past the initial issues and was able to launch the fuzzer, I thought there was a problem. It kept finishing very quickly with &#8220;Unable to communicate with fork server (OOM?)&#8221; <a href=\"https:\/\/github.com\/Metalnem\/sharpfuzz\/issues\/11#issuecomment-546723000\">even after setting the timeout, as suggested<\/a>. I eventually discovered that this is what happens when the input causes the program to hang. Oh my, our first critical bug!<\/p>\n<p>The problem ultimately boiled down to improper handling of <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/ff2121a60f0b2d230095095085900cb6727c091b\">option overloading<\/a>. In retrospect, this is not surprising, given that this option directly influences how the internal buffers are traversed. A few fixes later (<a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/ff2121a60f0b2d230095095085900cb6727c091b\">[1]<\/a>, <a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/bc12daff1d54ced4617b4acbf4bf0f589f81d29f\">[2]<\/a>) plus a regression test (<a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/67e7cb76d99a9a5bce832120ba1dd40da573b73d\">[3]<\/a>), and the hang was solved.<\/p>\n<p>Another run of the fuzzer thankfully did not show any more hangs. Instead, it found several more crashes. While they were reported as &#8220;uniq crash&#8221; results, there was really only one underlying bug. Again, option overloading was the culprit. This time it was a simple buffer overflow where it tried to read from a zero length data segment. In short order I was able to prepare a fix (<a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/580210a7686a1a6299c344bb916c25efc86e464d\">[1]<\/a>) and another regression test (<a href=\"https:\/\/github.com\/bobbymcr\/DhcpServer\/commit\/d59cd01a1e1f5d91eea9e038d459cae7bf2d3bf6\">[2]<\/a>). After running the fuzzer again with the corrected program for several hours, I have yet to see another failure.<\/p>\n<p>In summary, I highly recommend SharpFuzz for any .NET Core library that parses input. Despite the rough edges with Windows support and a learning curve, the results are worth it.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>The DHCP server project is still alive and well. It&#8217;s definitely sample code, for demonstration purposes only, and other disclaimers. Even so, it is a technology that processes arbitrary network data, and we ought to worry about malicious input. To&hellip; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[106,51],"tags":[],"class_list":["post-5714","post","type-post","status-publish","format-standard","hentry","category-security","category-testing"],"_links":{"self":[{"href":"https:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/5714","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5714"}],"version-history":[{"count":4,"href":"https:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/5714\/revisions"}],"predecessor-version":[{"id":5718,"href":"https:\/\/writeasync.net\/index.php?rest_route=\/wp\/v2\/posts\/5714\/revisions\/5718"}],"wp:attachment":[{"href":"https:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5714"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5714"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/writeasync.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5714"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}