Let’s do DHCP: fuzzing

The DHCP server project is still alive and well. It’s definitely sample code, for demonstration purposes only, and other disclaimers. Even so, it is a technology that processes arbitrary network data, and we ought to worry about malicious input. To quote from RFC 1122, “assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect.”

If we want this DHCP server to be robust, we cannot allow read operations to result in any undesirable behavior (hangs, crashes, etc.). Note that we will not attempt to protect write operations in the same way; these do not stem from untrusted input, but rather programmer errors (“boneheaded exceptions” in Eric Lippert’s classification).

Let’s start by devising a strategy to detect bad read behavior. Ideally, we want a set of operations which are simple and exhaustive — guaranteed to touch all relevant parts of the read buffer. String formatting is a good choice to meet both criteria. As of now, there are four general areas where this would be relevant:

1. The entire DHCP message header
2. The entire DHCP options buffer
3. DHCP sub-options contained in a DHCP option (e.g. relay agent information)
4. Data contained within a DHCP sub-option (e.g. RADIUS attributes)

In practice, this results in TryFormat methods on these data types:

• DhcpMessageBuffer
• DhcpMessageBuffer+OptionsSequence
• DhcpOption
• DhcpOption+SubOptionsSequence
• DhcpSubOption
• DhcpRelayAgentInformationSubOption
• DhcpRelayAgentInformationSubOption+Sequence
• RadiusAttribute
• RadiusAttribute+Sequence

Whew… hard work, but it’s for a good cause. And now we’re going to get to the fun part: fuzzing! We will use fuzz testing (“providing invalid, unexpected, or random data as inputs”) to make sure that we caught all the possible read errors we might see in the wild world of the Internet.

Our fuzzing tool of choice is SharpFuzz. It works on .NET Core and has a good track record of finding serious bugs (even in .NET Core itself). The one catch is that it doesn’t work natively on Windows, but WSL 2 will do fine.

I wanted this to be a fair fight, so to speak, so before running SharpFuzz I attempted to find and fix any obvious buffer handling issues. There were quite a few ([1], [2], [3], [4], [5], [6], [7], [8], [9]) — so many, that I thought, “Surely, there won’t be anything left for SharpFuzz to do.” (Famous last words…)

SharpFuzz needs a simple test and at least one input file to do its magic (I chose the basic DHCP request binary file I had been using for other unit tests). The test program just exercises all possible formatting scenarios, assuming that no exceptions should ever be thrown:

    internal sealed class Program
    {
        private static Memory<byte> bytes;
        private static Memory<char> chars;
        private static Lazy<DhcpMessageBuffer> lazyBuffer;

        private static void Main(string[] args)
        {
            ushort size = 300;
            if (args.Length > 0)
            {
                size = ushort.Parse(args[0]);
            }

            bytes = new Memory<byte>(new byte[size]);
            chars = new Memory<char>(new char[65536]);
            lazyBuffer = new Lazy<DhcpMessageBuffer>(() => new DhcpMessageBuffer(bytes));
            Fuzzer.Run(Run);
        }

        private static void Run(Stream stream)
        {
            DhcpMessageBuffer buffer = lazyBuffer.Value;
            Span<char> destination = chars.Span;
            ushort length = (ushort)stream.Read(buffer.Span);
            buffer.Load(length);
            buffer.TryFormat(destination, out _);
            buffer.Options.TryFormat(destination, out _);
            foreach (DhcpOption option in buffer.Options)
            {
                option.RelayAgentInformation().TryFormat(destination, out _);
                option.SubOptions.TryFormat(destination, out _);
                foreach (DhcpSubOption subOption in option.SubOptions)
                {
                    subOption.RadiusAttributes().TryFormat(destination, out _);
                }
            }
        }
    }

Note that since SharpFuzz uses special binary instrumentation, you cannot invoke the instrumented code outside of a Fuzzer.Run call. (I found this out the hard way.)

Once I got past the initial issues and was able to launch the fuzzer, I thought there was a problem. It kept finishing very quickly with “Unable to communicate with fork server (OOM?)” even after setting the timeout, as suggested. I eventually discovered that this is what happens when the input causes the program to hang. Oh my, our first critical bug!

The problem ultimately boiled down to improper handling of option overloading. In retrospect, this is not surprising, given that this option directly influences how the internal buffers are traversed. A few fixes later ([1], [2]) plus a regression test ([3]), and the hang was solved.

Another run of the fuzzer thankfully did not show any more hangs. Instead, it found several more crashes. While they were reported as “uniq crash” results, there was really only one underlying bug. Again, option overloading was the culprit. This time it was a simple buffer overflow where it tried to read from a zero length data segment. In short order I was able to prepare a fix ([1]) and another regression test ([2]). After running the fuzzer again with the corrected program for several hours, I have yet to see another failure.

In summary, I highly recommend SharpFuzz for any .NET Core library that parses input. Despite the rough edges with Windows support and a learning curve, the results are worth it.