Operation throttle: part 1

A while back I discussed some design points relating to fixed concurrency, such as when attempting to ensure a constant number of simultaneous active requests throughout a load test. Today I will jump back to the single-threaded world and discuss how to ensure a fixed operation rate.

Imagine you are trying to determine the scalability of a specific server implementation. You would probably start by trying to send a certain number of requests over a period of time and measure how the server responds under load. For instance, you might choose 1000 per second as your target operation rate and maintain this for several minutes at a time. A simple (synchronous) implementation might look like this:

while (!done)
{
    Send();
    Thread.Sleep(1);
}

Assuming Send() takes a negligible amount of time and the Sleep is precise to the millisecond, you just might approach 1000 ops/sec. But wait — this is load testing where we should never merely assume, but actually test our theories. Let’s do so with BenchmarkDotNet.

Here is our benchmark:

    public class SleepPerf
    {
        private UdpClient client;

        [Params(1)]
        public int DelayMS;

        [GlobalSetup]
        public void Setup()
        {
            this.client = new UdpClient("localhost", 17);
        }

        [Benchmark]
        public void Run()
        {
            byte[] request = Encoding.ASCII.GetBytes("QOTD");
            this.client.Send(request, request.Length);
            Thread.Sleep(this.DelayMS);
        }

        [GlobalCleanup]
        public void Cleanup()
        {
            this.client.Close();
        }
    }

We’re using the QOTD protocol client as a simple example, just to do some amount of trivial work on each request.

But, look! The measured time per operation is far longer (relatively speaking) than we expected:

Method | DelayMS |     Mean |     Error |    StdDev |
------ |-------- |---------:|----------:|----------:|
   Run |       1 | 1.994 ms | 0.0016 ms | 0.0015 ms |

Maybe the Send() call takes too long? Commenting it out and rerunning the benchmark actually doesn’t make much of a difference.

Method | DelayMS |     Mean |     Error |    StdDev |
------ |-------- |---------:|----------:|----------:|
   Run |       1 | 1.990 ms | 0.0097 ms | 0.0091 ms |

How could this be? We asked to sleep for exactly one millisecond, but in practice it waited almost two. The answer is on the MSDN page for the Win32 Sleep function:

The system clock “ticks” at a constant rate. If dwMilliseconds is less than the resolution of the system clock, the thread may sleep for less than the specified length of time. If dwMilliseconds is greater than one tick but less than two, the wait can be anywhere between one and two ticks, and so on.

Let’s vary the DelayMS parameter and rerun the benchmark see if it becomes more accurate. Here are the results (with the Send call still commented out):

Method | DelayMS |      Mean |     Error |    StdDev |
------ |-------- |----------:|----------:|----------:|
   Run |       1 |  1.995 ms | 0.0027 ms | 0.0026 ms |
   Run |       2 |  2.994 ms | 0.0036 ms | 0.0033 ms |
   Run |       3 |  3.993 ms | 0.0037 ms | 0.0035 ms |
   Run |       5 |  5.989 ms | 0.0044 ms | 0.0041 ms |
   Run |      10 | 10.982 ms | 0.0075 ms | 0.0070 ms |
   Run |      15 | 15.971 ms | 0.0153 ms | 0.0143 ms |
   Run |      20 | 20.970 ms | 0.0145 ms | 0.0136 ms |
   Run |      30 | 30.954 ms | 0.0141 ms | 0.0132 ms |
   Run |      50 | 50.901 ms | 0.0464 ms | 0.0434 ms |

The observed sleep time gets closer to the expected as the time gets bigger, but it does so slowly — we’re still almost 1 ms late to wake up even when sleeping for 50 ms. If we want to be more strict about this operation rate, we will have to account for at least two important factors:

1. The time spent doing the work, i.e. the operation latency.
2. The actual time spent waiting vs. our expectation.

Trying to inline all this logic into our operation method would start to get unwieldy:

Stopwatch stopwatch = Stopwatch.StartNew();
TimeSpan delay = TimeSpan.Zero;
while (!done)
{
    TimeSpan mark1 = stopwatch.Elapsed;
    Send();
    TimeSpan mark2 = stopwatch.Elapsed;
    delay += TimeSpan.FromMilliseconds(1.0d) - mark2 + mark1;
    if (delay >= TimeSpan.FromMilliseconds(1.0d))
    {
        Thread.Sleep(delay);
        TimeSpan mark3 = stopwatch.Elapsed;
        delay -= mark3 - mark2;
    }
}

We are making three time marks to get an idea of how long each phase takes and using this to scale the actual waiting time appropriately. Obviously it is rather specific to the operation rate being used in the current code and I’m honestly not even sure the algorithm above is correct (though it “looks right”).

Clearly we could do with a better abstraction here. Next time, we will try to actually build one.