Designing for fixed concurrency

Consider a simple load test which is trying to maintain a constant stream of parallel requests against a server. That is, at any given moment the server should be handling, say, 100 concurrent requests. The exact number is not important, just that the level of concurrency is fixed at a certain target value. The point of the test is not to place undue stress on the server but rather keep it busy enough to exercise basic concurrency behavior. How would you write such a test?

Multiple client processes

One approach would be to host load test clients in separate processes. Each client process would send a single request at a time in a loop; you would thus need as many client processes as the maximum parallel request count. These client processes may be on one machine or many depending on the requirements.

The client implementation would be relatively simple, e.g.:

private static void RunClient(TestClient client, int totalCalls)
{
    // basic loop . . .
    for (int i = 0; i < totalCalls; ++i)
    {
        // generate request data . . .
        result = client.Request(data);

        // do something with result . . .
    }
}

This is a good approach when process isolation is desirable or required. Maybe the client has some shared process state which you want to avoid touching such as a network connection pool (which could affect your performance benchmark). It also makes sense to place clients on remote machines when the requests must originate from distinct locations. Perhaps your server has some flow control logic to throttle traffic from the same incoming network endpoint.

Multiple client threads

The previous points notwithstanding, a single-threaded client implementation is likely to be inefficient and will pose scalability problems. Processes are expensive and moving to a design that can use many threads will generally result in higher client throughput with lower resource utilization. If you decide to use, for instance, 20 threads per process, you can reach 100 parallel requests by using only five client processes in total.

At a first cut, this would be a simple addition to the above client code to launch many threads:

private static void RunClientThreads(TestClient client, int totalCallsPerThread, int totalThreads)
{
    Thread[] threads = new Thread[totalThreads];
    for (int i = 0; i < totalThreads; ++i)
    {
        threads[i] = new Thread(() => RunClient(client, totalCallsPerThread));
        threads[i].Start();
    }

    // Wait for client threads to complete . . .
    foreach (Thread thread in threads)
    {
        thread.Join();
    }
}

However, there is still a practical limitation on the number of threads you can create in a process without affecting overall system responsiveness and latency. There is also the fact that client requests tend to be I/O bound so the majority of the time will be spent simply waiting for responses.

Multiple async client workflows

As you might have guessed, a superior approach in many cases is to think in terms of async client workflows rather than writing blocking code using a fixed number of threads. This tends to provide the best balance between resource utilization, latency, and throughput. So instead of 100 physical threads, you could have 100 asynchronous loops which will only actively use a thread when initiating a request and handling a response.

In the next post, I will explore a few ways to build fixed concurrency workflows with simple asynchronous programming patterns.