Fluent async testing

Cellfish writes:

When .net 4.5 was first in public beta most test frameworks had a bug making it impossible to write async test methods. I wish they had never fixed that because I don’t think those help you write good tests.

I completely agree and as a rule I never write async Task-returning test methods. The drawback is that doing this carefully means making sure the Task is in the desired state, unpacking inner exceptions from an expected AggregateException, and other tedium.

Really, though, this is a general problem in unit testing, when assertions pile up and become more structural than logical. Consider this sample test for a fictional ItemCounter:

public void GivenTwoPrefixMatchesFindByPrefixShouldReturnBothItems()
{
    ItemCounter counter = new ItemCounter();
    counter.Increment("A1");
    counter.Increment("A2");
    counter.Increment("A2");

    IDictionary<string, int> counts = counter.FindByPrefix("A");

    Assert.AreEqual(2, counts.Count);
    Assert.IsTrue(counts.ContainsKey("A1"));
    Assert.AreEqual(1, counts["A1"]);
    Assert.IsTrue(counts.ContainsKey("A2"));
    Assert.AreEqual(2, counts["A2"]);
}

While this isn’t the worst possible example, it is indicative of a lot of day-to-day code I see. As the number of still logically-related but structurally different checks increases, these assertions become visual clutter and make tests more tedious and opaque than they need to be.

As a first-level refactoring you can always hide the clutter and take a page from the xUnit Patterns playbook, Custom Assertion:

public void GivenTwoPrefixMatchesFindByPrefixShouldReturnBothItems()
{
    ItemCounter counter = new ItemCounter();
    counter.Increment("A1");
    counter.Increment("A2");
    counter.Increment("A2");

    IDictionary<string, int> counts = counter.FindByPrefix("A");

    Assert.AreEqual(2, counts.Count);
    AssertHasItem(counts, "A1", 1);
    AssertHasItem(counts, "A2", 2);
}

private static void AssertHasItem<TKey, TValue>(IDictionary<TKey, TValue> items, TKey key, TValue value)
{
    Assert.IsTrue(items.ContainsKey(key));
    Assert.AreEqual(value, items[key]);
}

This helps readability quite a bit. However, the style is still not quite “fluent” (as in, “reads like a natural sentence”) and the error messages are no better than before. You can spot fix those problems, expand your custom assertions to be more generic, and so on. But it’s not going to be trivial (see e.g. Owen Pellegrin wrestle with this problem).

Thankfully, it turns out Dennis Doomen and others have already solved many of these problems in a low-friction reusable library.

Enter Fluent Assertions.

Using Fluent Assertions, we can rewrite our test like so:

// ...
using FluentAssertions;
// ...
public void GivenTwoPrefixMatchesFindByPrefixShouldReturnBothItems()
{
    ItemCounter counter = new ItemCounter();
    counter.Increment("A1");
    counter.Increment("A2");
    counter.Increment("A2");

    IDictionary<string, int> counts = counter.FindByPrefix("A");

    counts.Should().HaveCount(2).And.Contain("A1", 1).And.Contain("A2", 2);
}

The full assertion now fits comfortably on a single line, is easy to write, and it doesn’t get much more fluent than this. The error messages are also a breath of fresh air and give exact details of what went wrong:

• Wrong count? "Expected dictionary {[A1, 1]} to have 2 item(s), but found 1."
• Missing key? "Expected dictionary to contain value 2 at key "A2", but the key was not found."
• Wrong value? "Expected dictionary to contain value 1 at key "A1", but found 4."
• It works even if the dictionary itself was null: "Expected 2 item(s), but found <null>."

Of course, it does more than just dictionaries; see the Fluent Assertions documentation on GitHub for many more examples.

So, jumping back to async and Tasks — this is sadly one area that is currently lacking in Fluent Assertions. There is basic support for invoking Func<Task> delegates and asserting exception throwing behavior but this has a lot of the same drawbacks alluded to above (e.g. possible test hangs due to calling Wait() on a Task that will never complete).

Luckily there is a good extensibility experience so we can fill in the gaps and write async tests the way we want. Here is my attempt at doing just that: FluentSample on GitHub. It contains methods for dealing with Task in the style of Fluent Assertions, cutting down on boilerplate and improving readability.

Here are a few examples that show basic assertions on Task states:

[TestMethod]
public void DelayZeroShouldCompleteImmediately()
{
    Task task = Task.Delay(TimeSpan.Zero);

    task.Should().BeCompletedSuccessfully();
}

[TestMethod]
public void DelayInfiniteShouldNeverComplete()
{
    Task task = Task.Delay(Timeout.InfiniteTimeSpan);

    task.Should().BePending();
}

[TestMethod]
public void DelayInfiniteWithLaterCancellationShouldBeCanceled()
{
    using (CancellationTokenSource cts = new CancellationTokenSource())
    {
        Task task = Task.Delay(Timeout.InfiniteTimeSpan, cts.Token);

        cts.Cancel();

        task.Should().BeCanceled();
    }
}

Here is a more advanced example showing how to chain exception verification. Compare this to the “classic” way shown below and you can see the real value:

// Class under test:
public sealed class ParallelLoop
{
    private readonly Func<int, Task> doAsync;

    public ParallelLoop(Func<int, Task> doAsync)
    {
        this.doAsync = doAsync;
    }

    public Task RunAsync(int count)
    {
        List<Task> tasks = new List<Task>();
        for (int i = 0; i < count; ++i)
        {
            tasks.Add(this.doAsync(i));
        }

        return Task.WhenAll(tasks);
    }
}

// Helper factory method
private static Func<int, Task> GetThrowFunc()
{
    TaskCompletionSource<bool>[] t = new TaskCompletionSource<bool>[]
    {
        new TaskCompletionSource<bool>(),
        new TaskCompletionSource<bool>()
    };
    t[0].SetException(new InvalidOperationException("I:0"));
    t[1].SetException(new InvalidProgramException("I:1"));            
    return i => t[i].Task;
}

// Fluent way
[TestMethod]
public void RunAsyncAggregatesAllExceptionsAtEndFluent()
{
    ParallelLoop loop = new ParallelLoop(GetThrowFunc());

    Task task = loop.RunAsync(2);

    task.Should().BeFaulted().WithException<InvalidOperationException>().WithMessage("I:0");
    task.Should().BeFaulted().WithException<InvalidProgramException>().WithMessage("I:1");
}

// "Classic" way
[TestMethod]
public void RunAsyncAggregatesAllExceptionsAtEndClassic()
{
    ParallelLoop loop = new ParallelLoop(GetThrowFunc());

    Task task = loop.RunAsync(2);

    Assert.IsTrue(task.IsFaulted);
    AggregateException exception = task.Exception;
    Assert.IsNotNull(exception);
    Dictionary<string, Exception> exceptions = exception.InnerExceptions.ToDictionary(e => e.Message);
    Assert.IsTrue(exceptions.ContainsKey("I:0"));
    Assert.IsInstanceOfType(exceptions["I:0"], typeof(InvalidOperationException));
    Assert.IsTrue(exceptions.ContainsKey("I:1"));
    Assert.IsInstanceOfType(exceptions["I:1"], typeof(InvalidProgramException));
}

Feel free to try out and modify or extend FluentSample as you see fit. Happy async testing!