Loopy tests

Loops are fundamental structures in almost every programming language (we’ll put aside APL for the time being). In unit tests, however, loops can be a problem. This is especially true of loops in the “Assert” section. For instance, consider this seemingly innocuous test of an integer range object:

void TestSubtractAllEvens()
{
    // Arrange
    const int N = 8;
    Range range = new Range(1, N);

    // Act
    for (int i = 1; i <= N / 2; ++i)
    {
        range.Subtract(new Range(2 * i, 2 * i));
    }

    // Assert
    // (Range implements IEnumerable<int>)
    foreach (int i in range)
    {
        Assert.IsTrue(i % 2 == 1, "{0} should be odd", i);
    }
}

With a bit of studying, we can easily work out what is happening in this test. The problem is that it is not actually doing its job! I am not saying the test is functionally incorrect — just that it will not reveal plausible defects in the system under test. Imagine these cases:

• Range mistakenly returned an empty sequence from its enumerator. The test would still pass (because no assertions would run).
• Range had some sort of off-by-one error (too few or too many elements). The test would still pass (because no explicit boundaries are checked).

You could come up with several spot fixes for these problems. Maybe you could use a library to replace explicit Assert loops with clearer and stricter collection checks. Maybe you add a counter and fail if zero items were found.

In my experience, there is usually a simpler solution — just get rid of the loop and be explicit! Not only is this immune to the common problems above, it improves readability and helps show what the object is really expected to do:

void TestSubtractAllEvens()
{
    // Arrange
    Range range = new Range(1, 8);

    // Act
    range.Subtract(new Range(2, 2));
    range.Subtract(new Range(4, 4));
    range.Subtract(new Range(6, 6));
    range.Subtract(new Range(8, 8));

    // Assert
    // (ToArray() from System.Linq)
    int[] items = range.ToArray();

    Assert.AreEqual(1, items[0]);
    Assert.AreEqual(3, items[1]);
    Assert.AreEqual(5, items[2]);
    Assert.AreEqual(7, items[3]);
}

Some would bristle at these nakedly unrolled loops, but there is clear value in being explicit in such behavioral specifications (being “WET” as Arlo Belshee would put it). If you still find yourself going to a “loopy” place — especially if you want more exhaustive verification than this sort of hand-rolled “spec by example” approach could scale to — you might be better off with property-based testing a la FsCheck. At least with that strategy the loops are hidden within the test engine and the generators, no longer taking up valuable conceptual real estate. After all, that is what it’s all about — getting as close as you can to the essence of expectations vs. reality vis-à-vis your code.