No time, no problem

Unit tests are supposed to be deterministic. If a test is called out as being nondeterministic, you can bet that it depends on time in some way. Any app big enough and with sufficient unit tests likely has something akin to ISystemClock, ITimer, etc. Entire libraries exist to remove direct dependencies on the system clock. Even here, I’ve written code to cope with unpredictable periodic execution to make testing fast and reliable. Time is a big deal!

This is why it’s good to see .NET tackle this problem, once(?) and for all(??). Welcome, TimeProvider! While the details are not finalized, the proposal has been floating around for a while now and is all but certain to land in some form in .NET 8. (The API has been reviewed and approved!)

There is even a plan to ship a manual time provider to allow deterministic testing. But some of us can’t wait that long. Our tests need to be fast and reliable now! As an exercise, I took the proposed API and made an interface (I don’t have the same restrictions as the .NET Runtime, so I’ll forgo the abstract class) with two implementations, FakeTimeProvider and RealTimeProvider. I then went ahead and TDD’d my way to a complete (enough) prototype. I know dozens of others must have done this already, but I tried to do so on my own without referring to other code.

Follow my journey here, via Git commits:

• DateTime
• Timestamp
• TimerImmediateNonPeriodic
• TimerDelayedNonPeriodic
• TimerDelayedPeriodic
• TimerChangeFromPeriodicToDisabled
• TimerChangeFromDisabledToPeriodic
• Refactor FakeTimeProvider – use LinkedList
• TwoAlternatingTimers
• TwoTimersFireManyDuringInterval
• MultipleTimersDifferentOrder
• WaitUntilCanceled
• WaitUntilDuration
• WaitRacesWithCanceled
• AsyncDispose
• Fix test instability

The first thing to note is that I intend to fulfill the same (single-threaded) unit tests for both the RealTimeProvider and FakeTimeProvider. Thus I use the Testcase Superclass (AKA Abstract Test Fixture) pattern. There is one abstract test class, TimeProviderTest with two abstract members:

public abstract class TimeProviderTest
{
    protected abstract ITimeProvider Init();

    protected abstract void Wait(ITimeProvider provider, TimeSpan duration);
}

The Init method creates the specific implementation under test and the Wait method does the correct wait step for the implementation. For the real implementation, we can just use Thread.Sleep to wait, but this will not work with the fake (you could say time is an illusion). The tests must therefore be somewhat constrained and not break out of their simulation sandbox so to speak. For example, this is how we can test a non-periodic timer which fires after some delay:

    [Fact]
    public void TimerDelayedNonPeriodic()
    {
        var evt = new CountdownEvent(1);
        ITimeProvider time = Init();

        using ITimer timer = time.CreateTimer(o => evt.Signal(int.Parse(o?.ToString() ?? "-1")), "1", TimeSpan.FromMilliseconds(50), TimeSpan.Zero);

        evt.Wait(TimeSpan.Zero).Should().BeFalse();

        Wait(time, TimeSpan.FromMilliseconds(100));

        evt.Wait(TimeSpan.Zero).Should().BeTrue();
    }

The test must wait for 100 milliseconds (real or virtual) and then check that the event has been signaled. This brings us to our second note — these tests are not 100% deterministic. It is totally possible (and expected) to make a FakeTimeProvider which never varies in its behavior. But the fact is that the RealTimeProvider will never be so — the nuances of thread scheduling and background execution would always get in the way. To that end, I wanted tests that could be predictable enough to pass for both implementations yet still have a degree of freedom to account for some acceptable jitter. Take a look at this test for disabling a periodic timer:

    [Fact]
    public void TimerChangeFromPeriodicToDisabled()
    {
        var evt = new CountdownEvent(10);
        ITimeProvider time = Init();

        using ITimer timer = time.CreateTimer(o => evt.Signal(int.Parse(o?.ToString() ?? "-1")), "1", TimeSpan.Zero, TimeSpan.FromMilliseconds(40));

        Wait(time, TimeSpan.FromMilliseconds(50));

        evt.CurrentCount.Should().Be(8);

        timer.Change(Timeout.InfiniteTimeSpan, Timeout.InfiniteTimeSpan).Should().BeTrue();

        Wait(time, TimeSpan.FromMilliseconds(50));

        evt.CurrentCount.Should().BeInRange(7, 8);

        Wait(time, TimeSpan.FromMilliseconds(50));

        evt.CurrentCount.Should().BeInRange(7, 8);
    }

The timing is tight enough that we cannot guarantee we won’t see an extra event fire between the change and the next wait. So the tests compromise a bit and allow a range of values. These tests could be strengthened even further to ensure the count never goes backwards (technically a value 7 followed by 8 would be allowed here). But I called this good enough for a prototype.

Given a TimeProvider API, calls to Task.Delay can be replaced with an extension method WaitAsync as follows:

    public static async Task WaitAsync(this ITimeProvider provider, TimeSpan duration, CancellationToken token = default)
    {
        // WARNING: THIS CODE HAS BUGS!
        var tcs = new TaskCompletionSource();
        using ITimer timer = provider.CreateTimer(o => ((TaskCompletionSource)o!).SetResult(), tcs, duration, TimeSpan.Zero);
        using CancellationTokenRegistration reg = token.Register((o, t) => ((TaskCompletionSource)o!).SetCanceled(t), tcs);
        await tcs.Task;
    }

But wait, there’s a bug! Do you see it? Well, if you run this test hundreds of times you may eventually find the issue(s):

    [Fact]
    public void WaitRacesWithCanceled()
    {
        ITimeProvider time = Init();
        using var cts = new CancellationTokenSource();

        using ITimer timer = time.CreateTimer(o => ((CancellationTokenSource)o!).Cancel(), cts, TimeSpan.FromMilliseconds(70), TimeSpan.Zero);
        Task task = time.WaitAsync(TimeSpan.FromMilliseconds(70), cts.Token);

        task.IsCompleted.Should().BeFalse();

        Wait(time, TimeSpan.FromMilliseconds(100));

        task.IsFaulted.Should().BeFalse();
        task.IsCompleted.Should().BeTrue();
    }

Here is one possible exception:

System.InvalidOperationException : An attempt was made to transition a task to a final state when it had already completed.

Stack Trace: 
    TaskCompletionSource.SetResult()
    <>c.<WaitAsync>b__3_0(Object o) line 20
...

Yes, that’s right, we’re using the unconditional SetResult/SetCanceled on TaskCompletionSource instead of the Try… variants.

As a final step, I ran the tests in a loop hundreds of times using this PowerShell snippet:

# for Debug
do { dotnet test } while ($LASTEXITCODE -eq 0)
# for Release
do { dotnet test -c Release } while ($LASTEXITCODE -eq 0)

That resulted in a few changes to loosen the assertions appropriately (e.g. using Should().BeInRange(...) instead of Should().Be(...)).

This prototype should be sufficient to use in unit tests which operate in a single-threaded fashion. To put it another way, the FakeTimeProvider is not thread-safe. I think this is a fair restriction, since the whole point is to allow as much determinism as possible. Trying to make multi-threaded “unit” tests but insisting on a fake timer may just be defeating the purpose somewhat.

We can’t be sure yet what shape this .NET proposal will take, so things may still shift before .NET 8 is finalized. Until then, the implementation above could be a good enough starting point. Only time will tell…