Watching a directory: composition and thread-safety

We have a testable DirectoryWatcher; now what? Given that the DirectoryWatcher can only watch files in a single directory, we can extend this to a whole directory tree. Perhaps the best way to achieve this is with composition (in the “has-a” sense, contrasted with inheritance’s “is-a”).

The basic idea is that our new class — let’s call it DirectoryTreeWatcher — will be initialized with a root directory. After that, the caller can ask to Subscribe to any file within the tree. Example:

DirectoryInfo root = new DirectoryInfo(@"X:\some\folder");
DirectoryTreeWatcher watcher = new DirecctoryTreeWatcher(root);
// watch for updates to `X:\some\folder\top.txt`:
IDisposable top = watcher.Subscribe(@"top.txt", /* callback */);
// watch for updates to `X:\some\folder\sub1\rel.txt`:
watcher.Subscribe(@"sub1\rel.txt", /* callback */);
// watch for updates to `X:\some\folder\sub2\sub3\deep.txt`:
watcher.Subscribe(@"sub2\sub3\deep.txt", /* callback */);
// stop watching for `...\top.txt`:
top.Dispose();
// stop watching everything
watcher.Dispose(); 

The scenario above demonstrates most of the requirements, namely:

• Any file in the tree can be watched; as with DirectoryWatcher, the file must exist.
• Individual file subscriptions can be disposed in order to stop watching.
• Disposing the watcher will stop all subscriptions.

Having learned our lesson last time, we will start with something testable. This means that instead of using a concrete class that directly constructs DirectoryWatcher instances, we will provide an abstract class DirectoryTreeWatcherBase with an abstract method Create which returns DirectoryWatcherBase. In the test implementation, we will use FakeDirectoryWatcher. We’ll get to the real implementation a bit later, after all the TDD fun.

With the similarity of the requirements to the original DirectoryWatcher[Base], most of the tests for DirectoryTreeWatcher[Base] will be very familiar. We have UpdateZeroSubscriptions, UpdateIrrelevantFileOneSubscription, etc. But there are a few new ones such as UpdateTwoFilesTwoSubscriptionsDifferentDir since we are allowed to specify relative paths.

After test driving, we have a general structure that seems to work. DirectoryTreeWatcherBase takes the input path, converts it to a full path in terms of the root, and then either looks up or creates a new DirectoryTreeWatcherBase instance on which to call Subscribe. However, we are now reaching a limitation of what microtests can tell us: there is a thread-safety issue here!

Imagine this sequence of calls:

1. Caller 1 on thread 1 invokes Subscribe with path “dir\file1.txt”
2. Caller 2 on thread 2 invokes Subscribe with path “dir\file2.txt”
3. Thread 1 reaches the GetOrAdd call and sees no existing instance
4. Thread 2 also reaches the GetOrAdd call and sees no existing instance
5. Thread 1 creates a new instance of DirectoryWatcherBase
6. Thread 2 also creates a new instance of DirectoryWatcherBase
7. Based on the ConcurrentDictionary logic, one of the instances above is discarded and only one is successfully added to the dictionary

The final state of the ConcurrentDictionary is consistent but we have just leaked a DirectoryWatcher! The observable behavior of the program will be correct (since the watcher will not have been subscribed to anything), but we will have to hope that the finalizer will kick in and clean up all the garbage we left behind. Surely there must be a better way.

Of course, there is! We will use the old trick of wrapping the constructor in a Lazy instance for our GetOrAdd callback. Now in the above sequence of calls the worst that will happen is that we create a Lazy instance that gets discarded later. Most importantly, only one DirectoryWatcher instance will ever be created per key because only one value will ever be returned after the GetOrAdd call. Internally, Lazy takes care of the thread safety so we still don’t need any explicit locking here. Here is resulting implementation: DirectoryTreeWatcher – update sample program; thread safe create.

To review, we have a DirectoryTreeWatcher and DirectoryWatcher. We have testability and composition, but not composability. We’ll expand on that point in the next post.