An even faster hash table

Last week, in my continuing exploration of a C++ Letter Boxed solver, I had to build a faster hash table to improve performance. Today we’ll make it even faster, hoping to catch up to the performance of my .NET Core solver app.

Before making the code changes, I first needed to fix the benchmark code. The find operations were not really working the way they should have been. (Luckily the numbers didn’t change all that much.)

Now, let’s start optimizing! First, note that Hashtable has a single vector for its buckets. Each entry looks like this:

struct Entry
{
    TKey key_;
    TValue value_;
    bool occupied_;
};

This means that every time we resize the table, we have to reallocate and copy these keys and values again. Perhaps we can get a performance edge by using two vectors — one for buckets, which would just be a list of ints, and another for the entries which would never need to change other than appending totally new key-value pairs. The number in each bucket would refer to the index of the entry stored there.

Since a vector zero-fills by default, we should take advantage of that fact by treating a bucket value of zero as empty. Then we just need to be careful to one-index the entry list by setting its initial size to one.

Unfortunately, this approach forces us into a near total rewrite of the Hashtable class. But no matter — we have precious CPU cycles at stake! After a lot more logic errors than I care to admit, I was able to come up with these changes: Hashtable.h diff. Overall, I think this actually looks simpler, even if some of the loop patterns are duplicated. Where possible, I extracted helper functions to avoid literal copy/paste code. Also note that we do not need to keep track of any additional bookkeeping information about occupied buckets, so now the Entry is just a typedef:

typedef std::pair<TKey, TValue> Entry;

Given the structure of this new code, we can even explore a new collision scheme. Recall that we are using linear probing, where we try successive buckets B, B + 1, B + 2, B + 3, etc. In practice, quadratic probing can be much faster than linear probing (with a few caveats about load factor being 0.5 or less). An example quadratic probing scheme would use buckets B, B + 1, B + 4, B + 9, etc. Using the fact that each perfect square is the sum of consecutive odd numbers (1¹ = 1, 2² = 1 + 3, 3² = 1 + 3 + 5, and so on), we can implement it as follows:

void next_index(int& index, int c) const
{
    index += 1 + (2 * c);
    int n = static_cast<int>(buckets_.size());
    if (index >= n)
    {
        index = index % n;
    }
}

In this case, c represents the (zero-based) collision count: commit.

So now we have implementation A (original), implementation B (use separate buckets and entries vectors), and implementation C (B with quadratic probing). Did any of this make a difference? Let’s run the benchmark again:

Paying attention the 0.5 load factor numbers (since this is what we chose for the trie’s use case), we are clearly doing better with B. C is roughly the same, though drastically faster in some situations as the load factor increases. For now we’ll declare C the winner and use it as our solver implementation.

The final contest will be a head-to-head tourney of the managed and native solver implementations, to see if our theoretical improvements translate to actual wall clock time savings:

[000.000] Loading trie...
[000.059] Loaded 162309 words.
[000.060] Finding valid words...
[000.072] Found 754 valid words.
[000.072] Finding solutions...
MARKETPLACE-EARWIG
WIGWAM-MARKETPLACE
PRAGMATIC-CAKEWALK
[000.080] Done.

[000.000] Loading trie...
[000.049] Loaded 162309 words.
[000.049] Finding valid words...
[000.052] Found 754 valid words.
[000.052] Finding solutions...
MARKETPLACE-EARWIG
WIGWAM-MARKETPLACE
PRAGMATIC-CAKEWALK
[000.054] Done.

[000.000] Loading trie...
[000.056] Loaded 162309 words.
[000.057] Finding valid words...
[000.071] Found 1436 valid words.
[000.072] Finding solutions...
ROBAND-DEFECTING
 -- snipped 72 other solutions -- 
OBTECTED-DRAFTING
[000.120] Done.

[000.000] Loading trie...
[000.049] Loaded 162309 words.
[000.050] Finding valid words...
[000.053] Found 1436 valid words.
[000.053] Finding solutions...
BAITED-DEFORCING
 -- snipped 72 others solutions --
OBTECTED-DRAFTING
[000.092] Done.

C++ wins! It “only” took a custom data structure and several iterations of optimizations, but we got there in the end. Was it worth it? Sure, if all we care about is personal enrichment. Otherwise, you should look at your situation and heed the advice of Carlos Bueno:

Unless your optimizations are going to stick around long enough to pay for the time you spend making them plus the opportunity cost of not doing something else, it’s a net loss.