Testing static_assert (CMake)

One of the many innovations in C++11 was static_assert. This allowed, at long last, a custom error message at compile time. Sure, we’ve had the #error directive for a while, but that only works for conditions that one could evaluate at preprocessing time. On the contrary, static_assert can (with enough metaprogramming) see anything the compiler can see. Here is an example from WIL where static_assert provides a clear, unambiguous error (instead of an inscrutable template error):

// Blocking get_value_string template types that are not already specialized - this gives a much friendlier compiler error message
template <typename T>
T get_value_string(HKEY /*key*/, _In_opt_ PCWSTR /*subkey*/, _In_opt_ PCWSTR /*value_name*/)
{
    static_assert(sizeof(T) != sizeof(T), "Unsupported type for get_value_string");
}

Yes, static_assert is a great feature — one which may help us get ever-closer to the ideal of making illegal states unrepresentable. This is especially valuable for library development; we would much prefer that a misuse of a feature simply fail to compile rather than blow up at runtime.

Ah, but there is the conundrum. With runtime errors, we can easily write unit tests to assert the error behavior. But static_assert generates compiler errors; how are we supposed to test for those?

As it turns out, this question has come up more than a few times. There is even a related cottage industry of sorts on how to achieve unit testing at compile time. But let’s not stray too far from our goal here — what we would like is for a static_assert in our code to be explicitly tested just like any other feature.

By far the best treatment of this subject is by Dr. Roland Bock who gave the CppCon 2016 talk “How to test static_assert.” His solution is somewhat involved but works quite well for his use case in his sqlpp11 library. In short, he recommends a particular formulation of static_assert which provides predictable cross-platform behavior at the cost of a bit template magic (see for yourself in portable_static_assert.h). This is a great solution, but I actually want to explore the option he rejected at the front of his talk — I want to test my static_asserts with the compiler but in a way that fits better with my unit tests.

Right off the bat, we need to be clear that any “test” of static_assert is going to involve invoking the compiler. In particular, we need to provide the compiler with a program that should fail compilation, and that failure is success for the static_assert test. Further, we want to know that the failure is really the one we expect; it is pretty easy to write a program that fails for any one of a million reasons but not the reason that we care about.

Let’s start with a contrived example of a static assertion — maybe not so useful in real life but good enough for demonstration purposes:

std::string get_name(const char* raw);

template <typename T>
class Example
{
    static_assert(std::is_same_v<decltype(T::name), const char*>, "Type must have 'name' field of type `const char*`");

public:
    Example(const T& input) : m_name{ get_name(input.name) }
    {}

    std::string operator()() const
    {
        return "Hello, " + m_name + "!";
    }

private:
    std::string m_name;
};

In this imaginary library, we have a template class Example which assumes that the type T has a const char* name field. Regardless of the assertion, the code would fail to compile if we break this assumption (when trying to evaluate the input.name expression). But let’s say we just like our custom error message better ("Type must have 'name' field of type `const char*`"). (And yes, we could use C++20 concepts for this, but bear with me here.)

Here are some unit tests for this type, which are limited to testing runtime behavior only:

// stassert_test.cpp
// ...

struct MyVal
{
    const char* name{};
};

TEST(example_test, has_name)
{
    MyVal val{ .name = "world" };
    Example<MyVal> hello{val};

    ASSERT_EQ("Hello, world!", hello());
}

TEST(example_test, has_null_name)
{
    MyVal val{};
    Example<MyVal> hello{ val };

    ASSERT_EQ("Hello, <null>!", hello());
}

What would be nice is if I could add the following test and somehow validate that it fails while compiling:

struct WrongType
{
    std::string name{};
};

TEST(example_test, name_has_wrong_type)
{
    WrongType val{};

    Example<WrongType> hello{ val };
}

Of course, if I add this code, my entire test suite will fail to compile (for static_assert reasons). To get anywhere with this approach, I need conditional compilation. Let’s enlist the help of the much-maligned preprocessor:

#ifdef STASSERT_NAME_HAS_WRONG_TYPE
struct WrongType
{
    std::string name{};
};

TEST(example_test, name_has_wrong_type)
{
    WrongType val{};

    Example<WrongType> hello{ val };
}
#endif

Now I can have the rest of the test suite compile while hiding this code section, as long as no one has defined STASSERT_NAME_HAS_WRONG_TYPE. The next bit of machinery we need is some sort of compiler invocation that #defines this symbol and spits out the expected error message. CMake actually does have a try_compile feature that seems like it could be useful here; in my exploration of it, though, I could not get it to work in a simple enough way to recommend it.

Instead, let’s try to capture the compiler command that we already used to build our test suite. This is actually a feature of CMake, enabled with CMAKE_EXPORT_COMPILE_COMMANDS:

# put this at the top of the relevant CMakeLists.txt file
set(CMAKE_EXPORT_COMPILE_COMMANDS true)

All the relevant commands would thus be saved in a compile_commands.json file in the CMAKE_BINARY_DIR. But we need some other process to interpret the command and invoke the modified build step to trigger our static_assert. We’re going to need JSON parsing, text wrangling, and handling an external process — clearly a job for an advanced scripting system. If we want to be relatively cross-platform, we can write a PowerShell Core script-based cmdlet:

# Test-StaticAssert.ps1
[CmdletBinding()]
param(
	[Parameter(Mandatory=$true)]
	[string]$Symbol,
	[Parameter(Mandatory=$true)]
	[string]$Pattern,
	[Parameter(Mandatory=$true)]
	[string]$WorkingDirectory,
	[Parameter(Mandatory=$false)]
	[string]$CompileCommands = 'compile_commands.json',
	[Parameter(Mandatory=$false)]
	[string]$TestFile = 'stassert_test.cpp'
)

if (!(Test-Path $WorkingDirectory)) {
	throw "Working directory '$WorkingDirectory' not found"
}

Push-Location $WorkingDirectory
Write-Host "Set working directory '$WorkingDirectory'"

if (!(Test-Path $CompileCommands)) {
    throw "Compile commands file '$CompileCommands' not found"
}

$item = Get-Content $CompileCommands |
    ConvertFrom-Json | Where-Object { $_.file -like "*$TestFile"}

if (!$item) {
    throw "Could not find compile command for test file '$TestFile'"
}

$command = $item.command -replace '@\S+\.modmap ', ''
$command += " /D$Symbol"
Write-Host "Invoking compilation command: $command"
$output = Invoke-Expression $command
if ($LASTEXITCODE -eq 0) {
    throw "Compilation unexpectedly succeeded for '$Symbol'"
}

Write-Host "Output: $output"

$matched = $false
$output | Select-String -Pattern $Pattern | ForEach-Object {
    $matched = $true
    Write-Host "Found matching output line for '$Symbol': $_"
}

if (!$matched) {
    throw "Did not find output line for '$Symbol' matching '$Pattern'"
}

The script here reads the relevant command from the .json file, executes a modified version of it (filtering out some sort of module map response file that caused me trouble in practice), and tries to match the output with an expected pattern. Importantly, it fails if the compile is successful (remember, we expect a compilation error!) or if there is no matching output message.

The last step is to incorporate this “test” into the CMake build. Here is one way:

# our normal test suite
add_executable(stassert-test
    "test/stassert_test.cpp"
)

# . . .
# our static_assert test infra
function(add_stassert_test symbol err_pattern)
    set(test_script "${CMAKE_CURRENT_SOURCE_DIR}/test/Test-StaticAssert.ps1")
    string(CONCAT test_script_args
        " -Symbol ${symbol}"
        " -Pattern \"${err_pattern}\""
        " -WorkingDirectory ${CMAKE_BINARY_DIR}"
    )
    add_custom_command(TARGET stassert-test POST_BUILD
        COMMAND pwsh ${test_script} ${test_script_args}
    )
endfunction()

# the first static_assert test!
add_stassert_test(STASSERT_NAME_HAS_WRONG_TYPE "Type must have 'name' field of type `const char\\*`")

We are using add_custom_command to incorporate a post-build step for the test executable. The command simply invokes the script above based on a preprocessor symbol and the error pattern we expect.

Amazingly, this actually works, as shown in the CMake build output:

[47/48] Linking CXX executable src\stassert\stassert-test.exe
Set working directory '.../writeasync-cpp/out/build/x64-release'
Invoking compilation command: ...\VC\Tools\MSVC\1439~1.335\bin\Hostx64\x64\cl.exe  /nologo /TP -DGTEST_LINKED_AS_SHARED_LIBRARY=1 -I...\writeasync-cpp\src\stassert\inc -external:I...\writeasync-cpp\out\build\x64-release\vcpkg_installed\x64-windows\include -external:W0 /DWIN32 /D_WINDOWS /GR /EHsc /O2 /Ob2 /DNDEBUG -std:c++20 -MD /W4 -WX /Fosrc\stassert\CMakeFiles\stassert-test.dir\test\stassert_test.cpp.obj /FdTARGET_COMPILE_PDB /FS -c ...\writeasync-cpp\src\stassert\test\stassert_test.cpp /DSTASSERT_NAME_HAS_WRONG_TYPE
Output: stassert_test.cpp ...\writeasync-cpp\src\stassert\inc\example.h(13): error C2338: static_assert failed: 'Type must have 'name' field of type `const char*`' ...
Found matching output line for 'STASSERT_NAME_HAS_WRONG_TYPE': ...\writeasync-cpp\src\stassert\inc\example.h(13): error C2338: static_assert failed: 'Type must have 'name' field of type `const char*`'

Most importantly, if we “break” the static_assert, say, by erroneously allowing this code to compile, we get a clear error:

  Exception: ...\writeasync-cpp\src\stassert\test\Test-StaticAssert.ps1:38
  Line |
    38 |      throw "Compilation unexpectedly succeeded for '$Symbol'"
       |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
       | Compilation unexpectedly succeeded for 'STASSERT_NAME_HAS_WRONG_TYPE'

In fact, we can even use TDD now for static_assert! Let’s say we want to also assert that we cannot handle objects of size 16. First, add a new test at the end of the CMake file with the expected symbol and error:

add_stassert_test(STASSERT_NAME_HAS_WRONG_SIZE "Type must not have size of 16")

This should fail the build step (as expected) because we haven’t added any of this code yet, and the test program would still compile. Now, let’s add the test program code:

#ifdef STASSERT_NAME_HAS_WRONG_SIZE
struct WrongSize
{
    const char* name{};
    const char* extra{};
};

TEST(example_test, name_has_wrong_size)
{
    WrongSize val{};

    Example<WrongSize> hello{ val };
}
#endif

We are still in a failing state because the program as of now still compiles. Finally, we add the static_assert:

template <typename T>
class Example
{
    static_assert(std::is_same_v<decltype(T::name), const char*>, "Type must have 'name' field of type `const char*`");
    // new assert follows:
    static_assert(sizeof(T) != 16, "Type must not have size of 16");
// ...

At this point the build succeeds, which means our static_assert works (i.e., fails the build) as expected. Check out the GitHub project with all of the associated code here: https://github.com/brian-dot-net/writeasync-cpp/tree/main/src/stassert

One could certainly find many issues with this approach. For one, it is relatively heavyweight. A program with hundreds of static_assert statements would need hundreds of compiler invocations to be fully tested. But this is simply the reality — the only true test of static_assert is indeed a program that provably fails the assert. Perhaps it is not great to harvest build commands from CMake — maybe try_compile has better odds of working in a fully cross-platform way. It’s a cool hack, but is it practical?

Nevertheless, I thought this was an interesting jumping off point to TDD your asserts. What do you think?