Use Sometimes assertions

You can get more out of your Antithesis testing by adding sometimes assertions to your code. Sometimes assertions are a better version of traditional code coverage metrics and give critical insight into whether your Antithesis testing is effective. They also improve the overall quality of your testing by helping test unusual cases more, which finds bugs traditional methods would miss.

Two types of assertions

Think of some conventional assertions you are familiar with:

assert x > 0;
assert y == 1;
assertTrue(condition);

These are what we call always assertions: You are interested in these things always being true. These are the normal kind of assertions that almost every programming language or framework provides. This kind of assertion is great! They help guard against unexpected or invalid program states, escalate asymptomatic bugs into test failures, and prevent silent data corruption in production.

You may be less familiar with the second type of assertion, which at Antithesis we call Sometimes Assertions. Just as an always assertion asserts that something is always true, a Sometimes Assertion asserts that something is sometimes true! Here are some examples:

assertSometimes x < 1;
assertSometimes y == 1;
assertSometimes(condition);

Do these seem a little bit strange? At first glance it seems like these assertions are pointless – after all, if the condition is true, then the assertion will pass (we asserted it was sometimes true), and if the condition is false, then we can’t really say whether it’s passed or failed. Maybe it will still pass the next time – after all, we didn’t say it was always true!

The key is not to think of a single, scripted test run through your software. Imagine instead running your software for a very long time, or running an entire suite or ensemble of tests. Imagine you’ve added a sometimes assertion to your code, and it never, over all these many thousands of test runs, gets triggered. What can you deduce from that? There are two possibilities: either you’re wrong, and the program state designated by the sometimes assertion is actually unreachable, or your tests are too weak to ever reach it. Either one could be important.

These are the two main reasons that sometimes assertions are useful, whether or not you’re using Antithesis:

1. They help you detect whether code is reachable. This can be very important! Imagine introducing a bug into your website that makes it impossible to complete the checkout flow, or a bug in your game that makes it impossible to win. Reachability of program states can be a vital correctness property.

2. They help you to assess whether your tests are comprehensive and effective. If you’re sure that a sometimes assertion is reachable, and it’s reached in production, but your tests never trigger it, then that’s a sign that your testing is leaving important areas uncovered.

By the way, do these funny assertions sound more familiar now? That’s right, you’ve seen them before, at least if you’ve ever looked at code coverage.

A generalization of code coverage

Coverage tools, which tell you which lines of code in your program are reached, are just a more restrictive kind of sometimes assertion. To be precise, they’re equivalent to adding assertSometimes(true) to every line of your program. Code coverage can be a good start to assessing the reachability of your code and the comprehensiveness of your tests, but it has two major weaknesses compared to the full power of sometimes assertions.

Weakness #1: Code coverage is not sensitive enough. All it tells you is that a particular location was reached. But it tells you nothing about, for instance, the values of variables in memory or the parameters a function was invoked with. Another way to put it is that code coverage only covers locations, while sometimes assertions cover situations.

Weakness #2: Code coverage is too sensitive. Not every line of code in your program is equal. In fact, the vast majority of it is probably unimportant – either it runs all the time, or it’s an unused part of some third-party library dependency and never runs at all. There are too many lines of code in your program for you to consider whether or not each one was covered. People respond to this deluge by computing coverage percentages, but these are a coarse tool that can hide information about whether particular vital locations were reached.

Sometimes assertions directly address both of these weaknesses. Instead of coverage merely informing you that a function was reached, you can instead check whether it was ever reached with a negative value passed for one of the parameters. And every sometimes assertion is placed deliberately, by a programmer. Whenever you hit (or don’t hit) a sometimes assertion, you know it was a situation that some human being at some point thought was important. Moreover, the overall number of them is small enough that you can look at individual examples if a code change or test change means that some are no longer hit.

How Antithesis uses sometimes assertions

Now we’ve seen what sometimes assertions are, and why they’re more powerful than generic code coverage. But when you test in Antithesis, sometimes assertions get even better. We can get the two usual benefits of assessing reachability, and assessing the strength of your tests. But within our simulation they also gain two additional benefits.

First, adding a sometimes assertion is equivalent to adding a special log line in your software, in a format that our system understands. Since Antithesis supports deterministically replaying your entire software system, we can treat these “log lines” as breakpoints and replay from the moment a sometimes assertion triggers. We can then optionally go a little forward or a little backward in time, and gather logs, core dumps, data files, or any other artifacts of interest from any of the processes running within Antithesis.

Second, we know that programmers usually add sometimes assertions to especially interesting, rare, or tricky areas of their code. Therefore, if the Antithesis platform encounters one during the course of testing, it will use its deterministic replay capabilities to treat that sometimes assertion as a “checkpoint” and restart a percentage of its tests from that point.

To see why this is so powerful, consider a situation where a bug requires two independent, rare events to occur in succession in order to trigger. Perhaps there first needs to be a failover in the system, and then there needs to be a database transaction rollback, and only if both these things occur is the bug discovered. If both events are rare, for instance if they both have a 1/1000 chance of happening, then we would normally have to wait for a million trials before they both occur and the bug is discovered.

Now imagine that you’ve added a sometimes assertion to the failover code. This is a very natural thing to do, since after all it’s a rare and tricky situation, and you want to confirm that the tests are covering it. Antithesis will encounter that assertion after a thousand trials on average. But once it’s seen the assertion, it will restart a significant number of the tests from that point. This amplifies its probability, since that subset of tests won’t need to stumble upon the failover again, as it’s already happened in their past. This means you will be much more likely to find this one in a million bug.

To summarize: in general, you can use sometimes assertions to determine whether program states are reachable and assess the quality of your tests. Within Antithesis, you gain two additional benefits: generalized breakpoints and checkpoints that increase the effectiveness of your tests.

How to start using sometimes assertions

The easiest way to get started with sometimes assertions is to integrate with the Antithesis SDK in your language of choice. Our SDK provides convenient methods for writing sometimes assertions with minimal performance overhead. Even better – the assertions are easily disabled in production. But if you leave them enabled when running outside of Antithesis, they will produce an open-format coverage report that is digestible with third-party tools.

If you can’t use our SDK, whether for policy reasons or because it isn’t available in your language of choice, you can still use sometimes assertions! Antithesis provides a fallback SDK guide for this purpose.