Rigorously testing distributed systems is arguably as hard as designing and implementing them in the first place. Earlier this year, we introduced a suite of skills that enable agents to automatically infer high-level properties from a codebase, and construct an Antithesis harness that checks those safety invariants under fault injection.

But how can we tell if the test that the agent (or the human!) wrote is robust enough to find subtle bugs in the future?

This is the classic test oracle problem. To help overcome it, we’ve just developed a new agent skill to perform mutation testing, injecting artificial bugs to validate or refine Antithesis tests until properties are falsifiable (i.e., the test should fail when a property-violating bug is present).

We’ve been using rqlite, an open-source fault-tolerant database built on top of SQLite, to test and demo our skills, because rqlite is incredibly well-designed, well-tested, and thoroughly documented.

rqlite replicates SQLite database files across a cluster of nodes, using Raft consensus to agree on a log of writes that describe the state of the database. All writes go through an elected leader node, the followers replicate, and reads can be served quickly unless linearizability is desired.

The interactive diagram below summarizes the invariants that our agent inferred, and how our tests held up under mutation testing — the rest of the post unpacks what’s happening and how we built this.

Building a test suite

We started by using the existing Antithesis skills to set up a testing harness for rqlite. Our agent used antithesis-research to infer a number of conceptual invariants for rqlite from the source code and docs. We then had it write a harness using the antithesis-workload skill to check the most straightforward distributed safety properties.

The workloads are Go programs that perform property-based testing, such as writing arbitrary rows with unique keys, recording what the cluster acknowledges, and then reading those keys back from replicas to validate durability, integrity, or consistency. The agent also set up the test system, a cluster of three voter nodes plus one read-replica in a docker-compose file.

Finally, it launched tests with Antithesis, which randomly schedules the test workload under network faults and node crashes to validate the safety properties. If anything fails, agents can pull up logs and replay the history to triage the issue, such as identifying a bug or addressing a false positive by refining the test itself.

Now, let’s say the report comes back green — every property assertion passes. How do we know that the test suite is actually valuable? If there’s a bug in the system today (or in the future), are the workloads and properties strong enough to detect it, or will they silently ignore any flaws?

Overcoming the oracle problem

So we turned our attention to mutation testing—an idea first introduced almost 50 years ago to craft high-quality test suites. The basic premise is that a test is valuable if it fails when the software is modified to intentionally behave incorrectly. For simple unit test frameworks, there exist many tools for systematically mutating a program under test by making syntactic code changes (for example, replacing an expression a + b with a - b, or replacing return True with return False) and then validating that the test fails, often referred to as having killed the mutant. If the test still passes, it means the deviation cannot be detected, and the mutant is said to survive, prompting a refinement of the test suite.

For end-to-end testing of distributed systems, however, simple syntactic changes are not good mutations because they mostly lead to run-time panics and short-circuit the validation of distributed safety properties such as consensus or durability.

Our new skill, called antithesis-mutation-testing, teaches agents to understand the source code and inject artificial bugs that induce subtle race conditions where most of the time everything works fine but bad things can happen in edge cases. This is exactly the kind of bug that Antithesis excels at uncovering, and we want to make sure that AI-generated test suites are able to detect these.

Injecting deep distributed system bugs

For example, in rqlite, consider the safety property fsm-index-monotonic, which asserts that “a node’s applied-log index never moves backwards while the process stays up”; that is, if a node’s view of the database is up-to-date as of the N-th write operation, it should not subsequently regress to a state that was valid before the N-th update. To validate that our test can catch violations of this property, our agent deliberately breaks it in a subtle part of the code: restoring snapshots.

Here’s how this works in normal rqlite: when a replica falls behind the leader node long enough (e.g., due to a network partition) it can subsequently catch up by downloading a full snapshot of the database state from the leader node. The receiver stores the snapshot file on disk and then swaps out the stale database for the updated one. The snapshot file is kept on disk to support recovery on future reboots. So, rqlite must handle a situation when there are multiple snapshot files on disk: an older snapshot that was already restored (or perhaps a previously-aborted snapshot-restore attempt) and a new snapshot that was just received.

Using the mutation testing skill, our AI agent injects a subtle deviation where the snapshot file used in the restoration operation is an older snapshot file instead of the newest one — see the patch below. Note that the source-code modification may look like an obvious bug; our objective is not to disguise it. Rather, the point is to stress the ability of our end-to-end tests to exercise deep logic — in this case, a situation where multiple network partitions lead to correspondingly multiple snapshot-restore operations. With the mutant patch applied, a replica’s use of an “older” snapshot can cause a regression of the database state to a previous version, and we want our tests to go red with a big error message when this happens!

// snapshot/state.go

@@ -18,7 +18,7 @@ func LatestIndexTerm(dir string) (uint64, uint64, error) {
   if err != nil {
     return 0, 0, err
   }
-  newest, ok := sset.Newest()
+  newest, ok := sset.Oldest()
   if !ok {
     return 0, 0, nil
   }

Note that these mutant patches are ephemeral and only used for validating Antithesis tests. The mutants are intended to be discarded once we are confident in the tests’ ability to catch these kinds of bugs.

How our skill helps agents improve tests

Starting with the workload for the 13 safety properties our agent extracted in the first pass, we ran the mutation testing skill with a budget of at most 100 test runs and stepped back.

The agent proceeded to invent its own mutants: it built forks of rqlite with these injected bugs, then ran Antithesis tests on them, and decided if the mutant falsified the safety property (i.e., the test actually caught artificial bugs that violated the property).

If the test results indicated that the property was not falsified (i.e. the test passed despite the bug we injected), then the agent autonomously iterated further by refining one or more of the mutant, workload, assertion, or test configuration.

After 19 mutant injections across 46 runs of 15-60 minutes each (~24 hours cumulative fuzzing duration), the agent ended the mutation testing campaign reporting that 11 of 13 safety properties were successfully falsified, thereby building our confidence that our test setup would actually safeguard them.

However, 2 other properties couldn’t be falsified, indicating that either the test setup needs further reinforcement (e.g., by configuring the endpoint to enable disk faults, as in the case of acked-write-survives-total-cluster-kill), or the conceptual properties themselves should be revisited by an expert.

Why you should do this yourself

Mutation testing is a powerful but under-appreciated technique, and we believe the takeaways from this post apply even beyond Antithesis. We now live in a world where agents are increasingly being used not only to produce implementations but also to validate them, some going as far as to vibe formal verification. Your confidence in your system is only strong as the specifications or properties against which your correctness checks are applied. And the best way to gain trust in translations of vaguely defined intent is through falsification. Go on, break your software, and see what happens.