Testing in the deploy loop

If you’re running a nightly test on main, you’ll probably start your day by looking at the nightly test results.

Start by checking if any properties failed.

curl --fail $AUTH \
"https://$ANTITHESIS_TENANT.antithesis.com/api/v0/runs/$RUN/properties?status=Failing"
# → {"data": [{"name": "Writes are durable", "status": "Failing",
#              "example_count": 0, "counterexample_count": 8, ...}],
#    "next_cursor": null}

snouty runs properties $RUN --failing

This is the programmatic equivalent of reading a triage report. By default you get every property’s status; here we filter down to just the failing ones, using the status param on the API or the --failing flag on Snouty.

One property, “Writes are durable,” is failing. To dig deeper, you can ask structured questions using event set search. For example, asking what kind of faults were happening just before the failure, were there any leader elections in close proximity of the failure, and more.

For instance, if you suspect the property failed due to your system not handling network failures well, you can check whether every assertion failure is closely preceded by a network partition.

curl --fail $AUTH -X POST \
https://$ANTITHESIS_TENANT.antithesis.com/api/v0/runs/$RUN/events/search \
-d '{"query": "matches({message: \"Writes are durable\", status: \"failing\"}).with_last({partition: contains({fault.name: \"partition\"})})"}'
# → each failing assertion Event, annotated with a `last_partition` field
#   pointing at the fault that preceded it (when one did)

snouty logs search $RUN --event-set \
'matches({message: "Writes are durable", status: "failing"}).with_last({partition: contains({fault.name: "partition"})})'

If every event is annotated with a last_partition, you can dig deeper into the logs around one of these and see how long it took for the system to fail after each network failure and more.

The Event sets reference covers the full set of query operations, and the event logs reference describes how Fault Injector events like network partitions are recorded.

From here you can decide the next steps to take. You found a bug, now what? tells you when to reach for the logs explorer, causality analysis, or the Multiverse Debugger.