Findings

The first thing you’ll see in the triage report is the findings section. Findings are calls to action derived from how your system has changed from run to run. They tell you when you have a new bug, or have successfully fixed one. Findings are grouped by test run, so they essentially present a diff of your software’s behavior from one run to the next.

We derive this diff from the assertions you have made about how your system should behave. We call these assertions test properties. On each test run, Antithesis puts your system into a multitude of situations to see if those properties are ever violated, and then we diff the results of the latest run against the previous ones.

As always, you can follow along in our interactive example triage report.

Types of Findings

“New” Findings indicate a regression that wasn’t happening the last time Antithesis ran your software. The presence of a New Finding indicates that your software is not behaving as you expect.
“Resolved” entries indicate that an outstanding issue may have been fixed.
“Rare” entries are behaviors that we find infrequently – they’re at the limit of what your current setup can test for. More details.
“Ongoing” Findings are hidden behind a toggle by default, and they’re either:
1. Property violations that Antithesis encountered on the run immediately prior to this one. Antithesis recognizes that it encountered the same issue, so the new report won’t highlight the Finding as “New.” Or
2. A hard-to-find Finding that we didn’t encounter on this test run, but aren’t yet sure is fixed. More details.

Common Workflows

Catch up on one or more recent test runs

Read through the Findings section from the top down until you encounter entries you’re familiar with. Chances are, you’ll want to act on most of the Findings you’re seeing for the first time.

You can look into each Finding individually, or all the Findings from a single run as a group (since they may share the same root cause). Runs where Antithesis didn’t find anything are hidden behind a toggle.

“New” Findings indicate a regression that wasn’t happening the last time Antithesis ran your software.
- You should send New Findings to the relevant code committer or team to examine – there’s a link you can copy in the detail view. This gives them access to relevant debugging information, such as logs and other artifacts, and they’ll also be able to start a Multiverse Debugging session by copying the relevant moment from the logs.
- You may also want to file a ticket in your issue tracker.
“Resolved” entries suggest that an outstanding issue may have been fixed, and you may want to update any associated tickets you’ve filed. If you look into the “Resolved” Finding, you’ll see its full history in the detail view, but only the “Resolved” entry will show up in the main report.
“Rare” entries are behaviors that we find infrequently – they’re at the limit of what your current setup can test for. The most important takeaway when you see a “Rare” Finding is that you need to expand your testing to make the behavior more common, or the bug might slip into production.

Investigate and fix a new bug

Let’s say you’ve just received a link to a “New” Finding, or group of Findings, and you need to investigate.

The headline is a user-defined description of the property being tested. You (or someone else on your team) directly stated how your system ought to behave, and Antithesis ran your system in a multitude of situations to see if those properties were ever violated.

Immediately below is a history of the Finding. Red ticks show test runs on which your software did not behave as specified; green ticks show test runs where it did.

Below the history is a selection of examples where the property was violated, drawn from the most recent test run. Every Antithesis run tests many thousands of execution paths, with your software subjected to different faults and environmental conditions each time. Usually, there are many examples for a single Finding, because Antithesis found multiple scenarios in which the property failed.

Selecting an example from the list changes the set of available logs and other debugging artifacts displayed below.

If you need access to a reproduction, you can start a Multiverse Debugging session by copying the relevant moment from the logs. This gives you shell access to a reproduction of your system at the moment of failure.

Once you’ve implemented a fix, you should kick off another test run (or ask your teammate to do so), which will generate a new triage report.
In the unhappy circumstance that we see the issue again, Antithesis will recognize that it encountered the same regression, so the new report won’t highlight this as a “New” Finding – since it just found the behavior on the previous run! Instead, the Finding will be labeled “Ongoing.” “Ongoing” findings are hidden from view by default but can be toggled to display.

If we don’t encounter the issue, this could be for one of two reasons:
- Your fix worked, and we’re confident that the bug was successfully resolved. In this case the same Finding (or findings) you saw will show up as “Resolved” in the new report. Congratulations!
- We got lucky (or unlucky, depending on your worldview). Since we randomly explore test scenarios, we expect not to encounter some hard-to-find findings on every run. We avoid calling these findings “Resolved” until a statistically significant amount of testing has passed (we do this to avoid spamming you with “New” and “Resolved” updates). You can look into the Finding for more details.
Once a Finding is “Resolved,” it stops appearing in subsequent reports, unless Antithesis encounters it again. In this case, it will be displayed as “New” once more – and you should pay attention!

Disentangle a known issue

It’s possible for several distinct bugs to trigger the same test property. For example, two bugs might both make your program crash, and both will then show up under the “no crashes” property. By default, all violations of the same property will be grouped within the same Finding, but sometimes it can be useful to separate them into different findings – for example if they represent very different kinds of issues, or have different severity levels.

For example, if there is a particular segfault that’s considered low priority because it happens during a shutdown routine, splitting that off into its own “Ongoing” Finding will allow you to remain sensitive to any new segfaults, while continuing to track that one as a backburner task.

One way to handle this is to use SDK assertions to track these behaviors separately, but sometimes this isn’t possible due to constraints of architecture, code ownership, or resourcing. If so, you can instead have Antithesis organize how these events are displayed in emails and reports to maintain a high proportion of signal to noise.
To do this, please speak with your professional services consultant to describe the set of events you’d like to track separately. They’ll define a Manual Finding that encompasses these events. Manual Findings can be toggled off to reduce clutter in your Findings panel going forward.
The Manual Finding will appear as “New” when initially created, and becomes “Ongoing” on subsequent runs, just like any other Finding.
If Antithesis encounters failures outside this defined set, e.g. segfaults before a shutdown routine starts, they’ll still be displayed as “New” Findings.
It’s also possible that you’ll introduce a change that fixes these known failures. In that case, the Manual Finding will be displayed as “Resolved,” and you might want to check that this is an expected outcome.

Investigate a rare issue

Antithesis will mark a finding as “Rare,” if we estimate the probability of encountering this behavior during a test run of your usual duration is under a certain threshold, by default 33%.

Each time we encounter an occurrence of “Rare” behavior, we’ll flag it in the Findings section. You can explore each occurrence via the logs or restore them in Multiverse Debugging.

When a Finding is “Rare,” we may require many test runs to determine the behavior has stopped. The Findings detail view shows our current estimate of how many runs we’ll need (behind the “Show findability information” toggle).

A “Rare” Finding is a signal to improve your testing, or next time the bug might slip into production. There are three main ways to do this:

Expand your test templates to make this rare behavior common.
Go through the checklist in our section on finding more bugs, or contact our solutions team for help.
Increase your average amount of testing per day.

Check if a feature will cause a regression

Most Antithesis users run regular tests on their main build, but if you’re developing a feature, you might want to compare your exploratory branch to main.

Antithesis enables you to do this kind of “what-if” test by running a one-off, ephemeral test that won’t muddle your mainline test results.

However, Antithesis doesn’t automatically know which branch a test was run from, so you’ll need to tell us via the following steps:

Ensure the build from your new branch is uploaded, and specify its images in the antithesis.images param.
Kick off a test using your usual testing webhook and add the parameter antithesis.is_ephemeral: “true”. You’ll need to provide the same values for any custom parameters in your webhook call and antithesis.source if you use it. We’ve provided an example webhook call below.
You’ll receive a triage report as usual, but any behaviors introduced by your feature branch will be reported as “New” Findings, and any bugs your branch has fixed will show up as “Resolved.”
The next time you test, this run won’t show up in your Findings history. Antithesis will derive Findings by comparing your latest test to your last test of main.

curl --fail -u 'user:password' \ 
-X POST https://<tenant>.antithesis.com/api/v1/launch/<my_test_name> \ 
-d '{"params": { 
        <...your_parameters>, 
        "antithesis.config_image":"config_image_with_tag", 
        "antithesis.images":"feature_build_images_with_version_list", 
        "antithesis.report.recipients":"foo@email.com;bar@email.com", 
        "antithesis.is_ephemeral":"true" 
    }}'