Controlling faults

The Antithesis fault injector continuously disrupts the environment that your software runs in. It can disrupt the network between your nodes, with added latency, dropped packets, partitions and clogs. It can also pause, throttle or terminate the nodes themselves, skip the clock forwards and backwards, and pause individual threads.

You can choose which of these faults Antithesis injects by setting a fault tolerance level. You can also exclude specified containers from faults. See Launching tests for how to set these parameters in the web app, Snouty CLI, or the API.

See Types of faults for more details on each fault type, and on how faults overlap.

Fault tolerance levels

The following fault tolerance levels are available:

Level	Faults injected
`none`	No faults.
`resilient`	The recommended level for `basic_test`. Faults that degrade the system without stopping any node or network link: baseline latency, `Slowed` or `Jammed` clogs, node pauses, CPU modulation, thread pausing, and clock skips. Packets may be dropped intermittently but no link drops packets persistently.
`high_availability`	Everything in `resilient`, plus partitions, `Stopped` clogs, and node termination. `Stopped` faults can drop packets persistently, including in one direction only.
`legacy`	The default level for `basic_test`. All network faults and thread pausing are enabled, other faults are disabled. Not recommended except for backwards compatibility with a custom notebook.

We recommend building up to high_availability in stages:

Run basic_test with none until your system starts reliably with no faults. Enable setup mode to iterate on setup with a setup assistant in the web app.
Run basic_test with resilient to find bugs while your system is slowed down and disrupted.
If your system is designed to keep working when a node goes down or the network splits, run basic_test with high_availability.

For thread pausing to take effect, you need to instrument your code. To add custom faults, talk to your forward-deployed engineer.

Excluding containers from faults

By default, no containers are excluded from fault injection.

To exclude containers, use the antithesis.fault_exclusions parameter. This parameter is a regular expression that uses Rust regex syntax. It matches container names, or pod names on Kubernetes. For example, this pattern excludes two containers named raft-client and raft-monitor:

^(raft-client|raft-monitor)$

Thread pausing applies to the whole machine rather than to individual containers, so containers cannot be excluded from this fault type.

Pausing faults

By default, faults are injected throughout the test timeline, interleaved randomly with your workload.

There are two ways to pause fault injection:

To pause fault injection during a test timeline, call the ANTITHESIS_STOP_FAULTS binary from your test command or workload to request a quiet period for a given duration. Antithesis injects this binary on every container or pod path. Antithesis will pause faults, restore killed containers or pods and not inject any faults for the requested duration.
To pause fault injection at the end of a test timeline, use an eventually or finally test command. These commands create a terminal pause at the end of a timeline, giving the system under test time to recover before final validation checks.

See Pausing faults for more details.

Understanding what faults occurred

Information about faults appears in three places after a run completes:

In the Triage report, as fault-injection events, next to your application’s logs and any assertion outcomes.
In the Logs Explorer, filterable by the fault injector category. The fault logs reference explains the logging format.
In the API responses that return logs, as events whose source.name is fault_injector.