> ## Types of faults > Fetch the complete documentation index at: https://antithesis.com/docs/llms.txt > Use this file to discover all available pages before exploring further. --- Antithesis tests your system by injecting faults — network disruptions, node failures, clock jumps, and more — while exploring different execution paths. The sections below describe each fault type and how it behaves. ## How faults are scoped Antithesis applies faults to *nodes* of your system. In Docker Compose, these nodes are individual containers; in Kubernetes, they are individual pods. Everything inside a node shares the same fate. If you put two services in a node, no network fault will ever happen between them. If you put them in separate units, they can be partitioned, lose their connection, or experience asymmetric loss between them. There are two exceptions to this: - Clock skips affect every node simultaneously because Antithesis simulates a shared wall clock, called virtual time, for the whole system. - Thread pausing acts at the thread level inside a unit, and requires Antithesis [instrumentation](/docs/reference/instrumentation/coverage_instrumentation/) to take effect. ## Network faults Network faults relate to problems with packet delivery. Incoming and outgoing packets are treated independently, so all network faults may be **asymmetric**. One direction may be disrupted while the other works normally. Network faults are enabled by default. ### Baseline latency This fault type simulates the normal variation in packet delivery time and a low baseline packet drop rate for all network traffic that flows through the Antithesis [environment](/docs/configuration/the_antithesis_environment/). ### Partitions This fault type simulates temporary failure of rack switches and similar hardware, which results in a network partition. Nodes in the network are temporarily separated into groups; nodes may communicate within their group, but communication between different groups is disrupted. The disruption takes one of three forms: `Stopped` (packets dropped entirely), `Slowed` (packets delayed with latency), or `Jammed` (packets queued for later delivery). Partitions can be configured by frequency, symmetry, number of partitioned groups, and duration. ### Clogs This fault type simulates failure of the networking stack of one or more machines, resulting in their inability to communicate with other machines. Affected nodes will have their packet delivery disrupted for inbound packets, outbound packets, or both. The disruption takes one of three forms: `Stopped` (packets dropped entirely), `Slowed` (packets delayed with latency), or `Jammed` (packets queued for later delivery). ### Restore This fault type simulates a network recovery event, which stops all ongoing network faults across all network links in the system. ## Node faults Node faults relate to the problems with individual nodes in the system, independent of the network links between them. To enable node faults, talk to your forward-deployed engineer. ### Node throttling This fault type limits one or more nodes' CPU resources or the bandwidth (in microseconds of processor time) they're permitted to use. This fault surfaces bugs that manifest under load. ### Node pause This fault type simulates a container frozen in place for a temporary period of time. It remains on the network but cannot process anything, so other nodes will see timeouts when trying to communicate with it. The node is restored to proper functioning after a set period of time has elapsed. ### Node termination This fault type simulates a node terminating either by shutting down gracefully (node stop) or by crashing (node kill). If the node is a container, it'll be restarted after a set period of time has elapsed. If the node is a pod, Antithesis cannot restart it and it's fully managed by Kubernetes. > **Note** > > If you've defined a restart policy in your Docker Compose configuration, the container may be restarted by docker-compose immediately, nullifying the Antithesis fault event. We recommend not defining a restart policy. Restarted nodes may change IP addresses upon rejoining the network. They may keep durably synced filesystem changes, or may lose all of their modified data and boot fresh from their image. If you are testing a data storage system, and want to configure customised durability behavior for particular directories, talk to your forward-deployed engineer. ## Other faults This section lists other fault types that are available. To enable thread pausing, you must [instrument your code](/docs/reference/instrumentation/coverage_instrumentation/). To enable clock faults or custom faults, talk to your forward-deployed engineer. ### Thread pausing This fault type simulates one individual thread taking an unexpectedly long time to execute. Threads are paused for small periods of time. These small pauses cause threads to interleave unexpectedly, and differently in each test run. For thread pausing to work, your system under test must be [instrumented](/docs/reference/instrumentation/coverage_instrumentation/). Thread pause events are not recorded in the log to avoid excessive logging. ### Clock skips This fault type simulates changes to the system clock by moving it forward or backwards. It affects all nodes equally. Jumps may be temporary (reversed after a set duration) or permanent, and multiple jumps stack cumulatively. These leaps in time surface bugs in processes that depend on small, consistent changes in system time. Calls to intrinsics -- such as `__rdtsc()` -- will not be affected. ### CPU modulation This fault type simulates running your entire cluster on different hardware. The clock speed of the simulated processor can be changed, and the relative performance of different low-level processor operations may also change (for example, changing the speed of particular instructions while keeping the same overall clock speed). This fault surfaces bugs by changing the order in which concurrent threads or processes execute and can trigger some of the same bugs as our [thread pausing](/docs/product/fault_injection/fault_types/#thread-pausing) fault without incurring the overhead of our instrumentation. CPU modulation events are not recorded in the log to avoid excessive logging. ### Custom You can create your own faults -- you know your software best! If you write a script or program that simulates some sort of a fault, you can have our fault injector invoke it at random intervals. Popular custom faults include connecting to administrative APIs and changing system configuration, triggering compaction or garbage-collection events, or forking troublesome background processes. Note that your custom fault can invoke our [SDKs](/docs/reference/sdk/) and, in particular, can use the SDK to draw random numbers which your custom fault can rely on. Your custom fault might therefore execute some default behavior 90% of the time, but a more pathological behavior 10% of the time. If you write a test action that simulates a fault, you could choose to define it as a custom fault, or you could have it be invoked [by your test template](/docs/product/test_templates/). A custom fault is more flexible, as you will be able to run it in any one of your nodes, whereas your workload will be in its own unit and will be less able to affect the rest of the system. However, making the workload in charge of invoking your custom fault will give you fine-grained control over when it runs. This might be better if you believe that the fault ought to be invoked in very specific circumstances or as part of a custom validation step. ## Overlapping faults Faults are scheduled independently and may overlap in time. The overlap behavior differs by fault type: - Network faults can overlap on the same target. When two network faults affect the same link at the same time, the more aggressive one wins for the duration of the overlap, e.g., a fault that drops packets on a link supersedes a slowdown on the same link. Once the more aggressive fault ends, the other resumes if its window has not yet expired. You will see overlapping network fault events emitted independently in the log; Antithesis does not collapse them into a single combined event. - Node faults do not overlap on the same target. Only one node fault can be active on a given container at a time. If subsequent node faults are scheduled against the same target, they're skipped while one is in progress. The skipped fault isn't logged. ## An example test run Here's what a 90 second window of a test run with all faults enabled might look like on a single execution branch from the fault injector's perspective. Times below use [virtual time](/docs/reference/glossary/#virtual-time) (`vtime`), Antithesis's simulated clock. - vtime 10 - test starts, system under test being set up, no active faults. - vtime 12 - `node pause` event targets node B (`max_duration: 8`); B becomes unresponsive on the network. - vtime 18 - `partition` event splits nodes `{A, C}` from node `{B, D}`; all cross-partition packets are dropped (`disruption_type: Stopped`, `max_duration: 10`). - vtime 20 - B's node pause expires. - vtime 28 - partition expires; cross-partition links restored. - vtime 40 - `skip` event jumps the clock forward by 65 ms (`max_duration: 1`). - vtime 41 - `skip` event jumps the clock back by 65 ms to reverse the offset. - vtime 75 - your workload runs `ANTITHESIS_STOP_FAULTS 10` before a liveness assertion; all faults pause until vtime 85. - vtime 85 - faults resume. Every one of those events appears in the JSON log with a virtual timestamp, and you can reconstruct the same picture from the log alone or look at the multiverse map by filtering fault injector events.