Event logs

Logs record events that happen during a test run. These can be general output logs, assertion logs, test command logs, or fault injection logs. This page is a reference for the fields you’ll see on those events.

You can access logs in the following places:

• In the triage report, events appear in the log viewer panel, grouped by category (your application’s output, assertion outcomes, fault injection events, and so on).
• In the Logs Explorer, events are searchable by category and field. Temporal queries (“preceded by” and “followed by”) let you correlate events across categories, for example an assertion failure that follows a fault. See the Logs Explorer search reference for the full list of searchable fields and operators.
• In API responses that return logs, each event is a JSON object with the fields described below.

General fields

Every event in the logs has source.name and moment.vtime fields, along with other relevant fields below:

Assertion logs

Assertion logs provide information about assertion events, such as the type of assertion and whether it passes or fails. Assertion logs have an antithesis_assert object alongside the general fields.

For example, the following log shows a “Database key values stay consistent” always assertion was hit and was found to be true:

{
  "antithesis_assert": {
    "assert_type": "always",
    "condition": true,
    "details": {
      "mismatch": null
    },
    "display_type": "Always",
    "hit": true,
    "id": "Database key values stay consistent",
    "location": {
      "begin_column": 0,
      "begin_line": 76,
      "class": "",
      "file": "/opt/antithesis/test/v1/main/consistency_test.py",
      "function": "<module>"
    },
    "message": "Database key values stay consistent",
    "must_hit": true
  },
  "source": {
    "container": "my-container",
    "name": "my-app",
    "pid": 49
  },
  "moment": {
    "input_hash": "...",
    "vtime": 25.227017703466117,
    "session_id": "..."
  }
}

Fields

Assertion logs may have the following fields inside the antithesis_assert object:

Whether an assertion is passing or failing is derived from assert_type, condition, and must_hit together. For example, an Always assertion is failing when condition is false, while an Unreachable assertion is failing whenever it’s hit at all. The Logs Explorer’s status search field applies this logic for you.

Test command logs

Test command logs provide extra information about test command events. Test command logs have a source.name of antithesis_test_composer.

For example, the following log shows that a parallel driver command has started:

{
  "command": "main/parallel_driver_generate_traffic.py",
  "command_type": "parallel_driver_",
  "container_id": "<CONTAINER_ID>",
  "IPT_bytes_out": 2113728,
  "source": {
    "name": "antithesis_test_composer",
    "pid": 1000
  },
  "started_task": "<CONTAINER_ID>_parallel_driver_generate_traffic.py",
  "task_status": "started",
  "tasks_len": "1",
  "moment": {
    "input_hash": "...",
    "vtime": 157.60831591766328,
    "session_id": "..."
  }
}

Fields

Test command logs may have the following fields alongside the standard source and moment fields:

Fault injector logs

Fault injector logs provide information about most fault injector events. The exceptions are thread pausing, CPU modulation and baseline latency faults. These events happen very frequently during a test run, so to prevent excessive logging they are not recorded in the logs.

Other types of faults are logged with a source.name of fault_injector, and a fault field.

The fault injector also logs pause and unpause events.

Fields

Fault injector logs may have the following fields inside the fault object:

Examples

These examples demonstrate the shape of the fault object for different types of fault.

Network partition

The following example shows a network partition fault log:

{
  "fault": {
    "affected_nodes": [
      "ALL"
    ],
    "details": {
      "asymmetric": true,
      "disruption_type": "Slowed",
      "drop_rate": 0,
      "latency": {
        "deviation": 1597.9999999999998,
        "mean": 1492.601977
      },
      "partitions": [["client-1", "client-2"], ["server", "client-3"]]
    },
    "max_duration": 0.183884736,
    "name": "partition",
    "type": "network"
  }
}

Nodes are split into groups by details.partitions. Network links between different groups experience the disruption type in details.disruption_type:

• Stopped: packets are dropped entirely.
• Slowed: packets are delayed with latency.
• Jammed: packets are “piled up” in a queue until a future deliver time.

Network links within the same group are not affected by this event (though they may be affected by an overlapping fault).

Network clog

The following example shows a network clog fault log:

{
  "fault": {
    "affected_nodes": ["server", "client-2"],
    "details": {
      "disruption_type": "Stopped"
    },
    "max_duration": 4.515860336,
    "name": "clog",
    "type": "network"
  }
}

Any connection to a node listed in affected_nodes experiences the details.disruption_type for max_duration:

• Stopped: packets are dropped entirely.
• Slowed: packets are delayed with latency.
• Jammed: packets are “piled up” in a queue until a future deliver time.

Network restore

The following example shows a network restore log:

{
    "fault":{
        "affected_nodes":["ALL"],
        "name":"restore",
        "type":"network"
    }
}

All ongoing network faults are stopped until new ones are scheduled in the future.

Node faults

The node fault types all share the same structure. The following example shows a node kill fault log:

{
    "fault":{
        "affected_nodes":["server-3"],
        "max_duration":1.7741677258234223,
        "name":"kill",
        "type":"node"
    }
}

The fault types are:

• kill / stop: The affected nodes are terminated (killed or stopped) for max_duration seconds, then restarted. Note that if the node is a Kubernetes pod, Antithesis cannot restart it because it’s fully managed by Kubernetes. In that case, max_duration will be 0 because Antithesis can’t control the restart.
• pause: The affected nodes are frozen in place for max_duration seconds. The container remains on the network but cannot process anything, so other containers will see timeouts when trying to communicate with it.
• throttle: The named node’s CPU is constrained for max_duration seconds.

Clock skips

System level clock skips move the time forward/backward by an offset (in seconds). The jump can be temporary or permanent:

• if the fault event contains a max_duration field, the offset is reversed after that duration
• if max_duration is missing, the offset is permanent. Clock offsets are cumulative. Each new skip event shifts the clock from wherever it already is.

The following example shows a clock skip fault log:

{
    "fault":{
        "affected_nodes":["ALL"],
        "details":{
            "offset":-0.11456344671067203
        },
        "max_duration":0.15177661326674713,
        "name":"skip",
        "type":"clock"
    }
}

Pause and unpause

Separately from fault event logs, the fault injector logs a status event whenever it is paused or unpaused. These have no fault field, so they don’t show up as fault events in the Logs Explorer’s fault injector category — but they do carry source.name: "fault_injector". Instead, search for them with the Log Explorer’s quiet search field:

The following log shows a pause event:

{
  "source": {
    "name": "fault_injector"
  },
  "info": {
    "message": "status",
    "details": {
      "paused": true
    }
  }
}