Simple multiverse debugging

The Multiverse Debugger lets you jump into any moment in an in-progress test run and run commands on your system. This gives you two superpowers.

You can time-travel to pull information from the past, present, or future of a reproduction of your bug, with extreme precision. This allows you to ask questions like:

What was the last network packet that was sent before your process died?
What was the internal state of your Raft algorithm at the moment a leader has been chosen?
What was the eventual health-check result of your system after minutes of quiescence?

You can also do destructive analysis, interacting with the system without fear of losing your reproduction. You can take actions like:

Cause a core dump by killing a key process.
Take down all secondary nodes and see if a suspected race condition still occurs.
First choose to step-over in your debugger and later choose to step-into.

You can use multiverse debugging to execute commands through the web app, with Snouty CLI, or with the API.

Prerequisites

To use multiverse debugging in the web app, you must set up SSO.

Get an in-progress test run

You need an in-progress Antithesis test run to use the Multiverse Debugger.

Snouty CLI and the API allow you to use an existing in-progress test run — this can be the one you want to debug a moment from, or a different one.

If you don’t have a suitable in-progress run, or you’re using the web app, you can launch a new debugging run from a completed run.

Launch a new run to debug from

With the web app

You can launch a run starting from the logs explorer or a triage report.

From the logs explorer

Select a log line in the log viewer on the right.
Click the Debug event button in the top right corner.
Optionally, update the pre-filled description of the debugging session and provide emails to notify when the session is ready.
Your debugging session will appear in the Debugging sessions list once it’s ready (usually in 10-30 minutes).

From the triage view

Select any example found.
Click the Get logs button. This will find the relevant log line in the Logs explorer.
Click the Debug event button in the top right corner.
Optionally, update the pre-filled description of the debugging session and provide emails to notify when the session is ready.
Your debugging session will appear in the Debugging sessions list once it’s ready (usually in 10-30 minutes).

With Snouty CLI

Use snouty debug to launch a debugging session for a given run ID, and a moment given as an input hash and a virtual time:

  snouty debug \
    --run-id <run-id> \
    --input-hash <input-hash> \
    --vtime <vtime> \
    --description "Optional description" \
    --recipients "team@example.com"

You can get the moment for an interesting event with the snouty runs events or snouty runs properties --detail commands.

Your debugging session will appear in the Debugging sessions list once it’s ready (usually in 10-30 minutes).

With the API

Call the Launch a debugging session endpoint with the run ID of the test run you want to debug, and a moment, given as an input hash and a virtual time:

curl --request POST \
  --url https://ANTITHESIS_TENANT.antithesis.com/api/v1/launch/debugging \
  --header 'authorization: Bearer ANTITHESIS_API_KEY' \
  --header 'content-type: application/json' \
  --data '{
  "params": {
    "antithesis.debugging.run_id": "<run-id>",
    "antithesis.debugging.input_hash": "<input-hash>",
    "antithesis.debugging.vtime": "<moment>",
    "antithesis.event_description": "Optional description",
    "antithesis.report.recipients": "team@example.com"
  }
}'

Replace ANTITHESIS_TENANT and ANTITHESIS_API_KEY with your tenant and API key.

You can get the moment for an interesting event with the List test run properties or Search events with event sets endpoints.

Your debugging session will appear in the Debugging sessions list once it’s ready (usually in 10-30 minutes).

Execute a command

The Multiverse Debugger allows you to run a bash script in any container at any point in time in the simulation.

From the web app

Select your session from the Debugging sessions tab, and select Open. This opens the Debug timeline view.

To execute a command:

Specify a moment in the test run (key in, or select a log line in the viewer).
Specify a container, or select (host) to run on the host machine.
Run your bash command.

If you’re using Kubernetes, select (host) and use kubectl to manage the cluster. For example, use kubectl get pods -A to list the pods, and kubectl exec <pod> -n <namespace> -c <container> -- <command> to run a command inside a pod.

With Snouty CLI

Use snouty runs exec with the run ID of your debugging session, a moment, given as an input hash and a virtual time, and the container to run the script in:

snouty runs exec <run_id> <input_hash> <vtime> --container <container> 'uname -a'

To run the script directly on the host machine instead of one of your containers, use --container "_ANTITHESIS_HOST".

If you’re using Kubernetes, set --container "_ANTITHESIS_HOST" and use kubectl to manage the cluster. For example, use kubectl get pods -A to list the pods, and kubectl exec <pod> -n <namespace> -c <container> -- <command> to run a command inside a pod.

You can also debug from a moment in a different run that is not currently in progress. To do that, pass the run id of the run the moment comes from with --source-run-id (or the session id with --source-session-id). You’ll need to wait for this moment to be “rewarmed” before debugging can start.

The command times out after 600 seconds by default; set --timeout to change this. Rewarming time doesn’t count toward the timeout.

You can get a moment with the snouty runs events or snouty runs properties --detail commands.

For multi-line scripts, pass it on stdin with a heredoc:

snouty runs exec <run_id> <input_hash> <vtime> --container <container> <<'EOF'
hpid=1275
grep NSpid /proc/$hpid/status
EOF

You can also redirect from a file: snouty runs exec <run_id> <input_hash> <vtime> --container <container> < script.sh.

With the API

Call the Execute a command endpoint with the run ID of your debugging session, a moment, given as an input hash and a virtual time, and the container to run the script in:

curl --request POST \
  --url https://ANTITHESIS_TENANT.antithesis.com/api/v0/runs/<run-id>/command \
  --header 'authorization: Bearer ANTITHESIS_API_KEY' \
  --header 'content-type: application/json' \
  --no-buffer \
  --data '{
  "moment": {
    "input_hash": "<input_hash>",
    "vtime": <vtime>
  },
  "container": "<container>",
  "script": "uname -a"
}'

Replace ANTITHESIS_TENANT and ANTITHESIS_API_KEY with your tenant and API key.

To run the script directly on the host machine instead of in a container, set "container": "_ANTITHESIS_HOST".

If you’re using Kubernetes, set "container": "_ANTITHESIS_HOST" and use kubectl to manage the cluster. For example, use kubectl get pods -A to list the pods, and kubectl exec <pod> -n <namespace> -c <container> -- <command> to run a command inside a pod.

You can also debug from a moment in a different run that is not currently in progress. To do that, pass the run id of the run the moment comes from with the source_run_id parameter (or the session id with source_session_id). You’ll need to wait for this moment to be “rewarmed” before debugging can start.

By default, the API waits 600 seconds for the command to return (this does not include rewarming time). To change the waiting time, set the timeout_seconds parameter.

Example commands

Here are some examples of useful bash commands that you can run.

Run a diagnostic script

Run a bash script (e.g. netstat) in a container you’re investigating.

netstat

Get a process id

If you’re here to debug a crashed process, get the pid of that process from the logs. Container processes have two pids, one found in the host machine and another in the container. The pid obtained from the logs is the host pid. For most debugging use-cases, you’ll want the container pid.

Map a host pid to its container pid

#!/usr/bin/env bash

hpid=1275

# Get all pids of the target process in the namespaces it participates in
# This command is run on the host machine
grep NSpid /proc/$hpid/status

Get the process id of a known process

#!/usr/bin/env bash

# if you know the process name
process_name="slirp4netns"
ps --format pid --no-headers -C $process_name | head -n 1

# If you have `psgrep` inside of your target container, that could be more ergonomic
pgrep $process_name

Get the container id from a host pid

If you’re here to investigate a process crash, here’s how to get the container id that was running the crashed process.

Grab the host pid of the crashed process from the logs.
Enter a time that’s a few seconds before that process crash.
Select the (host) container.
Run the following script.

#!/usr/bin/env bash

hpid=1275
cat /proc/$hpid/cgroup | grep -o 'libpod-[^.]*' /proc/$hpid/cgroup | sed 's/libpod-//'

# Containers are placed into distinct cgroups, so you can get the container id from the process's cgroup information
# The output will look similar to
# 0::/machine.slice/libpod-{CONTAINER_ID}.scope
# The important part is: "libpod-{CONTAINER_ID}.scope" to get the container id

# Optional: Inspect the container to find the image name, container name
# podman inspect <container-id>

Extract a file

In the web app, the Extract file button lets you print the output of a process or command with one click.

Select your session from the Debugging sessions tab, and select Open. This opens the Debug timeline view.

To extract a file:

Specify a moment in the test run (key in, or select a log line in the viewer).
Specify a container, or select (host) to run on the host machine.
Select Extract a file, and provide the path of the file.

If the file you want to extract already exists in a container, just specify the path to it.

If you want to extract the output of a process or command ran in the MVD, you can chain the two steps by writing a bash script and providing the path to the file to extract and run them together. Running them separately will not work.

Recover suppressed logs

If you used Retroactive logging, the verbose log lines it suppressed during fuzzing are recoverable. The UI help text includes instructions for retrieving them.

In the log viewer, right-click the line you’re interested in and choose Recover filtered logs. Antithesis replays the 10 seconds preceding that line with capture switched on, and the recovered lines appear inline in the timeline.

Recovery takes a moment to complete, since Antithesis is replaying that window of the simulation. If nothing was suppressed in that window, we’ll tell you rather than adding new lines.

Automate your workflow

Any set of debugging steps that you perform in one of your multiverse debugging sessions can be converted into a script that automatically runs in every one of your tests when a certain kind of bug is found. The results, output, artifacts, etc. are automatically attached to your triage report in the bug details section.

Suppose you often want to know what’s happening in your network three seconds before a segmentation fault. Our triage report can autonomously gather this information for you and have it ready when you open the report. You can still use the Multiverse Debugger for the truly hard cases where your scripted steps don’t work.

To get started with custom artifacts, contact us at support@antithesis.com.

Advanced mode

The Multiverse Debugger offers an advanced mode that uses our browser-based interactive notebook. It allows you to go beyond the capabilities here, but is also a little more complicated to use.

For instance, advanced mode lets you ask a counterfactual, or run an external debugger inside our simulation.

To launch advanced mode, open a debugging session and select “Advanced mode” from the ⋮ menu in the top right.