The Environment and its utilities#

Exploring the Universe#

In Exploring the multiverse, we explain that a moment is a single point in the tree of possible timelines, but we don’t talk much about what’s inside a moment.

The Environment is the virtual system where we run your containers – it’s the universe in some branch of the testing multiverse.

You can read more about the specifics of the system in our Antithesis Environment documentation, but for our purposes, you can think of it as a Linux system that’s been customized for testing.

The Environment object#

In each moment, the Environment is in some state. There are probably containers running your software, there may be Antithesis faults in progress, and there are certainly some background services supporting the system. Inside a multiverse debugging session, the Environment object represents the system as a whole.

When you begin your session, the Environment is already defined for you.

// `environment` here is your Environment
[environment, moment] = prepare_multiverse_debugging()

As a rule of thumb, we’ll typically use the Environment when we’re inspecting and configuring the system as a whole.

Environment event sets#

Sometimes, when you’re diagnosing a problem, it’s helpful to see what’s going on with the system as a whole. System logs are available through the Environment object as curated event sets.

Note

When events are the result of readable (non-binary) program output, an event is sent each time the output contains a newline character.

  • environment.containers.events: output from all of the containers

  • environment.containers.meta_events: changes to containers (create, start, stop, etc) as seen by the container management system

  • environment.sdk.all_events: all of the events that have been processed through the Antithesis SDK

  • environment.sdk.assertions: assertions defined with the Antithesis SDK which have been encountered

  • environment.events: the main event source, including the sources above

  • environment.fault_injector.events: cumulative statistics and other logs from the Antithesis fault injector (Warning: we have not yet invested in making this easy to understand, and the format is likely to change!)

  • environment.background_monitor.events: raw output from the Antithesis system utilization monitor (Warning: this is a new feature and its format may be unusually unstable!)

  • environment.journal.events : the main logger on the system, including for various daemons. It can be verbose, but it is the right place to notice (for instance) the system OOM-killer. (read more at man journald)

You can read more about ways to use event sets on their page.

Containers#

Since Antithesis runs your software inside containers, you’ll probably want to interact with them. If you want to run a command or extract a file, you’ll need a container.

Listing containers#

You can list all of your containers as of some moment.

[environment, moment] = prepare_multiverse_debugging()

print(environment.containers.list({moment}))

[
    {
        name: "myworkload",
        id: "5bf984f4b13cdb1aad334c80113d72b8b480984c4ae3d98da988c70975ca4af8",
        state: "running",
        image: "us-central1-docker.pkg.dev/molten-verve-216720/my-company-repository/my_db:latest",
        image_id: "20a003596f19ddcd16bd1c3ce30e0c3dedda40f959ead53e46a6e45e3452fa18"
    }
]

Running commands#

To run a command in one of your containers, you can use a bash fragment.

Any time you run a command, you can print the returned process object to view the output up to the end of the branch. (More on process objects below)

Note

Since we know that many lightweight containers don’t include bash, Antithesis mounts a shell and assorted shell tools into every container at runtime, making it possible to run bash commands even in minimal containers.

There are two main ways to run a bash fragment: in the foreground and the background. Either way, you need to supply a container, so we know where to run a command, and a branch, so we know when in the multiverse to run it. Running a bash command will return a process object.

Running a command with .run()#

Most of the time, when you want to run a command, you want to run it to completion. If you call .run() on a fragment, it will run until it finishes, advancing the branch until the command has exited.

If the command doesn’t exit, .run() will fail after 30 virtual minutes have passed. You can supply an optional timeout to change how long it waits.

print(bash`echo 'hello world!'`.run({container: "myworkload", timeout: Time.seconds(2), branch}))

78.513  info    hello world

Starting a command with .run_in_background()#

Sometimes, you want to start a long-running command and not wait for it to complete. When that happens, you can use .run_in_background(). Unlike .run(), .run_in_background() only advances time on the branch until after the command has been delivered. It might not even be started until time has advanced on the branch!

If you print the result, you’ll still see the output as it comes in (see the process object notes for more of an explanation).

print(bash`while true ; do du -hs /* && echo "" ; sleep 2 ; done`.run_in_background({container: "myworkload", branch}))

branch.wait(Time.seconds(10))

73.938  info    17404   /
73.938  info
76.111  info    17408   /
76.111  info
78.283  info    17413   /
78.283  info
80.455  info    17413   /
80.455  info
82.626  info    17420   /
82.627  info

Extracting files#

Sometimes, the best tool for inspecting a file is one you have locally. In cases like that, you can extract the file and download it.

// To extract a file, use the container name and the path in the container
link = extract_file({moment, path: "/run/myservice.db", container: "myworkload"})

// You can also extract a file from an image if you need to.
link = extract_file({moment, path: "/run/myservice.db", image: "my_db"})

// Print the result to get a download link
print(link)

Download myservice.log (808712 bytes)

Configuring fault injection#

If you’re exploring a moment from the middle of a test, Antithesis might have been disrupting the network or other containers. If you want to stop these faults, either to see whether your software recovers or to make other debugging clearer, you can use environment.fault_injector.pause, and if you want to resume, you cam use environment.fault_injector.unpause.

Profiling CPU usage#

The Environment includes a CPU profiler, which is a powerful way to investigate what’s happening during some part of a branch. Here’s what it looks like to use the profiler.

// Start the profiler on some branch
background_profiler = environment.profiler.start({branch})

// You can optionally supply a PID if you want to look at a particular process.
// background_profiler = environment.profiler.start({branch, pid: 1})

// Advance time on the branch
// Note that instead of waiting, you could run commands here.
// This is especially helpful for investigating the performance of a series of commands
branch.wait(Time.seconds(10))

// Stop the profiler on the branch
environment.profiler.stop({branch, background_profiler})

// View the results as of the end of the branch
print(environment.profiler.report({moment: branch.end}))
../_images/flame.png

Reference#

Process objects#

When you run a command in our environment, the result is a process object representing the execution of that process on the branch.

You can print a process to view its output.

proc = bash`ps`.run({container: "myworkload", branch})

print(proc)

73.446  info    Every 2.0s: ps
                2024-08-19 19:57:56
73.446  info
73.447  info    PID     USER    TIME    COMMAND
73.447  info      1     root    0:00    sleep infinity
73.447  info      2     root    0:00    watch ps
73.447  info      4     root    0:00    timeout 10 watch ps
73.447  info      5     root    0:00    ps

You can also call help(proc) for more details on what you can do with a process object.

Highlights#

  • Process-specific event sets, including proc.events, proc.stdout, proc.stderr, and proc.exits.

  • proc.pid is the PID of the process inside our virtual system.

  • proc.exited_by(moment) will return true if the process existed in the branch that ends with the supplied moment, and had exited.

  • proc.exit_code is the exit code that the process returned.

Background processes#

Whether you use .run() or .run_in_background(), the process will have the same shape, but the behavior may be slightly different.

In particular, when you get a Process object from .run_in_background(), its properties will be true as of the current state of the branch it was run on.

  • proc.exit_code will be undefined if the process hasn’t returned yet.

  • proc.pid will be undefined if the process hasn’t forked yet.

However, when you print() a process that results from .run_in_background(), you will see the events associated with the process on the branch that it was constructed from, as though the print were written below the last update to the branch in the notebook.

Bash fragments#

As we mentioned above, the multiverse debugging environment supports running shell commands via bash fragments. A bash fragment is a JavaScript tagged template literal containing the command that you want to run.

bash`echo 'hello world!'`

Like regular JavaScript templates, you can interpolate, but bash fragments escape intelligently.

s = 'This is a multicharacter string'

print(bash`echo ${s}`)

bash`echo "This is a multicharacter string"`

b = bash`ls /`

print(bash`MY_VARIABLE="foo" ${b}`) // equivalent to: bash`MY_VARIABLE="foo" ls /`

bash`MY_VARIABLE="foo" ls /`