The Environment and its utilities
Exploring the Universe
In Exploring the multiverse, we explain that a moment is a single point in the tree of possible timelines, but we don’t talk much about what’s inside a moment.
The Environment is the virtual system where we run your containers – it’s the universe in some branch of the testing multiverse.
You can read more about the specifics of the system in our Antithesis Environment documentation, but for our purposes, you can think of it as a Linux system that’s been customized for testing.
The Environment object
In each moment, the Environment is in some state. There are probably containers running your software, there may be Antithesis faults in progress, and there are certainly some background services supporting the system. Inside a multiverse debugging session, the Environment object represents the system as a whole.
When you begin your session, the Environment
is already defined for you.
// `environment` here is your Environment
[environment, moment] = prepare_multiverse_debugging()
As a rule of thumb, we’ll typically use the Environment when we’re inspecting and configuring the system as a whole.
Environment event sets
Sometimes, when you’re diagnosing a problem, it’s helpful to see what’s going on with the system as a whole. System logs are available through the Environment object as curated event sets.
When events are the result of readable (non-binary) program output, an event is sent each time the output contains a newline character.
environment.containers.events
: output from all of the containersenvironment.containers.meta_events
: changes to containers (create, start, stop, etc) as seen by the container management systemenvironment.sdk.all_events
: all of the events that have been processed through the Antithesis SDKenvironment.sdk.assertions
: assertions defined with the Antithesis SDK which have been encounteredenvironment.events
: the main event source, including the sources aboveenvironment.fault_injector.events
: cumulative statistics and other logs from the Antithesis fault injector (Warning: we have not yet invested in making this easy to understand, and the format is likely to change!)environment.background_monitor.events
: raw output from the Antithesis system utilization monitor (Warning: this is a new feature and its format may be unusually unstable!)environment.journal.events
: the main logger on the system, including for various daemons. It can be verbose, but it is the right place to notice (for instance) the system OOM-killer. (read more at man journald)
You can read more about ways to use event sets on their page.
Containers
Since Antithesis runs your software inside containers, you’ll probably want to interact with them. If you want to run a command or extract a file, you’ll need a container.
Listing containers
You can list all of your containers as of some moment.
[environment, moment] = prepare_multiverse_debugging()
print(environment.containers.list({moment}))
[
{
name: "myworkload",
id: "5bf984f4b13cdb1aad334c80113d72b8b480984c4ae3d98da988c70975ca4af8",
state: "running",
image: "us-central1-docker.pkg.dev/molten-verve-216720/my-company-repository/my_db:latest",
image_id: "20a003596f19ddcd16bd1c3ce30e0c3dedda40f959ead53e46a6e45e3452fa18"
}
]
Running commands
To run a command in one of your containers, you can use a bash fragment.
Any time you run a command, you can print the returned process object to view the output up to the end of the branch. (More on process objects below)
Since we know that many lightweight containers don’t include bash, Antithesis mounts a shell and assorted shell tools into every container at runtime, making it possible to run bash commands even in minimal containers.
There are two main ways to run a bash fragment: in the foreground and the background. Either way, you need to supply a container, so we know where to run a command, and a branch, so we know when in the multiverse to run it. Running a bash command will return a process object.
Running a command with .run()
Most of the time, when you want to run a command, you want to run it to completion. If you call .run()
on a fragment, it will run until it finishes, advancing the branch until the command has exited.
If the command doesn’t exit, .run()
will fail after 30 virtual minutes have passed. You can supply an optional timeout to change how long it waits.
print(
bash`echo 'hello world!'`.run({
container: "myworkload",
timeout: Time.seconds(2),
branch
})
)
78.513 info hello world
Starting a command with .run_in_background()
Sometimes, you want to start a long-running command and not wait for it to complete. When that happens, you can use .run_in_background()
. Unlike .run()
, .run_in_background()
only advances time on the branch until after the command has been delivered. It might not even be started until time has advanced on the branch!
If you print the result, you’ll still see the output as it comes in (see the process object notes for more of an explanation).
print(bash`while true ; do du -hs /* && echo "" ; sleep 2 ; done`.run_in_background({container: "myworkload", branch}))
branch.wait(Time.seconds(10))
73.938 info 17404 /
73.938 info
76.111 info 17408 /
76.111 info
78.283 info 17413 /
78.283 info
80.455 info 17413 /
80.455 info
82.626 info 17420 /
82.627 info
Extracting files
Sometimes, the best tool for inspecting a file is one you have locally. In cases like that, you can extract the file and download it.
// To extract a file, use the container name and the path in the container
link = environment.extract_file({moment, path: "/run/myservice.db", container: "myworkload"})
// You can also extract a file from an image if you need to.
link = environment.extract_file({moment, path: "/run/myservice.db", image: "my_db"})
// Print the result to get a download link
print(link)
Injecting files
Sometimes, you may want to inject a file in your container after it’s started. In cases like that, use inject_file()
.
// To inject a file, use the container name, the path in the container, and the file_data you want to inject.
stat = environment.inject_file({
branch,
path: "/run/testing.sh",
container: "mycontainer",
file_data: bash`echo hello`,
permissions: "0755"
})
// Print the result to see the stat results
print(stat)
Configuring fault injection
If you’re exploring a moment from the middle of a test, Antithesis might have been disrupting the network or other containers. If you want to stop these faults, either to see whether your software recovers or to make other debugging clearer, you can use environment.fault_injector.pause
, and if you want to resume, you cam use environment.fault_injector.unpause
.
Profiling CPU usage
The Environment includes a CPU profiler, which is a powerful way to investigate what’s happening during some part of a branch. Here’s what it looks like to use the profiler.
// Start the profiler on some branch
background_profiler = environment.profiler.start({branch})
// You can optionally supply a PID if you want to look at a particular process.
// background_profiler = environment.profiler.start({branch, pid: 1})
// Advance time on the branch
// Note that instead of waiting, you could run commands here.
// This is especially helpful for investigating the performance of a series of commands
branch.wait(Time.seconds(10))
// Stop the profiler on the branch
environment.profiler.stop({branch, background_profiler})
// View the results as of the end of the branch
print(environment.profiler.report({moment: branch.end}))
Reference
Process objects
When you run a command in our environment, the result is a process object representing the execution of that process on the branch.
You can print a process to view its output.
proc = bash`ps`.run({container: "myworkload", branch})
print(proc)
73.446 info Every 2.0s: ps
2024-08-19 19:57:56
73.446 info
73.447 info PID USER TIME COMMAND
73.447 info 1 root 0:00 sleep infinity
73.447 info 2 root 0:00 watch ps
73.447 info 4 root 0:00 timeout 10 watch ps
73.447 info 5 root 0:00 ps
You can also call help(proc)
for more details on what you can do with a process object.
Highlights
- Process-specific event sets, including
proc.events
,proc.stdout
,proc.stderr
, andproc.exits
. proc.pid
is the PID of the process inside our virtual system.proc.exited_by(moment)
will returntrue
if the process existed in the branch that ends with the supplied moment, and had exited.proc.exit_code
is the exit code that the process returned.
Background processes
Whether you use .run()
or .run_in_background()
, the process will have the same shape, but the behavior may be slightly different.
In particular, when you get a Process object from .run_in_background()
, its properties will be true as of the current state of the branch it was run on.
proc.exit_code
will beundefined
if the process hasn’t returned yet.proc.pid
will beundefined
if the process hasn’t forked yet.
However, when you print()
a process that results from .run_in_background()
, you will see the events associated with the process on the branch that it was constructed from, as though the print were written below the last update to the branch in the notebook.
Bash fragments
As we mentioned above, the multiverse debugging environment supports running shell commands via bash fragments. A bash fragment is a JavaScript tagged template literal containing the command that you want to run.
bash`echo 'hello world!'`
Like regular JavaScript templates, you can interpolate, but bash fragments escape intelligently.
s = 'This is a multicharacter string'
print(bash`echo ${s}`)
bash`echo "This is a multicharacter string"`
b = bash`ls /`
print(bash`MY_VARIABLE="foo" ${b}`)
// equivalent to: bash`MY_VARIABLE="foo" ls /`
b = bash`ls /`
bash`MY_VARIABLE="foo" ls /`