Querying with event sets

Event sets allow you to ask powerful questions about a moment’s history or future. An event set is an unevaluated query about the Antithesis multiverse.

These queries analyze a single timeline and return data in the form of events (see below) and include things like stdout messages, SDK assertion statuses, and journald messages.

An example

Here’s the scenario: we’re testing Chromium and believe it has a memory leak. We’ve already set up a naive memory monitor that logs percent memory usage by process name every few seconds as JSON.

First, we want to know when in the simulation Chromium’s memory usage is unacceptably high. To do this, we define the set of events where Chromium’s memory usage is above a threshold.

With this event set in hand, we can ask questions about both the past and future of a timeline in the multiverse.

In this example we use the Antithesis Notebook to ask:

A) In both the past and future of this timeline, what is the memory usage of Chromium? B) If we let the simulation continue forward, how long will it take for Chromium’s memory usage to pass our threshold?

To access the Notebook, see the workflow document.

// Step 1     
memory_events = environment.events.filter(
    (event) => event.source.includes('your_custom_memory_monitor')
)
parsed_set = memory_events.map((event) => {
    const parsed = JSON.parse(event.output_text)
    return {
        process_name: parsed.process_name,
        mem_usage: parsed.mem_usage,
    }
})
chromium_memory_events = parsed_set.filter((event) => event.process_name === 'chromium')

// Step 2
high_memory_usage_events = chromium_memory_events.filter((event) => event.mem_usage > 50)
high_memory_in_past = high_memory_usage_events.up_to(moment)
print(high_memory_in_past)

// Step 3 
future_branch = moment.branch()
future_branch.wait_until(high_memory_usage_events)
memory_events_in_future = chromium_memory_events.up_to(future_branch)
print(memory_events_in_future)

Let’s break this down:

Step 1: Parsing our data

Many things will be happening in the Antithesis simulation, but only some of them are interesting. We’re interested in a memory leak, so let’s focus on the relevant messages emitted by our memory monitor.

These messages are printed to stdout and are therefore events. Most events have a field source that contains information about the emitter of an event (including the container name). The source field can be manually set by your event emitter. Let’s assume we used the $ANTITHESIS_OUTPUT_DIR functionality (documented here) to set the source name for this memory monitor.

We can narrow our event set to relevant events by looking only at events whose source includes the string “your_custom_memory_monitor”:

memory_events = environment.events.filter(
    (event) => event.source?.name?.includes('your_custom_memory_monitor')
)

We know that our memory monitor writes logs as JSON. These can be parsed:

parsed_set = memory_events.map((event) => {
    const parsed = JSON.parse(event.output_text)
    return {
        process_name: parsed.process_name,
        mem_usage: parsed.mem_usage,
    }
})

Now we can access process_name and mem_usage as fields in our events and filter down further to only the relevant Chromium ones.

chromium_memory_events = parsed_set.filter((event) => event.process_name === 'chromium')

Step 2: Setting up conditions

After this setup, we can now ask a key question: has Chromium broken the memory threshold yet?

high_memory_usage_events = chromium_memory_events.filter((event) => event.mem_usage > 50)
high_memory_in_past = high_memory_usage_events.up_to(moment)
print(high_memory_in_past)

Here up_to (see reference below) takes our abstract EventSet query and executes it. It returns a concrete list of Events (we call this an EventSequence) that we can render nicely with our sequence viewer.

Step 3: Execute and Analyze

It seems like it hasn’t yet, so we ask our second question: will we pass the memory threshold in the future?

We spawn a new branch to run a wait_until campaign. This tells Antithesis to keep running our software until an event that matches our query is found. That is, the simulation will run until it encounters the first event where Chromium is using lots of memory.

future_branch = moment.branch()
future_branch.wait_until(high_memory_usage_events)

We can watch Chromium’s memory usage grow over time on this branch, by printing all of this branch’s Chromium memory events:

memory_events_in_future = chromium_memory_events.up_to(future_branch)
print(memory_events_in_future)

Event set reference

There are a couple of common event sets:

• environment.events, which is a set of all commonly useful events
• Process.stdout, which is all text printed to stdout by some process.
• Process.stderr, which is all text printed to stderr by some process.
• And there are more event sets available as part of the environment.

Querying

up_to is currently the only available query plan for event sets.

up_to is an instance method on EventSet. It accepts a single positional argument. For simple queries this argument is a branch or moment.

up_to returns an EventSequence containing events from the beginning of the simulation leading ‘up to’ the provided branch end or end moment. This EventSequence can be rendered by our sequence viewer.

If you’d like your results to start from a moment that is not the beginning of the simulation, you can provide a begin moment via an object of the shape {begin?: Moment, end: Branch | Moment} as the sole positional argument to up_to.

Here are a few examples:

print(environment.events.up_to(bug_branch))
// or to grab only part of a branch's history
print(environment.events.up_to({begin: start_moment, end: bug_branch})
// or to grab only particular kinds of events
print(environment.events.filter(event => event.output_text.includes("[WARN]")).up_to(bug_branch))

Creating other event sets

All event sets start from a root event set. In multiverse debugging this will typically be environment.events.

From here you can build derivative event sets by chaining together the following instance methods

EventSet::filter

environment.events.filter( event => event.moment.vtime > 20 )

Accepts an arbitrary function from event to any value. Returns the EventSet containing events where the function returns a truthy value.

EventSet::map

// The function returns an object with two new fields
// So events_with_new_fields is environment.events with these two new fields
events_with_new_fields = environment.events.map((event) => {
    const parsed = JSON.parse(event.output_text)
    return {
        process_name: parsed.process_name,
        mem_usage: parsed.mem_usage,
    }
})

Accepts an arbitrary function from event to object. Returns the originating EventSet with the object’s fields added to each event.

EventSet::contains

worker1_warnings = environment.events.contains({output_text: 'warning', source: 'worker1'})

Accepts an object with optional keys output_text, source, or stream and string values. Returns the EventSet representing events whose fields contain the string for every present key. This is case-sensitive.

EventSet::matches

server_events = environment.events.matches({source: 'my_db_server'})

Similar to contains. Accepts an object with optional keys output_text, source, or stream and string values. Returns the EventSet representing events whose fields are exactly equal to the provided string for every present key. This is case-sensitive.

EventSet::union

concerning_events = sev1_events.union(sev2_events, sev3_events)

Accepts arbitrarily many event sets. Returns the EventSet containing events that are contained in any of the given event sets.

A full list can be seen in the notebook with help(environment.events).

Events

Event emitters

An event represents anything that happened within the simulation. The following will emit events:

• stdout/stderr: Standard output of processes inside your containers will emit events
• SDK events: Our SDK assertions communicate with the rest of our system using events. You may also generate custom events using the SDK’s send_event() method.
• SDK capture directory: Files written to a special location will emit events.
• Antithesis environment: Systems like our fault injector, journald, and others will emit events. See their documentation for details.

Event fields

Events have the following fields (optional where noted):

• event.moment: The moment at which an event happened.

• event.moment.vtime: The time that passed within the simulation along this timeline, measured in seconds.

• event.source: The source that emitted an event.

• event.source.name: The human-readable name of the system or process that emitted the event.

  • For stdout/stderr and SDK assertion events it is your container name.
  • For the SDK capture directory, it is effectively the path into that directory and is documented here.
  • For Environment event emitters fault_injector, journal, and background_monitor it is the that emitter name.

• event.source.pid: Optional. The Linux process ID of the process that generated this event. PIDs are available for events emitted by the Antithesis SDK, or via writes to the SDK capture directory.

• event.source.container: Optional. The hostname of the container that generate this event, if any.

• event.source.stream: Optional. Current values include "error", "info", "internal".

  • For stdout/stderr this maps to "info" and "error" respectively.

• event.output_text: Optional. The text-payload of the event.

  • For stdout events this will be the verbatim text, with one event per newline.

• event.exit_code: Optional. If present, means that the Antithesis environment has terminated with an exit code.

  • This is an error in our setup of your system, contact your Professional Services representative or email support@antithesis.com.