How to Run a Test#

Webhooks#

Antithesis provides webhook endpoints that can be used to trigger preconfigured test setups on demand. We can provide various webhook endpoints to cover the testing configurations needed for your system.

Calling a webhook fetches all of the containers you have previously pushed (the default tags, specified in the notebook, will be used unless specified with the parameter below). After pulling the containers, our system will build the test environment, run your test for a configured duration, and report results by sending a link to one of our interactive triage reports to the email address(es) specified. Contact us if you prefer to receive your report links in your Slack, issue tracker, or somewhere else, and we can help you build that integration.

Running a webhook is simple. Run:

curl --fail -u 'user:password' -X POST https://<subdomain>.antithesis.com/api/v1/launch_experiment/<endpoint>

Where the variables have the following definitions:

user:password

the basic auth user and password for your tenant (provided by Antithesis)

<subdomain>

the subdomain for the tenant or the tenant name

<endpoint>

the endpoint to run (provided by Antithesis)

Webhook parameters#

Webhook parameters allow the caller to pass parameters to a test to control its behavior. Tests may define custom parameters; your consultant should provide you with a list of supported parameters for your test. Parameters are provided in the body of your request to start a test, as in the following:

curl --fail -u 'user:password' -X POST https://<subdomain>.antithesis.com/api/v1/launch_experiment/<endpoint> -d '{"params": { "parameter1_name": "parameter1_value", "parameter2_name":"parameter2_value" } }'

This is the template for the request body:

{
    "params": {
        "parameter1_name": "parameter1_value",
        "parameter2_name": "parameter2_value",
    }
}

In addition to parameters that have been customized for you, all Antithesis endpoints support a set of default parameters. Unless otherwise indicated, all such parameters are optional:

antithesis.integrations.type

The CI / Source Control integration type. Supported values are : ["none", "github"]

antithesis.integrations.callback_url

The URL Antithesis will call back to post the test results.

antithesis.integrations.token

A security token used to authenticate when calling back with results.

antithesis.report.recipients

Any additional recipients that should receive the report email. You can specify more than one additional address with a ‘;’ delimiter, for example, u1@site.com;u2@site.com

antithesis.images

The image versions that Antithesis will pull from the container registry to test. This is a ‘;’ delimited list. Each entry in the list specifies the container name and either a digest (recommended) or a tag in this format NAME[:TAG|@DIGEST]. e.g. container_1@sha25612341234;container_2:latest_tag

antithesis.source

An optional source control descriptor. When specified, the report will include it in its header & history information will be limited to runs with similar source.

The defaults are none for the integration type, and empty for the rest. When email recepients are empty, emails will be sent to the default users set up for the webhook.

Here is an example of how some of the default parameters are used in our GitHub integration:

{
    "params": {
        "antithesis.integrations.type": "github",
        "antithesis.integrations.callback_url": "https://api.github.com/repos/<org>/<repo>/statuses/<sha1>",
        "antithesis.integrations.token": "my_github_token",
    }
}

This is how the above could be called in curl:

curl --fail -u 'user:password' -X POST https://<subdomain>.antithesis.com/api/v1/launch_experiment/<endpoint> -d '{"params": { "antithesis.integrations.type":"github", "antithesis.integrations.callback_url":"https://api.github.com/repos/<org>/<repo>/statuses/<sha1>", "antithesis.integrations.token":"my_github_token", "antithesis.images":"my_images_with_version_list" } }'

Scheduling the webhook#

Antithesis tests are high-throughput and high-latency; thus, the webhook is not designed to be used interactively, or by a human user. Instead, it should be triggered on a recurring basis by an automated system, such as your CI server.

There are many benefits to running tests frequently. Among them are better bisection capability around when bugs were introduced, richer data around whether we can find the bug every time, and lower risk of a bug going unnoticed and being released in production. For these and other reasons, our current recommended cadence is nightly testing. You should configure your CI system so that each night, it builds all of the containers associated with your Antithesis setup, pushes them to our registry, and then triggers the webhook.

If you are unable to configure your CI server to call a remote endpoint, we can handle scheduling the webhook for you. Simply tell us which webhooks you want activated—and what time they should run—and we will take care of running them every day at that time.

We can also work with you to schedule your tests for you—just contact us.

Webhook results#

Curl (and similar tools) will exit successfully so long as the webhook returns anything at all, even if these returns are failure codes. To check for success of the underlying webhook, you must ensure that a 200 or 202 code was returned. Alternatively, you may (as we demonstrate) use curl with the –fail flag or whatever similar flag your tool might have.