Product FAQs

1. How can I validate my setup?

You can manually validate your setup by disconnecting from the internet, running docker-compose up locally and verifying that all services are running as expected. If this happens, you should be able to call basic_test webhook successfully.

To streamline this process, you can automate it using a GitHub Action. The example workflow includes steps to build the project, validate the local setup, run and validate test commands, and finally call the basic_test endpoint.

We recommend setting up either this or a similar workflow in your CI pipeline.

2. How can I validate test composer commands locally?

To validate your test composer commands locally:

• Run docker-compose up, without internet connection, to bring up your system.
• Run the individual test commands using docker exec <container_with_command> path/to/test-file/. <container_name> is the container containing the test command.

If the test commands run successfully then you’re good to kick off an actual test. This page has more details.

3. How often should we test?

Most Antithesis customers run tests nightly to catch bugs and regressions as early in development as possible, and some also run shorter tests on every commit to help reduce the feedback cycle for active development.

• Nightly tests:

  • Run for 6-8hrs.
  • Uncover bugs that are hard to find. The longer a test runs, the more states are explored.

• Short tests:

  • Run on a commit to the main development branch.
  • Run for ~30min - 1hr.
  • Catch simple bugs that slipped through conventional unit and end-to-end tests.

4. What are “test hours” and how are they measured?

“Test hours” refers to the total time the System Under Test (SUT) experiences across an entire test run.

It’s roughly equivalent to the total active CPU time during the run, and can be estimated as: antithesis.duration × number of CPU cores provisioned.

5. Is there a way to bias test runs? If I believe there’s a bug in subsystem X can I get Antithesis to apply more effort to a subset of our tests?

Yes, there are several ways to focus on a particular subsystem:

• Write more test drivers that call the subsystem.
• Put more assertions in the subsystem or the clients that exercise the subsystem.
• Talk to our solutions engineers to help tune the configuration: support@antithesis.com.

6. Which S3 mocks do you support?

We support MinIO and Localstack services. This page has more details, and a long list of mocks for other dependencies.

7. Is there a logging limit in Antithesis? Can I run my system with full debugging logs?

Best practice when testing with Antithesis is to use a moderate amount of logging – we recommend a logging limit of 200 MB/test hour – but remember that the platform’s determinism makes it easy to reproduce and replay any bug you find.

If your system exceeds the 200 MB/test hour limit, the output/logging limit property in your report will fail. This section has details on how to do this.

Can you run with full DEBUG logs? Well, it depends on:

• How verbose your DEBUG logs are.
• How many services are logging at DEBUG level.
• Where the logs are written (e.g., DEBUG to disk, INFO to stdout is preferable).
• The overall scale of your system.

In general, smaller-scale systems with flexible logging setups are more likely to stay within the limit.

8. What are some common uses of sometimes assertions?

Sometimes assertions are important signposts along the software execution paths that can be useful in many ways.

1. Making rare bugs more common: If you encounter a bug infrequently and you want to reproduce it with more logging, you should add a sometimes assertion in the relevant parts of code. Or if you’ve tackled a flaky issue and want to make sure it’s actually fixed, sometimes assertions help Antithesis hit those code blocks more often.

2. Hit more edge cases in a test: It’s useful to write sometimes assertions for known edge cases that you want to test, so you know that the test actually ran your edge case. If it doesn’t pass, maybe you’re either not running the simulation long enough or you need to strengthen your test template to hit this scenario. For example, if you’re writing a search algorithm, maybe you add a sometimes assertion to ensure that Antithesis has encountered scenarios where the item is at the end of the list.

3. Ensure the system is being exercised: Make sure that minimum amount of work is being done during each test. Adding simple and intuitive sometimes assertions of some work being done in the system provides assurance that your system was correctly exercised. For example, if your product is a database, maybe you add a sometimes assertion to ensure Antithesis has encountered scenarios where a table has more than 1000 rows.

9. Why isn’t MY question covered here?

Probably because we don’t know about it! Please drop us a line at support@antithesis.com or ask us on Discord.