Build and run an etcd cluster

In this getting started guide, we’ll walk you through how to set up an example system and test it in Antithesis. Our example system is etcd, a strongly consistent, distributed key-value datastore.

We’ll set up a 3-node etcd cluster in a local Docker environment, then deploy and test it in Antithesis.

You can find the source code for this tutorial on GitHub.

If you get stuck or have further questions, please don’t hesitate to contact us at support@antithesis.com or on Discord.

Before you start

To use Antithesis, you’ll need a container registry and credentials. To request these, contact us at support@antithesis.com.

You’ll receive a tenant name, a username and password. Use these to replace the $TENANT_NAME, $USER and $PASSWORD placeholders in this guide.

Set up the cluster

Create a working directory for this guide:

mkdir etcd-antithesis && cd etcd-antithesis

Pull the etcd container image:

docker pull bitnamilegacy/etcd:3.5

You’ll need this to test your setup locally.

Use Docker Compose to set up the etcd nodes and create a cluster. Create a new config directory and a docker-compose.yaml file inside it:

etcd-antithesis/
└── config/
    └── docker-compose.yaml

Inside docker-compose.yaml, define and configure the cluster:

services:
  etcd0:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd0
    hostname: etcd0
    environment: 
      ETCD_NAME: "etcd0"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd0:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd0.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

  etcd1:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd1
    hostname: etcd1
    environment: 
      ETCD_NAME: "etcd1"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd1:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd1.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

  etcd2:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd2
    hostname: etcd2
    environment: 
      ETCD_NAME: "etcd2"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd2:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd2.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

For each container, we define the image, container name and hostname, and set some environment variables. Let’s look at the container etcd0 in more detail:

Lines 4-5: It’s best practice to use the same name for the container_name and hostname when testing with Antithesis.
Line 7: Sets the name for the member node within the cluster. This name is used to identify member nodes in the cluster and is referenced in ETCD_INITIAL_CLUSTER.
Line 8: Sets the list of URLs the member node advertises to the rest of the cluster while bootstrapping. It’s referenced in ETCD_INITIAL_CLUSTER.
Lines 9-11: Sets the URLs the node listens on for peer and client traffic.
Lines 12-14: Sets the environment variables to bootstrap an initial cluster using the unique cluster token, cluster members, and cluster state.
Line 15: Allows users to connect to a etcd node without a password.

The containers etcd1 and etcd2 are configured similarly.

Start the cluster locally

Next, you’ll start the cluster locally and test that it’s working correctly.

Since there’s no internet access inside the Antithesis environment, when testing your setup locally, you should test it without internet access. On Linux machines, you can create a network namespace (for example, with unshare -n). From your config directory, run:

unshare -n

Alternatively, you can simply disconnect from the internet.

Then run docker-compose up to start the cluster:

docker-compose up

You’ll see a lot of logs from the cluster nodes. In a new terminal window, run:

docker ps

This command checks whether the Docker containers are running. It should list the three nodes, etcd0, etcd1, and etcd2 with an Up status.

Another way to check the cluster’s health is to ask a node to list the members of the cluster. Run:

docker exec etcd0 etcdctl member list

You should see the three nodes in the member list. You can run this command on other nodes as well by replacing etcd0.

Add a ready signal

When you run your etcd cluster in the Antithesis environment, Antithesis needs a way to know that the cluster is running and healthy before starting to test.

To do this, we’re going to add a healthcheck to the docker-compose.yaml file. The Docker Compose healthcheck attribute lets you define a command to run to determine whether or not the service container is “healthy”.

Once all the etcd nodes are healthy, you need to signal Antithesis that setup is complete and it should begin testing. To do this, you’ll create a health-checker container that depends on the etcd nodes to be “healthy” before sending a setup_complete message to Antithesis. You’ll then update your docker-compose.yaml file to include the new container.

Antithesis only expects to receive one setup_complete message from any of the containers in your system. Antithesis will treat the first such message sent by any running process as its signal to begin testing and injecting faults. Emitting further setup_complete messages has no effect, but if your system isn’t actually ready when the first one is sent, this can lead to unexpected problems.

Create the health-checker directory inside etcd-antithesis:

mkdir health-checker && cd health-checker

Then create the entrypoint.py health-checker script. Use the Antithesis Python SDK to add a setup_complete function call to emit the ready signal:

#!/usr/bin/env -S python3 -u

# This file serves as the client's entrypoint. It signals "setupComplete" using the Antithesis SDK

import time

from antithesis.lifecycle import (
    setup_complete,
)

print("Client [entrypoint]: cluster is healthy!")
# Here is the python format for setup_complete. At this point, our system is fully initialized and ready to test.
setup_complete({"Message":"ETCD cluster is healthy"})

Make the health-checker script an executable. From the health-checker directory, run:

chmod 777 entrypoint.py

You’ll run the health-checker in a separate container, so create a Dockerfile.health-checker inside the health-checker directory:

etcd-antithesis/
├── config/
│   └── docker-compose.yaml
└── health-checker/
    ├── Dockerfile.health-checker
    └── entrypoint.py

Add the following to Dockerfile.health-checker:

FROM python:3.12-slim

# Install dependencies and Antithesis Python SDK
RUN pip install --no-cache-dir antithesis cffi 

# Copy the python entrypoint. It contains the lifecycle setup_complete message.
COPY ./entrypoint.py /entrypoint.py

Now build the health-checker container image. From the health-checker directory, run:

docker build -f Dockerfile.health-checker -t us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-health-checker:v1 .

Replace $TENANT_NAME with your tenant’s name in this and all the following such commands.

Next, update your docker-compose.yaml file. Add a healthcheck attribute to each etcd service that calls etcdctl endpoint status to check the health of the cluster:

services:
  etcd0:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd0
    hostname: etcd0
    environment: 
      ETCD_NAME: "etcd0"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd0:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd0.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  etcd1:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd1
    hostname: etcd1
    environment: 
      ETCD_NAME: "etcd1"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd1:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd1.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  etcd2:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd2
    hostname: etcd2
    environment: 
      ETCD_NAME: "etcd2"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd2:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd2.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

Then add the health-checker service to docker-compose.yaml. Use the depends_on attribute to specify that it needs each of the etcd nodes to return a “healthy” signal before it runs the entrypoint.py script to let Antithesis know that setup has completed:

  health-checker:
    image: 'etcd-health-checker:v1'
    container_name: health-checker
    depends_on:
      etcd0:
        condition: service_healthy
      etcd1:
        condition: service_healthy
      etcd2:
        condition: service_healthy
    entrypoint: ['/entrypoint.py']

You’re now ready to test locally again. Start the cluster:

docker-compose up
...
...
...
health-checker  | Client [entrypoint]: cluster is healthy!

Once the cluster is ready, you should see the cluster is healthy! message shown above.

Package your configuration

You now have a working local setup. Next, you’ll package your config directory. To do this, make a Docker scratch image, a minimal base image that you add your docker-compose.yaml to. Later, Antithesis will extract the file from the config image to run your system.

Create a new Dockerfile inside the config directory:

etcd-antithesis/
├── config/
│   ├── docker-compose.yaml
│   └── Dockerfile.config
└── health-checker/
    ├── Dockerfile.health-checker
    └── entrypoint.py

Add the following to your Dockerfile.config:

FROM scratch
COPY docker-compose.yaml /docker-compose.yaml

Push your images

Next, you’ll push your images to the Antithesis registry.

When you become a customer, we configure a container registry for you and send you a credential file $TENANT_NAME.key.json.

To authenticate to your container registry, run the following command:

cat $TENANT_NAME.key.json | docker login -u _json_key https://us-central1-docker.pkg.dev --password-stdin

Now you’re locally authenticated to the registry and can run all other Docker commands as normal.

Push your health-checker and config images to us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/:

docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-health-checker:v1
docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:v1

Images that are publicly available (e.g. docker.io/bitnamilegacy/etcd:3.5) can be referenced directly in your config files and you do not need to copy them into the Antithesis registry.

Run your first test

You can now run your first test in Antithesis!

In this tutorial, you’ll run your test by calling our basic_test webhook endpoint, which starts a test in the Docker environment. You can also run tests through our GitHub integration.

Use this curl command to kick off a test:

curl --fail -u '$USER:$PASSWORD' \
-X POST https://$TENANT_NAME.antithesis.com/api/v1/launch/basic_test \
-d '{"params": { "antithesis.description":"basic_test on main",
    "antithesis.duration":"15",
    "antithesis.config_image":"etcd-config:v1", 
    "antithesis.report.recipients":"foo@email.com;bar@email.com"
    } }'

Use the $USER, $PASSWORD, and $TENANT_NAME provided by Antithesis, and update antithesis.report.recipients with the email recipients that you would like to receive the report.

Since you’re just learning the ropes here, we’ll set Antithesis up to test for 15 minutes, but once you’re up and running, you’ll want to do longer test runs.

For more information on these and other parameters, please consult the endpoint reference.

View your report

To check on the progress of your run, go to your Runs page at https://$TENANT_NAME.antithesis.com/runs. You should see your current run along with its status.

When your run completes, click the Triage results button to see your triage report, or click the link in the email you receive. You can also get results through Slack or Discord as well as email.

You’re now ready to exercise your cluster in part 2 of the tutorial!