> ## Build and run an etcd cluster

> Fetch the complete documentation index at: https://antithesis.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

---

In this getting started guide, we'll walk you through how to set up an example system and test it in Antithesis. Our example system is [etcd](https://etcd.io/), a strongly consistent, distributed key-value datastore.

We’ll set up a 3-node etcd cluster in a local Docker environment, then deploy and test it in Antithesis.

You can find the source code for this tutorial [on GitHub](https://github.com/antithesishq/examples/tree/main/etcd-docker).

If you get stuck or have further questions, please don't hesitate to contact us at [support@antithesis.com](mailto:support@antithesis.com) or on [Discord](https://discord.com/invite/antithesis).

## Before you start

To use Antithesis, you'll need a container registry and credentials. To request these, contact us at [support@antithesis.com](mailto:support@antithesis.com).

You'll receive a tenant name, a username and password. Use these to replace the `$TENANT_NAME`, `$USER` and `$PASSWORD` placeholders in this guide.

## Set up the cluster

Create a working directory for this guide:

```shell frame="none"
$ mkdir etcd-antithesis && cd etcd-antithesis
```

Pull the etcd container image:

```shell frame="none"
$ docker pull bitnamilegacy/etcd:3.5
```

You'll need this to test your setup locally.

Use Docker Compose to set up the etcd nodes and create a cluster. Create a new `config` directory and a [`docker-compose.yaml`](https://docs.docker.com/compose/) file inside it:

```console
etcd-antithesis/
└── config/
    └── docker-compose.yaml
```

Inside `docker-compose.yaml`, define and configure the cluster:

```yaml lines
services:
  etcd0:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd0
    hostname: etcd0
    environment:
      ETCD_NAME: "etcd0"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd0:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd0.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

  etcd1:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd1
    hostname: etcd1
    environment:
      ETCD_NAME: "etcd1"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd1:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd1.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

  etcd2:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd2
    hostname: etcd2
    environment:
      ETCD_NAME: "etcd2"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd2:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd2.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
```

For each container, we define the image, container name and hostname, and set some environment variables. Let's look at the container `etcd0` in more detail:

- **Lines 4-5**: It's best practice to use the same name for the `container_name` and `hostname` when testing with Antithesis.
- **Line 7**: Sets the name for the member node within the cluster. This name is used to identify member nodes in the cluster and is referenced in `ETCD_INITIAL_CLUSTER`.
- **Line 8**: Sets the list of URLs the member node advertises to the rest of the cluster while bootstrapping. It's referenced in `ETCD_INITIAL_CLUSTER`.
- **Lines 9-11**: Sets the URLs the node listens on for peer and client traffic.
- **Lines 12-14**: Sets the environment variables to bootstrap an initial cluster using the unique cluster token, cluster members, and cluster state.
- **Line 15**: Allows users to connect to a etcd node without a password.

The containers `etcd1` and `etcd2` are configured similarly.

## Start the cluster locally

Next, you'll start the cluster locally and test that it's working correctly.

Since there's no internet access inside the Antithesis environment, when testing your setup locally, you should test it without internet access. On Linux machines, you can create a network namespace (for example, with `unshare -n`). From your `config` directory, run:

```shell frame="none"
$ unshare -n
```

Alternatively, you can simply disconnect from the internet.

Then run `docker-compose up` to start the cluster:

```shell frame="none"
$ docker-compose up
```

You'll see a lot of logs from the cluster nodes. In a new terminal window, run:

```shell frame="none"
$ docker ps
```

This command checks whether the Docker containers are running. It should list the three nodes, `etcd0`, `etcd1`, and `etcd2` with an `Up` status.

Another way to check the cluster's health is to ask a node to list the members of the cluster. Run:

```shell frame="none"
$ docker exec etcd0 etcdctl member list
```

You should see the three nodes in the member list. You can run this command on other nodes as well by replacing `etcd0`.

## Add a ready signal

When you run your etcd cluster in the Antithesis environment, Antithesis needs a way to know that the cluster is running and healthy before starting to test.

To do this, we're going to add a healthcheck to the `docker-compose.yaml` file. The Docker Compose `healthcheck` attribute lets you define a command to run to determine whether or not the service container is "healthy".

Once all the etcd nodes are healthy, you need to signal Antithesis that setup is complete and it should begin testing. To do this, you'll create a `health-checker` container that depends on the etcd nodes to be "healthy" before sending a `setup_complete` message to Antithesis. You'll then update your `docker-compose.yaml` file to include the new container.

> **Warning**
>
> Antithesis only expects to receive one `setup_complete` message from any of the containers in your system. Antithesis will treat **the first such message sent by any running process** as its signal to begin testing and injecting faults.
> Emitting further `setup_complete` messages has no effect, but if your system isn't *actually* ready when the first one is sent, this can lead to unexpected problems.

Create the `health-checker` directory inside `etcd-antithesis`:

```shell frame="none"
$ mkdir health-checker && cd health-checker
```

Then create the `entrypoint.py` health-checker script. Use the Antithesis [Python SDK](/docs/reference/sdk/python/) to add a [`setup_complete`](https://antithesis.com/docs/generated/sdk/python/antithesis/lifecycle.html#setup_complete) function call to emit the ready signal:

```python
#!/usr/bin/env -S python3 -u

# This file serves as the client's entrypoint. It signals "setupComplete" using the Antithesis SDK

from antithesis.lifecycle import (
    setup_complete,
)

print("Client [entrypoint]: cluster is healthy!")
# Here is the python format for setup_complete. At this point, our system is fully initialized and ready to test.
setup_complete({"Message":"ETCD cluster is healthy"})
```

Make the health-checker script an executable. From the `health-checker` directory, run:

```shell frame="none"
$ chmod 777 entrypoint.py
```

You'll run the health-checker in a separate container, so create a `Dockerfile.health-checker` inside the `health-checker` directory:

```console ins={5}
etcd-antithesis/
├── config/
│   └── docker-compose.yaml
└── health-checker/
    ├── Dockerfile.health-checker
    └── entrypoint.py
```

Add the following to `Dockerfile.health-checker`:

```docker
FROM python:3.12-slim

# Install dependencies and Antithesis Python SDK
RUN pip install --no-cache-dir antithesis cffi

# Copy the python entrypoint. It contains the lifecycle setup_complete message.
COPY ./entrypoint.py /entrypoint.py
```

Now build the health-checker container image. From the `health-checker` directory, run:

```shell frame="none"
$ docker build -f Dockerfile.health-checker -t us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-health-checker:v1 .
```

Replace `$TENANT_NAME` with your tenant's name in this and all the following such commands.

Next, update your `docker-compose.yaml` file. Add a [`healthcheck`](https://docs.docker.com/reference/compose-file/services/#healthcheck) attribute to each etcd service that calls [`etcdctl endpoint status`](https://etcd.io/docs/v3.5/tutorials/how-to-check-cluster-status/) to check the health of the cluster:

```yaml ins={16-20, 36-40, 56-60}
services:
  etcd0:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd0
    hostname: etcd0
    environment:
      ETCD_NAME: "etcd0"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd0:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd0.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  etcd1:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd1
    hostname: etcd1
    environment:
      ETCD_NAME: "etcd1"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd1:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd1.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  etcd2:
    image: 'docker.io/bitnamilegacy/etcd:3.5'
    container_name: etcd2
    hostname: etcd2
    environment:
      ETCD_NAME: "etcd2"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd2:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd2.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3
```

Then add the `health-checker` service to `docker-compose.yaml`. Use the [`depends_on`](https://docs.docker.com/reference/compose-file/services/#depends_on) attribute to specify that it needs each of the etcd nodes to return a "healthy" signal before it runs the `entrypoint.py` script to let Antithesis know that setup has completed:

```yaml
  health-checker:
    image: 'etcd-health-checker:v1'
    container_name: health-checker
    depends_on:
      etcd0:
        condition: service_healthy
      etcd1:
        condition: service_healthy
      etcd2:
        condition: service_healthy
    entrypoint: ['/entrypoint.py']
```

You're now ready to test locally again. Start the cluster:

```shell frame="none"
$ docker-compose up
...
...
...
health-checker  | Client [entrypoint]: cluster is healthy!
```

Once the cluster is ready, you should see the `cluster is healthy!` message shown above.

## Package your configuration

You now have a working local setup. Next, you'll package your `config` directory. To do this, make a Docker [scratch image](https://docs.docker.com/develop/develop-images/baseimages/#create-a-simple-parent-image-using-scratch), a minimal base image that you add your `docker-compose.yaml` to. Later, Antithesis will extract the file from the `config` image to run your system.

Create a new Dockerfile inside the `config` directory:

```console ins={4}
etcd-antithesis/
├── config/
│   ├── docker-compose.yaml
│   └── Dockerfile.config
└── health-checker/
    ├── Dockerfile.health-checker
    └── entrypoint.py
```

Add the following to your `Dockerfile.config`:

```docker
FROM scratch
COPY docker-compose.yaml /docker-compose.yaml
```

## Push your images

Next, you'll push your images to the Antithesis registry.

When you become a customer, we configure a container registry for you and send you a credential file `$TENANT_NAME.key.json`.

To authenticate to your container registry, run the following command:

```shell frame="none"
$ cat $TENANT_NAME.key.json | docker login -u _json_key https://us-central1-docker.pkg.dev --password-stdin
```

Now you’re locally authenticated to the registry and can run all other Docker commands as normal.

Push your `health-checker` and `config` images to `us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/`:

```shell frame="none"
$ docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-health-checker:v1
$ docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:v1
```

Images that are publicly available (e.g. `docker.io/bitnamilegacy/etcd:3.5`) can be referenced directly in your config files and you do not need to copy them into the Antithesis registry.

## Run your first test

You can now run your first test in Antithesis!

In this tutorial, you’ll run the `Basic_test` that comes built into every Antithesis tenant.

In the web app, go to the “Test launchers” page at `https://$TENANT_NAME.antithesis.com/test-launchers`, and select `Basic_test`. Enter a duration of “15” and click “launch.”

You can also use [our webhook](/docs/reference/webhook/test_webhook/) to do this from command line.

Since you’re just learning the ropes here, we’ll set Antithesis up to test for 15 minutes, but once you’re up and running, you’ll want to do longer test runs. Later, you’ll also learn to write new tests in Antithesis.

## View your report

To check on the progress of your run, go to your **Runs** page at `https://$TENANT_NAME.antithesis.com/runs`. You should see your current run along with its status.

When your run completes, click the **Triage results** button to see your [triage report](/docs/product/reports/), or click the link in the email you receive. You can also get results through [Slack or Discord](/docs/product/tooling_integrations/discord_slack/) as well as email.

You're now ready to exercise your cluster in part 2 of the tutorial!