Build and run an etcd cluster

Before you start, please make sure you’ve acquired a container registry and credentials. If you get stuck, please don’t hesitate to contact us at support@antithesis.com or on Discord.

What is etcd?

Etcd is a strongly consistent, distributed key-value datastore.

We’ll set up a 3-node etcd cluster in a local Docker environment, then deploy and test it in Antithesis.

Here’s the source code covering this portion of the tutorial.

1. Create the project

$ mkdir etcd-antithesis && cd etcd-antithesis

2. Set up the cluster

First, pull the etcd container image. This will come in handy when you test your setup locally.

$ docker pull bitnami/etcd:3.5

Next, you’ll use container orchestration to set up the etcd nodes and create a cluster. Create a working directory and a docker-compose.yaml file inside it.

etcd-antithesis/
    config/
        docker-compose.yaml

Here’s the docker-compose.yaml file to define and configure the cluster – pay attention to the notes below:

version: '3.8'

services:

  etcd0:
    image: 'docker.io/bitnami/etcd:3.5'
    container_name: etcd0
    hostname: etcd0
    environment: 
      ETCD_NAME: "etcd0"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd0:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd0.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

  etcd1:
    image: 'docker.io/bitnami/etcd:3.5'
    container_name: etcd1
    hostname: etcd1
    environment: 
      ETCD_NAME: "etcd1"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd1:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd1.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

  etcd2:
    image: 'docker.io/bitnami/etcd:3.5'
    container_name: etcd2
    hostname: etcd2
    environment: 
      ETCD_NAME: "etcd2"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd2:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd2.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"

Lines 7-8 - It’s best practice to use the same name for the container_name and hostname when testing with Antithesis.

Line 10 - Sets the name for the member node within the cluster. This name is used to identify member nodes in the cluster and is referenced in ETCD_INITIAL_CLUSTER.

Line 11 - Sets the list of URLs the member node advertises to the rest of the cluster while bootstrapping. It’s referenced in ETCD_INITIAL_CLUSTER.

Lines 12-15 - Set the URLs the node listens on for peer and client traffic.

Lines 15-17 - Set the environment variables to bootstrap an initial cluster using the unique cluster token, cluster members, and cluster state.

Line 18 - Allows users to connect to a etcd node without a password.

Since there’s no internet access inside the Antithesis environment, when testing your setup locally, you should test it without internet access. On linux machines, you can enter a network namespace and then run docker-compose up. Or you can simply disconnect from the internet.

$ pwd
.../etcd-antithesis/config
$ unshare -n

[2] $ docker-compose up

You’ll see a lot of logs from the cluster nodes. There are two ways to check the health of the nodes and the cluster.

In a new terminal window, run:

$ docker ps

It should list the three nodes, etcd0, etcd1, and etcd2 with an Up status.

Another way to check the cluster’s health is to ask a node to list the members of the cluster. Run:

$ docker exec etcd0 etcdctl member list

You should see the three nodes in the member list. You can run this command on other nodes as well by replacing the etcd0.

4. Add a ready signal

When you run your etcd cluster in the Antithesis environment, Antithesis needs a way to know that the cluster is running and healthy before starting to test.

To do this, we’re going to use a healthcheck in the docker-compose.yaml file. The Docker Compose healthcheck attribute lets you define a command to run to determine whether or not the service container is “healthy”.

Once all the etcd nodes are healthy, you need to signal Antithesis that setup is complete and it should begin testing. To do this, you’ll create a health-checker container that depends on the etcd nodes to be “healthy” before sending a setup_complete message to Antithesis. Docker Compose’s depends_on field lets you decribe the service dependencies and specify an order in which the services should be started and stopped.

Antithesis only expects to receive one setup_complete message from any of the containers in your system. Antithesis will treat the first such message sent by any running process as its signal to begin testing and injecting faults. Emitting further setup_complete messages has no effect, but if your system isn’t actually ready when the first one is sent, this can lead to unexpected problems.

First, write the health-checker script. You’ll use Antithesis’ Python SDK to add a setup_complete function call to emit the ready signal.

Here’s the script – entrypoint.py:

$ pwd
.../etcd-antithesis
$ mkdir health-checker && cd health-checker
$ mkdir entrypoint && cd entrypoint

#!/usr/bin/env -S python3 -u

# This file serves as the client's entrypoint. It signals "setupComplete" using the Antithesis SDK

import time

from antithesis.lifecycle import (
    setup_complete,
)

print("Client [entrypoint]: cluster is healthy!")
# Here is the python format for setup_complete. At this point, our system is fully initialized and ready to test.
setup_complete({"Message":"ETCD cluster is healthy"})

Be sure to make the health-checker script an executable.

$ pwd
.../etcd-antithesis/health-checker/entrypoint
$ chmod 777 entrypoint.py

You’ll run the health-checker in a separate container, so create a Dockerfile.health-checker:

etcd-antithesis/
    health-checker/
        entrypoint/
            entrypoint.py
        Dockerfile.health-checker

FFROM docker.io/ubuntu:latest 

# Install dependencies
RUN apt-get update -y && apt-get install -y pip

# Install Python
RUN apt-get install -y python3

# Install Python etcd client
RUN apt install -y python3-etcd3

# Install Antithesis Python SDK
RUN pip install antithesis cffi --break-system-packages

# Copying entrypoint. It contains the lifecycle setup_complete message.
COPY ./entrypoint/entrypoint.py /entrypoint.py

Build the health-checker container image:

$ pwd
.../etcd-antithesis/health-checker
$ docker build -f Dockerfile.health-checker -t us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-health-checker:v1 .

Add the health-checker service to the docker-compose.yaml file and add the health-checks on etcd nodes.

Your updated docker-compose.yaml should look like this:

version: '3.8'

services:

  etcd0:
    image: 'docker.io/bitnami/etcd:3.5'
    container_name: etcd0
    hostname: etcd0
    environment: 
      ETCD_NAME: "etcd0"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd0:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd0.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  etcd1:
    image: 'docker.io/bitnami/etcd:3.5'
    container_name: etcd1
    hostname: etcd1
    environment: 
      ETCD_NAME: "etcd1"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd1:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd1.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  etcd2:
    image: 'docker.io/bitnami/etcd:3.5'
    container_name: etcd2
    hostname: etcd2
    environment: 
      ETCD_NAME: "etcd2"
      ETCD_INITIAL_ADVERTISE_PEER_URLS: "http://etcd2:2380"
      ETCD_LISTEN_PEER_URLS: "http://0.0.0.0:2380"
      ETCD_LISTEN_CLIENT_URLS: "http://0.0.0.0:2379"
      ETCD_ADVERTISE_CLIENT_URLS: "http://etcd2.etcd:2379"
      ETCD_INITIAL_CLUSTER_TOKEN: "etcd-cluster-1"
      ETCD_INITIAL_CLUSTER: "etcd0=http://etcd0:2380,etcd1=http://etcd1:2380,etcd2=http://etcd2:2380"
      ETCD_INITIAL_CLUSTER_STATE: "new"
      ALLOW_NONE_AUTHENTICATION: "yes"
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 5s
      timeout: 5s
      retries: 3

  health-checker:
    image: 'etcd-health-checker:v1'
    container_name: health-checker
    depends_on:
      etcd0:
        condition: service_healthy
      etcd1:
        condition: service_healthy
      etcd2:
        condition: service_healthy
    entrypoint: ['/entrypoint.py']

Test the cluster and health-checker setup locally:

$ docker-compose up
...
...
...
health-checker  | Client [entrypoint]: cluster is healthy!

You should see the cluster is healthy! message shown above.

5. Build a configuration image

Once your setup works locally, you can package the config directory by adding the configuration files to a scratch image. This is just a simple way to ship it to Antithesis. Antithesis will extract all the files from the config image to run your system.

Create a Dockerfile:

etcd-antithesis/
    config/
        docker-compose.yaml
        Dockerfile.config

FROM scratch
COPY docker-compose.yaml /docker-compose.yaml

Build a config image to upload to the registry:

$ docker build -f Dockerfile.config -t us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:v1 .

6. Push your containers

When you start a POC, we configure a container registry for you and send you a credential file $TENANT_NAME.key.json.

To authenticate to your container registry, run the following command:

$ cat $TENANT_NAME.key.json | docker login -u _json_key https://us-central1-docker.pkg.dev --password-stdin

Now you’re locally authenticated to the registry and can run all other Docker commands as normal.

Push your containers and config image to: us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/

$ docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-health-checker:v1
$ docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:v1

7. Run your first test

Antithesis provides webhook endpoints that allow you to run tests on demand. This curl command kicks off a test. You should have received a username and password when you started your POC.

Since you’re just learning the ropes here, we’ll set Antithesis up to test for 30 minutes, but once you’re up and running, you’ll want to do longer test runs.

curl --fail -u 'user:password' \
-X POST https://<tenant>.antithesis.com/api/v1/launch/basic_test \
-d '{"params": { "antithesis.description":"basic_test on main",
    "antithesis.duration":"30",
    "antithesis.config_image":"etcd-config:v1",
    "antithesis.images":"docker.io/bitnami/etcd:3.5;etcd-health-checker:v1", 
    "antithesis.report.recipients":"foo@email.com;bar@email.com"
    } }'

For more information on these and other parameters, please consult our webhook reference.

Within an hour, you’ll receive an email with a link to a triage report. We suggest you read about the triage report, and test properties, while you wait.

You can also get results through Slack or Discord as well as email.

You’re now ready to exercise your cluster in part 2 of the tutorial!