Build and run an etcd cluster

Before you start, please make sure you’ve acquired a container registry and credentials. If you get stuck, please don’t hesitate to contact us at support@antithesis.com or on Discord.

For this tutorial, you’ll need Docker and Helm installed.

You’ll also need a local Kubernetes cluster. We recommend K3s for parity with our environment. Kind or minikube also work for local testing.

Here’s the source code for this tutorial.

What is etcd?

Etcd is a strongly consistent, distributed key-value datastore.

We’ll set up a 3-node etcd cluster in a local kubernetes environment, then deploy and test it in Antithesis.

Create manifests

Create a working directory

$ mkdir etcd-antithesis && cd etcd-antithesis

Add the Bitnami Helm repo

$ helm repo add bitnami https://charts.bitnami.com/bitnami 
$ helm repo update

Create a values.yaml file

These values configure a 3-node etcd cluster for the Antithesis environment. We’ll use this file to template the Helm chart.

fullnameOverride: "etcd" # Ensures pods have predictable names like etcd-0, etcd-1, etcd-2
replicaCount: 3
image:
  repository: bitnamilegacy/etcd
  tag: 3.5
preUpgradeJob:
  enabled: false
persistence:
  size: 1Gi
auth:
  rbac:
    create: false
  client:
    enableAuthentication: false
  token:
    enabled: false

Template the helm chart

Use the values.yaml file to render the kubernetes manifests.

$ mkdir -p manifests && 
helm template etcd bitnami/etcd -f values.yaml > manifests/etcd.yaml

Start the cluster locally

Apply the generated manifests to your local Kubernetes cluster to confirm they start a healthy etcd cluster. This approximates how the setup will behave in Antithesis.

Apply your manifests

$ kubectl apply -f manifests/etcd.yaml

$ kapp deploy -a etcd -f manifests/ --yes

Check rollout status

$ kubectl rollout status statefulset/etcd --timeout=3m
$ kubectl get sts etcd
$ kubectl get pods

You should see 3/3 pods become ready, named etcd-0, etcd-1, etcd-2.

Build & upload a config image

Package your manifests/ directory into a minimal scratch image called the config image. Before tests run, Antithesis pulls the image, extracts /manifests, and applies the YAML files to run your system.

Create a Dockerfile

Create a Dockerfile.config alongside the manifests/ folder:

FROM scratch
COPY ./manifests/ /manifests/

Build the config image

$ docker build -f Dockerfile.config -t us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:k8s .

Push to the registry

When you become a customer, we configure a container registry for you and send you a credential file $TENANT_NAME.key.json.

To authenticate to your container registry, run the following command:

$ cat $TENANT_NAME.key.json | docker login -u _json_key https://us-central1-docker.pkg.dev --password-stdin

Now you’re locally authenticated to the registry and can run all other Docker commands as normal.

Push your custom and non-public images (including your config image) to: us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/

Images that are publicly available (e.g. docker.io/bitnamilegacy/etcd:3.5) can be referenced directly in your manifests with their fully qualified registry path, you do not need to copy them into the Antithesis registry.

$ docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:k8s

Run your first test

You can use this webhook endpoint to run tests on demand. Use the username and password you receive when you become a customer.

Remember to change <tenant> and antithesis.report.recipients accordingly.

Since you’re just learning the ropes here, we’ll set Antithesis up to test for 15 minutes, but once you’re up and running, you’ll want to run much longer tests.

curl --fail -u 'USER:PASSWORD' \
-X POST https://<TENANT>.antithesis.com/api/v1/launch/basic_k8s_test \
-d '{"params": { "antithesis.description":"basic_k8s_test on main",
    "antithesis.duration":"15",
    "antithesis.config_image":"etcd-config:k8s",
    "antithesis.images":"docker.io/bitnamilegacy/etcd:3.5", 
    "antithesis.report.recipients":"foo@email.com;bar@email.com"
    } }'

For more information on these and other parameters, please consult our webhook reference.

Soon after the run completes, you’ll receive an email with a link to a triage report. We suggest you read about the triage report, and test properties, while you wait.

You can also get results through Slack or Discord as well as email.

You’re now ready to exercise your cluster in part 2 of the tutorial!