Build and run an etcd cluster

Before you start, please make sure you’ve acquired a container registry and credentials. If you get stuck, please don’t hesitate to contact us at support@antithesis.com or on Discord.

For this tutorial, you’ll need Docker and Helm installed.

You’ll also need a local Kubernetes cluster. We recommend K3s for parity with our environment. Kind or minikube also work for local testing.

Here’s the source code for this tutorial.

What is etcd?

Etcd is a strongly consistent, distributed key-value datastore.

We’ll set up a 3-node etcd cluster in a local kubernetes environment, then deploy and test it in Antithesis.

Create manifests

Create a working directory

mkdir etcd-antithesis && cd etcd-antithesis

Add the Bitnami Helm repo

helm repo add bitnami https://charts.bitnami.com/bitnami 
helm repo update

Create a values.yaml file

These values configure a 3-node etcd cluster for the Antithesis environment. We’ll use this file to template the Helm chart.

fullnameOverride: "etcd" # Ensures pods have predictable names like etcd-0, etcd-1, etcd-2
replicaCount: 3
image:
  repository: bitnamilegacy/etcd
  tag: 3.5
preUpgradeJob:
  enabled: false
persistence:
  size: 1Gi
auth:
  rbac:
    create: false
  client:
    enableAuthentication: false
  token:
    enabled: false

Template the helm chart

Use the values.yaml file to render the kubernetes manifests.

mkdir -p manifests && 
helm template etcd bitnami/etcd -f values.yaml > manifests/etcd.yaml

Start the cluster locally

Apply the generated manifests to your local Kubernetes cluster to confirm they start a healthy etcd cluster. This approximates how the setup will behave in Antithesis.

Apply your manifests

kubectl apply -f manifests/etcd.yaml

kapp deploy -a etcd -f manifests/ --yes

Check rollout status

kubectl rollout status statefulset/etcd --timeout=3m
kubectl get sts etcd
kubectl get pods

You should see 3/3 pods become ready, named etcd-0, etcd-1, etcd-2.

Build & upload a config image

Package your manifests/ directory into a minimal scratch image called the config image. Before tests run, Antithesis pulls the image, extracts /manifests, and applies the YAML files to run your system.

Create a Dockerfile

Create a Dockerfile.config alongside the manifests/ folder:

FROM scratch
COPY ./manifests/ /manifests/

Build the config image

docker build -f Dockerfile.config -t us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:k8s .

Push to the registry

When you become a customer, we configure a container registry for you and send you a credential file $TENANT_NAME.key.json.

To authenticate to your container registry, run the following command:

cat $TENANT_NAME.key.json | docker login -u _json_key https://us-central1-docker.pkg.dev --password-stdin

Now you’re locally authenticated to the registry and can run all other Docker commands as normal.

Push your custom and non-public images (including your config image) to: us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/:

docker push us-central1-docker.pkg.dev/molten-verve-216720/$TENANT_NAME-repository/etcd-config:k8s

Images that are publicly available (e.g. docker.io/bitnamilegacy/etcd:3.5) can be referenced directly in your config files, you do not need to copy them into the Antithesis registry.

Run your first test

You can now run your first test in Antithesis!

In this tutorial, you’ll run your test by calling our basic_test webhook endpoint, which starts a test in the Docker environment. You can also run tests through our GitHub integration.

Use this curl command to kick off a test:

curl --fail -u 'USER:PASSWORD' \
-X POST https://<TENANT>.antithesis.com/api/v1/launch/basic_k8s_test \
-d '{"params": { "antithesis.description":"basic_k8s_test on main",
    "antithesis.duration":"15",
    "antithesis.config_image":"etcd-config:k8s",
    "antithesis.report.recipients":"foo@email.com;bar@email.com"
    } }'

Use the $USER, $PASSWORD, and $TENANT_NAME provided by Antithesis, and update antithesis.report.recipients with the email recipients that you would like to receive the report.

Since you’re just learning the ropes here, we’ll set Antithesis up to test for 15 minutes, but once you’re up and running, you’ll want to do longer test runs.

For more information on these and other parameters, please consult our webhook reference.

View your report

To check on the progress of your run, go to your Runs page at https://$TENANT_NAME.antithesis.com/runs. You should see your current run along with its status.

When your run completes, click the Triage results button to see your triage report, or click the link in the email you receive. You can also get results through Slack or Discord as well as email.

You’re now ready to exercise your cluster in part 2 of the tutorial!