Validating a distributed system is hard (not just hard work, but NP-hard) and all Cloud Native is distributed. If you write Cloud Native software, you’ll struggle to prevent subtle bugs.

etcd, a critical part of Cloud Native infrastructure, faced a crisis when consistency errors emerged in v3.5. This led to a significant investment to develop the etcd robustness testing framework, aiming for correctness in all conditions. The challenge for etcd, or any open-source project, is that maintaining such a guarantee requires unsustainable effort and expertise.

Here, Marek Siarkowicz, etcd Maintainer, and Marcus Hodgson, Forward-deployed Engineer at Antithesis, explain deterministic simulation testing, a technique for verifying entire distributed systems. This approach provides deterministic execution and fault injection, perfectly reproducing elusive bugs.

A CNCF-sponsored collaboration allowed the etcd team to use Antithesis to ensure etcd’s stability and trustworthiness, tackle more ambitious projects, and democratize involvement with a complex codebase.