Datadog’s Event Platform (EVP) sits at the heart of the Datadog platform, processing the data that customers rely on to understand and manage what’s happening across their production systems. Datadog collects more than 100 trillion events per day, and operating at this scale means that the platform has to continually evolve as customer environments become larger and more complex.

Joy Zhang, Senior Staff Engineer, works on the EVP Intake team, which is responsible for the pipelines that bring customer data into the platform. As the volume and complexity of those systems continue to grow, the Intake team looks for ways to make these pipelines more efficient.

To keep pace with that growth, the EVP Intake team began exploring a significant evolution of its architecture: moving from stateless communication between the Datadog Agent and Intake toward a stateful model that could substantially reduce the amount of data transmitted. Making that kind of change to a critical production system presented an equally significant testing challenge. How do you rebuild a 747 in midair?

The stateful encoding project

In the fall of 2025, the pipeline used a stateless HTTP-based architecture. The Datadog Agent, which runs in customer environments, would send customer telemetry data to Intake without maintaining decoding state between requests. As telemetry volumes continued to grow, the team saw an opportunity to make this communication more efficient by moving to a stateful model. By maintaining decoding state between the Agent and Intake, the new architecture could reduce the amount of data transmitted and the resources required to process it.

“We expect stateful encoding to significantly reduce the amount of data we need to transmit and process,” Joy explained. “That efficiency helps us continue to support growing telemetry volumes while maintaining the performance and reliability our customers depend on.”

Testing a stateful system

When the EVP Intake team began working with Antithesis, they had already built a partial prototype of the stateful pipeline. But before moving further, they needed a testing strategy that could give them confidence in the correctness of the new architecture.

“These services are load-bearing, highly critical, and very mature—they’re not greenfield projects,” Joy explained. “We were fundamentally changing the way two critical components of Datadog communicate. Maintaining this kind of stateful synchronization across proxies, with network delays and failures, is extremely tricky.”

The team had anticipated this challenge from the beginning. “Even in initial design, everybody knew this would be a complicated system, and we’d need to have a testing plan in place,” Joy said. “Going through development only strengthened our belief that the stateful encoding pipeline needed to be tested with something like Antithesis.”

Another Data Platform team at Datadog was already using Antithesis, giving the EVP Intake team an internal example to learn from. After hearing about that team’s experience, the EVP Intake team began evaluating whether Antithesis could provide the testing approach they needed for the stateful encoding project.

Putting the architecture to the test

Within a couple of days, the EVP Intake team integrated the components they’d built into an end-to-end prototype of the stateful encoding pipeline. They containerized the prototype and deployed it to Antithesis to begin testing the correctness of the protocol.

In parallel, the team used Antithesis to verify that the existing architecture upheld invariants such as preventing data loss. Given the size of the system under test, they were initially unsure how effectively Antithesis would be able to explore its state space. “We knew the test footprint was very large, so we were wondering how long it would take Antithesis to explore the state space and how resource-intensive it would be,” Joy said.

Early test runs surfaced issues in production code that could cause JVM crashes. Seeing Antithesis reproduce these kinds of failures increased the team’s confidence in using it to test the stateful encoding prototype.

“I initially thought we’d start using Antithesis when we moved closer to production, but I realized we should start during the POC,” Joy said. “Testing that early gave us a chance to find issues in the design implementation before they became harder to address, and gave us more confidence as we moved toward production.”

“Antithesis caught protocol-breaking bugs in stateful encoding, but what really gave us confidence was seeing it exercise complex timing interleavings that would have been difficult for us to reproduce with our existing tests,” she added. “The failures it surfaced made us reevaluate how we were testing the system.”

From test to harness

As the stateful encoding project progressed, the EVP Intake team was also experimenting with more agentic development workflows. Antithesis began to play a different role in how the team approached testing. “For the first six months, we were using Antithesis more as a traditional test on existing software releases,” Joy said. “Since then, we’ve started incorporating it earlier in our development workflow.”

That experience also influenced how the team thought about designing and testing new software. “The mindset has changed,” Joy said. “When our developers are designing new systems, they’re starting with invariants and test properties. Antithesis is one of the things that helped us make that shift.”

Michele Zoncheddu, a Software Engineer who worked on the stateful encoding project, put this approach into practice while building a new library for the Intake system that would underpin the pipeline’s durability guarantees.

“I was working on the new on-disk buffer for the stateful encoding pipeline and wrote down the invariants before writing almost any implementation code,” Michele said. “Then I pointed the antithesis-research skill at the journal currently deployed in production to check whether those same invariants held there. I gave it the repo, the new design, and a seed list of entry points.”

Within a couple of hours, Michele’s research agent identified a sequence of steps that violated a key durability invariant in the existing implementation. “It didn’t require any fault injection to trigger,” Michele said. “It was on the ordinary path under ordinary load and covered by existing tests, but those tests weren’t framed around the invariant.”

From POC to pilot

Following the POC, the EVP Intake team moved on to piloting the new architecture. “Our evaluations supported the assumptions behind the initial design, so we’ve started a pilot implementation on the metrics pipeline,” Joy said. “The stateful communication pattern is designed to be applicable across several types of telemetry data, giving us a path to evaluate it more broadly as we learn from the pilot.”

For the team, the project has also reinforced the value of bringing invariant-based testing earlier into the development process. Joel Barciauskas, Senior Director, Data Platform, described how the team is incorporating those lessons into its engineering workflows: “We’ve been taking the practices that help us increase development velocity and confidence and building them into the harness so that other teams can benefit from them. Our experience with EVP Intake showed us how testing approaches like Antithesis can help us take on ambitious changes to critical systems.”

The project also reflects a broader approach Datadog calls harness-first engineering: defining constraints and invariants around AI coding agents so engineers can use them effectively while maintaining rigorous validation. For the EVP Intake team, incorporating Antithesis into that harness provides another way to exercise complex system behavior and test whether critical invariants hold as development progresses.

As the team pilots stateful encoding in the metrics pipeline, these testing practices give engineers a way to evaluate the new architecture continuously as they move toward production—and to apply what they learn to future projects.