TW Lim

Humble scrivener

No power, no programs

April 2, 2025

With gigawatt-scale AI datacenters coming online every month, EVs proliferating, and more and more power coming from intermittent renewables, utilities around the world increasingly rely on grid observability technologies like PingThings’ PredictiveGrid platform to keep the lights on and the power flowing. “Everyone’s excited about the potential of AI… but there’s no AI without electricity,” PingThings CTO Michael Brown observes.

But in spite of its importance to literally every aspect of modern life, “Most VCs are only just starting to realize how fundamental this space is… it’s foundational to AI, essential for the clean energy transition, so many other things,” said Sean Murphy, PingThings’ CEO. “But funding hasn’t quite caught up. In 2021, we managed to raise what was considered a decent round in this sector… $4 million.” PingThings leveraged this funding to win Tier 1 utility customers and become cash flow positive.

Being scrappy, PingThings strives to do more with less. It relies on Antithesis to turbocharge the productivity of their engineering team.

“We use Antithesis as a force multiplier to achieve a much larger impact with a much smaller team. We’ve seen a real transformation of the development workflow over the year we’ve used it. I’d say Antithesis allowed us to pay down years of tech debt and 3x our engineering output.” – Sean Murphy, CEO

PredictiveGrid enables electric utilities to ingest, store, and ultimately make sense of data from hundreds of thousands of sensors connected to the electrical grid. These data streams overwhelm traditional relational databases and even scalable NOSQL data stores like Apache Cassandra. Modern time series databases can write approximately 1M points per second per node, while it’s not uncommon for utilities to generate 30 or more times that amount today.

Querying over time ranges and analytical workloads are correspondingly challenging, while sensor density is only increasing. Simply uniting and making sense of this disparate collection of data streams is a significant technical challenge, but unlocks correspondingly significant value.

High density data, at scale

PingThings, and the PredictiveGrid platform, were envisioned as a way to handle this high density telemetry or time series data, only to run into a utility-scale technical challenge.

“The grid is actually covered with sensors that span six or seven orders of magnitude and sample rates. You have everything from smart meters that sample once an hour to point-on-wave sensors that run 100,000 samples per second.” Brown said. “That’s an insane range in terms of time series data and at the slow end, you have issues around higher cardinality. You’re looking at tens or hundreds or thousands of streams simultaneously because the information isn’t in a single stream. It’s across streams. But then when you have 100,000 samples per second, one stream is rich in information.”

So PingThings quickly found themselves having to rebuild the plane they were flying. “We’ve had to expand our operating range from really focused on the highest frequency to encompass those six or seven orders of magnitude of sample rate… We started down this path about a year ago and let’s just say there was quite a bit of cleanup to do along the way,” said Brown.

Pressure from all sides

At the same time, adoption of the PredictiveGrid was accelerating. While it was initially pitched as an analytical sandbox for utilities to understand the complex demands they faced, early adopters, finding value in historical analytics, soon started relying on the platform for real-time use cases as well. “The thesis that all of your data matters, which every other industry has broadly adopted, is starting to resonate and so is the importance of what we’re doing. Your historical data is the driver for the creation of new, real-time use cases,” said Murphy.

“We’ve got some new customers that are starting to put some pretty stringent SLAs on us,” Brown noted. “We’re seeing more and more real time use cases pop up and when you do that, your availability has to go up as well. Instead of having two 9s or three 9s, now folks are saying, ‘we need this around the clock, we want four 9s of availability,’ a standard that’s typically been reserved for telecom services and other critical infrastructure.”

So PingThings found themselves under pressure from all sides, with exciting but imperative customer demands stacking onto what was already a complex rearchitecting of their system. The backend work was introducing bugs and creating issues in production, making on-call rotations increasingly fraught – but it was equally non-negotiable. With software quality suffering and fewer than 20 engineers on board, Murphy described “a situation that simply would not scale. Forget the number of engineers, the company hit a plateau. If we’d gotten five new customers, we would’ve died.”

Just knock the bugs off

PingThings was discovering, as many other companies have, that existing methods of testing software are simply not up to the demands of modern development, especially when you’re dealing with critical infrastructure.

“We might do, you know, 50, 75, hundred runs with our own test tools, but they didn’t uncover what we saw using Antithesis… being able to run a week’s worth of testing in a handful of > hours is a tremendous value. We’d run our test at night, then in the morning we’d get our report and we’d see what the bugs were and just knock them off.”

Antithesis allowed PingThings’ engineering team to uncover bugs in their distributed storage architecture, ensuring that it maintained correctness while PingThings carried out the aggressive performance optimizations they urgently needed. Antithesis testing helped the team find edge conditions around networking and race conditions threading that their normal integration and unit tests were unable to. On multiple occasions, it helped them find and fix innocuous looking changes that created potential for data corruption.

Brown and Murphy soon began to see the difference that the new workflow, as much as the step-change in test quality, made. Using Antithesis enabled the company to close out a series of “thorn in the side” issues that had been interrupting development work with manual interventions.

“That lower stress environment that engineers were able to work in allowed them to produce some of their best work, unlike the high stress environment we’d previously had. We know right away if something’s broken, and that’s hugely powerful.”

Murphy summed it up:

“Over the last year there’s been a transformation in the engineering group. With basically the same group of people, output is now significantly better. Antithesis has given us more confidence in what we’re delivering, lower stress development, more predictability… I look at Antithesis as a way of enabling us to scale, and that is just an enormous deal.”