Security at Antithesis: A Manifesto

At Antithesis, most of our security goals concern Tenants. A Tenant is the basic unit of security isolation. It comprises some resources (for example customer software, reports, and test configuration) and some Authorized Users (including both Antithesis employees and external users). Most of our customers run within a single Tenant, but we have the ability to offer multiple Tenants to a customer, for example if they have a secret project that they want isolated from their main Antithesis account.

We have the following goals for Tenants:

• Isolation: Any user or service without permissions for a Tenant will not be able to inspect or disrupt the operations of that Tenant.
• Confidentiality: Only Authorized Users of a Tenant should be able to read Tenant data.
• Integrity: Only Authorized Users of a Tenant should be able to write Tenant data.
• Availability: The availability of our products and services to a Tenant should not be compromised by actions of unauthorized individuals, or failures of another Tenant’s systems. This further implies that Tenants must be isolated from a performance standpoint, and that excessive load on one Tenant cannot result in an outage of another.
• Erasure: On the authorized request of a current or former Tenant, Antithesis should be able to reliably and permanently delete all Tenant data and/or the encryption keys required to decrypt all Tenant data.

Almost as important as our security goals are our security non-goals. At this time, our most significant non-goal is that we are not trying to provide robust, fine-grained permissions within a tenant. Of course we’re pragmatic about this, and when a customer asks us (for instance) to enable one of their employees to view reports without having the ability to launch tests, we’re game for building it. But this kind of intra-tenant permissioning is nowhere near as ironclad as the security between tenants, and we don’t make guarantees that it can’t be broken by dedicated, professional attacks.

Another good example of a security non-goal is that today we do not attempt to be robust against the subversion of our cloud providers. You can imagine a future Antithesis, perhaps using secure enclaves or perhaps only deploying in our own datacenters, that could make those sorts of promises, but that is not where we are today. In order to trust us, you need to trust AWS and GCP, which brings us to the next principle.

Antithesis comprises a vast amount of code, written in a plethora of languages, many of them memory unsafe. Statistically speaking, this code is full of bugs. If our security depended on all of this code being right, we would be utterly doomed, so we need a different approach.

Our response to this reality is to try to concentrate all of the security goals of our platform into a small and very clearly demarcated set of components. We refer to this as our Trusted Computing Base (TCB). Software that is not part of the TCB is untrusted, and should not be able to compromise our security goals even if it is actively malicious.

Here are some examples of things that must be in the TCB:

• Anything that has authorization to read or write data in more than one tenant (this follows directly from the security goals laid in the previous principle).
• Anything that sits on the boundary between our systems and the rest of the world.
• Any compiler or toolchain responsible for building the components of our TCB.
• Any library dependencies of anything in our TCB.
• Any operating system that our TCB components run on.

At this point you might be rolling your eyes, like presumably we aren’t auditing the code of our operating systems. Fair enough, we’re pragmatic about some of this, and aware of the limitations of how secure software can be in the year 2024. But actually we take this a good deal more seriously than you might expect. For instance, we do fanatically avoid taking third-party dependencies in our TCB components, and when we do, we audit their code to the best of our ability and then pin their versions so they don’t change under us.

Developing anything in the TCB at Antithesis is an extremely unpleasant experience. Even the most minor change you make generally involves multiple rounds of security reviews. If you try to import a new library, somebody will probably tell you “no”. You have zero discretion over what language you get to use (it’s almost all Rust). And for the absolutely most sensitive components, such as the proxy service that sits on our network boundary and handles all incoming requests, we actually maintain a security “proof” that the component cannot possibly subvert our security goals. Modifying these components often looks like a week or two of expanding the proof to accommodate whatever it is you want to do, then just a couple hours writing the actual code.

Developing the TCB is unpleasant so that developing everything else can be pleasant. An overwhelming goal of our architecture is to make the TCB as small as possible. This means we can be maximally meticulous about it, and that the process for changing it can be maximally annoying and onerous, without significantly impacting overall development velocity. We’re not saying we’re slapdash about everything else, but the knowledge that if you aren’t touching the TCB, then nothing you do can compromise our security goals is tremendously freeing. Most of our developers can worry about normal things like “does it work” and “is it performant”, without the nightmare possibility that a subtle bug could destroy every promise we’ve ever made.

An interesting consequence of all this is that it makes everybody naturally skeptical of third-party software and services that have to be trusted. At most companies it’s pretty easy to accidentally start depending on these. At Antithesis, adding a dependency usually triggers an argument, and over time we’ve moved in the direction of expunging trusted third-parties from our world. A counterintuitive consequence of this is that we don’t run antivirus programs on our production servers, as we’re pretty sure it would make them less secure.

Another beneficial consequence is that it exerts a steady pressure in the direction of architectural simplicity. Our developers want the TCB to be as small as possible, because they don’t want to go anywhere near it, because doing so will ruin their lives. This has played into major architectural choices that we’ll discuss under the next principle.

A great side-effect of our TCB-based approach to security is that it makes it very easy for security to be the default. Every system, service, API, or process in our entire company is tenant-scoped by default, because if it isn’t then somebody will argue that it’s part of the TCB and then the person developing that thing’s life will suck. This lets us avoid a very common security pathology of SaaS businesses.

Often, when you’re building a platform, security is the last thing on your mind. After all, you don’t know if anybody is even going to use the thing. So you build all your services in a multi-tenant fashion, with an architecture that looks like this:

One day, one of your customers asks about your security policies, and in a panic you retrofit some attempted isolation onto your existing architecture.

This is, to put it charitably, really hard to get right. You have accidentally made your entire software stack into your TCB. A bug almost anywhere could compromise your security goals. And if the individual components weren’t designed for security in the first place, there’s a good chance your ad-hoc security barriers have holes.

At Antithesis things work quite differently. We designed all of our systems to be single-tenant by default. Yes, that means we essentially wind up spinning up an entirely separate copy of our whole system architecture for each of our customers. At first that was a pain. Then we automated it and got really good at it. The resulting picture looks like this:

Anything that sits above, between, or within multiple tenants is definitionally part of the TCB, but for reasons we’ve discussed, there’s as little of that as possible.

This isn’t all picnics and roses, there are some things that this approach makes much harder. For example, “observability”, “analytics”, and other forms of spying on our customers. At many companies, enabling that stuff is a snap, because everybody is already in all the same systems. At Antithesis, the friction is much, much higher. This is a genuine cost of our security approach.

We consider any data produced by a tenant system to be radioactive unless proven otherwise. These days we do have systems that gingerly exfiltrate performance and other data from our tenant enclaves, but those systems are necessarily heavily reviewed and a pain to develop. Nothing they exfiltrate is permitted to contain any customer-produced strings. In some components, this is enforced at the type system.

Running so many copies of each of our services is also a bit of an operational headache, but on the plus side it gives us a certain amount of availability/uptime “blast radius” for free. It’s also made more bearable by the way we’ve applied the next principle.

Like many companies, we’ve gone all-in on infrastructure-as-code (IaC). The basic idea behind IaC is that you maintain a textual description of what your infrastructure ought to be, which is the true source of record for what is running in production. When you want to add a new service or delete an old one, you first update your textual description, and then run a process which gathers the current state of everything you have running, diffs it against the state described in your text files, and then starts or shuts down servers as needed to make the two things match exactly.

The most obvious advantage of this approach is that it’s basically impossible for somebody to accidentally leave a server running and forget about it. Every time any of our production configuration is modified (and also at the beginning and end of each workday as an extra precaution), we are reading the entire state of every machine and piece of cloud configuration and comparing it against what it should be. When people accidentally leave servers on (and this has happened), they stick out like a sore thumb in the resulting diff. It’s tremendously comforting to be able to read a file in source control and to know with perfect confidence that that’s exactly what’s running.

That gets rid of the problem where cloud environments are a big ball of mutable state that you can’t sort out without knowing the history of every operation ever performed on them. The problem with stopping there is that there’s another system which could be described that way: everybody’s computers. You thought it was easy to leave a server running? It’s even easier for you, or somebody unfriendly, to leave a process running on somebody’s server or on their developer machine.

So we’ve gone all the way, and embraced a fully-declarative, IaC-inspired operating system called NixOS. A machine running NixOS, like an IaC-managed cloud environment, is fully specified by a short and readable collection of text files. If you want to install a program or start a service, you edit these text files and run a command which… reads the whole configuration of your computer and diffs it against what it ought to be. It’s a bit of a pain to get started with, but the result is computing environments that are totally auditable, and where “works on my machine” is a thing of the past. As a bonus, since the full transitive closure of the software on your machine is known, it’s trivial to scan for vulnerabilities.

The true power comes from combining these two levels of IaC, because the result is totally fearless deployments. If you truly believe that there is no hidden state in your cloud because it’s all spelled out in declarative text files, and you truly believe that there is no hidden state on your servers or computers, because it’s all spelled out in declarative text files, then updating or making changes to your systems becomes much less scary. You obviously still want to test that the new configuration works, but you no longer need to live in fear that that server you’re considering replacing had some secret, undocumented, load-bearing other purpose. This in turn means that applying security patches and rebooting machines, etc. is something that we can do almost without thought, which makes reacting to zero-day vulnerabilities a lot less annoying.

Our developers and customer success (CS) engineers' normal accounts don’t have permission to access any tenant data or resources. To perform an operation that reads information belonging to a tenant, or which triggers some action on a tenant, our authorized employees may perform a tenant role assumption. This is a logged and audited operation that gives them the temporary ability to access that tenant. But the ability to perform this role assumption is itself locked down — CS engineers are only able to do it for the customers they’re working with directly, and our developers generally can’t do it at all unless they’re helping to debug an issue with our platform that’s affecting some particular customer. At any time, a customer can request a list of every employee at Antithesis who is able to access their tenant, and it’s usually between 2 and 5 people.

There’s a much smaller group of employees, at this time exactly 5 people, who are infrastructure admins and have the ability to modify our TCB and infrastructure at will. Importantly, however, their normal user accounts cannot do anything at all. To perform an administrative action, they must elevate to an admin role. This elevation requires a hardware TOTP second factor (not an MFA app on a computer capable of performing the elevation), and is sharply time-limited. This limits the damage that could be done by somebody who steals an admin’s computer. Every elevation to an admin role is logged, and the actions taken as an admin are also logged.

We follow similar patterns for our machine roles and service accounts. We avoid having accessible long-lived credentials anywhere in our system. Whenever possible, we take long-lived keys and lock them behind lambdas whose only job is generating short-lived derived credentials for those with the authority to invoke them. The goal throughout is that if or when an attacker is able to steal a credential, it won’t do much for them unless they also figure out how to somehow turn it into permanent access via a different mechanism.

Everybody hates process (well, most people who work at an early-stage startup do anyway), but it’s more bearable if you make a fun ritual out of it. For example, one of the rules in our security policy is that all employees must read the security policy at least once a year. So we make a company-wide calendar event and have a security policy reading party. Similarly, our security committee is required to meet at least once a quarter, and we ensure that those meetings are festive.

The quarterly security committee meeting is actually very valuable. One of the important things that happens at that meeting is the Tenant Access Review. We go through the entire list of tenants, and for each tenant the list of everybody who’s authorized to perform role assumption on that tenant, and if they don’t have a very good reason to continue having that power then we delete them. This is great for catching situations where somebody got authorized on a tenant to perform some debugging for a customer, or to do some unusual analysis, and nobody remembered to remove them.

We also make a little ritual out of the infrastructure changes that are performed by administrators. We’ll discuss this in the next section, but outside of dire emergencies, all infrastructure/TCB changes require signoff from more than one administrator. If it’s an anticipated change, this is handled asynchronously — one admin proposes the change, in writing, in a publicly viewable messaging system, and does not apply it until another one replies to sign off. If it’s an emergency change or exceptional debugging situation, the two administrators will gather at a computer, and solemnly enter their MFA keys together. This is great for reducing the temptation for an administrator to cut corners and do something on their own — it wouldn’t just be wrong, it would feel wrong. They would know in their guts that something was out of whack in the universe because the proper rituals are not being followed.

We also have very clear processes around how and when to notify customers in the event of a breach (our Breach Notification Policy), and how and when to notify open source projects if we discover security vulnerabilities in the course of testing customer software that relies on them (our Responsible Disclosure Policy). These are both situations where we could be tempted to bend the rules a little — in the first case, out of embarrassment; in the second case, because we come under pressure from the customer in question. Having a hard and fast rule, something written down, something that everybody is on the same page about, makes it easier to do the right thing when the stakes are high.

If we put ourselves in the shoes of an attacker, it’s immediately apparent that one of the softest targets in the company is our employees’ work computers. An attacker could try to gain access to one of our computers by bribing, blackmailing, or phishing an employee; or just by stealing the computer outright. The last possibility is mitigated by the fact that most of our employees work on desktops (yes, really), but we still need to worry about it.

The computers themselves are locked down. As mentioned above, we run NixOS on all of our machines, which enables us to keep them up to date very easily, to tell at a glance what the configuration of the machine is, and to rapidly push out new policies or configurations without having to tell everybody to take an action. We require that all work machines have full disk encryption, and enforce various login security requirements.

In the event that somebody did take control of one of our employees’ work machines, the damage they could do would be limited. The only tenants they could possibly attack would be the tenants that that employee happened to have access to. For most developers, this is zero of them. For most CS engineers, it’s the small handful of customers that they work with directly. In no case could compromising a machine give an attacker unfettered access to all of our customers.

If an attacker compromised an admin’s computer, they would not be able to take administrator actions without also compromising the same admin’s phone or hardware second factor. An attacker could try to expand their access by introducing a followup vulnerability into our production systems via a code push. But every change that affects our infrastructure or TCB requires the signoff of two or more infrastructure admins. A change that did not go through the normal review channels would be noticed the next time an admin performed an IaC infrastructure deployment.

One of the most devious possible attacks would be for an attacker to sign up as a customer, receive their tenant, and then use the software deployed in their tenant to attack another tenant through one of our CS engineer's workstations. Today this attack vector is mitigated by the fact that we work closely with all of our customers, and by some basic sandboxing (using Linux namespaces) of workstation commands that could be affected by the contents of a tenant. Our longer-term plan is to significantly harden this sandboxing and to treat it as a fully-fledged part of our TCB, but our efforts here are still early.

Every change affecting our TCB requires security review, and any change to our infrastructure requires multiple admins to approve it. This is important, because even the best-intentioned changes made by the smartest people can have bugs in them. We try to cultivate the mindset of rejoicing that somebody else is checking our work. Finally, many security properties of our system are under test within the Antithesis platform itself. We use Antithesis to test Antithesis.

But we also believe in external accountability. We hire professional penetration testers at least annually, and give them carte blanche to try and break our security. One test starts by assuming that particular resources have already been compromised, and asks them to see what they can do from there. The results of our penetration tests are available to customers by request on our trust center. We are also SOC 2 certified, and sailed through our most recent audit (also available at that link). The only thing that gave the auditors heartburn was that we conduct annual reviews by taking our employees to lunch, instead of trapping them in a conference room with paperwork. Going forward, we'll do both. :)