Property-based testing – how it works and when to use it

This article is co-written with David MacIver, who created Hypothesis, the property-based testing library for Python. At Antithesis, David is now working on Hegel, a family of property-based testing libraries that aims to bring Hypothesis-quality testing to every language.

Property-based testing (PBT) is an approach to software testing where you describe the properties that should always be true of your software, and then check them against many generated inputs. It finds the failures you didn’t anticipate, so you can take on complex projects with confidence.

This article explores how property-based testing works, how to implement it effectively, and which kinds of systems benefit most from it.

What is property-based testing?

To understand what property-based testing is, it’s easiest to first compare it to the common approach of example-based testing. In example-based testing, each test sets up a single example with specific inputs.

For example, say that you’re testing the login page of a website. You want to check that a user can log in to the site with a username, email, and password. An example-based test would use a hard-coded string for each of these fields.

In property-based testing, you instead ask what properties each field should have. For example, you might specify that the username can be any non-empty string, or that the email must match a valid email address pattern. Then you try logging in multiple times with random data that fits these properties, and see if it ever fails.

Both the example-based test and the property-based test might claim to test that “a user can log in with a valid username, email, and password,” but the example-based test only actually tests that “a user can log in with a username of alice, an email of alice@example.com and a password of password1.” The property-based test can throw random data at the login page and see if it holds up to situations you didn’t think of, like obscure Unicode characters in the password.

For a longer introduction, see David MacIver’s In praise of property-based testing.

As well as “property-based testing,” you may hear the terms “fuzzing” or “fuzz testing.” Property-based testing and fuzzing are very similar ideas, but they come from somewhat different traditions and have a difference in emphasis.

Property-based testing was popularized by the Haskell library QuickCheck. There are now property-based testing libraries for most common languages – examples include Hypothesis for Python and FsCheck for C# and other .NET languages.

These libraries are developer tools that give you rapid feedback on software you are writing. They assume that you have a fairly good understanding of your software, so you can write tests that are specifically tailored to it. If you find a bug, you will most likely fix it yourself as part of the development process.

Property-based tests normally generate valid input. In the login example above, the test generated valid usernames, passwords, and email addresses.

Fuzzers like AFL are normally run as a long-running process. Rather than generating inputs completely at random, they are usually coverage-guided: they track which parts of your code each input reaches, and build on inputs that reach somewhere new. They’re generally used in situations where finding a bug is high value, so a greater effort-to-bug ratio is acceptable.

Fuzzing is often used on other people’s software, where you have less understanding of the details and instead look for more generic property failures like crashes or security issues. If you find a bug with a fuzzer, you are more likely to report it upstream (or exploit it!) than fix it yourself.

In the login form example, a fuzzer is likely to send invalid data as well as valid data, and check that the site doesn’t crash or behave insecurely in response.

Antithesis combines elements of both traditions. It has the long-running guided exploration of a fuzzer, but the expressivity of property-based testing — you can write assertions that test specific properties of your software. Applying property-based testing to complex systems is not a new idea — for example, Quviq’s Erlang QuickCheck uses stateful property-based tests to find bugs in concurrent and distributed software — but Antithesis makes it available for systems written in any language, by testing the whole system in a simulated environment.

You may also come across the term “generative testing,” which is a looser term that’s been used to include both fuzzing and property-based testing. In the rest of this article, we’ll use “property-based testing” as a generic term for all these approaches.

How do I implement property-based testing?

How you implement property-based testing depends on the type of tool you’re using:

Property-based testing libraries like Hypothesis and QuickCheck are integrated into your normal testing workflow. You add your tests in the same place as your unit tests, and run them in CI. This is also the default way to use Hegel.
With fuzzers like AFL, you normally write a harness: a program that takes the fuzzer’s generated input and passes it to the code you want to test. You compile it with instrumentation so the fuzzer can track coverage, then leave it running for hours or days to look for inputs that cause crashes or hangs. This usually runs separately from your normal test suite.
Antithesis applies property-based testing to your whole software system, including your program and the environment it runs in. To use Antithesis, you package your system along with its dependencies and use language-specific SDKs to specify the properties you want to test. The Antithesis fuzzer then explores your system’s behavior while injecting faults such as network failures and crashed nodes. Because Antithesis uses deterministic simulation testing, you can reliably reproduce any bug that you see on a given run.

We want Hegel to scale through all of these modes of interaction. By default, you use it for rapid feedback like any other property-based testing library, but we also support long-running workloads and running inside Antithesis.

How does property-based testing work?

Property-based testing repeatedly executes your tests with a variety of generated data, to test whether the general claim you made holds up in circumstances you wouldn’t necessarily have thought about.

In libraries like Hypothesis and QuickCheck, the data is normally randomly generated. In longer-running systems like fuzzers, input generation is also coverage-guided, so the fuzzer can explore your code more thoroughly.

Once property-based testing finds a bug, there is often a shrinking process that looks for a simpler version of the bug that also makes the test fail. For example, if a list fails, it will look for a shorter list that also fails.

What are the strengths and limitations of property-based testing?

Property-based testing has a lot of advantages:

It makes you more confident about the software you’re writing. You can take on complex tasks and trust that your tools are going to be good at finding bugs.
Random choice of inputs can find “unknown unknowns” that you didn’t think to test for deliberately.
It’s great for testing interactions. Writing example-based tests that each test one thing is not so bad, but if there are multiple things you need to test in different combinations, the number of tests you need explodes.
Properties tend to be more enduring, higher-level features of the software than specific examples. This makes tests more maintainable as they are more resilient to software changes.
Even before you run your tests, thinking about what properties your software should guarantee can help you write better software. Software with clearly defined properties is easier to reason about and debug. For some common patterns, see Scott Wlaschin’s Choosing properties for property-based testing.

However, there are some limitations that you should be aware of:

Property-based testing libraries run each test a lot of times, so if your tests are slow this will take a while.
Getting started is pretty easy, but it takes skill to get good at generating data and finding good properties to test.
Sometimes the thing you’re testing is so specific that you may be best off with an example-based test. For example, snapshot tests run some code and check that it gives exactly the same result as last time. This works well when the correct output is hard to describe as a general rule, such as a rendered web page or a formatted error message, so the easiest thing is to check the exact result.
There may be extra setup that you need to do before you can use your property-based testing tool, such as mocking external dependencies.

What kinds of systems benefit most from property-based testing?

Property-based testing is especially useful if your software is complex or fiddly. Almost all software has some parts that are complex and fiddly! For example, any software that works with timezones, Unicode, or floating point arithmetic will have a lot of annoying details and edge cases, and property-based testing is great for finding bugs in it.

Beyond that, property-based testing is a particularly strong approach in domains where you have expected properties that you need to guarantee. For example, databases have consistency models that define what they are allowed to do.

Antithesis customers use property-based testing to check guarantees like these in a wide range of systems:

Databases and storage systems like MongoDB, FoundationDB, and Tigris.
Message queues like NATS.
Key-value stores like etcd.
Financial transaction engines like Formance.
Blockchain platforms like Ethereum and Sui.
Systems undergoing optimization work or rewrites, where you need to check that the new system satisfies the same properties as the old one, as at PingThings and Datadog.

For examples of properties you might want to test in your own system, see our system-specific guides.

Conclusion

Property-based testing lets you write a smaller number of more powerful, resilient tests than the traditional example-based approach, saving developer time and uncovering bugs in scenarios you didn’t think to test for directly.

Introducing property-based testing to your codebase can take some upfront work, but you can expect higher developer productivity and better confidence in your software.

Antithesis offers a powerful property-based testing platform for distributed systems. Book a demo for a walkthrough of how we test the systems no one else can.