TLA+ Explained: How Amazon Proves Distributed Designs Correct Before Writing Code
You have probably read about Raft, Paxos and the FLP impossibility result. However, there is a quieter question behind all of them: how do you know your own design is correct before anyone writes a line of code? Amazon’s answer is a specification language called TLA+, and the story behind it is surprisingly practical.
Why Testing Alone Misses the Rare Bugs
First, consider the problem Amazon faced. AWS engineers already used design reviews, code reviews, static analysis, stress testing and fault injection. Even so, their own paper explains that subtle bugs still hid in complex, fault-tolerant systems. Human intuition is simply poor at judging how likely an “extremely rare” combination of events is when a system handles millions of requests per second.
Moreover, many of these bugs live in the design, not in the code. The code faithfully implements the plan, but the plan fails in one unusual scenario. Testing cannot reach it, because the number of reachable states is astronomical. Consequently, the engineers went looking for a way to check the design itself.
What TLA+ Actually Is
TLA+ was created by Leslie Lamport, the same researcher behind Paxos. According to Wikipedia’s summary, the underlying logic appeared in 1994, the language itself in 1999, and Lamport’s book Specifying Systems followed in 2002. The language rests on plain set theory and predicates, so most engineers already know the maths involved.
In practice, a TLA+ specification describes every legal behavior of a system. Then the TLC model checker explores all of those behaviors and tests your correctness properties. The AWS team calls this “exhaustively testable pseudo-code”, which is a fair description. In addition, a companion language called PlusCal looks more like ordinary pseudo-code, and several Amazon engineers found it easier to start with.
Two kinds of properties matter here. Safety means nothing bad ever happens, for instance that committed data is never lost. Liveness means something good eventually happens, for instance that every request gets an answer.
A Tiny Spec You Can Run in Two Minutes
Next, let us see TLC catch a real bug. Two processes each try to increment a shared counter. Each one reads the value first and writes back the value plus one. The rule we want is simple: once both are done, the counter must equal 2.
You only need a recent Java runtime. The block below downloads the TLA+ tools, writes the spec and runs the checker. I ran this exact block on a clean directory with OpenJDK 21 and TLC 1.7.4.
mkdir tla-demo && cd tla-demo
curl -sL -o tla2tools.jar https://github.com/tlaplus/tlaplus/releases/latest/download/tla2tools.jar
cat > LostUpdate.tla <<'EOF'
---- MODULE LostUpdate ----
EXTENDS Naturals
CONSTANT N
VARIABLES counter, pc, tmp
vars == <<counter, pc, tmp>>
Procs == 1..N
Init == /\ counter = 0
/\ pc = [p \in Procs |-> "read"]
/\ tmp = [p \in Procs |-> 0]
Read(p) == /\ pc[p] = "read"
/\ tmp' = [tmp EXCEPT ![p] = counter]
/\ pc' = [pc EXCEPT ![p] = "write"]
/\ UNCHANGED counter
Write(p) == /\ pc[p] = "write"
/\ counter' = tmp[p] + 1
/\ pc' = [pc EXCEPT ![p] = "done"]
/\ UNCHANGED tmp
Next == \E p \in Procs : Read(p) \/ Write(p)
Spec == Init /\ [][Next]_vars
AllDone == \A p \in Procs : pc[p] = "done"
Correct == AllDone => counter = N
====
EOF
printf 'SPECIFICATION Spec\nCONSTANT N = 2\nINVARIANT Correct\n' > LostUpdate.cfg
java -cp tla2tools.jar tlc2.TLC -deadlock LostUpdate.tla
TLC stops almost instantly with “Invariant Correct is violated” and prints the failing run. The trace is only five states long, and it is the classic lost update:
| Step | What happened | counter | Process 1 | Process 2 |
|---|---|---|---|---|
| 1 | Start | 0 | read | read |
| 2 | Process 1 reads 0 | 0 | write | read |
| 3 | Process 2 also reads 0 | 0 | write | write |
| 4 | Process 1 writes 0 + 1 | 1 | done | write |
| 5 | Process 2 writes 0 + 1 | 1 | done | done |
Why the State Space Matters
Now compare the racy version with a fixed one where the read and write happen in a single atomic step. I also removed the invariant so that TLC explored every reachable state. As a result, the difference in size is striking.
The atomic version simply doubles with every process (4, 8, 16, 32, 64). The racy version grows from 13 states to 118,509 by N = 6. That jump is exactly why humans miss these bugs: every extra interleaving is one more path nobody thought to test.
What Amazon Found With It
Since 2011, according to the CACM version of the paper, AWS engineers have used TLA+ and model checking on critical designs. By the time of writing, they had applied it to 10 large systems and had 7 teams using it. Importantly, engineers from entry level to Principal learned it from scratch and got useful results in two to three weeks.
| Component | Lines | Language | Result |
|---|---|---|---|
| S3: fault-tolerant network algorithm | 804 | PlusCal | 2 bugs, plus more in proposed optimizations |
| S3: background data redistribution | 645 | PlusCal | 1 bug, and a bug in the first proposed fix |
| DynamoDB: replication and membership | 939 | TLA+ | 3 bugs, some needing 35-step traces |
| EBS: volume management | 102 | PlusCal | 3 bugs |
| Lock manager: lock-free data structure | 223 | PlusCal | More confidence, but a liveness bug was missed |
| Lock manager: replication algorithm | 318 | TLA+ | 1 bug, and an aggressive optimization verified |
Notice how small these specifications are. Even the largest has fewer than a thousand lines, while the production systems they describe are far bigger. The DynamoDB story shows why that is enough. The model checker found a data-loss bug whose shortest failing trace had 35 high-level steps. It had passed design reviews, code reviews and testing.
Likewise, an S3 engineer wrote his first spec for an algorithm with a known bug that had surfaced only after months of testing. After two weeks of learning TLA+, he watched TLC find the bug in a few seconds. Then the team’s proposed fix failed too, through a different trace, and a stronger fix passed.
It Is Not Just Amazon
Other teams publish similar experiences. Microsoft documents the five Azure Cosmos DB consistency levels with TLA+ specifications in a public repository. A later paper describes a client-focused spec of only 390 lines and uses it to explain the root cause of a 28-day outage inside Azure.
MongoDB has specified its replication protocol, initial sync and locking rules, and it reported its results honestly in “eXtreme Modelling in Practice”. Trace-checking the server against its spec proved impractical, whereas generating tests from a spec worked well for Realm Sync. Meanwhile, Diego Ongaro’s 2014 Raft spec sits in the TLA+ examples repository. In 2023, the Linux Foundation also launched the TLA+ Foundation, with Microsoft, AWS and Oracle as founding members.
Where This Meets Consensus and FLP
Here the topic connects to what many of you already know. The FLP result shows that no consensus protocol in a fully asynchronous system can promise termination if even one process may fail. In other words, it is a liveness result: the protocol may never decide, but nothing says it decides wrongly.
That distinction is precisely the safety and liveness split above. Model checkers handle safety well, because they search for a single bad state. Liveness is harder and needs explicit fairness assumptions, which is why the Amazon lock manager spec missed a liveness bug: nobody had told TLC to check it. Interestingly, the Amazon author first met TLA+ in an appendix to a Paxos paper by Lamport.
The Honest Limits
Of course, TLA+ is not magic. Amazon’s own answer to “does the code match the spec?” is that they do not know. A model also covers only small configurations, since TLC checks a bounded instance. Finally, the paper says it cannot predict emergent slowdowns such as retry storms, which involve no logic bug at all. Even so, getting the design right is a necessary first step.
Checklist: is TLA+ worth trying on your next design?
- The design involves concurrency, replication or failure recovery.
- A bug would damage or lose data, not just annoy a user.
- You can state the property in one sentence, such as “committed data is never lost”.
- Reviews and tests feel necessary but not sufficient.
What We Learned
In short, subtle design bugs hide in rare interleavings that reviews and tests rarely reach. TLA+ lets you describe a design in a few hundred lines and have TLC check every reachable state, which is how Amazon found a 35-step DynamoDB bug and how our tiny counter spec exposed a lost update in five steps. Microsoft and MongoDB report similar benefits. However, the spec is a model, not the code, and it checks only the properties and sizes you give it. Start small, write down what must stay true, and let the checker surprise you.
