An interactive walk through Chapter 9 of Designing Data-Intensive Applications (2nd ed.). This chapter is about everything that can go wrong once software runs on more than one machine: networks, clocks, paused processes, and nodes that can’t trust their own view of the world. Every section has something to play with. Press Next ▶ in the step-through demos, and drag the sliders in the labs.
A reliable system keeps working as a whole even when parts of it fail. It’s tempting to code for the happy path, because things usually work. To build something reliable you have to flip your mindset and think about what could go wrong, however unlikely, because at scale unlikely things happen all the time.
Distributed systems add a whole new set of things that can go wrong. This chapter turns pessimism up to the maximum: the network, clocks, and pauses in a process’s execution. Then it looks at how to reason about a system where no node can be sure of anything. Chapter 10 turns to solutions.
Click any node to jump there.
Deterministic: the same operation gives the same result. When hardware fails, the usual result is a total failure (kernel panic, blue screen) rather than a wrong answer. That’s a deliberate design choice: crashing is easier to deal with than being subtly wrong. The computer presents an idealized model of mathematical perfection. (Silent corruption does happen, but rarely enough to ignore.)
Some parts can be broken in unpredictable ways while others work fine: a partial failure. Partial failures are nondeterministic: the same operation sometimes works and sometimes doesn’t, and you may not even know whether it succeeded.
The physical world has lots of ways to fail. Coda Hale’s list from his experience includes long network partitions inside one datacenter, power distribution unit failures, switch failures, whole racks power-cycled by accident, datacenter-wide backbone and power failures, and a driver crashing a pickup truck into a datacenter’s air conditioning.
This book focuses on shared-nothing systems: machines with their own memory and disk that talk only over a network. (Mainframes instead made individual components redundant, e.g. RAID; shared-nothing systems use replication across machines.) Even “shared” storage such as object stores is reached over the network.
The internet and most datacenter networks are asynchronous packet networks: you can send a packet, but nothing guarantees when it arrives, or whether it arrives at all.
If you send a request and get no answer, any of these might have happened:
The sender can’t even tell whether its packet was delivered. The only way to find out is a reply, and the reply can be lost or delayed too. From the sender’s side these cases are indistinguishable: all it knows is that no response has arrived yet.
The usual remedy is a timeout: after a while, give up and assume no response is coming. Even then you don’t know whether the remote node got the request, and a request still sitting in a queue may yet be delivered after you gave up.
Packets are small (a few kB), so applications use TCP to turn a byte stream into packets and back. (Most of this applies equally to QUIC, SCTP as used in WebRTC, BitTorrent’s uTP, and similar protocols.) TCP is called “reliable” because it:
Writing to a socket only puts data into an OS buffer. Congestion control decides when to send each packet; the packet crosses switches and routers; the receiving OS puts it in a receive buffer and sends an acknowledgment; only then is the receiving application told new data is there.
The network is still unreliable underneath TCP:
TCP is still very useful: it carries messages bigger than one packet, and one connection can carry many requests and responses, typically each with a length header followed by the message (HTTP and many RPC protocols work like this).
Decades of networking haven’t made networks reliable, even inside one company’s datacenter:
A network partition (netsplit) is when part of the network is cut off from the rest. It’s not fundamentally different from other network interruptions. (It has nothing to do with sharding, which is also sometimes called partitioning.)
Handling a fault doesn’t have to mean tolerating it. If your network is usually reliable, showing users an error during a problem is a valid choice. What matters is that you know how the software behaves and that it recovers. Deliberately triggering network problems (fault injection) is how you find out.
Load balancers need to take dead nodes out of rotation, and single-leader databases need to promote a follower when the leader dies. But network uncertainty makes it hard to tell if a node works. Sometimes you get an explicit signal:
| Signal | When you get it | Limits |
|---|---|---|
RST / FIN packet | The machine is reachable but nothing listens on the port (the process crashed). The OS closes or refuses the connection. | Only if the OS and network are fine. |
| Crash notification script | The process died but the OS is up; a script tells other nodes so they can take over without waiting for a timeout (HBase does this). | Useless if the whole machine is down. |
| Switch management interface | Query the switches for link failures (e.g. the machine is powered off). | Not available over the internet, in shared datacenters, or when the management network itself is unreachable. |
| ICMP Destination Unreachable | A router is sure the address can’t be reached. | Routers have no magic failure detection either. |
You can’t count on any of these. In general you get no response at all. You retry a few times, wait for a timeout, and declare the node dead if you hear nothing. Since it might actually be alive, you’re trading false positives against false negatives.
A long wait before a dead node is declared dead. Users wait or see errors meanwhile.
Faster detection, but healthy nodes that just had a temporary slowdown (a load spike) get declared dead.
Declaring a node dead too early is harmful:
Imagine a network where every packet is delivered within time d or lost, and every working node handles a request within time r. Then every successful request gets its answer within 2d + r, and that would make a perfect timeout. Real systems have neither guarantee: asynchronous networks have unbounded delays (they try to be fast, but there’s no upper limit), and most servers can’t guarantee a maximum handling time. It isn’t enough to be fast most of the time: with a low timeout, one transient spike in round-trip time throws the system off balance.
As with traffic on roads, most of the variation in packet delay comes from queueing:
When TCP retransmits a lost packet, the application doesn’t see the loss, only the delay: waiting for the retransmission timeout, then for the retransmitted packet to be acknowledged.
Latency-sensitive applications like video calls and VoIP often use UDP. UDP has no flow control and no retransmission, so it avoids some causes of delay variability (it is still subject to switch queues and scheduling delays). It makes sense when late data is worthless: in a phone call there’s no time to retransmit a lost packet before its audio is due, so the app plays a moment of silence and moves on. The retry happens at the human layer: “Sorry, could you repeat that?”
Queueing delays vary most when a system is near capacity: spare capacity drains queues quickly, while a busy system builds long queues fast. In public clouds and multitenant datacenters everything is shared: links, switches, NICs, even CPUs when you run in VMs. A noisy neighbor saturating shared resources can make your delays highly variable, and you can’t see or control their usage.
So timeouts can only be chosen experimentally: measure round-trip times across many machines over a long period, then pick a trade-off between detection delay and false alarms. Better still, measure response times and their variability (jitter) continuously and adjust timeouts automatically. The Phi Accrual failure detector (used in Akka and Cassandra) does this: instead of a yes/no timeout it outputs a suspicion level φ, roughly −log₁₀ of the probability that a healthy node would be this slow. TCP’s retransmission timeouts adapt similarly. Try “Adaptive (φ accrual)” in the timeout lab above and raise the load: the timeout moves with the distribution.
Why not fix this in hardware? The fixed-line telephone network is extremely reliable. A call opens a circuit: a fixed amount of bandwidth reserved along the entire route for the duration of the call. An ISDN line runs at 4,000 frames per second, and a call gets 16 bits in each frame in each direction, so each side may send exactly 16 bits every 250 µs. That network is synchronous: data doesn’t queue, because its space in the next hop is already reserved, so the maximum end-to-end latency is fixed. This is a bounded delay.
A TCP connection is the opposite. It reserves nothing, grabs whatever bandwidth is available, finishes as fast as it can, and uses nothing while idle (beyond the occasional keepalive). Ethernet and IP are packet-switched: no circuits, so queueing and unbounded delays.
The reason is bursty traffic. A call needs a steady bit rate; a web page, email or file transfer just wants to finish as quickly as possible. On a circuit you’d have to guess the bandwidth: too low and the transfer is needlessly slow while capacity sits idle; too high and the circuit can’t be set up at all. TCP adapts to whatever capacity is available.
Variable delay is a consequence of dynamic resource partitioning. A wire between two telephone switches might carry 10,000 calls. It’s divided statically: your call gets the same slice even if the other 9,999 slots are empty. The internet shares bandwidth dynamically: senders compete for the wire and switches decide packet by packet. That causes queueing, but it maximizes utilization, and since the wire has a fixed cost, better utilization means cheaper bytes.
CPUs work the same way. Sharing a core dynamically between threads means a thread sometimes waits in the run queue, but it uses the hardware better than giving each thread fixed cycles. That’s also why clouds pack several customers’ VMs onto one machine.
QoS isn’t enabled in multitenant datacenters, public clouds or across the internet today. So the working assumption is: congestion, queueing and unbounded delays will happen, and there is no “correct” timeout. It has to be found by experiment.
Messages take time to cross the network: a message is always received after it was sent, but variable delay means you don’t know how much later. And each machine has its own clock, usually a quartz crystal oscillator, which runs slightly fast or slow. Clocks can be synchronized to a degree, most commonly with NTP (Network Time Protocol), which adjusts a computer’s clock using a group of servers, which in turn get their time from something more accurate such as a GPS receiver.
Applications ask clocks two different kinds of question. Sort these:
A time-of-day (wall-clock) clock returns the current date and time: Linux clock_gettime(CLOCK_REALTIME) or Java System.currentTimeMillis() give seconds or milliseconds since the epoch (midnight UTC, 1 January 1970, not counting leap seconds). (“Realtime” here has nothing to do with real-time operating systems.)
A monotonic clock (CLOCK_MONOTONIC or CLOCK_BOOTTIME on Linux, Java System.nanoTime()) is guaranteed to only move forward. It’s a stopwatch, not a wall clock:
Monotonic clocks need no synchronization, but time-of-day clocks are only useful if they’re set correctly, and that is less reliable than you might hope:
Very good accuracy is possible if you invest enough. The EU’s MiFID II rules require high-frequency trading firms to stay within 100 µs of UTC (to help investigate flash crashes and detect market manipulation). That takes GPS receivers and/or atomic clocks, the Precision Time Protocol (PTP), and careful deployment and monitoring. GPS alone is risky, since signals are easily jammed (frequently, near military facilities). Some cloud providers now offer high-accuracy sync for VMs, but a misconfigured NTP daemon or a firewall rule can still quietly let errors grow large.
Clocks look simple but have many pitfalls: a day may not have exactly 86,400 seconds, time-of-day clocks can go backwards, and two nodes can disagree considerably. As with the network, robust software has to be prepared for wrong clocks.
It’s tempting, but dangerous, to use clocks to decide which of two writes on different nodes came first. In the figure-9-3 scenario the clocks are synchronized better than most clusters manage, and it still loses data:
Last write wins (LWW) keeps the write with the greatest timestamp per key and drops the rest. The fix in the demo, always giving an overwrite a higher timestamp than the value it overwrites, costs an extra read. Cassandra and ScyllaDB skip that read and use the client’s clock with LWW, which has serious problems:
Even with tight NTP, a packet sent at 100 ms on the sender’s clock can arrive at 99 ms on the receiver’s, apparently before it was sent. NTP can’t fix this, because its own accuracy is limited by network round-trip time; correct ordering would need clock error much smaller than network delay. Logical clocks, based on incrementing counters, are the safer way to order events: they capture only before/after, not time of day. Time-of-day and monotonic clocks are physical clocks. (More on logical clocks in chapter 10.)
A clock may report microseconds, but that doesn’t make it accurate to microseconds. Quartz drift alone can easily be several milliseconds even with LAN NTP every minute; over the public internet, tens of milliseconds at best, spiking past 100 ms under congestion. A reading is better thought of as a range: “95% sure it’s between 10.3 and 10.5 seconds past the minute”. With ±100 ms of uncertainty, the microsecond digits are meaningless.
The uncertainty can be estimated: for a GPS or atomic clock attached to the machine, from the device and signal quality; when syncing from a server, roughly drift since the last sync + the server’s own uncertainty + the network round-trip time. Most APIs hide it: clock_gettime doesn’t say whether its error is five milliseconds or five years. Google Spanner’s TrueTime and Amazon’s ClockBound do report it, returning [earliest, latest]; the interval widens with time since the last sync.
Snapshot isolation (chapter 8) gives each read-only transaction a consistent snapshot using MVCC and monotonically increasing transaction IDs. On one node a counter will do. Across many machines and datacenters, a global ID has to reflect causality (if B reads or overwrites A’s write, B needs a larger ID), and handing out such IDs centrally becomes a bottleneck with many small, fast transactions.
Could synchronized timestamps serve as transaction IDs? Spanner does this, using intervals. If two intervals don’t overlap (Aearliest < Alatest < Bearliest < Blatest), B definitely happened after A. Only overlapping intervals leave the order in doubt. So Spanner waits out the length of the confidence interval before committing a read-write transaction. To keep that wait short it needs small uncertainty, so Google puts GPS receivers or atomic clocks in every datacenter, achieving about 7 ms. The precise clocks aren’t strictly required; the interval is what matters. YugabyteDB can use ClockBound on AWS, and other systems increasingly rely on clock synchronization.
In a database with one leader per shard, how does a node know it’s still the leader and may accept writes? One option is a lease: like a lock with a timeout, held by one node at a time. The leader renews it periodically; if it fails, it stops renewing and someone else takes over after expiry. A request loop might look like the code in the lab below. It has two problems:
| Cause | What happens |
|---|---|
| Lock / queue contention | Threads wait on shared resources; often worse with more cores, and hard to diagnose. |
| Garbage collection | Runtimes like the JVM sometimes stop all threads. “Stop-the-world” pauses once lasted minutes; modern collectors are much better but pauses can still be noticeable. |
| VM suspend / live migration | The hypervisor pauses the whole VM (saving memory to disk) and resumes it later; for migration, the pause depends on how fast processes write to memory. |
| End-user devices | Laptops and phones suspend whenever the lid closes. |
| Context switches / steal time | The OS or hypervisor switches away at any point; under heavy load the thread may wait in a long run queue. CPU time spent in other VMs is steal time. |
| Synchronous disk I/O | Waits for a slow disk, sometimes unexpectedly (Java lazily loads class files on first use). With network block storage like EBS, network variability adds up. I/O and GC pauses can compound. |
| Swapping (paging) | A memory access triggers a page fault that loads from disk. Under pressure the OS may spend most of its time swapping (thrashing), so servers often disable paging and prefer to kill a process. |
SIGSTOP | Ctrl-Z stops a process until SIGCONT. An operator can send it by accident. |
All of these can preempt a thread at any point and resume it later without it noticing. It’s like making code thread-safe on one machine, where you can’t assume anything about timing. But on one machine you have mutexes, semaphores, atomic counters, lock-free structures and blocking queues. A distributed system has no shared memory, only messages over an unreliable network, so those tools don’t carry over.
Pauses can be eliminated with enough effort. Systems that control physical things (aircraft, rockets, robots, cars) must meet deadlines: you don’t want an airbag delayed by a GC pause. These are hard real-time systems, where a missed deadline can mean failure of the whole system. (Not the loose web sense of “real-time” for pushing data to clients.) They need:
It’s expensive, severely restricts languages and tools, and is mostly used in safety-critical embedded devices. Real-time also isn’t high performance: it may have lower throughput because timeliness comes first. For most server-side systems it isn’t economical, so they live with pauses and clock instability.
A well-tuned modern collector usually pauses for at most a few milliseconds. Java offers CMS, G1, ZGC, Epsilon and Shenandoah for different memory profiles; Go has a simpler self-tuning concurrent mark-and-sweep. To avoid GC entirely, use a language without it: Swift uses automatic reference counting; Rust and Mojo track lifetimes in the type system. Within a GC language you can pool and reuse objects, or allocate off-heap.
A more radical trick treats each GC pause as a brief planned outage:
These measures reduce the impact of GC pauses but can’t prevent them completely.
So far: no shared memory, an unreliable network with variable delay, partial failures, unreliable clocks, and pauses. The consequence is disorienting. A node can’t know anything for sure about other nodes. It can only guess from the messages it receives, or doesn’t receive. If a node stops responding, there’s no way to tell a network problem from a node problem.
Rather than philosophize, we state our assumptions as a system model and design algorithms that are provably correct within it. Reliable behavior is achievable even when the model guarantees very little, but it isn’t easy.
Three nightmares:
The moral: a node can’t trust its own judgment of the situation, and the system can’t rely on any single node. So many algorithms use a quorum: decisions need a minimum number of votes. That includes declaring nodes dead. If a quorum says a node is dead, it is dead, and it must step down however alive it feels. The usual quorum is an absolute majority: with 3 nodes, 1 can fail; with 5, 2 can fail. It’s safe because only one majority can exist at a time, so two conflicting decisions can’t both win.
Leases are used whenever there must be only one of something:
Think about what happens if two nodes both believe they hold the lease. In the third case it wastes some computing. In the first two it loses or corrupts data. Here, a storage service lets one client at a time write a file, guarded by a lease from a lock service (often built on consensus, chapter 10):
A zombie is a former leaseholder that hasn’t realized it lost the lease and still acts as if it holds it. You can’t rule zombies out, so you have to make sure they can’t do damage: fencing them off.
Other names: Chubby (Google’s lock service) calls them sequencers, Kafka epoch numbers, Paxos ballot numbers, Raft term numbers. With ZooKeeper you can use zxid or the node version cversion; with etcd the revision plus lease ID; Hazelcast’s FencedLock generates one explicitly. Storage must be able to check tokens, or at least support a conditional write that succeeds only if nobody else wrote since you read (like an atomic CAS): Amazon S3 conditional writes, Azure Blob Storage conditional headers, Google Cloud Storage request preconditions.
If clients only write to one storage service that supports conditional writes, the lock service is somewhat redundant, since leases could be built on that service directly. A token, though, can protect many services or replicas. In a leaderless store with last-write-wins on client-supplied timestamps, put the token in the most significant digits of the timestamp:
It is not safe to assume only one node holds a lease at a time. With fencing tokens, zombies and delayed requests can’t do damage.
Fencing stops nodes that are wrong by accident. A node that wanted to subvert the system could simply send a fake token. This book assumes nodes are unreliable but honest: slow, unresponsive, outdated, but when they do answer, they play by the rules as far as they know. A node that may lie (send arbitrary or corrupted responses, or cast contradictory votes) has a Byzantine fault, and reaching agreement despite such nodes is the Byzantine Generals Problem.
It generalizes the two generals problem: two armies must agree on a plan, but can only communicate by messengers who may be delayed or captured.
In the Byzantine version, n generals must agree, and an unknown number of traitors send false messages:
The name: Byzantium (later Constantinople, now Istanbul) had no more traitors than anywhere else. “Byzantine” here means excessively complicated or devious. Lamport wanted a nationality that wouldn’t offend readers, and was told “The Albanian Generals Problem” was a bad idea.
| Where Byzantine fault tolerance matters | Where it doesn’t help much |
|---|---|
| Aerospace: radiation can corrupt memory or registers, so a node behaves arbitrarily; failure is catastrophic, so flight control must tolerate it. Mutually untrusting parties: participants may cheat. Cryptocurrencies and blockchains let strangers agree on transactions without a central authority. Peer-to-peer networks have no central authority either. | Your datacenter: you control every node, and radiation is low (orbital datacenters are being considered, though). Multitenant isolation uses firewalls, virtualization and access control, not BFT. Web clients: browsers can be malicious, but the server is the authority. Validate, sanitize and escape input (against SQL injection, XSS). Bugs: the same software on every node has the same bug. BFT needs a two-thirds supermajority of correct nodes, so you’d need four independent implementations and hope the bug is in only one. Attackers: if one node can be compromised, all probably can. Authentication, access control, encryption and firewalls remain the defence. |
BFT protocols are expensive and fault-tolerant embedded systems rely on hardware support, so in most server-side systems they aren’t practical.
Short of full BFT, it’s still worth guarding against “lying” caused by hardware issues, bugs and misconfiguration:
Algorithms for problems like consensus must not depend on the details of particular hardware. So we write down the faults we expect as a system model: an abstraction of the algorithm’s assumptions.
Correctness is defined by properties. A sorting algorithm’s output has the property that every element is smaller than the ones to its right. For a fencing-token generator we might require:
No two requests get the same token.
If request x completed before y began, then tx < ty.
A node that requests a token and doesn’t crash eventually gets a response.
An algorithm is correct in a system model if it always satisfies its properties in every situation the model allows. But if all nodes crash or all network delays become infinite, no algorithm can do anything. How can we still promise something useful?
If violated, you can point to the exact moment it broke (the operation that returned a duplicate token), and the damage can’t be undone.
May not hold at a given moment (a request has no response yet), but there’s always hope it will. The word “eventually” is the giveaway, and yes, eventual consistency is a liveness property.
(The informal slogans are imprecise, since “good” and “bad” are value judgments; the definitions above are the precise ones.) The distinction makes hard models tractable: distributed algorithms usually require safety always, in every situation the model allows, even if every node crashes or the whole network fails. Liveness can have caveats: “a response arrives if a majority of nodes are up and the network eventually recovers”. The partially synchronous model in fact assumes every outage is finite.
Models are simplifications, and reality bites back. Crash-recovery assumes disk survives crashes, but disks get corrupted or wiped by hardware errors and misconfiguration, and firmware bugs have made servers fail to recognize correctly attached drives on reboot. Quorum algorithms rely on nodes remembering what they claimed to store; a node with amnesia breaks the quorum and the algorithm. A model with “storage mostly survives” is harder to reason about.
A theoretical description can declare some things impossible; a real implementation may still need code for them, even if that code is just printf("Sucks to be you") and exit(666), leaving a human to clean up. That’s one difference between computer science and software engineering. Abstract models are still incredibly valuable: they reduce real complexity to a manageable set of faults you can reason about systematically.
Concurrency, partial failure and network delay create a huge number of possible states, and the properties must hold in all of them. Formal verification describes the algorithm mathematically and proves it correct for everything the model allows. A proven algorithm doesn’t guarantee a correct implementation, but analysis finds problems that might otherwise hide until unusual timing exposes them. Combine it with empirical, randomized testing: property-based testing, fuzzing, and deterministic simulation testing. AWS, FoundationDB and TigerBeetle use combinations of these successfully.
You write a specification in a purpose-built language such as TLA+, Gallina or FizzBee, describing the algorithm’s behavior without implementation details. A model checker then systematically tries everything that could happen and checks the invariants in every state it reaches.
Many bugs only show up when machines or networks fail. Fault injection deliberately causes those failures in a running system: network failures, crashes, disk corruption, paused processes. It runs in a production-like environment, or in production itself (Netflix’s Chaos Monkey; chaos engineering). A coordinator decides which faults to inject and when; scripts on the nodes inject them with tools like kill, umount and firewall rules; you watch the system during and after. Because so many tools are involved, frameworks such as Jepsen package them up, and Jepsen has found critical bugs in many widely used systems.
DST explores states like a model checker but runs your actual code. Network, I/O and clocks are replaced by mocks, so a simulator decides the exact order of everything, including timings and failures. It runs huge numbers of randomized executions, and any failing one can be replayed exactly from its seed, which fault injection on real machines can’t do.
To work, the simulator must control every source of nondeterminism. Three strategies:
| Level | How | Examples |
|---|---|---|
| Application | Built from day one for determinism: async I/O through a library you can swap for a simulated network, or all state changes in one event loop with mocked clocks. | FoundationDB (Flow), TigerBeetle |
| Runtime | Force the async runtime to run everything sequentially, and swap in deterministic library versions without changing app code. | FrostDB (patched Go runtime), Rust MadSim (Tokio, S3, Kafka libraries) |
| Machine | A custom hypervisor answers every normally nondeterministic call (clocks, network, storage) deterministically; run the whole system in containers inside it. | Antithesis |
Extra benefits: Antithesis branches executions to explore rare paths more often, and because clocks are mocked, simulations can run faster than real time; TigerBeetle simulates timeouts without waiting for them. Try both approaches on the lease bug:
Nondeterminism (concurrency, network delay, pauses, clock jumps, crashes) is behind every problem in this chapter. Making a system deterministic simplifies things a great deal, and the idea keeps coming back:
Full determinism takes care. Even after removing concurrency and mocking I/O, network, clocks and random numbers, some nondeterminism can remain: hash-table iteration order in some languages, or hitting resource limits such as allocation failures and stack overflows.
The defining characteristic of distributed systems is partial failure: any interaction with another node may fail, go slow, or get no answer at all. We build tolerance into software so the whole keeps working while parts are broken.
Detecting faults is hard: timeouts can’t distinguish node failure from network failure, variable delay causes false suspicion, and limping nodes are worse still. Tolerating them is hard too: there is no shared memory, no global variable, no common knowledge; nodes can’t even agree on the time. Major decisions can’t be made by one node, so we use protocols that get a quorum to agree.
If you can solve a problem on a single machine (with an embedded storage engine, say), it’s generally worth it, and a single machine can do a lot today. But scalability isn’t the only reason to distribute: fault tolerance and low latency (data near users) need multiple nodes. In principle a distributed system can run forever, because every fault and maintenance task happens at node level (though a bad config change pushed to every node will still bring it down). Bounded delays and real-time guarantees are possible but expensive, so most systems choose cheap and unreliable over expensive and reliable. Chapter 10 moves from problems to solutions.
| Assumption | Can you rely on it? | What to do instead |
|---|---|---|
| A sent message arrives | No | Timeouts, retries, and idempotent requests |
| No reply means it wasn’t processed | No | Application-level responses; deduplication on retry |
| Delay has an upper bound | No (asynchronous network) | Measure; adaptive timeouts (φ accrual) |
| TCP ack means processed | No | Wait for the application’s answer |
| Time-of-day clocks agree and only go forward | Roughly, not exactly | Monotonic clock for durations; monitor offsets; confidence intervals |
| Timestamps order events across nodes | No | Logical clocks, version vectors, or commit wait on bounded intervals |
| My thread runs continuously | No | Assume a pause between any two lines |
| My node knows whether it is alive and leader | No | Majority quorums decide |
| A lease means I’m the only holder | No | Fencing tokens checked by the resource |
| Nodes don’t lie | Usually, inside your datacenter | BFT only for untrusted parties; checksums, validation, NTP majority for weak lying |
| Disk survives a crash | Mostly | Crash-recovery model, plus code for the “impossible” |