Consensus is the mechanism by which a group of machines agrees on a sequence of decisions despite some of them failing. Nearly every stateful system you rely on has one at its core — the metadata layer of your database, your service registry, your scheduler, your configuration store. Almost nobody should be implementing one, and yet a surprising amount of production code contains a hand-rolled leader election that is subtly wrong.
The wrongness is rarely in the happy path. It appears during a network partition or a long garbage collection pause, at which point two nodes each believe they are the leader, and both of them act on it.
A lock is not a leader election
The common pattern is: acquire a key in Redis with a TTL, do work while holding it, renew periodically. It looks like mutual exclusion. It is not, because the holder can be paused — a stop-the-world pause, a suspended container, a node that lost its network — past the expiry, wake up believing it still holds the lock, and write. Meanwhile another node legitimately acquired it. Two writers, no error, corrupted state.
The fix is fencing. Every lock acquisition returns a monotonically increasing token, the token accompanies every write, and the resource rejects any write carrying a token lower than the highest it has seen. A delayed leader's writes are rejected by the storage layer rather than by its own belief about time.
# Fencing: the resource enforces exclusion, not the client's sense of time.
lease = coordinator.acquire('indexer', ttl=15) # lease.fence increases every time
while lease.valid():
batch = next_batch()
# The token travels with the write. A paused leader that wakes up late
# carries a stale token and is rejected at the storage layer.
ok = store.write(batch, fence=lease.fence)
if not ok:
log.warning('fenced out, standing down', fence=lease.fence)
break
lease.renew()
# Without the fence parameter, the line above is a race with a comfortable
# name. Every 'distributed lock' that lacks one is trusting a clock.What consensus costs
A quorum system needs a majority to make progress, which is the source of both its guarantee and its constraints. Three nodes tolerate one failure; five tolerate two. An even number buys nothing — four nodes still need three for a majority, so you have added a machine that can fail without adding fault tolerance.
Every committed write costs a round trip to a majority, so placement matters enormously. Spread a quorum across regions and every write pays inter-continental latency; put all members in one availability zone and a single zone failure takes the whole system down. Three zones in one region is the configuration that survives the failure people actually have.
- Use an existing implementation — etcd, Consul, a Raft library, or your database's built-in coordination. This is not the place for original work.
- Odd member counts only. Four nodes is three nodes with a higher bill.
- Keep consensus off the hot path. It coordinates metadata and leadership; bulk data should not need a quorum per write.
- Monitor leader elections per hour. Frequent elections mean flapping, and flapping means an unstable system that has not failed loudly yet.
- Rehearse quorum loss. Recovering a cluster that has lost its majority is a procedure, and reading it for the first time during an outage is not the plan.
Two nodes that each believe they are the leader will not report an error. They will both work, confidently, on the same data.
The practical takeaway is narrow: you need consensus far less often than it feels, and when you do need it, you need someone else's implementation plus fencing tokens at the point of use. The genuinely interesting engineering is in arranging your system so that the number of things requiring global agreement stays as small as possible — every one of them is a latency floor and an availability ceiling you have chosen to accept.