Someone set a one-hour TTL on a popular dashboard query. An hour after deploy, every cached entry expired within the same second, four thousand requests missed simultaneously, and all of them ran the same expensive query against the database at once. The database fell over, the cache stayed empty, and every retry made it worse.
That is a cache stampede, and it is the most common way caching causes an outage rather than preventing one. It is entirely avoidable, and the avoidance is three lines of code that almost nobody writes until after the first incident.
Jitter, locks, and early recomputation
Three defences, in increasing order of effort. Jitter the TTL so expiries spread rather than synchronise — this alone fixes most of it. Take a short lock on miss so only one request recomputes while the others wait briefly or serve stale. And for genuinely hot keys, recompute probabilistically before expiry, so the value is refreshed by one unlucky request while the old one is still being served.
import random, time
def get_cached(key, compute, ttl=3600, beta=1.0):
packed = cache.get(key)
if packed:
value, delta, expiry = packed # delta = how long compute() took
# Probabilistic early expiration: the closer to expiry and the more
# expensive the recompute, the likelier this request refreshes it.
if time.time() - delta * beta * math.log(random.random()) < expiry:
return value
# Only one worker recomputes; the rest serve stale or wait briefly.
if cache.add(f'{key}:lock', 1, ex=30):
try:
start = time.time()
value = compute()
delta = time.time() - start
jittered = ttl * random.uniform(0.85, 1.15) # never a bare TTL
cache.set(key, (value, delta, time.time() + jittered), ex=int(jittered * 2))
return value
finally:
cache.delete(f'{key}:lock')
return packed[0] if packed else compute() # stale beats a stampedePick a strategy per data type, not per system
Cache-aside is the sensible default: read from cache, fall back to the source, populate. It tolerates cache failure gracefully, and its weakness is a window of staleness after a write. Write-through updates both together and keeps the cache consistent at the cost of write latency and a cache that must be available to write at all. Write-behind is fast and will eventually lose data on failure — acceptable for counters and analytics, not for anything a customer can see.
Two smaller patterns prevent a disproportionate share of problems. Cache negative results, or every lookup for a non-existent record hammers the database — with a short TTL, because the thing may come into existence. And version your cache keys by including a schema or deploy identifier, so a change in the cached structure does not require a flush that causes the very stampede you were avoiding.
- Always jitter TTLs. Synchronised expiry is the single most common cache-induced outage.
- Decide explicitly how stale each piece of data may be. 'Fresh' is not a requirement, it is an unpriced assumption.
- Measure hit rate per key pattern. An aggregate hit rate of ninety percent can hide a hot key that misses constantly.
- Keep the system functional with the cache entirely down — degraded is acceptable, unavailable is a single point of failure you built on purpose.
- Invalidate on write rather than relying only on TTL for anything where users notice staleness immediately.
A cache does not make a system faster. It makes it faster when it is warm, and fragile in a new way when it is not.
The framing that helps most is to treat every cached value as a deliberate decision to serve possibly-stale data, with a documented tolerance. Written down, the awkward cases surface immediately — pricing, permissions, availability — and those are precisely the ones where a five-minute TTL quietly chosen by whoever wrote the endpoint becomes a support ticket about a customer seeing the wrong number.