A dependency got slow for forty seconds. We were down for twenty minutes. Nothing crashed, no disk filled, no deploy went out — every service in the path was politely retrying, three attempts each, with a timeout longer than the one its caller was using. By the time the dependency recovered, the retries alone were more traffic than the original load, and the system could not climb back out.
That shape has a name: metastable failure. The trigger is transient, the failure is not, because the system's own recovery behaviour became the load. Partial failure is the defining property of a distributed system, and the controls that survive it are unglamorous — a budget that propagates, retries expressed as a ratio, and the willingness to fail fast.
A timeout is not a number, it is a budget
Most services hardcode a per-hop timeout: five seconds to the database, three to the search cluster, ten to the third party. Stack four hops and the deepest call is still working on a request the user abandoned eleven seconds ago. Work that nobody is waiting for is pure fuel for a collapse — it holds connections, occupies workers, and produces nothing.
The fix is to propagate a deadline instead of a duration. The edge sets one, every hop passes the remaining time down, and each call takes the minimum of the remaining budget and its own ceiling. When the budget is gone, you stop — immediately, everywhere, at the same time.
import asyncio, random, time
class RetryBudget:
"""Retries are allowed only while they stay a small fraction of real traffic."""
def __init__(self, ratio=0.1):
self.ratio = ratio
self.calls = 0
self.retries = 0
def allow(self) -> bool:
return self.retries <= self.ratio * self.calls
async def call(fn, deadline, budget, attempts=3, per_try=2.0):
budget.calls += 1
for attempt in range(attempts):
remaining = deadline - time.monotonic()
if remaining <= 0:
raise DeadlineExceeded()
try:
return await fn(timeout=min(remaining, per_try))
except Retryable:
# Last attempt, or the dependency is unhealthy enough that the
# budget has run dry: stop amplifying.
if attempt == attempts - 1 or not budget.allow():
raise
budget.retries += 1
await asyncio.sleep(random.uniform(0, 0.2 * 2 ** attempt))Retry budgets beat retry counts
Three retries sounds modest. Three retries at three layers is twenty-seven requests for one user action, and the amplification arrives precisely when the dependency is least able to absorb it. A per-call count cannot see this, because it has no idea what the rest of the system is doing.
A budget can. Express retries as a share of successful traffic — ten percent is a reasonable starting point — and the behaviour becomes self-correcting: when a dependency is healthy, retries are rare and the budget is never touched; when it is failing, the share is exhausted in seconds and further retries are refused. Errors surface immediately instead of being converted into load.
- Retry only idempotent operations — everything else needs an idempotency key first.
- Retry at one layer, not every layer. Pick the one closest to the dependency and make the others pass failures through.
- Never retry a 4xx. It will be invalid the second time too.
- Always jitter the backoff; synchronized clients are how you turn a recovery into a second outage.
- Scope the budget per dependency, so a failing search cluster cannot consume the allowance for payments.
Shed load before you queue it
An unbounded queue looks like resilience and behaves like a delay line: requests accumulate, latency climbs past every client timeout, and the service works hard on results nobody will read. Bounded concurrency with fast rejection is almost always better. Rejecting ten percent of requests in two milliseconds keeps ninety percent healthy; accepting everything degrades all of it until the whole thing is useless.
Circuit breakers are the same instinct applied per dependency. After a threshold of failures, stop calling, serve a fallback or a clean error, and let a small number of probes test the water before reopening. The value is not the failing call you avoid — it is the connection pool you do not exhaust and the worker you leave free for the requests that can still succeed.
A fast failure is a feature. A slow success nobody is waiting for is a leak.
Hedging buys the tail back
For read paths where the tail matters, hedged requests are the cheapest latency win available: if the first attempt has not answered by roughly the p95, send a second to another replica and take whichever returns first. Capped at a couple of percent of traffic and charged to the same retry budget, it cuts p99 dramatically because slow responses are usually an unlucky replica rather than a slow system. Restrict it to idempotent reads and cancel the loser, or you have just built a retry storm with better marketing.
Test the failure, not the success
None of this is verifiable through normal testing, because normal testing exercises the path where everything works. Inject latency and errors into dependencies in staging, run the load test with a dependency artificially degraded rather than down, and measure time to recovery as a first-class metric alongside availability. The question worth answering before an incident is not whether you survive a dependency failing — it is whether you survive it getting slow, which is strictly harder and vastly more common.