The on-call rotation received roughly sixty pages a month. Perhaps four corresponded to something a user noticed. The rest were CPU above eighty percent, a pod restarting, a queue briefly deep, a replica lagging — all technically true, all conditions the system routinely recovered from on its own while a human was being woken to watch it happen.
That is alerting on causes. Service level objectives invert it: define what users need, measure whether they are getting it, and page only when they are not. The change is less about tooling than about deciding, explicitly, what 'working' means.
Pick indicators users would recognise
A good service level indicator is a ratio of good events to valid events, measured as close to the user as you can get. Availability as successful requests over total requests. Latency as the proportion of requests served under a threshold. For an asynchronous pipeline, freshness as the proportion of records processed within an agreed window.
The threshold matters more than the number of nines. Ninety-nine percent of requests under 300ms is a claim someone can verify and argue with. An average response time of 120ms hides every slow request behind a mean, which is precisely where user pain lives.
# Multi-window burn rate: page for fast burns, ticket for slow ones.
# 99.9% over 30 days = 43 minutes of budget. A 14.4x burn exhausts it in ~2 days.
groups:
- name: checkout-slo
rules:
- alert: CheckoutBudgetBurningFast
# Both windows must agree: the long one confirms it is real,
# the short one makes recovery reset the alert quickly.
expr: |
(checkout:error_ratio:rate1h > 14.4 * 0.001)
and
(checkout:error_ratio:rate5m > 14.4 * 0.001)
for: 2m
labels: { severity: page }
- alert: CheckoutBudgetBurningSlow
expr: |
(checkout:error_ratio:rate6h > 6 * 0.001)
and
(checkout:error_ratio:rate30m > 6 * 0.001)
for: 15m
labels: { severity: ticket } # a ticket, not a 3 a.m. phone callThe two-window structure is what makes burn-rate alerting usable. A single short window is noisy and a single long window is slow to notice a severe outage. Requiring both to agree gives you fast detection of genuine problems and rapid reset once they stop.
The budget is a decision tool
An objective of 99.9% is permission to be unavailable for about 43 minutes a month. That remainder is the error budget, and its real value is settling arguments that are otherwise decided by whoever is most senior in the room. Budget healthy: ship the risky migration, run the experiment. Budget exhausted: reliability work takes priority until it recovers.
- Set objectives from what users need, not from what you currently achieve. Then measure the gap honestly.
- Do not target more nines than your dependencies can support — you cannot be more available than what you call.
- Page only on user-visible symptoms. Everything else is a dashboard or a ticket.
- Keep cause-based alerts as context on the incident, not as the trigger for it.
- Review objectives quarterly with the product owner. An objective nobody outside engineering has agreed to is an engineering opinion.
If an alert fires and no user could tell, you have not detected a problem. You have detected a computer doing its job.
The rotation above went from roughly sixty pages a month to under ten, and the ones that remained were real. The reliability of the system did not change that week; what changed was that we stopped asking humans to respond to conditions the system already handled, and started measuring the thing customers were actually experiencing.