The outage lasted ninety minutes. Roughly fifteen of those were spent fixing it. The rest went to five engineers independently investigating the same theory, nobody knowing who was deciding anything, and a customer-facing team learning about the problem from a customer.
Incident response is mostly coordination. The engineering is usually within the capability of whoever is on call; what fails is that nobody owns the coordination, so it does not happen.
Separate command from investigation
The most valuable structure is also the simplest: whoever is coordinating is not debugging. An incident commander tracks what has been tried, decides what happens next, keeps a timeline, and handles communication. Engineers investigate. The moment the commander opens a terminal and starts looking at logs, coordination stops — and in a long incident, that is when it stretches.
This feels like overhead at small scale, and it is the thing that most reliably shortens incidents at any scale above two people. It also gives you a timeline written during the event rather than reconstructed afterwards from memory and chat scrollback.
- Declare early and downgrade freely. The cost of an unnecessary declaration is a few minutes; the cost of a late one is measured in customer impact.
- One channel per incident, with the current status pinned and updated. People joining should not have to read four hundred messages.
- Mitigate before diagnosing. Roll back, fail over, disable the feature — understanding can wait until customers are served.
- Communicate on a fixed cadence even with nothing new to report. Silence is read as chaos.
- Record timestamps as you go. Nobody remembers accurately afterwards, and the timeline is where the real lessons are.
## Incident 2026-02-19 — checkout failures (SEV2, 47 min)
**Impact:** 12% of checkout attempts failed, 14:02-14:49 UTC. 480 orders affected,
all recoverable. No data loss.
**Timeline (UTC)**
- 14:02 Error budget burn alert fires (fast-burn, checkout SLO)
- 14:06 Incident declared, IC assigned, #inc-0342 opened
- 14:11 Suspected payment provider; provider status page green
- 14:23 Connection pool exhaustion identified on billing service
- 14:31 Mitigation: pool size raised, service rolled
- 14:49 Error rate normal, incident closed
**What happened:** a deploy at 13:55 added a synchronous provider call inside a
request already holding a database connection. Under normal load the pool
absorbed it; at peak it did not.
**Why it took 21 minutes to find:** pool saturation was not on the service
dashboard. We inferred it from thread dumps.
**Actions**
- [ ] Pool utilisation on the service dashboard + alert at 80% — @owner, Mar 6
- [ ] Lint rule: no outbound HTTP while holding a DB connection — @owner, Mar 13
- [ ] Load test in staging at 2x peak before billing deploys — @owner, Mar 20The review is about the system, not the person
A postmortem that concludes someone should have been more careful has found nothing. The engineer who wrote that call was behaving reasonably given what was visible to them, and the interesting question is why the system made it easy to do and hard to detect. That is not a nicety — it is the difference between an action item that prevents recurrence and one that adds a line to a checklist nobody reads.
Two questions produce more value than the rest of the template combined: what made this hard to detect, and what made it hard to diagnose. Most organisations fix the cause and leave the detection gap in place, so the next distinct cause with the same symptom takes just as long to find.
Every action item without a name and a date is a sentence in a document, not a change to the system.
Track completion of those items as a metric and review it monthly. The most common failure of incident review is not the quality of the analysis — it is that a thoughtful document was written, everyone agreed, and none of the three actions were done by the time the same class of failure recurred four months later.