Incidents

During an incident the goal is to stop the bleeding. Understanding comes afterwards, and confusing the two extends outages.

Mitigate before diagnosing. Roll back, disable the flag, shift traffic. Engineers frequently spend twenty minutes finding root cause when a thirty-second rollback would have ended user impact — and the evidence is still there afterwards.

One incident commander. Their job is coordination and decisions, not debugging. Without a clear owner, three people investigate the same thing while nobody communicates and nobody decides.

Communicate on a schedule. Regular updates, even "still investigating, next update in 15 minutes", stop the stream of interruptions asking for status. Silence generates more load than any update.

Write down a timeline as you go. Memory reconstructs incidents wrongly, confidently. Timestamps of what you observed and what you changed are what make the review accurate.

Blameless postmortems, genuinely. If someone is punished for an outage, the next one gets hidden, and hidden incidents are how organisations stop learning. The useful question is never "who ran the command" — it is "why was it possible for a single command to do this, and why did nothing catch it?"

Root cause is usually plural. "The deploy broke it" is where analysis starts. Why did review pass? Why did staging not catch it? Why did alerting take eleven minutes? Each is a separate improvement.

Declare early, and downgrade freely. Declaring an incident that turns out to be minor costs a few minutes of several people's attention. Not declaring one costs an hour of a single engineer quietly working a problem nobody else knows about. Agree the severity levels in advance and tie them to user impact rather than to how alarming the graph looks, so the decision during an incident is a lookup rather than a negotiation.

Long incidents need handover. Fatigue degrades exactly the judgement an incident demands, and the errors start well before anyone admits to being tired. A handover needs the timeline, the current hypothesis, what has already been ruled out, and who has been told what — which is one more reason the running timeline is not optional. Customer communication is a separate role for a reason: it moves at a different pace from the investigation, and one person attempting both does neither.

Action items need owners and dates, or the postmortem was theatre. Fewer real items beat a long aspirational list.