How to Build an Incident Response Runbook Your Team Actually Uses

The runbook you wrote last quarter is probably wrong. Not because the process changed, but because the last three incidents did not match it, and nobody edited it after. Now it sits in a wiki page that opens the second-most in the year, right after your holiday policy.

A working incident response runbook is not a document. It is a habit encoded in a document. Here is how to write one that survives contact with a real 3am page.

What should an incident response runbook actually contain?

Strip it to the minimum useful set. If a section does not change what a responder does in the first 30 minutes, cut it.

• Severity definitions. Four tiers, defined by customer impact, not internal drama.
• Roles. Who is running the incident, who is talking to customers, who is writing things down.
• Declaration protocol. How to open an incident, what channel it lives in, who gets paged.
• Communication cadence. How often status updates go out and to whom.
• Escalation ladder. When and how to wake a second engineer, a VP, or the CEO.
• Closure criteria. What has to be true before you say the incident is over.
• Postmortem trigger. Which severities require a written postmortem and by when.

That is seven sections. Everything else belongs in a linked service runbook or a training doc.

How should you define severity levels?

The worst runbook you can write says "SEV1 is a bad problem." Every severity needs a concrete customer-impact test that a tired on-call engineer can apply in 30 seconds.

Notice what is missing: engineering effort estimates, "high priority" labels, and any language like "important." Severity is a public commitment about response, not a signal of internal urgency.

Who are the three incident roles you actually need?

Three roles cover 90% of incidents. Add more and you dilute accountability.

• Incident commander. Runs the response. Assigns work, calls escalations, decides when severity changes. Not the person fixing the thing. If your best debugger is commanding, you have burned your best debugger.
• Communications lead. Owns customer-facing status page updates, internal Slack broadcasts, and executive briefings. Speaks in complete sentences a customer would understand.
• Scribe. Captures every decision, hypothesis, and action to the timeline in real time. Often the same person as the comms lead in small teams; separate above SEV2.

Write down one sentence per role about what that person owes the channel every five minutes. The commander owes a "here is what we are doing next." The comms lead owes a customer-facing status. The scribe owes an entry on the timeline.

What is the right channel protocol for an incident?

One channel per incident. Not a thread. Not the general engineering channel. A dedicated channel with a naming convention like #inc-241-checkout that anyone in the org can find by searching inc-.

• Everything happens in the channel. No DMs, no side threads. If a fix happens in a DM, it did not happen for the timeline.
• Status is pinned. The current severity, commander, and one-line summary sit in the channel topic, updated on every state change.
• Silence the rest. Snooze notifications on unrelated channels for the responders. The last thing a commander needs is a marketing message during a database failover.

If your team runs incidents in threads, your timeline is already broken. Threads collapse. They do not export cleanly. Nobody scrolls back through 400 replies at hour six.

How do you write the escalation ladder?

The ladder answers one question: at what point does someone with more context or more authority get pulled in? Write it down before the incident, because in the incident you will hesitate.

• 5 minutes without ack. Secondary on-call is paged automatically.
• 15 minutes of active SEV2. Engineering manager is notified.
• 30 minutes of active SEV1, or 60 minutes of active SEV2. VP of Engineering joins the channel.
• Customer impact greater than 1 hour on SEV1. CEO and Head of Customer Success are pulled in for external communications decisions.
• Data loss or security exposure suspected. Legal and the DPO are pulled in immediately, regardless of severity.

The point of the ladder is to make escalation a rule, not a judgment call under stress. An on-call engineer should not be deciding whether to wake the VP at 3am. The runbook already decided.

When do you require a postmortem?

Every SEV1 and SEV2 gets a written postmortem, due within 5 business days of resolution. Every SEV3 gets a lightweight incident note in the same shared doc, due within the week. SEV4 gets nothing except a timeline entry.

The runbook must name the template, the reviewer, and the follow-up meeting. If it does not, postmortems become optional, and then they stop happening. A postmortem you skip is an incident you will run again.

How do you keep the runbook from decaying?

Two mechanisms. First, every postmortem produces at least one edit to the runbook, even if it is one sentence. That is the reviewer's job to enforce. Second, run a quarterly incident drill: a staged SEV2 with a real page, real channel, and a real 30-minute response, then a 10-minute debrief against the runbook.

Teams that skip the drill discover their runbook is wrong the first time a customer is watching. Teams that drill quarterly find the gaps on their own schedule, which is much cheaper.

The mistake to avoid

The temptation is to write the runbook as a compliance artifact: comprehensive, defensive, hedged. That runbook is unusable at 3am. Write it for the tired engineer with one hand on the keyboard and one on a coffee mug, then trust them to escalate the edge cases. A one-page runbook that gets used beats a 40-page runbook that gets referenced once a year during audit prep.