Why divide a runbook into two layers?
An incident runbook does not contain the correct answer to every outage. Unexpected dependencies, deployment changes, data states, and traffic patterns combine, and identical symptoms can have different causes. A good runbook fixes the response method rather than the cause. Agree who declares incidents, how severity is judged, who commands, how user harm is reduced, when recovery is declared, what is recorded and communicated, and what state transfers at handover.
NIST’s latest guidance also considers detailed advance procedures for every possible situation unrealistic.[6] Instead, define organizational roles and authority and prepare actionable procedures and playbooks for recurring, important, and time-sensitive activities. A runbook makes teams use shared operating rules during incidents rather than predicting incidents.
Putting every command and failure type into one runbook makes it stale quickly. Two layers are easier to maintain. The Incident Coordination Runbook contains common detection, triage, severity, command, communication, handoff, closure, and review rules. Linked technical runbooks or recovery procedures handle system-specific work such as database failover, queue backlogs, external-API workarounds, cache disabling, service restarts, and deployment rollback.
An incident may use several technical runbooks, but one coordination runbook controls who executes which procedure in which state and how outcomes are checked. This reduces the need to redesign incident management whenever the technical stack changes.
Agree on response states first
Responders can disagree even about current incident state. One person may think recovery is complete while another is still investigating causes. A shared model provides common language.
Detected → Triaged → Declared → Mitigating → Recovering → Monitoring → Resolved → Reviewed
This is an example, not the only standard-prescribed model. Each state’s objective matters more than its name. Mitigating stops harm spreading rather than achieving a perfect fix. Recovering returns to intended operation without relying on temporary measures. Monitoring verifies user outcomes and data recovery. Deterioration must allow a return to an earlier state.
Operations connected from observation to improvement — Respond using service indicators and alerts, then connect change records and incident reviews to improvements.
Detection: record what was found
Incidents do not always start with monitoring alerts. Users may report errors, support may notice repeated inquiries, providers may report outages, or deployment owners may first notice unexpected behavior.
Record initial facts rather than guessed causes: detection time and source, failing user actions, apparently affected users, regions, and functions, first symptoms, and supporting dashboards, logs, traces, and request IDs.
Receiving an alert and declaring an incident are distinct. Choosing Page, Ticket, or Dashboard belongs to alert routing. This article addresses what happens after a detected phenomenon enters incident response.
Triage: establish impact before cause
Premature certainty is dangerous: a recent deployment or rising database CPU may suggest hypotheses, but hypotheses must not become facts. Initial triage aims to build a shared understanding of operations, not complete root-cause analysis.
Ask whether users are actually affected, which critical journeys fail, whether scope is growing or limited to regions, tenants, functions, or versions, what recently changed, how dependencies behave, whether writes or consistency are at risk, whether security or privacy incidents are possible, and whether workarounds exist.
Infrastructure metrics are important causal signals but cannot alone determine user impact. Recovery must likewise connect to critical user-journey outcomes.
Severity: record reasoning, not only numbers
Severity determines staffing, decision participation, communication channels, and which other work to interrupt. It is more than a label. A universal table such as ten minutes means SEV-1 and thirty minutes means SEV-2 is not viable. NIST prioritizes incidents through scope, impact, time-critical nature, recoverability, and other risk factors.[6] Review at least the dimensions in the table.
Map these dimensions to organizational severity labels. Initial severity must not remain fixed: expanding impact or newly discovered data risk may raise it, while confirmed limited scope may justify adjustment. Record what changed and why in the timeline.
Dimension | Question |
|---|---|
| User impact | Are critical journeys interrupted? One function or the whole service? |
| Scope | Some users, regions, or tenants, or rapidly expanding impact? |
| Data | Could data be lost, duplicated, corrupted, or inconsistent? |
| Security / Safety | Could there be a security incident or personal-data exposure? |
| Time-criticality | Does delay accumulate damage or make recovery harder? |
| Workaround | Is a safe user workaround available? |
| Recoverability | Are mitigation and recovery paths known, or outcomes uncertain? |
| Coordination complexity | Can one team handle it, or are multiple teams and providers needed? |
| Uncertainty | Is impact scope itself unknown, as well as cause? |
Why does the Incident Commander not fix everything personally?
As incidents grow, coordination problems can exceed technical ones. More people simultaneously reading logs, editing the same configuration, and discussing separate hypotheses in different Slack threads can make understanding harder.
Google SRE separates Incident Commander, Communications Lead, and Operations Lead roles.[1] The commander manages overall state and priorities, operations focuses on mitigation and recovery, and communications manages information flow. Small teams may combine roles rather than assign three people. Two points must still be explicit: who has final coordination authority, and who may change production.
The commander need not be the best debugger. Executing every command can obscure overall state, priorities, and communication. The role needs authority and responsibility to maintain what is known, what is unknown, and which decision comes next, rather than every correct answer.
Mitigation: reduce harm before fixing the cause
Treating mitigation and recovery as synonyms destabilizes response. Mitigation reduces current user impact or limits further harm, even through temporary states. Disabling a faulty feature, rerouting traffic, restricting writes, or reducing overload-generating requests may need to precede root-cause investigation.
Rollback is an option when deployment is a suspected cause. Its feasibility depends on compatibility of configuration, database schemas, data, feature exposure, and traffic as well as code. Those mechanics belong to separate rollback design.
Rather than prescribing one recovery method, require the table’s records for each action. In particular, distinguish an action having executed from having worked.
Item | Record |
|---|---|
| Action | What will change |
| Owner | Who executes it |
| Objective | Which user impact it should reduce |
| Expected Signal | Which signal should change if it works |
| Risk | Additional risk the change could create |
| Safety Control | How to stop or recover if it fails |
| Result | Actual outcome |
| Evidence | Basis for assessing the result |
Evidence: what should mitigation preserve?
Urgency makes records easy to defer, but reconstruction needs pre-change state and response history. Where possible, preserve running application and deployment versions, configuration and feature flags, reference times for dashboards and queries, and relevant logs, traces, and request IDs before destructive changes. Add database and queue state, dependency status, who changed what and why, and resulting differences.
NIST recommends preserving integrity and provenance of incident data and metadata while distinguishing that not every incident requires legal-grade chain of custody.[6] For ordinary service incidents, the baseline goal is reconstructable records.
Suspected compromise or personal-data exposure requires security and legal procedures and evidence-handling standards. Evidence preservation must not leave ongoing harm unattended. If user impact is growing, the commander should explicitly prioritize containment and preservation.
Recovery: from temporary measures to normal operation
Reduced error rates after mitigation do not end an incident. A service made usable by disabling an essential function may be mitigated without being fully recovered.
Recovery returns to the agreed target state: removing temporary changes, restoring normal traffic, repairing data consistency, clearing backlogs, and normalizing dependencies. It may require rollback, roll-forward, failover, or restoration procedures. Distinguish completing recovery actions from confirming service recovery.
Recovery verification: is a green server enough?
Recovery declarations are easy to make prematurely. Lower CPU, running pods, and fewer error logs are useful signals, but do not establish that users can use the service normally again.
NIST Rev. 3 calls for confirming actual restoration of essential services, verifying integrity and normal behavior, monitoring afterward, and ending recovery against closure criteria.[6] Verification must return to the user outcomes originally affected.
Do failed critical journeys succeed again? Is latency still abnormal? Do missing, duplicate, or inconsistent data remain? Are queues still accumulating? Are some regions, tenants, or user segments still failing? Have dependencies returned to normal paths? Do monitoring and alerts work? Do temporary workarounds create new risks?
Observation duration depends on traffic, failure cycles, and data processing. Define service-specific recovery acceptance criteria and observation conditions instead of a universal rule such as fifteen healthy minutes means recovered.
Communication is a separate workstream
Communication is not an announcement postponed until later. Internal coordination among responders, stakeholder communication to leadership, support, and sales, and external communication to users or organizations may run simultaneously.
NIST treats coordination, notification, public communication, and information sharing as separate response activities.[6] Google SRE likewise emphasizes clear descriptions of current impact and response state alongside technical mitigation.[1]
Updates need at least incident identity, current state and severity, user impact, affected scope, and changes since the previous update. Add mitigation or recovery in progress, known workarounds and unknowns, the next decision point, and the next update.
Next update is not an unsupported recovery ETA. When recovery time is unknown, use an update after the next procedure or an agreed communication cadence, rather than guessing normal service within thirty minutes.
Handoff requires more than leaving a message
Long incidents make one person responding throughout a risk. The question is whether incident state transfers without loss, not whether handover is needed. Google SRE describes briefing the incoming commander and obtaining explicit acceptance of command.[1]
A handoff includes current user impact, severity and reasoning, commander and key roles, confirmed facts, active hypotheses, completed mitigation and results, ongoing work, failed or unsafe actions, and the next decision point. Include communication state, preserved-evidence locations, and remaining risks.
Leaving a document does not complete handover. Record when the incoming owner understands the state and explicitly accepts responsibility.
When should an incident close?
Incidents need not remain open until root cause is fully known, nor close as soon as error rates briefly fall. Agree beforehand on conditions for entering Resolved.
User harm has stopped; affected journeys meet recovery criteria; residual data, queue, and asynchronous-work issues are acceptable; key dependencies are checked; monitoring and alerts work; temporary measures are recorded and retained ones have owners; residual risks are recorded and acknowledged; required communications are updated; and a post-incident review owner is appointed.
NIST similarly treats ending recovery as verification, closure criteria, and documentation, rather than merely completed technical work.[6]
Review: go beyond one line of root cause
What caused it is necessary but insufficient; alone it may barely prevent the same incident. Improving response requires a wider review.
Detection: did the team notice before users, and were signals missing? Triage: was scope established quickly or time lost to wrong hypotheses? Severity: did the right people join early enough? Coordination: were command and production-change authority clear? Mitigation: was harm reduction prioritized appropriately? Recovery: were temporary mitigation and normal recovery distinguished? Verification: were user outcomes and data checked? Communication: did the right people share current state on time? Handoff: was context lost?
Google SRE postmortems aim to improve detection, mitigation, coordination, and communication as well as analyze causes.[5] Actions must enter the actual backlog. Blameless does not mean ownerless: examine why systems allowed or required behavior rather than blaming individuals, while assigning clear owners to fixes.
Follow-up: has writing the review changed anything?
A postmortem that ends in a document store will not change behavior in the next incident. Every follow-up needs at least the table’s fields.
Record emergency feature flags, firewall rules, traffic workarounds, and manual cron jobs as separate cleanup items. Response ends not with a postmortem meeting, but when discovered operating gaps acquire backlog ownership.
Field | Content |
|---|---|
| Gap | The failure possibility discovered |
| Action | The specific change to make |
| Owner | Person or team accountable through completion |
| Priority | Priority relative to other operating work |
| Target | Agreed completion target |
| Acceptance Evidence | What establishes completion |
| Tracking ID | Issue or ticket location |
| Status | Current state |
What should the Runbook Template contain?
This is not a technical runbook for one cause. Link service-specific commands and recovery procedures separately; this document manages common coordination states and responsibilities. Seventeen blocks run from metadata to runbook improvement, tracing detection, triage, severity, mitigation, evidence, recovery, communication, handoff, closure, and review.
Roles & Authority assigns owners and responsibilities to six roles. Small teams may combine roles, but command and production-change authority must remain explicit. When severity changes, append time, previous and new levels, reasoning, and decision-maker rather than overwriting history.
The status-update block puts communication fields into a reusable form. Handoff checks six items: outgoing explanation, incoming questions, ongoing production changes, agreement on the next decision point, explicit incoming acceptance, and notification of command transfer to other responders. Detailed deployment rollback stays in a separate rollback runbook.
- Incident Commander
- Manages overall state, priorities, and decisions; need not be the best debugger.
- Operations / Technical Lead
- Executes mitigation and recovery work.
- Communications Owner
- Communicates current state internally and externally.
- Scribe / Timeline Owner
- Records timelines, decisions, and evidence.
- SME
- Provides expertise in a particular system.
- Production Change Authority
- Has authority to approve and execute production changes.
Block | Record fields | Checks |
|---|---|---|
| Runbook Metadata | runbook_name · service · runbook_owner · last_reviewed_at · review_due_at · critical_user_journeys · related_alerts · technical_runbooks · recovery_procedures · rollback_guide · escalation_path · communication_channels | Check linked resources regularly. |
| Detection | incident_id · detected_at · detected_by · detection_source(alert, user_report, support, operator, dependency, other) · observed_symptom · suspected_user_impact · affected_journey · affected_scope · initial_confidence · initial_evidence(dashboards, logs, traces, request_ids, external_status) | Record initial facts before hypotheses. |
| Triage & Incident Declaration | declared(true, false) · declared_at · declaration_owner · current_state · initial_severity · severity_rationale · known_facts · hypotheses · unknowns | Actual user impact; affected critical journeys; scope and spread; recent deployments, configuration, and features; dependencies; data risk; security/privacy possibilities; workarounds; incident declaration. |
| Roles & Authority | Incident Commander · Operations / Technical Lead · Communications Owner · Scribe / Timeline Owner · SME · Production Change Authority: named owner and responsibility for each role | Small teams may combine roles; explicitly identify current command and production-change authority. |
| Severity Assessment | current_severity · assessed_at · assessed_by · user_impact · scope · data_risk · security_or_safety_risk · time_criticality · workaround · recoverability · coordination_complexity · uncertainty · rationale | Append time, previous level, new level, reasoning, and decision-maker when severity changes; do not overwrite. |
| Current Incident State | state(DETECTED, TRIAGED, DECLARED, MITIGATING, RECOVERING, MONITORING, RESOLVED) · current_objective · current_user_impact · current_scope · latest_decision · next_decision_point | One line stating the current objective for each state. |
| Mitigation Log | Action · Owner · Objective · Expected Signal · Risk / Safety Control · Status · Result · Evidence | One row per action; distinguish execution from effectiveness. |
| Evidence Preservation | evidence_location · retention_requirement · sensitive_data_present · security_or_legal_escalation | Deployment and artifact versions; configuration and flags; dashboard/query reference time; logs, traces, request IDs; stateful-system and dependency state; production actors, reasons, and outcomes; separate evidence procedures for security/privacy. |
| Recovery Plan | target_state · recovery_owner · prerequisites · technical_runbooks · recovery_actions · temporary_controls_to_remove · known_residual_risk | Use a separate rollback runbook for deployment mechanics. |
| Recovery Verification | verification_started_at · verification_owner · journey_evidence · data_evidence · dependency_evidence · monitoring_evidence · observation_condition · verification_result | Critical journeys succeed; acceptable user latency; data loss, duplication, and consistency checked; queues and asynchronous backlogs; regions, tenants, and segments; normal dependency paths; monitoring coverage and alerts; residual workaround risk; service-specific observation conditions. |
| Communication | Audience(Incident Responders · Internal Stakeholders · Users / Customers · External Provider · Legal / Security / Regulatory) · Owner · Content · Last Update · Next Update Trigger | One row per audience. |
| Status Update Template | Incident · Current State · Severity · User Impact · Affected Scope · Changes Since Previous Update · Mitigation / Recovery in Progress · Available Workaround · Known Unknowns · Next Decision Point · Next Update | Next update is an update point, not an unsupported recovery ETA. |
| Handoff | handoff_at · outgoing_owner · incoming_owner · current_user_impact · current_severity · severity_rationale · current_state · confirmed_facts · active_hypotheses · unknowns · completed_actions · actions_in_progress · failed_or_unsafe_actions · pending_actions · next_decision_point · communication_status · evidence_location · guardrails · incoming_owner_accepted_at | Outgoing explanation; time for incoming questions; ongoing production changes checked; next decision point agreed; explicit incoming acceptance; command change communicated to responders. |
| Closure | resolved_at · resolved_by · closure_rationale · remaining_risks · temporary_controls · post_incident_owner | User harm stopped; recovery checks pass; data, queues, dependencies, monitoring, and alerts checked; all temporary changes recorded; retained controls have owners; residual risks recorded; communications complete; review owner assigned. |
| Post-incident Review | review_owner · review_date · incident_summary · timeline · user_impact · technical_factors · contributing_factors · detection_review · triage_review · severity_review · coordination_review · mitigation_review · recovery_review · verification_review · communication_review · handoff_review · what_worked · what_did_not_work · remaining_unknowns | Do not stop at a one-line root cause. |
| Follow-up Ownership | ID · Gap · Action · Owner · Priority · Target · Acceptance Evidence · Tracking ID · Status | One row per item; emergency temporary measures become separate cleanup items. |
| Runbook Improvement | runbook_gap_found · runbook_change_required · change_owner · technical_runbook_change_required · alerting_change_required · observability_change_required · deployment_change_required · next_review_or_drill | Update the runbook itself after real incidents. |
When does runbook quality become visible?
A runbook should not be read for the first time during an incident. Regularly check whether service and on-call owners know severity criteria and commander authority, whether channels work, technical links remain live, and handoffs are possible.
Update the runbook after real incidents. NIST Rev. 3 treats lessons learned as improvement throughout response, not only one step after all recovery ends.[6]
A good runbook does not eliminate incidents. It reduces secondary incidents caused by people responding inconsistently. Not knowing the cause initially is acceptable. Knowing who decides, what to protect now, which changes were made, what evidence establishes recovery, and what to hand over turns response into repeatable operations.
