Building an Incident Response Workflow with Runbooks and Native Tools
In Part 1 we broke down why root cause analysis takes hours: scattered context, tribal knowledge, and the "what changed?" question that nobody can answer at 3AM. This article is the DIY fix. You are going to build an incident response runbook system using tools you almost certainly already run — Prometheus Alertmanager, PagerDuty or Grafana OnCall, and your CI pipeline. No new vendors, no procurement cycle.
The goal is narrow and concrete: when an alert fires, the responder should land on a page that tells them what to check, in what order, and when to escalate — instead of starting from a blank Slack channel. If you want the wider context on RCA methods and where runbooks fit, the root cause analysis guide covers the full landscape. Here, we build.
What a runbook has to contain
Most runbooks fail because they are written as documentation, not as procedures. A wiki page titled "API latency" that explains how the service works is not a runbook. A runbook is an if-this-then-that decision procedure a stressed human can execute without understanding the whole system.
Every entry in your incident response process needs seven parts:
- Trigger. The exact alert name and the threshold that fired it. One runbook per alert, not one runbook per service. If two alerts share a runbook, the runbook is too vague.
- Severity guidance. How to tell a page-worthy incident from noise, in terms the responder can evaluate in two minutes. "P1 if error rate is also elevated, P3 if latency is high but errors are flat."
- Impact check. The first command or dashboard that answers "are customers actually affected?" This determines everything downstream — comms, escalation, urgency.
- Decision tree. Ordered checks, each with an explicit "if yes, do X; if no, continue" branch. Order them by probability times speed: the most common cause that is fastest to check goes first. For most latency incidents, that check is "did we just deploy?"
- Verification steps. How to confirm the mitigation worked. "Latency recovered" means a specific query returning a specific range, not a vibe.
- Escalation. Who to page next, at what point, with what context. A time budget ("if not mitigated in 20 minutes, escalate") prevents the solo-hero-at-4AM failure mode.
- Rollback. The exact command to undo the most recent change, plus any preconditions. If rollback is not safe for this service (migrations, queue consumers), the runbook must say so explicitly — this is the single most expensive thing to discover mid-incident.
Two writing rules that matter more than any template. First, every command must be copy-pasteable: real namespaces, real label selectors, placeholders clearly marked. A responder at 3AM will paste exactly what you wrote. Second, write for the newest engineer on the rotation, not the person who built the service. If a step assumes knowledge, link it or inline it.
A runbook template you can copy today
Here is a complete sre runbook for a realistic alert. Replace the placeholders, drop it in your repo (runbooks belong in git, next to the alert definitions — more on why below), and use it as your runbook template for every other alert.
# Runbook: APIHighP99Latency
**Alert:** `APIHighP99Latency` — API p99 latency above 1.5s for 10 minutes
**Service:** api-gateway (owns: platform team)
**Dashboard:** https://grafana.example.com/d/api-overview
**Last verified:** 2026-06-30 by @lena
## Severity
- **P1** — p99 above 1.5s AND 5xx rate above 1%. Customers are timing out. Page secondary immediately.
- **P2** — p99 above 1.5s, error rate flat. Degraded but functional. Work it solo, 20-minute budget.
- **P3** — fired during a known batch window (02:00–02:30 UTC). Ack, verify it recovers, file a ticket to tune the alert.
## Impact check (2 minutes)
```bash
# Is the elevated latency reaching users, or absorbed by retries?
kubectl -n api top pods -l app=api-gateway
```
Check the dashboard panel "Requests by status". If 5xx is climbing, treat as P1 and post in #incidents before continuing.
## Decision tree
### 1. Did we just deploy? (most common cause)
Check the deploy annotations on the dashboard, or:
```bash
kubectl -n api rollout history deployment/api-gateway | tail -5
```
- **Deploy in the last 60 minutes → go to Rollback.** Do not debug first. Roll back, then debug.
- No recent deploy → continue.
### 2. Is one pod the outlier?
```bash
kubectl -n api top pods -l app=api-gateway --sort-by=cpu
```
- **One pod hot while others are idle →** delete it and let the ReplicaSet reschedule:
```bash
kubectl -n api delete pod <pod-name>
```
- All pods equally loaded → continue.
### 3. Is the database the bottleneck?
Check panel "DB connection pool saturation" on the dashboard.
- **Pool above 90% →** this is the downstream `postgres-primary` runbook's territory. Follow `runbooks/postgres-connection-saturation.md`.
- Pool healthy → continue.
### 4. Is traffic anomalous?
Check panel "Requests by client". A single API key above 3x its baseline means a customer script or an abuse pattern.
- **Yes →** apply the rate-limit override per `runbooks/rate-limit-override.md`.
- No → you are past the common causes. Escalate now rather than at the time budget.
## Rollback
Safe for this service (stateless, no migrations in the deploy pipeline):
```bash
kubectl -n api rollout undo deployment/api-gateway
kubectl -n api rollout status deployment/api-gateway --timeout=180s
```
## Verification
Run after any mitigation. Recovery means p99 back under 400ms for 10 consecutive minutes:
```bash
curl -sG 'https://prometheus.example.com/api/v1/query' \
--data-urlencode 'query=histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job="api-gateway"}[5m])) by (le))'
```
Then confirm the alert resolves in Alertmanager. Do not manually silence it to make it green.
## Escalation
- 20 minutes without mitigation, or any P1: page `platform-secondary` via the escalation policy.
- Database suspected: page `dba-oncall` with the pool saturation graph attached.
- Say what you ruled out, not just what is broken.
Notice what this template optimizes for: the deploy check is step one of the decision tree, rollback is a first-class section, and verification is a query with a number — not "confirm things look better."
Wire the alert to the runbook
A runbook nobody can find is a runbook that does not exist. The delivery mechanism matters as much as the content: the link must be on the page itself, not in a wiki the responder has to search. Every major alerting stack supports this natively.
Prometheus Alertmanager: the runbook_url annotation
The convention — used by kube-prometheus and most public mixins — is a runbook_url annotation on the alerting rule. Put it in the rule, and it travels with the alert everywhere Alertmanager sends it:
groups:
- name: api-gateway
rules:
- alert: APIHighP99Latency
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket{job="api-gateway"}[5m])) by (le)
) > 1.5
for: 10m
labels:
severity: page
team: platform
annotations:
summary: 'API p99 latency above 1.5s for 10 minutes'
runbook_url: 'https://github.com/acme/runbooks/blob/main/api/high-p99-latency.md'
Then make Alertmanager surface it. The route/receiver structure below follows the official configuration reference; the links block on pagerduty_configs turns the annotation into a clickable link on the PagerDuty incident:
route:
receiver: slack-default
group_by: ['alertname', 'team']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
routes:
- matchers:
- severity="page"
receiver: pagerduty-platform
receivers:
- name: pagerduty-platform
pagerduty_configs:
- routing_key_file: /etc/alertmanager/secrets/pd-routing-key
severity: 'critical'
links:
- href: '{{ (index .Alerts 0).Annotations.runbook_url }}'
text: 'Runbook'
details:
runbook: '{{ .CommonAnnotations.runbook_url }}'
firing: '{{ template "pagerduty.default.instances" .Alerts.Firing }}'
- name: slack-default
slack_configs:
- channel: '#alerts'
title: '{{ .CommonAnnotations.summary }}'
text: 'Runbook: {{ .CommonAnnotations.runbook_url }}'
actions:
- type: button
text: 'Open runbook'
url: '{{ .CommonAnnotations.runbook_url }}'
Enforce the annotation's presence in CI — a promtool check rules step plus a small script that fails the build if any rule with severity: page lacks runbook_url. That one gate is what keeps coverage from silently decaying as people add alerts.
PagerDuty: Event Orchestration
If some alerts reach PagerDuty from sources you do not control as tightly as Alertmanager (Datadog monitors, CloudWatch, custom webhooks), use Event Orchestration to normalize them. An orchestration rule can match on event content and add a note carrying the runbook link, so the incident always shows it regardless of sender.
In the orchestration UI (Automation → Event Orchestration), create a rule on the service orchestration:
- Condition:
event.summary matches part 'p99 latency' - Action — Annotate: add a note with a templated value pulled from the event payload:
Runbook: {{event.custom_details.runbook}}
Because the Alertmanager config above copies runbook_url into details.runbook, the note resolves to the real URL. For senders that use the Events API v2 directly, skip orchestration entirely and put the link in the payload — the API accepts a links array natively:
curl -s https://events.pagerduty.com/v2/enqueue \
-H 'Content-Type: application/json' \
-d '{
"routing_key": "YOUR_INTEGRATION_KEY",
"event_action": "trigger",
"payload": {
"summary": "API p99 latency above 1.5s",
"source": "prometheus",
"severity": "critical"
},
"links": [
{
"href": "https://github.com/acme/runbooks/blob/main/api/high-p99-latency.md",
"text": "Runbook: APIHighP99Latency"
}
]
}'
Either way, the responder's phone notification opens onto an incident with the runbook one tap away.
Grafana OnCall: routes and escalation chains
If you are on the Grafana stack, Grafana OnCall handles the paging side. Alert annotations — including runbook_url — pass through the Alertmanager integration into the alert group details. What you configure in OnCall is the routing and escalation: which alerts page whom, and what happens when nobody acks.
Escalation chains are configurable in the UI, but they belong in Terraform for the same reason runbooks belong in git — review and history:
resource "grafana_oncall_escalation_chain" "platform_critical" {
name = "platform-critical"
}
resource "grafana_oncall_escalation" "notify_primary" {
escalation_chain_id = grafana_oncall_escalation_chain.platform_critical.id
type = "notify_on_call_from_schedule"
notify_on_call_from_schedule = grafana_oncall_schedule.platform_primary.id
position = 0
}
resource "grafana_oncall_escalation" "wait_five_minutes" {
escalation_chain_id = grafana_oncall_escalation_chain.platform_critical.id
type = "wait"
duration = 300
position = 1
}
resource "grafana_oncall_escalation" "notify_secondary" {
escalation_chain_id = grafana_oncall_escalation_chain.platform_critical.id
type = "notify_on_call_from_schedule"
notify_on_call_from_schedule = grafana_oncall_schedule.platform_secondary.id
position = 2
}
resource "grafana_oncall_route" "critical_route" {
integration_id = grafana_oncall_integration.alertmanager.id
escalation_chain_id = grafana_oncall_escalation_chain.platform_critical.id
routing_regex = "\"severity\":\\s*\"page\""
position = 0
}
That encodes the escalation section of your runbook — primary paged immediately, secondary after five unacked minutes — as configuration instead of prose someone has to remember to follow.
Make "what changed?" answerable in one glance
Step one of the decision tree was "did we just deploy?" — because change is the leading cause of incidents, and Part 1 covered how much RCA time evaporates into reconstructing the change timeline. The fix is mechanical: every deploy posts a marker to the same place responders look at graphs.
Grafana's annotations API makes this a one-liner from any CI system. As the last step of your deploy job:
# GitHub Actions — final step of the deploy job
- name: Post deploy marker to Grafana
if: success()
env:
GRAFANA_URL: https://grafana.example.com
GRAFANA_TOKEN: ${{ secrets.GRAFANA_SA_TOKEN }}
run: |
curl -sf -X POST "$GRAFANA_URL/api/annotations" \
-H "Authorization: Bearer $GRAFANA_TOKEN" \
-H "Content-Type: application/json" \
-d "{
\"tags\": [\"deploy\", \"api-gateway\", \"production\"],
\"text\": \"Deploy ${GITHUB_SHA:0:7} by ${GITHUB_ACTOR} — ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/commit/${GITHUB_SHA}\"
}"
Omitting the time field timestamps the annotation at the moment of the call, which is what you want. Then enable the deploy tag as an annotation query on your service dashboards, and every latency graph gets a vertical line at each deploy. The 40-minute archaeology session from Part 1 becomes a glance: latency broke upward two minutes after the vertical line, done.
Do the same for anything else that changes production — feature flag flips, infrastructure applies, database migrations. Different tag per change type. The discipline generalizes: any system that can change production must announce that it did, in the place where symptoms are observed.
The honest limits
This setup is worth building. It will measurably shorten your incidents. It also has three failure modes that no amount of diligence fully eliminates, and you should go in knowing them.
Runbooks rot. The runbook describes the system as it was the day someone wrote it. Services get renamed, dashboards move, the "safe" rollback stops being safe after someone adds a migration step. The Last verified field in the template exists because unverified runbooks decay into a special kind of hazard: instructions that are confidently, specifically wrong. A quarterly review ritual helps; in practice, review always loses to feature work, and you find the rot during the incident.
Humans execute them at 3AM. A runbook is a program whose runtime is a sleep-deprived person under pressure. They skip steps, paste commands into the wrong terminal, and misread branch conditions — precisely because the situations where runbooks matter most are the situations where human execution is least reliable. Good formatting and copy-pasteable commands reduce the error rate. They do not remove the human from the loop, and the human is the slowest and most variable component in the system. This is the gap the runbook automation ladder exists to close — turning steps a human executes into steps a machine executes.
Coverage is never complete. You will write runbooks for the incidents you have already had. The incident that pages you next quarter is, with grim reliability, the one without a runbook — because if it matched a known pattern, you would probably have alerted on the precondition and prevented it. Runbooks compress the known; they are structurally silent on the novel. Expect them to handle the recurring 60–70% of pages and to leave your hardest incidents exactly as hard as before.
None of this is a reason to skip the work. A rotting runbook plus a deploy marker still beats a blank terminal. But be honest about what you have built: a paper procedure with good delivery, executed by hand.
What comes after paper
You now have a working incident response process: structured runbooks in git, links delivered on the alert, deploy markers answering "what changed." The obvious next question is the one the limits section raises — if the runbook is a decision procedure, and the decision procedure is written down, why is a human at 3AM still the execution engine?
That is Part 3: what happens when an agent executes the investigation instead — reading the same runbooks, checking the same deploy markers, and handing you a findings summary instead of a checklist. If you want to see that in practice first, CloudThinker's free tier includes 100 premium credits, no card required.