Opens on reph with the template ready to start. No account needed; sign in to keep your document.
Use it when
A service has an on-call rotation and whoever gets paged at 3am needs the dashboards, the diagnosis path and exact commands in one place
Who it's for
EngineerSRE
Part of a kit
Incident response, step 1 of 4. Runbooks and on-call handoffs before, status updates while it burns, a blameless postmortem after.
Filled example
The template filled in for a made-up company, so you can see what good looks like.
Search API runbook
Owner: Ana Lima · Last reviewed: 2026-09-20
Service overview
The Search API answers every product search on Fernhill's storefront and mobile app. It is tier 1: when it is down, customers cannot find anything to buy. It runs as search-api in the prod-eu and prod-us clusters, about 900 requests per second at peak.
Dependency
Direction
Impact if unavailable
OpenSearch cluster catalog-os
Downstream
All queries fail with 503
Pricing service
Downstream
Results show without prices (degraded, not down)
Catalog indexer
Upstream
New products stop appearing; search still works
Storefront and mobile BFF
Upstream callers
They show "search unavailable"
Dashboards & logs
Grafana "Search API overview": healthy is p95 latency under 180 ms and 5xx under 0.2%.
Loki saved search search-api errors: grep first for circuit_open and os_timeout.
Diagnosis path
Mermaid diagram (drawn as a picture in reph)
flowchart TD
A[Alert fires] --> B{Deploy of search-api in the last hour?}
B -- Yes --> R[Rollback]
B -- No --> C{catalog-os cluster red or yellow?}
C -- Yes --> E[Page the data platform on-call]
C -- No --> D{Pod CPU over 80% or queue depth over 500?}
D -- Yes --> S[Scale to 24 replicas]
D -- No --> F{One region or both?}
F -- One region --> FO[Failover to the healthy region]
F -- Both --> RS[Restart, then escalate]
Find the last good revision with kubectl -n search rollout history deployment/search-api.
Roll back, confirm 5xx recovers.
File a SRCH ticket to fix forward.
Escalation
Page the Search team secondary in PagerDuty (service "Search API"). After 30 minutes without mitigation, page the Discovery engineering manager, Marcus Webb.
Known issues
A latency spike every night at 02:00 UTC is the full reindex. It recovers by 02:20 on its own and does not need action.
The template
Show the blank template
Runbook
Owner: [your name] · Last reviewed: [date]
Service overview
What the service does, its tier (e.g. tier-1 customer-facing), and its upstream/downstream dependencies.
Dependency
Direction
Impact if unavailable
Dashboards & logs
Dashboard link: what "healthy" looks like on it
Log query saved search: what to grep for first
Diagnosis path
The questions to ask, in order, when an alert fires. Each leaf points at a procedure below.
Mermaid diagram (drawn as a picture in reph)
flowchart TD
A[Alert fires] --> B{Recent deploy in the last hour?}
B -- Yes --> R[Rollback]
B -- No --> C{Is a dependency unhealthy?}
C -- Yes --> E[Escalate to the dependency owner]
C -- No --> D{Saturated CPU, memory or queue depth?}
D -- Yes --> S[Scale]
D -- No --> F{One region or all regions?}
F -- One region --> FO[Failover]
F -- All regions --> RS[Restart, then escalate]
Alert → procedure
Alert
Meaning
Procedure
See below
Procedures
Restart
kubectl rollout restart deployment/<service>
Confirm the restart is safe (no in-flight jobs that can't resume).
Run the command above.
Watch the dashboard until error rate returns to baseline.
Scale
kubectl scale deployment/<service> --replicas=<n>
Check current load against the metric that triggers scaling.
Scale, then confirm the new replicas pass health checks.
Failover
./scripts/failover.sh --to <region>
Confirm the target region is healthy.
Run failover, verify traffic shift on the dashboard.
Announce in the incident channel.
Rollback
kubectl rollout undo deployment/<service>
Identify the last known-good revision.
Roll back, confirm error rate recovers.
File a ticket to fix-forward the root cause.
Escalation
Who to page next if the on-call can't resolve it, and how (paging tool, not just a Slack handle).
Known issues
A quirk that looks like an incident but isn't. Save the next on-call the debugging time
Questions
How often should it be reviewed?
After every incident that used it, and at least once a quarter by the service owner
Should commands be exact?
Yes. Copy-pasteable commands with placeholders in angle brackets, never prose descriptions of them
What if the flowchart does not cover the case?
Escalate, then add the missing branch in the postmortem action items
Fill it with your agent
Connect Claude to reph over MCP, then paste this. The draft arrives as suggestions you accept or reject section by section. How to connect Claude.
Find my reph doc "Runbook" with search_docs, then read it with read_doc. Fill the dependency table, alert to procedure table and diagnosis flowchart from the alert rules file, deploy manifests and last three postmortems I paste. Keep every command copy-pasteable and mark any you have not seen run. File it with suggest_edit, one replace_section op per section (heading line kept) so each can be accepted alone. Ask each open question (alerts with no procedure yet) with add_comment quoting that section's heading line, then call request_review with a one-line summary.