reph

Templates · Engineering

Incident postmortem template

Blameless review: timeline, impact, root cause, action items

Start with this template →Sign in

Opens on reph with the template ready to start. No account needed; sign in to keep your document.

Use it when

An incident is resolved and the team needs a blameless record of what happened, why, and what changes so it does not happen again

Who it's for

EngineerSREEngineering manager

Part of a kit

Incident response, step 4 of 4. Runbooks and on-call handoffs before, status updates while it burns, a blameless postmortem after.

Filled example

The template filled in for a made-up company, so you can see what good looks like.

Checkout outage: expired payment-gateway certificate

Author: Dmitri Sokolov · Date: 2026-09-14

Severity Duration Detected by Incident commander
SEV1 47 min Synthetic checkout probe Priya Raman

Summary

On 10 September, Kestrel's checkout service could not reach our payment gateway for 47 minutes because the client certificate it presents had expired. Every card payment failed with a generic error. We restored service by deploying a renewed certificate by hand.

Impact

Card checkouts failed for all web and mobile customers from 14:02 to 14:49 UTC. Orders paid with store credit were unaffected. Roughly 3,100 customers saw a failed payment, and an estimated $41,000 in orders was abandoned; about 60% of those customers retried successfully after recovery.

Duration
47 min
Users affected
3,100
Error budget burned
38%
▲ +38%
Time to detect
4 min

Timeline

Mermaid diagram (drawn as a picture in reph)
timeline
    title Incident timeline (UTC)
    14.02 : Gateway certificate expires, payments start failing
    14.06 : Synthetic checkout probe pages on-call
    14.11 : On-call acknowledged, incident declared
    14.31 : Root cause identified as expired client cert
    14.44 : Renewed cert deployed by hand
    14.49 : Payment success rate back to baseline
Time (UTC) Event
14:02 Client certificate for the gateway expires
14:06 Synthetic checkout probe fires, pages on-call
14:11 On-call (Dmitri) acknowledges, declares SEV1
14:31 TLS handshake error in gateway logs points to the cert
14:44 Renewed cert issued and deployed by hand
14:49 Payment success rate recovers to 99.4%

Root cause

  1. Why did payments fail? The gateway rejected our expired client certificate.
  2. Why was it expired? It was issued manually in 2025 with a one-year lifetime and never renewed.
  3. Why was it not renewed? It lives outside cert-manager, so no automated renewal covered it.
  4. Why did nobody notice? Our expiry alerts only watch certificates cert-manager issues.
  5. Why is it outside cert-manager? The gateway requires a CSR flow we never automated.

What went well

  • The synthetic probe detected the failure within four minutes.
  • The payments runbook pointed straight at the gateway logs.

What went poorly

  • Twenty minutes went to suspecting the morning's deploy before checking TLS errors.
  • Issuing a new cert needed a vendor portal login only two people had.

Where we got lucky

  • One of those two people was online. On a weekend this would have lasted hours.

Action items

Action Owner Priority Due
Automate gateway CSR flow in cert-manager (PAY-812) Dmitri Sokolov P0 2026-09-26
Alert on every TLS cert in the inventory, not just managed ones (OBS-301) Ana Lima P1 2026-10-03
Add TLS handshake errors to the payments dashboard (OBS-302) Ana Lima P2 2026-10-10
Share vendor portal access with the on-call group (SEC-77) Priya Raman P1 2026-09-19

Lessons

A certificate nobody's tooling knows about is a certificate that will expire. Inventory beats memory.

The template

Show the blank template

Incident postmortem

Author: [your name] · Date: [date]

Severity Duration Detected by Incident commander

Summary

What happened, in two or three sentences a VP could read and understand.

Impact

Who was affected, how (errors, latency, data loss), and for how long. Numbers, not adjectives: "12% of write requests 502'd for 34 minutes," not "some users had issues."

Duration
0 min
Users affected
0
Error budget burned
0%
Time to detect
0 min

Timeline

Times in UTC. The diagram is the at-a-glance version; the table keeps the detail.

Mermaid diagram (drawn as a picture in reph)
timeline
    title Incident timeline (UTC)
    Trigger : Change that introduced the issue (time)
    Detected : First alert fired (time)
    Acknowledged : On-call acknowledged (time)
    Mitigated : Mitigation applied (time)
    Resolved : Service back to normal (time)
Time (UTC) Event
First deploy / change that introduced the issue
First alert fired
On-call acknowledged
Root cause identified
Mitigation applied
Resolved

Root cause

Not "what broke" but "why did our system allow it to break this way." Five whys:

  1. Why did the incident happen?
  2. Why?
  3. Why?
  4. Why?
  5. Why?

What went well

What worked as designed during this incident (detection, runbooks, communication) is worth reinforcing.

What went poorly

Specific gaps in tooling, process or knowledge that made this worse or slower to resolve than it should have been.

Where we got lucky

  • Honest. This is what separates a real postmortem from a victory lap

Action items

Action Owner Priority Due

Every item above is tracked in the issue tracker, not just this doc. Link the ticket, don't duplicate the state.

Lessons

The one or two things worth remembering six months from now, independent of the specific fix.

Questions

When should the postmortem be written?
Within five working days, while memories and logs are fresh
Is it really blameless?
Yes. Name systems and decisions, not people, and ask why the system allowed the mistake
Which incidents need one?
Every SEV1 and SEV2, plus any near miss the team wants to learn from

Fill it with your agent

Connect Claude to reph over MCP, then paste this. The draft arrives as suggestions you accept or reject section by section. How to connect Claude.

Find my reph doc "Incident postmortem" with search_docs, then read it with read_doc. Draft the summary, impact numbers and timeline from the incident channel export, paging tool alert history and deploy log I paste. Keep it blameless, and leave root cause and action items as questions, not guesses. File it with suggest_edit, one replace_section op per section (heading line kept) so each can be accepted alone. Ask each open question (every timestamp or number you could not confirm) with add_comment quoting that section's heading line, then call request_review with a one-line summary.

Related templates

Runbook

Operate a service: dashboards, alerts, and step-by-step procedures

EngineerSRE

On-call handoff

What the next on-call needs to know

EngineerSRE

Incident status update

Stakeholder and customer communication during an active incident: current status, impact, what we know, next update time and a running update log

SREEngineerEngineering managerCommunicationsCustomer success

ADR

Architecture decision record: one decision, its context and consequences

EngineerTech leadEngineering manager

Agent instructions

The charter your AI coding agents work from, project overview through review checklist

EngineerTech leadEngineering managerSRE

AI feature eval plan

How you will measure an AI feature before and after launch: the task, a quality rubric, the eval dataset, metrics, baseline vs target, human review, regression gates and a cost and latency budget

EngineerTech leadProduct managerResearcher

Start with this template →Sign in