Opens on reph with the template ready to start. No account needed; sign in to keep your document.
Use it when
A change spans teams or is expensive to reverse, and reviewers need the reasoning before code is written
Who it's for
EngineerTech leadEngineering manager
Part of a kit
Engineering rituals, step 2 of 7. Propose, decide, build and review: RFC, design doc, ADR, spec, PR description and review checklist.
Filled example
The template filled in for a made-up company, so you can see what good looks like.
Webhook delivery with retries and a dead-letter queue
Author: Tomas Okafor · Date: 2026-04-21 · Status: In review · Reviewers: Priya Raman, Leo Marchetti
Lumen Labs sends webhooks inline from the API process, so a slow customer endpoint slows our API and a failed delivery is lost forever. This doc proposes a queue-backed delivery worker with retries and a dead-letter queue. We need sign-off from platform and SRE by April 28.
Context
About 1.2 million webhooks go out per day. Delivery is a synchronous HTTP call inside the request that triggered it. Last quarter, 0.6% of deliveries failed with no retry, and two customer timeouts (INC-2301, INC-2318) pushed API p99 latency from 310 ms to over 2 seconds.
Goals
No webhook is lost: every event is delivered or lands in a dead-letter queue within 24 hours
API p99 latency no longer depends on customer endpoint speed
Customers can see and replay failed deliveries
Non-goals
Ordering guarantees across events (we document at-least-once, unordered)
A new public event schema
Proposed design
The API writes an outbox row in the same transaction as the change. A relay publishes outbox rows to a queue, and a pool of delivery workers sends them with exponential backoff.
Mermaid diagram (drawn as a picture in reph)
sequenceDiagram
participant API
participant Outbox
participant Queue
participant Worker
participant Customer
API->>Outbox: insert event (same transaction)
Outbox->>Queue: relay publishes
Queue->>Worker: deliver attempt
Worker->>Customer: POST signed payload
Customer-->>Worker: 2xx or failure
Worker->>Queue: retry with backoff or move to DLQ
Data model
Entity
Field
Type
Notes
webhook_outbox
id
uuid
Also the idempotency key sent to customers
webhook_outbox
attempts
int
Capped at 12 (about 24 hours of backoff)
webhook_dlq
last_error
text
Shown in the customer dashboard
API / interfaces
New POST /v1/webhooks/deliveries/{id}/replay, scoped webhooks:write.
Failure modes
Failure
Detection
Impact
Mitigation
Queue unavailable
Relay lag alert above 2 minutes
Delivery delayed, not lost
Outbox keeps rows until published
Customer down for a day
DLQ growth per customer
Events land in DLQ
Replay from dashboard
Alternatives considered
Option
Pros
Cons
Why not
In-process retries
Small change
Still ties up API workers
Does not meet goal 2
Managed webhook vendor
Fast to ship
$4,800 per month, data leaves our region
Cost and residency
Security, privacy & compliance
Payloads are already signed with HMAC; the DLQ stores payloads for 14 days, matching existing log retention.
Rollout plan
Observability
Delivery success rate, attempts per event, relay lag and DLQ size per customer, with alerts on relay lag and success rate below 99%.
Open questions
Appendix
Load test: one worker sustains 180 deliveries per second against a 300 ms mock endpoint.
Review
Review by: 2026-04-28
Reviewer
Role
Status
Priya Raman
Platform tech lead
Approved
Leo Marchetti
SRE
Changes requested
Decision record
Decision
Date
Decided by
Notes
Needs changes
2026-04-27
Priya Raman
Add a per-customer concurrency cap before the 10% stage
One paragraph a busy reader can stop after: what you're building, why now, and what you need from reviewers.
Context
What exists today, what's wrong with it, and who feels the pain. Link data, incidents or tickets.
Goals
Outcome, stated so it can be checked afterwards
Non-goals
Plausible things this doc deliberately does not do (prevents scope creep in review)
Proposed design
The design at a level a new teammate could implement from. Start with the big picture, then zoom in.
Mermaid diagram (drawn as a picture in reph)
sequenceDiagram
participant Client
participant Service
participant Store
Client->>Service: request
Service->>Store: read / write
Store-->>Service: result
Service-->>Client: response
Data model
Entity
Field
Type
Notes
API / interfaces
New or changed endpoints, events, or function signatures.
Failure modes
Failure
Detection
Impact
Mitigation
Alternatives considered
Option
Pros
Cons
Why not
Security, privacy & compliance
New data collected, who can access it, threat model deltas.
Rollout plan
Observability
Metrics, logs and alerts that tell you it's working, and that it's broken.
Open questions
Appendix
Benchmarks, prior art, raw notes.
Review
Who must sign off, and by when. Reviewers record their call in the decision record so the outcome is findable later.
Review by: [date, usually five working days out]
Reviewer
Role
Status
Pending
Pending
Decision record
Decision
Date
Decided by
Notes
Accepted / Rejected / Needs changes
Questions
How long should it be?
Two to five pages, long enough to implement from and short enough to read in one sitting
When is a design doc overkill?
When the change is easy to reverse and touches one team, a good pull request description is enough
Who approves it?
The reviewers named in the Review section, recorded in the decision record at the end
Fill it with your agent
Connect Claude to reph over MCP, then paste this. The draft arrives as suggestions you accept or reject section by section. How to connect Claude.
Find my reph doc "Design doc" with search_docs, then read it with read_doc. Draft Context, Goals, Proposed design and Failure modes from the tickets, incident reports, metrics dashboards and code I link, fill Alternatives considered with at least two rejected options and why, and list reviewers in the Review section. File it with suggest_edit, one replace_section op per section (heading line kept) so each can be accepted alone. Ask each open question (anything you could not resolve) with add_comment quoting that section's heading line, then call request_review with a one-line summary.