Add retry with backoff to webhook delivery

Author: Diego Alvarez · Date: 2026-09-30 · Closes: #2184

What

Webhook deliveries that fail with a 5xx or a timeout are now retried up to 6 times with exponential backoff, instead of being dropped after the first failure.

Why

• Issue: #2184, "Webhooks lost when customer endpoint blips"
• Context: Lumen Labs support logged 37 tickets in August from customers whose endpoint was briefly down during a deploy. Every one of those events was lost for good.

How it works

• New retry_at and attempt columns on webhook_deliveries (migration 0091).
• The existing delivery worker now picks up rows where retry_at <= now(), so no new process is needed.
• Backoff is 30s, 2m, 10m, 30m, 2h, 6h, with 10% jitter. After the 6th failure the delivery is marked failed and shows in the dashboard.
• 4xx responses are not retried, since those are the customer rejecting the payload. Read delivery/retry.ts first.

How to test

1. Run make dev and register a webhook pointing at http://localhost:9999/hook (nothing listening).
2. Trigger an event with make fake-event. Expected: the delivery row shows attempt = 1 and a retry_at about 30 seconds out.
3. Start npx http-echo -p 9999 and wait for the retry. Expected: the row flips to delivered and the echo server logs the payload once.
4. Point the webhook at an endpoint returning 400. Expected: one attempt, then failed, no retries.

Automated tests: unit tests for the backoff schedule and the 4xx/5xx split; an integration test for the worker pickup. Jitter is not asserted exactly.

Screenshots

Risk and rollback

• Risk level: Medium
• What it touches: a migration on a table with 48 million rows (adds two nullable columns, no rewrite), and the delivery worker.
• Rollback: set WEBHOOK_RETRIES=0 to stop retrying without a deploy; the columns are harmless if left in place.

Checklist

• Tests added or updated, and passing locally
• Docs, changelog or release notes updated if behavior changed
• No secrets, debug logging or commented-out code
• Feature flag or migration reviewed by its owner (waiting on Sana from data platform)
• Accessibility checked for UI changes (keyboard, contrast, labels)