Incident: Delayed payment confirmations
Incident commander: Marco Ruiz · Date: 2026-09-22 · Started: 14:05 UTC · Comms lead: Hana Sato
| Status | Severity | Next update |
|---|---|---|
| Monitoring | SEV-2 | 16:30 UTC |
flowchart LR
A[Investigating 14:12] --> B[Identified 14:48]
B --> C[Monitoring 15:35]
C --> D[Resolved]
C -. regression .-> B
Current status
Payments made in Fernhill Pay are going through, and confirmations are arriving again. Some confirmations sent between 14:05 and 15:30 UTC arrived late; we are watching closely before we call this resolved.
Impact
- Affected: Merchants in the EU region. About 11% of payments between 14:05 and 15:30 UTC showed as "Pending" for up to 40 minutes before confirming.
- Not affected: No payments were lost or charged twice. US and APAC merchants were not affected.
- Workaround: None needed now. Merchants who refunded a "Pending" payment during the window should check the payment list; support can reverse duplicate refunds.
What we know
- A configuration change at 14:02 UTC reduced the capacity of the service that sends confirmations in the EU region.
- Rolling back the change at 15:31 UTC restored normal confirmation times (under 3 seconds).
- We are still confirming that every delayed confirmation has been delivered; so far 41,200 of 41,350 have.
What we are doing
- Replaying the remaining 150 confirmations, expected to finish by 16:15 UTC.
- Holding further configuration changes in the EU region until the review is complete.
Next update
Next update by 16:30 UTC.
Update log
| Time (UTC) | Status | Update |
|---|---|---|
| 15:40 | Monitoring | A fix is in place and confirmations are arriving normally. We are delivering confirmations delayed earlier today. |
| 15:05 | Identified | We have found a configuration change that caused delays and are rolling it back. |
| 14:35 | Investigating | Confirmations for some EU payments are delayed. Payments are not lost or duplicated. |
| 14:12 | Investigating | We are investigating reports of payments stuck in "Pending" for EU merchants. |
Internal notes (do not publish)
Incident channel: #inc-2026-09-22-eu-confirmations. Dashboard: Payments EU latency. Config change: CHG-4471. Postmortem owner: Marco Ruiz, draft due September 26.