Move file attachments from NFS to object storage
Author: Leo Marchetti · Date: 2026-07-06
Goal & scope
Move all 38 million customer attachments (21 TB) from the shared NFS volume to S3-compatible object storage, and switch the app to read and write there. Out of scope: thumbnails (regenerated on demand) and the legacy export archive.
Current vs target state
flowchart LR
subgraph Current
A[App servers] --> N[NFS volume]
end
subgraph Target
B[App servers] --> S[Object storage]
S --> C[CDN]
end
N -- backfill --> S
Phases
| Phase | Steps | Exit criteria |
|---|---|---|
| 1. Prepare | Create buckets, IAM, lifecycle rules | Write test passes from every app node |
| 2. Dual-write | New uploads go to both stores | 7 days with zero dual-write errors |
| 3. Backfill | Copy historical files in key order | 100% copied, checksums match |
| 4. Cutover | Flip reads to object storage | 48 hours with error rate under 0.01% |
| 5. Decommission old | Unmount NFS, keep snapshot 30 days | Finance confirms storage contract ended |
Dual-write / backfill strategy
The upload handler writes NFS first, then object storage, and records failures in a retry table. The backfill job copies files oldest first at up to 40 files per second, throttled when NFS latency passes 50 ms.
Verification
Nightly job compares object count and SHA-256 for a 1% random sample, plus a full count per customer. Shadow reads fetch from both stores for 5% of downloads and log mismatches.
Progress
Rollback plan
Trigger: download error rate above 0.5% for 10 minutes, or any confirmed missing file. Steps: flip the attachments_read_source flag back to nfs (takes effect in under a minute), keep dual-write on, and open an incident. NFS stays authoritative until phase 5.
Communication
- Support and customer success: two weeks' notice, with a macro for "file not found" reports
- Status updates in #eng-migrations every weekday at 10:00 during phases 3 and 4
Timeline
gantt
dateFormat YYYY-MM-DD
title Attachments migration
section Prepare
Prepare :p1, 2026-07-13, 5d
section Move
Dual-write :p2, after p1, 7d
Backfill :p3, after p2, 14d
Cutover :milestone, p4, after p3, 1d
section Clean up
Decommission old :p5, after p4, 30d
| Milestone | Date |
|---|---|
| Dual-write on | 2026-07-18 |
| Backfill complete | 2026-08-07 |
| Cutover | 2026-08-08 |
| NFS decommissioned | 2026-09-08 |
Risks
| Risk | Mitigation |
|---|---|
| Backfill slows production NFS | Throttle on latency, run hardest batches overnight |
| Filenames with unusual characters break keys | Keys are content hashes; names live in the database |