RFC: Move service-to-service auth from shared API keys to mTLS
Author: Priya Raman · Date: 2026-09-14 · Status: Open for comment
Northwind's internal services authenticate to each other with long-lived shared API keys. This RFC proposes replacing them with mutual TLS issued by our existing service mesh. We need a go or no-go from platform, payments and data by September 28 so the work can land in Q4 planning.
Summary
Every internal service call would carry a short-lived certificate identifying the calling service, issued and rotated automatically by the mesh. Shared API keys would be retired over one quarter, service by service.
Motivation
- Two of the last five security findings (SEC-112, SEC-131) were API keys committed to repos or pasted in chat.
- Rotating a key today takes a coordinated deploy across up to 9 services; we have not rotated the billing key in 14 months.
- Audit asks "which service called payments at 03:12?" and we cannot answer, because 6 services share one key.
Doing nothing means the next leaked key is a full incident with a multi-team rotation.
Proposal
What changes
- The mesh issues each workload a 24-hour certificate bound to its service identity.
- Services authorize callers by identity (an allowlist per endpoint) instead of by key.
- API keys are removed from the secret store once a service has had zero key-authenticated calls for 14 days.
What stays the same
- External, customer-facing auth is untouched.
- Application code that does not handle auth today needs no changes.
Alternatives
| Option | Pros | Cons | Why not |
|---|---|---|---|
| Do nothing | No work | Leaks stay likely, no caller identity | Fails the audit requirement |
| Signed JWTs per call | Works outside the mesh | Every service must add verification code | Twice the engineering effort |
| Rotate keys monthly | Small change | Still shared, still no identity | Treats the symptom only |
Drawbacks
- Debugging TLS handshake failures is harder than reading a 401.
- The two batch jobs outside the mesh (ledger export, CSV import) need a sidecar or an exception.
- About 3 engineer-weeks per team across 7 teams.
Unresolved questions
Review
Comment period: 2026-09-14 to 2026-09-28. Leave comments inline; summarize resolved threads here before deciding.
| Reviewer | Role | Status |
|---|---|---|
| Tomas Ek | Platform lead | Approved |
| Aisha Bello | Payments tech lead | Changes requested |
| Lin Zhou | Data engineering | Pending |
Review by: 2026-09-28
Decision record
| Decision | Date | Decided by | Notes |
|---|---|---|---|
| Accepted with changes | 2026-09-29 | Tomas Ek, Aisha Bello | Batch jobs keep scoped keys until Q1; design doc due October 10 |