One Customer's Config Took Down the Edge — Bug Hunt
Modelled on the Fastly global outage (8 June 2021): a latent bug shipped weeks earlier was triggered when one customer pushed a valid configuration change.
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Configuration, Defensive Coding
- Modelled on: Fastly · 2021
- Visible tests: an explicit backend is used; a missing backends map falls back; an empty backends map falls back
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-FASTLY
- Reported by: edge SRE
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a missing backends map falls back — Cannot read properties of undefined (reading 'a.example')
- an empty backends map falls back — expected "pool-d" but got undefined
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Fastly global outage (8 June 2021): a latent bug shipped weeks earlier was triggered when one customer pushed a valid configuration change. 85% of the CDN's network started returning errors within a minute.
router.js resolves a hostname to a backend and assumes the backend map is always populated. A customer whose config legitimately has no explicit backends takes the whole request path down.
Fix resolveBackend so an empty or missing map degrades to the default backend instead of failing.
Bug report
BUG-FASTLY · Priority: Critical · Reported by: edge SRE
resolveBackend(config, host) must return:
- config.backends[host] when that entry exists
- config.defaultBackend when it does not, or when backends is missing/empty
- the string "origin" when there is no default either
Observed: a config whose backends object is absent throws, and a config whose backends is empty returns undefined — both surface as a 503 for every request served by that edge node.
Logs
[edge] TypeError: Cannot read property 'www.example.com' of undefined
[edge] 85% of POPs returning 503The code as shipped
src/edge/router.js (editable)
// Maps a hostname to the backend that should serve it.
exports.resolveBackend = function (config, host) {
return config.backends[host];
};
Read-only context: src/edge/CONTRACT.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.