One Typo, Half the Internet — JavaScript Bug Hunt
Inspired by the 2017 AWS S3 outage: an engineer running a routine playbook mistyped one number and removed far more capacity than intended — taking a chunk…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Validation, Operations
- Modelled on: AWS S3 · 2017
- Visible tests: a sane removal removes exactly that many; an oversized request is clamped to keep MIN_ALIVE; garbage input throws instead of guessing
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-S3-0228
- Reported by: incident review
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- an oversized request is clamped to keep MIN_ALIVE — never below MIN_ALIVE: expected 2 but got 0
- garbage input throws instead of guessing — letter O is not a number: expected an error to be thrown
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by the 2017 AWS S3 outage: an engineer running a routine playbook mistyped one number and removed far more capacity than intended — taking a chunk of the internet down with it.
The tooling should have refused. Make decommission.js refuse.
Bug report
BUG-S3-0228 · Priority: Critical · Reported by: incident review
planRemoval(requestedCount, pool) rules:
- requestedCount must parse as a positive integer, else throw "invalid count"
- at least MIN_ALIVE (2) servers must remain: clamp the removal to pool.length - 2
- removing from a pool of 2 or fewer removes nothing
Observed: "1O0" (letter O) parsed as 1, "100" removed 100 of 10, pool hit zero.
Logs
[decom] pool=10 requested=100 removed=10 remaining=0
[decom] index-fleet offline; GET /* -> 500The code as shipped
src/ops/decommission.js (editable)
var MIN_ALIVE = 2;
// Plans a capacity removal. Returns the servers that remain.
exports.planRemoval = function (requestedCount, pool) {
var count = parseInt(requestedCount, 10);
return pool.slice(0, pool.length - count);
};
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.