The Re-Mirror Storm — JavaScript Bug Hunt
Modelled on the Amazon EBS outage of 21 April 2011: a network change routed traffic onto a low-capacity network.
- Language: JavaScript
- Layer: Backend
- Difficulty: Hard
- Concepts: Retries, Backpressure
- Modelled on: Amazon EBS · 2011
- Visible tests: volumes are placed when capacity exists; a full cluster does not produce a storm
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-EBS0421
- Reported by: storage
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- a full cluster does not produce a storm — backoff must limit retries, saw 640: expected a truthy value but got false
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Amazon EBS outage of 21 April 2011: a network change routed traffic onto a low-capacity network. Volumes that lost their mirror immediately searched the cluster for space, and because they retried without backoff they consumed the remaining capacity and blocked each other — a "re-mirroring storm" that took down Reddit, Quora and Heroku for days.
mirror.js retries the search on every tick with no backoff and no cap.
Fix reMirror so a volume that cannot find space backs off instead of retrying immediately, and the cluster's free capacity is never oversubscribed.
Bug report
BUG-EBS0421 · Priority: Critical · Reported by: storage
reMirror(cluster, volumes, ticks) must return { placed, attempts }:
- each tick, every unplaced volume whose backoff has expired tries once to take one unit of cluster.free
- a failed attempt doubles that volume's backoff (starting at 1), so it waits that many ticks before trying again
- cluster.free must never go negative
Observed: with no free capacity, every volume retries on every tick. The attempt count explodes and nothing ever gets placed.
Logs
[ebs] free=0 attempts=1284000 placed=0
[ebs] control plane saturated by re-mirror requestsThe code as shipped
src/storage/mirror.js (editable)
// Re-mirrors volumes that lost their replica.
exports.reMirror = function (cluster, volumes, ticks) {
var placed = 0;
var attempts = 0;
var state = [];
for (var i = 0; i < volumes.length; i++) state.push({ done: false });
for (var t = 0; t < ticks; t++) {
for (var v = 0; v < volumes.length; v++) {
if (state[v].done) continue;
attempts++;
if (cluster.free > 0) {
cluster.free--;
state[v].done = true;
placed++;
}
}
}
return { placed: placed, attempts: attempts };
};
Read-only context: src/storage/BACKOFF.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.