The Re-Mirror Storm

Hard

· Backend

Incident

SEV-1

BUG-EBS0421

Priority: Critical · reported by storage

Time to fix: 35 min

Failing checks on the shipped build

  • a full cluster does not produce a storm

    backoff must limit retries, saw 640: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Amazon EBS · 2011

Modelled on the Amazon EBS outage of 21 April 2011: a network change routed traffic onto a low-capacity network. Volumes that lost their mirror immediately searched the cluster for space, and because they retried without backoff they consumed the remaining capacity and blocked each other — a "re-mirroring storm" that took down Reddit, Quora and Heroku for days.

mirror.js retries the search on every tick with no backoff and no cap.

Fix reMirror so a volume that cannot find space backs off instead of retrying immediately, and the cluster's free capacity is never oversubscribed.

Code map

// Re-mirrors volumes that lost their replica.
exports.reMirror = function (cluster, volumes, ticks) {
  var placed = 0;
  var attempts = 0;
  var state = [];
  for (var i = 0; i < volumes.length; i++) state.push({ done: false });
  for (var t = 0; t < ticks; t++) {
    for (var v = 0; v < volumes.length; v++) {
      if (state[v].done) continue;
      attempts++;
      if (cluster.free > 0) {
        cluster.free--;
        state[v].done = true;
        placed++;
      }
    }
  }
  return { placed: placed, attempts: attempts };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Re-Mirror Storm — JavaScript Bug Hunt

Modelled on the Amazon EBS outage of 21 April 2011: a network change routed traffic onto a low-capacity network.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Retries, Backpressure
  • Modelled on: Amazon EBS · 2011
  • Visible tests: volumes are placed when capacity exists; a full cluster does not produce a storm
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-EBS0421
  • Reported by: storage
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a full cluster does not produce a storm — backoff must limit retries, saw 640: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Amazon EBS outage of 21 April 2011: a network change routed traffic onto a low-capacity network. Volumes that lost their mirror immediately searched the cluster for space, and because they retried without backoff they consumed the remaining capacity and blocked each other — a "re-mirroring storm" that took down Reddit, Quora and Heroku for days.

mirror.js retries the search on every tick with no backoff and no cap.

Fix reMirror so a volume that cannot find space backs off instead of retrying immediately, and the cluster's free capacity is never oversubscribed.

Bug report

BUG-EBS0421 · Priority: Critical · Reported by: storage

reMirror(cluster, volumes, ticks) must return { placed, attempts }:

  • each tick, every unplaced volume whose backoff has expired tries once to take one unit of cluster.free
  • a failed attempt doubles that volume's backoff (starting at 1), so it waits that many ticks before trying again
  • cluster.free must never go negative

Observed: with no free capacity, every volume retries on every tick. The attempt count explodes and nothing ever gets placed.

Logs

[ebs] free=0 attempts=1284000 placed=0
[ebs] control plane saturated by re-mirror requests

The code as shipped

src/storage/mirror.js (editable)

// Re-mirrors volumes that lost their replica.
exports.reMirror = function (cluster, volumes, ticks) {
  var placed = 0;
  var attempts = 0;
  var state = [];
  for (var i = 0; i < volumes.length; i++) state.push({ done: false });
  for (var t = 0; t < ticks; t++) {
    for (var v = 0; v < volumes.length; v++) {
      if (state[v].done) continue;
      attempts++;
      if (cluster.free > 0) {
        cluster.free--;
        state[v].done = true;
        placed++;
      }
    }
  }
  return { placed: placed, attempts: attempts };
};

Read-only context: src/storage/BACKOFF.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.