The Re-Mirror Storm — JavaScript Bug Hunt

Briefing

Modelled on the Amazon EBS outage of 21 April 2011: a network change routed traffic onto a low-capacity network. Volumes that lost their mirror immediately searched the cluster for space, and because they retried without backoff they consumed the remaining capacity and blocked each other — a "re-mirroring storm" that took down Reddit, Quora and Heroku for days.

mirror.js retries the search on every tick with no backoff and no cap.

Fix reMirror so a volume that cannot find space backs off instead of retrying immediately, and the cluster's free capacity is never oversubscribed.

Bug report

BUG-EBS0421 · Priority: Critical · Reported by: storage

reMirror(cluster, volumes, ticks) must return { placed, attempts }:

• each tick, every unplaced volume whose backoff has expired tries once to take one unit of cluster.free
• a failed attempt doubles that volume's backoff (starting at 1), so it waits that many ticks before trying again
• cluster.free must never go negative

Observed: with no free capacity, every volume retries on every tick. The attempt count explodes and nothing ever gets placed.

Logs

[ebs] free=0 attempts=1284000 placed=0
[ebs] control plane saturated by re-mirror requests

The code as shipped

src/storage/mirror.js (editable)

// Re-mirrors volumes that lost their replica.
exports.reMirror = function (cluster, volumes, ticks) {
  var placed = 0;
  var attempts = 0;
  var state = [];
  for (var i = 0; i < volumes.length; i++) state.push({ done: false });
  for (var t = 0; t < ticks; t++) {
    for (var v = 0; v < volumes.length; v++) {
      if (state[v].done) continue;
      attempts++;
      if (cluster.free > 0) {
        cluster.free--;
        state[v].done = true;
        placed++;
      }
    }
  }
  return { placed: placed, attempts: attempts };
};

Read-only context: src/storage/BACKOFF.js.