Every Region in One Flight — JavaScript Bug Hunt

Briefing

Modelled on the Microsoft Azure Storage outage of 18–19 November 2014. A performance change to Azure Storage had been tested and "flighted" on a small set of production clusters, but it was then applied across most regions at once instead of following the incremental flighting process. The change contained a bug that sent the blob front-ends into an infinite loop, and storage — and the services built on it — went down in many regions together. Microsoft's post-incident review said the standard flighting policy had not been followed and moved to enforcing it in the deployment tooling.

This reconstruction's planner.js splits a deployment into waves: one canary region, then small batches. A change marked alreadyTested skips all that and goes everywhere in a single wave.

Fix planWaves so every change is staged, no matter what the change claims about itself.

Bug report

BUG-FLIGHT · Priority: Critical · Reported by: incident review

planWaves(change, regions) returns the list of deployment waves:

• no regions -> []
• wave 1 is exactly the first region (the canary)
• the remaining regions follow in order, in batches of WAVE_SIZE (rollout-config.js; the last batch may be shorter)
• the shape depends ONLY on the region list. No property of the change (alreadyTested, priority, author ...) may shorten or skip the staging.

runner.run(waves, isHealthyAfter) stops after the first unhealthy wave, so a bad change must only ever reach the canary.

Observed: a change flagged alreadyTested is planned as one wave holding every region; when it misbehaves, every region is hit before anyone can stop it.

Logs

[deploy] change perf-7731 alreadyTested=true -> 1 wave, 5 regions
[blob-fe] us-east: request loop detected, CPU 100%
[blob-fe] eu-west: request loop detected, CPU 100%

The code as shipped

src/deploy/planner.js (editable)

var cfg = require("./rollout-config");

// Splits a change's deployment into waves of regions.
exports.planWaves = function (change, regions) {
  if (regions.length === 0) return [];
  if (change.alreadyTested) return [regions.slice()];
  var waves = [[regions[0]]];
  for (var i = 1; i < regions.length; i += cfg.WAVE_SIZE) {
    waves.push(regions.slice(i, i + cfg.WAVE_SIZE));
  }
  return waves;
};

Read-only context: src/deploy/rollout-config.js, src/deploy/runner.js.