Every Region in One Flight

Easy

· Backend

Incident

SEV-1

BUG-FLIGHT

Priority: Critical · reported by incident review

Time to fix: 10 min

Failing checks on the shipped build

  • a change that claims to be tested is staged too

    no flag skips flighting: expected [["a"],["b","c"],["d","e"]] but got [["a","b","c","d","e"]]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Microsoft Azure · 2014

Modelled on the Microsoft Azure Storage outage of 18–19 November 2014. A performance change to Azure Storage had been tested and "flighted" on a small set of production clusters, but it was then applied across most regions at once instead of following the incremental flighting process. The change contained a bug that sent the blob front-ends into an infinite loop, and storage — and the services built on it — went down in many regions together. Microsoft's post-incident review said the standard flighting policy had not been followed and moved to enforcing it in the deployment tooling.

This reconstruction's planner.js splits a deployment into waves: one canary region, then small batches. A change marked alreadyTested skips all that and goes everywhere in a single wave.

Fix planWaves so every change is staged, no matter what the change claims about itself.

Code map

var cfg = require("./rollout-config");

// Splits a change's deployment into waves of regions.
exports.planWaves = function (change, regions) {
  if (regions.length === 0) return [];
  if (change.alreadyTested) return [regions.slice()];
  var waves = [[regions[0]]];
  for (var i = 1; i < regions.length; i += cfg.WAVE_SIZE) {
    waves.push(regions.slice(i, i + cfg.WAVE_SIZE));
  }
  return waves;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Every Region in One Flight — JavaScript Bug Hunt

Modelled on the Microsoft Azure Storage outage of 18–19 November 2014.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Config, Deploys
  • Modelled on: Microsoft Azure · 2014
  • Visible tests: an ordinary change is staged canary-first; a change that claims to be tested is staged too
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-FLIGHT
  • Reported by: incident review
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • a change that claims to be tested is staged too — no flag skips flighting: expected [["a"],["b","c"],["d","e"]] but got [["a","b","c","d","e"]]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Microsoft Azure Storage outage of 18–19 November 2014. A performance change to Azure Storage had been tested and "flighted" on a small set of production clusters, but it was then applied across most regions at once instead of following the incremental flighting process. The change contained a bug that sent the blob front-ends into an infinite loop, and storage — and the services built on it — went down in many regions together. Microsoft's post-incident review said the standard flighting policy had not been followed and moved to enforcing it in the deployment tooling.

This reconstruction's planner.js splits a deployment into waves: one canary region, then small batches. A change marked alreadyTested skips all that and goes everywhere in a single wave.

Fix planWaves so every change is staged, no matter what the change claims about itself.

Bug report

BUG-FLIGHT · Priority: Critical · Reported by: incident review

planWaves(change, regions) returns the list of deployment waves:

  • no regions -> []
  • wave 1 is exactly the first region (the canary)
  • the remaining regions follow in order, in batches of WAVE_SIZE (rollout-config.js; the last batch may be shorter)
  • the shape depends ONLY on the region list. No property of the change (alreadyTested, priority, author ...) may shorten or skip the staging.

runner.run(waves, isHealthyAfter) stops after the first unhealthy wave, so a bad change must only ever reach the canary.

Observed: a change flagged alreadyTested is planned as one wave holding every region; when it misbehaves, every region is hit before anyone can stop it.

Logs

[deploy] change perf-7731 alreadyTested=true -> 1 wave, 5 regions
[blob-fe] us-east: request loop detected, CPU 100%
[blob-fe] eu-west: request loop detected, CPU 100%

The code as shipped

src/deploy/planner.js (editable)

var cfg = require("./rollout-config");

// Splits a change's deployment into waves of regions.
exports.planWaves = function (change, regions) {
  if (regions.length === 0) return [];
  if (change.alreadyTested) return [regions.slice()];
  var waves = [[regions[0]]];
  for (var i = 1; i < regions.length; i += cfg.WAVE_SIZE) {
    waves.push(regions.slice(i, i + cfg.WAVE_SIZE));
  }
  return waves;
};

Read-only context: src/deploy/rollout-config.js, src/deploy/runner.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.