The Canary That Failed and Was Overruled

Medium

· Backend

Incident

SEV-1

BUG-GCE0411

Priority: Critical · reported by SRE

Time to fix: 20 min

Failing checks on the shipped build

  • one failing canary check aborts

    expected {"status":"aborted","reason":"canary","pushed":[]} but got {"status":"done","reason":null,"pushed":["s1","s2"]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Google Compute Engine · 2016

Modelled on the Google Compute Engine outage of 11 April 2016, when GCE instances lost external connectivity in every region for about 18 minutes. Google's post-mortem: engineers removed an unused IP block from the network configuration, and a timing quirk in the management software produced an inconsistent, effectively empty configuration. The canary step correctly found the new configuration unsafe — but a second bug meant that failure did not stop the rollout, and every site withdrew its announcements of Google Cloud's IP blocks.

This reconstruction's propagate.js runs a set of canary checks and then pushes the config everywhere. It also never questions a config with no IP blocks in it.

Fix propagate so a failed canary aborts and an empty config is never pushed.

Code map

// Pushes a new IP-block announcement config to every site: the canary
// first, then everyone else.
exports.propagate = function (config, sites, runChecks) {
  var canary = sites[0];
  var results = runChecks(canary, config);
  var failed = false;
  for (var i = 0; i < results.length; i++) {
    failed = !results[i].ok;
  }
  if (failed) return { status: "aborted", reason: "canary", pushed: [] };
  return { status: "done", reason: null, pushed: sites.slice() };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Canary That Failed and Was Overruled — Bug Hunt

Modelled on the Google Compute Engine outage of 11 April 2016, when GCE instances lost external connectivity in every region for about 18 minutes.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Networking, Config, Deploys
  • Modelled on: Google Compute Engine · 2016
  • Visible tests: a healthy config reaches every site; one failing canary check aborts
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-GCE0411
  • Reported by: SRE
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • one failing canary check aborts — expected {"status":"aborted","reason":"canary","pushed":[]} but got {"status":"done","reason":null,"pushed":["s1","s2"]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Google Compute Engine outage of 11 April 2016, when GCE instances lost external connectivity in every region for about 18 minutes. Google's post-mortem: engineers removed an unused IP block from the network configuration, and a timing quirk in the management software produced an inconsistent, effectively empty configuration. The canary step correctly found the new configuration unsafe — but a second bug meant that failure did not stop the rollout, and every site withdrew its announcements of Google Cloud's IP blocks.

This reconstruction's propagate.js runs a set of canary checks and then pushes the config everywhere. It also never questions a config with no IP blocks in it.

Fix propagate so a failed canary aborts and an empty config is never pushed.

Bug report

BUG-GCE0411 · Priority: Critical · Reported by: SRE

propagate(config, sites, runChecks) returns { status, reason, pushed }:

  • config.blocks missing or empty -> { status: "aborted", reason: "empty-config", pushed: [] }, WITHOUT calling runChecks (the canary is not touched either)
  • otherwise runChecks(sites[0], config) returns [{ name, ok }, ...]; if ANY check is not ok -> { status: "aborted", reason: "canary", pushed: [] }
  • only when every check passes -> { status: "done", reason: null, pushed: <every site, in order> }

Observed: the canary reported a failing check, the rollout carried on, and all sites withdrew their routes.

Logs

[canary] site=us-central1 check=announcements FAILED check=bgp-session ok
[propagate] canary verdict: pass (last check ok)
[propagate] pushed config (0 blocks) to 14 sites

The code as shipped

src/net/propagate.js (editable)

// Pushes a new IP-block announcement config to every site: the canary
// first, then everyone else.
exports.propagate = function (config, sites, runChecks) {
  var canary = sites[0];
  var results = runChecks(canary, config);
  var failed = false;
  for (var i = 0; i < results.length; i++) {
    failed = !results[i].ok;
  }
  if (failed) return { status: "aborted", reason: "canary", pushed: [] };
  return { status: "done", reason: null, pushed: sites.slice() };
};

Read-only context: src/net/ROLLOUT.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.