The Canary That Failed and Was Overruled — JavaScript Bug Hunt

Briefing

Modelled on the Google Compute Engine outage of 11 April 2016, when GCE instances lost external connectivity in every region for about 18 minutes. Google's post-mortem: engineers removed an unused IP block from the network configuration, and a timing quirk in the management software produced an inconsistent, effectively empty configuration. The canary step correctly found the new configuration unsafe — but a second bug meant that failure did not stop the rollout, and every site withdrew its announcements of Google Cloud's IP blocks.

This reconstruction's propagate.js runs a set of canary checks and then pushes the config everywhere. It also never questions a config with no IP blocks in it.

Fix propagate so a failed canary aborts and an empty config is never pushed.

Bug report

BUG-GCE0411 · Priority: Critical · Reported by: SRE

propagate(config, sites, runChecks) returns { status, reason, pushed }:

• config.blocks missing or empty -> { status: "aborted", reason: "empty-config", pushed: [] }, WITHOUT calling runChecks (the canary is not touched either)
• otherwise runChecks(sites[0], config) returns [{ name, ok }, ...]; if ANY check is not ok -> { status: "aborted", reason: "canary", pushed: [] }
• only when every check passes -> { status: "done", reason: null, pushed: <every site, in order> }

Observed: the canary reported a failing check, the rollout carried on, and all sites withdrew their routes.

Logs

[canary] site=us-central1 check=announcements FAILED check=bgp-session ok
[propagate] canary verdict: pass (last check ok)
[propagate] pushed config (0 blocks) to 14 sites

The code as shipped

src/net/propagate.js (editable)

// Pushes a new IP-block announcement config to every site: the canary
// first, then everyone else.
exports.propagate = function (config, sites, runChecks) {
  var canary = sites[0];
  var results = runChecks(canary, config);
  var failed = false;
  for (var i = 0; i < results.length; i++) {
    failed = !results[i].ok;
  }
  if (failed) return { status: "aborted", reason: "canary", pushed: [] };
  return { status: "done", reason: null, pushed: sites.slice() };
};

Read-only context: src/net/ROLLOUT.js.