The Drain Order That Matched Too Much — JavaScript Bug Hunt

Briefing

Modelled on Google's network outage of 2 June 2019: a configuration change intended for a small number of servers in a single region was applied to a larger set of servers across several neighbouring regions. The maintenance automation then descheduled the network control plane jobs on all of them, those regions lost a large share of their network capacity, and Google Cloud, YouTube and Gmail were degraded for hours.

This project is a reconstruction. drain.js picks the servers a maintenance order applies to and deschedules their jobs — but it selects on the job tag alone, so an order written for one cluster reaches every server that carries the tag, anywhere.

Fix selectTargets so an order only ever reaches the exact region and clusters it names.

Bug report

BUG-NETCTL-0602 · Priority: P0 · Reported by: SRE on-call

selectTargets(fleet, scope) — scope is { tag, region, clusters: [...] }. A server is a target only when ALL of these hold:

• its tags include scope.tag
• its region is exactly scope.region (no prefix or family matching: "us-east" does not mean "us-east1")
• its cluster is one of scope.clusters (an empty list selects nothing)

Targets are returned as server names in fleet order.

deschedule(fleet, scope) empties the jobs of every target and returns how many jobs it stopped. Servers that are not targets keep their jobs.

Observed: an order scoped to us-east1-a descheduled the network control jobs in us-east1-b, us-east4 and us-central1 as well.

Logs

[maint] order MO-4471 scope=us-east1/[us-east1-a] tag=netctl
[maint] descheduling netctl on 5 servers: ue1a-01 ue1a-02 ue1b-01 ue4a-01 uc1a-01
[netmon] us-east4: control plane jobs 0/1 running, capacity withdrawn

The code as shipped

src/ops/drain.js (editable)

// Maintenance orders: pick the servers an order applies to, then stop their jobs.

// scope: { tag, region, clusters: [...] }
exports.selectTargets = function (fleet, scope) {
  var out = [];
  for (var i = 0; i < fleet.length; i++) {
    var server = fleet[i];
    if (server.tags.indexOf(scope.tag) !== -1) out.push(server.name);
  }
  return out;
};

exports.deschedule = function (fleet, scope) {
  var targets = exports.selectTargets(fleet, scope);
  var stopped = 0;
  for (var i = 0; i < fleet.length; i++) {
    if (targets.indexOf(fleet[i].name) !== -1) {
      stopped += fleet[i].jobs.length;
      fleet[i].jobs = [];
    }
  }
  return stopped;
};

Read-only context: src/ops/fleet.js, src/ops/RUNBOOK.js.