The Drain Order That Matched Too Much

Medium

· Backend

Incident

SEV-1

BUG-NETCTL-0602

Priority: P0 · reported by SRE on-call

Time to fix: 20 min

Failing checks on the shipped build

  • an order for one cluster stays in that cluster

    other clusters and regions must not be touched: expected ["ue1a-01","ue1a-02"] but got ["ue1a-01","ue1a-02","ue1b-01","ue4a-01","uc1a-01"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Google · 2019

Modelled on Google's network outage of 2 June 2019: a configuration change intended for a small number of servers in a single region was applied to a larger set of servers across several neighbouring regions. The maintenance automation then descheduled the network control plane jobs on all of them, those regions lost a large share of their network capacity, and Google Cloud, YouTube and Gmail were degraded for hours.

This project is a reconstruction. drain.js picks the servers a maintenance order applies to and deschedules their jobs — but it selects on the job tag alone, so an order written for one cluster reaches every server that carries the tag, anywhere.

Fix selectTargets so an order only ever reaches the exact region and clusters it names.

Code map

// Maintenance orders: pick the servers an order applies to, then stop their jobs.

// scope: { tag, region, clusters: [...] }
exports.selectTargets = function (fleet, scope) {
  var out = [];
  for (var i = 0; i < fleet.length; i++) {
    var server = fleet[i];
    if (server.tags.indexOf(scope.tag) !== -1) out.push(server.name);
  }
  return out;
};

exports.deschedule = function (fleet, scope) {
  var targets = exports.selectTargets(fleet, scope);
  var stopped = 0;
  for (var i = 0; i < fleet.length; i++) {
    if (targets.indexOf(fleet[i].name) !== -1) {
      stopped += fleet[i].jobs.length;
      fleet[i].jobs = [];
    }
  }
  return stopped;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Drain Order That Matched Too Much — JavaScript Bug Hunt

Modelled on Google's network outage of 2 June 2019: a configuration change intended for a small number of servers in a single region was applied to a larger…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Config, Networking
  • Modelled on: Google · 2019
  • Visible tests: an order for a whole region's clusters selects them; an order for one cluster stays in that cluster
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: P0)
  • Ticket: BUG-NETCTL-0602
  • Reported by: SRE on-call
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • an order for one cluster stays in that cluster — other clusters and regions must not be touched: expected ["ue1a-01","ue1a-02"] but got ["ue1a-01","ue1a-02","ue1b-01","ue4a-01","uc1a-01"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Google's network outage of 2 June 2019: a configuration change intended for a small number of servers in a single region was applied to a larger set of servers across several neighbouring regions. The maintenance automation then descheduled the network control plane jobs on all of them, those regions lost a large share of their network capacity, and Google Cloud, YouTube and Gmail were degraded for hours.

This project is a reconstruction. drain.js picks the servers a maintenance order applies to and deschedules their jobs — but it selects on the job tag alone, so an order written for one cluster reaches every server that carries the tag, anywhere.

Fix selectTargets so an order only ever reaches the exact region and clusters it names.

Bug report

BUG-NETCTL-0602 · Priority: P0 · Reported by: SRE on-call

selectTargets(fleet, scope) — scope is { tag, region, clusters: [...] }. A server is a target only when ALL of these hold:

  • its tags include scope.tag
  • its region is exactly scope.region (no prefix or family matching: "us-east" does not mean "us-east1")
  • its cluster is one of scope.clusters (an empty list selects nothing)

Targets are returned as server names in fleet order.

deschedule(fleet, scope) empties the jobs of every target and returns how many jobs it stopped. Servers that are not targets keep their jobs.

Observed: an order scoped to us-east1-a descheduled the network control jobs in us-east1-b, us-east4 and us-central1 as well.

Logs

[maint] order MO-4471 scope=us-east1/[us-east1-a] tag=netctl
[maint] descheduling netctl on 5 servers: ue1a-01 ue1a-02 ue1b-01 ue4a-01 uc1a-01
[netmon] us-east4: control plane jobs 0/1 running, capacity withdrawn

The code as shipped

src/ops/drain.js (editable)

// Maintenance orders: pick the servers an order applies to, then stop their jobs.

// scope: { tag, region, clusters: [...] }
exports.selectTargets = function (fleet, scope) {
  var out = [];
  for (var i = 0; i < fleet.length; i++) {
    var server = fleet[i];
    if (server.tags.indexOf(scope.tag) !== -1) out.push(server.name);
  }
  return out;
};

exports.deschedule = function (fleet, scope) {
  var targets = exports.selectTargets(fleet, scope);
  var stopped = 0;
  for (var i = 0; i < fleet.length; i++) {
    if (targets.indexOf(fleet[i].name) !== -1) {
      stopped += fleet[i].jobs.length;
      fleet[i].jobs = [];
    }
  }
  return stopped;
};

Read-only context: src/ops/RUNBOOK.js, src/ops/fleet.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.