The Drain Order That Matched Too Much — JavaScript Bug Hunt
Modelled on Google's network outage of 2 June 2019: a configuration change intended for a small number of servers in a single region was applied to a larger…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Config, Networking
- Modelled on: Google · 2019
- Visible tests: an order for a whole region's clusters selects them; an order for one cluster stays in that cluster
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: P0)
- Ticket: BUG-NETCTL-0602
- Reported by: SRE on-call
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- an order for one cluster stays in that cluster — other clusters and regions must not be touched: expected ["ue1a-01","ue1a-02"] but got ["ue1a-01","ue1a-02","ue1b-01","ue4a-01","uc1a-01"]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Google's network outage of 2 June 2019: a configuration change intended for a small number of servers in a single region was applied to a larger set of servers across several neighbouring regions. The maintenance automation then descheduled the network control plane jobs on all of them, those regions lost a large share of their network capacity, and Google Cloud, YouTube and Gmail were degraded for hours.
This project is a reconstruction. drain.js picks the servers a maintenance order applies to and deschedules their jobs — but it selects on the job tag alone, so an order written for one cluster reaches every server that carries the tag, anywhere.
Fix selectTargets so an order only ever reaches the exact region and clusters it names.
Bug report
BUG-NETCTL-0602 · Priority: P0 · Reported by: SRE on-call
selectTargets(fleet, scope) — scope is { tag, region, clusters: [...] }. A server is a target only when ALL of these hold:
- its tags include scope.tag
- its region is exactly scope.region (no prefix or family matching: "us-east" does not mean "us-east1")
- its cluster is one of scope.clusters (an empty list selects nothing)
Targets are returned as server names in fleet order.
deschedule(fleet, scope) empties the jobs of every target and returns how many jobs it stopped. Servers that are not targets keep their jobs.
Observed: an order scoped to us-east1-a descheduled the network control jobs in us-east1-b, us-east4 and us-central1 as well.
Logs
[maint] order MO-4471 scope=us-east1/[us-east1-a] tag=netctl
[maint] descheduling netctl on 5 servers: ue1a-01 ue1a-02 ue1b-01 ue4a-01 uc1a-01
[netmon] us-east4: control plane jobs 0/1 running, capacity withdrawnThe code as shipped
src/ops/drain.js (editable)
// Maintenance orders: pick the servers an order applies to, then stop their jobs.
// scope: { tag, region, clusters: [...] }
exports.selectTargets = function (fleet, scope) {
var out = [];
for (var i = 0; i < fleet.length; i++) {
var server = fleet[i];
if (server.tags.indexOf(scope.tag) !== -1) out.push(server.name);
}
return out;
};
exports.deschedule = function (fleet, scope) {
var targets = exports.selectTargets(fleet, scope);
var stopped = 0;
for (var i = 0; i < fleet.length; i++) {
if (targets.indexOf(fleet[i].name) !== -1) {
stopped += fleet[i].jobs.length;
fleet[i].jobs = [];
}
}
return stopped;
};
Read-only context: src/ops/RUNBOOK.js, src/ops/fleet.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.