The Reject Term Moved to the Top — JavaScript Bug Hunt
Modelled on Cloudflare's outage of 21 June 2022. A change to the network configuration in 19 of its data centres — locations that carry a large share of its…
- Language: JavaScript
- Layer: Backend
- Difficulty: Hard
- Concepts: Networking, Config
- Modelled on: Cloudflare · 2022
- Visible tests: withdrawn reports what a policy would not advertise; normalising a site policy keeps the site advertised
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-MCP-ORDER
- Reported by: network engineering
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- normalising a site policy keeps the site advertised — expected "accept" but got "reject"
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Cloudflare's outage of 21 June 2022. A change to the network configuration in 19 of its data centres — locations that carry a large share of its traffic — reordered the terms of the BGP policies that decide which IP prefixes are advertised. After the reorder, a term that rejects prefixes came before the terms that accept the sites' own prefixes, so a critical subset of prefixes was withdrawn and those data centres became unreachable until the change was reverted.
This reconstruction's normalise rewrites a policy into a house layout before deployment by grouping all reject terms first — including the catch-all reject that must always be last — and deploy pushes the result with no check on what it stops advertising.
Fix the ordering, and make deploy refuse a policy that would withdraw a critical prefix.
Bug report
BUG-MCP-ORDER · Priority: Critical · Reported by: network engineering
Policies are ordered term lists evaluated by evaluate() (locked): the first matching term decides; unmatched prefixes are rejected.
normalise(terms) -> a NEW array (the input is never modified):
- every term keeps its position relative to the others, except catch-all terms (matchAll: true), which move to the end, keeping their own relative order
withdrawn(terms, critical) -> the prefixes of critical, in input order, that the policy would not accept.
deploy(terms, critical):
- normalises the policy
- if withdrawn(normalised, critical) is non-empty, throws an Error and deploys nothing
- otherwise returns the normalised policy
Observed: after normalise the catch-all reject was the first term, every site prefix was withdrawn, and deploy pushed it anyway.
Logs
[policy] normalise: 3 reject terms moved ahead of 4 accept terms
[bgp] withdrawing 214 prefixes from 19 sites
[deploy] policy pushed to 19 sites (no pre-deploy checks configured)The code as shipped
src/bgp/policy.js (editable)
var evaluate = require("./evaluate").evaluate;
// Rewrites a policy into the house layout before it is pushed to the data
// centres. Returns a new array; the input is left alone.
exports.normalise = function (terms) {
var rejects = terms.filter(function (t) { return t.action === "reject"; });
var others = terms.filter(function (t) { return t.action !== "reject"; });
return rejects.concat(others);
};
// The prefixes from critical that this policy would stop advertising.
exports.withdrawn = function (terms, critical) {
return critical.filter(function (prefix) { return evaluate(terms, prefix) !== "accept"; });
};
// Normalises and returns the policy that will be deployed.
exports.deploy = function (terms, critical) {
return exports.normalise(terms);
};
Read-only context: src/bgp/evaluate.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.