The Reject Term Moved to the Top — JavaScript Bug Hunt

Briefing

Modelled on Cloudflare's outage of 21 June 2022. A change to the network configuration in 19 of its data centres — locations that carry a large share of its traffic — reordered the terms of the BGP policies that decide which IP prefixes are advertised. After the reorder, a term that rejects prefixes came before the terms that accept the sites' own prefixes, so a critical subset of prefixes was withdrawn and those data centres became unreachable until the change was reverted.

This reconstruction's normalise rewrites a policy into a house layout before deployment by grouping all reject terms first — including the catch-all reject that must always be last — and deploy pushes the result with no check on what it stops advertising.

Fix the ordering, and make deploy refuse a policy that would withdraw a critical prefix.

Bug report

BUG-MCP-ORDER · Priority: Critical · Reported by: network engineering

Policies are ordered term lists evaluated by evaluate() (locked): the first matching term decides; unmatched prefixes are rejected.

normalise(terms) -> a NEW array (the input is never modified):

• every term keeps its position relative to the others, except catch-all terms (matchAll: true), which move to the end, keeping their own relative order

withdrawn(terms, critical) -> the prefixes of critical, in input order, that the policy would not accept.

deploy(terms, critical):

• normalises the policy
• if withdrawn(normalised, critical) is non-empty, throws an Error and deploys nothing
• otherwise returns the normalised policy

Observed: after normalise the catch-all reject was the first term, every site prefix was withdrawn, and deploy pushed it anyway.

Logs

[policy] normalise: 3 reject terms moved ahead of 4 accept terms
[bgp] withdrawing 214 prefixes from 19 sites
[deploy] policy pushed to 19 sites (no pre-deploy checks configured)

The code as shipped

src/bgp/policy.js (editable)

var evaluate = require("./evaluate").evaluate;

// Rewrites a policy into the house layout before it is pushed to the data
// centres. Returns a new array; the input is left alone.
exports.normalise = function (terms) {
  var rejects = terms.filter(function (t) { return t.action === "reject"; });
  var others = terms.filter(function (t) { return t.action !== "reject"; });
  return rejects.concat(others);
};

// The prefixes from critical that this policy would stop advertising.
exports.withdrawn = function (terms, critical) {
  return critical.filter(function (prefix) { return evaluate(terms, prefix) !== "accept"; });
};

// Normalises and returns the policy that will be deployed.
exports.deploy = function (terms, critical) {
  return exports.normalise(terms);
};

Read-only context: src/bgp/evaluate.js.