The Term That Matched Every Route — JavaScript Bug Hunt
Modelled on Cloudflare's backbone outage of 17 July 2020. While relieving congestion on part of its private backbone, an engineer edited the routing policy…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Networking, Config
- Modelled on: Cloudflare · 2020 backbone
- Visible tests: a scoped term only touches its own prefixes; a term that lost its prefix condition matches nothing
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-BACKBONE-ATL
- Reported by: network engineering
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a term that lost its prefix condition matches nothing — expected {"routes":[{"prefix":"10.1.0.0/16","localPref":100},{"prefix":"10.2.0.0/16","localPref":100}],"skipped":["site-local"]} but got {"routes":[{"prefix":"10.1.0.0/16","localPref":200},{"prefix":"10.2.0.0/16","localPref":200}],"skipped":[]}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Cloudflare's backbone outage of 17 July 2020. While relieving congestion on part of its private backbone, an engineer edited the routing policy on a router in Atlanta. The edit left a policy term without the prefix-list condition that limited it to the site's own local routes, so the term applied to every route: the router advertised all of them with a high preference and pulled backbone traffic towards Atlanta instead of away from it, overloading it. Cloudflare's post-mortem describes a sharp drop in traffic across much of its network for roughly half an hour.
In this reconstruction the policy evaluator treats a term with no prefix condition as matching everything — exactly the behaviour that turned one edit into a network-wide event.
Fix applyPolicy so a term without a prefix condition fails closed instead of matching every route.
Bug report
BUG-BACKBONE-ATL · Priority: Critical · Reported by: network engineering
applyPolicy(routes, terms) — routes are { prefix }, terms are { name, prefixes, localPref }, evaluated in order:
- a term matches a route only when route.prefix is in term.prefixes
- a term whose
prefixesis missing or empty matches NO route (fail closed); its name is reported inskipped(policy order, once each) - the first matching term sets localPref; a route no term matches gets DEFAULT_LOCAL_PREF (100)
- returns { routes: [{ prefix, localPref }, ...] in input order, skipped }
Observed: after the "site-local" term lost its prefix list, every route on the router was exported with localPref 200.
Logs
[policy] term site-local: condition inactive
[bgp] exporting 1 of 1 terms -> 100% of routes localPref=200
[backbone] atl01 ingress 4x capacity, droppingThe code as shipped
src/backbone/policy.js (editable)
var DEFAULT_LOCAL_PREF = require("./defaults").DEFAULT_LOCAL_PREF;
function termMatches(term, route) {
if (!term.prefixes || term.prefixes.length === 0) return true;
return term.prefixes.indexOf(route.prefix) !== -1;
}
// Applies an ordered export policy to the routes a router advertises.
exports.applyPolicy = function (routes, terms) {
var skipped = [];
var out = routes.map(function (route) {
for (var i = 0; i < terms.length; i++) {
if (termMatches(terms[i], route)) {
return { prefix: route.prefix, localPref: terms[i].localPref };
}
}
return { prefix: route.prefix, localPref: DEFAULT_LOCAL_PREF };
});
return { routes: out, skipped: skipped };
};
Read-only context: src/backbone/defaults.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.