The Filter That Held Back the Flood — JavaScript Bug Hunt
Modelled on the Rogers outage of 8 July 2022: a maintenance step removed a routing filter from the core network.
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Networking, Fail Closed
- Modelled on: Rogers · 2022
- Visible tests: matching routes are sent; an empty filter list sends nothing; a batch over budget is refused
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-ROGERS0708
- Reported by: IP core
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- an empty filter list sends nothing — an empty allow-list must fail closed: expected [] but got ["10.1.0.0/16","192.168.0.0/16"]
- a batch over budget is refused — an over-budget batch must not reach the core: expected [] but got ["10.0.0.0/16","10.1.0.0/16","10.2.0.0/16","10.3.0.0/16","10.4.0.0/16","10.5.0.0/16","10.6.0.0/16","10.7.0.0/16","10.8.0.0/16","10.9.0.0/16"]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Rogers outage of 8 July 2022: a maintenance step removed a routing filter from the core network. Without it the routers received far more routing information than they could process, the IP core collapsed, and a third of Canada's internet — including 911 service — was down for around 15 hours.
distribution.js distributes routes to the core and treats an empty filter list as "allow everything".
Fix distribute so an empty filter list fails closed, and so a batch that exceeds the core's processing budget is refused.
Bug report
BUG-ROGERS0708 · Priority: Critical · Reported by: IP core
distribute(routes, filters, budget) must return { sent, blocked }:
- an EMPTY filters array means nothing is explicitly permitted, so nothing is sent — fail closed, not open
- otherwise send only routes whose prefix starts with one of the filters
- if the number of routes that would be sent exceeds budget, send none and report them all as blocked
Observed: removing the last filter turns the guard into a pass-through and the entire global table is pushed at the core.
Logs
[core] filters=[] sent=1043221 routes
[core] control plane unresponsive across all regionsThe code as shipped
src/net/distribution.js (editable)
// Distributes routes into the core network.
exports.distribute = function (routes, filters, budget) {
var sent = [];
var blocked = [];
for (var i = 0; i < routes.length; i++) {
var allowed = filters.length === 0;
for (var j = 0; j < filters.length; j++) {
if (routes[i].indexOf(filters[j]) === 0) allowed = true;
}
if (allowed) sent.push(routes[i]);
else blocked.push(routes[i]);
}
return { sent: sent, blocked: blocked };
};
Read-only context: src/net/GUARD.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.