The Filter That Held Back the Flood

Medium

· Backend

Incident

SEV-1

BUG-ROGERS0708

Priority: Critical · reported by IP core

Time to fix: 20 min

Failing checks on the shipped build

  • an empty filter list sends nothing

    an empty allow-list must fail closed: expected [] but got ["10.1.0.0/16","192.168.0.0/16"]

  • a batch over budget is refused

    an over-budget batch must not reach the core: expected [] but got ["10.0.0.0/16","10.1.0.0/16","10.2.0.0/16","10.3.0.0/16","10.4.0.0/16","10.5.0.0/16","10.6.0.0/16","10.7.0.0/16","10.8.0.0/16","10.9.0.0/16"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Rogers · 2022

Modelled on the Rogers outage of 8 July 2022: a maintenance step removed a routing filter from the core network. Without it the routers received far more routing information than they could process, the IP core collapsed, and a third of Canada's internet — including 911 service — was down for around 15 hours.

distribution.js distributes routes to the core and treats an empty filter list as "allow everything".

Fix distribute so an empty filter list fails closed, and so a batch that exceeds the core's processing budget is refused.

Code map

// Distributes routes into the core network.
exports.distribute = function (routes, filters, budget) {
  var sent = [];
  var blocked = [];
  for (var i = 0; i < routes.length; i++) {
    var allowed = filters.length === 0;
    for (var j = 0; j < filters.length; j++) {
      if (routes[i].indexOf(filters[j]) === 0) allowed = true;
    }
    if (allowed) sent.push(routes[i]);
    else blocked.push(routes[i]);
  }
  return { sent: sent, blocked: blocked };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Filter That Held Back the Flood — JavaScript Bug Hunt

Modelled on the Rogers outage of 8 July 2022: a maintenance step removed a routing filter from the core network.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Networking, Fail Closed
  • Modelled on: Rogers · 2022
  • Visible tests: matching routes are sent; an empty filter list sends nothing; a batch over budget is refused
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-ROGERS0708
  • Reported by: IP core
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • an empty filter list sends nothing — an empty allow-list must fail closed: expected [] but got ["10.1.0.0/16","192.168.0.0/16"]
  • a batch over budget is refused — an over-budget batch must not reach the core: expected [] but got ["10.0.0.0/16","10.1.0.0/16","10.2.0.0/16","10.3.0.0/16","10.4.0.0/16","10.5.0.0/16","10.6.0.0/16","10.7.0.0/16","10.8.0.0/16","10.9.0.0/16"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Rogers outage of 8 July 2022: a maintenance step removed a routing filter from the core network. Without it the routers received far more routing information than they could process, the IP core collapsed, and a third of Canada's internet — including 911 service — was down for around 15 hours.

distribution.js distributes routes to the core and treats an empty filter list as "allow everything".

Fix distribute so an empty filter list fails closed, and so a batch that exceeds the core's processing budget is refused.

Bug report

BUG-ROGERS0708 · Priority: Critical · Reported by: IP core

distribute(routes, filters, budget) must return { sent, blocked }:

  • an EMPTY filters array means nothing is explicitly permitted, so nothing is sent — fail closed, not open
  • otherwise send only routes whose prefix starts with one of the filters
  • if the number of routes that would be sent exceeds budget, send none and report them all as blocked

Observed: removing the last filter turns the guard into a pass-through and the entire global table is pushed at the core.

Logs

[core] filters=[] sent=1043221 routes
[core] control plane unresponsive across all regions

The code as shipped

src/net/distribution.js (editable)

// Distributes routes into the core network.
exports.distribute = function (routes, filters, budget) {
  var sent = [];
  var blocked = [];
  for (var i = 0; i < routes.length; i++) {
    var allowed = filters.length === 0;
    for (var j = 0; j < filters.length; j++) {
      if (routes[i].indexOf(filters[j]) === 0) allowed = true;
    }
    if (allowed) sent.push(routes[i]);
    else blocked.push(routes[i]);
  }
  return { sent: sent, blocked: blocked };
};

Read-only context: src/net/GUARD.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.