The Term That Matched Every Route

Medium

· Backend

Incident

SEV-1

BUG-BACKBONE-ATL

Priority: Critical · reported by network engineering

Time to fix: 20 min

Failing checks on the shipped build

  • a term that lost its prefix condition matches nothing

    expected {"routes":[{"prefix":"10.1.0.0/16","localPref":100},{"prefix":"10.2.0.0/16","localPref":100}],"skipped":["site-local"]} but got {"routes":[{"prefix":"10.1.0.0/16","localPref":200},{"prefix":"10.2.0.0/16","localPref":200}],"skipped":[]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Cloudflare · 2020 backbone

Modelled on Cloudflare's backbone outage of 17 July 2020. While relieving congestion on part of its private backbone, an engineer edited the routing policy on a router in Atlanta. The edit left a policy term without the prefix-list condition that limited it to the site's own local routes, so the term applied to every route: the router advertised all of them with a high preference and pulled backbone traffic towards Atlanta instead of away from it, overloading it. Cloudflare's post-mortem describes a sharp drop in traffic across much of its network for roughly half an hour.

In this reconstruction the policy evaluator treats a term with no prefix condition as matching everything — exactly the behaviour that turned one edit into a network-wide event.

Fix applyPolicy so a term without a prefix condition fails closed instead of matching every route.

Code map

var DEFAULT_LOCAL_PREF = require("./defaults").DEFAULT_LOCAL_PREF;

function termMatches(term, route) {
  if (!term.prefixes || term.prefixes.length === 0) return true;
  return term.prefixes.indexOf(route.prefix) !== -1;
}

// Applies an ordered export policy to the routes a router advertises.
exports.applyPolicy = function (routes, terms) {
  var skipped = [];
  var out = routes.map(function (route) {
    for (var i = 0; i < terms.length; i++) {
      if (termMatches(terms[i], route)) {
        return { prefix: route.prefix, localPref: terms[i].localPref };
      }
    }
    return { prefix: route.prefix, localPref: DEFAULT_LOCAL_PREF };
  });
  return { routes: out, skipped: skipped };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Term That Matched Every Route — JavaScript Bug Hunt

Modelled on Cloudflare's backbone outage of 17 July 2020. While relieving congestion on part of its private backbone, an engineer edited the routing policy…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Networking, Config
  • Modelled on: Cloudflare · 2020 backbone
  • Visible tests: a scoped term only touches its own prefixes; a term that lost its prefix condition matches nothing
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-BACKBONE-ATL
  • Reported by: network engineering
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a term that lost its prefix condition matches nothing — expected {"routes":[{"prefix":"10.1.0.0/16","localPref":100},{"prefix":"10.2.0.0/16","localPref":100}],"skipped":["site-local"]} but got {"routes":[{"prefix":"10.1.0.0/16","localPref":200},{"prefix":"10.2.0.0/16","localPref":200}],"skipped":[]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Cloudflare's backbone outage of 17 July 2020. While relieving congestion on part of its private backbone, an engineer edited the routing policy on a router in Atlanta. The edit left a policy term without the prefix-list condition that limited it to the site's own local routes, so the term applied to every route: the router advertised all of them with a high preference and pulled backbone traffic towards Atlanta instead of away from it, overloading it. Cloudflare's post-mortem describes a sharp drop in traffic across much of its network for roughly half an hour.

In this reconstruction the policy evaluator treats a term with no prefix condition as matching everything — exactly the behaviour that turned one edit into a network-wide event.

Fix applyPolicy so a term without a prefix condition fails closed instead of matching every route.

Bug report

BUG-BACKBONE-ATL · Priority: Critical · Reported by: network engineering

applyPolicy(routes, terms) — routes are { prefix }, terms are { name, prefixes, localPref }, evaluated in order:

  • a term matches a route only when route.prefix is in term.prefixes
  • a term whose prefixes is missing or empty matches NO route (fail closed); its name is reported in skipped (policy order, once each)
  • the first matching term sets localPref; a route no term matches gets DEFAULT_LOCAL_PREF (100)
  • returns { routes: [{ prefix, localPref }, ...] in input order, skipped }

Observed: after the "site-local" term lost its prefix list, every route on the router was exported with localPref 200.

Logs

[policy] term site-local: condition inactive
[bgp] exporting 1 of 1 terms -> 100% of routes localPref=200
[backbone] atl01 ingress 4x capacity, dropping

The code as shipped

src/backbone/policy.js (editable)

var DEFAULT_LOCAL_PREF = require("./defaults").DEFAULT_LOCAL_PREF;

function termMatches(term, route) {
  if (!term.prefixes || term.prefixes.length === 0) return true;
  return term.prefixes.indexOf(route.prefix) !== -1;
}

// Applies an ordered export policy to the routes a router advertises.
exports.applyPolicy = function (routes, terms) {
  var skipped = [];
  var out = routes.map(function (route) {
    for (var i = 0; i < terms.length; i++) {
      if (termMatches(terms[i], route)) {
        return { prefix: route.prefix, localPref: terms[i].localPref };
      }
    }
    return { prefix: route.prefix, localPref: DEFAULT_LOCAL_PREF };
  });
  return { routes: out, skipped: skipped };
};

Read-only context: src/backbone/defaults.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.