The Term That Matched Every Route — JavaScript Bug Hunt

Briefing

Modelled on Cloudflare's backbone outage of 17 July 2020. While relieving congestion on part of its private backbone, an engineer edited the routing policy on a router in Atlanta. The edit left a policy term without the prefix-list condition that limited it to the site's own local routes, so the term applied to every route: the router advertised all of them with a high preference and pulled backbone traffic towards Atlanta instead of away from it, overloading it. Cloudflare's post-mortem describes a sharp drop in traffic across much of its network for roughly half an hour.

In this reconstruction the policy evaluator treats a term with no prefix condition as matching everything — exactly the behaviour that turned one edit into a network-wide event.

Fix applyPolicy so a term without a prefix condition fails closed instead of matching every route.

Bug report

BUG-BACKBONE-ATL · Priority: Critical · Reported by: network engineering

applyPolicy(routes, terms) — routes are { prefix }, terms are { name, prefixes, localPref }, evaluated in order:

• a term matches a route only when route.prefix is in term.prefixes
• a term whose prefixes is missing or empty matches NO route (fail closed); its name is reported in skipped (policy order, once each)
• the first matching term sets localPref; a route no term matches gets DEFAULT_LOCAL_PREF (100)
• returns { routes: [{ prefix, localPref }, ...] in input order, skipped }

Observed: after the "site-local" term lost its prefix list, every route on the router was exported with localPref 200.

Logs

[policy] term site-local: condition inactive
[bgp] exporting 1 of 1 terms -> 100% of routes localPref=200
[backbone] atl01 ingress 4x capacity, dropping

The code as shipped

src/backbone/policy.js (editable)

var DEFAULT_LOCAL_PREF = require("./defaults").DEFAULT_LOCAL_PREF;

function termMatches(term, route) {
  if (!term.prefixes || term.prefixes.length === 0) return true;
  return term.prefixes.indexOf(route.prefix) !== -1;
}

// Applies an ordered export policy to the routes a router advertises.
exports.applyPolicy = function (routes, terms) {
  var skipped = [];
  var out = routes.map(function (route) {
    for (var i = 0; i < terms.length; i++) {
      if (termMatches(terms[i], route)) {
        return { prefix: route.prefix, localPref: terms[i].localPref };
      }
    }
    return { prefix: route.prefix, localPref: DEFAULT_LOCAL_PREF };
  });
  return { routes: out, skipped: skipped };
};

Read-only context: src/backbone/defaults.js.