The Route Leak Nobody Filtered

Medium

· Backend

Incident

SEV-1

BUG-AS396531

Priority: Critical · reported by network engineering

Time to fix: 20 min

Failing checks on the shipped build

  • an over-specific prefix is rejected

    a /24 beyond the ceiling must be filtered: expected false but got true

  • an unknown origin is rejected

    expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Verizon / Cloudflare · 2019

Modelled on the Verizon/Cloudflare BGP leak (24 June 2019): a small ISP's route optimiser re-announced huge swathes of the internet as its own, and Verizon accepted the announcements without any prefix or AS-path filter. Traffic for Cloudflare, Amazon and Linode was black-holed for around two hours.

bgp.js decides whether to accept an announced route and applies no policy at all.

Fix acceptRoute so announcements outside the peer's policy are rejected.

Code map

// Decides whether to install an announced route.
exports.acceptRoute = function (route, policy) {
  return true;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Route Leak Nobody Filtered — JavaScript Bug Hunt

Modelled on the Verizon/Cloudflare BGP leak (24 June 2019): a small ISP's route optimiser re-announced huge swathes of the internet as its own, and Verizon…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Networking, Validation
  • Modelled on: Verizon / Cloudflare · 2019
  • Visible tests: a policy-compliant route is accepted; an over-specific prefix is rejected; an unknown origin is rejected
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-AS396531
  • Reported by: network engineering
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • an over-specific prefix is rejected — a /24 beyond the ceiling must be filtered: expected false but got true
  • an unknown origin is rejected — expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Verizon/Cloudflare BGP leak (24 June 2019): a small ISP's route optimiser re-announced huge swathes of the internet as its own, and Verizon accepted the announcements without any prefix or AS-path filter. Traffic for Cloudflare, Amazon and Linode was black-holed for around two hours.

bgp.js decides whether to accept an announced route and applies no policy at all.

Fix acceptRoute so announcements outside the peer's policy are rejected.

Bug report

BUG-AS396531 · Priority: Critical · Reported by: network engineering

acceptRoute(route, policy) must accept a route only when ALL hold:

  • route.prefixLen is at most policy.maxPrefixLen (more specific is a red flag)
  • route.asPath.length is at most policy.maxAsPath
  • route.asPath[0] is in policy.allowedOrigins

Observed: every announcement from the peer is accepted, including a /24 with a 20-hop path originated by an AS we have never heard of.

Logs

[bgp] accepted prefix=104.16.0.0/24 asPath=20 hops origin=AS33154
[bgp] 20% of global traffic now pointed at a 20Gbps peer

The code as shipped

src/net/bgp.js (editable)

// Decides whether to install an announced route.
exports.acceptRoute = function (route, policy) {
  return true;
};

Read-only context: src/net/POLICY.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.