The /24 That Swallowed YouTube

Hard

· Backend

Incident

SEV-1

BUG-BGP-0802

Priority: Critical · reported by network engineering

Time to fix: 35 min

Failing checks on the shipped build

  • the more-specific hijack is invalid and ignored

    wrong origin, too specific: expected "invalid" but got "valid"

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Pakistan Telecom · 2008

Modelled on Pakistan Telecom and YouTube, 24 February 2008. Asked to block YouTube inside Pakistan, Pakistan Telecom (AS17557) announced 208.65.153.0/24 — a more specific slice of YouTube's 208.65.152.0/22. The announcement leaked to its upstream provider and on across the internet, and because routers prefer the longest matching prefix, much of the world's YouTube traffic went to Pakistan instead for roughly two hours. Route origin validation (RPKI ROAs, which name the authorised origin AS and a maximum prefix length) is the defence that later became standard.

This reconstruction's route table checks announcements against ROAs, but only asks whether a ROA covers the prefix.

Fix announce to perform full origin validation.

Code map

var ip = require("./ip");

function validate(net, originAs, roas) {
  for (var i = 0; i < roas.length; i++) {
    if (ip.covers(ip.parse(roas[i].prefix), net)) return "valid";
  }
  return "not-found";
}

exports.create = function (roas) {
  var routes = [];
  return {
    announce: function (prefix, originAs) {
      var net = ip.parse(prefix);
      var state = validate(net, originAs, roas);
      if (state === "invalid") return state;
      routes.push({ net: net, origin: originAs });
      return state;
    },
    lookup: function (address) {
      var n = ip.toInt(address);
      var best = null;
      for (var i = 0; i < routes.length; i++) {
        var r = routes[i];
        if (ip.covers(r.net, { base: n, len: 32 }) && (!best || r.net.len > best.net.len)) best = r;
      }
      return best ? best.origin : null;
    }
  };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The /24 That Swallowed YouTube — JavaScript Bug Hunt

Modelled on Pakistan Telecom and YouTube, 24 February 2008. Asked to block YouTube inside Pakistan, Pakistan Telecom (AS17557) announced 208.65.153.0/24 — a…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Networking, Security, Validation
  • Modelled on: Pakistan Telecom · 2008
  • Visible tests: the legitimate /22 is valid and routes; the more-specific hijack is invalid and ignored
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-BGP-0802
  • Reported by: network engineering
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the more-specific hijack is invalid and ignored — wrong origin, too specific: expected "invalid" but got "valid"

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Pakistan Telecom and YouTube, 24 February 2008. Asked to block YouTube inside Pakistan, Pakistan Telecom (AS17557) announced 208.65.153.0/24 — a more specific slice of YouTube's 208.65.152.0/22. The announcement leaked to its upstream provider and on across the internet, and because routers prefer the longest matching prefix, much of the world's YouTube traffic went to Pakistan instead for roughly two hours. Route origin validation (RPKI ROAs, which name the authorised origin AS and a maximum prefix length) is the defence that later became standard.

This reconstruction's route table checks announcements against ROAs, but only asks whether a ROA covers the prefix.

Fix announce to perform full origin validation.

Bug report

BUG-BGP-0802 · Priority: Critical · Reported by: network engineering

rib.create(roas).announce(prefix, originAs) returns one of:

  • "valid": some ROA covers the prefix, its asn === originAs, AND the prefix length <= that ROA's maxLength
  • "invalid": at least one ROA covers the prefix but none makes it valid
  • "not-found": no ROA covers the prefix

Invalid announcements are NOT installed; valid and not-found ones are. lookup(address) returns the origin AS of the installed route with the longest prefix containing the address (first installed wins a tie), or null.

Observed: AS17557's 208.65.153.0/24 was reported "valid" because YouTube's /22 ROA covers it, and took over the traffic by longest-prefix match.

Logs

[rib] announce 208.65.153.0/24 origin=AS17557 rov=valid
[rib] best path 208.65.153.238 -> AS17557 (/24 beats /22)

The code as shipped

src/bgp/rib.js (editable)

var ip = require("./ip");

function validate(net, originAs, roas) {
  for (var i = 0; i < roas.length; i++) {
    if (ip.covers(ip.parse(roas[i].prefix), net)) return "valid";
  }
  return "not-found";
}

exports.create = function (roas) {
  var routes = [];
  return {
    announce: function (prefix, originAs) {
      var net = ip.parse(prefix);
      var state = validate(net, originAs, roas);
      if (state === "invalid") return state;
      routes.push({ net: net, origin: originAs });
      return state;
    },
    lookup: function (address) {
      var n = ip.toInt(address);
      var best = null;
      for (var i = 0; i < routes.length; i++) {
        var r = routes[i];
        if (ip.covers(r.net, { base: n, len: 32 }) && (!best || r.net.len > best.net.len)) best = r;
      }
      return best ? best.origin : null;
    }
  };
};

Read-only context: src/bgp/ROA.js, src/bgp/ip.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.