More Routes Than the Router Could Hold

Medium

· Backend

Incident

SEV-1

BUG-OPTUS1108

Priority: Critical · reported by core network

Time to fix: 20 min

Failing checks on the shipped build

  • an over-capacity batch installs nothing

    a partial update is worse than none: expected 0 but got 2

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Optus · 2023

Modelled on the Optus outage of 8 November 2023: a routing information change propagated from an upstream peer exceeded a preset limit on the routers, which then disconnected themselves from the network. Around 10 million customers lost service for most of a day.

routing.js installs routes one at a time and stops when it runs out of room, leaving the table half-updated.

Fix installRoutes so an update that would exceed the limit is rejected atomically, leaving the existing table untouched.

Code map

// Installs a batch of routes into the table.
exports.installRoutes = function (table, routes, limit) {
  var installed = 0;
  for (var i = 0; i < routes.length; i++) {
    if (table.length >= limit) break;
    table.push(routes[i]);
    installed++;
  }
  return { installed: installed, table: table };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

More Routes Than the Router Could Hold — JavaScript Bug Hunt

Modelled on the Optus outage of 8 November 2023: a routing information change propagated from an upstream peer exceeded a preset limit on the routers, which…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Resource Limits, Atomicity
  • Modelled on: Optus · 2023
  • Visible tests: a batch that fits is installed; an over-capacity batch installs nothing
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-OPTUS1108
  • Reported by: core network
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • an over-capacity batch installs nothing — a partial update is worse than none: expected 0 but got 2

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Optus outage of 8 November 2023: a routing information change propagated from an upstream peer exceeded a preset limit on the routers, which then disconnected themselves from the network. Around 10 million customers lost service for most of a day.

routing.js installs routes one at a time and stops when it runs out of room, leaving the table half-updated.

Fix installRoutes so an update that would exceed the limit is rejected atomically, leaving the existing table untouched.

Bug report

BUG-OPTUS1108 · Priority: Critical · Reported by: core network

installRoutes(table, routes, limit) must return { installed, table }:

  • if table.length + routes.length would exceed limit, install nothing and return { installed: 0, table: <unchanged> }
  • otherwise append them all and return the new count

Observed: routes are appended until the limit is hit, so the table ends up containing part of an update. The router then withdraws from the network.

Logs

[rib] limit 1000 reached after 137 of 900 new prefixes
[rib] inconsistent table; process restarting

The code as shipped

src/net/routing.js (editable)

// Installs a batch of routes into the table.
exports.installRoutes = function (table, routes, limit) {
  var installed = 0;
  for (var i = 0; i < routes.length; i++) {
    if (table.length >= limit) break;
    table.push(routes[i]);
    installed++;
  }
  return { installed: installed, table: table };
};

Read-only context: src/net/RIB.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.