More Routes Than the Router Could Hold — JavaScript Bug Hunt

Briefing

Modelled on the Optus outage of 8 November 2023: a routing information change propagated from an upstream peer exceeded a preset limit on the routers, which then disconnected themselves from the network. Around 10 million customers lost service for most of a day.

routing.js installs routes one at a time and stops when it runs out of room, leaving the table half-updated.

Fix installRoutes so an update that would exceed the limit is rejected atomically, leaving the existing table untouched.

Bug report

BUG-OPTUS1108 · Priority: Critical · Reported by: core network

installRoutes(table, routes, limit) must return { installed, table }:

• if table.length + routes.length would exceed limit, install nothing and return { installed: 0, table: <unchanged> }
• otherwise append them all and return the new count

Observed: routes are appended until the limit is hit, so the table ends up containing part of an update. The router then withdraws from the network.

Logs

[rib] limit 1000 reached after 137 of 900 new prefixes
[rib] inconsistent table; process restarting

The code as shipped

src/net/routing.js (editable)

// Installs a batch of routes into the table.
exports.installRoutes = function (table, routes, limit) {
  var installed = 0;
  for (var i = 0; i < routes.length; i++) {
    if (table.length >= limit) break;
    table.push(routes[i]);
    installed++;
  }
  return { installed: installed, table: table };
};

Read-only context: src/net/RIB.js.