More Routes Than the Router Could Hold — JavaScript Bug Hunt
Modelled on the Optus outage of 8 November 2023: a routing information change propagated from an upstream peer exceeded a preset limit on the routers, which…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Resource Limits, Atomicity
- Modelled on: Optus · 2023
- Visible tests: a batch that fits is installed; an over-capacity batch installs nothing
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-OPTUS1108
- Reported by: core network
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- an over-capacity batch installs nothing — a partial update is worse than none: expected 0 but got 2
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Optus outage of 8 November 2023: a routing information change propagated from an upstream peer exceeded a preset limit on the routers, which then disconnected themselves from the network. Around 10 million customers lost service for most of a day.
routing.js installs routes one at a time and stops when it runs out of room, leaving the table half-updated.
Fix installRoutes so an update that would exceed the limit is rejected atomically, leaving the existing table untouched.
Bug report
BUG-OPTUS1108 · Priority: Critical · Reported by: core network
installRoutes(table, routes, limit) must return { installed, table }:
- if table.length + routes.length would exceed limit, install nothing and return { installed: 0, table: <unchanged> }
- otherwise append them all and return the new count
Observed: routes are appended until the limit is hit, so the table ends up containing part of an update. The router then withdraws from the network.
Logs
[rib] limit 1000 reached after 137 of 900 new prefixes
[rib] inconsistent table; process restartingThe code as shipped
src/net/routing.js (editable)
// Installs a batch of routes into the table.
exports.installRoutes = function (table, routes, limit) {
var installed = 0;
for (var i = 0; i < routes.length; i++) {
if (table.length >= limit) break;
table.push(routes[i]);
installed++;
}
return { installed: installed, table: table };
};
Read-only context: src/net/RIB.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.