The Extra Second That Pegged Every CPU — JavaScript Bug Hunt
Modelled on the leap second of 30 June 2012: inserting a 61st second left kernel timers with a deadline that had already passed.
- Language: JavaScript
- Layer: Backend
- Difficulty: Hard
- Concepts: Time, Livelock
- Modelled on: Linux leap second · 2012
- Visible tests: a due timer fires once and re-arms; a clock step backwards does not livelock
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-LEAPSEC
- Reported by: kernel team
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- a clock step backwards does not livelock — a stale deadline must fire once, not spin, saw 199: expected a truthy value but got false
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the leap second of 30 June 2012: inserting a 61st second left kernel timers with a deadline that had already passed. The timer loop re-armed a task that was instantly due again, so it spun without progressing. Reddit, Mozilla, LinkedIn and Qantas all went down at the same instant.
scheduler.js re-arms timers by adding an interval to the deadline, which never catches up once the clock jumps backwards.
Fix runDueTimers so it always makes progress, even when the clock steps.
Bug report
BUG-LEAPSEC · Priority: Critical · Reported by: kernel team
runDueTimers(now, timers) must:
- fire every timer whose deadline is at or before
now - re-arm each fired timer strictly AFTER
now - return { fired, timers } and terminate in bounded time
Observed: when the clock steps backwards, a re-armed timer is immediately due again and the loop spins until the iteration guard trips.
Logs
[timer] iterations=1000000 fired=1000000 pending=1
[timer] all cores at 100%; ntp step -1sThe code as shipped
src/kernel/scheduler.js (editable)
// Fires every timer that is due, re-arming it for its next interval.
exports.runDueTimers = function (now, timers) {
var fired = 0;
var guard = 0;
var due = true;
while (due) {
due = false;
for (var i = 0; i < timers.length; i++) {
if (timers[i].deadline <= now) {
fired++;
// Re-arm relative to the old deadline.
timers[i].deadline = timers[i].deadline + timers[i].interval;
due = true;
}
}
guard++;
if (guard > 1000000) break;
}
return { fired: fired, timers: timers };
};
Read-only context: src/kernel/TIMERS.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.