The Extra Second That Pegged Every CPU

Hard

· Backend

Incident

SEV-1

BUG-LEAPSEC

Priority: Critical · reported by kernel team

Time to fix: 35 min

Failing checks on the shipped build

  • a clock step backwards does not livelock

    a stale deadline must fire once, not spin, saw 199: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Linux leap second · 2012

Modelled on the leap second of 30 June 2012: inserting a 61st second left kernel timers with a deadline that had already passed. The timer loop re-armed a task that was instantly due again, so it spun without progressing. Reddit, Mozilla, LinkedIn and Qantas all went down at the same instant.

scheduler.js re-arms timers by adding an interval to the deadline, which never catches up once the clock jumps backwards.

Fix runDueTimers so it always makes progress, even when the clock steps.

Code map

// Fires every timer that is due, re-arming it for its next interval.
exports.runDueTimers = function (now, timers) {
  var fired = 0;
  var guard = 0;
  var due = true;
  while (due) {
    due = false;
    for (var i = 0; i < timers.length; i++) {
      if (timers[i].deadline <= now) {
        fired++;
        // Re-arm relative to the old deadline.
        timers[i].deadline = timers[i].deadline + timers[i].interval;
        due = true;
      }
    }
    guard++;
    if (guard > 1000000) break;
  }
  return { fired: fired, timers: timers };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Extra Second That Pegged Every CPU — JavaScript Bug Hunt

Modelled on the leap second of 30 June 2012: inserting a 61st second left kernel timers with a deadline that had already passed.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Time, Livelock
  • Modelled on: Linux leap second · 2012
  • Visible tests: a due timer fires once and re-arms; a clock step backwards does not livelock
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-LEAPSEC
  • Reported by: kernel team
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a clock step backwards does not livelock — a stale deadline must fire once, not spin, saw 199: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the leap second of 30 June 2012: inserting a 61st second left kernel timers with a deadline that had already passed. The timer loop re-armed a task that was instantly due again, so it spun without progressing. Reddit, Mozilla, LinkedIn and Qantas all went down at the same instant.

scheduler.js re-arms timers by adding an interval to the deadline, which never catches up once the clock jumps backwards.

Fix runDueTimers so it always makes progress, even when the clock steps.

Bug report

BUG-LEAPSEC · Priority: Critical · Reported by: kernel team

runDueTimers(now, timers) must:

  • fire every timer whose deadline is at or before now
  • re-arm each fired timer strictly AFTER now
  • return { fired, timers } and terminate in bounded time

Observed: when the clock steps backwards, a re-armed timer is immediately due again and the loop spins until the iteration guard trips.

Logs

[timer] iterations=1000000 fired=1000000 pending=1
[timer] all cores at 100%; ntp step -1s

The code as shipped

src/kernel/scheduler.js (editable)

// Fires every timer that is due, re-arming it for its next interval.
exports.runDueTimers = function (now, timers) {
  var fired = 0;
  var guard = 0;
  var due = true;
  while (due) {
    due = false;
    for (var i = 0; i < timers.length; i++) {
      if (timers[i].deadline <= now) {
        fired++;
        // Re-arm relative to the old deadline.
        timers[i].deadline = timers[i].deadline + timers[i].interval;
        due = true;
      }
    }
    guard++;
    if (guard > 1000000) break;
  }
  return { fired: fired, timers: timers };
};

Read-only context: src/kernel/TIMERS.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.