The Alarm Queue That Went Quiet

Hard

· Backend

Incident

SEV-1

BUG-EMS-0814

Priority: Critical (silent alarm loss) · reported by operations

Time to fix: 35 min

Failing checks on the shipped build

  • an alarm raised between check and sleep is not lost

    asleep with an alarm waiting: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Northeast blackout · 2003

Modelled on the Northeast blackout of August 14, 2003. The US–Canada task force found that the alarm function of the XA/21 energy management system at FirstEnergy's control room stalled without any indication — a race condition in GE's software was later identified as the cause — so operators did not see alarms as transmission lines tripped. The cascade that followed left about 50 million people without power.

This project reconstructs the class of bug, not GE's code: alarms.js is an alarm processor and its writers sharing one queue. The processor looks at the queue, and if it found it empty goes to sleep until a writer wakes it; writers only wake it on the empty → non-empty transition. Each function below is one atomic step, and the locked scheduler.js replays an interleaving of steps deterministically.

Fix the processor so no interleaving can leave it asleep with alarms waiting.

Code map

exports.create = function () {
  return { queue: [], shown: [], sleeping: false, sawEmpty: false };
};

// Processor: take one alarm off the queue, or note that it was empty.
exports.check = function (sys) {
  if (sys.sleeping) return "asleep";
  if (sys.queue.length > 0) {
    sys.shown.push(sys.queue.shift());
    sys.sawEmpty = false;
    return "shown";
  }
  sys.sawEmpty = true;
  return "empty";
};

// Processor: sleep until a writer wakes us, if the last check found nothing.
exports.sleep = function (sys) {
  if (sys.sawEmpty) sys.sleeping = true;
  return sys.sleeping ? "asleep" : "awake";
};

// Writer: queue an alarm. Only the empty -> non-empty edge needs a wake-up.
exports.raise = function (sys, alarm) {
  sys.queue.push(alarm);
  if (sys.queue.length === 1 && sys.sleeping) sys.sleeping = false;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Alarm Queue That Went Quiet — JavaScript Bug Hunt

Modelled on the Northeast blackout of August 14, 2003. The US–Canada task force found that the alarm function of the XA/21 energy management system at…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Concurrency, State
  • Modelled on: Northeast blackout · 2003
  • Visible tests: alarms raised between cycles are shown; an alarm raised between check and sleep is not lost
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (silent alarm loss))
  • Ticket: BUG-EMS-0814
  • Reported by: operations
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • an alarm raised between check and sleep is not lost — asleep with an alarm waiting: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Northeast blackout of August 14, 2003. The US–Canada task force found that the alarm function of the XA/21 energy management system at FirstEnergy's control room stalled without any indication — a race condition in GE's software was later identified as the cause — so operators did not see alarms as transmission lines tripped. The cascade that followed left about 50 million people without power.

This project reconstructs the class of bug, not GE's code: alarms.js is an alarm processor and its writers sharing one queue. The processor looks at the queue, and if it found it empty goes to sleep until a writer wakes it; writers only wake it on the empty → non-empty transition. Each function below is one atomic step, and the locked scheduler.js replays an interleaving of steps deterministically.

Fix the processor so no interleaving can leave it asleep with alarms waiting.

Bug report

BUG-EMS-0814 · Priority: Critical (silent alarm loss) · Reported by: operations

Steps (each atomic): check(sys) — processor takes one alarm off the queue (onto sys.shown) or notes it was empty; sleep(sys) — processor sleeps if its last look found nothing; raise(sys, alarm) — a writer queues an alarm. Required, for EVERY interleaving of these steps:

  • invariant: the processor is never asleep (sys.sleeping) while the queue holds an alarm
  • afterwards, scheduler.drain(sys) shows every raised alarm, in the order raised — including alarms raised later on
  • an alarm raised while the processor sleeps wakes it (this works today)

Observed: with the order check, raise, sleep the processor sleeps on a non-empty queue, and because later writers see a non-empty queue they never wake it. The display stays frozen.

Logs

14:14 [xa21] alarm list: last update 14:14:02
15:05 [xa21] line trip Harding-Chamberlin 345kV queued (depth 12)
15:32 [xa21] line trip Hanna-Juniper 345kV queued (depth 31) — processor state: asleep

The code as shipped

src/ems/alarms.js (editable)

exports.create = function () {
  return { queue: [], shown: [], sleeping: false, sawEmpty: false };
};

// Processor: take one alarm off the queue, or note that it was empty.
exports.check = function (sys) {
  if (sys.sleeping) return "asleep";
  if (sys.queue.length > 0) {
    sys.shown.push(sys.queue.shift());
    sys.sawEmpty = false;
    return "shown";
  }
  sys.sawEmpty = true;
  return "empty";
};

// Processor: sleep until a writer wakes us, if the last check found nothing.
exports.sleep = function (sys) {
  if (sys.sawEmpty) sys.sleeping = true;
  return sys.sleeping ? "asleep" : "awake";
};

// Writer: queue an alarm. Only the empty -> non-empty edge needs a wake-up.
exports.raise = function (sys, alarm) {
  sys.queue.push(alarm);
  if (sys.queue.length === 1 && sys.sleeping) sys.sleeping = false;
};

Read-only context: src/ems/scheduler.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.