The Alarm Queue That Went Quiet — JavaScript Bug Hunt

Briefing

Modelled on the Northeast blackout of August 14, 2003. The US–Canada task force found that the alarm function of the XA/21 energy management system at FirstEnergy's control room stalled without any indication — a race condition in GE's software was later identified as the cause — so operators did not see alarms as transmission lines tripped. The cascade that followed left about 50 million people without power.

This project reconstructs the class of bug, not GE's code: alarms.js is an alarm processor and its writers sharing one queue. The processor looks at the queue, and if it found it empty goes to sleep until a writer wakes it; writers only wake it on the empty → non-empty transition. Each function below is one atomic step, and the locked scheduler.js replays an interleaving of steps deterministically.

Fix the processor so no interleaving can leave it asleep with alarms waiting.

Bug report

BUG-EMS-0814 · Priority: Critical (silent alarm loss) · Reported by: operations

Steps (each atomic): check(sys) — processor takes one alarm off the queue (onto sys.shown) or notes it was empty; sleep(sys) — processor sleeps if its last look found nothing; raise(sys, alarm) — a writer queues an alarm. Required, for EVERY interleaving of these steps:

• invariant: the processor is never asleep (sys.sleeping) while the queue holds an alarm
• afterwards, scheduler.drain(sys) shows every raised alarm, in the order raised — including alarms raised later on
• an alarm raised while the processor sleeps wakes it (this works today)

Observed: with the order check, raise, sleep the processor sleeps on a non-empty queue, and because later writers see a non-empty queue they never wake it. The display stays frozen.

Logs

14:14 [xa21] alarm list: last update 14:14:02
15:05 [xa21] line trip Harding-Chamberlin 345kV queued (depth 12)
15:32 [xa21] line trip Hanna-Juniper 345kV queued (depth 31) — processor state: asleep

The code as shipped

src/ems/alarms.js (editable)

exports.create = function () {
  return { queue: [], shown: [], sleeping: false, sawEmpty: false };
};

// Processor: take one alarm off the queue, or note that it was empty.
exports.check = function (sys) {
  if (sys.sleeping) return "asleep";
  if (sys.queue.length > 0) {
    sys.shown.push(sys.queue.shift());
    sys.sawEmpty = false;
    return "shown";
  }
  sys.sawEmpty = true;
  return "empty";
};

// Processor: sleep until a writer wakes us, if the last check found nothing.
exports.sleep = function (sys) {
  if (sys.sawEmpty) sys.sleeping = true;
  return sys.sleeping ? "asleep" : "awake";
};

// Writer: queue an alarm. Only the empty -> non-empty edge needs a wake-up.
exports.raise = function (sys, alarm) {
  sys.queue.push(alarm);
  if (sys.queue.length === 1 && sys.sleeping) sys.sleeping = false;
};

Read-only context: src/ems/scheduler.js.