The Backup That Never Took Over — JavaScript Bug Hunt

Briefing

Modelled on the Tokyo Stock Exchange outage of 1 October 2020. A memory device in the exchange's trading system failed, and the automatic switch to the backup that should have followed did not happen — TSE and its vendor Fujitsu later attributed that to a setting. The exchange suspended trading in every listed stock for the whole day.

This reconstruction's fault monitor looks up a failover mode per fault kind in the operations team's settings, and falls back to halting when it finds none.

Fix handleFault so each fault kind reads the setting that was actually written for it.

Bug report

BUG-ARW-1001 · Priority: Critical · Reported by: market operations

handleFault(fault, settings, cluster) looks up the failover mode for the fault's kind in settings.failover (keys as written in settings.js): DISK_FAULT -> "disk", MEM_FAULT -> "memory-device", NET_FAULT -> "network", PWR_FAULT -> "power"

• mode "auto" and a standby exists: active = standby, standby = null, return { action: "failed-over", active: <new active> }
• otherwise ("manual", no setting, unknown fault code, or no standby): cluster.halted = true, return { action: "halted", active: <unchanged> }

Observed: a memory-device fault halted the whole market although the settings say "memory-device": "auto".

Logs

[arrowhead] 07:04:12 fault code=MEM_FAULT node=node-a
[arrowhead] 07:04:12 failover mode=manual (no setting found) -> HALT
[arrowhead] 07:04:13 trading suspended: all issues

The code as shipped

src/arrowhead/monitor.js (editable)

var FAULT_SETTING = {
  DISK_FAULT: "disk",
  MEM_FAULT: "memory",
  NET_FAULT: "network",
  PWR_FAULT: "power"
};

exports.handleFault = function (fault, settings, cluster) {
  var mode = settings.failover[FAULT_SETTING[fault.code]] || "manual";
  if (mode === "auto" && cluster.standby) {
    cluster.active = cluster.standby;
    cluster.standby = null;
    return { action: "failed-over", active: cluster.active };
  }
  cluster.halted = true;
  return { action: "halted", active: cluster.active };
};

Read-only context: src/arrowhead/settings.js.