The Backup That Never Took Over — JavaScript Bug Hunt
Modelled on the Tokyo Stock Exchange outage of 1 October 2020.
- Language: JavaScript
- Layer: Backend
- Difficulty: Easy
- Concepts: Config, Retries
- Modelled on: Tokyo Stock Exchange · 2020
- Visible tests: a disk fault fails over; a memory-device fault fails over
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-ARW-1001
- Reported by: market operations
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- a memory-device fault fails over — the setting says auto: expected {"action":"failed-over","active":"node-b"} but got {"action":"halted","active":"node-a"}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Tokyo Stock Exchange outage of 1 October 2020. A memory device in the exchange's trading system failed, and the automatic switch to the backup that should have followed did not happen — TSE and its vendor Fujitsu later attributed that to a setting. The exchange suspended trading in every listed stock for the whole day.
This reconstruction's fault monitor looks up a failover mode per fault kind in the operations team's settings, and falls back to halting when it finds none.
Fix handleFault so each fault kind reads the setting that was actually written for it.
Bug report
BUG-ARW-1001 · Priority: Critical · Reported by: market operations
handleFault(fault, settings, cluster) looks up the failover mode for the fault's kind in settings.failover (keys as written in settings.js): DISK_FAULT -> "disk", MEM_FAULT -> "memory-device", NET_FAULT -> "network", PWR_FAULT -> "power"
- mode "auto" and a standby exists: active = standby, standby = null, return { action: "failed-over", active: <new active> }
- otherwise ("manual", no setting, unknown fault code, or no standby): cluster.halted = true, return { action: "halted", active: <unchanged> }
Observed: a memory-device fault halted the whole market although the settings say "memory-device": "auto".
Logs
[arrowhead] 07:04:12 fault code=MEM_FAULT node=node-a
[arrowhead] 07:04:12 failover mode=manual (no setting found) -> HALT
[arrowhead] 07:04:13 trading suspended: all issuesThe code as shipped
src/arrowhead/monitor.js (editable)
var FAULT_SETTING = {
DISK_FAULT: "disk",
MEM_FAULT: "memory",
NET_FAULT: "network",
PWR_FAULT: "power"
};
exports.handleFault = function (fault, settings, cluster) {
var mode = settings.failover[FAULT_SETTING[fault.code]] || "manual";
if (mode === "auto" && cluster.standby) {
cluster.active = cluster.standby;
cluster.standby = null;
return { action: "failed-over", active: cluster.active };
}
cluster.halted = true;
return { action: "halted", active: cluster.active };
};
Read-only context: src/arrowhead/settings.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.