The Backup That Never Took Over

Easy

· Backend

Incident

SEV-1

BUG-ARW-1001

Priority: Critical · reported by market operations

Time to fix: 10 min

Failing checks on the shipped build

  • a memory-device fault fails over

    the setting says auto: expected {"action":"failed-over","active":"node-b"} but got {"action":"halted","active":"node-a"}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Tokyo Stock Exchange · 2020

Modelled on the Tokyo Stock Exchange outage of 1 October 2020. A memory device in the exchange's trading system failed, and the automatic switch to the backup that should have followed did not happen — TSE and its vendor Fujitsu later attributed that to a setting. The exchange suspended trading in every listed stock for the whole day.

This reconstruction's fault monitor looks up a failover mode per fault kind in the operations team's settings, and falls back to halting when it finds none.

Fix handleFault so each fault kind reads the setting that was actually written for it.

Code map

var FAULT_SETTING = {
  DISK_FAULT: "disk",
  MEM_FAULT: "memory",
  NET_FAULT: "network",
  PWR_FAULT: "power"
};

exports.handleFault = function (fault, settings, cluster) {
  var mode = settings.failover[FAULT_SETTING[fault.code]] || "manual";
  if (mode === "auto" && cluster.standby) {
    cluster.active = cluster.standby;
    cluster.standby = null;
    return { action: "failed-over", active: cluster.active };
  }
  cluster.halted = true;
  return { action: "halted", active: cluster.active };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Backup That Never Took Over — JavaScript Bug Hunt

Modelled on the Tokyo Stock Exchange outage of 1 October 2020.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Config, Retries
  • Modelled on: Tokyo Stock Exchange · 2020
  • Visible tests: a disk fault fails over; a memory-device fault fails over
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-ARW-1001
  • Reported by: market operations
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • a memory-device fault fails over — the setting says auto: expected {"action":"failed-over","active":"node-b"} but got {"action":"halted","active":"node-a"}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Tokyo Stock Exchange outage of 1 October 2020. A memory device in the exchange's trading system failed, and the automatic switch to the backup that should have followed did not happen — TSE and its vendor Fujitsu later attributed that to a setting. The exchange suspended trading in every listed stock for the whole day.

This reconstruction's fault monitor looks up a failover mode per fault kind in the operations team's settings, and falls back to halting when it finds none.

Fix handleFault so each fault kind reads the setting that was actually written for it.

Bug report

BUG-ARW-1001 · Priority: Critical · Reported by: market operations

handleFault(fault, settings, cluster) looks up the failover mode for the fault's kind in settings.failover (keys as written in settings.js): DISK_FAULT -> "disk", MEM_FAULT -> "memory-device", NET_FAULT -> "network", PWR_FAULT -> "power"

  • mode "auto" and a standby exists: active = standby, standby = null, return { action: "failed-over", active: <new active> }
  • otherwise ("manual", no setting, unknown fault code, or no standby): cluster.halted = true, return { action: "halted", active: <unchanged> }

Observed: a memory-device fault halted the whole market although the settings say "memory-device": "auto".

Logs

[arrowhead] 07:04:12 fault code=MEM_FAULT node=node-a
[arrowhead] 07:04:12 failover mode=manual (no setting found) -> HALT
[arrowhead] 07:04:13 trading suspended: all issues

The code as shipped

src/arrowhead/monitor.js (editable)

var FAULT_SETTING = {
  DISK_FAULT: "disk",
  MEM_FAULT: "memory",
  NET_FAULT: "network",
  PWR_FAULT: "power"
};

exports.handleFault = function (fault, settings, cluster) {
  var mode = settings.failover[FAULT_SETTING[fault.code]] || "manual";
  if (mode === "auto" && cluster.standby) {
    cluster.active = cluster.standby;
    cluster.standby = null;
    return { action: "failed-over", active: cluster.active };
  }
  cluster.halted = true;
  return { action: "halted", active: cluster.active };
};

Read-only context: src/arrowhead/settings.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.