Half the Servers Had the Old Config

Medium

· Backend

Incident

SEV-1

BUG-NYSE0708

Priority: Critical · reported by market operations

Time to fix: 20 min

Failing checks on the shipped build

  • mixed versions block the open

    a partial rollout must not open the market: expected {"open":false,"reason":"config-mismatch"} but got {"open":true,"reason":"ok"}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

NYSE · 2015

Modelled on the NYSE trading halt of 8 July 2015: a configuration update was loaded onto some gateway servers but not all of them. The mismatch produced inconsistent behaviour on the open, and trading was suspended for nearly four hours.

startup.js opens the market as soon as the servers are up, without checking that they agree.

Fix canOpenMarket so trading only starts when every server carries the same config version.

Code map

// Decides whether the market may open.
exports.canOpenMarket = function (servers) {
  for (var i = 0; i < servers.length; i++) {
    if (!servers[i].up) return { open: false, reason: "offline" };
  }
  return { open: true, reason: "ok" };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Half the Servers Had the Old Config — JavaScript Bug Hunt

Modelled on the NYSE trading halt of 8 July 2015: a configuration update was loaded onto some gateway servers but not all of them.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Configuration, Consistency
  • Modelled on: NYSE · 2015
  • Visible tests: matching versions open the market; mixed versions block the open
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-NYSE0708
  • Reported by: market operations
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • mixed versions block the open — a partial rollout must not open the market: expected {"open":false,"reason":"config-mismatch"} but got {"open":true,"reason":"ok"}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the NYSE trading halt of 8 July 2015: a configuration update was loaded onto some gateway servers but not all of them. The mismatch produced inconsistent behaviour on the open, and trading was suspended for nearly four hours.

startup.js opens the market as soon as the servers are up, without checking that they agree.

Fix canOpenMarket so trading only starts when every server carries the same config version.

Bug report

BUG-NYSE0708 · Priority: Critical · Reported by: market operations

canOpenMarket(servers) must return { open, reason }:

  • { open: false, reason: "offline" } if any server is not up
  • { open: false, reason: "config-mismatch" } if the servers do not all report the same configVersion
  • { open: true, reason: "ok" } otherwise

Observed: the market opens with mixed config versions and orders are handled inconsistently depending on which gateway receives them.

Logs

[gateway] open=true versions seen: {"2015.07.07":12,"2015.07.08":9}
[gateway] order routing inconsistent across gateways; halting

The code as shipped

src/market/startup.js (editable)

// Decides whether the market may open.
exports.canOpenMarket = function (servers) {
  for (var i = 0; i < servers.length; i++) {
    if (!servers[i].up) return { open: false, reason: "offline" };
  }
  return { open: true, reason: "ok" };
};

Read-only context: src/market/PREFLIGHT.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.