Half the Servers Had the Old Config — JavaScript Bug Hunt

Briefing

Modelled on the NYSE trading halt of 8 July 2015: a configuration update was loaded onto some gateway servers but not all of them. The mismatch produced inconsistent behaviour on the open, and trading was suspended for nearly four hours.

startup.js opens the market as soon as the servers are up, without checking that they agree.

Fix canOpenMarket so trading only starts when every server carries the same config version.

Bug report

BUG-NYSE0708 · Priority: Critical · Reported by: market operations

canOpenMarket(servers) must return { open, reason }:

• { open: false, reason: "offline" } if any server is not up
• { open: false, reason: "config-mismatch" } if the servers do not all report the same configVersion
• { open: true, reason: "ok" } otherwise

Observed: the market opens with mixed config versions and orders are handled inconsistently depending on which gateway receives them.

Logs

[gateway] open=true versions seen: {"2015.07.07":12,"2015.07.08":9}
[gateway] order routing inconsistent across gateways; halting

The code as shipped

src/market/startup.js (editable)

// Decides whether the market may open.
exports.canOpenMarket = function (servers) {
  for (var i = 0; i < servers.length; i++) {
    if (!servers[i].up) return { open: false, reason: "offline" };
  }
  return { open: true, reason: "ok" };
};

Read-only context: src/market/PREFLIGHT.js.