Half the Servers Had the Old Config — JavaScript Bug Hunt
Modelled on the NYSE trading halt of 8 July 2015: a configuration update was loaded onto some gateway servers but not all of them.
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Configuration, Consistency
- Modelled on: NYSE · 2015
- Visible tests: matching versions open the market; mixed versions block the open
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-NYSE0708
- Reported by: market operations
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- mixed versions block the open — a partial rollout must not open the market: expected {"open":false,"reason":"config-mismatch"} but got {"open":true,"reason":"ok"}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the NYSE trading halt of 8 July 2015: a configuration update was loaded onto some gateway servers but not all of them. The mismatch produced inconsistent behaviour on the open, and trading was suspended for nearly four hours.
startup.js opens the market as soon as the servers are up, without checking that they agree.
Fix canOpenMarket so trading only starts when every server carries the same config version.
Bug report
BUG-NYSE0708 · Priority: Critical · Reported by: market operations
canOpenMarket(servers) must return { open, reason }:
- { open: false, reason: "offline" } if any server is not up
- { open: false, reason: "config-mismatch" } if the servers do not all report the same configVersion
- { open: true, reason: "ok" } otherwise
Observed: the market opens with mixed config versions and orders are handled inconsistently depending on which gateway receives them.
Logs
[gateway] open=true versions seen: {"2015.07.07":12,"2015.07.08":9}
[gateway] order routing inconsistent across gateways; haltingThe code as shipped
src/market/startup.js (editable)
// Decides whether the market may open.
exports.canOpenMarket = function (servers) {
for (var i = 0; i < servers.length; i++) {
if (!servers[i].up) return { open: false, reason: "offline" };
}
return { open: true, reason: "ok" };
};
Read-only context: src/market/PREFLIGHT.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.