The Flag That Fired Old Code — JavaScript Bug Hunt
Inspired by Knight Capital's 2012 collapse: a decommissioned feature flag was repurposed, and servers still carrying the OLD code path blasted the market…
- Language: JavaScript
- Layer: Backend
- Difficulty: Hard
- Concepts: Feature Flags, Trading
- Modelled on: Knight Capital · 2012
- Visible tests: the dead flag changes nothing; quantity is clamped to the risk limit; normal orders pass through untouched
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Existential)
- Ticket: BUG-KCG-2012
- Reported by: risk desk
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- the dead flag changes nothing — one order in, one order out: expected 1 but got 50000
- quantity is clamped to the risk limit — MAX_QTY is 100: expected 100 but got 5000
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by Knight Capital's 2012 collapse: a decommissioned feature flag was repurposed, and servers still carrying the OLD code path blasted the market with orders. $460M gone in 45 minutes.
Here, the legacy module is locked (you can't delete it — that's the point). The router must treat the dead flag as dead and clamp order quantity, no matter what.
Bug report
BUG-KCG-2012 · Priority: Existential · Reported by: risk desk
Spec for route(order, flags):
- The "power_peg" flag is DEAD. Its presence must change nothing.
- Quantity is always clamped to MAX_QTY (100) — never more, never negative.
- Exactly ONE order comes out for one order in.
Observed: with the flag set, the legacy blaster emits 1000x child orders.
Logs
[router] flags=["power_peg"] in qty=50 -> out orders=50000 (!!)The code as shipped
src/trade/router.js (editable)
var legacy = require("./legacy");
var MAX_QTY = 100;
// Routes one inbound order. flags is an array of active flag names.
exports.route = function (order, flags) {
if (flags.indexOf("power_peg") !== -1) {
return legacy.blast(order);
}
return [{ symbol: order.symbol, qty: order.qty }];
};
Read-only context: src/trade/legacy.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.