The Flag That Fired Old Code

Hard

· Backend

Incident

SEV-1

BUG-KCG-2012

Priority: Existential · reported by risk desk

Time to fix: 35 min

Failing checks on the shipped build

  • the dead flag changes nothing

    one order in, one order out: expected 1 but got 50000

  • quantity is clamped to the risk limit

    MAX_QTY is 100: expected 100 but got 5000

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Knight Capital · 2012

Inspired by Knight Capital's 2012 collapse: a decommissioned feature flag was repurposed, and servers still carrying the OLD code path blasted the market with orders. $460M gone in 45 minutes.

Here, the legacy module is locked (you can't delete it — that's the point). The router must treat the dead flag as dead and clamp order quantity, no matter what.

Code map

var legacy = require("./legacy");

var MAX_QTY = 100;

// Routes one inbound order. flags is an array of active flag names.
exports.route = function (order, flags) {
  if (flags.indexOf("power_peg") !== -1) {
    return legacy.blast(order);
  }
  return [{ symbol: order.symbol, qty: order.qty }];
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Flag That Fired Old Code — JavaScript Bug Hunt

Inspired by Knight Capital's 2012 collapse: a decommissioned feature flag was repurposed, and servers still carrying the OLD code path blasted the market…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Feature Flags, Trading
  • Modelled on: Knight Capital · 2012
  • Visible tests: the dead flag changes nothing; quantity is clamped to the risk limit; normal orders pass through untouched
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Existential)
  • Ticket: BUG-KCG-2012
  • Reported by: risk desk
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the dead flag changes nothing — one order in, one order out: expected 1 but got 50000
  • quantity is clamped to the risk limit — MAX_QTY is 100: expected 100 but got 5000

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by Knight Capital's 2012 collapse: a decommissioned feature flag was repurposed, and servers still carrying the OLD code path blasted the market with orders. $460M gone in 45 minutes.

Here, the legacy module is locked (you can't delete it — that's the point). The router must treat the dead flag as dead and clamp order quantity, no matter what.

Bug report

BUG-KCG-2012 · Priority: Existential · Reported by: risk desk

Spec for route(order, flags):

  • The "power_peg" flag is DEAD. Its presence must change nothing.
  • Quantity is always clamped to MAX_QTY (100) — never more, never negative.
  • Exactly ONE order comes out for one order in.

Observed: with the flag set, the legacy blaster emits 1000x child orders.

Logs

[router] flags=["power_peg"] in qty=50 -> out orders=50000 (!!)

The code as shipped

src/trade/router.js (editable)

var legacy = require("./legacy");

var MAX_QTY = 100;

// Routes one inbound order. flags is an array of active flag names.
exports.route = function (order, flags) {
  if (flags.indexOf("power_peg") !== -1) {
    return legacy.blast(order);
  }
  return [{ symbol: order.symbol, qty: order.qty }];
};

Read-only context: src/trade/legacy.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.