The Opening Cross That Never Closed

Hard

· Backend

Incident

SEV-1

BUG-FBIPO

Priority: Critical · reported by market operations

Time to fix: 35 min

Failing checks on the shipped build

  • an empty book settles immediately

    Killed

  • a book with cancellations converges quickly

    Killed

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Nasdaq · Facebook IPO 2012

Modelled on the Nasdaq Facebook IPO (18 May 2012): the opening cross recalculated the opening price, but orders cancelled during the calculation were fed back in and forced another pass. The loop never converged, the IPO opened 30 minutes late, and confirmations were delayed for hours.

cross.js recomputes until nothing changed, but it re-queues the very events it generates.

Fix runCross so it converges: only events that arrived before the pass began may trigger another one, and the loop is bounded.

Code map

// Recomputes the opening price until the book settles.
exports.runCross = function (book) {
  var passes = 0;
  var price = book.reference;
  while (book.pending.length > 0) {
    passes++;
    if (passes > 100000) return { price: price, passes: passes };
    for (var i = 0; i < book.pending.length; i++) {
      price = price + book.pending[i].delta;
      // Every processed cancellation logs a re-quote back into the queue.
      book.pending.push({ delta: 0, requote: true });
    }
    book.pending = book.pending.slice(1);
  }
  return { price: price, passes: passes };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Opening Cross That Never Closed — JavaScript Bug Hunt

Modelled on the Nasdaq Facebook IPO (18 May 2012): the opening cross recalculated the opening price, but orders cancelled during the calculation were fed…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Concurrency, Convergence
  • Modelled on: Nasdaq · Facebook IPO 2012
  • Visible tests: an empty book settles immediately; a book with cancellations converges quickly
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-FBIPO
  • Reported by: market operations
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • an empty book settles immediately — Killed
  • a book with cancellations converges quickly — Killed

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Nasdaq Facebook IPO (18 May 2012): the opening cross recalculated the opening price, but orders cancelled during the calculation were fed back in and forced another pass. The loop never converged, the IPO opened 30 minutes late, and confirmations were delayed for hours.

cross.js recomputes until nothing changed, but it re-queues the very events it generates.

Fix runCross so it converges: only events that arrived before the pass began may trigger another one, and the loop is bounded.

Bug report

BUG-FBIPO · Priority: Critical · Reported by: market operations

runCross(book) must:

  • repeatedly recompute the price while pending cancellations remain
  • process each pass against a SNAPSHOT of the pending list taken at the start of that pass
  • return { price, passes } — passes must be small and bounded

Observed: each pass appends its own bookkeeping event back onto the pending list, so the loop runs until the 100000-iteration guard trips.

Logs

[cross] pass 99998 pending=1 price=42
[cross] convergence guard tripped; opening delayed

The code as shipped

src/market/cross.js (editable)

// Recomputes the opening price until the book settles.
exports.runCross = function (book) {
  var passes = 0;
  var price = book.reference;
  while (book.pending.length > 0) {
    passes++;
    if (passes > 100000) return { price: price, passes: passes };
    for (var i = 0; i < book.pending.length; i++) {
      price = price + book.pending[i].delta;
      // Every processed cancellation logs a re-quote back into the queue.
      book.pending.push({ delta: 0, requote: true });
    }
    book.pending = book.pending.slice(1);
  }
  return { price: price, passes: passes };
};

Read-only context: src/market/RULES.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.