Interest Sent as Orders

Medium

· Backend

Incident

SEV-1

BUG-GS-0820

Priority: Critical (erroneous trades) · reported by exchange surveillance

Time to fix: 20 min

Failing checks on the shipped build

  • an indication of interest is never routed

    IOIs are not orders: expected [] but got [i1@100]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Goldman Sachs · 2013

Modelled on Goldman Sachs' options misfire of August 20, 2013. Goldman said an internal system error caused indications of interest — messages that are not meant to trade — to be sent to US options exchanges as real orders, many at erroneous prices, some at $1. A large number of the resulting trades were later cancelled or adjusted by the exchanges.

This project is a reconstruction: OptionsRouter takes a batch of outgoing messages and decides what goes to the exchange. It treats every message as a live order, and a message without a price is filled in with the default from the locked RouterConfig.

Fix route so indications of interest never reach the exchange and no order is sent without an explicit, positive price.

Code map

class OptionsRouter {
    static RouteResult route(List<Message> batch) {
        RouteResult result = new RouteResult();
        for (Message m : batch) {
            int price = m.priceCents != null ? m.priceCents : RouterConfig.DEFAULT_PRICE_CENTS;
            result.routed.add(m.id + "@" + price);
        }
        return result;
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Interest Sent as Orders — Java Bug Hunt

Modelled on Goldman Sachs' options misfire of August 20, 2013.

  • Language: Java
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Money, Validation, Config
  • Modelled on: Goldman Sachs · 2013
  • Visible tests: a priced order is routed; an indication of interest is never routed
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (erroneous trades))
  • Ticket: BUG-GS-0820
  • Reported by: exchange surveillance
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • an indication of interest is never routed — IOIs are not orders: expected [] but got [i1@100]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Goldman Sachs' options misfire of August 20, 2013. Goldman said an internal system error caused indications of interest — messages that are not meant to trade — to be sent to US options exchanges as real orders, many at erroneous prices, some at $1. A large number of the resulting trades were later cancelled or adjusted by the exchanges.

This project is a reconstruction: OptionsRouter takes a batch of outgoing messages and decides what goes to the exchange. It treats every message as a live order, and a message without a price is filled in with the default from the locked RouterConfig.

Fix route so indications of interest never reach the exchange and no order is sent without an explicit, positive price.

Bug report

BUG-GS-0820 · Priority: Critical (erroneous trades) · Reported by: exchange surveillance

route(batch) returns a RouteResult with three lists, each in batch order:

  • a message of type "IOI" is NEVER routed: its id goes in held
  • a message of type "ORDER" with an explicit price > 0 cents is routed as "<id>@<priceCents>"
  • an "ORDER" with a missing (null) or non-positive price, and a message of any other type, is rejected: its id goes in rejected

RouterConfig.DEFAULT_PRICE_CENTS must never be used to price a live order.

Observed: indications of interest were routed as orders, and those without a price went out at 100 cents.

Logs

09:30:02 [router] routed IOI-88213 SPY 150820C00160000 x500 @100
09:30:02 [router] routed IOI-88214 AAPL 150820P00450000 x300 @100
09:30:03 [surveillance] member GS: burst of $1 option prints across exchanges

The code as shipped

src/router/OptionsRouter.java (editable)

class OptionsRouter {
    static RouteResult route(List<Message> batch) {
        RouteResult result = new RouteResult();
        for (Message m : batch) {
            int price = m.priceCents != null ? m.priceCents : RouterConfig.DEFAULT_PRICE_CENTS;
            result.routed.add(m.id + "@" + price);
        }
        return result;
    }
}

Read-only context: src/router/Message.java, src/router/RouterConfig.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.