Divided by the Sum, Not the Average

Easy

· Backend

Incident

SEV-1

BUG-VAR-CIO

Priority: Critical · reported by model review

Time to fix: 10 min

Failing checks on the shipped build

  • a relative change divides by the average of the two rates

    expected 0.4 but got 0.2

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

JPMorgan Chase · 2012

Modelled on JPMorgan Chase, 2012 (the "London Whale"): the bank's own task-force report into the Chief Investment Office's trading losses of more than $6 billion found that a new value-at-risk model was run through spreadsheets, and that one step — after subtracting an old rate from a new one — divided by their sum instead of their average. That muted the measured volatility by a factor of two and lowered the reported VaR.

risk.js is a reconstruction of that step: it turns a series of daily hazard-rate marks into relative changes, then into a volatility and a 95% VaR.

Fix relativeChange.

Code map

var stats = require("./stats");

// Relative change in a hazard rate between two daily marks.
function relativeChange(oldRate, newRate) {
  return (newRate - oldRate) / (oldRate + newRate);
}
exports.relativeChange = relativeChange;

exports.changes = function (rates) {
  var out = [];
  for (var i = 1; i < rates.length; i++) out.push(relativeChange(rates[i - 1], rates[i]));
  return out;
};

exports.volatility = function (rates) {
  return stats.round6(stats.stdev(exports.changes(rates)));
};

exports.valueAtRisk = function (position, rates) {
  return Math.round(stats.Z_95 * exports.volatility(rates) * position);
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Divided by the Sum, Not the Average — JavaScript Bug Hunt

Modelled on JPMorgan Chase, 2012 (the "London Whale"): the bank's own task-force report into the Chief Investment Office's trading losses of more than $6…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Money, Math
  • Modelled on: JPMorgan Chase · 2012
  • Visible tests: a flat series has no volatility; a relative change divides by the average of the two rates
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-VAR-CIO
  • Reported by: model review
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • a relative change divides by the average of the two rates — expected 0.4 but got 0.2

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on JPMorgan Chase, 2012 (the "London Whale"): the bank's own task-force report into the Chief Investment Office's trading losses of more than $6 billion found that a new value-at-risk model was run through spreadsheets, and that one step — after subtracting an old rate from a new one — divided by their sum instead of their average. That muted the measured volatility by a factor of two and lowered the reported VaR.

risk.js is a reconstruction of that step: it turns a series of daily hazard-rate marks into relative changes, then into a volatility and a 95% VaR.

Fix relativeChange.

Bug report

BUG-VAR-CIO · Priority: Critical · Reported by: model review

relativeChange(oldRate, newRate) = (newRate - oldRate) / average(oldRate, newRate) where average(a, b) = (a + b) / 2.

  • changes(rates): relativeChange for each consecutive pair
  • volatility(rates): stats.round6(population stdev of changes(rates))
  • valueAtRisk(position, rates): Math.round(stats.Z_95 volatility(rates) position)

Examples: relativeChange(0.02, 0.03) = 0.4; volatility([0.02, 0.03, 0.02, 0.03]) = 0.377124; valueAtRisk(1000000, that series) = 620369.

Observed: every volatility the model reports is exactly half the desk's own recalculation.

Logs

[var] book=synthetic-credit vol=0.188562 var95=310185
[model-review] independent recalculation: vol=0.377124 var95=620369

The code as shipped

src/risk/risk.js (editable)

var stats = require("./stats");

// Relative change in a hazard rate between two daily marks.
function relativeChange(oldRate, newRate) {
  return (newRate - oldRate) / (oldRate + newRate);
}
exports.relativeChange = relativeChange;

exports.changes = function (rates) {
  var out = [];
  for (var i = 1; i < rates.length; i++) out.push(relativeChange(rates[i - 1], rates[i]));
  return out;
};

exports.volatility = function (rates) {
  return stats.round6(stats.stdev(exports.changes(rates)));
};

exports.valueAtRisk = function (position, rates) {
  return Math.round(stats.Z_95 * exports.volatility(rates) * position);
};

Read-only context: src/risk/stats.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.