Someone Else's Bank Account

Medium

· Database

Incident

SEV-1

BUG-TSB18

Priority: Critical (data integrity) · reported by migration control

Time to fix: 20 min

Failing checks on the shipped build

  • accounts attach to the right customer

    join on customerId, not on position: expected [{"accountId":"acc-1","customerId":"cust-3","name":"Priya Nair"},{"accountId":"acc-2","customerId":"cust-9","name":"Dan Okafor"}] but got [{"accountId":"acc-1","customerId":"cust-3","name":"Dan Okafor"},{"accountId":"acc-2","customerId":"cust-9","name":…

  • an account with no customer is marked UNKNOWN

    expected [{"accountId":"acc-5","customerId":"cust-404","name":"UNKNOWN"}] but got [{"accountId":"acc-5","customerId":"cust-404","name":"A"}]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

TSB Bank · 2018

Modelled on the TSB Bank migration (April 2018): moving 1.3 billion customer records onto a new platform, records were re-keyed incorrectly and customers logging in were shown other people's accounts and balances. The outage ran for weeks and cost the bank around £330 million.

migrate.js re-keys accounts during the move and joins on the wrong field.

Fix migrateAccounts so every account stays with its own customer.

Code map

// Attaches each legacy account to its customer.
exports.migrateAccounts = function (legacyAccounts, customers) {
  var out = [];
  for (var i = 0; i < legacyAccounts.length; i++) {
    var acc = legacyAccounts[i];
    // The two extracts were exported together, so position i lines up.
    var customer = customers[i];
    out.push({ accountId: acc.accountId, customerId: acc.customerId, name: customer.name });
  }
  return out;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Someone Else's Bank Account — JavaScript Bug Hunt

Modelled on the TSB Bank migration (April 2018): moving 1.3 billion customer records onto a new platform, records were re-keyed incorrectly and customers…

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Migration, Identifiers
  • Modelled on: TSB Bank · 2018
  • Visible tests: accounts attach to the right customer; an account with no customer is marked UNKNOWN
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (data integrity))
  • Ticket: BUG-TSB18
  • Reported by: migration control
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • accounts attach to the right customer — join on customerId, not on position: expected [{"accountId":"acc-1","customerId":"cust-3","name":"Priya Nair"},{"accountId":"acc-2","customerId":"cust-9","name":"Dan Okafor"}] but got [{"accountId":"acc-1","customerId":"cust-3","name":"Dan Okafor"},{"accountId":"acc-2","customerId":"cust-9","name":…
  • an account with no customer is marked UNKNOWN — expected [{"accountId":"acc-5","customerId":"cust-404","name":"UNKNOWN"}] but got [{"accountId":"acc-5","customerId":"cust-404","name":"A"}]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the TSB Bank migration (April 2018): moving 1.3 billion customer records onto a new platform, records were re-keyed incorrectly and customers logging in were shown other people's accounts and balances. The outage ran for weeks and cost the bank around £330 million.

migrate.js re-keys accounts during the move and joins on the wrong field.

Fix migrateAccounts so every account stays with its own customer.

Bug report

BUG-TSB18 · Priority: Critical (data integrity) · Reported by: migration control

migrateAccounts(legacyAccounts, customers) must return, for each account, { accountId, customerId, name } where the name is the name of the customer whose id matches the account's customerId.

Observed: accounts are matched to customers by array position instead of by id. Because the two lists are not in the same order, customers see each other's accounts.

Logs

[migrate] account acc-2 (customerId cust-9) attached to name="Priya Nair" (cust-3)
[migrate] 21403 mismatched attachments in this batch

The code as shipped

src/migration/migrate.js (editable)

// Attaches each legacy account to its customer.
exports.migrateAccounts = function (legacyAccounts, customers) {
  var out = [];
  for (var i = 0; i < legacyAccounts.length; i++) {
    var acc = legacyAccounts[i];
    // The two extracts were exported together, so position i lines up.
    var customer = customers[i];
    out.push({ accountId: acc.accountId, customerId: acc.customerId, name: customer.name });
  }
  return out;
};

Read-only context: src/migration/EXTRACT.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.