Someone Else's Bank Account — JavaScript Bug Hunt
Modelled on the TSB Bank migration (April 2018): moving 1.3 billion customer records onto a new platform, records were re-keyed incorrectly and customers…
- Language: JavaScript
- Layer: Database
- Difficulty: Medium
- Concepts: Migration, Identifiers
- Modelled on: TSB Bank · 2018
- Visible tests: accounts attach to the right customer; an account with no customer is marked UNKNOWN
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (data integrity))
- Ticket: BUG-TSB18
- Reported by: migration control
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- accounts attach to the right customer — join on customerId, not on position: expected [{"accountId":"acc-1","customerId":"cust-3","name":"Priya Nair"},{"accountId":"acc-2","customerId":"cust-9","name":"Dan Okafor"}] but got [{"accountId":"acc-1","customerId":"cust-3","name":"Dan Okafor"},{"accountId":"acc-2","customerId":"cust-9","name":…
- an account with no customer is marked UNKNOWN — expected [{"accountId":"acc-5","customerId":"cust-404","name":"UNKNOWN"}] but got [{"accountId":"acc-5","customerId":"cust-404","name":"A"}]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the TSB Bank migration (April 2018): moving 1.3 billion customer records onto a new platform, records were re-keyed incorrectly and customers logging in were shown other people's accounts and balances. The outage ran for weeks and cost the bank around £330 million.
migrate.js re-keys accounts during the move and joins on the wrong field.
Fix migrateAccounts so every account stays with its own customer.
Bug report
BUG-TSB18 · Priority: Critical (data integrity) · Reported by: migration control
migrateAccounts(legacyAccounts, customers) must return, for each account, { accountId, customerId, name } where the name is the name of the customer whose id matches the account's customerId.
Observed: accounts are matched to customers by array position instead of by id. Because the two lists are not in the same order, customers see each other's accounts.
Logs
[migrate] account acc-2 (customerId cust-9) attached to name="Priya Nair" (cust-3)
[migrate] 21403 mismatched attachments in this batchThe code as shipped
src/migration/migrate.js (editable)
// Attaches each legacy account to its customer.
exports.migrateAccounts = function (legacyAccounts, customers) {
var out = [];
for (var i = 0; i < legacyAccounts.length; i++) {
var acc = legacyAccounts[i];
// The two extracts were exported together, so position i lines up.
var customer = customers[i];
out.push({ accountId: acc.accountId, customerId: acc.customerId, name: customer.name });
}
return out;
};
Read-only context: src/migration/EXTRACT.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.