The Cleanup That Deleted the Customers — JavaScript Bug Hunt
Modelled on the Atlassian Jira outage (April 2022): a script meant to remove a deprecated app was handed site IDs where it expected app IDs, and it ran with…
- Language: JavaScript
- Layer: Database
- Difficulty: Medium
- Concepts: Data Loss, Identifiers
- Modelled on: Atlassian Jira · 2022
- Visible tests: apps are deleted; sites are never deleted
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (data loss))
- Ticket: BUG-JIRA22
- Reported by: incident command
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- sites are never deleted — a site id must be refused: expected {"deleted":["app-3"],"skipped":["site-19"]} but got {"deleted":["site-19","app-3"],"skipped":[]}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Atlassian Jira outage (April 2022): a script meant to remove a deprecated app was handed site IDs where it expected app IDs, and it ran with a "permanent delete" flag. About 400 customer sites were wiped, and restoring them took two weeks.
cleanup.js deletes records by id without checking that the ids it was given are the right kind.
Fix removeApps so it only ever deletes app records, and reports ids it refuses.
Bug report
BUG-JIRA22 · Priority: Critical (data loss) · Reported by: incident command
removeApps(store, ids) must:
- delete only entries whose kind is "app"
- leave every other entry untouched
- return { deleted: [...], skipped: [...] } listing the ids in each group, in the order they were supplied
Observed: passing site ids deletes the sites. The function never looks at the record's kind.
Logs
[cleanup] deleted id=site-19 kind=site
[cleanup] 400 customer sites removed in 9 minutesThe code as shipped
src/ops/cleanup.js (editable)
// Removes the given ids from the store.
exports.removeApps = function (store, ids) {
var deleted = [];
var skipped = [];
for (var i = 0; i < ids.length; i++) {
var id = ids[i];
if (store[id]) {
delete store[id];
deleted.push(id);
} else {
skipped.push(id);
}
}
return { deleted: deleted, skipped: skipped };
};
Read-only context: src/ops/SCHEMA.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.