The Backup That Deleted the Primary — JavaScript Bug Hunt
Modelled on the FAA NOTAM system outage of 11 January 2023. The FAA's preliminary review found that contract personnel unintentionally deleted files while…
- Language: JavaScript
- Layer: Database
- Difficulty: Medium
- Concepts: Replication, State
- Modelled on: FAA NOTAM · 2023
- Visible tests: a changed file is copied to the backup; a file only the primary has is copied, not deleted
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-NOTAM-SYNC
- Reported by: NOTAM operations
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a file only the primary has is copied, not deleted — the primary is never modified: expected ["new.dat=n","notam-index.dat=i1"] but got ["notam-index.dat=i1"]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the FAA NOTAM system outage of 11 January 2023. The FAA's preliminary review found that contract personnel unintentionally deleted files while working to correct synchronisation between the live primary database and a backup database. The NOTAM system, which publishes notices to pilots, failed, and the FAA halted departures nationwide on the morning of 11 January.
This reconstruction's synchronise is meant to make the backup match the primary. Instead it mirrors differences in the wrong direction — deleting from the primary whatever the backup lacks — and nothing stops it removing files the service cannot run without.
Fix synchronise so the primary is the source of truth and required files are never deleted.
Bug report
BUG-NOTAM-SYNC · Priority: Critical · Reported by: NOTAM operations
synchronise(primary, backup) — each database is an object mapping file name to contents; the call updates backup in place:
- the primary is the source of truth and is NEVER modified
- every primary file is copied to the backup (added, or overwritten when the contents differ)
- a backup file the primary does not have is deleted from the backup, unless its name is in REQUIRED_FILES (src/notam/required.js) — those are never deleted from either database
- returns { deleted: [...] }, the names removed from the backup, sorted
Observed: after a sync run, files that existed only on the primary were gone from the primary.
Logs
[sync] primary=412 files backup=398 files
[sync] removed 14 files from primary
[notam] startup failed: notam-index.dat missingThe code as shipped
src/notam/sync.js (editable)
var REQUIRED = require("./required").REQUIRED_FILES;
// Brings the two databases back into step after they drifted apart.
// Each database is a map of file name -> contents.
exports.synchronise = function (primary, backup) {
var deleted = [];
Object.keys(primary).forEach(function (name) {
if (!Object.prototype.hasOwnProperty.call(backup, name)) {
delete primary[name];
deleted.push(name);
}
});
Object.keys(primary).forEach(function (name) {
backup[name] = primary[name];
});
deleted.sort();
return { deleted: deleted };
};
Read-only context: src/notam/required.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.