The Backup That Deleted the Primary — JavaScript Bug Hunt

Briefing

Modelled on the FAA NOTAM system outage of 11 January 2023. The FAA's preliminary review found that contract personnel unintentionally deleted files while working to correct synchronisation between the live primary database and a backup database. The NOTAM system, which publishes notices to pilots, failed, and the FAA halted departures nationwide on the morning of 11 January.

This reconstruction's synchronise is meant to make the backup match the primary. Instead it mirrors differences in the wrong direction — deleting from the primary whatever the backup lacks — and nothing stops it removing files the service cannot run without.

Fix synchronise so the primary is the source of truth and required files are never deleted.

Bug report

BUG-NOTAM-SYNC · Priority: Critical · Reported by: NOTAM operations

synchronise(primary, backup) — each database is an object mapping file name to contents; the call updates backup in place:

• the primary is the source of truth and is NEVER modified
• every primary file is copied to the backup (added, or overwritten when the contents differ)
• a backup file the primary does not have is deleted from the backup, unless its name is in REQUIRED_FILES (src/notam/required.js) — those are never deleted from either database
• returns { deleted: [...] }, the names removed from the backup, sorted

Observed: after a sync run, files that existed only on the primary were gone from the primary.

Logs

[sync] primary=412 files backup=398 files
[sync] removed 14 files from primary
[notam] startup failed: notam-index.dat missing

The code as shipped

src/notam/sync.js (editable)

var REQUIRED = require("./required").REQUIRED_FILES;

// Brings the two databases back into step after they drifted apart.
// Each database is a map of file name -> contents.
exports.synchronise = function (primary, backup) {
  var deleted = [];
  Object.keys(primary).forEach(function (name) {
    if (!Object.prototype.hasOwnProperty.call(backup, name)) {
      delete primary[name];
      deleted.push(name);
    }
  });
  Object.keys(primary).forEach(function (name) {
    backup[name] = primary[name];
  });
  deleted.sort();
  return { deleted: deleted };
};

Read-only context: src/notam/required.js.