The Backup That Deleted the Primary

Medium

· Database

Incident

SEV-1

BUG-NOTAM-SYNC

Priority: Critical · reported by NOTAM operations

Time to fix: 20 min

Failing checks on the shipped build

  • a file only the primary has is copied, not deleted

    the primary is never modified: expected ["new.dat=n","notam-index.dat=i1"] but got ["notam-index.dat=i1"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

FAA NOTAM · 2023

Modelled on the FAA NOTAM system outage of 11 January 2023. The FAA's preliminary review found that contract personnel unintentionally deleted files while working to correct synchronisation between the live primary database and a backup database. The NOTAM system, which publishes notices to pilots, failed, and the FAA halted departures nationwide on the morning of 11 January.

This reconstruction's synchronise is meant to make the backup match the primary. Instead it mirrors differences in the wrong direction — deleting from the primary whatever the backup lacks — and nothing stops it removing files the service cannot run without.

Fix synchronise so the primary is the source of truth and required files are never deleted.

Code map

var REQUIRED = require("./required").REQUIRED_FILES;

// Brings the two databases back into step after they drifted apart.
// Each database is a map of file name -> contents.
exports.synchronise = function (primary, backup) {
  var deleted = [];
  Object.keys(primary).forEach(function (name) {
    if (!Object.prototype.hasOwnProperty.call(backup, name)) {
      delete primary[name];
      deleted.push(name);
    }
  });
  Object.keys(primary).forEach(function (name) {
    backup[name] = primary[name];
  });
  deleted.sort();
  return { deleted: deleted };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Backup That Deleted the Primary — JavaScript Bug Hunt

Modelled on the FAA NOTAM system outage of 11 January 2023. The FAA's preliminary review found that contract personnel unintentionally deleted files while…

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Replication, State
  • Modelled on: FAA NOTAM · 2023
  • Visible tests: a changed file is copied to the backup; a file only the primary has is copied, not deleted
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-NOTAM-SYNC
  • Reported by: NOTAM operations
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a file only the primary has is copied, not deleted — the primary is never modified: expected ["new.dat=n","notam-index.dat=i1"] but got ["notam-index.dat=i1"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the FAA NOTAM system outage of 11 January 2023. The FAA's preliminary review found that contract personnel unintentionally deleted files while working to correct synchronisation between the live primary database and a backup database. The NOTAM system, which publishes notices to pilots, failed, and the FAA halted departures nationwide on the morning of 11 January.

This reconstruction's synchronise is meant to make the backup match the primary. Instead it mirrors differences in the wrong direction — deleting from the primary whatever the backup lacks — and nothing stops it removing files the service cannot run without.

Fix synchronise so the primary is the source of truth and required files are never deleted.

Bug report

BUG-NOTAM-SYNC · Priority: Critical · Reported by: NOTAM operations

synchronise(primary, backup) — each database is an object mapping file name to contents; the call updates backup in place:

  • the primary is the source of truth and is NEVER modified
  • every primary file is copied to the backup (added, or overwritten when the contents differ)
  • a backup file the primary does not have is deleted from the backup, unless its name is in REQUIRED_FILES (src/notam/required.js) — those are never deleted from either database
  • returns { deleted: [...] }, the names removed from the backup, sorted

Observed: after a sync run, files that existed only on the primary were gone from the primary.

Logs

[sync] primary=412 files backup=398 files
[sync] removed 14 files from primary
[notam] startup failed: notam-index.dat missing

The code as shipped

src/notam/sync.js (editable)

var REQUIRED = require("./required").REQUIRED_FILES;

// Brings the two databases back into step after they drifted apart.
// Each database is a map of file name -> contents.
exports.synchronise = function (primary, backup) {
  var deleted = [];
  Object.keys(primary).forEach(function (name) {
    if (!Object.prototype.hasOwnProperty.call(backup, name)) {
      delete primary[name];
      deleted.push(name);
    }
  });
  Object.keys(primary).forEach(function (name) {
    backup[name] = primary[name];
  });
  deleted.sort();
  return { deleted: deleted };
};

Read-only context: src/notam/required.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.