The Cleanup That Deleted the Customers

Medium

· Database

Incident

SEV-1

BUG-JIRA22

Priority: Critical (data loss) · reported by incident command

Time to fix: 20 min

Failing checks on the shipped build

  • sites are never deleted

    a site id must be refused: expected {"deleted":["app-3"],"skipped":["site-19"]} but got {"deleted":["site-19","app-3"],"skipped":[]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Atlassian Jira · 2022

Modelled on the Atlassian Jira outage (April 2022): a script meant to remove a deprecated app was handed site IDs where it expected app IDs, and it ran with a "permanent delete" flag. About 400 customer sites were wiped, and restoring them took two weeks.

cleanup.js deletes records by id without checking that the ids it was given are the right kind.

Fix removeApps so it only ever deletes app records, and reports ids it refuses.

Code map

// Removes the given ids from the store.
exports.removeApps = function (store, ids) {
  var deleted = [];
  var skipped = [];
  for (var i = 0; i < ids.length; i++) {
    var id = ids[i];
    if (store[id]) {
      delete store[id];
      deleted.push(id);
    } else {
      skipped.push(id);
    }
  }
  return { deleted: deleted, skipped: skipped };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Cleanup That Deleted the Customers — JavaScript Bug Hunt

Modelled on the Atlassian Jira outage (April 2022): a script meant to remove a deprecated app was handed site IDs where it expected app IDs, and it ran with…

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Data Loss, Identifiers
  • Modelled on: Atlassian Jira · 2022
  • Visible tests: apps are deleted; sites are never deleted
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (data loss))
  • Ticket: BUG-JIRA22
  • Reported by: incident command
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • sites are never deleted — a site id must be refused: expected {"deleted":["app-3"],"skipped":["site-19"]} but got {"deleted":["site-19","app-3"],"skipped":[]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Atlassian Jira outage (April 2022): a script meant to remove a deprecated app was handed site IDs where it expected app IDs, and it ran with a "permanent delete" flag. About 400 customer sites were wiped, and restoring them took two weeks.

cleanup.js deletes records by id without checking that the ids it was given are the right kind.

Fix removeApps so it only ever deletes app records, and reports ids it refuses.

Bug report

BUG-JIRA22 · Priority: Critical (data loss) · Reported by: incident command

removeApps(store, ids) must:

  • delete only entries whose kind is "app"
  • leave every other entry untouched
  • return { deleted: [...], skipped: [...] } listing the ids in each group, in the order they were supplied

Observed: passing site ids deletes the sites. The function never looks at the record's kind.

Logs

[cleanup] deleted id=site-19 kind=site
[cleanup] 400 customer sites removed in 9 minutes

The code as shipped

src/ops/cleanup.js (editable)

// Removes the given ids from the store.
exports.removeApps = function (store, ids) {
  var deleted = [];
  var skipped = [];
  for (var i = 0; i < ids.length; i++) {
    var id = ids[i];
    if (store[id]) {
      delete store[id];
      deleted.push(id);
    } else {
      skipped.push(id);
    }
  }
  return { deleted: deleted, skipped: skipped };
};

Read-only context: src/ops/SCHEMA.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.