The Cleanup That Deleted the Customers — JavaScript Bug Hunt

Briefing

Modelled on the Atlassian Jira outage (April 2022): a script meant to remove a deprecated app was handed site IDs where it expected app IDs, and it ran with a "permanent delete" flag. About 400 customer sites were wiped, and restoring them took two weeks.

cleanup.js deletes records by id without checking that the ids it was given are the right kind.

Fix removeApps so it only ever deletes app records, and reports ids it refuses.

Bug report

BUG-JIRA22 · Priority: Critical (data loss) · Reported by: incident command

removeApps(store, ids) must:

• delete only entries whose kind is "app"
• leave every other entry untouched
• return { deleted: [...], skipped: [...] } listing the ids in each group, in the order they were supplied

Observed: passing site ids deletes the sites. The function never looks at the record's kind.

Logs

[cleanup] deleted id=site-19 kind=site
[cleanup] 400 customer sites removed in 9 minutes

The code as shipped

src/ops/cleanup.js (editable)

// Removes the given ids from the store.
exports.removeApps = function (store, ids) {
  var deleted = [];
  var skipped = [];
  for (var i = 0; i < ids.length; i++) {
    var id = ids[i];
    if (store[id]) {
      delete store[id];
      deleted.push(id);
    } else {
      skipped.push(id);
    }
  }
  return { deleted: deleted, skipped: skipped };
};

Read-only context: src/ops/SCHEMA.js.