The Files Deleted Years Ago Came Back

Medium

· Database

Incident

SEV-1

BUG-RESURRECT

Priority: Critical · reported by support (users)

Time to fix: 20 min

Failing checks on the shipped build

  • a file deleted long ago stays deleted

    the tombstone's work must survive compaction: expected ["/notes"] but got ["/notes","/photos/2015"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Dropbox · 2017

Modelled on Dropbox in January 2017, when users reported that files and folders they had deleted — some of them years earlier — had reappeared in their accounts. Dropbox said a bug had caused a small number of users' deleted files to be restored and that it was contacting the people affected.

This reconstruction models one plausible way such a bug happens: a compaction job trims each path's event log before the log is replayed into the live file tree. It throws away old delete tombstones but keeps the old uploads they cancelled, so the next replay brings the files back. (Dropbox's own internal root cause was not published in this form; the project is a reconstruction of the failure, not of their code.)

Fix compact so compaction never changes which files are live.

Code map

// Compacts the event log before it is replayed into the live file tree.
// events: [{ path, op: "put" | "delete", at }] in the order they happened.
exports.compact = function (events, now, retentionMs) {
  return events.filter(function (e) {
    if (e.op === "put") return true;
    return now - e.at <= retentionMs;
  });
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Files Deleted Years Ago Came Back — JavaScript Bug Hunt

Modelled on Dropbox in January 2017, when users reported that files and folders they had deleted — some of them years earlier — had reappeared in their…

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Data, Sync, Time
  • Modelled on: Dropbox · 2017
  • Visible tests: recent events are kept as they are; a file deleted long ago stays deleted
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-RESURRECT
  • Reported by: support (users)
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a file deleted long ago stays deleted — the tombstone's work must survive compaction: expected ["/notes"] but got ["/notes","/photos/2015"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Dropbox in January 2017, when users reported that files and folders they had deleted — some of them years earlier — had reappeared in their accounts. Dropbox said a bug had caused a small number of users' deleted files to be restored and that it was contacting the people affected.

This reconstruction models one plausible way such a bug happens: a compaction job trims each path's event log before the log is replayed into the live file tree. It throws away old delete tombstones but keeps the old uploads they cancelled, so the next replay brings the files back. (Dropbox's own internal root cause was not published in this form; the project is a reconstruction of the failure, not of their code.)

Fix compact so compaction never changes which files are live.

Bug report

BUG-RESURRECT · Priority: Critical · Reported by: support (users)

compact(events, now, retentionMs) — events are { path, op: "put"|"delete", at } in the order they happened. It returns the events to keep, in their original order:

  • keep every event with now - at <= retentionMs
  • additionally keep, for each path whose LATEST event is a put, that latest put (however old — a live file is never compacted away)
  • drop everything else

Invariant: replay.livePaths(compact(events, ...)) === replay.livePaths(events).

Observed: a folder deleted two years ago is live again after the nightly compaction.

Logs

[compact] 3 tombstones older than 30d dropped
[replay] /photos/2015 restored (latest event: put @ day 10)

The code as shipped

src/sync/compact.js (editable)

// Compacts the event log before it is replayed into the live file tree.
// events: [{ path, op: "put" | "delete", at }] in the order they happened.
exports.compact = function (events, now, retentionMs) {
  return events.filter(function (e) {
    if (e.op === "put") return true;
    return now - e.at <= retentionMs;
  });
};

Read-only context: src/sync/replay.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.