The Backup That Always Said Yes

Easy

· Database

Incident

SEV-1

BUG-TS2-1998

Priority: Critical · reported by systems

Time to fix: 10 min

Failing checks on the shipped build

  • a full tape is a failed backup

    never report ok for a partial tape: expected {"bytes": 8, "files": 2, "ok": false} but got {"bytes": 8, "files": 2, "ok": true}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Pixar · 1998

Modelled on Pixar's Toy Story 2 deletion (1998): a recursive delete command run on the film's file server wiped out most of the production's files, and when the team turned to the backups they found they had been failing for some time without anyone noticing. The film was largely recovered from a copy a technical director happened to keep at home.

This project is a reconstruction. backup.py writes every file to a tape, but it swallows write errors and reports success based on what it tried to write — never checking what the tape actually holds.

Fix run_backup so failures are surfaced and the result is verified against the tape.

Code map

tape_mod = bug_require("./tape.py")


def run_backup(source, tape):
    """Copy every file in source (name -> data) to the tape."""
    files = 0
    size = 0
    for name in sorted(source):
        data = source[name]
        try:
            tape.write(name, data)
        except tape_mod.TapeFull:
            continue
        files += 1
        size += len(data)
    return {"ok": True, "files": files, "bytes": size}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Backup That Always Said Yes — Python Bug Hunt

Modelled on Pixar's Toy Story 2 deletion (1998): a recursive delete command run on the film's file server wiped out most of the production's files, and when…

  • Language: Python
  • Layer: Database
  • Difficulty: Easy
  • Concepts: Reliability, Validation
  • Modelled on: Pixar · 1998
  • Visible tests: a backup that fits is reported ok; a full tape is a failed backup
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-TS2-1998
  • Reported by: systems
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • a full tape is a failed backup — never report ok for a partial tape: expected {"bytes": 8, "files": 2, "ok": false} but got {"bytes": 8, "files": 2, "ok": true}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Pixar's Toy Story 2 deletion (1998): a recursive delete command run on the film's file server wiped out most of the production's files, and when the team turned to the backups they found they had been failing for some time without anyone noticing. The film was largely recovered from a copy a technical director happened to keep at home.

This project is a reconstruction. backup.py writes every file to a tape, but it swallows write errors and reports success based on what it tried to write — never checking what the tape actually holds.

Fix run_backup so failures are surfaced and the result is verified against the tape.

Bug report

BUG-TS2-1998 · Priority: Critical · Reported by: systems

run_backup(source, tape) copies source (name -> data) to an empty tape, in sorted name order, and returns {"ok", "files", "bytes"}:

  • on the first TapeFull error, stop writing (attempt no further files)
  • "files" and "bytes" describe what tape.catalogue() actually holds afterwards (count of entries, sum of sizes)
  • "ok" is True only if no write failed AND every source file is in the catalogue with its exact size; otherwise False

Observed: the nightly job has reported ok for months while the tape held only part of the tree.

Logs

[backup] 1998-11-02 nightly: ok (files=18422)
[backup] 1998-11-03 nightly: ok (files=18422)
[restore] tape 0412: 4,117 of 18,422 expected files present

The code as shipped

src/backup/backup.py (editable)

tape_mod = bug_require("./tape.py")


def run_backup(source, tape):
    """Copy every file in source (name -> data) to the tape."""
    files = 0
    size = 0
    for name in sorted(source):
        data = source[name]
        try:
            tape.write(name, data)
        except tape_mod.TapeFull:
            continue
        files += 1
        size += len(data)
    return {"ok": True, "files": files, "bytes": size}

Read-only context: src/backup/tape.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.