The Backup That Always Said Yes — Python Bug Hunt
Modelled on Pixar's Toy Story 2 deletion (1998): a recursive delete command run on the film's file server wiped out most of the production's files, and when…
- Language: Python
- Layer: Database
- Difficulty: Easy
- Concepts: Reliability, Validation
- Modelled on: Pixar · 1998
- Visible tests: a backup that fits is reported ok; a full tape is a failed backup
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-TS2-1998
- Reported by: systems
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- a full tape is a failed backup — never report ok for a partial tape: expected {"bytes": 8, "files": 2, "ok": false} but got {"bytes": 8, "files": 2, "ok": true}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Pixar's Toy Story 2 deletion (1998): a recursive delete command run on the film's file server wiped out most of the production's files, and when the team turned to the backups they found they had been failing for some time without anyone noticing. The film was largely recovered from a copy a technical director happened to keep at home.
This project is a reconstruction. backup.py writes every file to a tape, but it swallows write errors and reports success based on what it tried to write — never checking what the tape actually holds.
Fix run_backup so failures are surfaced and the result is verified against the tape.
Bug report
BUG-TS2-1998 · Priority: Critical · Reported by: systems
run_backup(source, tape) copies source (name -> data) to an empty tape, in sorted name order, and returns {"ok", "files", "bytes"}:
- on the first TapeFull error, stop writing (attempt no further files)
- "files" and "bytes" describe what tape.catalogue() actually holds afterwards (count of entries, sum of sizes)
- "ok" is True only if no write failed AND every source file is in the catalogue with its exact size; otherwise False
Observed: the nightly job has reported ok for months while the tape held only part of the tree.
Logs
[backup] 1998-11-02 nightly: ok (files=18422)
[backup] 1998-11-03 nightly: ok (files=18422)
[restore] tape 0412: 4,117 of 18,422 expected files presentThe code as shipped
src/backup/backup.py (editable)
tape_mod = bug_require("./tape.py")
def run_backup(source, tape):
"""Copy every file in source (name -> data) to the tape."""
files = 0
size = 0
for name in sorted(source):
data = source[name]
try:
tape.write(name, data)
except tape_mod.TapeFull:
continue
files += 1
size += len(data)
return {"ok": True, "files": files, "bytes": size}
Read-only context: src/backup/tape.py.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.