The Backup That Backed Up Nothing — Python Bug Hunt
Inspired by GitLab's 2017 database incident, where — mid-recovery — the team discovered that of five backup mechanisms, the ones they reached for first were…
- Language: Python
- Layer: Database
- Difficulty: Medium
- Concepts: Backups, Validation
- Modelled on: GitLab · 2017
- Visible tests: empty snapshots are never chosen; the newest verified backup wins; no eligible backup raises loudly
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Existential)
- Ticket: BUG-GL-2017
- Reported by: incident retro
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- empty snapshots are never chosen — expected "pg-dump" but got "s3-sync"
- the newest verified backup wins — expected "new" but got "old"
- no eligible backup raises loudly — must refuse rather than fake a restore: expected an error to be raised
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by GitLab's 2017 database incident, where — mid-recovery — the team discovered that of five backup mechanisms, the ones they reached for first were empty. Restore tooling must pick a candidate that is verified and non-empty, and prefer the newest.
restore.py currently grabs whatever is listed first.
Bug report
BUG-GL-2017 · Priority: Existential · Reported by: incident retro
choose_source(candidates) — each {name, size_bytes, verified, created_at}:
- eligible = verified AND size_bytes > 0
- pick the eligible candidate with the LARGEST created_at
- none eligible -> raise ValueError (loudly, before deleting anything!)
Observed: the tool picked an unverified, zero-byte snapshot and reported success.
Logs
[restore] using "s3-sync" (0 bytes, unverified)
[restore] restore completed in 0.4s <- of course it didThe code as shipped
src/backup/restore.py (editable)
# Picks which backup to restore from.
def choose_source(candidates):
return candidates[0]
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.