The Backup That Backed Up Nothing — Python Bug Hunt

Briefing

Inspired by GitLab's 2017 database incident, where — mid-recovery — the team discovered that of five backup mechanisms, the ones they reached for first were empty. Restore tooling must pick a candidate that is verified and non-empty, and prefer the newest.

restore.py currently grabs whatever is listed first.

Bug report

BUG-GL-2017 · Priority: Existential · Reported by: incident retro

choose_source(candidates) — each {name, size_bytes, verified, created_at}:

• eligible = verified AND size_bytes > 0
• pick the eligible candidate with the LARGEST created_at
• none eligible -> raise ValueError (loudly, before deleting anything!)

Observed: the tool picked an unverified, zero-byte snapshot and reported success.

Logs

[restore] using "s3-sync" (0 bytes, unverified)
[restore] restore completed in 0.4s   <- of course it did

The code as shipped

src/backup/restore.py (editable)

# Picks which backup to restore from.

def choose_source(candidates):
    return candidates[0]