The Backup That Backed Up Nothing

Medium

· Database

Incident

SEV-1

BUG-GL-2017

Priority: Existential · reported by incident retro

Time to fix: 20 min

Failing checks on the shipped build

  • empty snapshots are never chosen

    expected "pg-dump" but got "s3-sync"

  • the newest verified backup wins

    expected "new" but got "old"

  • no eligible backup raises loudly

    must refuse rather than fake a restore: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

GitLab · 2017

Inspired by GitLab's 2017 database incident, where — mid-recovery — the team discovered that of five backup mechanisms, the ones they reached for first were empty. Restore tooling must pick a candidate that is verified and non-empty, and prefer the newest.

restore.py currently grabs whatever is listed first.

Code map

# Picks which backup to restore from.

def choose_source(candidates):
    return candidates[0]

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Backup That Backed Up Nothing — Python Bug Hunt

Inspired by GitLab's 2017 database incident, where — mid-recovery — the team discovered that of five backup mechanisms, the ones they reached for first were…

  • Language: Python
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Backups, Validation
  • Modelled on: GitLab · 2017
  • Visible tests: empty snapshots are never chosen; the newest verified backup wins; no eligible backup raises loudly
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Existential)
  • Ticket: BUG-GL-2017
  • Reported by: incident retro
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • empty snapshots are never chosen — expected "pg-dump" but got "s3-sync"
  • the newest verified backup wins — expected "new" but got "old"
  • no eligible backup raises loudly — must refuse rather than fake a restore: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by GitLab's 2017 database incident, where — mid-recovery — the team discovered that of five backup mechanisms, the ones they reached for first were empty. Restore tooling must pick a candidate that is verified and non-empty, and prefer the newest.

restore.py currently grabs whatever is listed first.

Bug report

BUG-GL-2017 · Priority: Existential · Reported by: incident retro

choose_source(candidates) — each {name, size_bytes, verified, created_at}:

  • eligible = verified AND size_bytes > 0
  • pick the eligible candidate with the LARGEST created_at
  • none eligible -> raise ValueError (loudly, before deleting anything!)

Observed: the tool picked an unverified, zero-byte snapshot and reported success.

Logs

[restore] using "s3-sync" (0 bytes, unverified)
[restore] restore completed in 0.4s   <- of course it did

The code as shipped

src/backup/restore.py (editable)

# Picks which backup to restore from.

def choose_source(candidates):
    return candidates[0]

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.