The Runaway Retry

Easy

· Database

Incident

SEV-2

BUG-STAMPEDE

Priority: High · reported by SRE

Time to fix: 10 min

Failing checks on the shipped build

  • the schedule doubles from one second

    expected [1, 2, 4, 8, 16] but got [0, 1, 2, 4, 8, 16]

  • long schedules cap at sixty

    expected [1, 2, 4, 8, 16, 32, 60, 60] but got [0, 1, 2, 4, 8, 16, 32, 64, 128]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Retry storms

Inspired by the retry storms that finish off every wounded database: a hiccup triggers retries, uncapped exponential delays overflow into instant retries — and the "backoff" becomes a battering ram.

backoff.py: n retry delays, doubling from 1s, capped at 60s.

Code map

# Exponential backoff schedule for reconnect attempts.

CAP_SECONDS = 60

def delays(n):
    return [2 ** i // 2 for i in range(n + 1)]

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Runaway Retry — Python Bug Hunt

Inspired by the retry storms that finish off every wounded database: a hiccup triggers retries, uncapped exponential delays overflow into instant retries —…

  • Language: Python
  • Layer: Database
  • Difficulty: Easy
  • Concepts: Retries, Backoff
  • Modelled on: Retry storms
  • Visible tests: the schedule doubles from one second; long schedules cap at sixty
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-STAMPEDE
  • Reported by: SRE
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • the schedule doubles from one second — expected [1, 2, 4, 8, 16] but got [0, 1, 2, 4, 8, 16]
  • long schedules cap at sixty — expected [1, 2, 4, 8, 16, 32, 60, 60] but got [0, 1, 2, 4, 8, 16, 32, 64, 128]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the retry storms that finish off every wounded database: a hiccup triggers retries, uncapped exponential delays overflow into instant retries — and the "backoff" becomes a battering ram.

backoff.py: n retry delays, doubling from 1s, capped at 60s.

Bug report

BUG-STAMPEDE · Priority: High · Reported by: SRE

delays(n):

  • [1, 2, 4, 8, 16, 32, 60, 60, ...] — doubling, capped at 60, length n

Observed: the list starts at 0 (an INSTANT retry), never caps, and returns one extra element — retry #31 asks to wait 2^31 seconds.

Logs

[retry] delays(3) -> [0, 1, 2, 4]

The code as shipped

src/db/backoff.py (editable)

# Exponential backoff schedule for reconnect attempts.

CAP_SECONDS = 60

def delays(n):
    return [2 ** i // 2 for i in range(n + 1)]

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.