The Runaway Retry — Python Bug Hunt
Inspired by the retry storms that finish off every wounded database: a hiccup triggers retries, uncapped exponential delays overflow into instant retries —…
- Language: Python
- Layer: Database
- Difficulty: Easy
- Concepts: Retries, Backoff
- Modelled on: Retry storms
- Visible tests: the schedule doubles from one second; long schedules cap at sixty
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-2 (priority: High)
- Ticket: BUG-STAMPEDE
- Reported by: SRE
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- the schedule doubles from one second — expected [1, 2, 4, 8, 16] but got [0, 1, 2, 4, 8, 16]
- long schedules cap at sixty — expected [1, 2, 4, 8, 16, 32, 60, 60] but got [0, 1, 2, 4, 8, 16, 32, 64, 128]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by the retry storms that finish off every wounded database: a hiccup triggers retries, uncapped exponential delays overflow into instant retries — and the "backoff" becomes a battering ram.
backoff.py: n retry delays, doubling from 1s, capped at 60s.
Bug report
BUG-STAMPEDE · Priority: High · Reported by: SRE
delays(n):
- [1, 2, 4, 8, 16, 32, 60, 60, ...] — doubling, capped at 60, length n
Observed: the list starts at 0 (an INSTANT retry), never caps, and returns one extra element — retry #31 asks to wait 2^31 seconds.
Logs
[retry] delays(3) -> [0, 1, 2, 4]The code as shipped
src/db/backoff.py (editable)
# Exponential backoff schedule for reconnect attempts.
CAP_SECONDS = 60
def delays(n):
return [2 ** i // 2 for i in range(n + 1)]
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.