The Notification Echo

Easy

· Backend

Incident

SEV-2

BUG-BUZZ

Priority: High (app deletions) · reported by mobile

Time to fix: 10 min

Failing checks on the shipped build

  • different users are independent

    user_b was never notified: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Coinbase-class alerts

Inspired by the price-alert storms that buzz phones at 3 AM — the same alert, five times, because the dedupe state was shared globally instead of tracked per user and per alert.

alerts.py rate-limits notifications: one per (user, alert key) per hour.

Code map

# Per-user, per-alert notification rate limiting.

WINDOW_SECONDS = 3600

def should_send(user, key, now, state):
    last = state.get("last_sent")
    if last is not None and now - last < WINDOW_SECONDS:
        return False
    state["last_sent"] = now
    return True

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Notification Echo — Python Bug Hunt

Inspired by the price-alert storms that buzz phones at 3 AM — the same alert, five times, because the dedupe state was shared globally instead of tracked…

  • Language: Python
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Dedupe, State
  • Modelled on: Coinbase-class alerts
  • Visible tests: different users are independent; the same alert is suppressed within the window; the window expires
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High (app deletions))
  • Ticket: BUG-BUZZ
  • Reported by: mobile
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • different users are independent — user_b was never notified: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the price-alert storms that buzz phones at 3 AM — the same alert, five times, because the dedupe state was shared globally instead of tracked per user and per alert.

alerts.py rate-limits notifications: one per (user, alert key) per hour.

Bug report

BUG-BUZZ · Priority: High (app deletions) · Reported by: mobile

should_send(user, key, now, state):

  • send if this (user, key) pair has not fired within 3600 seconds
  • record the send time on success

Observed: ONE global timestamp for everything — after any alert fires, every other user's alerts are suppressed for an hour (and vice versa at reset).

Logs

[alerts] user_a BTC alert fired -> user_b's ETH alert suppressed

The code as shipped

src/notify/alerts.py (editable)

# Per-user, per-alert notification rate limiting.

WINDOW_SECONDS = 3600

def should_send(user, key, now, state):
    last = state.get("last_sent")
    if last is not None and now - last < WINDOW_SECONDS:
        return False
    state["last_sent"] = now
    return True

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.