Nine A.M. Somewhere

Easy

· Backend

Incident

SEV-3

Time to fix: 10 min

Failing checks on the shipped build

  • east-of-UTC users send earlier in UTC

    9 AM IST is 03:30 UTC -> hour 3: expected 3 but got 9

  • west-of-UTC users send later in UTC

    expected 14 but got 9

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Lifecycle email

Inspired by the digest emails that arrive at 3 AM: "send at 9 AM" was implemented as 9 AM server time, for every user on the planet.

digest.py converts each user's local send-hour to the UTC hour the scheduler runs on. Offsets are minutes east of UTC.

Code map

# UTC hour at which to send a user's local-time digest.

def send_hour_utc(local_hour, offset_minutes):
    return local_hour

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Nine A.M. Somewhere — Python Bug Hunt

Inspired by the digest emails that arrive at 3 AM: "send at 9 AM" was implemented as 9 AM server time, for every user on the planet.

  • Language: Python
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Timezones, Scheduling
  • Modelled on: Lifecycle email
  • Visible tests: east-of-UTC users send earlier in UTC; west-of-UTC users send later in UTC
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-3
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • east-of-UTC users send earlier in UTC — 9 AM IST is 03:30 UTC -> hour 3: expected 3 but got 9
  • west-of-UTC users send later in UTC — expected 14 but got 9

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the digest emails that arrive at 3 AM: "send at 9 AM" was implemented as 9 AM server time, for every user on the planet.

digest.py converts each user's local send-hour to the UTC hour the scheduler runs on. Offsets are minutes east of UTC.

Bug report

BUG-9AM · Reported by: lifecycle marketing

send_hour_utc(local_hour, offset_minutes):

  • the UTC hour at which the user's wall clock reads local_hour
  • Mumbai (+330): 9 AM local -> 3 UTC
  • New York (-300): 9 AM local -> 14 UTC

Observed: the offset is ignored — everyone gets the digest at server 9 AM.

Logs

[digest] IST users report 2:30 PM delivery; PST users 1 AM

The code as shipped

src/mail/digest.py (editable)

# UTC hour at which to send a user's local-time digest.

def send_hour_utc(local_hour, offset_minutes):
    return local_hour

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.