Nine A.M. Somewhere — Python Bug Hunt
Inspired by the digest emails that arrive at 3 AM: "send at 9 AM" was implemented as 9 AM server time, for every user on the planet.
- Language: Python
- Layer: Backend
- Difficulty: Easy
- Concepts: Timezones, Scheduling
- Modelled on: Lifecycle email
- Visible tests: east-of-UTC users send earlier in UTC; west-of-UTC users send later in UTC
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-3
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- east-of-UTC users send earlier in UTC — 9 AM IST is 03:30 UTC -> hour 3: expected 3 but got 9
- west-of-UTC users send later in UTC — expected 14 but got 9
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by the digest emails that arrive at 3 AM: "send at 9 AM" was implemented as 9 AM server time, for every user on the planet.
digest.py converts each user's local send-hour to the UTC hour the scheduler runs on. Offsets are minutes east of UTC.
Bug report
BUG-9AM · Reported by: lifecycle marketing
send_hour_utc(local_hour, offset_minutes):
- the UTC hour at which the user's wall clock reads local_hour
- Mumbai (+330): 9 AM local -> 3 UTC
- New York (-300): 9 AM local -> 14 UTC
Observed: the offset is ignored — everyone gets the digest at server 9 AM.
Logs
[digest] IST users report 2:30 PM delivery; PST users 1 AMThe code as shipped
src/mail/digest.py (editable)
# UTC hour at which to send a user's local-time digest.
def send_hour_utc(local_hour, offset_minutes):
return local_hour
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.