The Leap Day Meltdown

Medium

· Backend

Incident

SEV-2

BUG-0229

Priority: High · reported by billing on-call

Time to fix: 20 min

Failing checks on the shipped build

  • leap-year February clamps to the 29th

    2024 is a leap year: expected [2024, 2, 29] but got [2024, 2, 31]

  • non-leap February clamps to the 28th

    expected [2023, 2, 28] but got [2023, 2, 31]

  • 31st clamps into 30-day months

    expected [2024, 4, 30] but got [2024, 4, 31]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

GitHub · 2012

Inspired by the GitHub and Azure outages that both landed on February 29th. Monthly billing anniversaries get pushed "one month forward" — and every date library shortcut breaks on leap years.

billing.py computes the next billing date. Its month table thinks February always has 28 days, and it never clamps.

Code map

# Computes the next monthly billing anniversary.

DAYS_IN_MONTH = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]

def next_billing_date(year, month, day):
    month = month + 1
    if month > 12:
        month = 1
        year = year + 1
    return [year, month, day]

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Leap Day Meltdown — Python Bug Hunt

Inspired by the GitHub and Azure outages that both landed on February 29th.

  • Language: Python
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Dates, Leap Years
  • Modelled on: GitHub · 2012
  • Visible tests: leap-year February clamps to the 29th; non-leap February clamps to the 28th; 31st clamps into 30-day months
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-0229
  • Reported by: billing on-call
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • leap-year February clamps to the 29th — 2024 is a leap year: expected [2024, 2, 29] but got [2024, 2, 31]
  • non-leap February clamps to the 28th — expected [2023, 2, 28] but got [2023, 2, 31]
  • 31st clamps into 30-day months — expected [2024, 4, 30] but got [2024, 4, 31]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the GitHub and Azure outages that both landed on February 29th. Monthly billing anniversaries get pushed "one month forward" — and every date library shortcut breaks on leap years.

billing.py computes the next billing date. Its month table thinks February always has 28 days, and it never clamps.

Bug report

BUG-0229 · Priority: High · Reported by: billing on-call

next_billing_date(year, month, day) -> [year, month, day] rules:

  • move one month forward, clamping the day to the target month's length
  • February has 29 days in leap years (divisible by 4, except centuries unless divisible by 400)

Observed: Jan 31, 2024 -> "Feb 31" -> exception; Jan 31, 2023 -> Mar 3 drift.

Logs

[billing] anniversary(2024, 1, 31) raised: day is out of range for month

The code as shipped

src/billing.py (editable)

# Computes the next monthly billing anniversary.

DAYS_IN_MONTH = [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]

def next_billing_date(year, month, day):
    month = month + 1
    if month > 12:
        month = 1
        year = year + 1
    return [year, month, day]

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.