The Hang at 49.7 Days

Hard

· Backend

Incident

SEV-2

BUG-Q216641

Priority: High (hang) · reported by long-uptime test rig

Time to fix: 35 min

Failing checks on the shipped build

  • a timeout started just before the wrap expires after it

    1000 ms have passed: expected condition to hold

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Windows 95/98 · 1999

Modelled on the Windows 95 / Windows 98 hang documented by Microsoft in 1999 (Knowledge Base article Q216641): after 49.7 days of continuous uptime the system could stop responding. The cause was a 32-bit millisecond tick counter — 2^32 ms is about 49.7 days — that wraps back to zero, and timing code that did not survive the wrap. Few people noticed for years, because few Windows 9x machines stayed up that long.

This project is a reconstruction: TickTimer measures elapsed time and timeouts from readings of an unsigned 32-bit tick counter (held in a long, always in [0, 2^32)). Its arithmetic is correct only until the counter wraps; a deadline computed just before the wrap is never reached.

Fix elapsed and expired so they are correct across the wrap.

Code map

class TickTimer {
    // Milliseconds between two readings of the tick counter.
    static long elapsed(long start, long now) {
        return now - start;
    }

    static boolean expired(long start, long timeoutMs, long now) {
        long deadline = start + timeoutMs;
        return now >= deadline;
    }

    // Index of the first reading at which the timeout has expired, or -1.
    static int firstExpired(long start, long timeoutMs, long[] readings) {
        for (int i = 0; i < readings.length; i++) {
            if (expired(start, timeoutMs, readings[i])) return i;
        }
        return -1;
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Hang at 49.7 Days — Java Bug Hunt

Modelled on the Windows 95 / Windows 98 hang documented by Microsoft in 1999 (Knowledge Base article Q216641): after 49.7 days of continuous uptime the…

  • Language: Java
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Time, Overflow
  • Modelled on: Windows 95/98 · 1999
  • Visible tests: elapsed time within one lap; a timeout started just before the wrap expires after it
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High (hang))
  • Ticket: BUG-Q216641
  • Reported by: long-uptime test rig
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a timeout started just before the wrap expires after it — 1000 ms have passed: expected condition to hold

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Windows 95 / Windows 98 hang documented by Microsoft in 1999 (Knowledge Base article Q216641): after 49.7 days of continuous uptime the system could stop responding. The cause was a 32-bit millisecond tick counter — 2^32 ms is about 49.7 days — that wraps back to zero, and timing code that did not survive the wrap. Few people noticed for years, because few Windows 9x machines stayed up that long.

This project is a reconstruction: TickTimer measures elapsed time and timeouts from readings of an unsigned 32-bit tick counter (held in a long, always in [0, 2^32)). Its arithmetic is correct only until the counter wraps; a deadline computed just before the wrap is never reached.

Fix elapsed and expired so they are correct across the wrap.

Bug report

BUG-Q216641 · Priority: High (hang) · Reported by: long-uptime test rig

Tick values are unsigned 32-bit milliseconds in [0, 2^32) (see Tick.MOD).

  • elapsed(start, now) = milliseconds from start to now, modulo 2^32 — always in [0, 2^32). elapsed(4294967295, 5) is 6.
  • expired(start, timeoutMs, now) is true iff elapsed(start, now) >= timeoutMs (timeouts are always below 2^31).
  • firstExpired(start, timeoutMs, readings) is the index of the first reading at which the timeout has expired, or -1.

Observed: a 1-second wait started at tick 4294967000 never finishes — every reading after the wrap is "before the deadline".

Logs

[vmm] tick=4294966904 wait(timeout=1000) start
[vmm] tick=0000000312 deadline=4294967904 not reached
[vmm] tick=0003600000 deadline=4294967904 not reached (watchdog: none)

The code as shipped

src/timer/TickTimer.java (editable)

class TickTimer {
    // Milliseconds between two readings of the tick counter.
    static long elapsed(long start, long now) {
        return now - start;
    }

    static boolean expired(long start, long timeoutMs, long now) {
        long deadline = start + timeoutMs;
        return now >= deadline;
    }

    // Index of the first reading at which the timeout has expired, or -1.
    static int firstExpired(long start, long timeoutMs, long[] readings) {
        for (int i = 0; i < readings.length; i++) {
            if (expired(start, timeoutMs, readings[i])) return i;
        }
        return -1;
    }
}

Read-only context: src/timer/Tick.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.