The Clock That Ran Backwards — Java Bug Hunt
Inspired by Twitter's Snowflake ID generator — and the NTP clock-rollback duplicates it explicitly guards against.
- Language: Java
- Layer: Database
- Difficulty: Hard
- Concepts: IDs, Clocks
- Modelled on: Twitter Snowflake
- Visible tests: a new tick resets the sequence; the same tick increments the sequence; a backwards clock is refused
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-SNOW
- Reported by: storage
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- a new tick resets the sequence — expected [4100096, 0] but got [4100104, 8]
- a backwards clock is refused — clock rollback must throw: expected condition to hold
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by Twitter's Snowflake ID generator — and the NTP clock-rollback duplicates it explicitly guards against. IDs are (timestamp << 12) | sequence and must be strictly increasing; if the wall clock ever runs backwards, the generator must refuse rather than mint colliding IDs.
IdGen.java currently trusts the clock and never resets the sequence.
Bug report
BUG-SNOW · Priority: Critical · Reported by: storage
next(lastTimestamp, lastSeq, now):
- now < lastTimestamp -> throw (clock moved backwards!)
- now == lastTimestamp -> same tick: sequence + 1
- now > lastTimestamp -> new tick: sequence resets to 0
- returns id = now * 4096 + seq
Observed: duplicate IDs after an NTP correction, and sequences that never reset — overflowing the 12-bit field within a busy second.
Logs
[idgen] now=1699999999 < last=1700000042 -> id went BACKWARDS
[db] duplicate key: 6963200172032The code as shipped
IdGen.java (editable)
class IdGen {
// Returns { id, seqUsed } packed as long[2].
static long[] next(long lastTimestamp, long lastSeq, long now) {
long seq = lastSeq + 1;
long id = now * 4096 + seq;
return new long[] { id, seq };
}
}Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.