The Week That Belonged to Next Year — Java Bug Hunt
Modelled on Twitter for Android, 29 December 2014: many Android users were signed out of the app and could not sign back in.
- Language: Java
- Layer: Backend
- Difficulty: Medium
- Concepts: Time, Dates, Auth
- Modelled on: Twitter · 2014
- Visible tests: a mid-December instant; 29 December 2014 is still 2014
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-YYYY
- Reported by: mobile on-call
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- 29 December 2014 is still 2014 — calendar year, not week year: expected 2014-12-29 00:00 but got 2015-12-29 00:00
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Twitter for Android, 29 December 2014: many Android users were signed out of the app and could not sign back in. The cause, as Twitter's engineers explained, was a date format written with YYYY — the week-based year — instead of yyyy, the calendar year. In the last days of December the week that contains 1 January already belongs to the next week-based year, so the app believed it was 2015 a few days early.
This reconstruction stamps session times as strings and compares them to decide whether a session has expired.
Fix Stamps so a stamp always shows the calendar year.
Bug report
BUG-YYYY · Priority: Critical · Reported by: mobile on-call
Stamps.stamp(epochMillis) formats an instant in UTC as "yyyy-MM-dd HH:mm" using the CALENDAR year (Locale.US), e.g.
- 2014-12-15T09:30Z -> "2014-12-15 09:30"
- 2014-12-29T00:00Z -> "2014-12-29 00:00"
- 2014-12-31T23:59Z -> "2014-12-31 23:59"
- 2015-01-01T00:00Z -> "2015-01-01 00:00"
SessionGuard (locked) compares these strings, so a stamp that jumps a year ahead makes every session look expired.
Observed: from 28 December onwards every stamp reads 2015 and users are signed out on launch.
Logs
[session] now=2015-12-29 00:00 expires=2015-01-11 12:00 -> EXPIRED, signing out
[session] 1 of 1 sessions expired on launch (device clock 2014-12-29)The code as shipped
src/session/Stamps.java (editable)
import java.text.SimpleDateFormat;
class Stamps {
static final String PATTERN = "YYYY-MM-dd HH:mm";
static String stamp(long epochMillis) {
SimpleDateFormat f = new SimpleDateFormat(PATTERN, Locale.US);
f.setTimeZone(TimeZone.getTimeZone("UTC"));
return f.format(new Date(epochMillis));
}
}Read-only context: src/session/SessionGuard.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.