The Year That Didn't Fit in an Int — Java Bug Hunt
Modelled on Microsoft Exchange's "Y2K22" bug (1 January 2022): on-premises Exchange servers stopped delivering email at midnight on New Year's Day.
- Language: Java
- Layer: Backend
- Difficulty: Easy
- Concepts: Overflow, Time
- Modelled on: Microsoft Exchange · 2022
- Visible tests: a 2021 version parses; the first version of 2022 still delivers mail
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (mail flow stopped))
- Ticket: BUG-Y2K22
- Reported by: messaging ops
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- the first version of 2022 still delivers mail — Jan 1, 2022 00:01: expected DELIVER but got QUEUED
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Microsoft Exchange's "Y2K22" bug (1 January 2022): on-premises Exchange servers stopped delivering email at midnight on New Year's Day. The version of the malware-scanning engine's updates encodes a date as yyMMddHHmm, and it was being stored in a signed 32-bit integer; the first update of 2022, 2201010001, is larger than 2,147,483,647, so the conversion failed, the scanning engine could not load, and mail sat in the transport queues.
EngineVersion.java parses and compares those version stamps; MalwareFilter.java is the transport's gate.
Fix EngineVersion so every yyMMddHHmm version parses and compares correctly.
Bug report
BUG-Y2K22 · Priority: Critical (mail flow stopped) · Reported by: messaging ops
Engine versions are 10-digit strings "yyMMddHHmm".
- parse(v) returns the numeric value for every 10-digit version, including versions from 2022 onwards ("2201010001" -> 2201010001); anything that is not exactly 10 digits throws an IllegalArgumentException
- isNewer(a, b) is true when a's value is greater than b's
- latest(list) returns the newest version string in the list
- MalwareFilter.route(v) (locked) returns "DELIVER" for a loadable version, "QUEUED" otherwise
Observed: at 00:00 on 2022-01-01 the new engine version "2201010001" failed to load and every message stayed in the queue.
Logs
[FIPFS] Cannot convert "2201010001" to long.
[FIPFS] The FIP-FS Scan Process failed initialization.
[Transport] 18,402 messages in submission queueThe code as shipped
src/filter/EngineVersion.java (editable)
class EngineVersion {
static int parse(String version) {
if (version == null || version.length() != 10) {
throw new IllegalArgumentException("bad engine version: " + version);
}
return Integer.parseInt(version);
}
static boolean isNewer(String candidate, String current) {
return parse(candidate) > parse(current);
}
static String latest(List<String> versions) {
String best = null;
for (String v : versions) {
if (best == null || isNewer(v, best)) best = v;
}
return best;
}
}Read-only context: src/filter/MalwareFilter.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.