The Negative Shard — Java Bug Hunt
Inspired by a bug that has paged on-call engineers at every large company: routing by Math.abs(key.hashCode()) % shards works for years — until a key whose…
- Language: Java
- Layer: Database
- Difficulty: Medium
- Concepts: Hashing, Overflow
- Modelled on: Java hashCode lore
- Visible tests: ordinary keys land in range and stay stable; the Integer.MIN_VALUE hash routes safely
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-2 (priority: High)
- Ticket: BUG-SHARD-8
- Reported by: storage on-call
- Time to fix: 20 minutes, the target for a medium incident
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by a bug that has paged on-call engineers at every large company: routing by Math.abs(key.hashCode()) % shards works for years — until a key whose hash is exactly Integer.MIN_VALUE arrives. Math.abs of that number is still negative, and the router indexes shard -8.
Fun fact baked into the tests: the string "polygenelubricants" hashes to exactly Integer.MIN_VALUE.
Bug report
BUG-SHARD-8 · Priority: High · Reported by: storage on-call
shardFor(key, shardCount) must ALWAYS return a value in [0, shardCount) and be stable per key.
Observed: ArrayIndexOutOfBoundsException: -8, once in a blue moon, always the same mystery user.
Logs
[router] key="polygenelubricants" -> shard -8 of 16The code as shipped
ShardRouter.java (editable)
class ShardRouter {
static int shardFor(String key, int shardCount) {
return Math.abs(key.hashCode()) % shardCount;
}
}Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.