The Negative Shard

Medium

· Database

Incident

SEV-2

BUG-SHARD-8

Priority: High · reported by storage on-call

Time to fix: 20 min

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Java hashCode lore

Inspired by a bug that has paged on-call engineers at every large company: routing by Math.abs(key.hashCode()) % shards works for years — until a key whose hash is exactly Integer.MIN_VALUE arrives. Math.abs of that number is still negative, and the router indexes shard -8.

Fun fact baked into the tests: the string "polygenelubricants" hashes to exactly Integer.MIN_VALUE.

Code map

class ShardRouter {
    static int shardFor(String key, int shardCount) {
        return Math.abs(key.hashCode()) % shardCount;
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Negative Shard — Java Bug Hunt

Inspired by a bug that has paged on-call engineers at every large company: routing by Math.abs(key.hashCode()) % shards works for years — until a key whose…

  • Language: Java
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Hashing, Overflow
  • Modelled on: Java hashCode lore
  • Visible tests: ordinary keys land in range and stay stable; the Integer.MIN_VALUE hash routes safely
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-SHARD-8
  • Reported by: storage on-call
  • Time to fix: 20 minutes, the target for a medium incident

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by a bug that has paged on-call engineers at every large company: routing by Math.abs(key.hashCode()) % shards works for years — until a key whose hash is exactly Integer.MIN_VALUE arrives. Math.abs of that number is still negative, and the router indexes shard -8.

Fun fact baked into the tests: the string "polygenelubricants" hashes to exactly Integer.MIN_VALUE.

Bug report

BUG-SHARD-8 · Priority: High · Reported by: storage on-call

shardFor(key, shardCount) must ALWAYS return a value in [0, shardCount) and be stable per key.

Observed: ArrayIndexOutOfBoundsException: -8, once in a blue moon, always the same mystery user.

Logs

[router] key="polygenelubricants" -> shard -8 of 16

The code as shipped

ShardRouter.java (editable)

class ShardRouter {
    static int shardFor(String key, int shardCount) {
        return Math.abs(key.hashCode()) % shardCount;
    }
}

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.