Read Your Own Writes

Easy

· Database

Incident

SEV-2

BUG-RYW

Priority: High · reported by community team

Time to fix: 10 min

Failing checks on the shipped build

  • fresh writers read from the primary

    expected primary but got replica

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Replicated databases

Inspired by the "my comment vanished!" bug of every replicated database: a user writes to the primary, the very next read hits a lagging replica, and their data is briefly gone. The classic fix: pin reads to the primary for a short window after your own write.

ReadRouter.java ignores the window.

Code map

class ReadRouter {
    static final long PIN_WINDOW_MS = 500;

    static String routeRead(long msSinceOwnWrite) {
        return "replica";
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Read Your Own Writes — Java Bug Hunt

Inspired by the "my comment vanished!" bug of every replicated database: a user writes to the primary, the very next read hits a lagging replica, and their…

  • Language: Java
  • Layer: Database
  • Difficulty: Easy
  • Concepts: Replication, Consistency
  • Modelled on: Replicated databases
  • Visible tests: fresh writers read from the primary; old writers read from replicas
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-RYW
  • Reported by: community team
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • fresh writers read from the primary — expected primary but got replica

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the "my comment vanished!" bug of every replicated database: a user writes to the primary, the very next read hits a lagging replica, and their data is briefly gone. The classic fix: pin reads to the primary for a short window after your own write.

ReadRouter.java ignores the window.

Bug report

BUG-RYW · Priority: High · Reported by: community team

routeRead(msSinceOwnWrite) with a 500ms pin window:

  • writes newer than 500ms (including clock-skew negatives) -> "primary"
  • otherwise -> "replica"

Observed: everything goes to replicas; users see their posts flicker out of existence right after submitting.

Logs

[router] msSinceOwnWrite=40 -> replica (lag 180ms) -> row not found

The code as shipped

ReadRouter.java (editable)

class ReadRouter {
    static final long PIN_WINDOW_MS = 500;

    static String routeRead(long msSinceOwnWrite) {
        return "replica";
    }
}

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.