The Optimistic Lock That Wasn't

Medium

· Database

Incident

SEV-2

BUG-LOST-UPDATE

Priority: High · reported by CMS team

Time to fix: 20 min

Failing checks on the shipped build

  • a matching version applies and bumps

    version must increment: expected 5 but got 4

  • a stale version is rejected

    conflict must raise: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

CMS platforms

Inspired by the lost-update bugs behind countless "my changes disappeared" tickets. Two editors load version 4; both save; the second silently erases the first. Optimistic locking exists precisely to stop this — if anyone actually checks the version.

optimistic.py doesn't.

Code map

# Optimistic-concurrency row update.

def update_row(row, expected_version, changes):
    for key in changes:
        row[key] = changes[key]
    return row

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Optimistic Lock That Wasn't — Python Bug Hunt

Inspired by the lost-update bugs behind countless "my changes disappeared" tickets.

  • Language: Python
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Concurrency, Versioning
  • Modelled on: CMS platforms
  • Visible tests: a matching version applies and bumps; a stale version is rejected
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-LOST-UPDATE
  • Reported by: CMS team
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a matching version applies and bumps — version must increment: expected 5 but got 4
  • a stale version is rejected — conflict must raise: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the lost-update bugs behind countless "my changes disappeared" tickets. Two editors load version 4; both save; the second silently erases the first. Optimistic locking exists precisely to stop this — if anyone actually checks the version.

optimistic.py doesn't.

Bug report

BUG-LOST-UPDATE · Priority: High · Reported by: CMS team

update_row(row, expected_version, changes):

  • row["version"] != expected_version -> raise ValueError("conflict")
  • otherwise apply changes and INCREMENT row["version"]

Observed: stale writers win, and version never moves off 1.

Logs

[cms] doc 88 saved by editor A (v4) then editor B (v4) -> A's edits gone

The code as shipped

src/store/optimistic.py (editable)

# Optimistic-concurrency row update.

def update_row(row, expected_version, changes):
    for key in changes:
        row[key] = changes[key]
    return row

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.