The Half-Applied Batch

Medium

· Database

Incident

SEV-1

BUG-ATOMIC

Priority: Critical · reported by finance close

Time to fix: 20 min

Failing checks on the shipped build

  • a failing batch leaves the ledger untouched

    atomicity!: expected {"cash": 100, "rev": 0} but got {"cash": 60, "rev": 40}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Ledger systems

Inspired by every ledger that ever ended a day unbalanced: a batch of postings hits an invalid entry halfway through, the job aborts — and the first half of the batch stays applied. Transactions exist so that either everything happens or nothing does.

postings.py mutates as it goes.

Code map

# Applies a batch of postings to the ledger (acct -> balance).

def apply_batch(ledger, ops):
    for op in ops:
        acct = op["acct"]
        if acct not in ledger:
            raise ValueError("unknown account: " + acct)
        new_balance = ledger[acct] + op["delta"]
        if new_balance < 0:
            raise ValueError("overdraft on " + acct)
        ledger[acct] = new_balance
    return ledger

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Half-Applied Batch — Python Bug Hunt

Inspired by every ledger that ever ended a day unbalanced: a batch of postings hits an invalid entry halfway through, the job aborts — and the first half of…

  • Language: Python
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Transactions, Atomicity
  • Modelled on: Ledger systems
  • Visible tests: a valid batch applies fully; a failing batch leaves the ledger untouched
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-ATOMIC
  • Reported by: finance close
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a failing batch leaves the ledger untouched — atomicity!: expected {"cash": 100, "rev": 0} but got {"cash": 60, "rev": 40}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by every ledger that ever ended a day unbalanced: a batch of postings hits an invalid entry halfway through, the job aborts — and the first half of the batch stays applied. Transactions exist so that either everything happens or nothing does.

postings.py mutates as it goes.

Bug report

BUG-ATOMIC · Priority: Critical · Reported by: finance close

apply_batch(ledger, ops) — ops are {"acct", "delta"}:

  • if ANY op references a missing account, or would take any balance below zero, raise ValueError and leave the ledger COMPLETELY unchanged
  • otherwise apply all ops

Observed: a failing 3rd op leaves ops 1–2 applied. The books never balance.

Logs

[ledger] batch failed at op 3/5; partial state persisted

The code as shipped

src/ledger/postings.py (editable)

# Applies a batch of postings to the ledger (acct -> balance).

def apply_batch(ledger, ops):
    for op in ops:
        acct = op["acct"]
        if acct not in ledger:
            raise ValueError("unknown account: " + acct)
        new_balance = ledger[acct] + op["delta"]
        if new_balance < 0:
            raise ValueError("overdraft on " + acct)
        ledger[acct] = new_balance
    return ledger

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.