The Block Only Half the Network Accepted

Hard

· Backend

Incident

SEV-1

BUG-BIP50

Priority: Critical (consensus) · reported by core developers

Time to fix: 35 min

Failing checks on the shipped build

  • a big but legal block is accepted by both versions

    5998 locks is under the consensus limit: expected true but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Bitcoin · BIP 50

Modelled on the Bitcoin chain fork of 11 March 2013, documented in BIP 50. Version 0.8 had moved block storage from BerkeleyDB to LevelDB. A large block mined by a 0.8 node needed more database locks than 0.7 nodes' BerkeleyDB configuration allowed, so 0.7 nodes rejected a block that 0.8 nodes accepted, and the chain split in two. The fork was resolved within hours when miners moved back to 0.7-compatible software so the 0.7 side became the longest chain. The limit had never been a written rule — it was an accident of one version's storage layer — and BIP 50 records how later releases enforced a compatible limit explicitly for a period.

This reconstruction has two validators. The 0.7 one lets its local database lock limit decide whether a block is valid; the 0.8 one applies no limit at all.

Fix the validators so both apply the same explicit rule and can never disagree.

Code map

rules = bug_require("./rules.py")


def validate_v07(block, db_lock_limit):
    # 0.7 connects blocks through its local database; an update that needs
    # more locks than the database is configured for fails.
    if not rules.structurally_valid(block):
        return False
    if rules.locks_needed(block) > db_lock_limit:
        return False
    return True


def validate_v08(block):
    return rules.structurally_valid(block)


def chain_splits(blocks, db_lock_limit):
    return [b["hash"] for b in blocks
            if validate_v07(b, db_lock_limit) != validate_v08(b)]

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Block Only Half the Network Accepted — Python Bug Hunt

Modelled on the Bitcoin chain fork of 11 March 2013, documented in BIP 50. Version 0.8 had moved block storage from BerkeleyDB to LevelDB.

  • Language: Python
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Consensus, Limits
  • Modelled on: Bitcoin · BIP 50
  • Visible tests: a small block is valid everywhere; a big but legal block is accepted by both versions
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (consensus))
  • Ticket: BUG-BIP50
  • Reported by: core developers
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a big but legal block is accepted by both versions — 5998 locks is under the consensus limit: expected true but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Bitcoin chain fork of 11 March 2013, documented in BIP 50. Version 0.8 had moved block storage from BerkeleyDB to LevelDB. A large block mined by a 0.8 node needed more database locks than 0.7 nodes' BerkeleyDB configuration allowed, so 0.7 nodes rejected a block that 0.8 nodes accepted, and the chain split in two. The fork was resolved within hours when miners moved back to 0.7-compatible software so the 0.7 side became the longest chain. The limit had never been a written rule — it was an accident of one version's storage layer — and BIP 50 records how later releases enforced a compatible limit explicitly for a period.

This reconstruction has two validators. The 0.7 one lets its local database lock limit decide whether a block is valid; the 0.8 one applies no limit at all.

Fix the validators so both apply the same explicit rule and can never disagree.

Bug report

BUG-BIP50 · Priority: Critical (consensus) · Reported by: core developers

A block is valid iff rules.structurally_valid(block) AND rules.locks_needed(block) <= rules.MAX_BLOCK_LOCKS (10000, inclusive).

  • validate_v07(block, db_lock_limit) and validate_v08(block) must both return exactly that verdict. db_lock_limit is a node's local storage tuning; it must NOT influence validity.
  • chain_splits(blocks, db_lock_limit) -> hashes of the blocks on which the two versions disagree, in input order (after the fix: always []).

Observed: a block needing 5,998 locks was accepted by 0.8 nodes and rejected by 0.7 nodes configured with 5,000 locks; a block needing 10,001 locks is accepted by 0.8.

Logs

[v0.7] block 0000…a1f2 rejected: database lock limit exceeded
[v0.8] block 0000…a1f2 accepted height=225430
[net] two chains: 0.8 tip=225431, 0.7 tip=225429

The code as shipped

src/chain/validators.py (editable)

rules = bug_require("./rules.py")


def validate_v07(block, db_lock_limit):
    # 0.7 connects blocks through its local database; an update that needs
    # more locks than the database is configured for fails.
    if not rules.structurally_valid(block):
        return False
    if rules.locks_needed(block) > db_lock_limit:
        return False
    return True


def validate_v08(block):
    return rules.structurally_valid(block)


def chain_splits(blocks, db_lock_limit):
    return [b["hash"] for b in blocks
            if validate_v07(b, db_lock_limit) != validate_v08(b)]

Read-only context: src/chain/rules.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.