The Fsync That Lied on Retry

Hard

· Database

Incident

SEV-1

BUG-FSYNCGATE

Priority: Critical (silent data loss) · reported by storage team

Time to fix: 35 min

Failing checks on the shipped build

  • a failed fsync panics and is never retried

    a failed fsync must PANIC: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

PostgreSQL · fsyncgate 2018

Modelled on PostgreSQL's "fsyncgate" (2018): developers found that when fsync failed, Linux could report the error once, mark the dirty pages clean and drop them. PostgreSQL's checkpointer retried the fsync, the retry succeeded, and the data it believed durable had never reached disk. The fix, released in February 2019, was to treat a failed fsync as fatal: PANIC, and recover from the write-ahead log on restart.

This project is a reconstruction with a fake OS that behaves the way Linux did. checkpoint.py writes buffers, fsyncs, and truncates the WAL — retrying once when the fsync fails.

Fix checkpoint so a failed fsync is never retried and never trusted.

Code map

errors = bug_require("./errors.py")


def checkpoint(os_, buffers, wal):
    for page in sorted(buffers):
        os_.write(page, buffers[page])
    try:
        os_.fsync()
    except OSError:
        os_.fsync()
    flushed = len(wal["records"])
    wal["records"] = []
    wal["redo_lsn"] += flushed
    return flushed

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Fsync That Lied on Retry — Python Bug Hunt

Modelled on PostgreSQL's "fsyncgate" (2018): developers found that when fsync failed, Linux could report the error once, mark the dirty pages clean and drop…

  • Language: Python
  • Layer: Database
  • Difficulty: Hard
  • Concepts: Retries, State
  • Modelled on: PostgreSQL · fsyncgate 2018
  • Visible tests: a clean checkpoint makes pages durable and truncates the WAL; a failed fsync panics and is never retried
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (silent data loss))
  • Ticket: BUG-FSYNCGATE
  • Reported by: storage team
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a failed fsync panics and is never retried — a failed fsync must PANIC: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on PostgreSQL's "fsyncgate" (2018): developers found that when fsync failed, Linux could report the error once, mark the dirty pages clean and drop them. PostgreSQL's checkpointer retried the fsync, the retry succeeded, and the data it believed durable had never reached disk. The fix, released in February 2019, was to treat a failed fsync as fatal: PANIC, and recover from the write-ahead log on restart.

This project is a reconstruction with a fake OS that behaves the way Linux did. checkpoint.py writes buffers, fsyncs, and truncates the WAL — retrying once when the fsync fails.

Fix checkpoint so a failed fsync is never retried and never trusted.

Bug report

BUG-FSYNCGATE · Priority: Critical (silent data loss) · Reported by: storage team

checkpoint(os_, buffers, wal) — wal is {"records": [...], "redo_lsn": int}:

  • writes every buffer (sorted by page name) with os_.write, then calls os_.fsync() exactly once
  • on success: redo_lsn += number of records, records = [], return that number
  • on ANY failure of fsync: raise errors.Panic — do not call fsync again, and leave the WAL exactly as it was, so crash recovery (recovery.recover) can replay it

Observed: after an EIO the checkpoint retried, succeeded, truncated the WAL, and the pages were nowhere on disk.

Logs

[checkpointer] fsync failed: EIO, retrying
[checkpointer] fsync ok, checkpoint complete redo_lsn=412
[startup] page p1 missing, no WAL to replay

The code as shipped

src/storage/checkpoint.py (editable)

errors = bug_require("./errors.py")


def checkpoint(os_, buffers, wal):
    for page in sorted(buffers):
        os_.write(page, buffers[page])
    try:
        os_.fsync()
    except OSError:
        os_.fsync()
    flushed = len(wal["records"])
    wal["records"] = []
    wal["redo_lsn"] += flushed
    return flushed

Read-only context: src/storage/errors.py, src/storage/fakeos.py, src/storage/recovery.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.