Two Billion Transactions Later — Java Bug Hunt

Briefing

Modelled on Sentry's July 2015 outage, described in its post-mortem "Transaction ID Wraparound in Postgres". PostgreSQL transaction IDs are 32-bit and compared circularly; when the oldest unfrozen transaction gets too far behind the current one, Postgres stops accepting writes to protect the data from wraparound. Sentry's primary database reached that point and writes stopped until the long vacuum work completed.

This project is a reconstruction of the monitoring side: XidMonitor compares transaction ids and reports how close the database is to that limit, using the thresholds in the locked XidLimits. It compares ids as plain numbers and never raises a warning before the stop threshold.

Fix precedes, age and status so they are circular and the monitor warns in time.

Bug report

BUG-SENTRY-XID · Priority: Critical (write outage) · Reported by: on-call

Transaction ids are unsigned 32-bit values in [0, 2^32) that wrap.

• precedes(a, b): a is older than b in circular order — true iff (a - b) mod 2^32, read as a signed 32-bit integer, is negative. precedes(4294967000, 100) is true; precedes(100, 4294967000) is false.
• age(current, oldest) = (current - oldest) mod 2^32, in [0, 2^32).
• status(current, oldest): "STOP" if age >= XidLimits.STOP_AGE, else "WARN" if age >= XidLimits.WARN_AGE, else "OK".

Observed: no warning ever fired, and after the counter wrapped status() reported "OK" for a database minutes away from refusing writes.

Logs

[pg-monitor] xid=852032704 oldest=3000000000 age=-2147967296 status=OK
postgres: ERROR: database is not accepting commands to avoid wraparound data loss in database "sentry"

The code as shipped

src/pg/XidMonitor.java (editable)

class XidMonitor {
    // True when transaction id a is older than b.
    static boolean precedes(long a, long b) {
        return a < b;
    }

    // How many transactions current is ahead of the oldest unfrozen one.
    static long age(long current, long oldest) {
        return current - oldest;
    }

    static String status(long current, long oldest) {
        long age = age(current, oldest);
        if (age >= XidLimits.STOP_AGE) return "STOP";
        return "OK";
    }
}

Read-only context: src/pg/XidLimits.java.