Two Billion Transactions Later — Java Bug Hunt
Modelled on Sentry's July 2015 outage, described in its post-mortem "Transaction ID Wraparound in Postgres".
- Language: Java
- Layer: Database
- Difficulty: Hard
- Concepts: Overflow, Time, Limits
- Modelled on: Sentry · 2015
- Visible tests: ordering of nearby ids; the monitor warns before the stop threshold
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (write outage))
- Ticket: BUG-SENTRY-XID
- Reported by: on-call
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- the monitor warns before the stop threshold — age 1.6 billion: expected WARN but got OK
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Sentry's July 2015 outage, described in its post-mortem "Transaction ID Wraparound in Postgres". PostgreSQL transaction IDs are 32-bit and compared circularly; when the oldest unfrozen transaction gets too far behind the current one, Postgres stops accepting writes to protect the data from wraparound. Sentry's primary database reached that point and writes stopped until the long vacuum work completed.
This project is a reconstruction of the monitoring side: XidMonitor compares transaction ids and reports how close the database is to that limit, using the thresholds in the locked XidLimits. It compares ids as plain numbers and never raises a warning before the stop threshold.
Fix precedes, age and status so they are circular and the monitor warns in time.
Bug report
BUG-SENTRY-XID · Priority: Critical (write outage) · Reported by: on-call
Transaction ids are unsigned 32-bit values in [0, 2^32) that wrap.
- precedes(a, b): a is older than b in circular order — true iff (a - b) mod 2^32, read as a signed 32-bit integer, is negative. precedes(4294967000, 100) is true; precedes(100, 4294967000) is false.
- age(current, oldest) = (current - oldest) mod 2^32, in [0, 2^32).
- status(current, oldest): "STOP" if age >= XidLimits.STOP_AGE, else "WARN" if age >= XidLimits.WARN_AGE, else "OK".
Observed: no warning ever fired, and after the counter wrapped status() reported "OK" for a database minutes away from refusing writes.
Logs
[pg-monitor] xid=852032704 oldest=3000000000 age=-2147967296 status=OK
postgres: ERROR: database is not accepting commands to avoid wraparound data loss in database "sentry"The code as shipped
src/pg/XidMonitor.java (editable)
class XidMonitor {
// True when transaction id a is older than b.
static boolean precedes(long a, long b) {
return a < b;
}
// How many transactions current is ahead of the oldest unfrozen one.
static long age(long current, long oldest) {
return current - oldest;
}
static String status(long current, long oldest) {
long age = age(current, oldest);
if (age >= XidLimits.STOP_AGE) return "STOP";
return "OK";
}
}Read-only context: src/pg/XidLimits.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.