Two Billion Events and One More — Java Bug Hunt
Modelled on the Basecamp 3 outage of November 2018. The primary key of the events table — a row is written for nearly every action in the product — was a…
- Language: Java
- Layer: Database
- Difficulty: Easy
- Concepts: Overflow, SQL, Limits
- Modelled on: Basecamp · 2018
- Visible tests: the last int-sized id can still be written; inserts continue past 2,147,483,647
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-BC3-IDS
- Reported by: ops
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- inserts continue past 2,147,483,647 — Out of range value for column 'id' (INT)
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Basecamp 3 outage of November 2018. The primary key of the events table — a row is written for nearly every action in the product — was a signed 32-bit integer. When the ids reached 2,147,483,647, the database refused new rows, and Basecamp 3 was effectively read-only for hours while the column was migrated to a 64-bit type.
In this reconstruction EventsSchema chooses the id column type and the threshold at which the capacity monitor pages someone. The locked EventsTable rejects an insert once the column's maximum is reached.
Fix EventsSchema so the table keeps accepting events and the monitor warns well before any ceiling.
Bug report
BUG-BC3-IDS · Priority: Critical · Reported by: ops
EventsSchema:
- idColumn() is the type of events.id; it must be a 64-bit column (IdColumn.BIGINT) so inserts continue past 2,147,483,647
- nearCapacity(lastId, column) must page when the column is at least 90% used: true when lastId >= column.maxValue - column.maxValue / 10 (integer division), false below that
Observed: insert after id 2147483647 throws "Out of range value for column 'id' (INT)"; the capacity monitor never fired before that moment.
Logs
[db] INSERT INTO events ... -> Out of range value for column 'id' (INT)
[app] 500 on every write since 2147483647
[monitor] events.id capacity check: OKThe code as shipped
src/db/EventsSchema.java (editable)
class EventsSchema {
// events.id -- a row is appended for nearly every action in the product.
static IdColumn idColumn() {
return IdColumn.INT;
}
// Paging threshold for the id-capacity monitor.
static boolean nearCapacity(long lastId, IdColumn column) {
return lastId >= column.maxValue;
}
}
Read-only context: src/db/EventsTable.java, src/db/IdColumn.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.