Two Billion Events and One More — Java Bug Hunt

Briefing

Modelled on the Basecamp 3 outage of November 2018. The primary key of the events table — a row is written for nearly every action in the product — was a signed 32-bit integer. When the ids reached 2,147,483,647, the database refused new rows, and Basecamp 3 was effectively read-only for hours while the column was migrated to a 64-bit type.

In this reconstruction EventsSchema chooses the id column type and the threshold at which the capacity monitor pages someone. The locked EventsTable rejects an insert once the column's maximum is reached.

Fix EventsSchema so the table keeps accepting events and the monitor warns well before any ceiling.

Bug report

BUG-BC3-IDS · Priority: Critical · Reported by: ops

EventsSchema:

• idColumn() is the type of events.id; it must be a 64-bit column (IdColumn.BIGINT) so inserts continue past 2,147,483,647
• nearCapacity(lastId, column) must page when the column is at least 90% used: true when lastId >= column.maxValue - column.maxValue / 10 (integer division), false below that

Observed: insert after id 2147483647 throws "Out of range value for column 'id' (INT)"; the capacity monitor never fired before that moment.

Logs

[db] INSERT INTO events ... -> Out of range value for column 'id' (INT)
[app] 500 on every write since 2147483647
[monitor] events.id capacity check: OK

The code as shipped

src/db/EventsSchema.java (editable)

class EventsSchema {
    // events.id -- a row is appended for nearly every action in the product.
    static IdColumn idColumn() {
        return IdColumn.INT;
    }

    // Paging threshold for the id-capacity monitor.
    static boolean nearCapacity(long lastId, IdColumn column) {
        return lastId >= column.maxValue;
    }
}

Read-only context: src/db/EventsTable.java, src/db/IdColumn.java.