The Two-Terabyte Wall — JavaScript Bug Hunt
Modelled on Instapaper's February 2017 outage: the service was down for well over a day because its MySQL database could no longer write to its bookmarks…
- Language: JavaScript
- Layer: Database
- Difficulty: Medium
- Concepts: Limits, Storage
- Modelled on: Instapaper · 2017
- Visible tests: a row goes into the active ext4 partition; a full ext3 partition rolls over
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (writes failing))
- Ticket: BUG-EFBIG
- Reported by: on-call DBA
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a full ext3 partition rolls over — EFBIG: bookmarks_0 would exceed the ext3 file size limit
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Instapaper's February 2017 outage: the service was down for well over a day because its MySQL database could no longer write to its bookmarks table. The table had been created before April 2014, so its data file lived on an ext3 filesystem with a 2 TB per-file limit; once the file reached that size, inserts failed. Recovery meant moving the data onto a new database instance.
planner.js chooses which partition file a new row goes into. It does roll over to a fresh partition before a file is full — but it measures "full" against one number for every file instead of the limit of the filesystem that file actually lives on.
Fix insert so no write ever pushes a file past its own filesystem's limit.
Bug report
BUG-EFBIG · Priority: Critical (writes failing) · Reported by: on-call DBA
store = { partitions: [{ name, fs, bytes }, ...] }; the last partition is the active one. limits.FS_FILE_LIMIT gives each filesystem's per-file limit.
insert(store, rowBytes):
- if the row fits in the active partition — bytes + rowBytes <= FS_FILE_LIMIT[active.fs]; reaching the limit exactly is fine — write it there
- otherwise first append a new partition { name: "bookmarks_" + <its index in the list>, fs: limits.NEW_PARTITION_FS, bytes: 0 } and write the row there
- a row too large for even a new partition throws, and nothing changes
- returns the name of the partition the row was written to
Observed: the active partition (ext3, created 2013) reached 2 TiB and every insert failed with EFBIG; the planner never rolled over because the file was nowhere near the 16 TiB table limit.
Logs
[mysql] ERROR 1114 (HY000): The table 'bookmarks' is full
[disk] EFBIG: bookmarks_0 would exceed the ext3 file size limit
[planner] active partition bookmarks_0 at 12.5% of MAX_TABLE_BYTESThe code as shipped
src/storage/planner.js (editable)
var disk = require("./disk");
var limits = require("./limits");
// Chooses the partition file a new row is written to.
exports.insert = function (store, rowBytes) {
var active = store.partitions[store.partitions.length - 1];
if (active.bytes + rowBytes > limits.MAX_TABLE_BYTES) {
active = { name: "bookmarks_" + store.partitions.length, fs: limits.NEW_PARTITION_FS, bytes: 0 };
store.partitions.push(active);
}
disk.write(active, rowBytes);
return active.name;
};
Read-only context: src/storage/disk.js, src/storage/limits.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.