The Two-Terabyte Wall — JavaScript Bug Hunt

Briefing

Modelled on Instapaper's February 2017 outage: the service was down for well over a day because its MySQL database could no longer write to its bookmarks table. The table had been created before April 2014, so its data file lived on an ext3 filesystem with a 2 TB per-file limit; once the file reached that size, inserts failed. Recovery meant moving the data onto a new database instance.

planner.js chooses which partition file a new row goes into. It does roll over to a fresh partition before a file is full — but it measures "full" against one number for every file instead of the limit of the filesystem that file actually lives on.

Fix insert so no write ever pushes a file past its own filesystem's limit.

Bug report

BUG-EFBIG · Priority: Critical (writes failing) · Reported by: on-call DBA

store = { partitions: [{ name, fs, bytes }, ...] }; the last partition is the active one. limits.FS_FILE_LIMIT gives each filesystem's per-file limit.

insert(store, rowBytes):

• if the row fits in the active partition — bytes + rowBytes <= FS_FILE_LIMIT[active.fs]; reaching the limit exactly is fine — write it there
• otherwise first append a new partition { name: "bookmarks_" + <its index in the list>, fs: limits.NEW_PARTITION_FS, bytes: 0 } and write the row there
• a row too large for even a new partition throws, and nothing changes
• returns the name of the partition the row was written to

Observed: the active partition (ext3, created 2013) reached 2 TiB and every insert failed with EFBIG; the planner never rolled over because the file was nowhere near the 16 TiB table limit.

Logs

[mysql] ERROR 1114 (HY000): The table 'bookmarks' is full
[disk] EFBIG: bookmarks_0 would exceed the ext3 file size limit
[planner] active partition bookmarks_0 at 12.5% of MAX_TABLE_BYTES

The code as shipped

src/storage/planner.js (editable)

var disk = require("./disk");
var limits = require("./limits");

// Chooses the partition file a new row is written to.
exports.insert = function (store, rowBytes) {
  var active = store.partitions[store.partitions.length - 1];
  if (active.bytes + rowBytes > limits.MAX_TABLE_BYTES) {
    active = { name: "bookmarks_" + store.partitions.length, fs: limits.NEW_PARTITION_FS, bytes: 0 };
    store.partitions.push(active);
  }
  disk.write(active, rowBytes);
  return active.name;
};

Read-only context: src/storage/disk.js, src/storage/limits.js.