The Player That Wore Out Disks

Hard

· Database

Incident

SEV-2

BUG-SSD-WRITES

Priority: High · reported by community forum

Time to fix: 35 min

Failing checks on the shipped build

  • small changes are appended, not rewritten

    no full rewrites for 50 changes: expected 0 but got 50

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Spotify · 2016

Modelled on the Spotify desktop client disk-write reports of November 2016: users noticed the app writing tens to hundreds of gigabytes a day to disk even while idle — a real concern for SSD lifespan. The writes were traced to the client's local database being rewritten over and over; Spotify shipped a fix in an update later that month.

store.js is a reconstruction: a small persistent key/value cache that rewrites its entire snapshot file on every change. The file system is injected (fakefs.js counts full rewrites and appends), so you can see exactly what it does.

Rewrite store.js to log changes by appending and only compact into a snapshot after enough changes have accumulated, exactly as the spec describes.

Code map

var config = require("./config");

exports.open = function (fs) {
  var state = {};
  var snap = fs.readFile(config.SNAPSHOT);
  if (snap) state = JSON.parse(snap);

  function persist() {
    fs.writeFile(config.SNAPSHOT, JSON.stringify(state));
  }

  return {
    get: function (key) {
      return Object.prototype.hasOwnProperty.call(state, key) ? state[key] : null;
    },
    set: function (key, value) {
      state[key] = value;
      persist();
    }
  };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Player That Wore Out Disks — JavaScript Bug Hunt

Modelled on the Spotify desktop client disk-write reports of November 2016: users noticed the app writing tens to hundreds of gigabytes a day to disk even…

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Hard
  • Concepts: Caching, Limits, State
  • Modelled on: Spotify · 2016
  • Visible tests: values survive a reopen; small changes are appended, not rewritten
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-SSD-WRITES
  • Reported by: community forum
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • small changes are appended, not rewritten — no full rewrites for 50 changes: expected 0 but got 50

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Spotify desktop client disk-write reports of November 2016: users noticed the app writing tens to hundreds of gigabytes a day to disk even while idle — a real concern for SSD lifespan. The writes were traced to the client's local database being rewritten over and over; Spotify shipped a fix in an update later that month.

store.js is a reconstruction: a small persistent key/value cache that rewrites its entire snapshot file on every change. The file system is injected (fakefs.js counts full rewrites and appends), so you can see exactly what it does.

Rewrite store.js to log changes by appending and only compact into a snapshot after enough changes have accumulated, exactly as the spec describes.

Bug report

BUG-SSD-WRITES · Priority: High · Reported by: community forum

open(fs) -> { get, set }. Paths and limits are in config.js.

Loading: state = JSON.parse(fs.readFile(SNAPSHOT)) if it exists (else {}), then every non-empty line of fs.readFile(LOG) (if any) is a JSON [key, value] applied in order. The count of changes since the last compaction starts at the number of log lines replayed.

get(key) -> the value, or null if never set.

set(key, value) (values are strings, numbers or booleans):

  • if state already holds exactly that value (===) -> no I/O at all
  • otherwise update state and fs.appendFile(LOG, JSON.stringify([key, value]) + "\n") and count one change
  • when the count reaches COMPACT_AFTER, compact: fs.writeFile(SNAPSHOT, JSON.stringify(state)) then fs.writeFile(LOG, "") and reset the count to 0

So 50 changes = 50 appends and 0 rewrites; 100 changes = 100 appends and 2 rewrites (snapshot + log truncate); 250 changes = 4 rewrites.

Observed: every single set rewrites the full snapshot.

Logs

[storage] state.json rewritten (4.1 MB) reason=set
[storage] state.json rewritten (4.1 MB) reason=set
[disk] 81 GB written in the last 24h by the desktop client

The code as shipped

src/cache/store.js (editable)

var config = require("./config");

exports.open = function (fs) {
  var state = {};
  var snap = fs.readFile(config.SNAPSHOT);
  if (snap) state = JSON.parse(snap);

  function persist() {
    fs.writeFile(config.SNAPSHOT, JSON.stringify(state));
  }

  return {
    get: function (key) {
      return Object.prototype.hasOwnProperty.call(state, key) ? state[key] : null;
    },
    set: function (key, value) {
      state[key] = value;
      persist();
    }
  };
};

Read-only context: src/cache/config.js, src/cache/fakefs.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.