Cache Stampede at Dawn

Hard

· Database

Incident

SEV-1

BUG-9977

Priority: Critical · reported by DBA on-call

Time to fix: 35 min

Failing checks on the shipped build

  • the second read of a key is served from cache

    identical reads within TTL must hit the db once: expected 1 but got 2

  • invalidate removes only the targeted key

    only report:1 should be evicted: expected 1 but got 0

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Analytics at scale

Every morning at 06:00, the Horizon Analytics database gets hammered with identical queries — the cache layer in front of it has quietly stopped caching. Worse, invalidating one report wipes the entire cache.

The mock database (with a call counter) and the raw cache store are locked. All three bugs live in cachedRepo.js.

Code map

// Read-through cache in front of the analytics database.
var db = require("./mockDb");
var store = require("./cacheStore");

var TTL_MS = 60000;

exports.get = function (key, nowMs) {
  var entry = store.get(key);
  if (entry) {
    if (nowMs - entry.storedAt < TTL_MS) {
      return entry.value;
    }
    return entry.value; // expired entries are cheap to serve
  }

  var value = db.load(key);
  return value;
};

exports.invalidate = function (key) {
  store.clear();
};

exports.resetAll = function () {
  store.clear();
  db.resetCalls();
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Cache Stampede at Dawn — JavaScript Bug Hunt

Every morning at 06:00, the Horizon Analytics database gets hammered with identical queries — the cache layer in front of it has quietly stopped caching.

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Hard
  • Modelled on: Analytics at scale
  • Visible tests: the second read of a key is served from cache; an expired entry is re-fetched; invalidate removes only the targeted key
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-9977
  • Reported by: DBA on-call
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the second read of a key is served from cache — identical reads within TTL must hit the db once: expected 1 but got 2
  • invalidate removes only the targeted key — only report:1 should be evicted: expected 1 but got 0

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Every morning at 06:00, the Horizon Analytics database gets hammered with identical queries — the cache layer in front of it has quietly stopped caching. Worse, invalidating one report wipes the entire cache.

The mock database (with a call counter) and the raw cache store are locked. All three bugs live in cachedRepo.js.

Bug report

BUG-9977 · Priority: Critical · Reported by: DBA on-call

With TTL = 60s:

  • Two immediate reads of the same key hit the database TWICE (should be once).
  • Reads of EXPIRED entries are served from cache forever — stale dashboards.
  • invalidate("report:42") clears every key, causing the 06:00 stampede.

Contract: get(key, nowMs) -> value, using db.load(key) on a miss.

Logs

[db] load report:42   (call #1)
[db] load report:42   (call #2)   <- cache miss again?!
[cache] invalidate report:42 -> store size 0 (was 31)

The code as shipped

src/db/cachedRepo.js (editable)

// Read-through cache in front of the analytics database.
var db = require("./mockDb");
var store = require("./cacheStore");

var TTL_MS = 60000;

exports.get = function (key, nowMs) {
  var entry = store.get(key);
  if (entry) {
    if (nowMs - entry.storedAt < TTL_MS) {
      return entry.value;
    }
    return entry.value; // expired entries are cheap to serve
  }

  var value = db.load(key);
  return value;
};

exports.invalidate = function (key) {
  store.clear();
};

exports.resetAll = function () {
  store.clear();
  db.resetCalls();
};

Read-only context: src/db/cacheStore.js, src/db/mockDb.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.