Cache Stampede at Dawn — JavaScript Bug Hunt
Every morning at 06:00, the Horizon Analytics database gets hammered with identical queries — the cache layer in front of it has quietly stopped caching.
- Language: JavaScript
- Layer: Database
- Difficulty: Hard
- Modelled on: Analytics at scale
- Visible tests: the second read of a key is served from cache; an expired entry is re-fetched; invalidate removes only the targeted key
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-9977
- Reported by: DBA on-call
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- the second read of a key is served from cache — identical reads within TTL must hit the db once: expected 1 but got 2
- invalidate removes only the targeted key — only report:1 should be evicted: expected 1 but got 0
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Every morning at 06:00, the Horizon Analytics database gets hammered with identical queries — the cache layer in front of it has quietly stopped caching. Worse, invalidating one report wipes the entire cache.
The mock database (with a call counter) and the raw cache store are locked. All three bugs live in cachedRepo.js.
Bug report
BUG-9977 · Priority: Critical · Reported by: DBA on-call
With TTL = 60s:
- Two immediate reads of the same key hit the database TWICE (should be once).
- Reads of EXPIRED entries are served from cache forever — stale dashboards.
- invalidate("report:42") clears every key, causing the 06:00 stampede.
Contract: get(key, nowMs) -> value, using db.load(key) on a miss.
Logs
[db] load report:42 (call #1)
[db] load report:42 (call #2) <- cache miss again?!
[cache] invalidate report:42 -> store size 0 (was 31)The code as shipped
src/db/cachedRepo.js (editable)
// Read-through cache in front of the analytics database.
var db = require("./mockDb");
var store = require("./cacheStore");
var TTL_MS = 60000;
exports.get = function (key, nowMs) {
var entry = store.get(key);
if (entry) {
if (nowMs - entry.storedAt < TTL_MS) {
return entry.value;
}
return entry.value; // expired entries are cheap to serve
}
var value = db.load(key);
return value;
};
exports.invalidate = function (key) {
store.clear();
};
exports.resetAll = function () {
store.clear();
db.resetCalls();
};
Read-only context: src/db/cacheStore.js, src/db/mockDb.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.