Everyone Tried to Fix It at Once

Hard

· Backend

Incident

SEV-1

BUG-FB2010

Priority: Critical · reported by site reliability

Time to fix: 35 min

Failing checks on the shipped build

  • an invalid value queries once and gives up

    one failed lookup must not become a retry storm: expected 1 but got 100001

  • a negative result is remembered

    the failure must be cached: expected 0 but got 100001

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Facebook · 2010

Modelled on the Facebook outage of 23 September 2010: a configuration value was judged invalid, so every client cleared its cache and queried the database for a correct value — which was still the invalid one. Hundreds of thousands of clients hammered the database in a self-sustaining loop, and the site had to be taken offline to break it.

config.js has the same shape: a failed lookup invalidates the cache and retries without limit.

Fix resolve so a persistently invalid value is cached as a negative result instead of driving unbounded traffic.

Code map

function valid(v) {
  return v !== null && v !== "invalid";
}

exports.resolve = function (cache, db, key) {
  var queries = 0;
  while (true) {
    if (cache.hasOwnProperty(key) && valid(cache[key])) {
      return { value: cache[key], queries: queries };
    }
    // The cached value looks wrong — throw it away and ask the source again.
    delete cache[key];
    queries++;
    cache[key] = db.read(key);
    if (queries > 100000) return { value: null, queries: queries };
  }
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Everyone Tried to Fix It at Once — JavaScript Bug Hunt

Modelled on the Facebook outage of 23 September 2010: a configuration value was judged invalid, so every client cleared its cache and queried the database…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Feedback Loops, Caching
  • Modelled on: Facebook · 2010
  • Visible tests: a valid cached value costs no queries; an invalid value queries once and gives up; a negative result is remembered
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-FB2010
  • Reported by: site reliability
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • an invalid value queries once and gives up — one failed lookup must not become a retry storm: expected 1 but got 100001
  • a negative result is remembered — the failure must be cached: expected 0 but got 100001

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Facebook outage of 23 September 2010: a configuration value was judged invalid, so every client cleared its cache and queried the database for a correct value — which was still the invalid one. Hundreds of thousands of clients hammered the database in a self-sustaining loop, and the site had to be taken offline to break it.

config.js has the same shape: a failed lookup invalidates the cache and retries without limit.

Fix resolve so a persistently invalid value is cached as a negative result instead of driving unbounded traffic.

Bug report

BUG-FB2010 · Priority: Critical · Reported by: site reliability

resolve(cache, db, key) must:

  • return the cached value when present and valid
  • otherwise query the database at most ONCE per call
  • when the database also returns an invalid value, remember that (a negative cache entry) and return null
  • return { value, queries } so the caller can see the database load

Observed: an invalid value makes the function clear the cache and loop, issuing thousands of queries for a single lookup.

Logs

[config] key="threshold" queries=98211 value=invalid
[config] database at 100% CPU; site taken offline to break the loop

The code as shipped

src/config/config.js (editable)

function valid(v) {
  return v !== null && v !== "invalid";
}

exports.resolve = function (cache, db, key) {
  var queries = 0;
  while (true) {
    if (cache.hasOwnProperty(key) && valid(cache[key])) {
      return { value: cache[key], queries: queries };
    }
    // The cached value looks wrong — throw it away and ask the source again.
    delete cache[key];
    queries++;
    cache[key] = db.read(key);
    if (queries > 100000) return { value: null, queries: queries };
  }
};

Read-only context: src/config/NOTES.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.