Everyone Tried to Fix It at Once — JavaScript Bug Hunt

Briefing

Modelled on the Facebook outage of 23 September 2010: a configuration value was judged invalid, so every client cleared its cache and queried the database for a correct value — which was still the invalid one. Hundreds of thousands of clients hammered the database in a self-sustaining loop, and the site had to be taken offline to break it.

config.js has the same shape: a failed lookup invalidates the cache and retries without limit.

Fix resolve so a persistently invalid value is cached as a negative result instead of driving unbounded traffic.

Bug report

BUG-FB2010 · Priority: Critical · Reported by: site reliability

resolve(cache, db, key) must:

• return the cached value when present and valid
• otherwise query the database at most ONCE per call
• when the database also returns an invalid value, remember that (a negative cache entry) and return null
• return { value, queries } so the caller can see the database load

Observed: an invalid value makes the function clear the cache and loop, issuing thousands of queries for a single lookup.

Logs

[config] key="threshold" queries=98211 value=invalid
[config] database at 100% CPU; site taken offline to break the loop

The code as shipped

src/config/config.js (editable)

function valid(v) {
  return v !== null && v !== "invalid";
}

exports.resolve = function (cache, db, key) {
  var queries = 0;
  while (true) {
    if (cache.hasOwnProperty(key) && valid(cache[key])) {
      return { value: cache[key], queries: queries };
    }
    // The cached value looks wrong — throw it away and ask the source again.
    delete cache[key];
    queries++;
    cache[key] = db.read(key);
    if (queries > 100000) return { value: null, queries: queries };
  }
};

Read-only context: src/config/NOTES.js.