Rate Limiter Free-for-All

Hard

· Backend

Incident

SEV-1

BUG-7010

Priority: Critical · reported by SRE

Time to fix: 35 min

Failing checks on the shipped build

  • the fourth request inside the window is throttled

    4th request within 10s must be blocked: expected 429 but got 200

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Public API platforms

The public API of Relay Notify is supposed to allow 3 requests per 10 seconds per client. Under load testing, one client got 40 requests through — and separate clients are mysteriously throttling each other.

The clock and the app wiring are locked. The sliding-window logic in rateLimiter.js has (at least) two independent bugs.

Code map

// Sliding-window rate limiter: LIMIT requests per WINDOW_MS per client.
var LIMIT = 3;
var WINDOW_MS = 10000;

var hits = [];

exports.allow = function (clientId, nowMs) {
  // Drop hits that have left the window
  hits = hits.filter(function (h) {
    return h.at > nowMs - WINDOW_MS && h.at !== nowMs - WINDOW_MS;
  });

  hits.push({ client: clientId, at: nowMs });

  var count = 0;
  for (var i = 0; i < hits.length; i++) {
    count++;
  }
  return count <= LIMIT + 1;
};

exports.resetAll = function () {
  hits = [];
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Rate Limiter Free-for-All — JavaScript Bug Hunt

The public API of Relay Notify is supposed to allow 3 requests per 10 seconds per client.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Modelled on: Public API platforms
  • Visible tests: the first three requests pass; the fourth request inside the window is throttled; clients are limited independently
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-7010
  • Reported by: SRE
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the fourth request inside the window is throttled — 4th request within 10s must be blocked: expected 429 but got 200

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

The public API of Relay Notify is supposed to allow 3 requests per 10 seconds per client. Under load testing, one client got 40 requests through — and separate clients are mysteriously throttling each other.

The clock and the app wiring are locked. The sliding-window logic in rateLimiter.js has (at least) two independent bugs.

Bug report

BUG-7010 · Priority: Critical · Reported by: SRE

Observed with limit=3, windowMs=10000:

  • client A fires 5 requests instantly -> ALL 5 allowed (expected 3)
  • client A then gets blocked... and so does client B, who sent nothing
  • after waiting 10+ seconds, clients stay blocked longer than they should

Contract: allow(clientId, nowMs) -> true/false.

Logs

[limit] allow A t=0     -> true
[limit] allow A t=1     -> true
[limit] allow A t=2     -> true
[limit] allow A t=3     -> true    <- should be false
[limit] allow B t=4     -> false   <- B never sent anything before!

The code as shipped

src/mw/rateLimiter.js (editable)

// Sliding-window rate limiter: LIMIT requests per WINDOW_MS per client.
var LIMIT = 3;
var WINDOW_MS = 10000;

var hits = [];

exports.allow = function (clientId, nowMs) {
  // Drop hits that have left the window
  hits = hits.filter(function (h) {
    return h.at > nowMs - WINDOW_MS && h.at !== nowMs - WINDOW_MS;
  });

  hits.push({ client: clientId, at: nowMs });

  var count = 0;
  for (var i = 0; i < hits.length; i++) {
    count++;
  }
  return count <= LIMIT + 1;
};

exports.resetAll = function () {
  hits = [];
};

Read-only context: src/mw/app.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.