Zero Usage Meant Zero Quota

Medium

· Backend

Incident

SEV-1

BUG-GOOG1214

Priority: Critical · reported by identity SRE

Time to fix: 20 min

Failing checks on the shipped build

  • a stale report keeps the previous allocation

    missing data must never mean zero quota: expected 900 but got 0

  • a null report keeps the previous allocation

    Cannot read properties of null (reading 'usage')

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Google auth · 2020

Modelled on Google's authentication outage (14 December 2020): a storage migration left the user-ID service reporting zero usage. The automated quota system read that as "needs nothing" and cut its allocation to zero, so every service requiring a Google login failed for about 45 minutes.

quota.js sets an allocation straight from the reported usage, with no way to tell "genuinely idle" from "no data".

Fix computeQuota so a missing or stale report keeps the previous allocation rather than dropping to zero.

Code map

// Recomputes a service's quota allocation.
exports.computeQuota = function (report, previous, minimum) {
  return report.usage * 2;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Zero Usage Meant Zero Quota — JavaScript Bug Hunt

Modelled on Google's authentication outage (14 December 2020): a storage migration left the user-ID service reporting zero usage.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Quotas, Missing Data
  • Modelled on: Google auth · 2020
  • Visible tests: a live report doubles the usage; a stale report keeps the previous allocation; a null report keeps the previous allocation
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-GOOG1214
  • Reported by: identity SRE
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a stale report keeps the previous allocation — missing data must never mean zero quota: expected 900 but got 0
  • a null report keeps the previous allocation — Cannot read properties of null (reading 'usage')

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Google's authentication outage (14 December 2020): a storage migration left the user-ID service reporting zero usage. The automated quota system read that as "needs nothing" and cut its allocation to zero, so every service requiring a Google login failed for about 45 minutes.

quota.js sets an allocation straight from the reported usage, with no way to tell "genuinely idle" from "no data".

Fix computeQuota so a missing or stale report keeps the previous allocation rather than dropping to zero.

Bug report

BUG-GOOG1214 · Priority: Critical · Reported by: identity SRE

computeQuota(report, previous, minimum) must return:

  • previous, when report is missing/null or report.stale is true — an absent signal is not evidence of idleness
  • max(minimum, report.usage * 2) otherwise, so a live service always keeps a usable floor

Observed: a stale report with usage 0 sets the allocation to 0, and the service cannot serve a single request afterwards.

Logs

[quota] service=user-id allocation 0 (reported usage: 0, report age: 41h)
[quota] user-id rejecting 100% of requests: OUT_OF_QUOTA

The code as shipped

src/quota/quota.js (editable)

// Recomputes a service's quota allocation.
exports.computeQuota = function (report, previous, minimum) {
  return report.usage * 2;
};

Read-only context: src/quota/RULES.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.