Zero Usage Meant Zero Quota — JavaScript Bug Hunt
Modelled on Google's authentication outage (14 December 2020): a storage migration left the user-ID service reporting zero usage.
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Quotas, Missing Data
- Modelled on: Google auth · 2020
- Visible tests: a live report doubles the usage; a stale report keeps the previous allocation; a null report keeps the previous allocation
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-GOOG1214
- Reported by: identity SRE
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a stale report keeps the previous allocation — missing data must never mean zero quota: expected 900 but got 0
- a null report keeps the previous allocation — Cannot read properties of null (reading 'usage')
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Google's authentication outage (14 December 2020): a storage migration left the user-ID service reporting zero usage. The automated quota system read that as "needs nothing" and cut its allocation to zero, so every service requiring a Google login failed for about 45 minutes.
quota.js sets an allocation straight from the reported usage, with no way to tell "genuinely idle" from "no data".
Fix computeQuota so a missing or stale report keeps the previous allocation rather than dropping to zero.
Bug report
BUG-GOOG1214 · Priority: Critical · Reported by: identity SRE
computeQuota(report, previous, minimum) must return:
- previous, when report is missing/null or report.stale is true — an absent signal is not evidence of idleness
- max(minimum, report.usage * 2) otherwise, so a live service always keeps a usable floor
Observed: a stale report with usage 0 sets the allocation to 0, and the service cannot serve a single request afterwards.
Logs
[quota] service=user-id allocation 0 (reported usage: 0, report age: 41h)
[quota] user-id rejecting 100% of requests: OUT_OF_QUOTAThe code as shipped
src/quota/quota.js (editable)
// Recomputes a service's quota allocation.
exports.computeQuota = function (report, previous, minimum) {
return report.usage * 2;
};
Read-only context: src/quota/RULES.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.