Zero Usage Meant Zero Quota — JavaScript Bug Hunt

Briefing

Modelled on Google's authentication outage (14 December 2020): a storage migration left the user-ID service reporting zero usage. The automated quota system read that as "needs nothing" and cut its allocation to zero, so every service requiring a Google login failed for about 45 minutes.

quota.js sets an allocation straight from the reported usage, with no way to tell "genuinely idle" from "no data".

Fix computeQuota so a missing or stale report keeps the previous allocation rather than dropping to zero.

Bug report

BUG-GOOG1214 · Priority: Critical · Reported by: identity SRE

computeQuota(report, previous, minimum) must return:

• previous, when report is missing/null or report.stale is true — an absent signal is not evidence of idleness
• max(minimum, report.usage * 2) otherwise, so a live service always keeps a usable floor

Observed: a stale report with usage 0 sets the allocation to 0, and the service cannot serve a single request afterwards.

Logs

[quota] service=user-id allocation 0 (reported usage: 0, report age: 41h)
[quota] user-id rejecting 100% of requests: OUT_OF_QUOTA

The code as shipped

src/quota/quota.js (editable)

// Recomputes a service's quota allocation.
exports.computeQuota = function (report, previous, minimum) {
  return report.usage * 2;
};

Read-only context: src/quota/RULES.js.