The Blank Field That Crashed Every Region — Java Bug Hunt

Briefing

Modelled on the Google Cloud outage of 12 June 2025: Service Control, which checks API requests against policy and quota, had gained a new quota-policy check that shipped without appropriate error handling and without a feature flag. When a policy change containing unintended blank fields was replicated globally, that code path hit a null pointer and the binary went into a crash loop in every region, taking a long list of Google Cloud products down with it.

ServiceControl.check evaluates one quota policy against a request's usage. It assumes every field of the policy is populated.

Fix it so a policy with missing or blank fields is treated as no quota policy — the check fails open — instead of crashing.

Bug report

BUG-SVCCTL · Priority: Critical (global) · Reported by: SRE

check(policy, usage) returns "ALLOW" or "DENY:<metric>":

• policy null -> "ALLOW"
• a policy whose metric is null or blank (empty / whitespace only), or whose limit is null, is incomplete: treat it as no quota policy -> "ALLOW". Never throw.
• otherwise usage > limit -> "DENY:" + the trimmed metric, else "ALLOW" (usage equal to the limit is allowed)

admit(policies, usage) is true unless some policy's check denies. An incomplete policy in the list never throws and never blocks.

Observed: a replicated policy with blank fields crashed every instance with a NullPointerException, on every restart.

Logs

[service-control] applying policy update p-7781 (replicated)
[service-control] java.lang.NullPointerException at ServiceControl.check
[service-control] task restarted (crash loop, all regions)

The code as shipped

src/control/ServiceControl.java (editable)

class ServiceControl {
    static String check(QuotaPolicy policy, long usage) {
        if (policy == null) return "ALLOW";
        String metric = policy.metric.trim();
        long limit = policy.limit;
        return usage > limit ? "DENY:" + metric : "ALLOW";
    }

    static boolean admit(List<QuotaPolicy> policies, long usage) {
        for (QuotaPolicy p : policies) {
            if (check(p, usage).startsWith("DENY")) return false;
        }
        return true;
    }
}

Read-only context: src/control/QuotaPolicy.java.