The Blank Field That Crashed Every Region

Medium

· Backend

Incident

SEV-1

BUG-SVCCTL

Priority: Critical (global) · reported by SRE

Time to fix: 20 min

Failing checks on the shipped build

  • a policy with blank fields does not crash

    Cannot invoke "String.trim()" because "<parameter1>.metric" is null

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Google Cloud · 2025

Modelled on the Google Cloud outage of 12 June 2025: Service Control, which checks API requests against policy and quota, had gained a new quota-policy check that shipped without appropriate error handling and without a feature flag. When a policy change containing unintended blank fields was replicated globally, that code path hit a null pointer and the binary went into a crash loop in every region, taking a long list of Google Cloud products down with it.

ServiceControl.check evaluates one quota policy against a request's usage. It assumes every field of the policy is populated.

Fix it so a policy with missing or blank fields is treated as no quota policy — the check fails open — instead of crashing.

Code map

class ServiceControl {
    static String check(QuotaPolicy policy, long usage) {
        if (policy == null) return "ALLOW";
        String metric = policy.metric.trim();
        long limit = policy.limit;
        return usage > limit ? "DENY:" + metric : "ALLOW";
    }

    static boolean admit(List<QuotaPolicy> policies, long usage) {
        for (QuotaPolicy p : policies) {
            if (check(p, usage).startsWith("DENY")) return false;
        }
        return true;
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Blank Field That Crashed Every Region — Java Bug Hunt

Modelled on the Google Cloud outage of 12 June 2025: Service Control, which checks API requests against policy and quota, had gained a new quota-policy…

  • Language: Java
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Validation, Config
  • Modelled on: Google Cloud · 2025
  • Visible tests: a complete policy denies over its limit; a policy with blank fields does not crash
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (global))
  • Ticket: BUG-SVCCTL
  • Reported by: SRE
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a policy with blank fields does not crash — Cannot invoke "String.trim()" because "<parameter1>.metric" is null

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Google Cloud outage of 12 June 2025: Service Control, which checks API requests against policy and quota, had gained a new quota-policy check that shipped without appropriate error handling and without a feature flag. When a policy change containing unintended blank fields was replicated globally, that code path hit a null pointer and the binary went into a crash loop in every region, taking a long list of Google Cloud products down with it.

ServiceControl.check evaluates one quota policy against a request's usage. It assumes every field of the policy is populated.

Fix it so a policy with missing or blank fields is treated as no quota policy — the check fails open — instead of crashing.

Bug report

BUG-SVCCTL · Priority: Critical (global) · Reported by: SRE

check(policy, usage) returns "ALLOW" or "DENY:<metric>":

  • policy null -> "ALLOW"
  • a policy whose metric is null or blank (empty / whitespace only), or whose limit is null, is incomplete: treat it as no quota policy -> "ALLOW". Never throw.
  • otherwise usage > limit -> "DENY:" + the trimmed metric, else "ALLOW" (usage equal to the limit is allowed)

admit(policies, usage) is true unless some policy's check denies. An incomplete policy in the list never throws and never blocks.

Observed: a replicated policy with blank fields crashed every instance with a NullPointerException, on every restart.

Logs

[service-control] applying policy update p-7781 (replicated)
[service-control] java.lang.NullPointerException at ServiceControl.check
[service-control] task restarted (crash loop, all regions)

The code as shipped

src/control/ServiceControl.java (editable)

class ServiceControl {
    static String check(QuotaPolicy policy, long usage) {
        if (policy == null) return "ALLOW";
        String metric = policy.metric.trim();
        long limit = policy.limit;
        return usage > limit ? "DENY:" + metric : "ALLOW";
    }

    static boolean admit(List<QuotaPolicy> policies, long usage) {
        for (QuotaPolicy p : policies) {
            if (check(p, usage).startsWith("DENY")) return false;
        }
        return true;
    }
}

Read-only context: src/control/QuotaPolicy.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.