The Certificate Nobody Renewed

Medium

· Backend

Incident

SEV-1

BUG-CERT-1206

Priority: Critical · reported by network operations

Time to fix: 20 min

Failing checks on the shipped build

  • ten days left is ten days, and due for renewal

    days left: expected 10 but got 240

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Ericsson · O2 UK 2018

Modelled on the Ericsson outage of 6 December 2018: an expired certificate in Ericsson's SGSN-MME core-network software took mobile data down for millions of customers of O2 in the UK and SoftBank in Japan, among other operators, for most of a day. Ericsson said the cause was an expired certificate in the software versions installed with those customers.

This project is a reconstruction of the safeguard that should have caught it: a renewal check that flags a certificate once it enters its renewal window. CertRenewal.java computes how many days a certificate has left — and the renewal alert never fires.

Fix CertRenewal so the days-to-expiry figure, and everything built on it, follows the spec.

Code map

class CertRenewal {
    // Whole days before the certificate expires; negative once it has.
    static long daysUntilExpiry(long nowMillis, long notAfterMillis) {
        return (notAfterMillis - nowMillis) / CertPolicy.MILLIS_PER_HOUR;
    }

    static boolean shouldRenew(long nowMillis, long notAfterMillis) {
        return daysUntilExpiry(nowMillis, notAfterMillis) <= CertPolicy.RENEW_WINDOW_DAYS;
    }

    static String status(long nowMillis, long notAfterMillis) {
        long days = daysUntilExpiry(nowMillis, notAfterMillis);
        if (days < 0) return "EXPIRED";
        if (days <= CertPolicy.RENEW_WINDOW_DAYS) return "RENEW";
        return "OK";
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Certificate Nobody Renewed — Java Bug Hunt

Modelled on the Ericsson outage of 6 December 2018: an expired certificate in Ericsson's SGSN-MME core-network software took mobile data down for millions…

  • Language: Java
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Time, Config, Security
  • Modelled on: Ericsson · O2 UK 2018
  • Visible tests: a certificate with 100 days left is fine; ten days left is ten days, and due for renewal
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-CERT-1206
  • Reported by: network operations
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • ten days left is ten days, and due for renewal — days left: expected 10 but got 240

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Ericsson outage of 6 December 2018: an expired certificate in Ericsson's SGSN-MME core-network software took mobile data down for millions of customers of O2 in the UK and SoftBank in Japan, among other operators, for most of a day. Ericsson said the cause was an expired certificate in the software versions installed with those customers.

This project is a reconstruction of the safeguard that should have caught it: a renewal check that flags a certificate once it enters its renewal window. CertRenewal.java computes how many days a certificate has left — and the renewal alert never fires.

Fix CertRenewal so the days-to-expiry figure, and everything built on it, follows the spec.

Bug report

BUG-CERT-1206 · Priority: Critical · Reported by: network operations

All times are epoch milliseconds (long).

daysUntilExpiry(now, notAfter)

  • whole days between now and notAfter, rounded toward negative infinity: floor((notAfter - now) / CertPolicy.MILLIS_PER_DAY)
  • 10 days left -> 10; 29 days 23 hours left -> 29; 1 ms past expiry -> -1

shouldRenew(now, notAfter)

  • true when daysUntilExpiry <= CertPolicy.RENEW_WINDOW_DAYS (30), including after expiry

status(now, notAfter)

  • "EXPIRED" when daysUntilExpiry < 0
  • "RENEW" when 0 <= daysUntilExpiry <= 30
  • "OK" otherwise

Observed: a certificate with 10 days left reports 240 days, status "OK"; nobody is paged until the node refuses to start.

Logs

[certwatch] sgsn-mme-07 cert notAfter=2018-12-06T00:00Z days_left=240 status=OK
[certwatch] no renewals due
[sgsn-mme-07] TLS handshake failed: certificate has expired

The code as shipped

CertRenewal.java (editable)

class CertRenewal {
    // Whole days before the certificate expires; negative once it has.
    static long daysUntilExpiry(long nowMillis, long notAfterMillis) {
        return (notAfterMillis - nowMillis) / CertPolicy.MILLIS_PER_HOUR;
    }

    static boolean shouldRenew(long nowMillis, long notAfterMillis) {
        return daysUntilExpiry(nowMillis, notAfterMillis) <= CertPolicy.RENEW_WINDOW_DAYS;
    }

    static String status(long nowMillis, long notAfterMillis) {
        long days = daysUntilExpiry(nowMillis, notAfterMillis);
        if (days < 0) return "EXPIRED";
        if (days <= CertPolicy.RENEW_WINDOW_DAYS) return "RENEW";
        return "OK";
    }
}

Read-only context: CertPolicy.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.