The Certificate Nobody Renewed — Java Bug Hunt

Briefing

Modelled on the Ericsson outage of 6 December 2018: an expired certificate in Ericsson's SGSN-MME core-network software took mobile data down for millions of customers of O2 in the UK and SoftBank in Japan, among other operators, for most of a day. Ericsson said the cause was an expired certificate in the software versions installed with those customers.

This project is a reconstruction of the safeguard that should have caught it: a renewal check that flags a certificate once it enters its renewal window. CertRenewal.java computes how many days a certificate has left — and the renewal alert never fires.

Fix CertRenewal so the days-to-expiry figure, and everything built on it, follows the spec.

Bug report

BUG-CERT-1206 · Priority: Critical · Reported by: network operations

All times are epoch milliseconds (long).

daysUntilExpiry(now, notAfter)

• whole days between now and notAfter, rounded toward negative infinity: floor((notAfter - now) / CertPolicy.MILLIS_PER_DAY)
• 10 days left -> 10; 29 days 23 hours left -> 29; 1 ms past expiry -> -1

shouldRenew(now, notAfter)

• true when daysUntilExpiry <= CertPolicy.RENEW_WINDOW_DAYS (30), including after expiry

status(now, notAfter)

• "EXPIRED" when daysUntilExpiry < 0
• "RENEW" when 0 <= daysUntilExpiry <= 30
• "OK" otherwise

Observed: a certificate with 10 days left reports 240 days, status "OK"; nobody is paged until the node refuses to start.

Logs

[certwatch] sgsn-mme-07 cert notAfter=2018-12-06T00:00Z days_left=240 status=OK
[certwatch] no renewals due
[sgsn-mme-07] TLS handshake failed: certificate has expired

The code as shipped

CertRenewal.java (editable)

class CertRenewal {
    // Whole days before the certificate expires; negative once it has.
    static long daysUntilExpiry(long nowMillis, long notAfterMillis) {
        return (notAfterMillis - nowMillis) / CertPolicy.MILLIS_PER_HOUR;
    }

    static boolean shouldRenew(long nowMillis, long notAfterMillis) {
        return daysUntilExpiry(nowMillis, notAfterMillis) <= CertPolicy.RENEW_WINDOW_DAYS;
    }

    static String status(long nowMillis, long notAfterMillis) {
        long days = daysUntilExpiry(nowMillis, notAfterMillis);
        if (days < 0) return "EXPIRED";
        if (days <= CertPolicy.RENEW_WINDOW_DAYS) return "RENEW";
        return "OK";
    }
}

Read-only context: CertPolicy.java.