The Storage Certificate That Lapsed — Java Bug Hunt

Briefing

Modelled on the Windows Azure Storage outage of 22 February 2013. The SSL certificate used for HTTPS traffic to Azure Storage expired, and HTTPS requests to storage failed worldwide for many hours, taking services that depended on storage down with them. HTTP traffic was unaffected; the fix was deploying a renewed certificate.

This reconstruction is a certificate monitor that is supposed to flag any certificate within 30 days of expiry. It compares a remaining time in milliseconds with a window computed in seconds, so it only notices a certificate in the last three-quarters of an hour of its life — or once it has already expired.

Fix CertMonitor so certificates are flagged for renewal a full 30 days ahead.

Bug report

BUG-WAS-CERT · Priority: Critical · Reported by: storage front-end on-call

CertMonitor.status(cert, nowMillis), all times in epoch milliseconds:

• "EXPIRED" when nowMillis >= cert.notAfterMillis
• "RENEW" when the time remaining (notAfterMillis - nowMillis) is at most RENEW_WINDOW_DAYS (30) days
• "OK" otherwise

dueForRenewal(certs, nowMillis) returns the hosts of every certificate that is not "OK", sorted alphabetically.

Observed: a certificate with 10 days left reported "OK" every day until it reported "EXPIRED".

Logs

[certmon] *.blob.core.windows.net remaining=864000000ms status=OK
[certmon] *.blob.core.windows.net remaining=0ms status=EXPIRED
[fe] TLS handshake failed: certificate has expired

The code as shipped

src/tls/CertMonitor.java (editable)

class CertMonitor {
    static final int RENEW_WINDOW_DAYS = 30;

    static String status(Certificate cert, long nowMillis) {
        long remaining = cert.notAfterMillis - nowMillis;
        if (remaining <= 0) return "EXPIRED";
        if (remaining <= RENEW_WINDOW_DAYS * 24 * 60 * 60) return "RENEW";
        return "OK";
    }

    static List<String> dueForRenewal(List<Certificate> certs, long nowMillis) {
        List<String> hosts = new ArrayList<>();
        for (Certificate c : certs) {
            if (!status(c, nowMillis).equals("OK")) hosts.add(c.host);
        }
        Collections.sort(hosts);
        return hosts;
    }
}

Read-only context: src/tls/Certificate.java.