The Storage Certificate That Lapsed — Java Bug Hunt
Modelled on the Windows Azure Storage outage of 22 February 2013.
- Language: Java
- Layer: Backend
- Difficulty: Medium
- Concepts: Time, Overflow, Security
- Modelled on: Microsoft Azure · 2013
- Visible tests: a certificate with a year left is OK; ten days before expiry the monitor asks for renewal
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-WAS-CERT
- Reported by: storage front-end on-call
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- ten days before expiry the monitor asks for renewal — expected RENEW but got OK
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Windows Azure Storage outage of 22 February 2013. The SSL certificate used for HTTPS traffic to Azure Storage expired, and HTTPS requests to storage failed worldwide for many hours, taking services that depended on storage down with them. HTTP traffic was unaffected; the fix was deploying a renewed certificate.
This reconstruction is a certificate monitor that is supposed to flag any certificate within 30 days of expiry. It compares a remaining time in milliseconds with a window computed in seconds, so it only notices a certificate in the last three-quarters of an hour of its life — or once it has already expired.
Fix CertMonitor so certificates are flagged for renewal a full 30 days ahead.
Bug report
BUG-WAS-CERT · Priority: Critical · Reported by: storage front-end on-call
CertMonitor.status(cert, nowMillis), all times in epoch milliseconds:
- "EXPIRED" when nowMillis >= cert.notAfterMillis
- "RENEW" when the time remaining (notAfterMillis - nowMillis) is at most RENEW_WINDOW_DAYS (30) days
- "OK" otherwise
dueForRenewal(certs, nowMillis) returns the hosts of every certificate that is not "OK", sorted alphabetically.
Observed: a certificate with 10 days left reported "OK" every day until it reported "EXPIRED".
Logs
[certmon] *.blob.core.windows.net remaining=864000000ms status=OK
[certmon] *.blob.core.windows.net remaining=0ms status=EXPIRED
[fe] TLS handshake failed: certificate has expiredThe code as shipped
src/tls/CertMonitor.java (editable)
class CertMonitor {
static final int RENEW_WINDOW_DAYS = 30;
static String status(Certificate cert, long nowMillis) {
long remaining = cert.notAfterMillis - nowMillis;
if (remaining <= 0) return "EXPIRED";
if (remaining <= RENEW_WINDOW_DAYS * 24 * 60 * 60) return "RENEW";
return "OK";
}
static List<String> dueForRenewal(List<Certificate> certs, long nowMillis) {
List<String> hosts = new ArrayList<>();
for (Certificate c : certs) {
if (!status(c, nowMillis).equals("OK")) hosts.add(c.host);
}
Collections.sort(hosts);
return hosts;
}
}
Read-only context: src/tls/Certificate.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.