The Intermediate Nobody Was Watching — JavaScript Bug Hunt
Modelled on Mozilla Firefox, May 2019 ("Armagadd-on"): an intermediate certificate in the chain Mozilla used to sign Firefox add-ons expired.
- Language: JavaScript
- Layer: Backend
- Difficulty: Hard
- Concepts: Security, Time, Monitoring
- Modelled on: Mozilla Firefox · 2019
- Visible tests: a leaf inside the window is reported; an intermediate inside the window is reported
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-ARMAGADDON
- Reported by: release engineering
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- an intermediate inside the window is reported — the intermediate is five days from lapsing: expected [{"serial":"I1","subject":"signing-ca","daysLeft":5}] but got []
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Mozilla Firefox, May 2019 ("Armagadd-on"): an intermediate certificate in the chain Mozilla used to sign Firefox add-ons expired. Firefox verifies every add-on's signature, so once that certificate lapsed nearly every installed extension was disabled at once. Mozilla pushed a replacement certificate through its Studies system and shipped it in Firefox 66.0.4.
This project is a reconstruction of the safety net that should have fired first: an expiry monitor that walks the signing chains and reports what is about to lapse. It only ever looks at one certificate per chain.
Fix expiringCerts so every certificate in every chain is watched.
Bug report
BUG-ARMAGADDON · Priority: Critical · Reported by: release engineering
expiringCerts(chains, nowMs, windowDays) — each chain is an array of certs ordered leaf first, then intermediates, then the root. A cert is { serial, subject, notAfter } (notAfter in epoch ms).
It must report EVERY certificate in EVERY chain (leaf, intermediates and root) whose daysLeft <= windowDays, where daysLeft = time.daysBetween(nowMs, notAfter) (floored whole days; negative once the cert has already expired — an expired cert is still reported).
- each report is { serial, subject, daysLeft } (in that key order)
- a certificate shared by several chains is reported once (by serial)
- sorted by daysLeft ascending, ties by serial ascending
Observed: an intermediate five days from expiry produced no alert at all; only leaf certificates ever show up in the report.
Logs
[certwatch] scanned 3 chains, 0 certificates inside the 30-day window
[addons] signature verification failed: certificate expired (intermediate "signing-ca")
[addons] disabled 41 of 42 installed extensionsThe code as shipped
src/certs/monitor.js (editable)
var time = require("./time");
// Scans every signing chain we ship and reports certificates that expire
// inside the alert window, soonest first.
exports.expiringCerts = function (chains, nowMs, windowDays) {
var seen = {};
var out = [];
for (var i = 0; i < chains.length; i++) {
var cert = chains[i][0];
if (!cert || seen[cert.serial]) continue;
seen[cert.serial] = true;
var daysLeft = time.daysBetween(nowMs, cert.notAfter);
if (daysLeft <= windowDays) {
out.push({ serial: cert.serial, subject: cert.subject, daysLeft: daysLeft });
}
}
out.sort(function (a, b) {
if (a.daysLeft !== b.daysLeft) return a.daysLeft - b.daysLeft;
return a.serial < b.serial ? -1 : a.serial > b.serial ? 1 : 0;
});
return out;
};
Read-only context: src/certs/CHAINS.js, src/certs/time.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.