One Domain Checked Three Times — JavaScript Bug Hunt
Modelled on the Let's Encrypt CAA rechecking bug (found 29 February 2020): when a certificate request covered several domain names whose CAA records needed…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Security, Concurrency, Validation
- Modelled on: Let's Encrypt · 2020
- Visible tests: a single-name order is checked; every name is checked once
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (mis-issuance))
- Ticket: BUG-CAA-2020
- Reported by: incident response
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- every name is checked once — expected ["a.example","b.example","c.example"] but got ["c.example","c.example","c.example"]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Let's Encrypt CAA rechecking bug (found 29 February 2020): when a certificate request covered several domain names whose CAA records needed rechecking, Boulder — Let's Encrypt's CA software — checked one of those names N times instead of each name once. The Go code captured a reference to the loop variable, so every check saw the same name. Let's Encrypt announced it would revoke about 3 million affected certificates.
caa.js reconstructs it in JavaScript: the rechecks are queued as closures inside a for loop and run afterwards by pool.js, the way Boulder fanned them out.
Fix recheck so every name on the order is checked exactly once.
Bug report
BUG-CAA-2020 · Priority: Critical (mis-issuance) · Reported by: incident response
recheck(domains, lookup) — lookup(domain) returns true when CAA permits issuance.
- lookup is called exactly once for each entry of domains, with that entry, in input order (duplicates in the input are each checked)
- returns { ok, failed }: failed lists the domains whose lookup returned false, in input order; ok is true exactly when failed is empty
- an empty list -> { ok: true, failed: [] }
Observed: for ["a.example", "b.example", "c.example"] lookup was called with "c.example" three times; a.example and b.example were never checked.
Logs
[boulder] recheckCAA order=4418 names=3
[boulder] caa lookup c.example ok
[boulder] caa lookup c.example ok
[boulder] caa lookup c.example okThe code as shipped
src/ca/caa.js (editable)
var pool = require("./pool");
// Re-checks CAA for every name on an order before issuance.
exports.recheck = function (domains, lookup) {
var tasks = [];
for (var i = 0; i < domains.length; i++) {
var domain = domains[i];
tasks.push(function () {
return { domain: domain, ok: lookup(domain) };
});
}
var results = pool.runAll(tasks);
var failed = results
.filter(function (r) { return !r.ok; })
.map(function (r) { return r.domain; });
return { ok: failed.length === 0, failed: failed };
};
Read-only context: src/ca/pool.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.