One Domain Checked Three Times

Medium

· Backend

Incident

SEV-1

BUG-CAA-2020

Priority: Critical (mis-issuance) · reported by incident response

Time to fix: 20 min

Failing checks on the shipped build

  • every name is checked once

    expected ["a.example","b.example","c.example"] but got ["c.example","c.example","c.example"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Let's Encrypt · 2020

Modelled on the Let's Encrypt CAA rechecking bug (found 29 February 2020): when a certificate request covered several domain names whose CAA records needed rechecking, Boulder — Let's Encrypt's CA software — checked one of those names N times instead of each name once. The Go code captured a reference to the loop variable, so every check saw the same name. Let's Encrypt announced it would revoke about 3 million affected certificates.

caa.js reconstructs it in JavaScript: the rechecks are queued as closures inside a for loop and run afterwards by pool.js, the way Boulder fanned them out.

Fix recheck so every name on the order is checked exactly once.

Code map

var pool = require("./pool");

// Re-checks CAA for every name on an order before issuance.
exports.recheck = function (domains, lookup) {
  var tasks = [];
  for (var i = 0; i < domains.length; i++) {
    var domain = domains[i];
    tasks.push(function () {
      return { domain: domain, ok: lookup(domain) };
    });
  }
  var results = pool.runAll(tasks);
  var failed = results
    .filter(function (r) { return !r.ok; })
    .map(function (r) { return r.domain; });
  return { ok: failed.length === 0, failed: failed };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

One Domain Checked Three Times — JavaScript Bug Hunt

Modelled on the Let's Encrypt CAA rechecking bug (found 29 February 2020): when a certificate request covered several domain names whose CAA records needed…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Security, Concurrency, Validation
  • Modelled on: Let's Encrypt · 2020
  • Visible tests: a single-name order is checked; every name is checked once
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (mis-issuance))
  • Ticket: BUG-CAA-2020
  • Reported by: incident response
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • every name is checked once — expected ["a.example","b.example","c.example"] but got ["c.example","c.example","c.example"]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Let's Encrypt CAA rechecking bug (found 29 February 2020): when a certificate request covered several domain names whose CAA records needed rechecking, Boulder — Let's Encrypt's CA software — checked one of those names N times instead of each name once. The Go code captured a reference to the loop variable, so every check saw the same name. Let's Encrypt announced it would revoke about 3 million affected certificates.

caa.js reconstructs it in JavaScript: the rechecks are queued as closures inside a for loop and run afterwards by pool.js, the way Boulder fanned them out.

Fix recheck so every name on the order is checked exactly once.

Bug report

BUG-CAA-2020 · Priority: Critical (mis-issuance) · Reported by: incident response

recheck(domains, lookup) — lookup(domain) returns true when CAA permits issuance.

  • lookup is called exactly once for each entry of domains, with that entry, in input order (duplicates in the input are each checked)
  • returns { ok, failed }: failed lists the domains whose lookup returned false, in input order; ok is true exactly when failed is empty
  • an empty list -> { ok: true, failed: [] }

Observed: for ["a.example", "b.example", "c.example"] lookup was called with "c.example" three times; a.example and b.example were never checked.

Logs

[boulder] recheckCAA order=4418 names=3
[boulder] caa lookup c.example ok
[boulder] caa lookup c.example ok
[boulder] caa lookup c.example ok

The code as shipped

src/ca/caa.js (editable)

var pool = require("./pool");

// Re-checks CAA for every name on an order before issuance.
exports.recheck = function (domains, lookup) {
  var tasks = [];
  for (var i = 0; i < domains.length; i++) {
    var domain = domains[i];
    tasks.push(function () {
      return { domain: domain, ok: lookup(domain) };
    });
  }
  var results = pool.runAll(tasks);
  var failed = results
    .filter(function (r) { return !r.ok; })
    .map(function (r) { return r.domain; });
  return { ok: failed.length === 0, failed: failed };
};

Read-only context: src/ca/pool.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.