The Key That Was Marked Keep

Medium

· Backend

Incident

SEV-1

BUG-AAD-0315

Priority: Critical (sign-in outage) · reported by identity SRE

Time to fix: 20 min

Failing checks on the shipped build

  • a key marked retain survives rotation

    k0 was marked for retention: expected [{"kid":"k0","active":false,"retain":true},{"kid":"k1","active":false,"retain":true},{"kid":"k2","active":true,"retain":false}] but got [{"kid":"k1","active":false,"retain":true},{"kid":"k2","active":true,"retain":false}]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Microsoft Azure AD · 2021

Modelled on the Microsoft Azure Active Directory outage of March 15, 2021. Microsoft's root-cause analysis says that during an automated rotation of signing keys, a key that had been marked for retention was removed because of a bug in the rotation process. Tokens signed with that key then failed validation, and sign-ins to many Microsoft and customer services failed for hours.

This project is a reconstruction: rotation.js builds the next key set when a new signing key is introduced, and validator.js accepts a token whose kid is still in the set. The rotation keeps only the key it is retiring and silently drops everything flagged retain.

Fix rotate so every key marked for retention survives the rotation.

Code map

// Builds the next signing-key set when a new key is introduced.
exports.rotate = function (keys, newKid) {
  var next = [];
  for (var i = 0; i < keys.length; i++) {
    var k = keys[i];
    if (k.active) {
      next.push({ kid: k.kid, active: false, retain: true });
    }
  }
  next.push({ kid: newKid, active: true, retain: false });
  return next;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Key That Was Marked Keep — JavaScript Bug Hunt

Modelled on the Microsoft Azure Active Directory outage of March 15, 2021.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Auth, Config, State
  • Modelled on: Microsoft Azure AD · 2021
  • Visible tests: the new key becomes the active one; a key marked retain survives rotation
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (sign-in outage))
  • Ticket: BUG-AAD-0315
  • Reported by: identity SRE
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a key marked retain survives rotation — k0 was marked for retention: expected [{"kid":"k0","active":false,"retain":true},{"kid":"k1","active":false,"retain":true},{"kid":"k2","active":true,"retain":false}] but got [{"kid":"k1","active":false,"retain":true},{"kid":"k2","active":true,"retain":false}]

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Microsoft Azure Active Directory outage of March 15, 2021. Microsoft's root-cause analysis says that during an automated rotation of signing keys, a key that had been marked for retention was removed because of a bug in the rotation process. Tokens signed with that key then failed validation, and sign-ins to many Microsoft and customer services failed for hours.

This project is a reconstruction: rotation.js builds the next key set when a new signing key is introduced, and validator.js accepts a token whose kid is still in the set. The rotation keeps only the key it is retiring and silently drops everything flagged retain.

Fix rotate so every key marked for retention survives the rotation.

Bug report

BUG-AAD-0315 · Priority: Critical (sign-in outage) · Reported by: identity SRE

rotate(keys, newKid) takes the current key set — an array of { kid, active, retain } — and returns the NEXT key set:

  • every key with active: true is retired to { kid, active: false, retain: true } (tokens it signed are still in flight)
  • every key with retain: true is kept exactly as it is
  • a key that is neither active nor retained is dropped
  • surviving keys keep their original order; the new key { kid: newKid, active: true, retain: false } is appended last
  • the input array and its objects are not modified

Observed: after the 19:00 rotation, tokens signed with a key flagged retain fail validation — the key is gone from the published set.

Logs

[keys] rotate: introduced k-2103, retired k-2102
[keys] published set: k-2102, k-2103 (3 keys dropped)
[auth] token kid=k-2031 rejected: unknown signing key (x41,208/min)

The code as shipped

src/keys/rotation.js (editable)

// Builds the next signing-key set when a new key is introduced.
exports.rotate = function (keys, newKid) {
  var next = [];
  for (var i = 0; i < keys.length; i++) {
    var k = keys[i];
    if (k.active) {
      next.push({ kid: k.kid, active: false, retain: true });
    }
  }
  next.push({ kid: newKid, active: true, retain: false });
  return next;
};

Read-only context: src/keys/NOTES.js, src/keys/validator.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.