The Key That Was Marked Keep — JavaScript Bug Hunt

Briefing

Modelled on the Microsoft Azure Active Directory outage of March 15, 2021. Microsoft's root-cause analysis says that during an automated rotation of signing keys, a key that had been marked for retention was removed because of a bug in the rotation process. Tokens signed with that key then failed validation, and sign-ins to many Microsoft and customer services failed for hours.

This project is a reconstruction: rotation.js builds the next key set when a new signing key is introduced, and validator.js accepts a token whose kid is still in the set. The rotation keeps only the key it is retiring and silently drops everything flagged retain.

Fix rotate so every key marked for retention survives the rotation.

Bug report

BUG-AAD-0315 · Priority: Critical (sign-in outage) · Reported by: identity SRE

rotate(keys, newKid) takes the current key set — an array of { kid, active, retain } — and returns the NEXT key set:

• every key with active: true is retired to { kid, active: false, retain: true } (tokens it signed are still in flight)
• every key with retain: true is kept exactly as it is
• a key that is neither active nor retained is dropped
• surviving keys keep their original order; the new key { kid: newKid, active: true, retain: false } is appended last
• the input array and its objects are not modified

Observed: after the 19:00 rotation, tokens signed with a key flagged retain fail validation — the key is gone from the published set.

Logs

[keys] rotate: introduced k-2103, retired k-2102
[keys] published set: k-2102, k-2103 (3 keys dropped)
[auth] token kid=k-2031 rejected: unknown signing key (x41,208/min)

The code as shipped

src/keys/rotation.js (editable)

// Builds the next signing-key set when a new key is introduced.
exports.rotate = function (keys, newKid) {
  var next = [];
  for (var i = 0; i < keys.length; i++) {
    var k = keys[i];
    if (k.active) {
      next.push({ kid: k.kid, active: false, retain: true });
    }
  }
  next.push({ kid: newKid, active: true, retain: false });
  return next;
};

Read-only context: src/keys/NOTES.js, src/keys/validator.js.