The Key That Was Marked Keep — JavaScript Bug Hunt
Modelled on the Microsoft Azure Active Directory outage of March 15, 2021.
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Auth, Config, State
- Modelled on: Microsoft Azure AD · 2021
- Visible tests: the new key becomes the active one; a key marked retain survives rotation
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (sign-in outage))
- Ticket: BUG-AAD-0315
- Reported by: identity SRE
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a key marked retain survives rotation — k0 was marked for retention: expected [{"kid":"k0","active":false,"retain":true},{"kid":"k1","active":false,"retain":true},{"kid":"k2","active":true,"retain":false}] but got [{"kid":"k1","active":false,"retain":true},{"kid":"k2","active":true,"retain":false}]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Microsoft Azure Active Directory outage of March 15, 2021. Microsoft's root-cause analysis says that during an automated rotation of signing keys, a key that had been marked for retention was removed because of a bug in the rotation process. Tokens signed with that key then failed validation, and sign-ins to many Microsoft and customer services failed for hours.
This project is a reconstruction: rotation.js builds the next key set when a new signing key is introduced, and validator.js accepts a token whose kid is still in the set. The rotation keeps only the key it is retiring and silently drops everything flagged retain.
Fix rotate so every key marked for retention survives the rotation.
Bug report
BUG-AAD-0315 · Priority: Critical (sign-in outage) · Reported by: identity SRE
rotate(keys, newKid) takes the current key set — an array of { kid, active, retain } — and returns the NEXT key set:
- every key with active: true is retired to { kid, active: false, retain: true } (tokens it signed are still in flight)
- every key with retain: true is kept exactly as it is
- a key that is neither active nor retained is dropped
- surviving keys keep their original order; the new key { kid: newKid, active: true, retain: false } is appended last
- the input array and its objects are not modified
Observed: after the 19:00 rotation, tokens signed with a key flagged retain fail validation — the key is gone from the published set.
Logs
[keys] rotate: introduced k-2103, retired k-2102
[keys] published set: k-2102, k-2103 (3 keys dropped)
[auth] token kid=k-2031 rejected: unknown signing key (x41,208/min)The code as shipped
src/keys/rotation.js (editable)
// Builds the next signing-key set when a new key is introduced.
exports.rotate = function (keys, newKid) {
var next = [];
for (var i = 0; i < keys.length; i++) {
var k = keys[i];
if (k.active) {
next.push({ kid: k.kid, active: false, retain: true });
}
}
next.push({ kid: newKid, active: true, retain: false });
return next;
};
Read-only context: src/keys/NOTES.js, src/keys/validator.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.