The Distribution List That Replied to Itself — Bug Hunt
Modelled on Microsoft's "Bedlam DL3" incident of October 1997. A message went to an internal distribution list with a very large membership.
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Limits, Email, Resilience
- Modelled on: Microsoft · Bedlam DL3 1997
- Visible tests: an ordinary message is delivered; reply-all to a huge list is blocked
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-2 (priority: High)
- Ticket: BUG-BEDLAM
- Reported by: messaging ops
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- reply-all to a huge list is blocked — expected {"ok":false,"error":"reply_all_blocked","delivered":0} but got {"ok":true,"error":null,"delivered":600}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Microsoft's "Bedlam DL3" incident of October 1997. A message went to an internal distribution list with a very large membership. Recipients hit Reply All — many of them to ask to be removed from the list, others to tell everyone to stop replying — and every reply went to the whole list again. The cascade of messages overwhelmed Microsoft's Exchange servers, and "Bedlam DL3" became the Exchange team's shorthand for a reply-all storm.
This reconstruction's send.js expands the list and delivers, every time.
Fix send so reply-all to a huge list is blocked and a repeating storm on one thread is throttled.
Bug report
BUG-BEDLAM · Priority: High · Reported by: messaging ops
send(msg, directory, history, now) — msg = { to, subject, replyAll }:
- recipients = lists.expand(directory, msg.to)
- msg.replyAll and recipients.length > REPLY_ALL_MAX_RECIPIENTS (limits.js) -> { ok: false, error: "reply_all_blocked", delivered: 0 }; exactly the limit is allowed; a message that is not a reply-all is never blocked by this
- storm throttle: normalise a subject by trimming, removing any number of leading "RE:" prefixes (any case, optional spaces) and lower-casing. If history already holds STORM_THRESHOLD or more entries with the same
toand the same normalised subject whose age now - at is < STORM_WINDOW_MS -> { ok: false, error: "throttled", delivered: 0 } - a refused message is not recorded
- otherwise push { to, subject: <normalised>, at: now } onto history and return { ok: true, error: null, delivered: recipients.length }
Observed: thousands of "RE: RE: please remove me" messages to the whole list within minutes.
Logs
[exchange] queue depth 1,204,331 (bedlam-dl3)
[exchange] RE: RE: RE: remove me -> 600 recipients
[exchange] store.exe: message rate exceeds capacityThe code as shipped
src/mail/send.js (editable)
var lists = require("./lists");
exports.send = function (msg, directory, history, now) {
var recipients = lists.expand(directory, msg.to);
history.push({ to: msg.to, subject: msg.subject, at: now });
return { ok: true, error: null, delivered: recipients.length };
};
Read-only context: src/mail/limits.js, src/mail/lists.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.