The Distribution List That Replied to Itself — JavaScript Bug Hunt

Briefing

Modelled on Microsoft's "Bedlam DL3" incident of October 1997. A message went to an internal distribution list with a very large membership. Recipients hit Reply All — many of them to ask to be removed from the list, others to tell everyone to stop replying — and every reply went to the whole list again. The cascade of messages overwhelmed Microsoft's Exchange servers, and "Bedlam DL3" became the Exchange team's shorthand for a reply-all storm.

This reconstruction's send.js expands the list and delivers, every time.

Fix send so reply-all to a huge list is blocked and a repeating storm on one thread is throttled.

Bug report

BUG-BEDLAM · Priority: High · Reported by: messaging ops

send(msg, directory, history, now) — msg = { to, subject, replyAll }:

• recipients = lists.expand(directory, msg.to)
• msg.replyAll and recipients.length > REPLY_ALL_MAX_RECIPIENTS (limits.js) -> { ok: false, error: "reply_all_blocked", delivered: 0 }; exactly the limit is allowed; a message that is not a reply-all is never blocked by this
• storm throttle: normalise a subject by trimming, removing any number of leading "RE:" prefixes (any case, optional spaces) and lower-casing. If history already holds STORM_THRESHOLD or more entries with the same to and the same normalised subject whose age now - at is < STORM_WINDOW_MS -> { ok: false, error: "throttled", delivered: 0 }
• a refused message is not recorded
• otherwise push { to, subject: <normalised>, at: now } onto history and return { ok: true, error: null, delivered: recipients.length }

Observed: thousands of "RE: RE: please remove me" messages to the whole list within minutes.

Logs

[exchange] queue depth 1,204,331 (bedlam-dl3)
[exchange] RE: RE: RE: remove me -> 600 recipients
[exchange] store.exe: message rate exceeds capacity

The code as shipped

src/mail/send.js (editable)

var lists = require("./lists");

exports.send = function (msg, directory, history, now) {
  var recipients = lists.expand(directory, msg.to);
  history.push({ to: msg.to, subject: msg.subject, at: now });
  return { ok: true, error: null, delivered: recipients.length };
};

Read-only context: src/mail/limits.js, src/mail/lists.js.