One Typo, Half the Internet — JavaScript Bug Hunt

Briefing

Inspired by the 2017 AWS S3 outage: an engineer running a routine playbook mistyped one number and removed far more capacity than intended — taking a chunk of the internet down with it.

The tooling should have refused. Make decommission.js refuse.

Bug report

BUG-S3-0228 · Priority: Critical · Reported by: incident review

planRemoval(requestedCount, pool) rules:

• requestedCount must parse as a positive integer, else throw "invalid count"
• at least MIN_ALIVE (2) servers must remain: clamp the removal to pool.length - 2
• removing from a pool of 2 or fewer removes nothing

Observed: "1O0" (letter O) parsed as 1, "100" removed 100 of 10, pool hit zero.

Logs

[decom] pool=10 requested=100 removed=10 remaining=0
[decom] index-fleet offline; GET /* -> 500

The code as shipped

src/ops/decommission.js (editable)

var MIN_ALIVE = 2;

// Plans a capacity removal. Returns the servers that remain.
exports.planRemoval = function (requestedCount, pool) {
  var count = parseInt(requestedCount, 10);
  return pool.slice(0, pool.length - count);
};