One Typo, Half the Internet

Medium

· Backend

Incident

SEV-1

BUG-S3-0228

Priority: Critical · reported by incident review

Time to fix: 20 min

Failing checks on the shipped build

  • an oversized request is clamped to keep MIN_ALIVE

    never below MIN_ALIVE: expected 2 but got 0

  • garbage input throws instead of guessing

    letter O is not a number: expected an error to be thrown

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

AWS S3 · 2017

Inspired by the 2017 AWS S3 outage: an engineer running a routine playbook mistyped one number and removed far more capacity than intended — taking a chunk of the internet down with it.

The tooling should have refused. Make decommission.js refuse.

Code map

var MIN_ALIVE = 2;

// Plans a capacity removal. Returns the servers that remain.
exports.planRemoval = function (requestedCount, pool) {
  var count = parseInt(requestedCount, 10);
  return pool.slice(0, pool.length - count);
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

One Typo, Half the Internet — JavaScript Bug Hunt

Inspired by the 2017 AWS S3 outage: an engineer running a routine playbook mistyped one number and removed far more capacity than intended — taking a chunk…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Validation, Operations
  • Modelled on: AWS S3 · 2017
  • Visible tests: a sane removal removes exactly that many; an oversized request is clamped to keep MIN_ALIVE; garbage input throws instead of guessing
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-S3-0228
  • Reported by: incident review
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • an oversized request is clamped to keep MIN_ALIVE — never below MIN_ALIVE: expected 2 but got 0
  • garbage input throws instead of guessing — letter O is not a number: expected an error to be thrown

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the 2017 AWS S3 outage: an engineer running a routine playbook mistyped one number and removed far more capacity than intended — taking a chunk of the internet down with it.

The tooling should have refused. Make decommission.js refuse.

Bug report

BUG-S3-0228 · Priority: Critical · Reported by: incident review

planRemoval(requestedCount, pool) rules:

  • requestedCount must parse as a positive integer, else throw "invalid count"
  • at least MIN_ALIVE (2) servers must remain: clamp the removal to pool.length - 2
  • removing from a pool of 2 or fewer removes nothing

Observed: "1O0" (letter O) parsed as 1, "100" removed 100 of 10, pool hit zero.

Logs

[decom] pool=10 requested=100 removed=10 remaining=0
[decom] index-fleet offline; GET /* -> 500

The code as shipped

src/ops/decommission.js (editable)

var MIN_ALIVE = 2;

// Plans a capacity removal. Returns the servers that remain.
exports.planRemoval = function (requestedCount, pool) {
  var count = parseInt(requestedCount, 10);
  return pool.slice(0, pool.length - count);
};

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.