Scaling Up Past the Thread Limit

Medium

· Backend

Incident

SEV-1

BUG-KINESIS

Priority: Critical · reported by capacity engineering

Time to fix: 20 min

Failing checks on the shipped build

  • a scale-up past the limit is refused

    the fleet must not grow past the thread limit: expected {"applied":false,"servers":895} but got {"applied":true,"servers":1095}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

AWS Kinesis · 2020

Modelled on the AWS Kinesis outage (25 November 2020): adding capacity to the front-end fleet pushed the number of OS threads per server past a configured limit, because each server maintains a thread per peer. The fleet failed to build its shard map and the outage cascaded into CloudWatch, Cognito and beyond.

fleet.js adds servers without checking the per-server thread budget.

Fix addServers so a change that would exceed the limit is refused, leaving the fleet as it was.

Code map

// Grows the front-end fleet.
exports.addServers = function (fleet, count) {
  fleet.servers += count;
  return { applied: true, servers: fleet.servers };
};

exports.threadsPerServer = function (fleet) {
  return fleet.servers - 1;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Scaling Up Past the Thread Limit — JavaScript Bug Hunt

Modelled on the AWS Kinesis outage (25 November 2020): adding capacity to the front-end fleet pushed the number of OS threads per server past a configured…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Capacity, Resource Limits
  • Modelled on: AWS Kinesis · 2020
  • Visible tests: a safe scale-up is applied; a scale-up past the limit is refused
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-KINESIS
  • Reported by: capacity engineering
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a scale-up past the limit is refused — the fleet must not grow past the thread limit: expected {"applied":false,"servers":895} but got {"applied":true,"servers":1095}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the AWS Kinesis outage (25 November 2020): adding capacity to the front-end fleet pushed the number of OS threads per server past a configured limit, because each server maintains a thread per peer. The fleet failed to build its shard map and the outage cascaded into CloudWatch, Cognito and beyond.

fleet.js adds servers without checking the per-server thread budget.

Fix addServers so a change that would exceed the limit is refused, leaving the fleet as it was.

Bug report

BUG-KINESIS · Priority: Critical · Reported by: capacity engineering

addServers(fleet, count):

  • each server keeps one thread per OTHER server, so a fleet of n servers uses n - 1 threads per server
  • the change must be refused if the resulting per-server thread count would exceed fleet.threadLimit
  • returns { applied: bool, servers: <resulting count> }

Observed: the fleet grows regardless, the servers exceed the limit, and none of them can complete the shard-map handshake.

Logs

[fleet] servers=1024 threadsPerServer=1023 limit=900
[fleet] shard map construction failed on 100% of front-ends

The code as shipped

src/fleet/fleet.js (editable)

// Grows the front-end fleet.
exports.addServers = function (fleet, count) {
  fleet.servers += count;
  return { applied: true, servers: fleet.servers };
};

exports.threadsPerServer = function (fleet) {
  return fleet.servers - 1;
};

Read-only context: src/fleet/CAPACITY.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.