Scaling Up Past the Thread Limit — JavaScript Bug Hunt

Briefing

Modelled on the AWS Kinesis outage (25 November 2020): adding capacity to the front-end fleet pushed the number of OS threads per server past a configured limit, because each server maintains a thread per peer. The fleet failed to build its shard map and the outage cascaded into CloudWatch, Cognito and beyond.

fleet.js adds servers without checking the per-server thread budget.

Fix addServers so a change that would exceed the limit is refused, leaving the fleet as it was.

Bug report

BUG-KINESIS · Priority: Critical · Reported by: capacity engineering

addServers(fleet, count):

• each server keeps one thread per OTHER server, so a fleet of n servers uses n - 1 threads per server
• the change must be refused if the resulting per-server thread count would exceed fleet.threadLimit
• returns { applied: bool, servers: <resulting count> }

Observed: the fleet grows regardless, the servers exceed the limit, and none of them can complete the shard-map handshake.

Logs

[fleet] servers=1024 threadsPerServer=1023 limit=900
[fleet] shard map construction failed on 100% of front-ends

The code as shipped

src/fleet/fleet.js (editable)

// Grows the front-end fleet.
exports.addServers = function (fleet, count) {
  fleet.servers += count;
  return { applied: true, servers: fleet.servers };
};

exports.threadsPerServer = function (fleet) {
  return fleet.servers - 1;
};

Read-only context: src/fleet/CAPACITY.js.