Scaling Up Past the Thread Limit — JavaScript Bug Hunt
Modelled on the AWS Kinesis outage (25 November 2020): adding capacity to the front-end fleet pushed the number of OS threads per server past a configured…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Capacity, Resource Limits
- Modelled on: AWS Kinesis · 2020
- Visible tests: a safe scale-up is applied; a scale-up past the limit is refused
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-KINESIS
- Reported by: capacity engineering
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a scale-up past the limit is refused — the fleet must not grow past the thread limit: expected {"applied":false,"servers":895} but got {"applied":true,"servers":1095}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the AWS Kinesis outage (25 November 2020): adding capacity to the front-end fleet pushed the number of OS threads per server past a configured limit, because each server maintains a thread per peer. The fleet failed to build its shard map and the outage cascaded into CloudWatch, Cognito and beyond.
fleet.js adds servers without checking the per-server thread budget.
Fix addServers so a change that would exceed the limit is refused, leaving the fleet as it was.
Bug report
BUG-KINESIS · Priority: Critical · Reported by: capacity engineering
addServers(fleet, count):
- each server keeps one thread per OTHER server, so a fleet of n servers uses n - 1 threads per server
- the change must be refused if the resulting per-server thread count would exceed fleet.threadLimit
- returns { applied: bool, servers: <resulting count> }
Observed: the fleet grows regardless, the servers exceed the limit, and none of them can complete the shard-map handshake.
Logs
[fleet] servers=1024 threadsPerServer=1023 limit=900
[fleet] shard map construction failed on 100% of front-endsThe code as shipped
src/fleet/fleet.js (editable)
// Grows the front-end fleet.
exports.addServers = function (fleet, count) {
fleet.servers += count;
return { applied: true, servers: fleet.servers };
};
exports.threadsPerServer = function (fleet) {
return fleet.servers - 1;
};
Read-only context: src/fleet/CAPACITY.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.