Scaled Down Overnight, Trampled at Dawn — Bug Hunt
Modelled on Slack's outage of 4 January 2021: the fleet scaled down over the quiet holidays, then the first working day of the year brought everyone back at…
- Language: JavaScript
- Layer: Backend
- Difficulty: Medium
- Concepts: Autoscaling, Thundering Herd
- Modelled on: Slack · 2021
- Visible tests: scaling up is immediate; scaling down is gradual; the floor is respected
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-2 (priority: High)
- Ticket: BUG-SLACK0104
- Reported by: capacity SRE
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- scaling down is gradual — at most 10 instances may be removed per step: expected 390 but got 1
- the floor is respected — expected 20 but got 0
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Slack's outage of 4 January 2021: the fleet scaled down over the quiet holidays, then the first working day of the year brought everyone back at once. The autoscaler could not add capacity fast enough, and the resulting overload took the service down for hours.
autoscaler.js sizes the fleet purely on current load, with no floor and no rate limit.
Fix scaleTarget so it never drops below a floor and never scales down faster than the configured step.
Bug report
BUG-SLACK0104 · Priority: High · Reported by: capacity SRE
scaleTarget(current, load, config) must return the next fleet size, where:
- the raw target is load / config.perInstanceCapacity, rounded up
- the result is never below config.minInstances
- a scale-DOWN never removes more than config.maxScaleDownStep instances at once
- scaling up is not rate limited
Observed: an overnight lull takes the fleet from 400 instances to 2 in one step, and the morning ramp cannot recover.
Logs
[autoscale] target 2 (from 400) load=95 rps
[autoscale] 09:02 load=42000 rps, healthy instances 4, queue depth 1.2MThe code as shipped
src/scale/autoscaler.js (editable)
// Computes the next fleet size from the current load.
exports.scaleTarget = function (current, load, config) {
return Math.ceil(load / config.perInstanceCapacity);
};
Read-only context: src/scale/CONFIG.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.