Scaled Down Overnight, Trampled at Dawn

Medium

· Backend

Incident

SEV-2

BUG-SLACK0104

Priority: High · reported by capacity SRE

Time to fix: 20 min

Failing checks on the shipped build

  • scaling down is gradual

    at most 10 instances may be removed per step: expected 390 but got 1

  • the floor is respected

    expected 20 but got 0

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Slack · 2021

Modelled on Slack's outage of 4 January 2021: the fleet scaled down over the quiet holidays, then the first working day of the year brought everyone back at once. The autoscaler could not add capacity fast enough, and the resulting overload took the service down for hours.

autoscaler.js sizes the fleet purely on current load, with no floor and no rate limit.

Fix scaleTarget so it never drops below a floor and never scales down faster than the configured step.

Code map

// Computes the next fleet size from the current load.
exports.scaleTarget = function (current, load, config) {
  return Math.ceil(load / config.perInstanceCapacity);
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Scaled Down Overnight, Trampled at Dawn — Bug Hunt

Modelled on Slack's outage of 4 January 2021: the fleet scaled down over the quiet holidays, then the first working day of the year brought everyone back at…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Autoscaling, Thundering Herd
  • Modelled on: Slack · 2021
  • Visible tests: scaling up is immediate; scaling down is gradual; the floor is respected
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-SLACK0104
  • Reported by: capacity SRE
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • scaling down is gradual — at most 10 instances may be removed per step: expected 390 but got 1
  • the floor is respected — expected 20 but got 0

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Slack's outage of 4 January 2021: the fleet scaled down over the quiet holidays, then the first working day of the year brought everyone back at once. The autoscaler could not add capacity fast enough, and the resulting overload took the service down for hours.

autoscaler.js sizes the fleet purely on current load, with no floor and no rate limit.

Fix scaleTarget so it never drops below a floor and never scales down faster than the configured step.

Bug report

BUG-SLACK0104 · Priority: High · Reported by: capacity SRE

scaleTarget(current, load, config) must return the next fleet size, where:

  • the raw target is load / config.perInstanceCapacity, rounded up
  • the result is never below config.minInstances
  • a scale-DOWN never removes more than config.maxScaleDownStep instances at once
  • scaling up is not rate limited

Observed: an overnight lull takes the fleet from 400 instances to 2 in one step, and the morning ramp cannot recover.

Logs

[autoscale] target 2 (from 400) load=95 rps
[autoscale] 09:02 load=42000 rps, healthy instances 4, queue depth 1.2M

The code as shipped

src/scale/autoscaler.js (editable)

// Computes the next fleet size from the current load.
exports.scaleTarget = function (current, load, config) {
  return Math.ceil(load / config.perInstanceCapacity);
};

Read-only context: src/scale/CONFIG.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.