One Customer's Config Took Down the Edge

Medium

· Backend

Incident

SEV-1

BUG-FASTLY

Priority: Critical · reported by edge SRE

Time to fix: 20 min

Failing checks on the shipped build

  • a missing backends map falls back

    Cannot read properties of undefined (reading 'a.example')

  • an empty backends map falls back

    expected "pool-d" but got undefined

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Fastly · 2021

Modelled on the Fastly global outage (8 June 2021): a latent bug shipped weeks earlier was triggered when one customer pushed a valid configuration change. 85% of the CDN's network started returning errors within a minute.

router.js resolves a hostname to a backend and assumes the backend map is always populated. A customer whose config legitimately has no explicit backends takes the whole request path down.

Fix resolveBackend so an empty or missing map degrades to the default backend instead of failing.

Code map

// Maps a hostname to the backend that should serve it.
exports.resolveBackend = function (config, host) {
  return config.backends[host];
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

One Customer's Config Took Down the Edge — Bug Hunt

Modelled on the Fastly global outage (8 June 2021): a latent bug shipped weeks earlier was triggered when one customer pushed a valid configuration change.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Configuration, Defensive Coding
  • Modelled on: Fastly · 2021
  • Visible tests: an explicit backend is used; a missing backends map falls back; an empty backends map falls back
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-FASTLY
  • Reported by: edge SRE
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a missing backends map falls back — Cannot read properties of undefined (reading 'a.example')
  • an empty backends map falls back — expected "pool-d" but got undefined

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Fastly global outage (8 June 2021): a latent bug shipped weeks earlier was triggered when one customer pushed a valid configuration change. 85% of the CDN's network started returning errors within a minute.

router.js resolves a hostname to a backend and assumes the backend map is always populated. A customer whose config legitimately has no explicit backends takes the whole request path down.

Fix resolveBackend so an empty or missing map degrades to the default backend instead of failing.

Bug report

BUG-FASTLY · Priority: Critical · Reported by: edge SRE

resolveBackend(config, host) must return:

  • config.backends[host] when that entry exists
  • config.defaultBackend when it does not, or when backends is missing/empty
  • the string "origin" when there is no default either

Observed: a config whose backends object is absent throws, and a config whose backends is empty returns undefined — both surface as a 503 for every request served by that edge node.

Logs

[edge] TypeError: Cannot read property 'www.example.com' of undefined
[edge] 85% of POPs returning 503

The code as shipped

src/edge/router.js (editable)

// Maps a hostname to the backend that should serve it.
exports.resolveBackend = function (config, host) {
  return config.backends[host];
};

Read-only context: src/edge/CONTRACT.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.