The Node Update That Cut the Network

Medium

· Backend

Incident

SEV-1

BUG-DD0308

Priority: Critical · reported by platform

Time to fix: 20 min

Failing checks on the shipped build

  • a network update drains first

    workloads must move before the network restarts: expected {"applied":true,"drained":true} but got {"applied":true,"drained":false}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Datadog · 2023

Modelled on the Datadog outage of 8 March 2023: a routine security update to systemd-networkd restarted networking on nodes that were still running workloads, severing their container network. Tens of thousands of nodes across three clouds went down at once.

updater.js applies updates to a node without checking whether the update class is disruptive.

Fix applyNodeUpdate so a network-class update drains the node's workloads first.

Code map

// Applies a maintenance update to a node.
exports.applyNodeUpdate = function (node, update) {
  node.version = update.version;
  return { applied: true, drained: false };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Node Update That Cut the Network — JavaScript Bug Hunt

Modelled on the Datadog outage of 8 March 2023: a routine security update to systemd-networkd restarted networking on nodes that were still running…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Deployments, Draining
  • Modelled on: Datadog · 2023
  • Visible tests: a benign update applies in place; a network update drains first
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-DD0308
  • Reported by: platform
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a network update drains first — workloads must move before the network restarts: expected {"applied":true,"drained":true} but got {"applied":true,"drained":false}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Datadog outage of 8 March 2023: a routine security update to systemd-networkd restarted networking on nodes that were still running workloads, severing their container network. Tens of thousands of nodes across three clouds went down at once.

updater.js applies updates to a node without checking whether the update class is disruptive.

Fix applyNodeUpdate so a network-class update drains the node's workloads first.

Bug report

BUG-DD0308 · Priority: Critical · Reported by: platform

applyNodeUpdate(node, update) must return { applied, drained }:

  • an update with kind "network" requires the node to be drained first — set node.workloads to [] and drained to true before applying
  • any other kind applies in place with drained false
  • a node already marked cordoned:false must be cordoned before draining

Observed: a network update is applied while workloads are still scheduled, and they lose connectivity with no chance to move.

Logs

[node] applied kind=network workloads_running=48
[node] 0/48 workloads reachable after restart

The code as shipped

src/ops/updater.js (editable)

// Applies a maintenance update to a node.
exports.applyNodeUpdate = function (node, update) {
  node.version = update.version;
  return { applied: true, drained: false };
};

Read-only context: src/ops/LIFECYCLE.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.