The Node Update That Cut the Network — JavaScript Bug Hunt

Briefing

Modelled on the Datadog outage of 8 March 2023: a routine security update to systemd-networkd restarted networking on nodes that were still running workloads, severing their container network. Tens of thousands of nodes across three clouds went down at once.

updater.js applies updates to a node without checking whether the update class is disruptive.

Fix applyNodeUpdate so a network-class update drains the node's workloads first.

Bug report

BUG-DD0308 · Priority: Critical · Reported by: platform

applyNodeUpdate(node, update) must return { applied, drained }:

• an update with kind "network" requires the node to be drained first — set node.workloads to [] and drained to true before applying
• any other kind applies in place with drained false
• a node already marked cordoned:false must be cordoned before draining

Observed: a network update is applied while workloads are still scheduled, and they lose connectivity with no chance to move.

Logs

[node] applied kind=network workloads_running=48
[node] 0/48 workloads reachable after restart

The code as shipped

src/ops/updater.js (editable)

// Applies a maintenance update to a node.
exports.applyNodeUpdate = function (node, update) {
  node.version = update.version;
  return { applied: true, drained: false };
};

Read-only context: src/ops/LIFECYCLE.js.