The Label That Kubernetes Removed

Medium

· Backend

Incident

SEV-1

BUG-REDDIT314

Priority: Critical · reported by compute

Time to fix: 20 min

Failing checks on the shipped build

  • the modern label is recognised

    expected "control-plane" but got "worker"

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Reddit · Pi-Day 2023

Modelled on Reddit's Pi-Day outage (14 March 2023): a Kubernetes upgrade to 1.24 removed the long-deprecated master node label. Calico's route calculation still keyed on it, so the cluster's network layer silently misrouted and the site was down for several hours.

topology.js reads only the legacy label and has no fallback.

Fix roleOf so the modern label is honoured and the legacy one remains a fallback.

Code map

// Determines a node's role from its labels.
exports.roleOf = function (node) {
  if (node.labels["node-role.kubernetes.io/master"] !== undefined) {
    return "control-plane";
  }
  return "worker";
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Label That Kubernetes Removed — JavaScript Bug Hunt

Modelled on Reddit's Pi-Day outage (14 March 2023): a Kubernetes upgrade to 1.24 removed the long-deprecated master node label.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Migration, Deprecation
  • Modelled on: Reddit · Pi-Day 2023
  • Visible tests: the modern label is recognised; the legacy label still works
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-REDDIT314
  • Reported by: compute
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • the modern label is recognised — expected "control-plane" but got "worker"

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Reddit's Pi-Day outage (14 March 2023): a Kubernetes upgrade to 1.24 removed the long-deprecated master node label. Calico's route calculation still keyed on it, so the cluster's network layer silently misrouted and the site was down for several hours.

topology.js reads only the legacy label and has no fallback.

Fix roleOf so the modern label is honoured and the legacy one remains a fallback.

Bug report

BUG-REDDIT314 · Priority: Critical · Reported by: compute

roleOf(node) must return the node's role, preferring the modern label:

  • "node-role.kubernetes.io/control-plane" wins if present
  • otherwise the legacy "node-role.kubernetes.io/master"
  • otherwise "worker"

Observed: after the upgrade only the legacy key exists on old nodes and only the modern key on new ones. Reading one key alone mislabels half the cluster.

Logs

[topology] node ip-10-0-3-14 role=worker (expected control-plane)
[topology] route table built from 0 control-plane nodes

The code as shipped

src/k8s/topology.js (editable)

// Determines a node's role from its labels.
exports.roleOf = function (node) {
  if (node.labels["node-role.kubernetes.io/master"] !== undefined) {
    return "control-plane";
  }
  return "worker";
};

Read-only context: src/k8s/LABELS.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.