Twenty-One Fields, Twenty Supplied

Medium

· Backend

Incident

SEV-1

BUG-CS0719

Priority: Critical · reported by sensor team

Time to fix: 20 min

Failing checks on the shipped build

  • a short record is rejected outright

    a read past the end must be refused: expected {"ok":false,"values":[]} but got {"ok":true,"values":[null]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

CrowdStrike · 2024

Modelled on the CrowdStrike Falcon outage (19 July 2024): a rapid-response content update carried fewer input fields than the sensor's template expected. The sensor read past the end of the supplied array, faulted in kernel mode, and around 8.5 million Windows machines blue-screened worldwide.

channel.js indexes fields straight out of the record with no count check.

Fix readTemplate so a record with the wrong field count is rejected before anything is read.

Code map

// Extracts the fields a template needs from a channel record.
exports.readTemplate = function (record, template) {
  var values = [];
  for (var i = 0; i < template.indexes.length; i++) {
    values.push(record.fields[template.indexes[i]]);
  }
  return { ok: true, values: values };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Twenty-One Fields, Twenty Supplied — JavaScript Bug Hunt

Modelled on the CrowdStrike Falcon outage (19 July 2024): a rapid-response content update carried fewer input fields than the sensor's template expected.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Parsing, Bounds Checking
  • Modelled on: CrowdStrike · 2024
  • Visible tests: a well-formed record is read; a short record is rejected outright
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-CS0719
  • Reported by: sensor team
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a short record is rejected outright — a read past the end must be refused: expected {"ok":false,"values":[]} but got {"ok":true,"values":[null]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the CrowdStrike Falcon outage (19 July 2024): a rapid-response content update carried fewer input fields than the sensor's template expected. The sensor read past the end of the supplied array, faulted in kernel mode, and around 8.5 million Windows machines blue-screened worldwide.

channel.js indexes fields straight out of the record with no count check.

Fix readTemplate so a record with the wrong field count is rejected before anything is read.

Bug report

BUG-CS0719 · Priority: Critical · Reported by: sensor team

readTemplate(record, template) must return { ok, values }:

  • template.indexes lists the field positions the template needs
  • if record.fields has fewer entries than the highest index requires, the record is invalid: return { ok: false, values: [] }
  • otherwise return { ok: true, values: <the fields at those indexes> }

Observed: a record with 20 fields is fed to a template that reads index 20, and the read returns undefined — which downstream code dereferences.

Logs

[sensor] template 291 read index 20 of a 20-field record -> undefined
[sensor] PAGE_FAULT_IN_NONPAGED_AREA

The code as shipped

src/sensor/channel.js (editable)

// Extracts the fields a template needs from a channel record.
exports.readTemplate = function (record, template) {
  var values = [];
  for (var i = 0; i < template.indexes.length; i++) {
    values.push(record.fields[template.indexes[i]]);
  }
  return { ok: true, values: values };
};

Read-only context: src/sensor/FORMAT.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.