The Task That Ran Out of Stack

Hard

· Backend

Incident

SEV-1

BUG-ETCS

Priority: Critical (safety) · reported by powertrain

Time to fix: 35 min

Failing checks on the shipped build

  • a too-deep graph fails safe instead of crashing

    an over-deep graph must fail to a closed throttle: expected {"ok":false,"throttle":0} but got {"ok":true,"throttle":5001}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Toyota ETCS · Bookout 2013

Modelled on the findings in the Toyota unintended-acceleration case (Bookout v. Toyota, 2013): the engine control software's stack usage was far larger than analysed, and there was no protection when it overflowed — a task could die silently and leave the throttle in its last commanded state.

ecu.js walks a nested task graph recursively with no depth limit and no fallback if the walk fails.

Fix computeThrottle so deep graphs cannot blow the stack, and a failed evaluation returns the safe default rather than the last value.

Code map

// Sums the demand across the task graph.
function walk(node) {
  var total = node.demand;
  for (var i = 0; i < node.children.length; i++) {
    total += walk(node.children[i]);
  }
  return total;
}

exports.computeThrottle = function (graph, maxDepth) {
  return { ok: true, throttle: walk(graph) };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Task That Ran Out of Stack — JavaScript Bug Hunt

Modelled on the findings in the Toyota unintended-acceleration case (Bookout v.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Recursion, Resource Limits
  • Modelled on: Toyota ETCS · Bookout 2013
  • Visible tests: a shallow graph sums correctly; a too-deep graph fails safe instead of crashing
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (safety))
  • Ticket: BUG-ETCS
  • Reported by: powertrain
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a too-deep graph fails safe instead of crashing — an over-deep graph must fail to a closed throttle: expected {"ok":false,"throttle":0} but got {"ok":true,"throttle":5001}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the findings in the Toyota unintended-acceleration case (Bookout v. Toyota, 2013): the engine control software's stack usage was far larger than analysed, and there was no protection when it overflowed — a task could die silently and leave the throttle in its last commanded state.

ecu.js walks a nested task graph recursively with no depth limit and no fallback if the walk fails.

Fix computeThrottle so deep graphs cannot blow the stack, and a failed evaluation returns the safe default rather than the last value.

Bug report

BUG-ETCS · Priority: Critical (safety) · Reported by: powertrain

computeThrottle(graph, maxDepth) must:

  • sum the demand of every node in the tree
  • refuse graphs deeper than maxDepth, returning { ok: false, throttle: 0 }
  • return { ok: true, throttle: <sum> } otherwise

Observed: a deeply nested graph overflows the stack. The caller keeps the previous throttle value, which on the road means the pedal stops responding.

Logs

[ecu] RangeError: Maximum call stack size exceeded at depth 11
[ecu] throttle held at last commanded value

The code as shipped

src/ecu/ecu.js (editable)

// Sums the demand across the task graph.
function walk(node) {
  var total = node.demand;
  for (var i = 0; i < node.children.length; i++) {
    total += walk(node.children[i]);
  }
  return total;
}

exports.computeThrottle = function (graph, maxDepth) {
  return { ok: true, throttle: walk(graph) };
};

Read-only context: src/ecu/SAFETY.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.