Touchdown at Forty Metres

Medium

· Backend

Incident

SEV-1

BUG-MPL

Priority: Critical · reported by EDL team

Time to fix: 20 min

Failing checks on the shipped build

  • a deployment transient does not cut the engine

    a transient during leg deployment must be ignored: expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

NASA · Mars Polar Lander 1999

Modelled on the Mars Polar Lander loss (December 1999): the leg-deployment jolt produced a spurious touchdown signal. The flight software latched that transient as "landed" and shut the descent engine down about 40 metres up. The lander fell.

descent.js latches the touchdown flag the moment it first sees it.

Fix shouldCutEngine so a transient during leg deployment cannot latch, and touchdown is only believed once it is stable and the altitude agrees.

Code map

// Decides whether the descent engine may be shut down.
exports.shouldCutEngine = function (samples, altitude, legDeploymentEndsAt) {
  var latched = false;
  for (var i = 0; i < samples.length; i++) {
    if (samples[i].touchdown) latched = true;
  }
  return latched;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Touchdown at Forty Metres — JavaScript Bug Hunt

Modelled on the Mars Polar Lander loss (December 1999): the leg-deployment jolt produced a spurious touchdown signal.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Sensors, State Latching
  • Modelled on: NASA · Mars Polar Lander 1999
  • Visible tests: a sustained touchdown near the ground cuts the engine; a deployment transient does not cut the engine
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-MPL
  • Reported by: EDL team
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a deployment transient does not cut the engine — a transient during leg deployment must be ignored: expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Mars Polar Lander loss (December 1999): the leg-deployment jolt produced a spurious touchdown signal. The flight software latched that transient as "landed" and shut the descent engine down about 40 metres up. The lander fell.

descent.js latches the touchdown flag the moment it first sees it.

Fix shouldCutEngine so a transient during leg deployment cannot latch, and touchdown is only believed once it is stable and the altitude agrees.

Bug report

BUG-MPL · Priority: Critical · Reported by: EDL team

shouldCutEngine(samples, altitude, legDeploymentEndsAt) must return true only when BOTH hold:

  • the last 3 samples all read touchdown=true, and none of those samples has an index below legDeploymentEndsAt (transients during deployment are ignored)
  • altitude is at or below 2 metres

Observed: one true sample anywhere in the record latches the state and the engine is cut, whatever the altitude.

Logs

[edl] engine cut at altitude=40.1m
[edl] touchdown asserted at sample 3 (leg deployment ends at sample 12)

The code as shipped

src/edl/descent.js (editable)

// Decides whether the descent engine may be shut down.
exports.shouldCutEngine = function (samples, altitude, legDeploymentEndsAt) {
  var latched = false;
  for (var i = 0; i < samples.length; i++) {
    if (samples[i].touchdown) latched = true;
  }
  return latched;
};

Read-only context: src/edl/SEQUENCE.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.