Negative Altitude, Underground

Medium

· Backend

Incident

SEV-1

BUG-EDM16

Priority: Critical · reported by GNC

Time to fix: 20 min

Failing checks on the shipped build

  • a saturated sample is discarded

    a saturated reading must not be integrated: expected 700 but got -97200

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

ESA · Schiaparelli 2016

Modelled on the Schiaparelli lander (October 2016): the inertial unit saturated during parachute deployment and reported a rotation rate far beyond its range. The navigation filter integrated that value, concluded the lander was below ground level, and jettisoned the parachute and shut off the thrusters at about 3.7 km altitude.

navfilter.js integrates every sample it is handed, saturated or not.

Fix estimateAltitude so saturated samples are discarded and the estimate can never go negative.

Code map

// Integrates descent rates into an altitude estimate.
exports.estimateAltitude = function (start, samples, maxRate) {
  var altitude = start;
  for (var i = 0; i < samples.length; i++) {
    altitude = altitude - samples[i].rate;
  }
  return altitude;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Negative Altitude, Underground — JavaScript Bug Hunt

Modelled on the Schiaparelli lander (October 2016): the inertial unit saturated during parachute deployment and reported a rotation rate far beyond its range.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Sensors, Saturation
  • Modelled on: ESA · Schiaparelli 2016
  • Visible tests: normal samples integrate; a saturated sample is discarded
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-EDM16
  • Reported by: GNC
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a saturated sample is discarded — a saturated reading must not be integrated: expected 700 but got -97200

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Schiaparelli lander (October 2016): the inertial unit saturated during parachute deployment and reported a rotation rate far beyond its range. The navigation filter integrated that value, concluded the lander was below ground level, and jettisoned the parachute and shut off the thrusters at about 3.7 km altitude.

navfilter.js integrates every sample it is handed, saturated or not.

Fix estimateAltitude so saturated samples are discarded and the estimate can never go negative.

Bug report

BUG-EDM16 · Priority: Critical · Reported by: GNC

estimateAltitude(start, samples, maxRate):

  • integrates each sample's rate (metres lost per step)
  • a sample whose |rate| exceeds maxRate is SATURATED and must be discarded, with the previous accepted rate used in its place (0 if there is none)
  • the returned altitude is clamped to a floor of 0

Observed: one saturated sample drives the estimate to a large negative altitude and the descent sequence fires early.

Logs

[gnc] altitude=-2400.0 after sample rate=98000.0 (max 500.0)
[gnc] parachute jettisoned at true altitude 3.7km

The code as shipped

src/gnc/navfilter.js (editable)

// Integrates descent rates into an altitude estimate.
exports.estimateAltitude = function (start, samples, maxRate) {
  var altitude = start;
  for (var i = 0; i < samples.length; i++) {
    altitude = altitude - samples[i].rate;
  }
  return altitude;
};

Read-only context: src/gnc/IMU.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.