The Clock Started Eleven Hours Early

Medium

· Backend

Incident

SEV-1

BUG-OFT-MET

Priority: Critical · reported by flight software IV&V

Time to fix: 20 min

Failing checks on the shipped build

  • countdown events do not start the clock

    MET must count from liftoff: expected 300 but got 39900

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Boeing Starliner · 2019

Modelled on Boeing Starliner's Orbital Flight Test (20 December 2019): the capsule's flight computer read its mission elapsed time from the Atlas V launch vehicle at the wrong point in the countdown, so its clock was off by about 11 hours. After separation it believed it was in a different phase of the mission and fired its thrusters accordingly, burning so much propellant that the planned docking with the International Space Station was called off.

MissionClock.java is a reconstruction: it receives each launch-sequence event with the booster's clock reading and must latch the mission epoch at liftoff. FlightPhases maps elapsed time to the phase — and to the thruster plan — the software believes it is in.

Fix MissionClock so mission elapsed time is measured from liftoff and nothing else.

Code map

class MissionClock {
    private long epoch;
    private boolean latched;

    // Called for every launch-sequence event with the booster's clock
    // reading (seconds since the booster's own power-on).
    void onEvent(String event, long boosterTime) {
        if (!latched) {
            epoch = boosterTime;
            latched = true;
        }
    }

    long met(long boosterTime) {
        if (!latched) return -1;
        return boosterTime - epoch;
    }

    String phase(long boosterTime) {
        return FlightPhases.phaseAt(met(boosterTime));
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Clock Started Eleven Hours Early — Java Bug Hunt

Modelled on Boeing Starliner's Orbital Flight Test (20 December 2019): the capsule's flight computer read its mission elapsed time from the Atlas V launch…

  • Language: Java
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Time, State
  • Modelled on: Boeing Starliner · 2019
  • Visible tests: liftoff alone starts the clock; countdown events do not start the clock
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-OFT-MET
  • Reported by: flight software IV&V
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • countdown events do not start the clock — MET must count from liftoff: expected 300 but got 39900

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Boeing Starliner's Orbital Flight Test (20 December 2019): the capsule's flight computer read its mission elapsed time from the Atlas V launch vehicle at the wrong point in the countdown, so its clock was off by about 11 hours. After separation it believed it was in a different phase of the mission and fired its thrusters accordingly, burning so much propellant that the planned docking with the International Space Station was called off.

MissionClock.java is a reconstruction: it receives each launch-sequence event with the booster's clock reading and must latch the mission epoch at liftoff. FlightPhases maps elapsed time to the phase — and to the thruster plan — the software believes it is in.

Fix MissionClock so mission elapsed time is measured from liftoff and nothing else.

Bug report

BUG-OFT-MET · Priority: Critical · Reported by: flight software IV&V

onEvent(event, boosterTime) is called for every launch-sequence event, in order, with the booster's clock (seconds since booster power-on).

  • the epoch is latched from the FIRST "LIFTOFF" event only; every other event, before or after it, and any repeated "LIFTOFF", leaves it alone

met(boosterTime)

  • before liftoff has been latched -> -1
  • after -> boosterTime - epoch

phase(boosterTime) -> FlightPhases.phaseAt(met(boosterTime))

Observed: with the countdown events delivered first, met is ~11 hours (39,600 s) too large and the capsule plans a burn for the wrong phase.

Logs

[fsw] event TERMINAL_COUNT booster_t=1000 -> MET epoch latched
[fsw] event LIFTOFF booster_t=40600
[fsw] T+300 MET=39900 phase=COAST — executing coast-phase attitude plan

The code as shipped

MissionClock.java (editable)

class MissionClock {
    private long epoch;
    private boolean latched;

    // Called for every launch-sequence event with the booster's clock
    // reading (seconds since the booster's own power-on).
    void onEvent(String event, long boosterTime) {
        if (!latched) {
            epoch = boosterTime;
            latched = true;
        }
    }

    long met(long boosterTime) {
        if (!latched) return -1;
        return boosterTime - epoch;
    }

    String phase(long boosterTime) {
        return FlightPhases.phaseAt(met(boosterTime));
    }
}

Read-only context: FlightPhases.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.