Tenths of a Second Since 2000

Medium

· Backend

Incident

SEV-1

BUG-DI-2013

Priority: Critical · reported by mission operations

Time to fix: 20 min

Failing checks on the shipped build

  • the August 2013 boundary does not reset the computer

    one second across 2^32 tenths: expected NOMINAL but got CLOCK_FAULT_RESET

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

NASA Deep Impact · 2013

Modelled on the loss of NASA's Deep Impact spacecraft in August 2013. Contact was lost in August 2013 and the mission was declared over the following month. The probable cause NASA's team described was a time-keeping limit: the fault-protection software counted time in tenths of a second since 1 January 2000, and that count no longer fitted in 32 bits once it passed 2^32 tenths — in August 2013 — leaving the onboard computers unable to keep a sensible clock.

This reconstruction keeps the flight software's time tags in a long but still masks them to 32 bits, so the tag wraps back to a small number at the boundary and fault protection sees time run backwards.

Fix SpacecraftClock so time tags keep counting past 2^32 tenths.

Code map

class SpacecraftClock {
    // Time tags count tenths of a second since 2000-01-01T00:00:00, stored in
    // the 32-bit field the flight software was specified with.
    static final long TAG_MASK = 0xFFFFFFFFL;

    static long toTag(long secondsSince2000) {
        return (secondsSince2000 * 10) & TAG_MASK;
    }

    static long toSeconds(long tag) {
        return tag / 10;
    }

    static long elapsedTenths(long earlierTag, long laterTag) {
        return laterTag - earlierTag;
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Tenths of a Second Since 2000 — Java Bug Hunt

Modelled on the loss of NASA's Deep Impact spacecraft in August 2013. Contact was lost in August 2013 and the mission was declared over the following month.

  • Language: Java
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Overflow, Time
  • Modelled on: NASA Deep Impact · 2013
  • Visible tests: an early mission time converts to tenths; the August 2013 boundary does not reset the computer
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-DI-2013
  • Reported by: mission operations
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • the August 2013 boundary does not reset the computer — one second across 2^32 tenths: expected NOMINAL but got CLOCK_FAULT_RESET

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the loss of NASA's Deep Impact spacecraft in August 2013. Contact was lost in August 2013 and the mission was declared over the following month. The probable cause NASA's team described was a time-keeping limit: the fault-protection software counted time in tenths of a second since 1 January 2000, and that count no longer fitted in 32 bits once it passed 2^32 tenths — in August 2013 — leaving the onboard computers unable to keep a sensible clock.

This reconstruction keeps the flight software's time tags in a long but still masks them to 32 bits, so the tag wraps back to a small number at the boundary and fault protection sees time run backwards.

Fix SpacecraftClock so time tags keep counting past 2^32 tenths.

Bug report

BUG-DI-2013 · Priority: Critical · Reported by: mission operations

SpacecraftClock:

  • toTag(secondsSince2000) returns tenths of a second since 2000-01-01T00:00:00 — exactly secondsSince2000 * 10, for any mission time representable in a long (no wrap at 2^32 = 4,294,967,296 tenths)
  • toSeconds(tag) is the inverse (tag / 10)
  • elapsedTenths(earlier, later) is later - earlier

FaultProtection.check(previous, current) (locked) resets the computer when time goes backwards or jumps more than a day between readings.

Observed: two readings one second apart, straddling 429,496,729.6 s after 2000 (August 2013), produce tags 4294967290 and 4 — fault protection resets, and keeps resetting.

Logs

[fp] tag prev=4294967290 cur=4 step=-4294967286
[fp] CLOCK_FAULT_RESET (1 of n)
[fp] CLOCK_FAULT_RESET (2 of n)

The code as shipped

src/clock/SpacecraftClock.java (editable)

class SpacecraftClock {
    // Time tags count tenths of a second since 2000-01-01T00:00:00, stored in
    // the 32-bit field the flight software was specified with.
    static final long TAG_MASK = 0xFFFFFFFFL;

    static long toTag(long secondsSince2000) {
        return (secondsSince2000 * 10) & TAG_MASK;
    }

    static long toSeconds(long tag) {
        return tag / 10;
    }

    static long elapsedTenths(long earlierTag, long laterTag) {
        return laterTag - earlierTag;
    }
}

Read-only context: src/clock/FaultProtection.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.