Tenths of a Second Since 2000 — Java Bug Hunt
Modelled on the loss of NASA's Deep Impact spacecraft in August 2013. Contact was lost in August 2013 and the mission was declared over the following month.
- Language: Java
- Layer: Backend
- Difficulty: Medium
- Concepts: Overflow, Time
- Modelled on: NASA Deep Impact · 2013
- Visible tests: an early mission time converts to tenths; the August 2013 boundary does not reset the computer
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-DI-2013
- Reported by: mission operations
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- the August 2013 boundary does not reset the computer — one second across 2^32 tenths: expected NOMINAL but got CLOCK_FAULT_RESET
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the loss of NASA's Deep Impact spacecraft in August 2013. Contact was lost in August 2013 and the mission was declared over the following month. The probable cause NASA's team described was a time-keeping limit: the fault-protection software counted time in tenths of a second since 1 January 2000, and that count no longer fitted in 32 bits once it passed 2^32 tenths — in August 2013 — leaving the onboard computers unable to keep a sensible clock.
This reconstruction keeps the flight software's time tags in a long but still masks them to 32 bits, so the tag wraps back to a small number at the boundary and fault protection sees time run backwards.
Fix SpacecraftClock so time tags keep counting past 2^32 tenths.
Bug report
BUG-DI-2013 · Priority: Critical · Reported by: mission operations
SpacecraftClock:
- toTag(secondsSince2000) returns tenths of a second since 2000-01-01T00:00:00 — exactly secondsSince2000 * 10, for any mission time representable in a long (no wrap at 2^32 = 4,294,967,296 tenths)
- toSeconds(tag) is the inverse (tag / 10)
- elapsedTenths(earlier, later) is later - earlier
FaultProtection.check(previous, current) (locked) resets the computer when time goes backwards or jumps more than a day between readings.
Observed: two readings one second apart, straddling 429,496,729.6 s after 2000 (August 2013), produce tags 4294967290 and 4 — fault protection resets, and keeps resetting.
Logs
[fp] tag prev=4294967290 cur=4 step=-4294967286
[fp] CLOCK_FAULT_RESET (1 of n)
[fp] CLOCK_FAULT_RESET (2 of n)The code as shipped
src/clock/SpacecraftClock.java (editable)
class SpacecraftClock {
// Time tags count tenths of a second since 2000-01-01T00:00:00, stored in
// the 32-bit field the flight software was specified with.
static final long TAG_MASK = 0xFFFFFFFFL;
static long toTag(long secondsSince2000) {
return (secondsSince2000 * 10) & TAG_MASK;
}
static long toSeconds(long tag) {
return tag / 10;
}
static long elapsedTenths(long earlierTag, long laterTag) {
return laterTag - earlierTag;
}
}
Read-only context: src/clock/FaultProtection.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.