Priority Inversion on Mars — Java Bug Hunt
Modelled on Mars Pathfinder (July 1997): days after landing, the spacecraft began resetting itself.
- Language: Java
- Layer: Backend
- Difficulty: Hard
- Concepts: Concurrency, State
- Modelled on: NASA Mars Pathfinder · 1997
- Visible tests: without contention the medium task simply pre-empts meteo; the bus task meets its watchdog deadline
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (spacecraft resets))
- Ticket: BUG-MPF-0797
- Reported by: flight software
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- the bus task meets its watchdog deadline — bus finished at tick 12: expected condition to hold
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Mars Pathfinder (July 1997): days after landing, the spacecraft began resetting itself. JPL traced it to priority inversion in its VxWorks software: a low-priority meteorological task held a mutex guarding the information bus, the high-priority bus-management task blocked waiting for it, and medium-priority tasks kept pre-empting the low-priority holder. A watchdog saw that the bus task had not completed in time and reset the system. JPL fixed it by uploading a change that enabled priority inheritance on that mutex.
This project reconstructs it with a deterministic tick scheduler (Task.java, locked). BusMutex is a plain lock: a task blocked on it does nothing for the owner.
Fix BusMutex so it implements priority inheritance.
Bug report
BUG-MPF-0797 · Priority: Critical (spacecraft resets) · Reported by: flight software
Scheduler.run executes one op per tick, always picking the runnable task with the highest effectivePriority (ties: the task listed first). At the start of every tick it calls mutex.block(t) for each task waiting on the held mutex.
BusMutex must keep the public field owner and implement:
- tryLock(t): take the mutex if free (owner = t), return whether it did
- block(t): t is waiting; the owner's effectivePriority is raised to max(owner.effectivePriority, t.effectivePriority) (priority inheritance)
- unlock(t): if t owns it, release it and restore t.effectivePriority to t.basePriority
With meteo (priority 1, arrives tick 0: LOCK WORK WORK UNLOCK), bus (priority 3, tick 1: LOCK WORK UNLOCK) and comms (priority 2, tick 2: six WORK), the bus task must finish by tick 8 (Watchdog.fed(bus, 8)); the trace is meteo x4, bus x3, comms x6.
Observed: comms runs ticks 2–7 while bus waits; bus finishes at tick 12 and the watchdog resets the lander.
Logs
[vxworks] bc_dist missed deadline; bc_sched watchdog expired
[vxworks] SYSTEM RESET
[ground] reset traced to mutex held by ASI/MET taskThe code as shipped
src/rtos/BusMutex.java (editable)
class BusMutex {
Task owner;
List<Task> waiters = new ArrayList<>();
boolean tryLock(Task t) {
if (owner != null) return false;
owner = t;
waiters.remove(t);
return true;
}
void block(Task t) {
if (!waiters.contains(t)) waiters.add(t);
}
void unlock(Task t) {
if (owner == t) owner = null;
}
}
Read-only context: src/rtos/Task.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.