Priority Inversion on Mars — Java Bug Hunt

Briefing

Modelled on Mars Pathfinder (July 1997): days after landing, the spacecraft began resetting itself. JPL traced it to priority inversion in its VxWorks software: a low-priority meteorological task held a mutex guarding the information bus, the high-priority bus-management task blocked waiting for it, and medium-priority tasks kept pre-empting the low-priority holder. A watchdog saw that the bus task had not completed in time and reset the system. JPL fixed it by uploading a change that enabled priority inheritance on that mutex.

This project reconstructs it with a deterministic tick scheduler (Task.java, locked). BusMutex is a plain lock: a task blocked on it does nothing for the owner.

Fix BusMutex so it implements priority inheritance.

Bug report

BUG-MPF-0797 · Priority: Critical (spacecraft resets) · Reported by: flight software

Scheduler.run executes one op per tick, always picking the runnable task with the highest effectivePriority (ties: the task listed first). At the start of every tick it calls mutex.block(t) for each task waiting on the held mutex.

BusMutex must keep the public field owner and implement:

• tryLock(t): take the mutex if free (owner = t), return whether it did
• block(t): t is waiting; the owner's effectivePriority is raised to max(owner.effectivePriority, t.effectivePriority) (priority inheritance)
• unlock(t): if t owns it, release it and restore t.effectivePriority to t.basePriority

With meteo (priority 1, arrives tick 0: LOCK WORK WORK UNLOCK), bus (priority 3, tick 1: LOCK WORK UNLOCK) and comms (priority 2, tick 2: six WORK), the bus task must finish by tick 8 (Watchdog.fed(bus, 8)); the trace is meteo x4, bus x3, comms x6.

Observed: comms runs ticks 2–7 while bus waits; bus finishes at tick 12 and the watchdog resets the lander.

Logs

[vxworks] bc_dist missed deadline; bc_sched watchdog expired
[vxworks] SYSTEM RESET
[ground] reset traced to mutex held by ASI/MET task

The code as shipped

src/rtos/BusMutex.java (editable)

class BusMutex {
    Task owner;
    List<Task> waiters = new ArrayList<>();

    boolean tryLock(Task t) {
        if (owner != null) return false;
        owner = t;
        waiters.remove(t);
        return true;
    }

    void block(Task t) {
        if (!waiters.contains(t)) waiters.add(t);
    }

    void unlock(Task t) {
        if (owner == t) owner = null;
    }
}

Read-only context: src/rtos/Task.java.