Priority Inversion on Mars

Hard

· Backend

Incident

SEV-1

BUG-MPF-0797

Priority: Critical (spacecraft resets) · reported by flight software

Time to fix: 35 min

Failing checks on the shipped build

  • the bus task meets its watchdog deadline

    bus finished at tick 12: expected condition to hold

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

NASA Mars Pathfinder · 1997

Modelled on Mars Pathfinder (July 1997): days after landing, the spacecraft began resetting itself. JPL traced it to priority inversion in its VxWorks software: a low-priority meteorological task held a mutex guarding the information bus, the high-priority bus-management task blocked waiting for it, and medium-priority tasks kept pre-empting the low-priority holder. A watchdog saw that the bus task had not completed in time and reset the system. JPL fixed it by uploading a change that enabled priority inheritance on that mutex.

This project reconstructs it with a deterministic tick scheduler (Task.java, locked). BusMutex is a plain lock: a task blocked on it does nothing for the owner.

Fix BusMutex so it implements priority inheritance.

Code map

class BusMutex {
    Task owner;
    List<Task> waiters = new ArrayList<>();

    boolean tryLock(Task t) {
        if (owner != null) return false;
        owner = t;
        waiters.remove(t);
        return true;
    }

    void block(Task t) {
        if (!waiters.contains(t)) waiters.add(t);
    }

    void unlock(Task t) {
        if (owner == t) owner = null;
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Priority Inversion on Mars — Java Bug Hunt

Modelled on Mars Pathfinder (July 1997): days after landing, the spacecraft began resetting itself.

  • Language: Java
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Concurrency, State
  • Modelled on: NASA Mars Pathfinder · 1997
  • Visible tests: without contention the medium task simply pre-empts meteo; the bus task meets its watchdog deadline
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (spacecraft resets))
  • Ticket: BUG-MPF-0797
  • Reported by: flight software
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the bus task meets its watchdog deadline — bus finished at tick 12: expected condition to hold

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Mars Pathfinder (July 1997): days after landing, the spacecraft began resetting itself. JPL traced it to priority inversion in its VxWorks software: a low-priority meteorological task held a mutex guarding the information bus, the high-priority bus-management task blocked waiting for it, and medium-priority tasks kept pre-empting the low-priority holder. A watchdog saw that the bus task had not completed in time and reset the system. JPL fixed it by uploading a change that enabled priority inheritance on that mutex.

This project reconstructs it with a deterministic tick scheduler (Task.java, locked). BusMutex is a plain lock: a task blocked on it does nothing for the owner.

Fix BusMutex so it implements priority inheritance.

Bug report

BUG-MPF-0797 · Priority: Critical (spacecraft resets) · Reported by: flight software

Scheduler.run executes one op per tick, always picking the runnable task with the highest effectivePriority (ties: the task listed first). At the start of every tick it calls mutex.block(t) for each task waiting on the held mutex.

BusMutex must keep the public field owner and implement:

  • tryLock(t): take the mutex if free (owner = t), return whether it did
  • block(t): t is waiting; the owner's effectivePriority is raised to max(owner.effectivePriority, t.effectivePriority) (priority inheritance)
  • unlock(t): if t owns it, release it and restore t.effectivePriority to t.basePriority

With meteo (priority 1, arrives tick 0: LOCK WORK WORK UNLOCK), bus (priority 3, tick 1: LOCK WORK UNLOCK) and comms (priority 2, tick 2: six WORK), the bus task must finish by tick 8 (Watchdog.fed(bus, 8)); the trace is meteo x4, bus x3, comms x6.

Observed: comms runs ticks 2–7 while bus waits; bus finishes at tick 12 and the watchdog resets the lander.

Logs

[vxworks] bc_dist missed deadline; bc_sched watchdog expired
[vxworks] SYSTEM RESET
[ground] reset traced to mutex held by ASI/MET task

The code as shipped

src/rtos/BusMutex.java (editable)

class BusMutex {
    Task owner;
    List<Task> waiters = new ArrayList<>();

    boolean tryLock(Task t) {
        if (owner != null) return false;
        owner = t;
        waiters.remove(t);
        return true;
    }

    void block(Task t) {
        if (!waiters.contains(t)) waiters.add(t);
    }

    void unlock(Task t) {
        if (owner == t) owner = null;
    }
}

Read-only context: src/rtos/Task.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.