The Thruster That Spun It Faster — Python Bug Hunt
Modelled on the loss of JAXA's Hitomi (ASTRO-H) X-ray observatory in March 2016.
- Language: Python
- Layer: Backend
- Difficulty: Hard
- Concepts: Config, State, Validation
- Modelled on: JAXA Hitomi · 2016
- Visible tests: safe-hold damps an x-axis spin; safe-hold damps a z-axis spin
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (loss of vehicle))
- Ticket: BUG-ASTRO-H
- Reported by: attitude control team
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- safe-hold damps a z-axis spin — z rate should be inside the deadband, got 1.5000000000000002: expected a truthy value but got false
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the loss of JAXA's Hitomi (ASTRO-H) X-ray observatory in March 2016. JAXA's investigation found that the attitude system wrongly believed the craft was rotating and tried to correct it, which made it actually rotate; when it then entered safe mode, the thrusters fired using incorrect parameters that had been uploaded after a software change and not properly checked. The firing increased the spin instead of stopping it, and parts of the spacecraft, including the solar array paddles, broke off.
This project is a reconstruction. safehold.py damps body rates by firing the thruster that produces the opposing torque, looked up in a hand-entered command table. The thruster geometry in geometry.py (mounting point r, thrust direction f, torque = r × f) is the ground truth; sim.py applies the resulting torque.
Fix the command table so every entry matches the geometry and safe-hold removes spin on every axis.
Bug report
BUG-ASTRO-H · Priority: Critical (loss of vehicle) · Reported by: attitude control team
COMMAND_TABLE maps a demanded torque ("+x", "-x", "+y", "-y", "+z", "-z") to the thruster that produces it: geometry.torque(COMMAND_TABLE[key]) must be a unit vector along exactly that axis, with that sign.
select(rate) returns, in x, y, z order, the thrusters to fire for one pulse:
- axes with abs(rate) <= DEADBAND (0.05 rad/s) fire nothing
- otherwise fire the thruster demanded by the OPPOSITE sign of that axis' rate (rate > 0 demands "-axis", rate < 0 demands "+axis")
Running sim.run(rate, select, pulses) long enough must bring every axis inside the deadband.
Observed: a z-axis spin gets faster with every safe-hold pulse.
Logs
[aocs] SAFE_HOLD entered rate=(0.00, 0.00, 0.50) rad/s
[aocs] pulse 5 rate_z=1.00
[aocs] pulse 10 rate_z=1.50 — structural limit exceededThe code as shipped
src/aocs/safehold.py (editable)
# Thruster selection for safe-hold rate damping.
# Key: the torque the controller demands.
COMMAND_TABLE = {
"+x": "RCS-2",
"-x": "RCS-1",
"+y": "RCS-3",
"-y": "RCS-4",
"+z": "RCS-6",
"-z": "RCS-5",
}
DEADBAND = 0.05
AXES = ["x", "y", "z"]
def select(rate):
"""Thrusters to fire this pulse to oppose the measured body rate."""
fired = []
for i in range(3):
if abs(rate[i]) <= DEADBAND:
continue
demand = ("-" if rate[i] > 0 else "+") + AXES[i]
fired.append(COMMAND_TABLE[demand])
return fired
Read-only context: src/aocs/geometry.py, src/aocs/sim.py.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.