Nobody Left to Fly Christmas — Python Bug Hunt
Modelled on American Airlines, November 2017. A glitch in the airline's pilot scheduling system let pilots drop trips over the December holiday period, and…
- Language: Python
- Layer: Backend
- Difficulty: Easy
- Concepts: Validation, State, Time
- Modelled on: American Airlines · 2017
- Visible tests: a drop from an overstaffed trip is approved; a drop that would leave the trip short is denied
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-CREW-1712
- Reported by: crew scheduling
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- a drop that would leave the trip short is denied — the trip needs both pilots: expected false but got true
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on American Airlines, November 2017. A glitch in the airline's pilot scheduling system let pilots drop trips over the December holiday period, and thousands of flights — reportedly around 15,000 — ended up without a pilot assigned. The airline said it would pay pilots extra to pick the trips back up. The precise root cause was not published.
This reconstruction models the failure as a trip-drop request that is approved whenever the pilot is on the trip, without checking what the trip is left with.
Fix request_drop so no approval leaves a trip under its minimum crew.
Bug report
BUG-CREW-1712 · Priority: Critical · Reported by: crew scheduling
request_drop(schedule, trip_id, pilot) returns True and removes the pilot from schedule[trip_id]["assigned"] ONLY if:
- the trip exists
- the pilot is currently assigned to it
- after removal the trip still has at least rules.min_crew(trip) pilots
Otherwise it returns False and leaves the schedule unchanged.
Observed: drops on fully-staffed December trips were approved, leaving those trips with fewer pilots than they need.
Logs
[crew] drop approved trip=AA1217 pilot=P-4410 assigned=1 required=2
[crew] drop approved trip=AA1217 pilot=P-2981 assigned=0 required=2
[crew] 2017-12 open trips without crew: risingThe code as shipped
src/crew/drops.py (editable)
rules = bug_require("./rules.py")
def request_drop(schedule, trip_id, pilot):
trip = schedule.get(trip_id)
if trip is None:
return False
if pilot not in trip["assigned"]:
return False
trip["assigned"].remove(pilot)
return True
Read-only context: src/crew/rules.py.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.