Two Waypoints, One Name — Java Bug Hunt
Modelled on the NATS flight-plan failure of 28 August 2023, the UK bank-holiday Monday when air traffic control had to fall back to manual flight-plan…
- Language: Java
- Layer: Backend
- Difficulty: Hard
- Concepts: Parsing, Safety, Resilience
- Modelled on: NATS · 2023
- Visible tests: a route of unique waypoints resolves; a duplicated name resolves by the route around it; one bad plan goes to the manual queue and the rest continue
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-NATS0828
- Reported by: operations
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- a duplicated name resolves by the route around it — cannot resolve KOBA
- one bad plan goes to the manual queue and the rest continue — cannot resolve NOWHERE
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the NATS flight-plan failure of 28 August 2023, the UK bank-holiday Monday when air traffic control had to fall back to manual flight-plan processing. NATS's investigation found that a single flight plan referred to two different waypoints that share the same identifier. The flight-plan processing system could not resolve the route, treated it as a critical error, and — to protect safety — both the primary and the backup system stopped processing, triggering widespread cancellations. This reconstruction's RouteResolver refuses any identifier with more than one match, and FlightPlanProcessor.processAll lets that one exception stop the whole batch.
Fix the resolver to use the route's context, and the processor to send a single unresolvable plan to the manual queue.
Bug report
BUG-NATS0828 · Priority: Critical · Reported by: operations
RouteResolver.resolve(route, db):
- an id with exactly one entry resolves to it
- an unknown id throws RouteException
- an id with several entries resolves to the candidate NEAREST (squared distance, Waypoint.dist2) to the previous resolved waypoint of the route
- if the ambiguous id is the first waypoint (no context), or two candidates are equally near, throw RouteException
FlightPlanProcessor.processAll(plans, db):
- each plan is resolved independently; success -> its callsign is appended to
processed - a RouteException -> its callsign is appended to
manualand processing CONTINUES with the next plan; processAll never throws for one bad plan
Observed: one plan with a duplicated waypoint name stopped processing for every plan behind it.
Logs
[fprsa] plan BAW123 route ... KOBA ... -> 2 candidates for KOBA
[fprsa] critical exception, entering maintenance mode
[fprsa-backup] same plan, same exception, entering maintenance modeThe code as shipped
src/atc/FlightPlanProcessor.java (editable)
class RouteResolver {
static List<Waypoint> resolve(List<String> route, WaypointDb db) {
List<Waypoint> out = new ArrayList<>();
for (String id : route) {
List<Waypoint> candidates = db.lookup(id);
if (candidates.size() != 1) {
throw new RouteException("cannot resolve " + id);
}
out.add(candidates.get(0));
}
return out;
}
}
class FlightPlanProcessor {
final List<String> processed = new ArrayList<>();
final List<String> manual = new ArrayList<>();
void processAll(List<FlightPlan> plans, WaypointDb db) {
for (FlightPlan p : plans) {
RouteResolver.resolve(p.route, db);
processed.add(p.callsign);
}
}
}Read-only context: src/atc/Charts.java, src/atc/Waypoints.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.