The Break That Left the Switch — Java Bug Hunt
Modelled on the AT&T long-distance network collapse of 15 January 1990: a software update to the 4ESS switches contained a C break statement inside an if…
- Language: Java
- Layer: Backend
- Difficulty: Hard
- Concepts: State, Networking
- Modelled on: AT&T · 1990
- Visible tests: messages from a healthy peer are delivered; a recovering peer with an empty buffer is marked in service; a busy buffer never drops the message
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: P0)
- Ticket: BUG-4ESS
- Reported by: network operations
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- a busy buffer never drops the message — the message must still be processed: expected [m1] but got []
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the AT&T long-distance network collapse of 15 January 1990: a software update to the 4ESS switches contained a C break statement inside an if that was itself inside a switch. The break was meant to leave the if, but it left the switch, skipping the processing that followed. When one switch reset and came back, the messages it sent made its neighbours reset too, and the failure rippled across the network for about nine hours.
This project is a reconstruction in Java, where break has the same meaning. Handler.handle processes signalling messages: an incoming message from a peer that was marked out of service should bring the peer back into service (when the write buffer is empty) and then be delivered — always.
Fix handle so every incoming message is delivered.
Bug report
BUG-4ESS · Priority: P0 · Reported by: network operations
Handler.handle(state, msg):
- PEER_DOWN: add msg.from to state.peersDown
- INCOMING: - if msg.from is in peersDown AND state.writeBufferDepth == 0: remove it from peersDown and append msg.from to state.inServiceSent - if the buffer is not empty, the peer stays down for now (it will be marked in service by a later message) - in EVERY case, append msg.id to state.delivered, exactly once
- any other type throws IllegalStateException
Handler.handleAll(state, msgs) handles each message in order.
Observed: messages from a recovering peer arriving while the write buffer is busy are silently dropped.
Logs
[4ess] peer NYC-04 back from reset, sending status
[4ess] INCOMING from NYC-04 while buffer depth=3 -> no delivery
[4ess] self-reset triggeredThe code as shipped
src/signal/Handler.java (editable)
class Handler {
static void handle(SwitchState st, Msg m) {
switch (m.type) {
case Msg.PEER_DOWN:
st.peersDown.add(m.from);
break;
case Msg.INCOMING:
if (st.peersDown.contains(m.from)) {
if (st.writeBufferDepth == 0) {
st.peersDown.remove(m.from);
st.inServiceSent.add(m.from);
} else {
break;
}
}
st.delivered.add(m.id);
break;
default:
throw new IllegalStateException("unknown message type " + m.type);
}
}
static void handleAll(SwitchState st, List<Msg> msgs) {
for (Msg m : msgs) handle(st, m);
}
}
Read-only context: src/signal/Msg.java, src/signal/SwitchState.java.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.