The Break That Left the Switch — Java Bug Hunt

Briefing

Modelled on the AT&T long-distance network collapse of 15 January 1990: a software update to the 4ESS switches contained a C break statement inside an if that was itself inside a switch. The break was meant to leave the if, but it left the switch, skipping the processing that followed. When one switch reset and came back, the messages it sent made its neighbours reset too, and the failure rippled across the network for about nine hours.

This project is a reconstruction in Java, where break has the same meaning. Handler.handle processes signalling messages: an incoming message from a peer that was marked out of service should bring the peer back into service (when the write buffer is empty) and then be delivered — always.

Fix handle so every incoming message is delivered.

Bug report

BUG-4ESS · Priority: P0 · Reported by: network operations

Handler.handle(state, msg):

• PEER_DOWN: add msg.from to state.peersDown
• INCOMING: - if msg.from is in peersDown AND state.writeBufferDepth == 0: remove it from peersDown and append msg.from to state.inServiceSent - if the buffer is not empty, the peer stays down for now (it will be marked in service by a later message) - in EVERY case, append msg.id to state.delivered, exactly once
• any other type throws IllegalStateException

Handler.handleAll(state, msgs) handles each message in order.

Observed: messages from a recovering peer arriving while the write buffer is busy are silently dropped.

Logs

[4ess] peer NYC-04 back from reset, sending status
[4ess] INCOMING from NYC-04 while buffer depth=3 -> no delivery
[4ess] self-reset triggered

The code as shipped

src/signal/Handler.java (editable)

class Handler {
    static void handle(SwitchState st, Msg m) {
        switch (m.type) {
            case Msg.PEER_DOWN:
                st.peersDown.add(m.from);
                break;
            case Msg.INCOMING:
                if (st.peersDown.contains(m.from)) {
                    if (st.writeBufferDepth == 0) {
                        st.peersDown.remove(m.from);
                        st.inServiceSent.add(m.from);
                    } else {
                        break;
                    }
                }
                st.delivered.add(m.id);
                break;
            default:
                throw new IllegalStateException("unknown message type " + m.type);
        }
    }

    static void handleAll(SwitchState st, List<Msg> msgs) {
        for (Msg m : msgs) handle(st, m);
    }
}

Read-only context: src/signal/Msg.java, src/signal/SwitchState.java.