The Break That Left the Switch

Hard

· Backend

Incident

SEV-1

BUG-4ESS

Priority: P0 · reported by network operations

Time to fix: 35 min

Failing checks on the shipped build

  • a busy buffer never drops the message

    the message must still be processed: expected [m1] but got []

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

AT&T · 1990

Modelled on the AT&T long-distance network collapse of 15 January 1990: a software update to the 4ESS switches contained a C break statement inside an if that was itself inside a switch. The break was meant to leave the if, but it left the switch, skipping the processing that followed. When one switch reset and came back, the messages it sent made its neighbours reset too, and the failure rippled across the network for about nine hours.

This project is a reconstruction in Java, where break has the same meaning. Handler.handle processes signalling messages: an incoming message from a peer that was marked out of service should bring the peer back into service (when the write buffer is empty) and then be delivered — always.

Fix handle so every incoming message is delivered.

Code map

class Handler {
    static void handle(SwitchState st, Msg m) {
        switch (m.type) {
            case Msg.PEER_DOWN:
                st.peersDown.add(m.from);
                break;
            case Msg.INCOMING:
                if (st.peersDown.contains(m.from)) {
                    if (st.writeBufferDepth == 0) {
                        st.peersDown.remove(m.from);
                        st.inServiceSent.add(m.from);
                    } else {
                        break;
                    }
                }
                st.delivered.add(m.id);
                break;
            default:
                throw new IllegalStateException("unknown message type " + m.type);
        }
    }

    static void handleAll(SwitchState st, List<Msg> msgs) {
        for (Msg m : msgs) handle(st, m);
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Break That Left the Switch — Java Bug Hunt

Modelled on the AT&T long-distance network collapse of 15 January 1990: a software update to the 4ESS switches contained a C break statement inside an if…

  • Language: Java
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: State, Networking
  • Modelled on: AT&T · 1990
  • Visible tests: messages from a healthy peer are delivered; a recovering peer with an empty buffer is marked in service; a busy buffer never drops the message
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: P0)
  • Ticket: BUG-4ESS
  • Reported by: network operations
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a busy buffer never drops the message — the message must still be processed: expected [m1] but got []

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the AT&T long-distance network collapse of 15 January 1990: a software update to the 4ESS switches contained a C break statement inside an if that was itself inside a switch. The break was meant to leave the if, but it left the switch, skipping the processing that followed. When one switch reset and came back, the messages it sent made its neighbours reset too, and the failure rippled across the network for about nine hours.

This project is a reconstruction in Java, where break has the same meaning. Handler.handle processes signalling messages: an incoming message from a peer that was marked out of service should bring the peer back into service (when the write buffer is empty) and then be delivered — always.

Fix handle so every incoming message is delivered.

Bug report

BUG-4ESS · Priority: P0 · Reported by: network operations

Handler.handle(state, msg):

  • PEER_DOWN: add msg.from to state.peersDown
  • INCOMING: - if msg.from is in peersDown AND state.writeBufferDepth == 0: remove it from peersDown and append msg.from to state.inServiceSent - if the buffer is not empty, the peer stays down for now (it will be marked in service by a later message) - in EVERY case, append msg.id to state.delivered, exactly once
  • any other type throws IllegalStateException

Handler.handleAll(state, msgs) handles each message in order.

Observed: messages from a recovering peer arriving while the write buffer is busy are silently dropped.

Logs

[4ess] peer NYC-04 back from reset, sending status
[4ess] INCOMING from NYC-04 while buffer depth=3 -> no delivery
[4ess] self-reset triggered

The code as shipped

src/signal/Handler.java (editable)

class Handler {
    static void handle(SwitchState st, Msg m) {
        switch (m.type) {
            case Msg.PEER_DOWN:
                st.peersDown.add(m.from);
                break;
            case Msg.INCOMING:
                if (st.peersDown.contains(m.from)) {
                    if (st.writeBufferDepth == 0) {
                        st.peersDown.remove(m.from);
                        st.inServiceSent.add(m.from);
                    } else {
                        break;
                    }
                }
                st.delivered.add(m.id);
                break;
            default:
                throw new IllegalStateException("unknown message type " + m.type);
        }
    }

    static void handleAll(SwitchState st, List<Msg> msgs) {
        for (Msg m : msgs) handle(st, m);
    }
}

Read-only context: src/signal/Msg.java, src/signal/SwitchState.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.