Locked Out of Everything — Python Bug Hunt

Briefing

Inspired by Facebook's October 2021 outage: a routine command withdrew the routes announcing their OWN backbone — including the ones engineers needed to log in and fix it. Badge readers stopped working. Someone reportedly needed an angle grinder.

routes.py plans a route withdrawal. It must protect the management network and never withdraw everything.

Bug report

BUG-BGP-104 · Priority: Existential · Reported by: network eng

plan_withdrawal(announced, to_withdraw) rules:

• "mgmt/16" is sacred: it survives every withdrawal
• at least one route must remain announced, else raise ValueError
• returns the routes still announced, in their original order

Observed: the audit command withdrew mgmt/16 and then everything else.

Logs

[bgp] withdrawing 214/214 routes
[noc] dashboards unreachable; DNS gone; door badges offline

The code as shipped

src/net/routes.py (editable)

# Plans a BGP route withdrawal.

MGMT_ROUTE = "mgmt/16"

def plan_withdrawal(announced, to_withdraw):
    remaining = []
    for route in announced:
        if route not in to_withdraw:
            remaining.append(route)
    return remaining