Locked Out of Everything

Medium

· Backend

Incident

SEV-1

BUG-BGP-104

Priority: Existential · reported by network eng

Time to fix: 20 min

Failing checks on the shipped build

  • the management route is untouchable

    expected ["mgmt/16"] but got []

  • withdrawing every route is refused

    cannot go fully dark: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Facebook · 2021

Inspired by Facebook's October 2021 outage: a routine command withdrew the routes announcing their OWN backbone — including the ones engineers needed to log in and fix it. Badge readers stopped working. Someone reportedly needed an angle grinder.

routes.py plans a route withdrawal. It must protect the management network and never withdraw everything.

Code map

# Plans a BGP route withdrawal.

MGMT_ROUTE = "mgmt/16"

def plan_withdrawal(announced, to_withdraw):
    remaining = []
    for route in announced:
        if route not in to_withdraw:
            remaining.append(route)
    return remaining

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Locked Out of Everything — Python Bug Hunt

Inspired by Facebook's October 2021 outage: a routine command withdrew the routes announcing their OWN backbone — including the ones engineers needed to log…

  • Language: Python
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Networking, Validation
  • Modelled on: Facebook · 2021
  • Visible tests: a normal withdrawal removes the listed routes; the management route is untouchable; withdrawing every route is refused
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Existential)
  • Ticket: BUG-BGP-104
  • Reported by: network eng
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • the management route is untouchable — expected ["mgmt/16"] but got []
  • withdrawing every route is refused — cannot go fully dark: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by Facebook's October 2021 outage: a routine command withdrew the routes announcing their OWN backbone — including the ones engineers needed to log in and fix it. Badge readers stopped working. Someone reportedly needed an angle grinder.

routes.py plans a route withdrawal. It must protect the management network and never withdraw everything.

Bug report

BUG-BGP-104 · Priority: Existential · Reported by: network eng

plan_withdrawal(announced, to_withdraw) rules:

  • "mgmt/16" is sacred: it survives every withdrawal
  • at least one route must remain announced, else raise ValueError
  • returns the routes still announced, in their original order

Observed: the audit command withdrew mgmt/16 and then everything else.

Logs

[bgp] withdrawing 214/214 routes
[noc] dashboards unreachable; DNS gone; door badges offline

The code as shipped

src/net/routes.py (editable)

# Plans a BGP route withdrawal.

MGMT_ROUTE = "mgmt/16"

def plan_withdrawal(announced, to_withdraw):
    remaining = []
    for route in announced:
        if route not in to_withdraw:
            remaining.append(route)
    return remaining

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.