The Migration That Unblocked Everyone

Medium

· Database

Incident

SEV-1

BUG-BLOCKLIST

Priority: Critical (privacy) · reported by trust & safety

Time to fix: 20 min

Failing checks on the shipped build

  • a block survives an update that does not mention it

    the block must survive: expected true but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Facebook · 2018

Modelled on Facebook's block-list bug (disclosed July 2018): for about a week in late May and early June 2018, a bug temporarily unblocked people that more than 800,000 users had blocked — a blocked person could, for instance, contact them on Messenger again. Facebook fixed the bug, restored the blocks and notified the affected users.

Facebook did not publish the code involved; this project is a reconstruction of one way such a bug happens. migrate.py applies a batch of relationship-setting updates, normalising every flag to a boolean — so a flag the update never mentioned is rewritten to False.

Fix apply_updates so an update only changes the fields it actually carries.

Code map

relations = bug_require("./relations.py")

FIELDS = ("blocked", "following", "muted")


def apply_updates(rows, updates):
    matched = 0
    for u in updates:
        row = relations.find(rows, u["a"], u["b"])
        if row is None:
            continue
        for field in FIELDS:
            row[field] = bool(u.get(field, False))
        matched += 1
    return matched

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Migration That Unblocked Everyone — Python Bug Hunt

Modelled on Facebook's block-list bug (disclosed July 2018): for about a week in late May and early June 2018, a bug temporarily unblocked people that more…

  • Language: Python
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Security, State
  • Modelled on: Facebook · 2018
  • Visible tests: an update sets the flag it carries; a block survives an update that does not mention it
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (privacy))
  • Ticket: BUG-BLOCKLIST
  • Reported by: trust & safety
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a block survives an update that does not mention it — the block must survive: expected true but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Facebook's block-list bug (disclosed July 2018): for about a week in late May and early June 2018, a bug temporarily unblocked people that more than 800,000 users had blocked — a blocked person could, for instance, contact them on Messenger again. Facebook fixed the bug, restored the blocks and notified the affected users.

Facebook did not publish the code involved; this project is a reconstruction of one way such a bug happens. migrate.py applies a batch of relationship-setting updates, normalising every flag to a boolean — so a flag the update never mentioned is rewritten to False.

Fix apply_updates so an update only changes the fields it actually carries.

Bug report

BUG-BLOCKLIST · Priority: Critical (privacy) · Reported by: trust & safety

apply_updates(rows, updates) — each update is {"a", "b", ...some of "blocked" / "following" / "muted"}:

  • the row for (a, b) is found with relations.find; an update with no matching row is skipped
  • ONLY the flags present in the update are written (as booleans); every flag the update does not mention keeps its current value
  • an update may explicitly set "blocked": False (a real unblock)
  • returns the number of rows that matched an update

Observed: a bulk update that only touched "following" cleared "blocked" on every row it matched.

Logs

[migrate] batch rel-settings-0529 applied to 812000 rows
[support] ticket: "someone I blocked just messaged me"

The code as shipped

src/social/migrate.py (editable)

relations = bug_require("./relations.py")

FIELDS = ("blocked", "following", "muted")


def apply_updates(rows, updates):
    matched = 0
    for u in updates:
        row = relations.find(rows, u["a"], u["b"])
        if row is None:
            continue
        for field in FIELDS:
            row[field] = bool(u.get(field, False))
        matched += 1
    return matched

Read-only context: src/social/relations.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.