The Migration That Unblocked Everyone — Python Bug Hunt
Modelled on Facebook's block-list bug (disclosed July 2018): for about a week in late May and early June 2018, a bug temporarily unblocked people that more…
- Language: Python
- Layer: Database
- Difficulty: Medium
- Concepts: Security, State
- Modelled on: Facebook · 2018
- Visible tests: an update sets the flag it carries; a block survives an update that does not mention it
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (privacy))
- Ticket: BUG-BLOCKLIST
- Reported by: trust & safety
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a block survives an update that does not mention it — the block must survive: expected true but got false
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Facebook's block-list bug (disclosed July 2018): for about a week in late May and early June 2018, a bug temporarily unblocked people that more than 800,000 users had blocked — a blocked person could, for instance, contact them on Messenger again. Facebook fixed the bug, restored the blocks and notified the affected users.
Facebook did not publish the code involved; this project is a reconstruction of one way such a bug happens. migrate.py applies a batch of relationship-setting updates, normalising every flag to a boolean — so a flag the update never mentioned is rewritten to False.
Fix apply_updates so an update only changes the fields it actually carries.
Bug report
BUG-BLOCKLIST · Priority: Critical (privacy) · Reported by: trust & safety
apply_updates(rows, updates) — each update is {"a", "b", ...some of "blocked" / "following" / "muted"}:
- the row for (a, b) is found with relations.find; an update with no matching row is skipped
- ONLY the flags present in the update are written (as booleans); every flag the update does not mention keeps its current value
- an update may explicitly set "blocked": False (a real unblock)
- returns the number of rows that matched an update
Observed: a bulk update that only touched "following" cleared "blocked" on every row it matched.
Logs
[migrate] batch rel-settings-0529 applied to 812000 rows
[support] ticket: "someone I blocked just messaged me"The code as shipped
src/social/migrate.py (editable)
relations = bug_require("./relations.py")
FIELDS = ("blocked", "following", "muted")
def apply_updates(rows, updates):
matched = 0
for u in updates:
row = relations.find(rows, u["a"], u["b"])
if row is None:
continue
for field in FIELDS:
row[field] = bool(u.get(field, False))
matched += 1
return matched
Read-only context: src/social/relations.py.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.