The Cleanup Job That Ran Against Production — Bug Hunt
Modelled on the AWS Elastic Load Balancing outage of 24 December 2012 in US-East.
- Language: Python
- Layer: Database
- Difficulty: Medium
- Concepts: Access, Config, Data
- Modelled on: AWS ELB · 2012
- Visible tests: a destructive job on its own dev environment runs; a staging job pointed at production is refused
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-ELB1224
- Reported by: incident review
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- a staging job pointed at production is refused — wrong environment: expected an error to be raised
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the AWS Elastic Load Balancing outage of 24 December 2012 in US-East. Amazon's summary: a maintenance process was inadvertently run against the production ELB state data and deleted part of it. The process was started by one of a small number of developers who had access to production for that purpose, and at the time such runs did not require per-change approval. Load balancers began to fail over Christmas Eve — Netflix streaming was among the services hit — and AWS changed its access controls so that production state changes need specific approval.
This reconstruction's job runner checks only that the operator may reach the target environment. A cleanup written for staging runs happily against production.
Fix run_job so destructive jobs only run where they were meant to, with approval for production.
Bug report
BUG-ELB1224 · Priority: Critical · Reported by: incident review
run_job(job, user, target_env, store, approvals) — job is { name, destructive, env, matches }:
- the user must have access to target_env (access.can_access), else raise PermissionError
- a destructive job must declare job["env"] and it must equal target_env exactly, else raise PermissionError
- a destructive job against "production" additionally needs approvals.get(job["name"]) to be a non-empty change id, else raise PermissionError
- a refused job leaves store["rows"] untouched
- when allowed, a destructive job removes every row for which job["matches"](row) is true; any allowed job returns "ran"
Observed: a staging cleanup run by an engineer with production access deleted production rows.
Logs
[ops] dev-ana ran cleanup-stale-lbs env=production (job authored for staging)
[elb] 1,204 load balancer state rows deleted
[elb] control plane: state missing for lb-7f3a, scaling disabledThe code as shipped
src/ops/jobs.py (editable)
access = bug_require("./access.py")
def run_job(job, user, target_env, store, approvals):
if not access.can_access(user, target_env):
raise PermissionError("no access to " + target_env)
if job["destructive"]:
store["rows"] = [r for r in store["rows"] if not job["matches"](r)]
return "ran"
Read-only context: src/ops/access.py.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.