The Cleanup Job That Ran Against Production — Python Bug Hunt

Briefing

Modelled on the AWS Elastic Load Balancing outage of 24 December 2012 in US-East. Amazon's summary: a maintenance process was inadvertently run against the production ELB state data and deleted part of it. The process was started by one of a small number of developers who had access to production for that purpose, and at the time such runs did not require per-change approval. Load balancers began to fail over Christmas Eve — Netflix streaming was among the services hit — and AWS changed its access controls so that production state changes need specific approval.

This reconstruction's job runner checks only that the operator may reach the target environment. A cleanup written for staging runs happily against production.

Fix run_job so destructive jobs only run where they were meant to, with approval for production.

Bug report

BUG-ELB1224 · Priority: Critical · Reported by: incident review

run_job(job, user, target_env, store, approvals) — job is { name, destructive, env, matches }:

• the user must have access to target_env (access.can_access), else raise PermissionError
• a destructive job must declare job["env"] and it must equal target_env exactly, else raise PermissionError
• a destructive job against "production" additionally needs approvals.get(job["name"]) to be a non-empty change id, else raise PermissionError
• a refused job leaves store["rows"] untouched
• when allowed, a destructive job removes every row for which job["matches"](row) is true; any allowed job returns "ran"

Observed: a staging cleanup run by an engineer with production access deleted production rows.

Logs

[ops] dev-ana ran cleanup-stale-lbs env=production (job authored for staging)
[elb] 1,204 load balancer state rows deleted
[elb] control plane: state missing for lb-7f3a, scaling disabled

The code as shipped

src/ops/jobs.py (editable)

access = bug_require("./access.py")


def run_job(job, user, target_env, store, approvals):
    if not access.can_access(user, target_env):
        raise PermissionError("no access to " + target_env)
    if job["destructive"]:
        store["rows"] = [r for r in store["rows"] if not job["matches"](r)]
    return "ran"

Read-only context: src/ops/access.py.