The Cleanup Job That Ran Against Production

Medium

· Database

Incident

SEV-1

BUG-ELB1224

Priority: Critical · reported by incident review

Time to fix: 20 min

Failing checks on the shipped build

  • a staging job pointed at production is refused

    wrong environment: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

AWS ELB · 2012

Modelled on the AWS Elastic Load Balancing outage of 24 December 2012 in US-East. Amazon's summary: a maintenance process was inadvertently run against the production ELB state data and deleted part of it. The process was started by one of a small number of developers who had access to production for that purpose, and at the time such runs did not require per-change approval. Load balancers began to fail over Christmas Eve — Netflix streaming was among the services hit — and AWS changed its access controls so that production state changes need specific approval.

This reconstruction's job runner checks only that the operator may reach the target environment. A cleanup written for staging runs happily against production.

Fix run_job so destructive jobs only run where they were meant to, with approval for production.

Code map

access = bug_require("./access.py")


def run_job(job, user, target_env, store, approvals):
    if not access.can_access(user, target_env):
        raise PermissionError("no access to " + target_env)
    if job["destructive"]:
        store["rows"] = [r for r in store["rows"] if not job["matches"](r)]
    return "ran"

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Cleanup Job That Ran Against Production — Bug Hunt

Modelled on the AWS Elastic Load Balancing outage of 24 December 2012 in US-East.

  • Language: Python
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Access, Config, Data
  • Modelled on: AWS ELB · 2012
  • Visible tests: a destructive job on its own dev environment runs; a staging job pointed at production is refused
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-ELB1224
  • Reported by: incident review
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a staging job pointed at production is refused — wrong environment: expected an error to be raised

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the AWS Elastic Load Balancing outage of 24 December 2012 in US-East. Amazon's summary: a maintenance process was inadvertently run against the production ELB state data and deleted part of it. The process was started by one of a small number of developers who had access to production for that purpose, and at the time such runs did not require per-change approval. Load balancers began to fail over Christmas Eve — Netflix streaming was among the services hit — and AWS changed its access controls so that production state changes need specific approval.

This reconstruction's job runner checks only that the operator may reach the target environment. A cleanup written for staging runs happily against production.

Fix run_job so destructive jobs only run where they were meant to, with approval for production.

Bug report

BUG-ELB1224 · Priority: Critical · Reported by: incident review

run_job(job, user, target_env, store, approvals) — job is { name, destructive, env, matches }:

  • the user must have access to target_env (access.can_access), else raise PermissionError
  • a destructive job must declare job["env"] and it must equal target_env exactly, else raise PermissionError
  • a destructive job against "production" additionally needs approvals.get(job["name"]) to be a non-empty change id, else raise PermissionError
  • a refused job leaves store["rows"] untouched
  • when allowed, a destructive job removes every row for which job["matches"](row) is true; any allowed job returns "ran"

Observed: a staging cleanup run by an engineer with production access deleted production rows.

Logs

[ops] dev-ana ran cleanup-stale-lbs env=production (job authored for staging)
[elb] 1,204 load balancer state rows deleted
[elb] control plane: state missing for lb-7f3a, scaling disabled

The code as shipped

src/ops/jobs.py (editable)

access = bug_require("./access.py")


def run_job(job, user, target_env, store, approvals):
    if not access.can_access(user, target_env):
        raise PermissionError("no access to " + target_env)
    if job["destructive"]:
        store["rows"] = [r for r in store["rows"] if not job["matches"](r)]
    return "ran"

Read-only context: src/ops/access.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.