The Uploader Nobody Checksummed — Python Bug Hunt
Modelled on the Codecov Bash Uploader compromise (disclosed April 2021): an attacker modified the uploader script Codecov served, and for about two months…
- Language: Python
- Layer: Backend
- Difficulty: Easy
- Concepts: Security, Validation
- Modelled on: Codecov · 2021
- Visible tests: a script matching its checksum runs; a tampered script is refused and never runs
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (supply chain))
- Ticket: BUG-UPLOADER
- Reported by: security
- Time to fix: 10 minutes, the target for an easy incident
Failing checks on the shipped build
- a tampered script is refused and never runs — must raise IntegrityError: expected a truthy value but got false
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the Codecov Bash Uploader compromise (disclosed April 2021): an attacker modified the uploader script Codecov served, and for about two months CI pipelines that downloaded it and piped it straight into a shell sent their environment variables — credentials included — to a server the attacker controlled. It was noticed when a customer found the script did not match its published checksum.
installer.py downloads the uploader and runs it. It computes the checksum, but a mismatch only produces a warning.
Fix install so a script is run only when its SHA-256 matches the published value.
Bug report
BUG-UPLOADER · Priority: Critical (supply chain) · Reported by: security
install(fetch, published_sha256, run, log):
- script = fetch() (bytes); digest = the SHA-256 hex digest of script
- the published value is compared case-insensitively, surrounding whitespace ignored
- match: call run(script) exactly once, return the digest (lower-case hex), append nothing to log
- mismatch, or a missing/empty published value: append one line starting with "refused:" to log, raise errors.IntegrityError, and never call run
Observed: a tampered uploader logged "warning: checksum mismatch" and ran anyway.
Logs
[ci] fetched codecov uploader (5.2 KB)
[ci] warning: checksum mismatch 9f2c...e1
[ci] running uploaderThe code as shipped
src/ci/installer.py (editable)
import hashlib
errors = bug_require("./errors.py")
def install(fetch, published_sha256, run, log):
script = fetch()
digest = hashlib.sha256(script).hexdigest()
if digest != published_sha256:
log.append("warning: checksum mismatch " + digest)
run(script)
return digest
Read-only context: src/ci/errors.py.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.