The Uploader Nobody Checksummed

Easy

· Backend

Incident

SEV-1

BUG-UPLOADER

Priority: Critical (supply chain) · reported by security

Time to fix: 10 min

Failing checks on the shipped build

  • a tampered script is refused and never runs

    must raise IntegrityError: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Codecov · 2021

Modelled on the Codecov Bash Uploader compromise (disclosed April 2021): an attacker modified the uploader script Codecov served, and for about two months CI pipelines that downloaded it and piped it straight into a shell sent their environment variables — credentials included — to a server the attacker controlled. It was noticed when a customer found the script did not match its published checksum.

installer.py downloads the uploader and runs it. It computes the checksum, but a mismatch only produces a warning.

Fix install so a script is run only when its SHA-256 matches the published value.

Code map

import hashlib

errors = bug_require("./errors.py")


def install(fetch, published_sha256, run, log):
    script = fetch()
    digest = hashlib.sha256(script).hexdigest()
    if digest != published_sha256:
        log.append("warning: checksum mismatch " + digest)
    run(script)
    return digest

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Uploader Nobody Checksummed — Python Bug Hunt

Modelled on the Codecov Bash Uploader compromise (disclosed April 2021): an attacker modified the uploader script Codecov served, and for about two months…

  • Language: Python
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Security, Validation
  • Modelled on: Codecov · 2021
  • Visible tests: a script matching its checksum runs; a tampered script is refused and never runs
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (supply chain))
  • Ticket: BUG-UPLOADER
  • Reported by: security
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • a tampered script is refused and never runs — must raise IntegrityError: expected a truthy value but got false

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Codecov Bash Uploader compromise (disclosed April 2021): an attacker modified the uploader script Codecov served, and for about two months CI pipelines that downloaded it and piped it straight into a shell sent their environment variables — credentials included — to a server the attacker controlled. It was noticed when a customer found the script did not match its published checksum.

installer.py downloads the uploader and runs it. It computes the checksum, but a mismatch only produces a warning.

Fix install so a script is run only when its SHA-256 matches the published value.

Bug report

BUG-UPLOADER · Priority: Critical (supply chain) · Reported by: security

install(fetch, published_sha256, run, log):

  • script = fetch() (bytes); digest = the SHA-256 hex digest of script
  • the published value is compared case-insensitively, surrounding whitespace ignored
  • match: call run(script) exactly once, return the digest (lower-case hex), append nothing to log
  • mismatch, or a missing/empty published value: append one line starting with "refused:" to log, raise errors.IntegrityError, and never call run

Observed: a tampered uploader logged "warning: checksum mismatch" and ran anyway.

Logs

[ci] fetched codecov uploader (5.2 KB)
[ci] warning: checksum mismatch 9f2c...e1
[ci] running uploader

The code as shipped

src/ci/installer.py (editable)

import hashlib

errors = bug_require("./errors.py")


def install(fetch, published_sha256, run, log):
    script = fetch()
    digest = hashlib.sha256(script).hexdigest()
    if digest != published_sha256:
        log.append("warning: checksum mismatch " + digest)
    run(script)
    return digest

Read-only context: src/ci/errors.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.