The Track That Forgot Its Past

Hard

· Backend

Incident

SEV-1

BUG-TRACK-HIST

Priority: Critical (safety) · reported by perception review

Time to fix: 35 min

Failing checks on the shipped build

  • reclassification keeps the history

    moving 2 m/s towards the lane: expected [0.0, 0.0] but got null

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Uber ATG · 2018

Modelled on the Uber ATG test-vehicle crash in Tempe, Arizona, March 2018. The NTSB's investigation found that the automated driving system detected the pedestrian several seconds before impact but repeatedly changed its classification of her — between vehicle, bicycle and "other". Each time the classification changed, the system did not use the object's previous tracking history, so it could not form a path prediction for her in time.

This reconstruction's tracker starts a fresh, empty history whenever an object's label changes, even though it is the same track id. An object that keeps being relabelled never accumulates the two observations a motion estimate needs.

Fix Tracker so an object's history survives reclassification.

Code map

motion = bug_require("./motion.py")


class Tracker:
    def __init__(self):
        self.tracks = {}

    def observe(self, track_id, label, t, x, y):
        track = self.tracks.get(track_id)
        if track is None or track["label"] != label:
            track = {"label": label, "history": []}
            self.tracks[track_id] = track
        track["history"].append((t, x, y))

    def label(self, track_id):
        track = self.tracks.get(track_id)
        return track["label"] if track else None

    def predict(self, track_id, dt):
        track = self.tracks.get(track_id)
        if track is None:
            return None
        v = motion.velocity(track["history"])
        if v is None:
            return None
        t, x, y = track["history"][-1]
        return [round(x + v[0] * dt, 3), round(y + v[1] * dt, 3)]

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Track That Forgot Its Past — Python Bug Hunt

Modelled on the Uber ATG test-vehicle crash in Tempe, Arizona, March 2018.

  • Language: Python
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: State, Safety
  • Modelled on: Uber ATG · 2018
  • Visible tests: a steadily labelled object gets a prediction; reclassification keeps the history
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (safety))
  • Ticket: BUG-TRACK-HIST
  • Reported by: perception review
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • reclassification keeps the history — moving 2 m/s towards the lane: expected [0.0, 0.0] but got null

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Uber ATG test-vehicle crash in Tempe, Arizona, March 2018. The NTSB's investigation found that the automated driving system detected the pedestrian several seconds before impact but repeatedly changed its classification of her — between vehicle, bicycle and "other". Each time the classification changed, the system did not use the object's previous tracking history, so it could not form a path prediction for her in time.

This reconstruction's tracker starts a fresh, empty history whenever an object's label changes, even though it is the same track id. An object that keeps being relabelled never accumulates the two observations a motion estimate needs.

Fix Tracker so an object's history survives reclassification.

Bug report

BUG-TRACK-HIST · Priority: Critical (safety) · Reported by: perception review

Tracker, per track_id:

  • observe(track_id, label, t, x, y) appends (t, x, y) to that track's history and sets its label to the latest one. A change of label is a reclassification of the SAME object: the history is kept.
  • label(track_id) -> the latest label, or None for an unknown id
  • predict(track_id, dt) -> [x, y] extrapolated dt seconds past the last observation using motion.velocity(history) (first to last observation), each coordinate rounded to 3 places; None for an unknown id or when velocity is None (fewer than two observations)
  • tracks are independent: nothing about one id affects another

Observed: a pedestrian observed for three frames as vehicle, bicycle, other has no prediction at all.

Logs

[perception] track 17 label vehicle -> bicycle (history reset)
[perception] track 17 label bicycle -> other (history reset)
[planner] track 17: no predicted path

The code as shipped

src/perception/tracker.py (editable)

motion = bug_require("./motion.py")


class Tracker:
    def __init__(self):
        self.tracks = {}

    def observe(self, track_id, label, t, x, y):
        track = self.tracks.get(track_id)
        if track is None or track["label"] != label:
            track = {"label": label, "history": []}
            self.tracks[track_id] = track
        track["history"].append((t, x, y))

    def label(self, track_id):
        track = self.tracks.get(track_id)
        return track["label"] if track else None

    def predict(self, track_id, dt):
        track = self.tracks.get(track_id)
        if track is None:
            return None
        v = motion.velocity(track["history"])
        if v is None:
            return None
        t, x, y = track["history"][-1]
        return [round(x + v[0] * dt, 3), round(y + v[1] * dt, 3)]

Read-only context: src/perception/motion.py.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Python bug hunts.