Somebody Else's Videos in My Takeout

Medium

· Database

Incident

SEV-1

BUG-TAKEOUT-1121

Priority: Critical (privacy) · reported by user report

Time to fix: 20 min

Failing checks on the shipped build

  • interleaved jobs keep their own items

    u1 must not receive u2's video: expected {"user":"u1","files":["v1","v2"]} but got {"user":"u1","files":["v1","w1","v2"]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Google Photos · 2019

Modelled on Google Takeout and Google Photos, November 2019. Google told affected users that for several days in late November 2019 a technical issue caused some videos from Google Photos to be exported into unrelated users' archives, and that fewer than 0.01% of Google Photos users who used Takeout in that window were affected. Google did not publish a detailed root cause, so this project is a reconstruction of one classic way it happens.

Here the export worker interleaves several users' jobs and collects items for all of them in one shared buffer.

Fix the exporter so every archive holds exactly its own user's items.

Code map

var archive = require("./archive");

exports.createExporter = function () {
  var jobs = {};
  var buffer = [];
  return {
    startJob: function (jobId, userId) {
      jobs[jobId] = { userId: userId };
    },
    addItem: function (jobId, item) {
      if (!jobs[jobId]) throw new Error("unknown job " + jobId);
      buffer.push(item.id);
    },
    finishJob: function (jobId) {
      var job = jobs[jobId];
      if (!job) throw new Error("unknown job " + jobId);
      var out = archive.seal(job.userId, buffer);
      buffer = [];
      delete jobs[jobId];
      return out;
    }
  };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Somebody Else's Videos in My Takeout — JavaScript Bug Hunt

Modelled on Google Takeout and Google Photos, November 2019.

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Concurrency, State, Security
  • Modelled on: Google Photos · 2019
  • Visible tests: a single job exports its items; interleaved jobs keep their own items
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (privacy))
  • Ticket: BUG-TAKEOUT-1121
  • Reported by: user report
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • interleaved jobs keep their own items — u1 must not receive u2's video: expected {"user":"u1","files":["v1","v2"]} but got {"user":"u1","files":["v1","w1","v2"]}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on Google Takeout and Google Photos, November 2019. Google told affected users that for several days in late November 2019 a technical issue caused some videos from Google Photos to be exported into unrelated users' archives, and that fewer than 0.01% of Google Photos users who used Takeout in that window were affected. Google did not publish a detailed root cause, so this project is a reconstruction of one classic way it happens.

Here the export worker interleaves several users' jobs and collects items for all of them in one shared buffer.

Fix the exporter so every archive holds exactly its own user's items.

Bug report

BUG-TAKEOUT-1121 · Priority: Critical (privacy) · Reported by: user report

createExporter() returns { startJob(jobId, userId), addItem(jobId, item), finishJob(jobId) } and jobs may interleave freely:

  • addItem(jobId, item) adds item.id to that job's archive; it throws if the job is unknown or item.ownerId is not the job's user (and adds nothing)
  • finishJob(jobId) returns archive.seal(userId, ids) -> { user, files } with exactly the ids added to THAT job, in the order they were added
  • a finished job is forgotten: finishing it again, or an unknown id, throws

Observed: a user's archive contained another user's videos; the other user's archive was missing them.

Logs

[takeout] job=j1 user=u1 add v1
[takeout] job=j2 user=u2 add w1
[takeout] job=j1 sealed files=[v1,w1]

The code as shipped

src/takeout/exporter.js (editable)

var archive = require("./archive");

exports.createExporter = function () {
  var jobs = {};
  var buffer = [];
  return {
    startJob: function (jobId, userId) {
      jobs[jobId] = { userId: userId };
    },
    addItem: function (jobId, item) {
      if (!jobs[jobId]) throw new Error("unknown job " + jobId);
      buffer.push(item.id);
    },
    finishJob: function (jobId) {
      var job = jobs[jobId];
      if (!job) throw new Error("unknown job " + jobId);
      var out = archive.seal(job.userId, buffer);
      buffer = [];
      delete jobs[jobId];
      return out;
    }
  };
};

Read-only context: src/takeout/archive.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.