Somebody Else's Videos in My Takeout — JavaScript Bug Hunt
Modelled on Google Takeout and Google Photos, November 2019.
- Language: JavaScript
- Layer: Database
- Difficulty: Medium
- Concepts: Concurrency, State, Security
- Modelled on: Google Photos · 2019
- Visible tests: a single job exports its items; interleaved jobs keep their own items
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (privacy))
- Ticket: BUG-TAKEOUT-1121
- Reported by: user report
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- interleaved jobs keep their own items — u1 must not receive u2's video: expected {"user":"u1","files":["v1","v2"]} but got {"user":"u1","files":["v1","w1","v2"]}
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on Google Takeout and Google Photos, November 2019. Google told affected users that for several days in late November 2019 a technical issue caused some videos from Google Photos to be exported into unrelated users' archives, and that fewer than 0.01% of Google Photos users who used Takeout in that window were affected. Google did not publish a detailed root cause, so this project is a reconstruction of one classic way it happens.
Here the export worker interleaves several users' jobs and collects items for all of them in one shared buffer.
Fix the exporter so every archive holds exactly its own user's items.
Bug report
BUG-TAKEOUT-1121 · Priority: Critical (privacy) · Reported by: user report
createExporter() returns { startJob(jobId, userId), addItem(jobId, item), finishJob(jobId) } and jobs may interleave freely:
- addItem(jobId, item) adds item.id to that job's archive; it throws if the job is unknown or item.ownerId is not the job's user (and adds nothing)
- finishJob(jobId) returns archive.seal(userId, ids) -> { user, files } with exactly the ids added to THAT job, in the order they were added
- a finished job is forgotten: finishing it again, or an unknown id, throws
Observed: a user's archive contained another user's videos; the other user's archive was missing them.
Logs
[takeout] job=j1 user=u1 add v1
[takeout] job=j2 user=u2 add w1
[takeout] job=j1 sealed files=[v1,w1]The code as shipped
src/takeout/exporter.js (editable)
var archive = require("./archive");
exports.createExporter = function () {
var jobs = {};
var buffer = [];
return {
startJob: function (jobId, userId) {
jobs[jobId] = { userId: userId };
},
addItem: function (jobId, item) {
if (!jobs[jobId]) throw new Error("unknown job " + jobId);
buffer.push(item.id);
},
finishJob: function (jobId) {
var job = jobs[jobId];
if (!job) throw new Error("unknown job " + jobId);
var out = archive.seal(job.userId, buffer);
buffer = [];
delete jobs[jobId];
return out;
}
};
};
Read-only context: src/takeout/archive.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.