The Reply Meant for the Last User

Hard

· Backend

Incident

SEV-1

BUG-POOL-0320

Priority: Critical (cross-user data exposure) · reported by incident review

Time to fix: 35 min

Failing checks on the shipped build

  • a cancelled call does not leak its reply

    bob must never read alice's reply: expected {"reply":"titles:bob"} but got {"reply":"titles:alice"}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

OpenAI · 2023

Modelled on OpenAI's ChatGPT outage of 20 March 2023. OpenAI's post-mortem traced it to a bug in the redis-py client library: a request cancelled after its command was sent but before the reply was read left that connection in the shared pool with the reply still waiting. The next request to use the connection read the stale reply — data belonging to a different user. Some users could see titles from other users' chat histories, and OpenAI said payment-related information of 1.2% of ChatGPT Plus subscribers active in a nine-hour window may have been visible. ChatGPT was taken offline while it was fixed.

In this reconstruction the client library (locked) releases a cancelled connection back to the pool as-is.

Fix the pool so a connection that is not clean is never handed out again.

Code map

exports.create = function (factory) {
  var idle = [];
  var created = 0;
  var discarded = 0;
  return {
    acquire: function () {
      if (idle.length > 0) return idle.pop();
      created++;
      return factory(created);
    },
    release: function (conn) {
      idle.push(conn);
    },
    stats: function () {
      return { idle: idle.length, created: created, discarded: discarded };
    }
  };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Reply Meant for the Last User — JavaScript Bug Hunt

Modelled on OpenAI's ChatGPT outage of 20 March 2023. OpenAI's post-mortem traced it to a bug in the redis-py client library: a request cancelled after its…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Concurrency, Caching, Security
  • Modelled on: OpenAI · 2023
  • Visible tests: sequential calls get their own replies; a cancelled call does not leak its reply
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (cross-user data exposure))
  • Ticket: BUG-POOL-0320
  • Reported by: incident review
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a cancelled call does not leak its reply — bob must never read alice's reply: expected {"reply":"titles:bob"} but got {"reply":"titles:alice"}

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on OpenAI's ChatGPT outage of 20 March 2023. OpenAI's post-mortem traced it to a bug in the redis-py client library: a request cancelled after its command was sent but before the reply was read left that connection in the shared pool with the reply still waiting. The next request to use the connection read the stale reply — data belonging to a different user. Some users could see titles from other users' chat histories, and OpenAI said payment-related information of 1.2% of ChatGPT Plus subscribers active in a nine-hour window may have been visible. ChatGPT was taken offline while it was fixed.

In this reconstruction the client library (locked) releases a cancelled connection back to the pool as-is.

Fix the pool so a connection that is not clean is never handed out again.

Bug report

BUG-POOL-0320 · Priority: Critical (cross-user data exposure) · Reported by: incident review

pool.create(factory) returns { acquire(), release(conn), stats() }:

  • acquire() returns an idle connection if there is one, else a new one from factory(n) where n counts connections created (1, 2, …)
  • release(conn) returns conn to the idle list ONLY if it is open and has no unread replies (conn.pendingReplies() === 0). Otherwise the pool closes it (conn.close()), counts it as discarded and never hands it out again
  • stats() is { idle, created, discarded }

Observed: after a cancelled request, the next caller on that connection received the cancelled request's reply.

Logs

[api] GET /conversations user=alice cancelled (client disconnected)
[redis] conn#1 returned to pool, pending_replies=1
[api] GET /conversations user=bob -> titles:alice

The code as shipped

src/cache/pool.js (editable)

exports.create = function (factory) {
  var idle = [];
  var created = 0;
  var discarded = 0;
  return {
    acquire: function () {
      if (idle.length > 0) return idle.pop();
      created++;
      return factory(created);
    },
    release: function (conn) {
      idle.push(conn);
    },
    stats: function () {
      return { idle: idle.length, created: created, discarded: discarded };
    }
  };
};

Read-only context: src/cache/client.js, src/cache/conn.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.