The Reply Meant for the Last User — JavaScript Bug Hunt

Briefing

Modelled on OpenAI's ChatGPT outage of 20 March 2023. OpenAI's post-mortem traced it to a bug in the redis-py client library: a request cancelled after its command was sent but before the reply was read left that connection in the shared pool with the reply still waiting. The next request to use the connection read the stale reply — data belonging to a different user. Some users could see titles from other users' chat histories, and OpenAI said payment-related information of 1.2% of ChatGPT Plus subscribers active in a nine-hour window may have been visible. ChatGPT was taken offline while it was fixed.

In this reconstruction the client library (locked) releases a cancelled connection back to the pool as-is.

Fix the pool so a connection that is not clean is never handed out again.

Bug report

BUG-POOL-0320 · Priority: Critical (cross-user data exposure) · Reported by: incident review

pool.create(factory) returns { acquire(), release(conn), stats() }:

• acquire() returns an idle connection if there is one, else a new one from factory(n) where n counts connections created (1, 2, …)
• release(conn) returns conn to the idle list ONLY if it is open and has no unread replies (conn.pendingReplies() === 0). Otherwise the pool closes it (conn.close()), counts it as discarded and never hands it out again
• stats() is { idle, created, discarded }

Observed: after a cancelled request, the next caller on that connection received the cancelled request's reply.

Logs

[api] GET /conversations user=alice cancelled (client disconnected)
[redis] conn#1 returned to pool, pending_replies=1
[api] GET /conversations user=bob -> titles:alice

The code as shipped

src/cache/pool.js (editable)

exports.create = function (factory) {
  var idle = [];
  var created = 0;
  var discarded = 0;
  return {
    acquire: function () {
      if (idle.length > 0) return idle.pop();
      created++;
      return factory(created);
    },
    release: function (conn) {
      idle.push(conn);
    },
    stats: function () {
      return { idle: idle.length, created: created, discarded: discarded };
    }
  };
};

Read-only context: src/cache/client.js, src/cache/conn.js.