Release the Connection!

Medium

· Database

Incident

SEV-1

BUG-POOL-DRAIN

Priority: Critical · reported by 3 AM pager

Time to fix: 20 min

Failing checks on the shipped build

  • a throwing query still releases

    and the connection must come back: expected 2 but got 1

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Connection pools

Inspired by the slow-motion outage every service hits eventually: a query throws, the error path skips release(), and the pool bleeds one connection per failure until nothing is left at 3 AM.

The pool is locked. withConnection must release on every path.

Code map

var pool = require("./pool");

exports.withConnection = function (fn) {
  var conn = pool.acquire();
  var result = fn(conn);
  pool.release();
  return result;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Release the Connection! — JavaScript Bug Hunt

Inspired by the slow-motion outage every service hits eventually: a query throws, the error path skips release(), and the pool bleeds one connection per…

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Concepts: Connection Pool, Error Handling
  • Modelled on: Connection pools
  • Visible tests: the happy path acquires, runs and releases; a throwing query still releases
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-POOL-DRAIN
  • Reported by: 3 AM pager
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a throwing query still releases — and the connection must come back: expected 2 but got 1

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the slow-motion outage every service hits eventually: a query throws, the error path skips release(), and the pool bleeds one connection per failure until nothing is left at 3 AM.

The pool is locked. withConnection must release on every path.

Bug report

BUG-POOL-DRAIN · Priority: Critical · Reported by: 3 AM pager

withConnection(pool, fn):

  • acquires, runs fn(conn), returns its result
  • the connection is released whether fn returns OR throws
  • the original error still propagates

Observed: pool.available() ratchets down after every failed query, then "pool exhausted" takes the whole service down.

Logs

[pool] available: 10 -> 9 -> 8 -> ... -> 0
[api] Error: pool exhausted

The code as shipped

src/db/withConnection.js (editable)

var pool = require("./pool");

exports.withConnection = function (fn) {
  var conn = pool.acquire();
  var result = fn(conn);
  pool.release();
  return result;
};

Read-only context: src/db/pool.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.