Access Denied at Midnight

Medium

· Database

Incident

SEV-1

BUG-4090

Priority: Blocker · reported by on-call (02:14)

Time to fix: 20 min

Failing checks on the shipped build

  • percent-encoded password is decoded

    userinfo must be percent-decoded: expected "S3cure@pass%" but got "S3cure%40pass%25"

  • default MySQL port is 3306

    mysql defaults to 3306, not 5432: expected 3306 but got 5432

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Production on-call

After a credential rotation, the reporting service can no longer reach its MySQL database — but only in production. The new password contains special characters, which the platform team URL-encodes into the connection string, exactly per spec.

The pool builder and the service entrypoint are locked. The bug is in parseConnectionString.js.

Decode what the spec requires, and the suite goes green.

Code map

// Parses mysql://user:password@host:port/database?params
// Returns { user, password, host, port, database }.
exports.parseConnectionString = function (url) {
  var rest = url.slice("mysql://".length);

  var atIndex = rest.indexOf("@");
  var creds = rest.slice(0, atIndex);
  var location = rest.slice(atIndex + 1);

  var colon = creds.indexOf(":");
  var user = creds.slice(0, colon);
  var password = creds.slice(colon + 1);

  var qIndex = location.indexOf("?");
  if (qIndex !== -1) location = location.slice(0, qIndex);

  var slash = location.indexOf("/");
  var hostPort = location.slice(0, slash);
  var database = location.slice(slash + 1);

  var hpColon = hostPort.indexOf(":");
  var host = hpColon === -1 ? hostPort : hostPort.slice(0, hpColon);
  var port = hpColon === -1 ? 5432 : parseInt(hostPort.slice(hpColon + 1), 10);

  return { user: user, password: password, host: host, port: port, database: database };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Access Denied at Midnight — JavaScript Bug Hunt

After a credential rotation, the reporting service can no longer reach its MySQL database — but only in production.

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Medium
  • Modelled on: Production on-call
  • Visible tests: plain credentials parse cleanly; percent-encoded password is decoded; default MySQL port is 3306
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Blocker)
  • Ticket: BUG-4090
  • Reported by: on-call (02:14)
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • percent-encoded password is decoded — userinfo must be percent-decoded: expected "S3cure@pass%" but got "S3cure%40pass%25"
  • default MySQL port is 3306 — mysql defaults to 3306, not 5432: expected 3306 but got 5432

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

After a credential rotation, the reporting service can no longer reach its MySQL database — but only in production. The new password contains special characters, which the platform team URL-encodes into the connection string, exactly per spec.

The pool builder and the service entrypoint are locked. The bug is in parseConnectionString.js.

Decode what the spec requires, and the suite goes green.

Bug report

BUG-4090 · Priority: Blocker · Reported by: on-call (02:14)

After rotating the DB password to S3cure@pass% the service fails to boot. The connection string set in the environment is:

mysql://reporter:S3cure%40pass%25@db.internal:3306/analytics

Spec (RFC 3986): userinfo is percent-ENCODED in the URL and must be DECODED before use. The host must obviously never contain pieces of the password.

Staging (password without special characters) connects fine, which is why this slipped through review.

Logs

[pool] connecting as user="reporter" host="pass%25@db.internal" ...
[pool] ERROR Access denied for user 'reporter'@'10.0.3.17' (using password: YES)
[boot] FATAL could not initialize reporting database

The code as shipped

src/db/parseConnectionString.js (editable)

// Parses mysql://user:password@host:port/database?params
// Returns { user, password, host, port, database }.
exports.parseConnectionString = function (url) {
  var rest = url.slice("mysql://".length);

  var atIndex = rest.indexOf("@");
  var creds = rest.slice(0, atIndex);
  var location = rest.slice(atIndex + 1);

  var colon = creds.indexOf(":");
  var user = creds.slice(0, colon);
  var password = creds.slice(colon + 1);

  var qIndex = location.indexOf("?");
  if (qIndex !== -1) location = location.slice(0, qIndex);

  var slash = location.indexOf("/");
  var hostPort = location.slice(0, slash);
  var database = location.slice(slash + 1);

  var hpColon = hostPort.indexOf(":");
  var host = hpColon === -1 ? hostPort : hostPort.slice(0, hpColon);
  var port = hpColon === -1 ? 5432 : parseInt(hostPort.slice(hpColon + 1), 10);

  return { user: user, password: password, host: host, port: port, database: database };
};

Read-only context: src/db/index.js, src/db/pool.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.