Happy New Year, Dropped Messages

Medium

· Backend

Incident

SEV-2

BUG-NYE

Priority: High · reported by messaging core

Time to fix: 20 min

Failing checks on the shipped build

  • at capacity the oldest is dropped

    expected ["m2","m3","m4"] but got ["m1","m2","m3","m4"]

  • the queue never exceeds cap

    expected 4 but got 5

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

WhatsApp NYE

Inspired by the messaging-app meltdowns every New Year's Eve at midnight, when send queues overflow worldwide. Policy for the bounded outbox: when full, drop the oldest — the newest message is the one the user just hit send on.

outbox.js does the opposite, and its capacity check is off by one.

Code map

// Bounded send queue. Returns the new queue (never mutates the input).
exports.enqueue = function (queue, msg, cap) {
  if (queue.length > cap) {
    return queue.slice();
  }
  return queue.concat([msg]);
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Happy New Year, Dropped Messages — JavaScript Bug Hunt

Inspired by the messaging-app meltdowns every New Year's Eve at midnight, when send queues overflow worldwide.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Medium
  • Concepts: Queues, Bounds
  • Modelled on: WhatsApp NYE
  • Visible tests: under capacity just appends; at capacity the oldest is dropped; the queue never exceeds cap
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-NYE
  • Reported by: messaging core
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • at capacity the oldest is dropped — expected ["m2","m3","m4"] but got ["m1","m2","m3","m4"]
  • the queue never exceeds cap — expected 4 but got 5

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Inspired by the messaging-app meltdowns every New Year's Eve at midnight, when send queues overflow worldwide. Policy for the bounded outbox: when full, drop the oldest — the newest message is the one the user just hit send on.

outbox.js does the opposite, and its capacity check is off by one.

Bug report

BUG-NYE · Priority: High · Reported by: messaging core

enqueue(queue, msg, cap) rules:

  • the returned queue never exceeds cap
  • when full, the OLDEST message is dropped to admit the new one

Observed: at capacity the NEW message is silently discarded (users' midnight wishes vanish), and the queue actually grows to cap+1 first.

Logs

[outbox] cap=3 size=4
[outbox] dropped incoming msg id=m5 (queue kept stale m1)

The code as shipped

src/msg/outbox.js (editable)

// Bounded send queue. Returns the new queue (never mutates the input).
exports.enqueue = function (queue, msg, cap) {
  if (queue.length > cap) {
    return queue.slice();
  }
  return queue.concat([msg]);
};

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.