Drill or Live, Same Prompt

Hard

· Frontend

Incident

SEV-1

BUG-HI-EMA

Priority: Critical · reported by agency review

Time to fix: 35 min

Failing checks on the shipped build

  • a live alert gets its own prompt and needs an approver

    expected {"templateId":"PACOM-CDW","operator":"E1","prompt":"LIVE ALERT: this will be broadcast to the public. Type LIVE to confirm. A second approver is required.","requiresApprover":true} but got {"templateId":"PACOM-CDW","operator":"E1","prompt":"Are you sure that you want to send this Alert?","…

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the value the screen shows back through the state and the pure helpers that compute it; the view is usually only the messenger.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Hawaii EMA · 2018

Modelled on the Hawaii false missile alert of 13 January 2018. During a shift-change drill at the Hawaii Emergency Management Agency, a live ballistic missile alert was sent to phones, TV and radio across the state; a correction took 38 minutes. The FCC's investigation found, among other failures, that the alert software offered drill and live templates side by side and asked the same confirmation question for both, and that nothing required a second person before a live alert went out.

This reconstruction's alert console asks one generic question for every template and sends a live alert on the same "yes" as a drill.

Fix prepare and send so a live alert has its own confirmation and a second approver.

Code map

var templates = require("./templates");

function findTemplate(id) {
  for (var i = 0; i < templates.TEMPLATES.length; i++) {
    if (templates.TEMPLATES[i].id === id) return templates.TEMPLATES[i];
  }
  throw new Error("unknown template " + id);
}

// Step 1: the operator picks a template from the menu.
exports.prepare = function (templateId, operator) {
  var t = findTemplate(templateId);
  return {
    templateId: t.id,
    operator: operator,
    prompt: "Are you sure that you want to send this Alert?",
    requiresApprover: false
  };
};

// Step 2: the operator answers the prompt.
exports.send = function (pending, input) {
  var t = findTemplate(pending.templateId);
  if (input.typed !== "yes") return { sent: false, channel: null };
  return { sent: true, channel: t.live ? "public" : "drill" };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Drill or Live, Same Prompt — JavaScript Bug Hunt

Modelled on the Hawaii false missile alert of 13 January 2018.

  • Language: JavaScript
  • Layer: Frontend
  • Difficulty: Hard
  • Concepts: UX, Validation, State
  • Modelled on: Hawaii EMA · 2018
  • Visible tests: a drill goes out on yes; a live alert gets its own prompt and needs an approver
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-HI-EMA
  • Reported by: agency review
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • a live alert gets its own prompt and needs an approver — expected {"templateId":"PACOM-CDW","operator":"E1","prompt":"LIVE ALERT: this will be broadcast to the public. Type LIVE to confirm. A second approver is required.","requiresApprover":true} but got {"templateId":"PACOM-CDW","operator":"E1","prompt":"Are you sure that you want to send this Alert?","…

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the value the screen shows back through the state and the pure helpers that compute it; the view is usually only the messenger.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Hawaii false missile alert of 13 January 2018. During a shift-change drill at the Hawaii Emergency Management Agency, a live ballistic missile alert was sent to phones, TV and radio across the state; a correction took 38 minutes. The FCC's investigation found, among other failures, that the alert software offered drill and live templates side by side and asked the same confirmation question for both, and that nothing required a second person before a live alert went out.

This reconstruction's alert console asks one generic question for every template and sends a live alert on the same "yes" as a drill.

Fix prepare and send so a live alert has its own confirmation and a second approver.

Bug report

BUG-HI-EMA · Priority: Critical · Reported by: agency review

prepare(templateId, operator) -> { templateId, operator, prompt, requiresApprover }; an unknown templateId throws.

  • drill template (live: false): prompt = PROMPTS.DRILL, requiresApprover = false
  • live template (live: true): prompt = PROMPTS.LIVE, requiresApprover = true

send(pending, input), input = { typed, approver }; returns { sent, channel } — decided from the TEMPLATE's live flag, never trusting fields of pending other than templateId and operator:

  • drill: sent when typed === "yes" -> { sent: true, channel: "drill" }
  • live: sent ONLY when typed === "LIVE" AND approver is a non-empty string different from pending.operator -> { sent: true, channel: "public" }
  • anything else -> { sent: false, channel: null }

Observed: the live template showed the same "Are you sure…" prompt as the drill, and "yes" from a single operator broadcast it to the public.

Logs

[console] op=E1 selected PACOM-CDW prompt="Are you sure that you want to send this Alert?"
[console] op=E1 confirmed "yes" -> channel=public
[console] correction issued +38m

The code as shipped

src/alerts/console.js (editable)

var templates = require("./templates");

function findTemplate(id) {
  for (var i = 0; i < templates.TEMPLATES.length; i++) {
    if (templates.TEMPLATES[i].id === id) return templates.TEMPLATES[i];
  }
  throw new Error("unknown template " + id);
}

// Step 1: the operator picks a template from the menu.
exports.prepare = function (templateId, operator) {
  var t = findTemplate(templateId);
  return {
    templateId: t.id,
    operator: operator,
    prompt: "Are you sure that you want to send this Alert?",
    requiresApprover: false
  };
};

// Step 2: the operator answers the prompt.
exports.send = function (pending, input) {
  var t = findTemplate(pending.templateId);
  if (input.typed !== "yes") return { sent: false, channel: null };
  return { sent: true, channel: t.live ? "public" : "drill" };
};

Read-only context: src/alerts/templates.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.