Both Halves Thought They Were Primary

Hard

· Database

Incident

SEV-1

BUG-GH1021

Priority: Critical (data divergence) · reported by database SRE

Time to fix: 35 min

Failing checks on the shipped build

  • the minority side refuses to commit

    one of three is not a majority: expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

GitHub · 2018

Modelled on the GitHub outage of 21 October 2018: a 43-second network partition let automated failover promote a second primary on the other coast. Writes landed on both sides, and reconciling them took over 24 hours of degraded service.

replica.js accepts a write on whichever node receives it, with no quorum check.

Fix commitWrite so a node only commits when it can see a majority of the cluster.

Code map

// Commits a write on the given node.
exports.commitWrite = function (cluster, nodeId, value) {
  var node = cluster.nodes[nodeId];
  node.log.push(value);
  return true;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Both Halves Thought They Were Primary — JavaScript Bug Hunt

Modelled on the GitHub outage of 21 October 2018: a 43-second network partition let automated failover promote a second primary on the other coast.

  • Language: JavaScript
  • Layer: Database
  • Difficulty: Hard
  • Concepts: Consensus, Split Brain
  • Modelled on: GitHub · 2018
  • Visible tests: a node with a majority commits; the minority side refuses to commit
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (data divergence))
  • Ticket: BUG-GH1021
  • Reported by: database SRE
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the minority side refuses to commit — one of three is not a majority: expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the GitHub outage of 21 October 2018: a 43-second network partition let automated failover promote a second primary on the other coast. Writes landed on both sides, and reconciling them took over 24 hours of degraded service.

replica.js accepts a write on whichever node receives it, with no quorum check.

Fix commitWrite so a node only commits when it can see a majority of the cluster.

Bug report

BUG-GH1021 · Priority: Critical (data divergence) · Reported by: database SRE

commitWrite(cluster, nodeId, value) must:

  • commit only if the node is reachable AND a strict majority of the cluster's nodes are reachable from it
  • on commit, append the value to that node's log and return true
  • otherwise change nothing and return false

Observed: after a partition, both sides accept writes and the logs diverge.

Logs

[repl] node=us-east committed seq=91002 (visible peers: 2 of 5)
[repl] node=us-west committed seq=91002 (visible peers: 3 of 5)

The code as shipped

src/db/replica.js (editable)

// Commits a write on the given node.
exports.commitWrite = function (cluster, nodeId, value) {
  var node = cluster.nodes[nodeId];
  node.log.push(value);
  return true;
};

Read-only context: src/db/QUORUM.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.