Both Halves Thought They Were Primary — JavaScript Bug Hunt
Modelled on the GitHub outage of 21 October 2018: a 43-second network partition let automated failover promote a second primary on the other coast.
- Language: JavaScript
- Layer: Database
- Difficulty: Hard
- Concepts: Consensus, Split Brain
- Modelled on: GitHub · 2018
- Visible tests: a node with a majority commits; the minority side refuses to commit
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical (data divergence))
- Ticket: BUG-GH1021
- Reported by: database SRE
- Time to fix: 35 minutes, the target for a hard incident
Failing checks on the shipped build
- the minority side refuses to commit — one of three is not a majority: expected false but got true
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Modelled on the GitHub outage of 21 October 2018: a 43-second network partition let automated failover promote a second primary on the other coast. Writes landed on both sides, and reconciling them took over 24 hours of degraded service.
replica.js accepts a write on whichever node receives it, with no quorum check.
Fix commitWrite so a node only commits when it can see a majority of the cluster.
Bug report
BUG-GH1021 · Priority: Critical (data divergence) · Reported by: database SRE
commitWrite(cluster, nodeId, value) must:
- commit only if the node is reachable AND a strict majority of the cluster's nodes are reachable from it
- on commit, append the value to that node's log and return true
- otherwise change nothing and return false
Observed: after a partition, both sides accept writes and the logs diverge.
Logs
[repl] node=us-east committed seq=91002 (visible peers: 2 of 5)
[repl] node=us-west committed seq=91002 (visible peers: 3 of 5)The code as shipped
src/db/replica.js (editable)
// Commits a write on the given node.
exports.commitWrite = function (cluster, nodeId, value) {
var node = cluster.nodes[nodeId];
node.log.push(value);
return true;
};
Read-only context: src/db/QUORUM.js.
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.