Both Halves Thought They Were Primary — JavaScript Bug Hunt

Briefing

Modelled on the GitHub outage of 21 October 2018: a 43-second network partition let automated failover promote a second primary on the other coast. Writes landed on both sides, and reconciling them took over 24 hours of degraded service.

replica.js accepts a write on whichever node receives it, with no quorum check.

Fix commitWrite so a node only commits when it can see a majority of the cluster.

Bug report

BUG-GH1021 · Priority: Critical (data divergence) · Reported by: database SRE

commitWrite(cluster, nodeId, value) must:

• commit only if the node is reachable AND a strict majority of the cluster's nodes are reachable from it
• on commit, append the value to that node's log and return true
• otherwise change nothing and return false

Observed: after a partition, both sides accept writes and the logs diverge.

Logs

[repl] node=us-east committed seq=91002 (visible peers: 2 of 5)
[repl] node=us-west committed seq=91002 (visible peers: 3 of 5)

The code as shipped

src/db/replica.js (editable)

// Commits a write on the given node.
exports.commitWrite = function (cluster, nodeId, value) {
  var node = cluster.nodes[nodeId];
  node.log.push(value);
  return true;
};

Read-only context: src/db/QUORUM.js.