The Late Enactor and the Empty Record — JavaScript Bug Hunt

Briefing

Modelled on the AWS us-east-1 outage of 19–20 October 2025: DynamoDB's DNS automation has a planner that produces numbered DNS plans and independent enactors that apply them. One enactor was badly delayed; by the time it applied its old plan, another enactor had already applied a much newer one — and that enactor's clean-up then deleted the old plan the delayed one had just made live. The regional endpoint was left with an empty DNS record, and the failure cascaded through the services that depend on DynamoDB.

enactor.js applies plans and cleans up old ones. The tests interleave the steps of two enactors explicitly, so the race is deterministic.

Fix applyPlan and cleanup so a stale plan can never replace a newer live one and the live plan is never deleted.

Bug report

BUG-DDBDNS · Priority: Critical (regional outage) · Reported by: DNS on-call

dns.js (locked) keeps store = { plans: { <generation>: [ips] }, live: <generation> | null }.

applyPlan(store, generation):

• a generation with no published plan -> false, nothing changes
• a generation OLDER than store.live is stale -> false, live unchanged
• otherwise (nothing live yet, or the same or a newer generation) -> store.live = generation, return true

cleanup(store, olderThan):

• deletes every plan whose generation < olderThan, EXCEPT the live plan
• returns the deleted generations as numbers, ascending

dns.resolve(store) must never come back empty while an applied plan exists.

Observed: enactor B applied plan 5; delayed enactor A then applied plan 1 over it; B's cleanup of plans older than 3 deleted plan 1 — now the live one. resolve() returned [].

Logs

[enactor-b] applied plan 5 to dynamodb.us-east-1
[enactor-a] applied plan 1 to dynamodb.us-east-1 (started 14m ago)
[enactor-b] cleanup: deleted plans [1, 2]
[resolver] dynamodb.us-east-1 -> NOERROR, 0 answers

The code as shipped

src/dns/enactor.js (editable)

// Applies DNS plans produced by the planner to the live endpoint record.
exports.applyPlan = function (store, generation) {
  if (!store.plans[generation]) return false;
  store.live = generation;
  return true;
};

// Removes plans that are too old to be useful.
exports.cleanup = function (store, olderThan) {
  var deleted = [];
  Object.keys(store.plans).forEach(function (key) {
    var generation = Number(key);
    if (generation < olderThan) {
      delete store.plans[key];
      deleted.push(generation);
    }
  });
  return deleted.sort(function (a, b) { return a - b; });
};

Read-only context: src/dns/dns.js.