The Rotation That Ate the Backups — JavaScript Bug Hunt
Inspired by retention scripts gone wrong the world over (and the reason "test your restores" is a proverb): the nightly rotation is supposed to keep the…
- Language: JavaScript
- Layer: Database
- Difficulty: Medium
- Concepts: Backups, Retention
- Modelled on: Retention scripts
- Visible tests: the newest N survive, newest first; the newest full backup is always retained
- Reward: 50 XP for a complete fix
Incident
- Severity: SEV-1 (priority: Critical)
- Ticket: BUG-ROTATE
- Reported by: storage
- Time to fix: 20 minutes, the target for a medium incident
Failing checks on the shipped build
- the newest N survive, newest first — expected ["d","b"] but got ["a","c"]
- the newest full backup is always retained — expected ["inc-2","inc-1","full-old"] but got ["full-old","inc-1"]
How to work this incident
- Read the report: write down the expected behaviour and the actual one in a sentence each.
- Reproduce it: run the visible tests and find the failing check that matches the report.
- Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
- Follow the data through the query or the repository code; compare the rows it returns with the rows the report expects.
- Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.
Briefing
Inspired by retention scripts gone wrong the world over (and the reason "test your restores" is a proverb): the nightly rotation is supposed to keep the newest N snapshots plus, always, at least one full backup. This one keeps the oldest and deletes every full.
retention.js decides what survives.
Bug report
BUG-ROTATE · Priority: Critical · Reported by: storage
keep(snapshots, keepN):
- keep the keepN NEWEST snapshots (by created, descending in the result)
- if no full backup made the cut, ALSO keep the newest full one
- everything else is deleted
Observed: the script keeps the oldest N, and full backups age out first because they're the biggest and oldest. Restore drill failed this morning.
Logs
[rotate] kept: [day-001, day-002, day-003] — all incrementals from last yearThe code as shipped
src/backup/retention.js (editable)
// Decides which snapshots survive the nightly rotation.
exports.keep = function (snapshots, keepN) {
var sorted = snapshots.slice().sort(function (a, b) {
return a.created - b.created;
});
return sorted.slice(0, keepN);
};
Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.