Back to 1970

Easy

· Backend

Incident

SEV-1

BUG-1970

Priority: Critical · reported by device reliability

Time to fix: 10 min

Failing checks on the shipped build

  • the epoch with a large positive offset clamps to zero

    elapsed time must never go negative: expected 0 but got -39600

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Apple iOS · 2016

Modelled on the iOS "1 January 1970" bug (2016): setting a 64-bit iPhone's date to the very start of the Unix epoch caused timezone offsets to push internal timestamps below zero. Code that assumed a non-negative epoch spun forever, and affected devices bricked on the next reboot.

uptime.js computes elapsed seconds and has the same missing floor.

Fix elapsedSince so a timestamp at or before the epoch cannot produce a negative or looping result.

Code map

// Seconds elapsed between two epoch timestamps, offset-adjusted.
exports.elapsedSince = function (nowEpoch, thenEpoch, tzOffsetSeconds) {
  var adjusted = thenEpoch + tzOffsetSeconds;
  return nowEpoch - adjusted;
};

// Retries once per elapsed second, capped by the caller.
exports.retryCount = function (nowEpoch, thenEpoch, tzOffsetSeconds) {
  var n = exports.elapsedSince(nowEpoch, thenEpoch, tzOffsetSeconds);
  var count = 0;
  while (count < n) count++;
  return count;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

Back to 1970 — JavaScript Bug Hunt

Modelled on the iOS "1 January 1970" bug (2016): setting a 64-bit iPhone's date to the very start of the Unix epoch caused timezone offsets to push internal…

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Dates, Underflow
  • Modelled on: Apple iOS · 2016
  • Visible tests: a normal interval is measured; the epoch with a large positive offset clamps to zero
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical)
  • Ticket: BUG-1970
  • Reported by: device reliability
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • the epoch with a large positive offset clamps to zero — elapsed time must never go negative: expected 0 but got -39600

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the iOS "1 January 1970" bug (2016): setting a 64-bit iPhone's date to the very start of the Unix epoch caused timezone offsets to push internal timestamps below zero. Code that assumed a non-negative epoch spun forever, and affected devices bricked on the next reboot.

uptime.js computes elapsed seconds and has the same missing floor.

Fix elapsedSince so a timestamp at or before the epoch cannot produce a negative or looping result.

Bug report

BUG-1970 · Priority: Critical · Reported by: device reliability

elapsedSince(nowEpoch, thenEpoch, tzOffsetSeconds):

  • returns the number of seconds elapsed, after applying the timezone offset to the "then" value
  • must never return a negative number; clamp to 0 instead

Observed: with thenEpoch=0 and a large positive offset the result goes below zero and the retry loop that consumes it never terminates.

Logs

[time] elapsed=-39600 now=0 then=0 tz=39600
[time] boot watchdog: retry loop exceeded 100000 iterations

The code as shipped

src/time/uptime.js (editable)

// Seconds elapsed between two epoch timestamps, offset-adjusted.
exports.elapsedSince = function (nowEpoch, thenEpoch, tzOffsetSeconds) {
  var adjusted = thenEpoch + tzOffsetSeconds;
  return nowEpoch - adjusted;
};

// Retries once per elapsed second, capped by the caller.
exports.retryCount = function (nowEpoch, thenEpoch, tzOffsetSeconds) {
  var n = exports.elapsedSince(nowEpoch, thenEpoch, tzOffsetSeconds);
  var count = 0;
  while (count < n) count++;
  return count;
};

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.