When 65536 Read as Zero — JavaScript Bug Hunt

Briefing

Modelled on the Excel 2007 display bug (September 2007): typing =850*77.1 showed 100000 instead of 65535. The value in the cell was right; the code that turned certain floating-point results lying a hair below 65535 or 65536 into text for the grid printed the wrong digits. Microsoft confirmed the problem was confined to how a handful of values were displayed and shipped a fix within weeks.

This project reconstructs a formatter with the same kind of fault in the same place: a fast path for near-integer values that memoises digit strings in a 16-bit slot (cell16.js). Its range check looks at the value before it is rounded.

Fix display so every near-integer value shows the integer it rounds to.

Bug report

BUG-XL-65535 · Priority: High · Reported by: a customer's quarterly report

display(x) returns the text the grid shows for a numeric cell:

• if x is within 1e-9 of an integer n, show String(n) (display(850 * 77.1) -> "65535", display(65535.99999999999) -> "65536")
• otherwise show String(Math.round(x * 100) / 100) (3.14159 -> "3.14")
• cell16.render may be used as a fast path, but only for integers n with 0 <= n <= cell16.MAX (65535); everything else goes through general()

Observed: values just below 65536 display as "0".

Logs

[grid] B7 value=65535.99999999999 text="0"
[grid] B8 value=65536 text="65536"

The code as shipped

src/grid/format.js (editable)

var cell16 = require("./cell16");
var EPS = 1e-9;

// Text shown in the grid for a numeric cell value.
exports.display = function (x) {
  if (x >= 0 && x < cell16.MAX + 1) {
    var whole = Math.floor(x);
    var frac = x - whole;
    if (frac < EPS) return cell16.render(whole);
    if (1 - frac < EPS) return cell16.render(whole + 1);
  }
  return exports.general(x);
};

exports.general = function (x) {
  var n = Math.round(x);
  if (Math.abs(x - n) < EPS) return String(n);
  return String(Math.round(x * 100) / 100);
};

Read-only context: src/grid/cell16.js.