The Config Value That Crashed Every App

Medium

· Frontend

Incident

SEV-1

BUG-SDK-0710

Priority: Critical (crash on launch) · reported by app developers, all at once

Time to fix: 20 min

Failing checks on the shipped build

  • a filter of the wrong type is dropped, not fatal

    raw.dataFilters[event].map is not a function

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the value the screen shows back through the state and the pure helpers that compute it; the view is usually only the messenger.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Facebook iOS SDK · 2020

Modelled on the Facebook iOS SDK crash of 10 July 2020: a change to a server-side configuration that the Facebook SDK downloads at launch sent a value the SDK did not expect, and apps embedding it — Spotify, Tinder and Pinterest among them — crashed on launch until Facebook reverted the change. A similar incident had happened in May 2020.

config.js is a reconstruction of the SDK's launch step: it merges the server's settings into the defaults and trusts every field to have the right shape.

Fix applyServerConfig so it validates every field, falls back to defaults for anything malformed, and never throws.

Code map

var defaults = require("./defaults");

// Applies the server-delivered app settings at launch.
exports.applyServerConfig = function (raw) {
  var out = defaults.copyDefaults();
  if (raw.sampleRate !== undefined) out.sampleRate = raw.sampleRate;
  if (raw.dataFilters !== undefined) {
    Object.keys(raw.dataFilters).forEach(function (event) {
      out.filters[event] = raw.dataFilters[event].map(function (p) {
        return p.toLowerCase();
      });
    });
  }
  if (raw.endpoints !== undefined) out.endpoints = raw.endpoints.slice();
  return out;
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Config Value That Crashed Every App — Bug Hunt

Modelled on the Facebook iOS SDK crash of 10 July 2020: a change to a server-side configuration that the Facebook SDK downloads at launch sent a value the…

  • Language: JavaScript
  • Layer: Frontend
  • Difficulty: Medium
  • Concepts: Config, Validation, Parsing
  • Modelled on: Facebook iOS SDK · 2020
  • Visible tests: a well-formed config is applied; a filter of the wrong type is dropped, not fatal
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (crash on launch))
  • Ticket: BUG-SDK-0710
  • Reported by: app developers, all at once
  • Time to fix: 20 minutes, the target for a medium incident

Failing checks on the shipped build

  • a filter of the wrong type is dropped, not fatal — raw.dataFilters[event].map is not a function

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the value the screen shows back through the state and the pure helpers that compute it; the view is usually only the messenger.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Facebook iOS SDK crash of 10 July 2020: a change to a server-side configuration that the Facebook SDK downloads at launch sent a value the SDK did not expect, and apps embedding it — Spotify, Tinder and Pinterest among them — crashed on launch until Facebook reverted the change. A similar incident had happened in May 2020.

config.js is a reconstruction of the SDK's launch step: it merges the server's settings into the defaults and trusts every field to have the right shape.

Fix applyServerConfig so it validates every field, falls back to defaults for anything malformed, and never throws.

Bug report

BUG-SDK-0710 · Priority: Critical (crash on launch) · Reported by: app developers, all at once

applyServerConfig(raw) returns { sampleRate, filters, endpoints } (in that key order) and must never throw, whatever JSON value raw is.

  • raw not a plain object (null, array, string, number, boolean) -> a fresh copy of the defaults
  • sampleRate: a finite number with 0 <= n <= 1, else the default (1)
  • dataFilters -> filters: must be a plain object (not null, not an array), else {}. Each entry is kept only if its value is an array whose items are all strings; kept items are lower-cased. Other entries are dropped. Key order follows the input.
  • endpoints: a non-empty array whose items are all strings (copied), else the default (["graph"])

Observed: a dataFilters entry whose value is true instead of an array crashes every app on launch.

Logs

[FBSDK] fetched app settings v=2020-07-10
TypeError: raw.dataFilters[event].map is not a function
    at applyServerConfig (config.js:9)

The code as shipped

src/sdk/config.js (editable)

var defaults = require("./defaults");

// Applies the server-delivered app settings at launch.
exports.applyServerConfig = function (raw) {
  var out = defaults.copyDefaults();
  if (raw.sampleRate !== undefined) out.sampleRate = raw.sampleRate;
  if (raw.dataFilters !== undefined) {
    Object.keys(raw.dataFilters).forEach(function (event) {
      out.filters[event] = raw.dataFilters[event].map(function (p) {
        return p.toLowerCase();
      });
    });
  }
  if (raw.endpoints !== undefined) out.endpoints = raw.endpoints.slice();
  return out;
};

Read-only context: src/sdk/defaults.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.