The Fetch That Reached the Metadata Service

Hard

· Backend

Incident

SEV-1

BUG-CAPONE

Priority: Critical (SSRF) · reported by security

Time to fix: 35 min

Failing checks on the shipped build

  • the metadata endpoint is refused

    link-local metadata must never be reachable: expected {"allowed":false,"reason":"blocked"} but got {"allowed":true,"reason":"ok"}

  • loopback and private ranges are refused

    expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Capital One · 2019

Modelled on the Capital One breach (July 2019): a misconfigured proxy let an attacker make the server fetch a URL of their choosing. They pointed it at the cloud metadata endpoint 169.254.169.254, retrieved the instance's credentials, and used them to read 100 million customer records.

fetcher.js fetches whatever URL it is given.

Fix checkUrl so internal and link-local addresses are refused.

Code map

var ALLOWED_HOSTS = ["images.example.com", "cdn.example.com", "assets.example.org"];
exports.ALLOWED_HOSTS = ALLOWED_HOSTS;

function hostOf(url) {
  var withoutScheme = url.replace("http://", "").replace("https://", "");
  var slash = withoutScheme.indexOf("/");
  return slash === -1 ? withoutScheme : withoutScheme.substring(0, slash);
}
exports.hostOf = hostOf;

exports.checkUrl = function (url) {
  return { allowed: true, reason: "ok" };
};

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Fetch That Reached the Metadata Service — Bug Hunt

Modelled on the Capital One breach (July 2019): a misconfigured proxy let an attacker make the server fetch a URL of their choosing.

  • Language: JavaScript
  • Layer: Backend
  • Difficulty: Hard
  • Concepts: Security, SSRF
  • Modelled on: Capital One · 2019
  • Visible tests: an allow-listed host is permitted; the metadata endpoint is refused; loopback and private ranges are refused
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-1 (priority: Critical (SSRF))
  • Ticket: BUG-CAPONE
  • Reported by: security
  • Time to fix: 35 minutes, the target for a hard incident

Failing checks on the shipped build

  • the metadata endpoint is refused — link-local metadata must never be reachable: expected {"allowed":false,"reason":"blocked"} but got {"allowed":true,"reason":"ok"}
  • loopback and private ranges are refused — expected false but got true

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the Capital One breach (July 2019): a misconfigured proxy let an attacker make the server fetch a URL of their choosing. They pointed it at the cloud metadata endpoint 169.254.169.254, retrieved the instance's credentials, and used them to read 100 million customer records.

fetcher.js fetches whatever URL it is given.

Fix checkUrl so internal and link-local addresses are refused.

Bug report

BUG-CAPONE · Priority: Critical (SSRF) · Reported by: security

checkUrl(url) must return { allowed, reason } for a URL of the form "http://<host>/<path>":

  • reject anything whose host is not in the public allow-list — reason "blocked"
  • specifically reject the metadata address 169.254.169.254, localhost, 127.0.0.1, and the private ranges 10.x and 192.168.x
  • an allowed host returns { allowed: true, reason: "ok" }

Observed: every URL is fetched, including the instance metadata endpoint.

Logs

[proxy] fetched http://169.254.169.254/latest/meta-data/iam/security-credentials/
[proxy] response contained an access key and session token

The code as shipped

src/proxy/fetcher.js (editable)

var ALLOWED_HOSTS = ["images.example.com", "cdn.example.com", "assets.example.org"];
exports.ALLOWED_HOSTS = ALLOWED_HOSTS;

function hostOf(url) {
  var withoutScheme = url.replace("http://", "").replace("https://", "");
  var slash = withoutScheme.indexOf("/");
  return slash === -1 ? withoutScheme : withoutScheme.substring(0, slash);
}
exports.hostOf = hostOf;

exports.checkUrl = function (url) {
  return { allowed: true, reason: "ok" };
};

Read-only context: src/proxy/SSRF.js.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More JavaScript bug hunts.