The Dotless I in Istanbul

Easy

· Backend

Incident

SEV-2

BUG-TR-0142

Priority: High · reported by Istanbul office

Time to fix: 10 min

Failing checks on the shipped build

  • keywords containing i match

    limit: expected condition to hold

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Inspired by

Java · Turkish locale

Modelled on the "Turkey test", a long-documented Java pitfall: String.toUpperCase() and toLowerCase() without an argument use the JVM's default locale, and in Turkish the lowercase i uppercases to a dotted İ while I lowercases to a dotless ı. Code that matches keywords or identifiers this way works everywhere until it runs on a machine configured for Turkish — then "limit".toUpperCase() is no longer "LIMIT". The standard fix is to pass Locale.ROOT for machine-readable text.

Here the production hosts' default locale is modelled by ServerLocale.DEFAULT, and the SQL keyword matcher uses it.

Fix SqlKeywords so keyword and identifier handling is locale-independent.

Code map

class SqlKeywords {
    static final Set<String> KEYWORDS = new HashSet<>(Arrays.asList(
        "SELECT", "FROM", "WHERE", "INSERT", "INTO", "LIMIT", "DISTINCT", "LIKE", "IN", "ORDER", "BY"));

    static boolean isKeyword(String word) {
        return KEYWORDS.contains(word.toUpperCase(ServerLocale.DEFAULT));
    }

    static String normalizeIdentifier(String name) {
        return name.toLowerCase(ServerLocale.DEFAULT);
    }
}

Sign in to edit the file, run the tests and submit a fix

A free account — no card, no trial clock. You come straight back here.

The Dotless I in Istanbul — Java Bug Hunt

Modelled on the "Turkey test", a long-documented Java pitfall: String.toUpperCase() and toLowerCase() without an argument use the JVM's default locale, and…

  • Language: Java
  • Layer: Backend
  • Difficulty: Easy
  • Concepts: Unicode, Parsing, Config
  • Modelled on: Java · Turkish locale
  • Visible tests: plain keywords match in any case; keywords containing i match
  • Reward: 50 XP for a complete fix

Incident

  • Severity: SEV-2 (priority: High)
  • Ticket: BUG-TR-0142
  • Reported by: Istanbul office
  • Time to fix: 10 minutes, the target for an easy incident

Failing checks on the shipped build

  • keywords containing i match — limit: expected condition to hold

How to work this incident

  1. Read the report: write down the expected behaviour and the actual one in a sentence each.
  2. Reproduce it: run the visible tests and find the failing check that matches the report.
  3. Read the logs and the failing checks for the first value that is wrong — the earliest wrong value is nearer the cause than the loudest error.
  4. Follow the request through the handler, the validation and the service it calls; find the first step whose output disagrees with the report.
  5. Fix the cause rather than the symptom, run the visible tests again, and submit against the hidden ones — then write the root cause down.

Briefing

Modelled on the "Turkey test", a long-documented Java pitfall: String.toUpperCase() and toLowerCase() without an argument use the JVM's default locale, and in Turkish the lowercase i uppercases to a dotted İ while I lowercases to a dotless ı. Code that matches keywords or identifiers this way works everywhere until it runs on a machine configured for Turkish — then "limit".toUpperCase() is no longer "LIMIT". The standard fix is to pass Locale.ROOT for machine-readable text.

Here the production hosts' default locale is modelled by ServerLocale.DEFAULT, and the SQL keyword matcher uses it.

Fix SqlKeywords so keyword and identifier handling is locale-independent.

Bug report

BUG-TR-0142 · Priority: High · Reported by: Istanbul office

SqlKeywords.isKeyword(word) is true iff the word, uppercased locale-independently (ASCII rules), is in KEYWORDS — for any casing: "limit", "Limit", "LIMIT", "distinct", "like", "in", "into". normalizeIdentifier(name) lowercases locale-independently: "TITLE" -> "title", "ID" -> "id". Words that are not keywords ("users", "limits", "selection") are not.

Observed: on the tr_TR hosts every query using LIMIT, DISTINCT, LIKE, IN or INTO fails to parse, and the column "TITLE" is looked up as "tıtle".

Logs

[sql] host=db-ist-2 user.language=tr
[sql] parse error near "limit": expected keyword, got identifier "LİMİT"
[sql] unknown column "tıtle"

The code as shipped

src/sql/SqlKeywords.java (editable)

class SqlKeywords {
    static final Set<String> KEYWORDS = new HashSet<>(Arrays.asList(
        "SELECT", "FROM", "WHERE", "INSERT", "INTO", "LIMIT", "DISTINCT", "LIKE", "IN", "ORDER", "BY"));

    static boolean isKeyword(String word) {
        return KEYWORDS.contains(word.toUpperCase(ServerLocale.DEFAULT));
    }

    static String normalizeIdentifier(String name) {
        return name.toLowerCase(ServerLocale.DEFAULT);
    }
}

Read-only context: src/sql/ServerLocale.java.

Open the hunt to start the incident clock, edit the files, run the visible tests and submit against the hidden ones, then write the root cause down. More Java bug hunts.