The Dotless I in Istanbul — Java Bug Hunt

Briefing

Modelled on the "Turkey test", a long-documented Java pitfall: String.toUpperCase() and toLowerCase() without an argument use the JVM's default locale, and in Turkish the lowercase i uppercases to a dotted İ while I lowercases to a dotless ı. Code that matches keywords or identifiers this way works everywhere until it runs on a machine configured for Turkish — then "limit".toUpperCase() is no longer "LIMIT". The standard fix is to pass Locale.ROOT for machine-readable text.

Here the production hosts' default locale is modelled by ServerLocale.DEFAULT, and the SQL keyword matcher uses it.

Fix SqlKeywords so keyword and identifier handling is locale-independent.

Bug report

BUG-TR-0142 · Priority: High · Reported by: Istanbul office

SqlKeywords.isKeyword(word) is true iff the word, uppercased locale-independently (ASCII rules), is in KEYWORDS — for any casing: "limit", "Limit", "LIMIT", "distinct", "like", "in", "into". normalizeIdentifier(name) lowercases locale-independently: "TITLE" -> "title", "ID" -> "id". Words that are not keywords ("users", "limits", "selection") are not.

Observed: on the tr_TR hosts every query using LIMIT, DISTINCT, LIKE, IN or INTO fails to parse, and the column "TITLE" is looked up as "tıtle".

Logs

[sql] host=db-ist-2 user.language=tr
[sql] parse error near "limit": expected keyword, got identifier "LİMİT"
[sql] unknown column "tıtle"

The code as shipped

src/sql/SqlKeywords.java (editable)

class SqlKeywords {
    static final Set<String> KEYWORDS = new HashSet<>(Arrays.asList(
        "SELECT", "FROM", "WHERE", "INSERT", "INTO", "LIMIT", "DISTINCT", "LIKE", "IN", "ORDER", "BY"));

    static boolean isKeyword(String word) {
        return KEYWORDS.contains(word.toUpperCase(ServerLocale.DEFAULT));
    }

    static String normalizeIdentifier(String name) {
        return name.toLowerCase(ServerLocale.DEFAULT);
    }
}

Read-only context: src/sql/ServerLocale.java.