Re: [PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31 + NFC) behind a root-controlled system switch
Date:
Thu, 23 Jul 2026 22:22:18 +0200
User-agent:
Mozilla Thunderbird
Hello Andreas,
Both are fair questions. Short answers: confusables are real but are not
something an identifier *syntax* can decide, and yes -- you are right
about RTL scripts; the patch removes only the part of that problem that
syntax can remove. Details below; v3 is attached, because your first
question pointed at something I could act on directly.
> What about confusables, like B vs. Β vs. В?
Deliberately out of scope for UAX #31, and the reason is structural:
confusability is not a property of an identifier, it is a property of a
*pair* of identifiers in a context. Β is a spoof next to a Latin B and a
perfectly ordinary name in Greek code. That is why Unicode puts it in a
separate document, UTS #39, and why no language I know of rejects
confusables in its grammar.
Two things are worth separating:
- The pixel-identical class is already gone. Canonically equivalent
spellings -- precomposed ä vs. a + U+0308 -- render identically by
definition, in every font, and the patch rejects the non-NFC form
outright, so identifier equality stays byte equality. What remains
is the merely *similar-looking* class.
- That class is the one ASCII has always had: $l vs. $I vs. $1, $O vs.
$0, $rn vs. $m. Rust's RFC 2457 says the same about within-script
confusables -- "equivalent to l vs I being confusable in some
fonts". Nobody proposes restricting ASCII names over it; the
mitigation has always been review and naming discipline. Non-ASCII
enlarges the set, it does not create the category.
What the ecosystem does with the rest: rustc warns, it does not reject.
confusable_idents compares the UTS #39 skeleton of every identifier
against every other identifier in the same crate; mixed_script_confusables
fires only when a script appears *solely* through characters confusable
with another script; uncommon_codepoints flags anything outside the
UTS #39 general security profile. C23, C++, Python, Java, JavaScript and
zsh do nothing at all at the language level.
The rustc approach cannot be transplanted to a shell: skeleton comparison
needs a compilation unit, and bash has none. Names arrive from the
environment, from read, from eval, from sourced files, at arbitrary times
during execution. There is no set to compare against, and comparing every
assignment against every variable in scope is not a cost I would want on
the assignment path.
But one part of UTS #39 *is* a plain per-codepoint property, and v3 adopts
it: section 3.1, the general security profile (Identifier_Status=Allowed).
The generated table is now intersected with it, which costs nothing at run
time -- it only makes the table smaller -- and removes in advance exactly
the material rustc merely warns about. The effect is larger than I
expected: the repertoire drops from about 136000 code points to about
33000, while every everyday letter of the living scripts stays (verified
in the generator's self-check for German, Greek, Cyrillic, Han, Kana,
Thai, Arabic, Hebrew and Hangul).
A concrete example in your direction: U+00B5 MICRO SIGN is Restricted
precisely because it is confusable with U+03BC GREEK SMALL LETTER MU. In
v2 both were legal and distinct; in v3 the micro sign is rejected and the
letter remains. Another: U+2126 OHM SIGN now drops out one stage earlier
than the NFC check. In fact, after the profile no NFC_Quick_Check=No
character survives at all -- Unicode has already restricted them -- so the
NFC table now carries only the Maybe cases.
The cost is that the generator needs IdentifierStatus.txt from the UCD
security data, which Python's unicodedata does not ship; it is now a
required command-line argument, and both source versions are recorded in
the generated header. They need not match: intersecting with a newer
security profile can only shrink what the UCD in use permits.
The other half, mixed-script restriction (UTS #39 restriction levels), is
also per-name and would catch your example directly: Cyrillic В followed
by Latin "ank" rejected, while both "Bank" and "Ванк" stay legal. It
needs a Script_Extensions table of roughly the size of the existing ones
and the Highly Restrictive level to keep the legitimate Japanese and
Korean combinations working. The trade-off is that it also forbids honest
mixtures like myПеременная, which is why Rust warns rather than
rejects. I have no strong view and will implement whichever you and Chet
consider the right default.
> Doesn't that issue also exist with Arabic and other right-to-left
> scripts, which (IIUC) can blend into neighboring bidi-neutral or
> bidi-mirroring characters?
Yes. You are right, and my phrasing was too narrow. Strong RTL characters
reorder adjacent neutrals under UAX #9, and mirrored pairs render mirrored
inside an RTL run, so ${عربي} does not look the way an ASCII reader
expects.
The line I would draw is between what a grammar can fix and what it
cannot:
- The explicit overrides and isolates (RLO/LRO/PDF, RLI/LRI/PDI) let
the reordering escape the token and misrepresent arbitrary
surrounding text -- the Trojan Source mechanism, where a comment can
be made to look like a statement. Those are not XID, and since v2 no
Default_Ignorable code point can appear in a name at all. What
remains is bounded by the run of strong characters itself; in a line
whose first strong character is ASCII, the surrounding structure
keeps its order.
- The residual effect is simply how a terminal renders Arabic or
Hebrew, and bash renders it today -- in string literals, comments,
filenames, here-documents, $'...'. echo "مرحبا" and a variable named
مرحبا have the same display behaviour. The patch does not introduce
RTL text to the shell; it introduces it to one more token class.
Excluding RTL scripts is the one restriction I would argue hardest
against. It would tell precisely the users this feature exists for that
their writing system is second-class, and it would not remove the
rendering effect from the rest of the shell in any case. Rust hit the
same question during stabilization and documented it as a known
limitation rather than banning the scripts. UAX #9 and UTS #55 put this
where I think it belongs: with the display layer -- editors, terminals,
pagers -- which can isolate and annotate. A grammar cannot.
So v3 also extends the manual entry, which previously warned only about
portability. It now states both limitations plainly: that non-ASCII names
can be visually confusable with one another, as ASCII l/I/1 already are,
and that names in right-to-left scripts reorder neighboring characters
when displayed. Anyone turning the switch on should know both.
Thanks for the two questions -- they were the right ones to ask, and the
first one made the patch measurably better.
Regards,
Luca Rodenhäuser