Re: [PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31 + NFC) behind a root-controlled system switch
Date:
Sat, 25 Jul 2026 17:05:47 +0200
User-agent:
Mozilla Thunderbird
Hello Robert,
This was the most useful message in the thread. Two of your three
engineering suggestions are in v4 (attached), one factual claim needs
correcting, and on the repertoire question I want to separate what is my
choice from what is Unicode's -- because it turns out almost none of it
is mine.
The US-legislation joke, first, because it cuts the other way
-------------------------------------------------------------
Unicode, Inc. is a California 501(c)(3) in Mountain View; ASCII is ANSI
X3.4, and ANSI is a Washington, D.C. non-profit. Both are US bodies. So
the insane-president scenario is not an argument for "everything from
0x80 up" over UAX #31 -- it applies with equal force to `$', `_' and the
ASCII letters, which the same government could reclassify tomorrow. No
character rule anywhere is safe from that; it is not a differentiator
between the two designs. What Unicode actually offers is narrower and
keepable: not "the categories will never change" but "a string that is a
valid identifier today stays one tomorrow." That is a monotonicity
promise about the XID sets, and even if a future version broke it, a
bash built against an older table is frozen against exactly the table it
shipped with. Which leads straight to your best point:
Load the table at startup / generate at build time
--------------------------------------------------
You're right, and v4 moves in that direction. The four tables are now a
single generated one, and the generator's data source
(IdentifierStatus.txt from the UCD) ships in the tree as
support/unicode-identifier-status.txt, with a `make regen-mbidtbl'
target. I kept the generated header in the tree rather than generating
unconditionally during build, for the same reason bash ships y.tab.c:
bison is expected on build machines, Python is not, and I didn't want to
add a hard Python build dependency. If Chet would rather generate
unconditionally, that's a one-line Makefile change.
A runtime-loadable external table I'd push back on, though, and the
reason is this feature specifically. The whole security argument for
UAX #31 + NFC -- byte equality is identity, no invisibles, no
unreviewable names -- evaporates if the accepted set becomes
site-editable data: now every host can define its own notion of a valid
name, and a script's meaning depends on a file the reader never sees.
For a portability-and-review feature, a fixed table the reader can
reason about is the point, not an accident.
One table, not three
--------------------
Entirely agreed, and this was worth doing regardless of the rest. v4 has
a single MBID_RANGE table; each range carries a flags byte (start /
continue / NFC-maybe) and the combining class. Adjacent code points
coalesce only when all attributes match, which -- pleasingly -- gives
fewer ranges than the old XID_Continue table alone (about 1660 vs. about
1600+1500+... before), because the previous design stored the same
boundaries three times. One binary search now answers every question.
Thank you; the code is simpler and smaller.
SARA AM -- you found a real thing, but it isn't a generator bug
--------------------------------------------------------------
You're right that U+0E33 SARA AM is category Lo and excluded from
starters, and that looked inconsistent to me too until I checked: SARA
AM has XID_Start=false in the UCD itself, not by any decision in my
generator. The reason is mechanical -- it has a canonical decomposition
(to U+0E4D + U+0E32), and characters that decompose that way are kept
out of XID_Start so that identifier identity survives normalization. So
it's excluded, but by Unicode, and for the NFC-identity reason that is
the spine of this whole patch, not arbitrarily. Same mechanism, inverted,
is why it *is* a valid continuation character.
Which is the honest framing for the rest of your Thai points:
The Mn vowels, and "any combination of Thai letters"
----------------------------------------------------
You're right, and my "any combination of Thai letters" was wrong -- SARA
I and the other Mn vowels are not starters, so the literal transliteration
of `i=1' cannot be written, and that is a real asymmetry with Latin `i'.
Where I'd correct the framing: this isn't my design choice, it's UAX #31's
XID_Start, which is letters (L*) plus a few Nl, and excludes the nonspacing
combining marks (Mn) as *starters* precisely because a name beginning with
a combining mark has nothing to combine onto -- it would render as a
dotted circle. SARA I is a dependent vowel: in Thai text it never begins
a cluster, it attaches above a preceding consonant. So Unicode lets it
continue an identifier (กิ is fine) but not start one. That is not Latin
chauvinism in my table; it's the same rule Rust, C23, C++ and Python
all get from the same Unicode property. I'll happily concede it's
imperfect for Thai -- but it's the imperfection the entire industry
standardized on, and diverging from it means bash alone disagrees with
every other language about what a Thai identifier is.
The digits
----------
Here your reasoning actually lands, and I think you've half-talked me
out of the current behaviour. You're right that Thai digits aren't used
for arithmetic -- Arabic numerals are used for that in Thailand -- so the
$0/$1 parsing rationale that forces the ASCII digit restriction has no
Thai analogue, and none of your `$(( ๑๒๓ + 456 ))' examples work or
should. The reason U+0E50..U+0E59 are continue-only in my table is,
again, not a decision I made: they're category Nd, and UAX #31 puts Nd
in XID_Continue but not XID_Start, uniformly, for every script's digits.
The consistency argument for keeping it is that `๑foo' and `1foo' then
fail the same way. The argument you're making -- that non-ASCII digits
carry none of the syntactic baggage that motivated the ASCII rule -- is
reasonable, and if Chet wanted Nd characters allowed as starters too, I
wouldn't object; it's a small generator change. I'd just want it to be
a deliberate divergence from UAX #31, documented as such, rather than
an accident.
Symbols, chess pieces, `Three*Six'
----------------------------------
This is the real design fork, and here I'll hold, but let me narrow it
to the actual disagreement. You're right on the mechanics: above 0x7F
nothing is sh syntax, so `*' in a name is only special because ASCII `*'
is special, and So characters carry no such baggage. `${Three*Six}'
could be made to work.
Where we differ is what the boundary is *for*. You're optimizing for
"the shell can cope, so let it"; the patch optimizes for "a name means
one thing to the reader." Those genuinely conflict, and I don't think
either is objectively right -- but for a feature whose entire
justification is that it's safe enough to turn on, letter-based
identifiers are the defensible default and "any symbol except the
syntax characters" is not, because that's the door through which the
invisibles and the confusables come back. On CJK specifically: you're
right that many CJK symbols and radicals are So, but the actual writing
system -- the tens of thousands of Han characters people write names in
-- is Lo, and those are all XID_Start and all present. 漢字=1 works.
What's excluded is ♈ and ♠ and the like, which are symbols in any
script. So CJK text is not collateral damage here; ideographs are
letters and they're in.
Why match C/Rust at all -- you asked directly
---------------------------------------------
Fair question, and "bash isn't compatible with its cousins so why be
compatible with C" is a real point. The answer isn't C the language --
you're right that bash needn't care about C's identifier rules. It's
that UAX #31 is the one cross-language *specification* of "what is an
identifier character" that exists, it was designed by the people who
assign the code points, and it comes with the security annexes (NFC,
UTS #39) that a shell actually benefits from. Matching it isn't
deference to C; it's using the spec instead of inventing a private
repertoire that every reviewer would then have to learn. When zsh went
the other way -- iswalnum(), locale-defined -- it got a set that varies
by platform and libc, which is why स्व is a valid zsh identifier on
one box and not another. I'd rather bash's answer not depend on the C
library's locale data.
Empirically, by the way: I built zsh, ksh93 and fish and tried the same
names. All three accept non-ASCII identifiers already, none normalizes
(zsh happily holds U+2126 and U+03A9 as two distinct variables that
render identically), and all three inherit iswalnum()'s quirks -- Thai
vowel signs accepted, Devanagari virama rejected, purely by libc
accident. PowerShell ships both models at once: bare $name is Unicode
categories (Lu/Ll/Lt/Lm/Lo/Nd, so no Mn either), while ${...} is
literally your "anything but the delimiter," invisibles and all. So the
spread of existing practice runs from "letters only" to "anything," and
this patch sits at the strict, specified end of it deliberately.
Where that leaves v4: single table, shipped generator and data, factual
Thai errors fixed, and the manual now states the confusable and RTL
limitations plainly. The digit and symbol questions are genuine design
forks rather than bugs, and on both I've tried to mark exactly where I'd
defer to Chet if he wants the more permissive side.
Thanks again -- this materially improved the patch.
Note (thanks to Eduardo):
The implementation, parts of the architecture, and some of the code were
generated by AI (Opus 4.8, Max). I manually test each patch in a
real-world environment.
About my name:
Luca is my real name, and Ótzï is my nickname
Regards,
Luca Rodenhäuser