Re: [PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31 + NFC) behind a root-controlled system switch
Date:
Thu, 23 Jul 2026 19:30:32 +0200
User-agent:
Mozilla Thunderbird
(Replying in the thread here: my first copy of this reply accidentally
went to Robert off-list.)
Hello Robert,
thank you for actually reading the patch and the tables -- this is exactly
the review I was hoping for. Answers inline; the short version is that one
of your points found a real improvement (v2 attached), one is a fair
review request I'm happy to act on, and on the central design question --
tables vs. anything-goes -- I'd like to defend the tables, because the
properties they buy are the point of the patch.
> I believe this to be a poor idea, scripts should always remain
> portable.
For scripts meant to travel, agreed without reservation -- which is why
the feature is off by default, triple-gated (root-created consent file,
UTF-8 locale, explicit warning at enable time), and byte-for-byte inert
when off. But bash already lets people write deliberately non-portable
scripts on an opt-in basis (arrays, [[, extglob, globstar...), and for
interactive use and site-local scripts that trade-off is routinely made.
zsh -- the default login shell on macOS and Kali -- has shipped multibyte
identifiers for many years with the same "not portable" caveat in its
manual; the demand and the practice exist, the question is only whether
bash offers a well-defined version of it.
> I have no problem with only supporting UTF-8 encodings, though others
> who still use neither that nor ascii might.
Deliberate: UTF-8 is self-synchronizing and no byte of a multibyte
sequence collides with ASCII syntax characters, which is not true of the
older East Asian encodings. In a non-UTF-8 locale the option quietly
suspends itself, so those users keep exactly today's behavior.
> It also looks as if your patch may have some irrelevant changes
> included, that are not inside the HANDLE_MULTIBYTE blocks.
Fair, and there are exactly two categories. First, the declared
incidental fix: toggle_shopts() now honors the return value of shopt set
functions (previously ignored), and the hostcomplete wrapper is
normalized to return 0 like every other set function -- needed so that
enabling this option can fail cleanly. I'm happy to split that into a
separate preliminary patch. Second, call sites like valid_identifier()
and the ${!name}/${#name} starters use MBIDENT_ACTIVE()/
MBIDENT_STARTER_P(), which are defined as constant 0 when
HANDLE_MULTIBYTE is not defined, so the compiler removes them and the
object code is unchanged; that null-macro pattern keeps the hot paths
free of #ifdef noise. If you'd prefer explicit guards there instead,
that's a mechanical change I can make.
> the unicode people keep adding new characters, and whole new code
> planes (for additional languages). Deciding now, to apply forever,
> what is to be acceptable and what is not, once stepping outside
> simple ascii, seems like a very poor idea.
This is the strongest objection, and it is the one UAX #31's XID
properties were created to answer. The Unicode Consortium's stability
policy makes a formal guarantee: "All strings that are valid default
Unicode identifiers will continue to be valid default Unicode
identifiers in all subsequent versions of Unicode." XID_Start/
XID_Continue only ever grow. So nothing is decided "forever" against
anyone: a bash built with an older table accepts a subset, everything it
accepts stays valid under every future table, and a newly encoded script
becomes available by regenerating the header (one command; the ~200-line
generator ships in support/, tables are ~46KB of read-only ranges with a
binary search). This is precisely the versioned-table model C23, C++
(P1949R7, applied as a DR to older standards), Rust, GCC and Clang
settled on after weighing the same concern.
The proposal that really does decide forever is the other one: freezing
"everything from 0x80 up, assigned or not" means today's unassigned code
point can become tomorrow's combining mark or format control, silently
changing how existing names normalize and display.
> Beyond that, for fun, I had a look at your tables in the Thai section,
> (\u0E00 .. \u0E7F, though not all of those are used) and what's there
> looks to be nonsense to me.
Let me make that concrete, because no word-plausibility test exists
anywhere in the patch -- I agree completely that names need not be words.
In U+0E00..U+0E7F the tables contain: all 55 Thai letters (category Lo)
plus MAIYAMOK as start characters; SARA AM, all 16 combining vowel/tone
marks (Mn) and the ten Thai digits as continue characters (they cannot
lead a name, exactly like ASCII digits); excluded are only U+0E3F THAI
CURRENCY SYMBOL BAHT and the three punctuation characters U+0E4F, U+0E5A,
U+0E5B. That is the same letters/marks/digits vs. punctuation/symbols
line that [A-Za-z0-9_] draws for ASCII, mechanically derived from
DerivedCoreProperties.txt -- any combination of Thai letters is a valid
name, sensible as a word or not. If some specific code point in there
struck you as wrong, please name it: the table is generated, so a
discrepancy would be a generator bug and I very much want to know.
> I'm actually not sure why, if non-ascii var names were to be allowed,
> there needs to be any restriction at all on what (non-ascii)
> characters would be used for them [...] they might just as well all
> be allowed (even things like non-breaking space, and zero-width
> space, and stuff like that) as they're not sh syntax either.
You're right that above 0x7F the *syntax* no longer forces a boundary --
the boundary has to be chosen, and the question is which choice creates
the fewest problems downstream. "Everything" has three known ones:
1. Invisible characters. With ZWSP allowed, $Straße and
$Straße<U+200B> are two different variables that render
identically in every terminal and diff; an NBSP inside ${foo bar}
looks like a syntax error but isn't. This is the "Trojan Source"
class (CVE-2021-42574) that made GCC and Clang add warnings and
pushed every language that has revisited identifiers since toward
UAX #31 rather than anything-goes. Reviewability of shell scripts
is worth a lot; scripts are reviewed by eye more than most code.
2. Bidi format controls are also "not sh syntax", and allowing them
in names lets displayed order diverge from byte order in the
middle of a command line.
3. Aliasing without normalization: precomposed a-umlaut and
a + U+0308 would be two distinct, visually identical variables --
an equivalence-class problem ASCII never had. The patch's
NFC-or-reject rule keeps byte equality == identity, so grep and
humans stay reliable.
And your instinct about invisibles caught something real: the UCD leaves
variation selectors in XID_Continue, and my generated table had them.
UAX #31 itself recommends a profile that excludes default-ignorable code
points from general-purpose identifiers; v2 (attached) applies exactly
that profile -- variation selectors, ZWJ/ZWNJ, fillers, CGJ and soft
hyphen are now excluded whatever UCD generates the tables, with a
generator self-check and test cases. Thank you for pushing on that
line; it made the patch better.
> People could use emoji's as variable names!
Under anything-goes, yes -- under this patch no, and I'd argue that's
the consistent side of the line: emoji are symbols (So), the same
category as $ and ~, not letters. (zsh's iswalnum() rejects them too.)
If demand ever materializes, UAX #31 profiles are the sanctioned way to
add them as a deliberate decision rather than as fallout.
So: I'll gladly split out the shopt return-value fix, add explicit
guards if preferred, and rebase onto devel. The one design point I'd
argue to keep is UAX #31 + NFC itself -- the tables aren't incidental
complexity, they are what makes the feature reviewable, stable across
Unicode versions by formal guarantee, and aligned with what C, C++,
Rust and Python users already know as "an identifier".
Regards,
Luca Rodenhäuser