bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31


From: address@hidden
Subject: Re: [PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31 + NFC) behind a root-controlled system switch
Date: Tue, 28 Jul 2026 01:08:34 +0200
User-agent: Mozilla Thunderbird

Hello Robert,

  | where exactly is E33 in there?   If it isn't there, how is it valid?

It isn't there, and it wasn't valid. You're right, and I was wrong twice
in the same paragraph: wrong that SARA AM was a valid continuation
character, and wrong about why it was missing. Thank you for checking the
table instead of taking my word for it.

The correct explanation: U+0E33 has a *compatibility* decomposition, not
a canonical one, so NFC leaves it entirely alone and has nothing to do
with its absence. It disappeared because of the UTS #39 general security
profile I added in v3, which restricts it.

That profile was a mistake on my part, and the reason is a mismatch I
introduced: it is derived with NFKC in mind and restricts everything NFKC
would rewrite, while this patch requires NFC. Applying it wholesale cost
ordinary letters of scripts in daily use -- U+0E33, U+0EB3 LAO VOWEL SIGN
AM, U+0EDC and U+0EDD LAO HO NO and HO MO, U+0587 ARMENIAN SMALL LIGATURE
ECH YIWN, the four Arabic hamza letters, the Kana digraphs. In practice
คำ, ทำ, น้ำ, จำ, ดำ, นำ could not be used as variable names at
all. I had checked Thai against the property tables and not against
actual words, which is exactly the mistake you were pointing at.

And the exclusion bought nothing. น้ำ has at least three NFC-stable,
pairwise distinct encodings that render the same:

    0E19 0E49 0E33          with SARA AM
    0E19 0E4D 0E49 0E32     NIKHAHIT before the tone mark
    0E19 0E49 0E4D 0E32     NIKHAHIT after the tone mark

NFKC unifies only the first and third. The second stays distinct and is
fully permitted even under the untouched profile, because U+0E4D, U+0E49
and U+0E32 are all Allowed. So banning U+0E33 removed the natural
spelling while leaving the ambiguity it was supposed to prevent.

v5 (attached) re-admits a restricted character when it is NFC-stable, has
a compatibility mapping to two or more characters, and every character of
that mapping lies in its own 256-code-point block and is itself an
allowed identifier character. All four conditions matter: without the
last, the Tibetan VOCALIC RR and LL would come back too, since they map
onto U+0F81, which is itself restricted. Currently that yields eleven
characters, listed by name in the generator, with a self-check that fails
the build if a UCD update changes the set. คำ=1 works now; A=1, ﻻ=1 and
the Latin digraphs still do not.

What I should state plainly rather than let you find it: this re-admission
adds a spelling. คำ and the sequence U+0E04 U+0E4D U+0E32 are now two
distinct variables that a conforming renderer draws identically, and the
same holds for the other ten characters. NFC does not unify them; only
NFKC would. My earlier claim that NFC makes byte equality equal identity
is true, but I overstated what it buys: it removes the
canonical-equivalence class only. Scripts that admit several identically
rendering encodings still have them, with or without this rule -- as the
three spellings above show -- which is why I think restoring the natural
spelling is the better trade. The manual now says so, and so does the
generator comment.

Two corrections to my previous message while I am at it. The Tibetan
vowel signs I listed as collateral damage were not: U+0F73, U+0F75,
U+0F76, U+0F78 and U+0F81 have canonical decompositions that NFC itself
applies, so my own NFC requirement excludes them and the profile is not
to blame; U+0F77 and U+0F79 are restricted and stay restricted. And my
summary of the re-admission rule left out the condition on the mapping's
characters, which is what makes the count eleven rather than thirteen.

On the remaining points, briefly, since neither of us needs them
re-argued. The Mn starter asymmetry stands -- SARA I still cannot begin a
name where Latin i can. That comes from UAX #31 rather than from me, but
choosing UAX #31 was my decision, and this episode is a fair
demonstration of what adopting a profile wholesale costs when nobody
checks what it does to each script. I would rather say that than keep
leaning on "the standard says so." On guarantees: Unicode does publish a
formal normalization stability policy, but your broader point holds, and
the practical answer is the one you already accepted.

One warning about the attachment: v5 is around 280 KB, because it now
vendors the UCD security data file so `make regen-mbidtbl' works without
network access. If that is unwelcome on the list I can drop the file and
leave the download URL in the generator instead.

Thanks again -- these would have shipped.

Regards,
Luca Rodenhäuser

Attachment: v5-0001-Add-multibyte-UTF-8-shell-variable-names-behind-a.patch
Description: Text Data


reply via email to

[Prev in Thread] Current Thread [Next in Thread]