bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31 + NF


From: Ótzï fón GlüϷgor
Subject: [PATCH] multibyte_identifiers: opt-in UTF-8 variable names (UAX #31 + NFC) behind a root-controlled system switch
Date: Tue, 21 Jul 2026 16:14:48 +0000

Hello,

the attached patch against bash-5.3 adds an experimental, disabled-by-default
facility that lets shell variable names contain non-ASCII UTF-8 characters,
behind a system-wide switch that only root can operate. I would like to
propose it for inclusion and am happy to rework it based on your feedback.

Why
---
UTF-8 is the de-facto default locale encoding on every current platform, and
for many people -- students in teaching contexts, casual scripters, speakers
of languages that do not fit into ASCII -- the natural names for things are
not ASCII. Bash already meets non-ASCII variable names in the wild today:
the environment can carry them (which is why invalid_env exists to pass them
through to children). This patch makes such names first-class, but only where
an administrator has explicitly said yes.

What the identifier syntax is
-----------------------------
The accepted repertoire is exactly the default identifier syntax of Unicode
UAX #31: the first character may be any character with the XID_Start
property, or `_' (all ASCII letters have XID_Start, so nothing changes for
them); every further character may be any character with the XID_Continue
property (which includes the ASCII letters, digits and `_'). Non-ASCII
characters are valid in any position, including the first: ÄÖÜẞ and äöüß are
both accepted names. Names are additionally required to be in Normalization
Form C, enforced with the NFC quick check of UAX #15 (characters whose
NFC_QC property is No or Maybe are rejected outright, and combining marks
must be canonically ordered).

This is the profile the wider ecosystem has converged on:

  - Closest to home: zsh has accepted multibyte characters in identifiers
    for many years. With POSIX_IDENTIFIERS unset and MULTIBYTE set -- both
    defaults in UTF-8 locales -- "any alphanumeric characters in the local
    character set may be used in identifiers", and its manual carries the
    same portability caveat this patch does. Since zsh is the login shell
    that macOS and Kali Linux ship by default, multibyte identifiers in an
    interactive POSIX-family shell are already everyday reality. zsh's set
    is locale-dependent (iswalnum(3)), which is known to vary between
    platforms; this patch instead uses the locale-independent UAX #31 set
    with an NFC guarantee -- the stricter, more predictable variant of the
    same idea.
  - C23 adopted UAX #31 for identifiers (XID_Start/XID_Continue), conforming
    to requirements R1 and R4 (normalized equivalence).
  - C++ adopted the same via P1949R7 -- XID + NFC -- and applied it as a
    defect report to earlier standards. GCC, Clang and MSVC accept raw UTF-8
    identifiers today.
  - Rust has used exactly XID + NFC since non-ASCII identifiers were
    stabilized in 1.53; the patch matches Rust's accepted set (verified by
    compiling the same names, äöüß and ÄÖÜẞ, with rustc).
  - Python has allowed Unicode identifiers per PEP 3131 (UAX #31-based,
    with NFKC) since Python 3.0, at very large scale, without incident.

Why this is safe to have in the tree
------------------------------------
  - With the option off -- the default -- behavior is byte-for-byte identical
    to an unpatched bash. The generated Unicode tables deliberately start at
    U+0080, so ASCII handling never changes and the hot paths keep using the
    historical legal_variable_starter/legal_variable_char macros. POSIX mode
    is unaffected.
  - NFC-or-reject means identifier equality remains byte equality. There is
    never a second spelling of the same name, so grep/sed/diff and humans
    keep working; no new aliasing classes are introduced.
  - The UTF-8 decoder is strict: overlong encodings, surrogates and
    codepoints above U+10FFFF are rejected. Bidirectional control characters
    and other format characters are not XID, so they cannot appear in names
    (relevant after the "Trojan Source" work).
  - Everything is self-contained: no new library dependencies (no ICU, no
    libidn), just generated range tables (currently UCD 15.0.0) with a
    binary search, plus the Python generator in support/ so the tables are
    reproducible and upgradable.

How it is gated
---------------
The consent file /etc/bash_multibyte_identifiers (path overridable via
MBIDENT_SYSTEM_FILE) is the system-wide switch. A new root-only support
script manages it:

    bash-mbident enable     # warns that such names are not portable and
                            # asks for confirmation (locale-aware yesexpr,
                            # so j/J work in a German locale); requires a
                            # UTF-8 locale; creates the file
    bash-mbident disable    # removes it
    bash-mbident status     # exit 0/1; usable by anyone

A newly started shell adopts the state of the switch; while it is on and the
locale uses UTF-8, the extended syntax is available to every user with no
further ceremony. Switching the locale away from UTF-8 quietly suspends it.
Root may override the state of a single running shell with `shopt -s/-u
multibyte_identifiers'; only real state changes require root, while
redundant toggles succeed as no-ops for everyone, so scripts that reset
every shell option, and BASHOPTS inheritance into child shells, keep working
unprivileged. Querying is free for all users via `shopt -q'.

The opt-in shape follows established practice: bash itself ships opt-in
changes to parsing and semantics as shopt options (extglob, globstar,
lastpipe); the userland relaxes historical name rules only behind explicit
root-operated flags (useradd --badname, adduser --allow-bad-names); and
ASCII-core Internet protocols grew optional, backward-compatible Unicode
layers (SMTPUTF8/RFC 6531 and the EAI family, IDNA for domain names).

Implementation summary
----------------------
New files: mbident.c/mbident.h (strict decoder, table lookup, NFC quick
check), mbidtbl.h (generated), support/mkmbidtbl.py (generator),
support/bash-mbident.sh, tests/mbident.tests with a run-mbident driver.
Touched: general.c (valid_identifier, assignment), subst.c (the
legal_variable_starter sites for ${!name}, ${#name}, ${!prefix*} and the
$name scanner in param_expand), expr.c (readtok, with the same NFC
guarantee for arithmetic assignments), variables.c (adopting
environment strings whose names become valid when the option is turned on
in a running shell), builtins/shopt.def, shell.c (startup), docs
(bash.1, bashref.texi) and the test suite. One incidental fix:
toggle_shopts() now honors the return value of shopt set functions so that
toggling this option can fail cleanly; the hostcomplete wrapper was
normalized to return 0 like every other set function.

The full test suite passes with the switch on and with it off; run-shopt
and run-invocation normalize the option out of shopt/$BASHOPTS listings so
suite results do not depend on the system state, and run-mbident skips
cleanly where the prerequisites are missing.

Demo (Ubuntu 24.04, de_DE.UTF-8; prompts shortened, screenshot available)
-------------------------------------------------------------------------
    $ sudo install -m 755 support/bash-mbident.sh /usr/local/bin/bash-mbident
    $ bash-mbident status
    bash-mbident: disabled (/etc/bash_multibyte_identifiers does not exist)
    $ ./bash
    $ Straße=1
    bash: Straße=1: command not found
    $ sudo bash-mbident enable
    This permits bash shell options to accept multibyte (non-ASCII)
    variable names on this system.  Scripts using such names are not
    portable, and other programs that parse shell code or process the
    environment may have difficulties with them.  Enable with care.

    Do you want to continue? [y/N] y
    bash-mbident: created /etc/bash_multibyte_identifiers
    bash-mbident: newly started bash shells will accept multibyte identifiers 
in UTF-8 locales
    $ ./bash
    $ Straße=1
    $ echo $Straße
    1
    $ ÄÖÜẞ=1
    $ echo $ÄÖÜẞ
    1
    $ äöüß=2
    $ echo $äöüß
    2
    $ bash-mbident status
    bash-mbident: enabled (/etc/bash_multibyte_identifiers exists)
    $ sudo bash-mbident disable
    bash-mbident: removed /etc/bash_multibyte_identifiers
    $ ./bash
    $ echo $Straße
    ße
    $ Straße=1
    bash: Straße=1: command not found

(The `ße' line is the historical behavior with the option off: $Stra
expands empty and the rest is literal -- unchanged from stock bash. The
ÄÖÜẞ example also exercises U+1E9E LATIN CAPITAL LETTER SHARP S as a
leading character.)

Honest limitations
------------------
The switch is read at shell startup; running shells keep their state (root
can follow up with shopt). Scripts that use such names are deliberately
non-portable -- that is what the warning, the man page text and the
experimental label are for. Arithmetic-context names and assignment
statements enforce the same NFC rule, so no path can bind a non-NFC name.

Logistics
---------
The patch is git format-patch output against the bash-5.3 release tarball
and applies cleanly with `git apply'; I am happy to rebase it onto the
devel branch, to split it differently, or to adjust any of the design
choices above. I am also prepared to complete the FSF copyright assignment
paperwork for a contribution of this size.

Thank you for considering it, and for maintaining bash.


Regards,
Luca Rodenhäuser

Attachment: preview.webp
Description: preview.webp

Attachment: 0001-Add-multibyte-UTF-8-shell-variable-names-behind-a-ro.patch
Description: 0001-Add-multibyte-UTF-8-shell-variable-names-behind-a-ro.patch


reply via email to

[Prev in Thread] Current Thread [Next in Thread]