coreutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: fold: add the --characters option


From: Collin Funk
Subject: Re: fold: add the --characters option
Date: Fri, 22 Aug 2025 22:28:00 -0700
User-agent: Gnus/5.13 (Gnus v5.13)

PƔdraig Brady <P@draigBrady.com> writes:

> That is better. I agree the focus on UTF8 is prudent.
> Given the tradeoffs here, this seems like the best approach.
>
> This is 4x faster (than i18n patch) in the normal case,
> and 2x slower in the LC_ALL=C case.
>
> Fold being focused on text, and usually reasonable amounts of text,
> this is a good tradeoff.
>
> Lower level (set) operations like sort, uniq, join
> would have more important perf constraints.

Yes, agreed. Hopefully I will come up with some Gnulib modules to handle
reading from files and handling strings without doing:

   if (ascii)
     fast_case (...)
   else
     slow_case (...)

Since at that point we are implementing the program twice.

> I'd tweak NEWS to spell out "count multi-byte characters"
> rather than just "count characters".

Sure, done. I struggled a bit to word NEWS and the documentation since
fold can operate on characters, bytes, or columns. The 小 Unicode
character, for example, is 1 character, 2 columns (based on
EastAsianWidth.txt [1]), and 3 bytes when UTF-8 encoded. But explaining
all of that feels too technical for the Coreutils manual.

> Since we're changing --spaces handling too,
> to would be good to incorporate spaces mentioned
> in tests/wc/wc-nbsp.sh in fold-characters.sh

Good point. I added fold-spaces.sh for testing some breaking Unicode
space characters. And fold-nbsp.sh to test some Unicode non-breaking
space characters.

This actually caught a bug in my v2 patch. When using 'fold --spaces' it
improperly assumed the space was a single byte. This isn't the case
using U+2002 EN SPACE or various other characters.

Anyways, I pushed the attatched patch.

> On the subject of testing, this is one place where the i18n patch lacked,
> though it does have some adjustments for testing fold.
> It would be good to incorporate it's tests/.../fold.pl adjustments.
>
> Also it would be good to add tests for invalid multi-byte characters
> to see that they're handled appropriately.

Yes, it would be nice to add tests if they are already written.

Who wrote that patch though? Is it someone who has a copyright
assignment? I didn't reference it for my implementation, so no issues
now. Just not sure if we can import the tests if not.

Collin

[1] https://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt

Attachment: v3-0001-fold-add-the-characters-option.patch
Description: Text Data


reply via email to

[Prev in Thread] Current Thread [Next in Thread]