PƔdraig Brady <P@draigBrady.com> writes:
> That is better. I agree the focus on UTF8 is prudent.
> Given the tradeoffs here, this seems like the best approach.
>
> This is 4x faster (than i18n patch) in the normal case,
> and 2x slower in the LC_ALL=C case.
>
> Fold being focused on text, and usually reasonable amounts of text,
> this is a good tradeoff.
>
> Lower level (set) operations like sort, uniq, join
> would have more important perf constraints.
Yes, agreed. Hopefully I will come up with some Gnulib modules to handle
reading from files and handling strings without doing:
if (ascii)
fast_case (...)
else
slow_case (...)
Since at that point we are implementing the program twice.
> I'd tweak NEWS to spell out "count multi-byte characters"
> rather than just "count characters".
Sure, done. I struggled a bit to word NEWS and the documentation since
fold can operate on characters, bytes, or columns. The å° Unicode
character, for example, is 1 character, 2 columns (based on
EastAsianWidth.txt [1]), and 3 bytes when UTF-8 encoded. But explaining
all of that feels too technical for the Coreutils manual.
> Since we're changing --spaces handling too,
> to would be good to incorporate spaces mentioned
> in tests/wc/wc-nbsp.sh in fold-characters.sh
Good point. I added fold-spaces.sh for testing some breaking Unicode
space characters. And fold-nbsp.sh to test some Unicode non-breaking
space characters.
This actually caught a bug in my v2 patch. When using 'fold --spaces' it
improperly assumed the space was a single byte. This isn't the case
using U+2002 EN SPACE or various other characters.
Anyways, I pushed the attatched patch.
> On the subject of testing, this is one place where the i18n patch lacked,
> though it does have some adjustments for testing fold.
> It would be good to incorporate it's tests/.../fold.pl adjustments.
>
> Also it would be good to add tests for invalid multi-byte characters
> to see that they're handled appropriately.
Yes, it would be nice to add tests if they are already written.
Who wrote that patch though? Is it someone who has a copyright
assignment? I didn't reference it for my implementation, so no issues
now. Just not sure if we can import the tests if not.
Collin
[1] https://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt