Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110419 Red Hat/3.1.10-1.el6_0 Mnenhy/0.8.3 Thunderbird/3.1.10
retitle 8871 RFE enhance sort --debug -i
tag 8871 wishlist
thanks
On 06/15/2011 09:42 AM, Al Bogner wrote:
> Hi,
>
> this looks like a bug for me:
Thanks for the report. However, most likely this is not a bug in sort,
but a misunderstanding on your part about how locales affect which bytes
(or byte sequences, in multi-byte locales) are deemed printable.
>
> var="蠁胃喂谓蠈蟺蠅蟻慰,魏喂蟽蟽蠈蟼,桅蠉位位慰"
>
>
> echo "$var" | sed -e 's/.*/\L&/' -e 's/,/_/g' | tr '_' '\n' | \
Wow, that's a complex way to change comma into newline. Why not just:
var="蠁胃喂谓蠈蟺蠅蟻慰
魏喂蟽蟽蠈蟼
桅蠉位位慰"
echo "$var" | sort ...
[I'm assuming you've distilled this from a larger example where the
complex processing was actually useful rather than starting from the
right string to begin with]
> sort -f -u
> 魏喂蟽蟽蠈蟼
> 蠁胃喂谓蠈蟺蠅蟻慰
> 蠁蠉位位慰
>
> echo "$var" | sed -e 's/.*/\L&/' -e 's/,/_/g' | tr '_' '\n' | \
> sort -f -i -u
> 蠁胃喂谓蠈蟺蠅蟻慰
Let's put the new 'sort --debug' option to use to point out the
difference a locale makes (and note that on my machine, the C locale
deems non-ASCII bytes as non-printable, even though they still render
just fine on my terminal). First, without -i:
$ echo "$var" | LC_ALL=en_US.UTF-8 sort --debug -fu
sort: using `en_US.UTF-8' sorting rules
魏喂蟽蟽蠈蟼
______
蠁胃喂谓蠈蟺蠅蟻慰
_________
桅蠉位位慰
_____
$ echo "$var" | LC_ALL=C sort --debug -fu
sort: using simple byte comparison
桅蠉位位慰
__________
魏喂蟽蟽蠈蟼
____________
蠁胃喂谓蠈蟺蠅蟻慰
__________________
Did you notice how the line lengths differ between the en_US.UTF-8
locale (which knows how to treat multi-byte characters as single
characters) and the C locale (where every byte is a character, and the
multi-byte UTF-8 entities are treated as multiple non-printable characters)?
Then adding -i:
$ echo "$var" | LC_ALL=en_US.UTF-8 sort --debug -fui
sort: using `en_US.UTF-8' sorting rules
魏喂蟽蟽蠈蟼
______
蠁胃喂谓蠈蟺蠅蟻慰
_________
桅蠉位位慰
_____
$ echo "$var" | LC_ALL=C sort --debug -fui
coreutils/src/sort: using simple byte comparison
蠁胃喂谓蠈蟺蠅蟻慰
__________________
When all of the bytes are ignored as non-printable, then all three lines
are identical, hence -u prints only one line.
However, I think this report _did_ find a valid tangential issue - the
'sort --debug' option ought to be enhanced to use a different character
than '_' when flagging which bytes were ignored by -i as unprintable
characters. That is, I would find it much nicer to see:
$ echo 'a蠁c' | LC_ALL=C sort --debug -i
a蠁c
_.._
to make it obvious that the two bytes for 蠁 were being ignored from the
particular sort field that I requested, because -i was in effect. Same
thing goes for other sort options, such as 'sort -k1n' ignoring
characters after the end of the first parsed number.
--
Eric Blake address@hidden +1-801-349-2682
Libvirt virtualization library http://libvirt.org
signature.asc Description: OpenPGP digital signature