Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1
forcemerge 11967 11968
tag 11967 notabug
thanks
On 07/17/2012 12:17 PM, Jaime Gaspar wrote:
> I think that there is a bug in "uniq" (version 8.13).
Is this your distro's build? However, I repeated your claim with the
latest coreutils.git (post-8.17)., so this is not likely to be a bug in
a distro-specific multibyte patch.
>
> The file "bug.txt" attached consists of two lines:
> - the first one containing a character that
> looks like a "v" and a line break;
> - the second one containing a character that
> looks like a upside down "v" and a line break.
> In hex:
>
> E2 88 A8 0A
> E2 88 A7 0A
Those glyphs that you describe line up with Unicode characters. I bet
you are using a locale with UTF-8 character encoding.
>
> When we run "uniq bug.txt" in a terminal, "uniq" outputs a single line, so
> "uniq" thinks that the two lines are equal, but they are not.
I can reproduce your symptoms, but only when I fudge my locale:
$ LC_ALL=C uniq ../bug.txt
β¨
β§
$ LC_ALL=en_US.UTF-8 uniq ../bug.txt
β¨
$
Remember, 'uniq' is required by POSIX to use the same line comparison
techniques as 'sort'; and 'sort' is required to use strcoll() (not
strcmp) to compare lines. And in your particular choice of locale,
strcoll() happens to state that 'β¨' and 'β§' collate identically; hence
uniq is correct in stating that you have a duplicated line according to
your current locale.
$ LC_ALL=en_US.UTF-8 sort ../bug.txt -u --debug
sort: using βen_US.UTF-8β sorting rules
β¨
_
$
So I'm closing this as not a bug, along with a final pointer to our FAQ:
https://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021
--
Eric Blake address@hidden +1-919-301-3266
Libvirt virtualization library http://libvirt.org
signature.asc Description: OpenPGP digital signature