From MAILER-DAEMON Sat Jul 02 18:19:56 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bJTGW-0006MB-NU
	for mharc-coreutils@gnu.org; Sat, 02 Jul 2016 18:19:56 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55230)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bJTGU-0006Ls-0l
	for coreutils@gnu.org; Sat, 02 Jul 2016 18:19:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bJTGM-0007Wj-Sr
	for coreutils@gnu.org; Sat, 02 Jul 2016 18:19:49 -0400
Received: from mail-qk0-x22d.google.com ([2607:f8b0:400d:c09::22d]:34822)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bJTGM-0007Wb-OO
	for coreutils@gnu.org; Sat, 02 Jul 2016 18:19:46 -0400
Received: by mail-qk0-x22d.google.com with SMTP id u124so15516834qkh.2
	for <coreutils@gnu.org>; Sat, 02 Jul 2016 15:19:46 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=CfIrLMuQ1byBvvmFDh/Njtvm0uoP31fuKX44/2bfd+I=;
	b=Tt0h/xAmN6cbtFRXvdNhjcDGi7GEt1qwilQRBl9Zc8SabgbZ8iNpSpSZ9uhNxqQtAc
	mqG1GEkaXq4X4UntRc+pchuFJqT3DxaAw9BD7qUjTHRkim9eCoPg8E2V2H2hl8FXxsI0
	ZNCxYtMBXgeQxvwWv/ktD8zTfKUStZX3mbDjTrZsj61O4iA7al1jpStw1nk4gkTizW7F
	EkH7vclO6OE6ZyuzliqGE3CsDqAwM5VVfmqiAL3hlgHYf+i2yWTDNy67VsSFGXW4lBzY
	MfzFaNH937AntOJ0HPvXGOnjocbHNU4LXRkE6XpX3WOtukscypfAR1dPPT7hpz/y1eTO
	GKtw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=CfIrLMuQ1byBvvmFDh/Njtvm0uoP31fuKX44/2bfd+I=;
	b=QjdpZiaOtLX8AItx+vw+StmV2OF3e4yTy2mzQOWe2hslFb1ix4iaGenZBJXViJWl5W
	DuewkBa+2+9fs9l5gBDHO9hO2I3vRDj7woxnzBi28aG0RA5uwtf8d6sC9GIhAqbxNIlh
	lwE4SG2uJGZ2NozLB/ClggKlyY4sRyScpWlEIHCusqEa7ukswAGe3kETvFX9YfnljP6m
	EUdE//e3zD9gQ0oItgr1IfmZzYwUZjTv4u3kLZCV4nwaTIkJLWKM/lEXSTQaMOm7DYvB
	5wf0LkkBGoiIW7gK/qh46MyJLNYTR2iN1un7vwXOiYwdsQq+XxQmgE4lyuP8QNZ5zn0Q
	8IEg==
X-Gm-Message-State: ALyK8tK34GcwipSv/aMIW3jAvBBvGo5h9HcPbPFMTNKTlwvgubCKrfsXDLcO16ox293Kag==
X-Received: by 10.55.47.6 with SMTP id v6mr6586175qkh.11.1467497986161;
	Sat, 02 Jul 2016 15:19:46 -0700 (PDT)
Received: from [10.3.80.88] ([69.74.14.178]) by smtp.gmail.com with ESMTPSA id
	b66sm2177890qkj.39.2016.07.02.15.19.45
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Sat, 02 Jul 2016 15:19:45 -0700 (PDT)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: prevent seq from inflow inflow on I/O errors
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <572752B5.7040508@draigBrady.com>
Date: Sat, 2 Jul 2016 18:19:45 -0400
Cc: Bernhard Voelker <mail@bernhard-voelker.de>, Coreutils <coreutils@gnu.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <792F60B2-527E-4116-B010-A678E416C97E@gmail.com>
References: <788E7E16-EFE1-47D0-96CD-A35FB080BBA6@gmail.com>
	<570256BD.7010302@draigBrady.com>
	<57037EDD.2020604@bernhard-voelker.de>
	<1BC925DA-15D0-4E54-BE80-92AD9D64C0E7@gmail.com>
	<57136AE1.6070007@draigBrady.com>
	<3534D2B6-109B-45E2-95A3-8AEB8566F588@gmail.com>
	<6883FEEC-7D8E-4D06-9568-29BCDACA68A3@gmail.com>
	<571F35CB.7020103@draigBrady.com>
	<CAHN-Gmx33iNRK1h1Jn2cNMoFfkENLrWpTqebEJoDjh_f8d=zWA@mail.gmail.com>
	<572228BC.3070506@draigBrady.com>
	<F10ED182-3E12-4336-BFA8-E82A7BC8C510@gmail.com>
	<572752B5.7040508@draigBrady.com>
To: =?utf-8?Q?P=C3=A1draig_Brady?= <P@draigBrady.com>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c09::22d
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 02 Jul 2016 22:19:55 -0000


> On May 2, 2016, at 09:14, P=C3=A1draig Brady <P@draigBrady.com> wrote:
>=20
> +1

Sorry for the long delay.

Pushed at:
=
http://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=3D383e4b2ce589f6=
ecf0a375e94ab56fab6eff88f1
tests: improve recent seq EPIPE test

Regards,
 - assaf



From MAILER-DAEMON Wed Jul 06 13:16:16 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bKqQq-000797-Sb
	for mharc-coreutils@gnu.org; Wed, 06 Jul 2016 13:16:16 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:42631)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bKqQn-00078d-O2
	for coreutils@gnu.org; Wed, 06 Jul 2016 13:16:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bKqQi-0005BA-M4
	for coreutils@gnu.org; Wed, 06 Jul 2016 13:16:13 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:51234)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bKqQi-0005Al-FF
	for coreutils@gnu.org; Wed, 06 Jul 2016 13:16:08 -0400
Received: from [192.168.1.80] (unknown [109.79.58.221])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id D144E16D;
	Wed,  6 Jul 2016 18:16:03 +0100 (IST)
Subject: Re: [Bug 984910] rm fails to detect errors in readdir(3)
To: "P. Benie" <pjb1008@cam.ac.uk>
References: <bug-984910-27849-NcmPmFN8vK@http.bugzilla.opensuse.org/>
	<57698E6F.9000502@bernhard-voelker.de>
	<576A6FC7.1070700@draigBrady.com>
	<19613e89-aecb-2be1-2ab4-3c0b338d1fdb@bernhard-voelker.de>
	<576A8926.3010607@draigBrady.com>
	<9f5c9083-a51f-6352-e71a-1bed462a9946@bernhard-voelker.de>
	<alpine.LSU.2.00.1606221433250.30919@akane.eng.cam.ac.uk>
	<576AA68F.6050709@draigBrady.com> <57703A5D.4060307@draigBrady.com>
	<57703F8D.3060902@draigBrady.com>
Cc: Andreas Stieger <astieger@suse.com>, Philipp Thomas <pth@suse.de>,
	Coreutils <coreutils@gnu.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <577D3CD2.3050306@draigBrady.com>
Date: Wed, 6 Jul 2016 18:16:02 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <57703F8D.3060902@draigBrady.com>
Content-Type: multipart/mixed; boundary="------------040502030304020505020306"
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 17:16:14 -0000

This is a multi-part message in MIME format.
--------------040502030304020505020306
Content-Type: text/plain; charset=iso-8859-15
Content-Transfer-Encoding: quoted-printable

On 26/06/16 21:48, P=E1draig Brady wrote:
> On 26/06/16 21:26, P=E1draig Brady wrote:
>> On 22/06/16 15:54, P=E1draig Brady wrote:
>>> Cool, I'll add that test in your name to coreutils.
>>
>> Attached.
>=20
> Small adjustment attached.

Bernhard Voelker noticed a problem with that on 32 bit,
which the attached should fix.

thanks,
P=E1draig


--------------040502030304020505020306
Content-Type: text/x-patch;
 name="tests-readdir64.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="tests-readdir64.patch"

>From 54c1397510cb08433680b5b7da46a8201770e9ee Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <P@draigBrady.com>
Date: Wed, 6 Jul 2016 18:08:32 +0100
Subject: [PATCH] tests: avoid false failure on 32 bit in readdir() test

* tests/rm/rm-readdir-fail.sh: Since we use the returned dirent
from the readdir wrapper it must be the correct type and not
just cast.  Therefore setup so that we only have to define a
wrapper for readdir(), which works appropriately on 32 and 64 bit.
Issue reported by Bernhard Voelker, where rm was seen to invoke
rmdir() on invalid file names.
---
 tests/rm/rm-readdir-fail.sh | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/tests/rm/rm-readdir-fail.sh b/tests/rm/rm-readdir-fail.sh
index 15ef1d6..d9d8efd 100755
--- a/tests/rm/rm-readdir-fail.sh
+++ b/tests/rm/rm-readdir-fail.sh
@@ -26,6 +26,12 @@ mkdir -p dir/notempty || framework_failure_
 # Simulate "readdir" failure.
 cat > k.c <<\EOF || framework_failure_
 #define _GNU_SOURCE
+
+/* Setup so we don't have to worry about readdir64.  */
+#ifndef __LP64__
+# define _FILE_OFFSET_BITS 64
+#endif
+
 #include <dlfcn.h>
 #include <dirent.h>
 #include <errno.h>
@@ -72,11 +78,6 @@ struct dirent *readdir (DIR *dirp)
   errno = ENOENT;
   return NULL;
 }
-
-struct dirent64 *readdir64 (DIR *dirp)
-{
-  return (struct dirent64 *) readdir (dirp);
-}
 EOF
 
 # Then compile/link it:
-- 
2.5.5


--------------040502030304020505020306--


From MAILER-DAEMON Wed Jul 06 17:01:47 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bKtx5-0002li-Pb
	for mharc-coreutils@gnu.org; Wed, 06 Jul 2016 17:01:47 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:41030)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <timbuecher@mailbox.org>) id 1bKtsI-00011x-3n
	for coreutils@gnu.org; Wed, 06 Jul 2016 16:56:51 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <timbuecher@mailbox.org>) id 1bKtsD-0008Kr-44
	for coreutils@gnu.org; Wed, 06 Jul 2016 16:56:49 -0400
Received: from mx1.mailbox.org ([80.241.60.212]:33128)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <timbuecher@mailbox.org>) id 1bKtsC-0008IZ-Pz
	for coreutils@gnu.org; Wed, 06 Jul 2016 16:56:45 -0400
Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.mailbox.org (Postfix) with ESMTPS id 94E4443ADA
	for <coreutils@gnu.org>; Wed,  6 Jul 2016 22:56:38 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mailbox.org; h=
	content-type:content-type:mime-version:message-id:subject
	:subject:from:from:date:date:received; s=mail20150812; t=
	1467838597; bh=2/SIKvhrVXWWhLwgYwksjgezMcT0EkyUt5HIUUphKf8=; b=d
	vO3Dxw290TnbXLjXcXXILRf6Vj2Nn30GMUSZSerwV+ZskXuA3Kcyt5BvkPR2rpVe
	D10yqfL/yRmxjA7+W/LhdZdMU6w6JuWEOCCwtVtUYF5znwi4M3hacnTcbtvb5JEO
	g24cJZR09y8ltdxjDJ+AxTUxAerT4rJOTVLNnefow3LOn5lfiIGxED6MGoCdh5YT
	RBzNyv50TIwqJyA2tN2tD/NRGO68eAs29yo8/VsZ+76xE3BkJ8mc2VdVuMYz0473
	A9PUs2RdVMRx3zGci6gNol0QaLQKwdTkQSFKi4737sWLQsAjeqXKgfBBYKiq+oUj
	p1PNAynF3Vy82fAlsgsLg==
X-Virus-Scanned: amavisd-new at heinlein-support.de
Received: from smtp1.mailbox.org ([80.241.60.240])
	by gerste.heinlein-support.de (gerste.heinlein-support.de
	[91.198.250.173]) (amavisd-new, port 10030)
	with ESMTP id 7To71B1mb74j for <coreutils@gnu.org>;
	Wed,  6 Jul 2016 22:56:37 +0200 (CEST)
Date: Wed, 06 Jul 2016 22:56:36 +0200
From: Tim =?iso-8859-1?q?B=FCcher?= <timbuecher@mailbox.org>
Subject: [PATCH] factor: remove outdated conditional
To: coreutils@gnu.org
Message-Id: <1467838596.3494.0@smtp.mailbox.org>
MIME-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 80.241.60.212
X-Mailman-Approved-At: Wed, 06 Jul 2016 17:01:46 -0400
Content-Type: text/plain; charset=utf-8; format=flowed
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 20:56:51 -0000

diff --git a/src/factor.c b/src/factor.c
index 8fee759..27e836b 100644
--- a/src/factor.c
+++ b/src/factor.c
@@ -215,12 +215,6 @@ const unsigned char factor_clz_tab[129] =

 #endif

-#if !defined __clz_tab && !defined UHWtype
-/* Without this seemingly useless conditional, gcc -Wunused-macros
-   warns that each of the two tested macros is unused on Fedora 18.
-   FIXME: this is just an ugly band-aid.  Fix it properly.  */
-#endif
-
 /* 2*3*5*7*11...*101 is 128 bits, and has 26 prime factors */
 #define MAX_NFACTS 26

From MAILER-DAEMON Wed Jul 06 17:03:25 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bKtyf-0004Wz-1T
	for mharc-coreutils@gnu.org; Wed, 06 Jul 2016 17:03:25 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:45057)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bKtyd-0004Ws-AE
	for coreutils@gnu.org; Wed, 06 Jul 2016 17:03:24 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bKtyY-0003y8-C8
	for coreutils@gnu.org; Wed, 06 Jul 2016 17:03:22 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:51628)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bKtyY-0003xS-6V
	for coreutils@gnu.org; Wed, 06 Jul 2016 17:03:18 -0400
Received: from [192.168.1.80] (unknown [109.79.58.221])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 27E2B16D;
	Wed,  6 Jul 2016 22:03:15 +0100 (IST)
Subject: Re: [PATCH] factor: remove outdated conditional
To: =?UTF-8?Q?Tim_B=c3=bccher?= <timbuecher@mailbox.org>, coreutils@gnu.org
References: <1467838596.3494.0@smtp.mailbox.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <577D7212.8080001@draigBrady.com>
Date: Wed, 6 Jul 2016 22:03:14 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <1467838596.3494.0@smtp.mailbox.org>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 21:03:24 -0000

On 06/07/16 21:56, Tim B=C3=BCcher wrote:
> diff --git a/src/factor.c b/src/factor.c
> index 8fee759..27e836b 100644
> --- a/src/factor.c
> +++ b/src/factor.c
> @@ -215,12 +215,6 @@ const unsigned char factor_clz_tab[129] =3D
>=20
>  #endif
>=20
> -#if !defined __clz_tab && !defined UHWtype
> -/* Without this seemingly useless conditional, gcc -Wunused-macros
> -   warns that each of the two tested macros is unused on Fedora 18.
> -   FIXME: this is just an ugly band-aid.  Fix it properly.  */
> -#endif
> -
>  /* 2*3*5*7*11...*101 is 128 bits, and has 26 prime factors */
>  #define MAX_NFACTS 26
>=20

Did you test it with -Wunused-macros ?
It still gives an error here with gcc 5.3.1




From MAILER-DAEMON Wed Jul 06 18:48:19 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bKvcB-0001Ho-CT
	for mharc-coreutils@gnu.org; Wed, 06 Jul 2016 18:48:19 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:49161)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <timbuecher@mailbox.org>) id 1bKuE6-0000ih-5t
	for coreutils@gnu.org; Wed, 06 Jul 2016 17:19:23 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <timbuecher@mailbox.org>) id 1bKuE1-000843-TU
	for coreutils@gnu.org; Wed, 06 Jul 2016 17:19:21 -0400
Received: from mx1.mailbox.org ([80.241.60.212]:37882)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <timbuecher@mailbox.org>) id 1bKuE1-00083h-JJ
	for coreutils@gnu.org; Wed, 06 Jul 2016 17:19:17 -0400
Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.mailbox.org (Postfix) with ESMTPS id 0F088427FA;
	Wed,  6 Jul 2016 23:19:12 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mailbox.org; h=
	content-type:content-type:mime-version:references:in-reply-to
	:message-id:subject:subject:from:from:date:date:received; s=
	mail20150812; t=1467839950; bh=0geFuEg9YOUv+5dohAGnnd3lhku5YZ1nC
	cmFLIKIanU=; b=i2F7EROxQ3wW/zf3epvAiMxEwbKYouLU48HViTTG8KHOl3+3D
	BKIZ+y12X8FakbSmWi9qkbMuq1mpCisC4Puef77EqPXNW4d+B8SC9vEj4kO+JD/e
	H292evQeHE7/izDUdvHQObBhNWFxBPIQILvA2NxsKyP2sKZc7+7uTXDXzPcQn1pq
	Gvi/Wqx6h+uIw+0e6No6u89jk8NMaBPnuPj5X0MaSVuGBgqxqQyVKqHcPVrM3y1g
	u1XhGy1ZJTYFYVYfyRXpltLjES63lU3tc0vh3kDEFnIO7kiWJohI3Ll/V2nSdM7x
	yDxk3j9Z5HKzaK0j1Z3Lgeh+22t887U9Sgl2g==
X-Virus-Scanned: amavisd-new at heinlein-support.de
Received: from smtp1.mailbox.org ([80.241.60.240])
	by gerste.heinlein-support.de (gerste.heinlein-support.de
	[91.198.250.173]) (amavisd-new, port 10030)
	with ESMTP id f9RZJqDCL_5a; Wed,  6 Jul 2016 23:19:10 +0200 (CEST)
Date: Wed, 06 Jul 2016 23:19:09 +0200
From: Tim =?iso-8859-1?q?B=FCcher?= <timbuecher@mailbox.org>
Subject: Re: [PATCH] factor: remove outdated conditional
To: =?iso-8859-1?q?P=E1draig?= Brady <P@draigBrady.com>
Cc: coreutils@gnu.org
Message-Id: <1467839949.3494.1@smtp.mailbox.org>
In-Reply-To: <577D7212.8080001@draigBrady.com>
References: <1467838596.3494.0@smtp.mailbox.org>
	<577D7212.8080001@draigBrady.com>
MIME-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 80.241.60.212
X-Mailman-Approved-At: Wed, 06 Jul 2016 18:48:17 -0400
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Jul 2016 21:19:23 -0000

I'm sorry, seems like i messed something up there.
Good night.

Am Mi, 6. Jul, 2016 um 11:03 schrieb P=C3=A1draig Brady <P@draigBrady.com>:
> On 06/07/16 21:56, Tim B=C3=BCcher wrote:
>>  diff --git a/src/factor.c b/src/factor.c
>>  index 8fee759..27e836b 100644
>>  --- a/src/factor.c
>>  +++ b/src/factor.c
>>  @@ -215,12 +215,6 @@ const unsigned char factor_clz_tab[129] =3D
>>=20
>>   #endif
>>=20
>>  -#if !defined __clz_tab && !defined UHWtype
>>  -/* Without this seemingly useless conditional, gcc -Wunused-macros
>>  -   warns that each of the two tested macros is unused on Fedora 18.
>>  -   FIXME: this is just an ugly band-aid.  Fix it properly.  */
>>  -#endif
>>  -
>>   /* 2*3*5*7*11...*101 is 128 bits, and has 26 prime factors */
>>   #define MAX_NFACTS 26
>>=20
>=20
> Did you test it with -Wunused-macros ?
> It still gives an error here with gcc 5.3.1
>=20
>=20
=

From MAILER-DAEMON Thu Jul 07 02:24:14 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bL2jO-0001UC-39
	for mharc-coreutils@gnu.org; Thu, 07 Jul 2016 02:24:14 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55467)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bL2jL-0001Ti-GL
	for coreutils@gnu.org; Thu, 07 Jul 2016 02:24:12 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bL2jH-00089D-76
	for coreutils@gnu.org; Thu, 07 Jul 2016 02:24:10 -0400
Received: from mout.kundenserver.de ([212.227.17.13]:59502)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bL2jG-00088Q-TA
	for coreutils@gnu.org; Thu, 07 Jul 2016 02:24:07 -0400
Received: from [192.168.101.10] ([217.87.208.222]) by mrelayeu.kundenserver.de
	(mreue104) with ESMTPSA (Nemesis) id 0M5gVc-1bVpND1GZc-00xcnv;
	Thu, 07 Jul 2016 08:23:55 +0200
Subject: Re: [Bug 984910] rm fails to detect errors in readdir(3)
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>,
	"P. Benie" <pjb1008@cam.ac.uk>
References: <bug-984910-27849-NcmPmFN8vK@http.bugzilla.opensuse.org/>
	<57698E6F.9000502@bernhard-voelker.de>
	<576A6FC7.1070700@draigBrady.com>
	<19613e89-aecb-2be1-2ab4-3c0b338d1fdb@bernhard-voelker.de>
	<576A8926.3010607@draigBrady.com>
	<9f5c9083-a51f-6352-e71a-1bed462a9946@bernhard-voelker.de>
	<alpine.LSU.2.00.1606221433250.30919@akane.eng.cam.ac.uk>
	<576AA68F.6050709@draigBrady.com> <57703A5D.4060307@draigBrady.com>
	<57703F8D.3060902@draigBrady.com> <577D3CD2.3050306@draigBrady.com>
Cc: Andreas Stieger <astieger@suse.com>, Philipp Thomas <pth@suse.de>,
	Coreutils <coreutils@gnu.org>
From: Bernhard Voelker <mail@bernhard-voelker.de>
Message-ID: <577DF576.6000400@bernhard-voelker.de>
Date: Thu, 7 Jul 2016 08:23:50 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.7.0
MIME-Version: 1.0
In-Reply-To: <577D3CD2.3050306@draigBrady.com>
Content-Type: text/plain; charset=iso-8859-15
Content-Transfer-Encoding: 8bit
X-Provags-ID: V03:K0:oVMGFBuZ6AIZVtBRs7jcdwSlRTO3/bx+RyBztfFn+rOp0VCa4kK
	q+uG9Xo6pqJnKiK/TkabKi/Wx+pLQMdQuv17RVGh8iaEcETLLk7ACm8MJkV2r03js5xmRZU
	KEU05EW1gy8CcyyxHXyFu3PrvtmUWWkbdljOaq2qhE0gFJdYOoWrh1K2G+NCkpyYMKTQQcH
	xg+V624+XtdpEgCzsYNWA==
X-UI-Out-Filterresults: notjunk:1;V01:K0:pihxrdUtSS4=:I2CLThRXJKMQ9wkeoFJrpS
	QfOUpHYYqljZpROnF2jstFruEDavZoEq8Ne69GgqSjWyfl72U/0GkSsqC1/GmqwmOfeSkBHRJ
	CHANn0YHVyfiX+LwUXxH2L6UDjZK2/wTHcHmo/l27R0AtZkjEV9VKZZVqPDHmkpKd0LkARcyF
	/gzzs6P29clOPW4R+0+kPo7FAFweCeb4XzhR9Esu4s6711/l43BCgAiEECo+ShCR16vzQFprb
	qAGSMO+9DP4Jyfa6CiOakbthrSqRFpt235eENHoAhYXCRTA/yJVoFql1tgMXwfvWMRjbVGkUm
	VKJ7sCL0oXLy3Rpu4AhHSZT9Moq4gP19mhvDLNLT82hRxnCwWOx/0yzVBafS+ZaM+blyuS71R
	oJN/DtN4Eu4PBGqK8O4AZ6rQPIXOb2+u3ecwasW1u3tZWBTkVqo9QOpKYnjkjONQQC/u4wJjw
	pxtU0JSfNlbr1KeKVfUb6Y2wcOIt76WUs7dyD0Z/5E3wEd3RtA8f09Gb29IaWa1o/A5b8XETL
	fjiNcx43WFc7V9cn1Jb+H27GHRPy/JJfASvfEId7xERp6QCyvzhjnPMlRfh0zOKrEFIZpBfrz
	KH5Nm+WErYiSNwOaKHdGvx5KMiMZpvXv5nXa8EI4fpTGovEIJu7pMtiZQ86yuP/M1vawx7xzT
	/6oU2pyjEauBd2Oy80lYvTRo2vjzN5keu2ircyoZY+hYslceyez+walMV5cGghp+3rJUN3nYR
	6R0t4HD/7NM5rqag
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.17.13
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 06:24:12 -0000

On 07/06/2016 07:16 PM, Pádraig Brady wrote:
> On 26/06/16 21:48, Pádraig Brady wrote:
>> On 26/06/16 21:26, Pádraig Brady wrote:
>>> On 22/06/16 15:54, Pádraig Brady wrote:
>>>> Cool, I'll add that test in your name to coreutils.
>>>
>>> Attached.
>>
>> Small adjustment attached.
> 
> Bernhard Voelker noticed a problem with that on 32 bit,
> which the attached should fix.

Thanks, now the test passes successfully at least on:

  i586, x86_64, armv7l, and ppc64

It's still building here - on OBS, build.opensuse.org - for aarch64.

Have a nice day.
Berny


From MAILER-DAEMON Thu Jul 07 02:25:44 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bL2kq-0002li-LO
	for mharc-coreutils@gnu.org; Thu, 07 Jul 2016 02:25:44 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:56016)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bL2kp-0002cV-CP
	for coreutils@gnu.org; Thu, 07 Jul 2016 02:25:44 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bL2kl-0008VS-8z
	for coreutils@gnu.org; Thu, 07 Jul 2016 02:25:42 -0400
Received: from mout.kundenserver.de ([212.227.17.24]:56464)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bL2kk-0008VO-S8
	for coreutils@gnu.org; Thu, 07 Jul 2016 02:25:39 -0400
Received: from [192.168.101.10] ([217.87.208.222]) by mrelayeu.kundenserver.de
	(mreue103) with ESMTPSA (Nemesis) id 0M2oZ4-1bbsoZ2XQ6-00seSu;
	Thu, 07 Jul 2016 08:25:29 +0200
Subject: Re: [Bug 984910] rm fails to detect errors in readdir(3)
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>,
	"P. Benie" <pjb1008@cam.ac.uk>
References: <bug-984910-27849-NcmPmFN8vK@http.bugzilla.opensuse.org/>
	<57698E6F.9000502@bernhard-voelker.de>
	<576A6FC7.1070700@draigBrady.com>
	<19613e89-aecb-2be1-2ab4-3c0b338d1fdb@bernhard-voelker.de>
	<576A8926.3010607@draigBrady.com>
	<9f5c9083-a51f-6352-e71a-1bed462a9946@bernhard-voelker.de>
	<alpine.LSU.2.00.1606221433250.30919@akane.eng.cam.ac.uk>
	<576AA68F.6050709@draigBrady.com> <57703A5D.4060307@draigBrady.com>
	<57703F8D.3060902@draigBrady.com> <577D3CD2.3050306@draigBrady.com>
	<577DF576.6000400@bernhard-voelker.de>
Cc: Andreas Stieger <astieger@suse.com>, Philipp Thomas <pth@suse.de>,
	Coreutils <coreutils@gnu.org>
From: Bernhard Voelker <mail@bernhard-voelker.de>
Message-ID: <577DF5D6.5080708@bernhard-voelker.de>
Date: Thu, 7 Jul 2016 08:25:26 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.7.0
MIME-Version: 1.0
In-Reply-To: <577DF576.6000400@bernhard-voelker.de>
Content-Type: text/plain; charset=iso-8859-15
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K0:hWP+PMdlsYqApJNlBp7sE7C3ozaSHKUMQuJ+A4N8tQkJFaLyOq1
	GY+HDY2EQ1a77znyvuAweWUr/Wj4lMYcnDb1TYoDG/KcagS0VsWFuEorhEyVrIkdnQMkKdA
	xqCldEg3VsQonqkYRQnknvmh0YWBR7HErKOpzi3ibU9tOXvWIq6LhO3WSTBErBK8KRArlhW
	Nl0KbZTrivtlG3ZF93bbg==
X-UI-Out-Filterresults: notjunk:1;V01:K0:/4MzUrzybF4=:WrveFrLBOe0FmKi/6yFcRZ
	rEkg2f6U1P5OXmxXtUlEg9gPzBKK9OFDYWzh6k4cZBgKFVNbJGHwZd6DY+bjUbjpA6065KD5j
	XbS/+ftQ4p/W3fJK6K47eX3r437JrChdMbr1SVSl3WWmJXBBuPeiDRBi45FkeowaLt6D4jhcT
	5zca/sKW5b9x1mCyxF4dPDNZvA6h1Y7boaQIHwa6rnhWCQSqh4BnsQpR2yvV+8brzFA4ucQVm
	p6onm7PPOP0uPw7PQFKqWn0XqVyGa1+qV8iAf2tCsXZ+Okt9mxW10DFqV2F5jYagJw62ng99O
	q2uCHoHipKjIYSV9MBrUhb4SNw7f5U2B0ZVvkvp5qE+q3gik6b6JpSMs8+FENV9qsmSW1+qyW
	KLkH1RdEze3yAyCFnYD2e4E0YX01ScpGpIPJXBWoMhKdKv/f98PFzPd1ejAzrG2RrhvD4mVcM
	UMw5eQ4j1FcoIxhelLDH1oBY2z+gihw9DYZXzvgrEgQ97CioGw0Fxgc/qtdYxwOlTdlXt+rT+
	MKtysjQKlbW0pTD+3YhkDJyaUfUCDMtAyaIrWMXTLGnqEj97R6pikuamLHDETLCk4Mjg+wN0S
	uRrWQreP576OT24axCTGgYbkzM4igca64P8jVMuEGY6HpvUrQhUnNUPv5m9qH+LZUBnEmom89
	Ml9/CvGvrf6u3/glPn39Q6XfDXQz7OoA2dMQi7VV6T1d35IPN0msuJl/90cJLzQl23cPa8Fi5
	mhKS3nBeEezH+6NX
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.17.24
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Jul 2016 06:25:44 -0000

On 07/07/2016 08:23 AM, Bernhard Voelker wrote:
> Thanks, now the test passes successfully at least on:
> 
>   i586, x86_64, armv7l, and ppc64
> 
> It's still building here - on OBS, build.opensuse.org - for aarch64.

... and it just passed on aarch64, too. ;-)

Please push.

Thanks & have a nice day,
Berny


From MAILER-DAEMON Mon Jul 11 11:50:11 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bMdTH-0005th-R2
	for mharc-coreutils@gnu.org; Mon, 11 Jul 2016 11:50:11 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:52614)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <ablacktshirt@gmail.com>) id 1bMdOR-0002iD-8a
	for coreutils@gnu.org; Mon, 11 Jul 2016 11:45:12 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <ablacktshirt@gmail.com>) id 1bMdON-0004tI-4k
	for coreutils@gnu.org; Mon, 11 Jul 2016 11:45:10 -0400
Received: from mail-pa0-x243.google.com ([2607:f8b0:400e:c03::243]:34351)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <ablacktshirt@gmail.com>) id 1bMdOM-0004sw-US
	for coreutils@gnu.org; Mon, 11 Jul 2016 11:45:07 -0400
Received: by mail-pa0-x243.google.com with SMTP id hh10so3393745pac.1
	for <coreutils@gnu.org>; Mon, 11 Jul 2016 08:45:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=to:from:subject:message-id:date:user-agent:mime-version
	:content-transfer-encoding;
	bh=jLLWndUKTZE/07xwdW6Y25mkZmXkguSaWLfW9jSVjT4=;
	b=iCfHIpYIUCwYLHS6/RFsBvovaovZffP9oufUTF049IeNX4DvSvgRssxSd9GbBz2VAI
	JkcILi1Qo/EJKyMXu8JVPd0eSRUtUdptnPmqB4joPjhonNy4sZvLoUctilffYoYHULMd
	bQ+e0o0NfF5jB84uA5C4Rdp1k5nftHSN9ld2zawZpT2Fj2UYYj0S28c12WXKkEqLgDcB
	U4nD/52W89GEohU4Cnqo6qJeD6KbttUt/FuuaBzzjpDp/F39ObNS2cPCJtghaN445TCf
	xRfdWPz7Z2E+yJciKze2YttiiEnmTw3fU+dj2azPoPpLinF3IMI3CYDgyUadt4Bs8w1C
	T7pg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:to:from:subject:message-id:date:user-agent
	:mime-version:content-transfer-encoding;
	bh=jLLWndUKTZE/07xwdW6Y25mkZmXkguSaWLfW9jSVjT4=;
	b=bU69l8WNCuKcZd0cLftzRQP3T0SIui7NXCbF9wR/XfXG63bBUvettJduTAHczbWdTP
	ejbv7g8QSB3YDUU+Kqt1Eqd3LvwGa3uwfYWMnj6RHoCSNb5uCY1m8jxMgrxVJJJsFyst
	g8OUDSd168t2Hx69SFpW2w7F9bxvKhMNE6SS4QG3djRuwZp/JPL55Vo+R7qzUAuZVGYH
	fSTHx2xcTS1aVh8HEFhCVOENrCtrK3bZJBUMLgBmplKjL140yjJBXEDSN3eh4fDDqLx1
	GQKqu15meCQVXTvpIifrIh0dqkeeYxITvnhBpP1hTfcah9Wc1Tc8i+RzRQPOvJDcesqI
	78xQ==
X-Gm-Message-State: ALyK8tLqqM5ZMWZzI8CRtdmIBPQUxbFFgtiN/kHifLSVdnuJpY+d2kwtBmPpg0ilMRqAGg==
X-Received: by 10.66.178.49 with SMTP id cv17mr36621276pac.157.1468251905934; 
	Mon, 11 Jul 2016 08:45:05 -0700 (PDT)
Received: from [192.168.10.106] ([110.64.91.55])
	by smtp.gmail.com with ESMTPSA id 6sm2755328pfs.29.2016.07.11.08.45.04
	for <coreutils@gnu.org>
	(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
	Mon, 11 Jul 2016 08:45:05 -0700 (PDT)
To: coreutils@gnu.org
From: Yubin Ruan <ablacktshirt@gmail.com>
Subject: what does AT mean in a linker scripts ?
Message-ID: <5783BEFE.9010807@gmail.com>
Date: Mon, 11 Jul 2016 23:45:02 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.8.0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400e:c03::243
X-Mailman-Approved-At: Mon, 11 Jul 2016 11:50:10 -0400
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 11 Jul 2016 15:45:12 -0000

Hi,
I am reading some C code and some linker scripts for `ld`.
I find something really hard to understand:

     .text : AT(0x100000) {
         *(.text .stub .text.* .gnu.linkonce.t.*)
     }

I understand what the `.text` and the wild char * do, but what does the 
`AT` after the colon do ?
Need help.
And also, why is there no man page describing the syntax of linker 
scripts ?

Regards,
Ruan


From MAILER-DAEMON Mon Jul 11 11:55:32 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bMdYS-0001AO-NP
	for mharc-coreutils@gnu.org; Mon, 11 Jul 2016 11:55:32 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55225)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bMdYR-00014d-0q
	for coreutils@gnu.org; Mon, 11 Jul 2016 11:55:32 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bMdYN-00074x-Ps
	for coreutils@gnu.org; Mon, 11 Jul 2016 11:55:30 -0400
Received: from mx1.redhat.com ([209.132.183.28]:56305)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bMdYN-00074s-Jv
	for coreutils@gnu.org; Mon, 11 Jul 2016 11:55:27 -0400
Received: from int-mx11.intmail.prod.int.phx2.redhat.com
	(int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.redhat.com (Postfix) with ESMTPS id CC47264070;
	Mon, 11 Jul 2016 15:55:26 +0000 (UTC)
Received: from [10.3.116.127] (ovpn-116-127.phx2.redhat.com [10.3.116.127])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id u6BFtQ9s006361; Mon, 11 Jul 2016 11:55:26 -0400
Subject: Re: what does AT mean in a linker scripts ?
To: Yubin Ruan <ablacktshirt@gmail.com>, coreutils@gnu.org
References: <5783BEFE.9010807@gmail.com>
From: Eric Blake <eblake@redhat.com>
Openpgp: url=http://people.redhat.com/eblake/eblake.gpg
Organization: Red Hat, Inc.
Message-ID: <5783C16E.50409@redhat.com>
Date: Mon, 11 Jul 2016 09:55:26 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <5783BEFE.9010807@gmail.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="Cal5EK0rB1NNOd3r4nEIK9apkxf23IFLQ"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(mx1.redhat.com [10.5.110.38]);
	Mon, 11 Jul 2016 15:55:26 +0000 (UTC)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 209.132.183.28
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 11 Jul 2016 15:55:32 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--Cal5EK0rB1NNOd3r4nEIK9apkxf23IFLQ
From: Eric Blake <eblake@redhat.com>
To: Yubin Ruan <ablacktshirt@gmail.com>, coreutils@gnu.org
Message-ID: <5783C16E.50409@redhat.com>
Subject: Re: what does AT mean in a linker scripts ?
References: <5783BEFE.9010807@gmail.com>
In-Reply-To: <5783BEFE.9010807@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 07/11/2016 09:45 AM, Yubin Ruan wrote:
> Hi,
> I am reading some C code and some linker scripts for `ld`.

However, you've asked your question on the coreutils list, which is not
responsible for 'ld', nor do we have any custom linker scripts in use in
this project. You may be better off asking your questions on the
binutils mailing list, as the owner of 'ld'.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--Cal5EK0rB1NNOd3r4nEIK9apkxf23IFLQ
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJXg8FuAAoJEKeha0olJ0Nq3tsIAIRsA9tnv6CFabW0v/leRtNo
l97UAeTsxGdXsriYqpk4b0pHw8SwmAN928gPaiGvpHGAxpnK2K2hro6ECkWgv8Q0
B62Jp9gW6sZrLTBIuxTMb2/W5dyV6QZ8W2650VvcgCu7kXrdN+HMYLCpJQDlB39M
s9NKmpucrHkV/MBhvHO/NzeZ41CPfFgpiooVgyfLgda2zGhZ7yFfoi9cOjQrgSw9
pVyWm6KhkzMXGFeL/7Tw0EDMV4BIvc9O66xffwiw9hsICNlkkg/B9lvk2YTonUPf
F0zUZ4hbs34GP8fdynW0RDUjEBReJl5XGeTKaCoT6p51cup6wZOwmW0+wZKFNAc=
=eetg
-----END PGP SIGNATURE-----

--Cal5EK0rB1NNOd3r4nEIK9apkxf23IFLQ--


From MAILER-DAEMON Sat Jul 16 10:29:40 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bOQb6-0006i5-ST
	for mharc-coreutils@gnu.org; Sat, 16 Jul 2016 10:29:40 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:45505)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <alsoijw@yandex.ru>) id 1bOQHg-0003Nv-6b
	for coreutils@gnu.org; Sat, 16 Jul 2016 10:09:36 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <alsoijw@yandex.ru>) id 1bOQHb-0007ZH-TL
	for coreutils@gnu.org; Sat, 16 Jul 2016 10:09:35 -0400
Received: from forward18j.cmail.yandex.net ([5.255.227.237]:45295)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <alsoijw@yandex.ru>) id 1bOQHa-0007Ub-Hw
	for coreutils@gnu.org; Sat, 16 Jul 2016 10:09:31 -0400
Received: from mxback5m.mail.yandex.net (mxback5m.mail.yandex.net
	[37.140.138.65])
	by forward18j.cmail.yandex.net (Yandex) with ESMTP id EB45C210FD
	for <coreutils@gnu.org>; Sat, 16 Jul 2016 17:09:17 +0300 (MSK)
Received: from web3m.yandex.ru (web3m.yandex.ru [37.140.138.94])
	by mxback5m.mail.yandex.net (nwsmtp/Yandex) with ESMTP id
	RhhmpehNdb-9HZ8ZQ8N; Sat, 16 Jul 2016 17:09:17 +0300
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail;
	t=1468678157; bh=OhISJ+B0wJKzFbZenU5z0qgVCxS1O5kYZbQM43UNWLA=;
	h=From:To:Subject:Message-Id:Date;
	b=YgaVZl4yJSdR53kh1jc0v/cJWoVIimCdvi6yjzymRxcw/PyfhDlIAY+05Tm3p1g95
	eap7aghtYAie4eOEUnvcSMH6ftuW3aWlsUKRsXDUWl5rH7QrE8ZBM0RxLlB9HpjjJo
	mS1HlXdTUaS4yLqRhwzZagvOwDuCo3cjo6zJvUn0=
Authentication-Results: mxback5m.mail.yandex.net; dkim=pass header.i=@yandex.ru
Received: by web3m.yandex.ru with HTTP;
	Sat, 16 Jul 2016 17:09:17 +0300
From: al so <alsoijw@yandex.ru>
Envelope-From: alsoijw@yandex.ua
To: coreutils@gnu.org
Subject: Command arguments of echo
MIME-Version: 1.0
Message-Id: <2101468678157@web3m.yandex.ru>
X-Mailer: Yamail [ http://yandex.ru ] 5.0
Date: Sat, 16 Jul 2016 17:09:17 +0300
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
	[fuzzy]
X-Received-From: 5.255.227.237
X-Mailman-Approved-At: Sat, 16 Jul 2016 10:29:40 -0400
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Jul 2016 14:09:37 -0000

How about add support "-u" command line argument?


From MAILER-DAEMON Sat Jul 16 10:33:34 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bOQes-0001XZ-AJ
	for mharc-coreutils@gnu.org; Sat, 16 Jul 2016 10:33:34 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:48366)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bOQeq-0001TT-8R
	for coreutils@gnu.org; Sat, 16 Jul 2016 10:33:33 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bOQem-0002wy-E2
	for coreutils@gnu.org; Sat, 16 Jul 2016 10:33:31 -0400
Received: from mx1.redhat.com ([209.132.183.28]:38166)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bOQem-0002wt-8w
	for coreutils@gnu.org; Sat, 16 Jul 2016 10:33:28 -0400
Received: from int-mx11.intmail.prod.int.phx2.redhat.com
	(int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.redhat.com (Postfix) with ESMTPS id 8D9127F7BA;
	Sat, 16 Jul 2016 14:33:27 +0000 (UTC)
Received: from [10.3.116.197] (ovpn-116-197.phx2.redhat.com [10.3.116.197])
	by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id u6GEXRw4017372; Sat, 16 Jul 2016 10:33:27 -0400
Subject: Re: Command arguments of echo
To: al so <alsoijw@yandex.ru>, coreutils@gnu.org
References: <2101468678157@web3m.yandex.ru>
From: Eric Blake <eblake@redhat.com>
Openpgp: url=http://people.redhat.com/eblake/eblake.gpg
Organization: Red Hat, Inc.
Message-ID: <578A45B6.6050803@redhat.com>
Date: Sat, 16 Jul 2016 08:33:26 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <2101468678157@web3m.yandex.ru>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="Egx0x167l4S4ImQxED4VjB2Q8SOxK7hRj"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(mx1.redhat.com [10.5.110.28]);
	Sat, 16 Jul 2016 14:33:27 +0000 (UTC)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 209.132.183.28
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Jul 2016 14:33:33 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--Egx0x167l4S4ImQxED4VjB2Q8SOxK7hRj
From: Eric Blake <eblake@redhat.com>
To: al so <alsoijw@yandex.ru>, coreutils@gnu.org
Message-ID: <578A45B6.6050803@redhat.com>
Subject: Re: Command arguments of echo
References: <2101468678157@web3m.yandex.ru>
In-Reply-To: <2101468678157@web3m.yandex.ru>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 07/16/2016 08:09 AM, al so wrote:
> How about add support "-u" command line argument?

That does what? Without an explanation of what you want it to do, the
answer is an unequivocal no.  And even if you did say what it should do,
the answer is still probably no - 'echo' is rather strictly defined by
POSIX to not take arguments because of back-compat reasons, so adding
new arguments makes it tricky to still comply with POSIX.  You'd have to
have a REALLY strong argument for why your new option is worthwhile, and
why it won't break existing scripts that use 'echo -u' and expect a
literal '-u' to be output, based on POSIX rules.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--Egx0x167l4S4ImQxED4VjB2Q8SOxK7hRj
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJXikW2AAoJEKeha0olJ0NqPHQH/jsIK9tW/8U+7TISEvM62ryF
Skkdeo6FGyNlJfwtasEcXxVj+gxb4qAGrWMMHgkzLN66L4316DNvjQBhZThkv/JI
KdGtwwl6hSN5VUY/P9ySyKrLj/616zM6UhCi5FheIkgkAMoF+YE0B96076+Rxtyd
t39FttEQUo+jTVM9LLnlueDOsHj/jSEeV1rNevIkhVaQ4vPwL4hnsxnbP6OrnR86
YFclYRNYMFuAeUoC8EbCOYCRu3qJbQ+LT0dCzrfdNxM2GV/wrbqXOPFCUz1jjZOy
53iPtroHEBgwyFTQ2j8FNrRdKN+Q9ET4ITqHfj30F60vfZc+MXLVip3RxljHLec=
=X4gL
-----END PGP SIGNATURE-----

--Egx0x167l4S4ImQxED4VjB2Q8SOxK7hRj--


From MAILER-DAEMON Sat Jul 16 21:52:19 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bObFj-0000zI-OT
	for mharc-coreutils@gnu.org; Sat, 16 Jul 2016 21:52:19 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:56828)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bObFh-0000zC-2a
	for coreutils@gnu.org; Sat, 16 Jul 2016 21:52:18 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bObFc-0001Ky-C6
	for coreutils@gnu.org; Sat, 16 Jul 2016 21:52:16 -0400
Received: from mail-qk0-x22f.google.com ([2607:f8b0:400d:c09::22f]:34073)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bObFc-0001Kl-6S
	for coreutils@gnu.org; Sat, 16 Jul 2016 21:52:12 -0400
Received: by mail-qk0-x22f.google.com with SMTP id o67so132021300qke.1
	for <coreutils@gnu.org>; Sat, 16 Jul 2016 18:52:10 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc:message-id:references
	:to; bh=Z9syOzDM5MwkmVqbEhQ3uRBcqPPILtvaTz2sRYuKLK0=;
	b=GA6r1oBEAwWbI82RdrM18awfO15iPxnF/TaFQ2R9fnUI4yw00AJan5FqKQlPwawyAZ
	+5r7wFo6o63kvbGPwDNleZy6KuS7RF74GzHQ3NKRKrLKIRch9c9DzE6jOqwQpBf1CXMo
	LMh1w1MzfINMsgnoJTwG8SvTYIII5p7BEFI09v50R0VwWzUsWQTC03cHHqxGZiEZ8p6q
	NUAjAuiaC4OVCumGJ3aEwxAAQdO4lvaUnfc+/7hQzIlLqzT2SDLUEkmqzYFj51dvYJTe
	br2+XmmcAJHZrTD1xLXlJ8jCfXAaWK9VSUtLm+PDhCOs5X6zNXcmlQCbMDUZaNutxu3J
	We+w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:message-id:references:to;
	bh=Z9syOzDM5MwkmVqbEhQ3uRBcqPPILtvaTz2sRYuKLK0=;
	b=D4aAZHWKnIG1YyjQJRBrtCAM/0cmspEAxN5IxtFcif2YMDhEssONwGYIaPaN/f3iv+
	97mo4jVhUPVbDmHpieroG/xqFvWeCOw7yNZOkucArhUomCK0w3OEN9mlOGQ106fBW/IG
	qDriMiWwdkYACETVXGhzPXAkKu8A/30T+oheAgRiLcIkMSRn/bWnWjugMnhHbvzQVw5A
	9G/GT/4rQeh3fdEIJiWaw6YqKZ4m8GCUfvKFfOxOPhdXVs1gmQHIMqxMa5bBx+32wXrW
	vZ8qdGmmbh+O1hoqoC0FAVYSi3H3oXG/BM1Xqtsei7z34GKBtMKfPSKuxZ95uz/nIu/M
	exZg==
X-Gm-Message-State: ALyK8tKApeYxHbGToGJGbr9NLs1+7+1tkyLhLGiJhrrV6esST0dFMVAzhP/mSFr9TTUppA==
X-Received: by 10.55.188.198 with SMTP id m189mr34074502qkf.134.1468720330395; 
	Sat, 16 Jul 2016 18:52:10 -0700 (PDT)
Received: from [172.31.98.104] (rrcs-50-74-108-28.nyc.biz.rr.com.
	[50.74.108.28])
	by smtp.gmail.com with ESMTPSA id q13sm1075081qkq.6.2016.07.16.18.52.09
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Sat, 16 Jul 2016 18:52:09 -0700 (PDT)
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_DCB16250-34FA-4DA7-8168-FD8FF271AA7F"
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: expand/unexpand: add tests, refactor common code
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <57710666.8030502@draigBrady.com>
Date: Sat, 16 Jul 2016 21:52:08 -0400
Cc: Coreutils <coreutils@gnu.org>,
 jonathan.russ.buchanan@gmail.com
Message-Id: <04456105-8B1F-4DCD-8B75-FE4B57ACA07C@gmail.com>
References: <6CFC7CFB-3C94-4284-A531-6E1C559D855F@gmail.com>
	<576E6912.8040304@draigBrady.com>
	<C8562056-8D19-4313-B672-02A933E3618B@gmail.com>
	<57710666.8030502@draigBrady.com>
To: =?utf-8?Q?P=C3=A1draig_Brady?= <P@draigBrady.com>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c09::22f
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Jul 2016 01:52:18 -0000


--Apple-Mail=_DCB16250-34FA-4DA7-8168-FD8FF271AA7F
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=windows-1252

Hello,

> On Jun 27, 2016, at 06:56, P=E1draig Brady <P@draigBrady.com> wrote:
>=20
> On 27/06/16 06:17, Assaf Gordon wrote:
>> Hello P=E1draig and all,
>>=20
>>> On Jun 25, 2016, at 07:20, P=E1draig Brady <P@draigBrady.com> wrote:
>>>=20
>>> As part of this, or at least before looking at multibyte changes,
>>> it would be worth considering this proposal for changing the
>>> unexpand algorithm: http://bugs.gnu.org/23335
>>=20
>> The above bug-report addresses this TODO item:
>> =3D=3D=3D
>> unexpand: =
[http://www.opengroup.org/onlinepubs/007908799/xcu/unexpand.html]
>>  printf 'x\t \t y\n'|unexpand -t 8,9 should print its input, =
unmodified.
>>  printf 'x\t \t y\n'|unexpand -t 5,8 should print "x\ty\n"
>> =3D=3D=3D
>=20
> I think the second command is wrong there actually?
> Surely it should print "x\t\t y\n"

Digging a bit deeper about various 'unexpand' implementation, it seems =
there are more differences.
Attached is a summary of most of coreutil's unexpand tests on various =
systems.
The trivial cases give the same results, but more tricky cases (e.g. the =
'blanks' and 'posix' tests) do differ.

The test script is here: =
http://files.housegordon.org/tmp/test-unexpand-2.sh
(the last 'ff' octet for AIX can be ignored, I suspect a bug in AIX's =
unexpand when lines are not '\n' terminated).

Example (the inputs are 'blank-1' and 'blank-11' from =
<coreutils>/tests/misc/unexpand.pl):

blanks-1   AIX-1                09 62 09 09 63 09 09 09 64
blanks-1   Darwin-14.4.0        20 62 09 20 63 09 09 20 64=20
blanks-1   FreeBSD-10.1-RELEASE 20 62 09 20 63 09 09 20 64=20
blanks-1   Linux-3.16.0-4-amd64 09 62 09 09 63 09 09 09 64
blanks-1   SunOS-5.11           20 62 20 20 63 20 20 20 64

blanks-11  AIX-1                09 09 34
blanks-11  Darwin-14.4.0        09 34=20
blanks-11  FreeBSD-10.1-RELEASE 09 34=20
blanks-11  Linux-3.16.0-4-amd64 09 09 34
blanks-11  SunOS-5.11           09 20 34


And so I wonder if it's best to leave unexpand's algorithm as-is, for =
the sake of backwards-compatability (if someone is expecting coreutils' =
expected behavior),
and then focus back on multibyte character processing in 'expand' (with =
or without using the refactoring patches).


--Apple-Mail=_DCB16250-34FA-4DA7-8168-FD8FF271AA7F
Content-Disposition: attachment;
	filename=unexpand-comparison.txt.xz
Content-Type: application/octet-stream;
	name="unexpand-comparison.txt.xz"
Content-Transfer-Encoding: base64

/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4DY8BTxdADCMQBrEEklqhP44G3m78WnBTAD3vfj8KzPp
a7cwsVXknnuCQ9AicpYUBIxkkHWB5UZ3bVxzY9PEge6JUmwPfJ65ek3L6JkwZg9xeY8LHhmwjrr8
uCKQyPE+yOQh98VpPrQlzkDHAEbbtBf4RCQQEYkZJzpXDFojo/zuLEANqfnh3AMycxDi59zam9pm
NJol9QHYkC19RQ0ajevJlwV51n0w7aqDEGcPxXHMk6wPtjs+ezYb3JrQghrfVinL7u90gZdQGhj5
2QBAhwn8MpaAmtGogFiOzeQb+BRc6ECXQfQU/KfQqCg99BmBNozTujQz8rTtWfbC31fFQeRIC30q
hWDUbt7wFL6YoYAYMhhWxe+I3EyRcUifW7vzmBQcQj/FEEX7b9/IaKw02gSQT/vQrZLNXm3tUzca
drBKf7XeN4qwgwXCFQnOaizxWybVkmIrQnNpKmojB9FHYqqUcnA6hu7jkexCGIftf6Q9n/Ueg5Kh
fR0Zk1iMxNoep0i5AD+8KFnLBcGhf4MnhJPM9+imcbWpXP0DodukqqmB6KA6cy7SNZF5UwcHNwPy
3fmP4GDa5le+wYqGHSGkDMUlYlR+SUZ0bpzfBsKob9uYPVXc7w/hR6b15pAbfU9DfZT5mYRAwiT9
9ucE4x7MVh9ra2La1u6JaplComRMdx1uK9P+1xJXigku6VFUs5aQtkOUqyE7wX5lY2nB/BBX+2vB
Fb6VuI/YmvEaLjY+RacWxEaChhu7TUCEQhcCQTzU2h4bCjbt0GCAFGg36xYUlRZQY1yPduauw7FR
t/7LTJTwxBbl2GxsEAP92zMSgQClJhxa+0GOTqegQo+EJKWIIelzLvrPCjlzH2t0qISzjo1n6CyP
6x33lQPDeekI6PZVwv/OgJTPStr7qZO4ooMoBLk7g4yl221aoYHpxWqX2LJ1JnEamP5EkCjwJscO
qL1DQtouB1yzRDuVPZYJJjnhLuibVoQJ7SAWQ2ugqbOHcYBm1XNR4COpy8FFTv4NJavbCBfVhALe
SkulUxRE7ziUK9vw+UDbLPBsF6SU4PgFirMngIT8pAXaf3KWvSP5doaHh7RtHflTub6cB1YvVh8h
mRI+b8F3KBtcURUDRK5irOZQunanoBhmdPh+qEJVF4LPC7G8v+qePSRXdDR/D4MvLLg8KqvKmh6t
NnsgeVcNL9dBeyGxjS1pN+aRkVPQfRjDaNCLGOfR2h/isavUO7gBOovagCEhPxj2+IROEyzUB+gw
UqEepmIiEYkHqKCCviEl1mXwIp4DOzADpeBDxjSTFSu7FfC3ZDxuAj93MDUsOEIjM3rGJ5lhRwwO
4+TqKp1441W/cfQwDo65DSn9N3e4jlkQGCV2CfU6gMqFs07yA4J+JXioWdurRNs0bEeb2MZ1eOHq
3uYDbC34w1exoFIr/xDXuKb0PqOqfeQgkqhn62idK6J/JOgg6xJFYM9nnIW7Vpbb8JUete1nN6tW
/i97jUYmdSW3J/TrmxWLN9wDxnKDRwjin39Ozm/2L3Ljnx1cKarst9+3RxZl1y9mGvpRYyA2/CB/
bv+SJ8qdQu1Sku6w92o8oznPLohVv5b32ApE9dbYlVHBEitkLpUi75GBSev5ODI9rCcUCH1ov+yD
63rdYec1fsiJjxw1HIlFkvCFNlY+s4PMBOhpAG4zE+QmrCFEpYGk4cR/TSkyUIr3iKR05hwhKjtU
anemr9gX9pGhGnJ18S2ijHpwA4FiqtJauQzKIJxLnWa48869kk4+jGOBXE14Rv1IsnSZRLHkaMOR
euBlABDLct/j2fluAAHYCr1sAAA99l3ZscRn+wIAAAAABFla
--Apple-Mail=_DCB16250-34FA-4DA7-8168-FD8FF271AA7F
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=windows-1252




regards,
 - assaf


--Apple-Mail=_DCB16250-34FA-4DA7-8168-FD8FF271AA7F--


From MAILER-DAEMON Sun Jul 17 08:50:02 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bOlWE-0006T6-LT
	for mharc-coreutils@gnu.org; Sun, 17 Jul 2016 08:50:02 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:46426)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bOlWC-0006Ss-2b
	for coreutils@gnu.org; Sun, 17 Jul 2016 08:50:01 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bOlW6-0002PH-NA
	for coreutils@gnu.org; Sun, 17 Jul 2016 08:49:59 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:50290)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bOlW6-0002P5-Gf
	for coreutils@gnu.org; Sun, 17 Jul 2016 08:49:54 -0400
Received: from [192.168.1.80] (unknown [109.77.17.63])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 39BA7245;
	Sun, 17 Jul 2016 13:49:50 +0100 (IST)
Subject: Re: expand/unexpand: add tests, refactor common code
To: Assaf Gordon <assafgordon@gmail.com>
References: <6CFC7CFB-3C94-4284-A531-6E1C559D855F@gmail.com>
	<576E6912.8040304@draigBrady.com>
	<C8562056-8D19-4313-B672-02A933E3618B@gmail.com>
	<57710666.8030502@draigBrady.com>
	<04456105-8B1F-4DCD-8B75-FE4B57ACA07C@gmail.com>
Cc: Coreutils <coreutils@gnu.org>, jonathan.russ.buchanan@gmail.com
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <578B7EED.4090209@draigBrady.com>
Date: Sun, 17 Jul 2016 13:49:49 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <04456105-8B1F-4DCD-8B75-FE4B57ACA07C@gmail.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 17 Jul 2016 12:50:01 -0000

On 17/07/16 02:52, Assaf Gordon wrote:
> Hello,
>=20
>> On Jun 27, 2016, at 06:56, P=E1draig Brady <P@draigBrady.com> wrote:
>>
>> On 27/06/16 06:17, Assaf Gordon wrote:
>>> Hello P=E1draig and all,
>>>
>>>> On Jun 25, 2016, at 07:20, P=E1draig Brady <P@draigBrady.com> wrote:
>>>>
>>>> As part of this, or at least before looking at multibyte changes,
>>>> it would be worth considering this proposal for changing the
>>>> unexpand algorithm: http://bugs.gnu.org/23335
>>>
>>> The above bug-report addresses this TODO item:
>>> =3D=3D=3D
>>> unexpand: [http://www.opengroup.org/onlinepubs/007908799/xcu/unexpand=
.html]
>>>  printf 'x\t \t y\n'|unexpand -t 8,9 should print its input, unmodifi=
ed.
>>>  printf 'x\t \t y\n'|unexpand -t 5,8 should print "x\ty\n"
>>> =3D=3D=3D
>>
>> I think the second command is wrong there actually?
>> Surely it should print "x\t\t y\n"
>=20
> Digging a bit deeper about various 'unexpand' implementation, it seems =
there are more differences.
> Attached is a summary of most of coreutil's unexpand tests on various s=
ystems.
> The trivial cases give the same results, but more tricky cases (e.g. th=
e 'blanks' and 'posix' tests) do differ.
>=20
> The test script is here: http://files.housegordon.org/tmp/test-unexpand=
-2.sh
> (the last 'ff' octet for AIX can be ignored, I suspect a bug in AIX's u=
nexpand when lines are not '\n' terminated).
>=20
> Example (the inputs are 'blank-1' and 'blank-11' from <coreutils>/tests=
/misc/unexpand.pl):
>=20
> blanks-1   AIX-1                09 62 09 09 63 09 09 09 64
> blanks-1   Darwin-14.4.0        20 62 09 20 63 09 09 20 64=20
> blanks-1   FreeBSD-10.1-RELEASE 20 62 09 20 63 09 09 20 64=20
> blanks-1   Linux-3.16.0-4-amd64 09 62 09 09 63 09 09 09 64
> blanks-1   SunOS-5.11           20 62 20 20 63 20 20 20 64
>=20
> blanks-11  AIX-1                09 09 34
> blanks-11  Darwin-14.4.0        09 34=20
> blanks-11  FreeBSD-10.1-RELEASE 09 34=20
> blanks-11  Linux-3.16.0-4-amd64 09 09 34
> blanks-11  SunOS-5.11           09 20 34
>=20
>=20
> And so I wonder if it's best to leave unexpand's algorithm as-is, for t=
he sake of backwards-compatability (if someone is expecting coreutils' ex=
pected behavior),
> and then focus back on multibyte character processing in 'expand' (with=
 or without using the refactoring patches).

I think the existing algorithm is fine,
and the refactoring patch should go in now.

We should move the two items from TODO to tests though,
to record this investigation.

# comment that this should arguably minimize translation
# as is done on Solaris, and not modify input, but at least
# verify prints "x\t\t\t y\n"
printf 'x\t \t y\n'|unexpand -t 8,9

# verify prints "x\t\t y\n"
printf 'x\t \t y\n'|unexpand -t 5,8

That with the previous 'extern' patch adjustment I sent
and it's good to push.

thanks!
P=E1draig


From MAILER-DAEMON Wed Jul 20 02:11:27 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bPkj9-0006rE-BZ
	for mharc-coreutils@gnu.org; Wed, 20 Jul 2016 02:11:27 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:51341)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bPkj7-0006r7-CY
	for coreutils@gnu.org; Wed, 20 Jul 2016 02:11:26 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bPkj3-0000AD-1W
	for coreutils@gnu.org; Wed, 20 Jul 2016 02:11:24 -0400
Received: from mail-qk0-x242.google.com ([2607:f8b0:400d:c09::242]:34171)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bPkj2-0000A9-RH
	for coreutils@gnu.org; Wed, 20 Jul 2016 02:11:20 -0400
Received: by mail-qk0-x242.google.com with SMTP id p126so3227280qke.1
	for <coreutils@gnu.org>; Tue, 19 Jul 2016 23:11:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=from:content-transfer-encoding:subject:message-id:date:to
	:mime-version; bh=uyTJDsx9ruOXuzGRkaeHObWVY/NlKLWt8QXrL3fIEo4=;
	b=CMzIVZk3AYf1HVyhHuJo2N5mZ1XaPaE1WaCuysxvEmZYAme9b21IuTgwmXTaoa4+gq
	pQP9R2z53kIWd378ir321bHvigYn8es7KNYbs1Lwg+5oVpzFzdRTyKCbDBkzjD6JkRs/
	a6qsxbDZupNjuMxpqXIqZ8fdRclnGSoZTq0UG6/OeCOXSDTtw4XFCX8U2lQJ+xd1zx4h
	mDvZTSExgczSVYmDMgSEzDC01BeUT6CxGiR237oTob/0pzjB09Xy8Iid8BDYa09ZmoCq
	K7f3nrJJd+ZTPXmjOo/m1qrjK19+ZFZnbmC2nlNGQ2i/VyMPNFr7KbDXkCzYvQd+FKi3
	8vpg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:from:content-transfer-encoding:subject
	:message-id:date:to:mime-version;
	bh=uyTJDsx9ruOXuzGRkaeHObWVY/NlKLWt8QXrL3fIEo4=;
	b=SyyJ72UO+drZlnqjqDFtjxPICYXm0oFNm91OAEFEnYkBgfTBg2zpsyAPVWmOJPaZcj
	u0iOtAr/rPY1P0pAiFv+L9pQOG74ibzj/rwJ9I6PaczHn2i8k87J65j+2U7FMUbr49I9
	kUSkubPEy4N1l3RfLFUE/NFO3SZ/AR5cM7MSybiLTqxQbqSWR7Y1oEhWpLZBwola0G+a
	2UCNx8zJoPKx6xRq7m553pw26mkkCbdnpccKzJA/sNmgp4C1XHiRw34HBK7S6je6SFAh
	OzD5shHslpsFaWYszS84WPw4xJ/ChlNMkcd6MTUb689QtuoOZ0o+cNhj4u++hwv9UllH
	KEug==
X-Gm-Message-State: ALyK8tJ8IUzr2GwmFvp4/s+uMTQDeT67FSYwGcKx96ZgOk88j9NnFtCzhPG9RA/pAfDViA==
X-Received: by 10.55.91.70 with SMTP id p67mr51905196qkb.67.1468995080260;
	Tue, 19 Jul 2016 23:11:20 -0700 (PDT)
Received: from ix.home (pool-100-38-105-55.nycmny.fios.verizon.net.
	[100.38.105.55])
	by smtp.gmail.com with ESMTPSA id a82sm637348qkj.0.2016.07.19.23.11.19
	for <coreutils@gnu.org>
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Tue, 19 Jul 2016 23:11:19 -0700 (PDT)
From: Assaf Gordon <assafgordon@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: multibyte processing - handling invalid sequences (long)
Message-Id: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
Date: Wed, 20 Jul 2016 02:11:18 -0400
To: Coreutils <coreutils@gnu.org>
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c09::242
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Jul 2016 06:11:26 -0000

Hello all,

I'd like to discuss few aspect of multibyte processing in coreutils, as =
a preparation for future improvements.

To start with an "easy" topic: how to handle invalid input (i.e. input =
octets that result in invalid multibyte sequence).
Previous discussion said no internal conversion to wchar_t so that =
invalid sequences can be handled as C locale ( =
https://lists.gnu.org/archive/html/coreutils/2010-09/msg00051.html ).
P=C3=A1draig's i18n plan left the handling issue open ("How do we handle =
invalid encodings; substitution, elision, leaving in place?",  =
http://www.pixelbeat.org/docs/coreutils_i18n/).

Is there an agreement on how to handle those?

Do we want to fall-back to C locale, and does that imply going back and =
revising invalid octets and re-processing them as single-byte characters =
?
If so, the implementation need to keep the N octets (up to MB_CUR_MAX), =
and be able to go back and process them. Alternatively, we can treat =
only the last octet (the offending one that caused the sequence to be =
invalid) as a single-byte character, thus possibly losing data.


One possibility is to have all programs print an informative warning to =
stderr upon the detection of the first invalid multibyte sequence, then =
resort to 'best-effort' (e.g. only the last octet, or something else =
that's easy to implement).
My rational is that for an input file with invalid sequences, there is =
no one correct solution that would satisfy all cases: some users would =
think the obvious correct solution is to output invalid sequences as-is, =
others would think they should be silently ignored (i.e. a program =
should never generate invalid output even on invalid input).
The best we could do is warn them, and document a way to fix invalid =
files (along the lines of 'iconv --byte-subst=3D"<0x%x>"'). Users could =
always fallback to forcing C locale and then all input bytes will be =
processed.



To be more concrete, here are some examples:

The unicode code-point U+2460 is 'CIRCLED DIGIT ONE',
in UTF-8 octal: printf '\342\221\240'
I'll use the invalid sequence '\342\221\300' as input below.

What should be the output in the following cases:

'cut': should it print '\300' or '\342' ?

    printf '\342\221\300' | LC_ALL=3Den_US.UTF-8 cut -c1


'wc': should it print 1 (counting only '\300') or 3 (counting all =
octets) or 0 ?
currently it prints 0 because it doesn't count invalid multibyte =
characters.

    printf '\342\221\300' | LC_ALL=3Den_US.UTF-8 wc -m

similar issue, but perhaps with different logic and rationale, with "wc =
-L".


'expand': should this be expanded to '\300' + 7 spaces + 'A',
or '\342\221\300' + 5 spaces + 'A' ? or something else ?

    printf '\342\221\300\tA\n' | LC_ALL=3Den_US.UTF-8 expand



'fold': should this print: 'aa\342\n\221\300b\n' (treating them as =
single-bytes), or
'aa\300\nb\n' (using only the last octet), or something else?

    printf 'aa\342\221\300b\n' | LC_ALL=3Den_US.UTF-8 fold -w 3


'printf' - deals only with bytes. e.g. the following should be printed =
as-is:

    env printf '%s\n' "$(env printf '\342\221\300')"
    env printf "$(env printf '\342\221\300')"


'fmt' and 'pr': I assume they should print the invalid sequence as is, =
as they do not break mid-words.

'head', 'tail', 'split' - not relevant as they deal with bytes, not =
characters.

'csplit': only indirectly relevant, as I seem to remember that standard =
regex should never
match an invalid multibyte sequence?

'shuf','paste' - not relevant as it deals with complete lines.

'yes' - prints input as-is, e.g. the following works:

    yes "$(env printf '\342\221\300')"

'test' - operators '-n' and '-z' work correctly with invalid sequences.

'expr': regex operations should never match (IIUC).
for 'substr', should this return '\300' or '\342' ?

    LC_ALL=3Den_US.UTF-8 expr substr "$(printf '\342\221\300')" 1 1

for 'length', should this return 3 (treating as 3 single-bytes) or 1 =
(counting the last offending octet)?

    LC_ALL=3Den_US.UTF-8 expr length "$(printf '\342\221\300')"

for 'index', both STRING and CHAR might be invalid. Should an invalid =
CHAR parameter be rejected outright ?

'numfmt' - as long as it doesn't get confused with a digit character, =
invalid sequences should be printed 'as-is'.

'seq' - doesn't take any input.

'date' - should print invalid characters in format string as-is.


For now I'm going to side-step sort+join+uniq, as I think they present a =
more complicated set of issues when it comes to multibyte processing.


comments very welcomed,
 - assaf




From MAILER-DAEMON Wed Jul 20 08:21:15 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bPqV1-0008LU-GE
	for mharc-coreutils@gnu.org; Wed, 20 Jul 2016 08:21:15 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:47545)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bPqUy-0008LK-Vf
	for coreutils@gnu.org; Wed, 20 Jul 2016 08:21:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bPqUu-0005RP-JJ
	for coreutils@gnu.org; Wed, 20 Jul 2016 08:21:11 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:59800)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bPqUu-0005Qj-9Y
	for coreutils@gnu.org; Wed, 20 Jul 2016 08:21:08 -0400
Received: from [192.168.1.80] (unknown [109.79.204.233])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 15065243;
	Wed, 20 Jul 2016 13:21:02 +0100 (IST)
Subject: Re: multibyte processing - handling invalid sequences (long)
To: Assaf Gordon <assafgordon@gmail.com>, Coreutils <coreutils@gnu.org>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <578F6CAD.8080205@draigBrady.com>
Date: Wed, 20 Jul 2016 13:21:01 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Jul 2016 12:21:14 -0000

On 20/07/16 07:11, Assaf Gordon wrote:
> Hello all,
>=20
> I'd like to discuss few aspect of multibyte processing in coreutils, as=
 a preparation for future improvements.
>=20
> To start with an "easy" topic: how to handle invalid input (i.e. input =
octets that result in invalid multibyte sequence).
> Previous discussion said no internal conversion to wchar_t so that inva=
lid sequences can be handled as C locale ( https://lists.gnu.org/archive/=
html/coreutils/2010-09/msg00051.html ).
> P=C3=A1draig's i18n plan left the handling issue open ("How do we handl=
e invalid encodings; substitution, elision, leaving in place?",  http://w=
ww.pixelbeat.org/docs/coreutils_i18n/).
>=20
> Is there an agreement on how to handle those?
>=20
> Do we want to fall-back to C locale, and does that imply going back and=
 revising invalid octets and re-processing them as single-byte characters=
 ?
> If so, the implementation need to keep the N octets (up to MB_CUR_MAX),=
 and be able to go back and process them. Alternatively, we can treat onl=
y the last octet (the offending one that caused the sequence to be invali=
d) as a single-byte character, thus possibly losing data.
>=20
>=20
> One possibility is to have all programs print an informative warning to=
 stderr upon the detection of the first invalid multibyte sequence, then =
resort to 'best-effort' (e.g. only the last octet, or something else that=
's easy to implement).
> My rational is that for an input file with invalid sequences, there is =
no one correct solution that would satisfy all cases: some users would th=
ink the obvious correct solution is to output invalid sequences as-is, ot=
hers would think they should be silently ignored (i.e. a program should n=
ever generate invalid output even on invalid input).
> The best we could do is warn them, and document a way to fix invalid fi=
les (along the lines of 'iconv --byte-subst=3D"<0x%x>"'). Users could alw=
ays fallback to forcing C locale and then all input bytes will be process=
ed.
>=20
>=20
>=20
> To be more concrete, here are some examples:
>=20
> The unicode code-point U+2460 is 'CIRCLED DIGIT ONE',
> in UTF-8 octal: printf '\342\221\240'
> I'll use the invalid sequence '\342\221\300' as input below.
>=20
> What should be the output in the following cases:
>=20
> 'cut': should it print '\300' or '\342' ?
>=20
>     printf '\342\221\300' | LC_ALL=3Den_US.UTF-8 cut -c1
>=20
>=20
> 'wc': should it print 1 (counting only '\300') or 3 (counting all octet=
s) or 0 ?
> currently it prints 0 because it doesn't count invalid multibyte charac=
ters.
>=20
>     printf '\342\221\300' | LC_ALL=3Den_US.UTF-8 wc -m
>=20
> similar issue, but perhaps with different logic and rationale, with "wc=
 -L".
>=20
>=20
> 'expand': should this be expanded to '\300' + 7 spaces + 'A',
> or '\342\221\300' + 5 spaces + 'A' ? or something else ?
>=20
>     printf '\342\221\300\tA\n' | LC_ALL=3Den_US.UTF-8 expand
>=20
>=20
>=20
> 'fold': should this print: 'aa\342\n\221\300b\n' (treating them as sing=
le-bytes), or
> 'aa\300\nb\n' (using only the last octet), or something else?
>=20
>     printf 'aa\342\221\300b\n' | LC_ALL=3Den_US.UTF-8 fold -w 3
>=20
>=20
> 'printf' - deals only with bytes. e.g. the following should be printed =
as-is:
>=20
>     env printf '%s\n' "$(env printf '\342\221\300')"
>     env printf "$(env printf '\342\221\300')"
>=20
>=20
> 'fmt' and 'pr': I assume they should print the invalid sequence as is, =
as they do not break mid-words.
>=20
> 'head', 'tail', 'split' - not relevant as they deal with bytes, not cha=
racters.
>=20
> 'csplit': only indirectly relevant, as I seem to remember that standard=
 regex should never
> match an invalid multibyte sequence?
>=20
> 'shuf','paste' - not relevant as it deals with complete lines.
>=20
> 'yes' - prints input as-is, e.g. the following works:
>=20
>     yes "$(env printf '\342\221\300')"
>=20
> 'test' - operators '-n' and '-z' work correctly with invalid sequences.
>=20
> 'expr': regex operations should never match (IIUC).
> for 'substr', should this return '\300' or '\342' ?
>=20
>     LC_ALL=3Den_US.UTF-8 expr substr "$(printf '\342\221\300')" 1 1
>=20
> for 'length', should this return 3 (treating as 3 single-bytes) or 1 (c=
ounting the last offending octet)?
>=20
>     LC_ALL=3Den_US.UTF-8 expr length "$(printf '\342\221\300')"
>=20
> for 'index', both STRING and CHAR might be invalid. Should an invalid C=
HAR parameter be rejected outright ?
>=20
> 'numfmt' - as long as it doesn't get confused with a digit character, i=
nvalid sequences should be printed 'as-is'.
>=20
> 'seq' - doesn't take any input.
>=20
> 'date' - should print invalid characters in format string as-is.
>=20
>=20
> For now I'm going to side-step sort+join+uniq, as I think they present =
a more complicated set of issues when it comes to multibyte processing.
>=20
>=20
> comments very welcomed,

It's worth considering having a separate (already existing?) util
to fix data before processing. That could have options to:
  drop invalid chars, replace with replacement char,
  apply various http://unicode.org/reports/tr15/#Norm_Forms,
  convert enclosed forms like =E3=8A=B7 to 42 etc.
I.E. we should avoid complicating each util where possible,
and at least avoid having options on each util that could be
hoisted to a more general util like above.

Silently dropping invalid characters probably isn't a great idea,
and warnings to stderr is a bit messy and could be seen to contradict
POSIX which suggests exiting with failure if anything output to stderr.
A compromise might be to just replace invalid chars with
the replacement character =EF=BF=BD and then include that in
normal character processing, to make issues in input apparent.

cheers,
P=C3=A1draig


From MAILER-DAEMON Wed Jul 20 09:04:59 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bPrBL-0006Xt-HI
	for mharc-coreutils@gnu.org; Wed, 20 Jul 2016 09:04:59 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:59266)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bPrBJ-0006Rs-3o
	for coreutils@gnu.org; Wed, 20 Jul 2016 09:04:58 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bPrBE-0006hm-OV
	for coreutils@gnu.org; Wed, 20 Jul 2016 09:04:55 -0400
Received: from mx1.redhat.com ([209.132.183.28]:60220)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bPrBE-0006hh-GB
	for coreutils@gnu.org; Wed, 20 Jul 2016 09:04:52 -0400
Received: from int-mx14.intmail.prod.int.phx2.redhat.com
	(int-mx14.intmail.prod.int.phx2.redhat.com [10.5.11.27])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.redhat.com (Postfix) with ESMTPS id E81837F0BC;
	Wed, 20 Jul 2016 13:04:51 +0000 (UTC)
Received: from [10.3.116.181] (ovpn-116-181.phx2.redhat.com [10.3.116.181])
	by int-mx14.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id u6KD4pBD032092; Wed, 20 Jul 2016 09:04:51 -0400
Subject: Re: multibyte processing - handling invalid sequences (long)
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>,
	Assaf Gordon <assafgordon@gmail.com>, Coreutils <coreutils@gnu.org>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com>
From: Eric Blake <eblake@redhat.com>
Openpgp: url=http://people.redhat.com/eblake/eblake.gpg
Organization: Red Hat, Inc.
Message-ID: <578F76F3.1070206@redhat.com>
Date: Wed, 20 Jul 2016 07:04:51 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <578F6CAD.8080205@draigBrady.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="svaj8rDgd57nQmsoQim6FqMp8HRAUtjm5"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.27
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(mx1.redhat.com [10.5.110.26]);
	Wed, 20 Jul 2016 13:04:52 +0000 (UTC)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 209.132.183.28
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Jul 2016 13:04:58 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--svaj8rDgd57nQmsoQim6FqMp8HRAUtjm5
From: Eric Blake <eblake@redhat.com>
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>,
 Assaf Gordon <assafgordon@gmail.com>, Coreutils <coreutils@gnu.org>
Message-ID: <578F76F3.1070206@redhat.com>
Subject: Re: multibyte processing - handling invalid sequences (long)
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
 <578F6CAD.8080205@draigBrady.com>
In-Reply-To: <578F6CAD.8080205@draigBrady.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 07/20/2016 06:21 AM, P=C3=A1draig Brady wrote:

> It's worth considering having a separate (already existing?) util
> to fix data before processing. That could have options to:
>   drop invalid chars, replace with replacement char,
>   apply various http://unicode.org/reports/tr15/#Norm_Forms,
>   convert enclosed forms like =E3=8A=B7 to 42 etc.
> I.E. we should avoid complicating each util where possible,
> and at least avoid having options on each util that could be
> hoisted to a more general util like above.
>=20
> Silently dropping invalid characters probably isn't a great idea,
> and warnings to stderr is a bit messy and could be seen to contradict
> POSIX which suggests exiting with failure if anything output to stderr.=

> A compromise might be to just replace invalid chars with
> the replacement character =EF=BF=BD and then include that in
> normal character processing, to make issues in input apparent.

Since there are several plausible error-handling methods (silently
discard invalid input, flag input as invalid with an error and no
further output, convert invalid input into replacement character and
proceed with output), all of which can be considered desirable in some
circumstances, I wonder if we should give ALL utilities a common
--encoding-error=3DPOLICY option that allows runtime selection between th=
e
three policies, and/or an environment variable that selects the default
policy in absence of a command line choice.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--svaj8rDgd57nQmsoQim6FqMp8HRAUtjm5
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEbBAEBCAAGBQJXj3bzAAoJEKeha0olJ0NqDWkH+JYLEe9A4VO+l52f4p+WpAzc
4N97pqGn6ZK41iVNfOZmCkApDvF6MyrdVkhP6ka01g+MUQ1zjyPQ+eQAmyPuIbON
GdMRnsxRZVBmh7Kn+4Y3svwe0YDs3YMWDVIPaz1r/LkpxB1QylO0/3DwQx9/yjmN
fWDzgdwthUh9m5oC8ZciRuvaeQMUQ7hUIh2DNlVjyghUe1AYLnSdxai/jssFqV57
KIu76W6VLB5PY1gROmAUS903HI5yMQsfvHVdFllyRq8fvkpIyH/mIeaxKyQRuNJW
5GKxIT/xOSygkj/aO/SzkRTK1Dcaol7kWB6hk/e821NpB4JhWURgrs9Taqhnow==
=U0PY
-----END PGP SIGNATURE-----

--svaj8rDgd57nQmsoQim6FqMp8HRAUtjm5--


From MAILER-DAEMON Thu Jul 21 02:04:23 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bQ75r-0007Bz-7S
	for mharc-coreutils@gnu.org; Thu, 21 Jul 2016 02:04:23 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:37883)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bQ75o-0007Bt-NB
	for coreutils@gnu.org; Thu, 21 Jul 2016 02:04:22 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bQ75j-0004H8-QU
	for coreutils@gnu.org; Thu, 21 Jul 2016 02:04:19 -0400
Received: from mail-qt0-x231.google.com ([2607:f8b0:400d:c0d::231]:33551)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bQ75j-0004Gx-Lf
	for coreutils@gnu.org; Thu, 21 Jul 2016 02:04:15 -0400
Received: by mail-qt0-x231.google.com with SMTP id w38so38622190qtb.0
	for <coreutils@gnu.org>; Wed, 20 Jul 2016 23:04:15 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=yXoYSXzqdn2FHMPdQ4KYLCgiWZdO0Yyt1TfJ3uBj8Pw=;
	b=ySvjG528UIykLLYQ33nZdlG5viEik2N2EP3PqRnDz2wONGoM6811O6y7Tgx2vbint9
	ekLNoNQGAuS5XnnxiS2h1tWdSQ2nunfhl0vsMEyNhiN/BsTU0035ALR9IPgXGxz1taKk
	SJykm7Q0PxSbGaba3vNsiWWtonJHaKPug67j6OktbCB1zG3CSg1PEXmSIfLyuiQrkzFm
	djqZxUToMkYSbrzwhR7LJuLMke6RrvckOoOr330hTwsmp5Tl3Hs5vUgicP+q+8R5r/rM
	xg0r1ulxkAV7nBYV39yTD7v+63bceJdX4sQJl0DSCUMn5/drJbPRR225q3Q9jX9WGoHm
	ZqPw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=yXoYSXzqdn2FHMPdQ4KYLCgiWZdO0Yyt1TfJ3uBj8Pw=;
	b=kuCq+OXDNOrL53hGoJdv272SBIJnwn7HojpYXFCa4d1m5MHdyF3subw4GCr+fuL7A1
	FKbxLwtt86z34OUhertt6paaruWUcSmeg5NY0ars6JlpAAJb+grMucKvp9Ai9RNJJEPk
	PCOuHLUkObjds2NMgRowoudj5V+p3XUJoUALkV6hFKd8hqIdIueCM+KjaDKB8YZ/ST1y
	J9rQxkU4KOE5AlkQwDP4MxNFBRqcWaVeewJghHDnmmdfkrwY2Gu62o+gdm15WJAGmxPZ
	LLlIrsVFfxND+YAkvtpSIA0Y0/Ai4qlyY/kemftA1jlYHcWuaaTP0Hwg5YA6Hgz3h6VZ
	T5rQ==
X-Gm-Message-State: ALyK8tKjJ290MocLGulpGDm07wqGFTzacpIQCzTLKGAyR5T8zgbhZxB/4+s8Mh7teu8EbQ==
X-Received: by 10.237.45.135 with SMTP id i7mr80030546qtd.30.1469081054781;
	Wed, 20 Jul 2016 23:04:14 -0700 (PDT)
Received: from ix.home (pool-100-38-105-55.nycmny.fios.verizon.net.
	[100.38.105.55]) by smtp.gmail.com with ESMTPSA id
	f94sm3484454qtb.13.2016.07.20.23.04.13
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Wed, 20 Jul 2016 23:04:14 -0700 (PDT)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: multibyte processing - handling invalid sequences (long)
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <578F76F3.1070206@redhat.com>
Date: Thu, 21 Jul 2016 02:04:13 -0400
Cc: Coreutils <coreutils@gnu.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
To: Eric Blake <eblake@redhat.com>,
	=?utf-8?Q?P=C3=A1draig_Brady?= <P@draigBrady.com>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c0d::231
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Jul 2016 06:04:22 -0000

Hello,

> On Jul 20, 2016, at 09:04, Eric Blake <eblake@redhat.com> wrote:
>=20
> On 07/20/2016 06:21 AM, P=C3=A1draig Brady wrote:
>=20
>> It's worth considering having a separate (already existing?) util
>> to fix data before processing.
[...]
>=20
> [...] I wonder if we should give ALL utilities a common
> --encoding-error=3DPOLICY option that allows runtime selection between =
the
> three policies, and/or an environment variable that selects the =
default
> policy in absence of a command line choice.

I like the idea of a dedicated program - does one thing and hopefully =
does it well...
Even if we end up implementing the policy logic in all the common =
utilities, it's a good start to discuss the policy itself.

Attached is a suggestion for such a program ('mbfix' is just a temporary =
name).
Usage examples below, and of course it's just a baseline for further =
discussion.
The program does not perform any conversions between encoding: it only =
checks for valid sequences in the current locale.

I'm aware of four existing utilities that have overlapping =
functionality:

1.
gnu glibc's iconv: seems like the default iconv that comes with the =
default glibc does not support the --byte-subst parameter, so can't be =
easily used to replace invalid sequences.

2.
gnu's libiconv: iconv from libiconv does support "--byte-subst" and "-c" =
options. However, it seems this package is not readily available on many =
gnu/linux distributions, and anecdotally proved tricky to install from =
git or latest tarball.

3.
ICU's uconv (http://site.icu-project.org): uconv supports several =
methods to handle invalid data (called 'callbacks' in their man page). =
Examples:
   $ printf 'ab\342cdef' | uconv
   Conversion to Unicode from codepage failed at input byte position 2. =
Bytes: e2 Error: Illegal character found
   $ printf 'ab\342cdef' | uconv --callback substitute
   ab=EF=BF=BDcdef
   $ printf 'ab\342cdef' | uconv --callback escape-c
   ab\xE2cdef

However, the ICU package is large, requires C++, and brings in lots of =
dependancies. This might not be desirable in all environments.

4.
'recode' - from a cursory look it supports silently ignoring invalid =
character, but I found no way to transform invalid input. I'm also not =
sure if this program is still actively maintained (comments are welcomed =
about it).


=3D=3D=3D

As such, it might be worth considering adding a dedicated program.
Below are usage examples of the attached program.

Feedback and suggestions are very welcomed,
 - assaf





Valid input is printed as-is:

  $ printf 'ab\u2461cdef' | ./src/mbfix=20
  ab=E2=91=A1cdef


'--check' option allows scripting:

  $ printf 'ab\u2461cdef' | ./src/mbfix --check && echo ok
  ok

  $ printf 'ab\342cdef' | ./src/mbfix --check && echo ok || echo fail
  ./src/mbfix: '(stdin)': line 1 char 2 (byte 2): found invalid =
multibyte sequence, octet 0xe2 / 0342
  fail


encoding policies: 'discard'

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Ddiscard
  abcdef

'discard' with verbose printing:

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Ddiscard --verbose > a
  ./src/mbfix: '(stdin)': line 1 char 2 (byte 2): found invalid =
multibyte sequence, octet 0xe2 / 0342

  $ cat a
  abcdef


'abort' policy stops at first invalid sequence:

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Dabort
  ab./src/mbfix: '(stdin)': line 1 char 2 (byte 2): found invalid =
multibyte sequence, octet 0xe2 / 0342


'replace' policy uses a fixed replacement character:

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Dreplace
  ab=EF=BF=BDcdef

'replace' can use a custom unicode character:

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Dreplace =
--replace-char=3D0x2665
  ab=E2=99=A5cdef

'recode' uses printf to output the invalid octet:

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Drecode
  ab<0xe2>cdef

'recode' with custom format:

  $ printf 'ab\342cdef' | ./src/mbfix --policy=3Drecode =
--recode-format=3D"<INVALID=3D0x%03o>"
  ab<INVALID=3D0x342>cdef


$ ./src/mbfix --help
Usage: ./src/mbfix [OPTION]... [FILE]...
Fix and adjust multibyte character in files

Mandatory arguments to long options are mandatory for short options too.
  -c, --check          validate input, no output
  -p, --policy=3DPOLICY  invalid-input policy: discard, abort
                       replace (default), recode
      --replace-char=3DN
                       with 'replace' policy, use unicode character N
                       (default: 0xFFFD 'REPLACEMENT CHARACTER')
      --recode-format=3DFMT
                       with 'recode' policy, recode invalid octets
                       using FMT printf-format (default: '<0x%02x>')
  -v, --verbose        report location of invalid input
  -z, --zero-terminated    line delimiter is NUL, not newline
      --help     display this help and exit
      --version  output version information and exit

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/mbfix>
or available locally via: info '(coreutils) mbfix invocation'




From MAILER-DAEMON Thu Jul 21 06:09:04 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bQAue-00032s-Iq
	for mharc-coreutils@gnu.org; Thu, 21 Jul 2016 06:09:04 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:60213)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQAub-00032l-Ru
	for coreutils@gnu.org; Thu, 21 Jul 2016 06:09:03 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQAuW-000605-QN
	for coreutils@gnu.org; Thu, 21 Jul 2016 06:09:01 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:34770)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQAuW-0005zm-H8
	for coreutils@gnu.org; Thu, 21 Jul 2016 06:08:56 -0400
Received: from [192.168.1.80] (unknown [109.77.141.179])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 0FAC394B8;
	Thu, 21 Jul 2016 11:08:55 +0100 (IST)
Subject: Re: multibyte processing - handling invalid sequences (long)
To: Assaf Gordon <assafgordon@gmail.com>, Eric Blake <eblake@redhat.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
Cc: Coreutils <coreutils@gnu.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <57909F2C.7060403@draigBrady.com>
Date: Thu, 21 Jul 2016 11:08:44 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Jul 2016 10:09:03 -0000

On 21/07/16 07:04, Assaf Gordon wrote:
> Hello,
>=20
>> On Jul 20, 2016, at 09:04, Eric Blake <eblake@redhat.com> wrote:
>>
>> On 07/20/2016 06:21 AM, P=C3=A1draig Brady wrote:
>>
>>> It's worth considering having a separate (already existing?) util
>>> to fix data before processing.
> [...]
>>
>> [...] I wonder if we should give ALL utilities a common
>> --encoding-error=3DPOLICY option that allows runtime selection between=
 the
>> three policies, and/or an environment variable that selects the defaul=
t
>> policy in absence of a command line choice.
>=20
> I like the idea of a dedicated program - does one thing and hopefully d=
oes it well...
> Even if we end up implementing the policy logic in all the common utili=
ties, it's a good start to discuss the policy itself.
>=20
> Attached is a suggestion for such a program ('mbfix' is just a temporar=
y name).
> Usage examples below, and of course it's just a baseline for further di=
scussion.
> The program does not perform any conversions between encoding: it only =
checks for valid sequences in the current locale.
>=20
> I'm aware of four existing utilities that have overlapping functionalit=
y:
>=20
> 1.
> gnu glibc's iconv: seems like the default iconv that comes with the def=
ault glibc does not support the --byte-subst parameter, so can't be easil=
y used to replace invalid sequences.
>=20
> 2.
> gnu's libiconv: iconv from libiconv does support "--byte-subst" and "-c=
" options. However, it seems this package is not readily available on man=
y gnu/linux distributions, and anecdotally proved tricky to install from =
git or latest tarball.
>=20
> 3.
> ICU's uconv (http://site.icu-project.org): uconv supports several metho=
ds to handle invalid data (called 'callbacks' in their man page). Example=
s:
>    $ printf 'ab\342cdef' | uconv
>    Conversion to Unicode from codepage failed at input byte position 2.=
 Bytes: e2 Error: Illegal character found
>    $ printf 'ab\342cdef' | uconv --callback substitute
>    ab=EF=BF=BDcdef
>    $ printf 'ab\342cdef' | uconv --callback escape-c
>    ab\xE2cdef
>=20
> However, the ICU package is large, requires C++, and brings in lots of =
dependancies. This might not be desirable in all environments.
>=20
> 4.
> 'recode' - from a cursory look it supports silently ignoring invalid ch=
aracter, but I found no way to transform invalid input. I'm also not sure=
 if this program is still actively maintained (comments are welcomed abou=
t it).
>=20
>=20
> =3D=3D=3D
>=20
> As such, it might be worth considering adding a dedicated program.
> Below are usage examples of the attached program.
>=20
> Feedback and suggestions are very welcomed,
>  - assaf
>=20
>=20
>=20
>=20
>=20
> Valid input is printed as-is:
>=20
>   $ printf 'ab\u2461cdef' | ./src/mbfix=20
>   ab=E2=91=A1cdef
>=20
>=20
> '--check' option allows scripting:
>=20
>   $ printf 'ab\u2461cdef' | ./src/mbfix --check && echo ok
>   ok
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --check && echo ok || echo fail
>   ./src/mbfix: '(stdin)': line 1 char 2 (byte 2): found invalid multiby=
te sequence, octet 0xe2 / 0342
>   fail
>=20
>=20
> encoding policies: 'discard'
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Ddiscard
>   abcdef
>=20
> 'discard' with verbose printing:
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Ddiscard --verbose > a
>   ./src/mbfix: '(stdin)': line 1 char 2 (byte 2): found invalid multiby=
te sequence, octet 0xe2 / 0342
>=20
>   $ cat a
>   abcdef
>=20
>=20
> 'abort' policy stops at first invalid sequence:
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Dabort
>   ab./src/mbfix: '(stdin)': line 1 char 2 (byte 2): found invalid multi=
byte sequence, octet 0xe2 / 0342
>=20
>=20
> 'replace' policy uses a fixed replacement character:
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Dreplace
>   ab=EF=BF=BDcdef
>=20
> 'replace' can use a custom unicode character:
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Dreplace --replace-char=
=3D0x2665
>   ab=E2=99=A5cdef
>=20
> 'recode' uses printf to output the invalid octet:
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Drecode
>   ab<0xe2>cdef
>=20
> 'recode' with custom format:
>=20
>   $ printf 'ab\342cdef' | ./src/mbfix --policy=3Drecode --recode-format=
=3D"<INVALID=3D0x%03o>"
>   ab<INVALID=3D0x342>cdef
>=20
>=20
> $ ./src/mbfix --help
> Usage: ./src/mbfix [OPTION]... [FILE]...
> Fix and adjust multibyte character in files
>=20
> Mandatory arguments to long options are mandatory for short options too=
.
>   -c, --check          validate input, no output
>   -p, --policy=3DPOLICY  invalid-input policy: discard, abort
>                        replace (default), recode
>       --replace-char=3DN
>                        with 'replace' policy, use unicode character N
>                        (default: 0xFFFD 'REPLACEMENT CHARACTER')
>       --recode-format=3DFMT
>                        with 'recode' policy, recode invalid octets
>                        using FMT printf-format (default: '<0x%02x>')
>   -v, --verbose        report location of invalid input
>   -z, --zero-terminated    line delimiter is NUL, not newline
>       --help     display this help and exit
>       --version  output version information and exit

It seems like --normalization=3D{NFKD,NFKD,NFC,NFD} functionality would
also be quite cohesive in such a util.

A related thread:
http://lists.gnu.org/archive/html/bug-coreutils/2009-02/threads.html#0022=
4

thanks,
P=C3=A1draig


From MAILER-DAEMON Thu Jul 21 23:23:33 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bQR3k-0007KO-Qj
	for mharc-coreutils@gnu.org; Thu, 21 Jul 2016 23:23:32 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:37789)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bQR3i-0007H1-Hf
	for coreutils@gnu.org; Thu, 21 Jul 2016 23:23:31 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bQR3d-0003y0-Fs
	for coreutils@gnu.org; Thu, 21 Jul 2016 23:23:29 -0400
Received: from mail-qk0-x234.google.com ([2607:f8b0:400d:c09::234]:34635)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bQR3d-0003xL-9V
	for coreutils@gnu.org; Thu, 21 Jul 2016 23:23:25 -0400
Received: by mail-qk0-x234.google.com with SMTP id o67so91306707qke.1
	for <coreutils@gnu.org>; Thu, 21 Jul 2016 20:23:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc:message-id:references
	:to; bh=Rpt06Ex54DDvXRIB8zWcb0PBDHmHjcF10h+g0mWtQdY=;
	b=AA7tW/ZVC0FErBGOYujYN9StxxiM1+4FnxA2Nkqhnec5kibCOYWahGsX/l2U7xm9Mh
	RJNGtvh8CX3Kb740PTZ0J55mI4/jUhY7BchB8dmWggjlZQPjlyFeO5QLFpixIjIIP0s5
	wDzVlRCV4XqgBYHEDvdDP9RwwgwMJmIBuoowOoDp88MLE+ECr8H7AQGY1nkQChY46js+
	hzHMKpymbrRW/0RpRwY4l7GVTazLivUyNj+W7HPRCeymCxMRsG/QDl7MLgp2lH0eIbtP
	W0oVldwkLBMNiGRBuAOnU2DWyetcrkGPsvFd+njsOkESxfjcTMB0Y4srgcPdrYSy4uUl
	EZ6w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:message-id:references:to;
	bh=Rpt06Ex54DDvXRIB8zWcb0PBDHmHjcF10h+g0mWtQdY=;
	b=GJxwFmXVu+LzhYRKheRV8UuzaYpX5G4yfju5kGnE/zVGRde3oCrSOcE/+71+5M7kyu
	KFt96Q3sKB7piR9OcDdwf/1iVYsWkMb0cv4d/yoBGBJ7ZcBbmoiYpwX8fjAh5AoyxMsT
	vklBPppBnxfpGsHLTLd1yCGF8VOiR5qCj76SlKDzhRBXLMUExvqxLP9a/eR4eLB61PQ9
	/4oEPExH7zKtHV7wWelLOP8bXKm4K0iwDS5myiBU9ljqgTSG18D/gPI3u++40X1Q5TF2
	owttz8G3+/quJboNUQG6uMQlTYhlDPDp/DdjYP1YIZSIGxp1SicZzUwNWLvftRMxpuAo
	KNCw==
X-Gm-Message-State: AEkooutdzoiJvReMrBZjZ0nU5E/opDm6Q4rvKVS3yyV80ttsPQRg9EAiwzTYjdYbnq+RaA==
X-Received: by 10.55.161.73 with SMTP id k70mr1965199qke.189.1469157804906;
	Thu, 21 Jul 2016 20:23:24 -0700 (PDT)
Received: from ix.home (pool-100-38-105-55.nycmny.fios.verizon.net.
	[100.38.105.55]) by smtp.gmail.com with ESMTPSA id
	o70sm6188475qka.29.2016.07.21.20.23.23
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Thu, 21 Jul 2016 20:23:23 -0700 (PDT)
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_FD7D67CE-9F80-4BA6-956C-9CA5413DED19"
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: multibyte processing - handling invalid sequences (long)
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <57909F2C.7060403@draigBrady.com>
Date: Thu, 21 Jul 2016 23:23:22 -0400
Cc: Eric Blake <eblake@redhat.com>,
 Coreutils <coreutils@gnu.org>
Message-Id: <654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
To: =?utf-8?Q?P=C3=A1draig_Brady?= <P@draigBrady.com>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c09::234
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Jul 2016 03:23:32 -0000


--Apple-Mail=_FD7D67CE-9F80-4BA6-956C-9CA5413DED19
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hello,

> On Jul 21, 2016, at 06:08, P=C3=A1draig Brady <P@draigBrady.com> =
wrote:
> [...]
> It seems like --normalization=3D{NFKD,NFKD,NFC,NFD} functionality =
would
> also be quite cohesive in such a util.

Attached an improved version with unicode normalization support.

Before continuing with other stuff (e.g. more tests, documentation, =
news, etc.),
it's worth discussing if this is the path to take (or if we want to add =
this to each individual utility).
Also, do we keep these options or modify them?
e.g. 'uconv' uses different terminology for handling invalid sequences: =
stop, skip, substitute, escape (corresponding to abort, discard, =
replace, recode below).

To keep the implementation simple, unicode normalization requires UTF-8 =
locales - is this a valid requirement?

And of course, what about the name?

Comments welcomed,
 - assaf




Example (from 'Unicode Explained' book):
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
$ printf '\uFB01anc\u00E9\n'
=EF=AC=81anc=C3=A9

$ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfd | od -An -tx1c
  ef  ac  81  61  6e  63  65  cc  81  0a
   ?   ? 201   a   n   c   e   ? 201  \n

$ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfc | od -An -tx1c
  ef  ac  81  61  6e  63  c3  a9  0a
   ?   ? 201   a   n   c   ?   ?  \n

$ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfkd | od -An -tx1c
  66  69  61  6e  63  65  cc  81  0a
   f   i   a   n   c   e   ? 201  \n

$ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfkc | od -An -tx1c
  66  69  61  6e  63  c3  a9  0a
   f   i   a   n   c   ?   ?  \n

$ ./src/mbfix --help
Usage: ./src/mbfix [OPTION]... [FILE]...
Fix and adjust multibyte character in files

Mandatory arguments to long options are mandatory for short options too.
  -A, --abort          same as --policy=3Dabort
  -C, --recode         same as --policy=3Drecode
  -c, --check          validate input, no output
  -D, --discard        same as --policy=3Ddiscard
  -n, --normalization=3DNORM
                       apply unicode normalization NORM:, one of:
                       nfd, nfc, nfkd, nfkc. Normalization requires
                       UTF-8 locales.
  -p, --policy=3DPOLICY  invalid-input policy: discard, abort
                       replace (default), recode
  -R, --replace        same as --policy=3Dreplace
      --replace-char=3DN
                       with 'replace' policy, use unicode character N
                       (default: 0xFFFD 'REPLACEMENT CHARACTER')
      --recode-format=3DFMT
                       with 'recode' policy, recode invalid octets
                       using FMT printf-format (default: '<0x%02x>')
  -v, --verbose        report location of invalid input
  -z, --zero-terminated    line delimiter is NUL, not newline
      --help     display this help and exit
      --version  output version information and exit

GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/mbfix>
or available locally via: info '(coreutils) mbfix invocation'
=3D=3D=3D=3D



--Apple-Mail=_FD7D67CE-9F80-4BA6-956C-9CA5413DED19
Content-Disposition: attachment;
	filename=0001-mbfix-a-new-program-to-fix-invalid-multibyte-files.patch.xz
Content-Type: application/octet-stream;
	name="0001-mbfix-a-new-program-to-fix-invalid-multibyte-files.patch.xz"
Content-Transfer-Encoding: base64

/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4IPwJlFdACMcieb23yib8AcGkz2cFdzyhoBeMBbtoW07
PFRvCDMIq8dNJQoJBli5o2A9Ee7rUqVoJ8qnFxHi0OQIzky9/Qspjfu6w2bxYEn2N/CichE+8pEP
hff6xYh4Q4IDkast3amA2FT0iR2fSw0C9C/tp5Bei8vJbXR53Ho2noffjuLvLGmOYnxpcnQoT3ne
M2PrHbSN/GohiJF5PeLSjPwlUdFsF1gSTGos/hMu0Jvb/Isa9FtEAmPW1JpMvmk4AgTNGbWES1sw
+Q2GED/ngvCaG5xh+u4pZAmLqq8ja777x7fWEAu+4L8DYL23UxB1DYrxQmApJFNYm/qg2yd9Dxb6
nd8JJYWI+I7DXXtkw1Lbe+OJKyO4sAAExvAIrNjVFFQFhGS7D19CB33qn0eS66QuxCZwMb50Ol2b
Ty18HnnFjRbBcFUhxn6Blkpk2gJIeuCVdOehAderKpfRTEPfmve8xuInCohKy/BjutERrQ69xVIX
YZubJa9hy82dmTu4iC67x/ZXq+BXeIKX1S54uPpuWfR8Rs3ekWS7bH605FDxHL9Fwh3U9TNl9gqa
T0yPG7df9OYuZ7HZnxso22v+3ucqSbjoTbg2OXorRXkjHE1ziTQ6C4+6glyXJse5JVaqXo+M9i8F
gvsCrn4YQHjyLhNk7o8pW9Em2OewbY7ANudy+XP+jkNG/U/OCtJG7jF71fSHvKetzygKndVG1P6T
bD6LLBm0iQA4HvA9cJ4261/TyWXLMYvgHmmB49OZNLVEGj8RlNoKlARnnbpxgrjVkJSiFVJ+LrTb
pD3U2QvfOtDLN2pujgOIieB+wcyCsiaCgFFRcYjwx0Yo2kbEWor3308K3qVhNyckb6MGRi5fj2ub
EUIfhgllcKWFlFVXpruqgJAljvxAEtR+ZAlHfgXM/TmZUFMshRK+/EoRzLyhJZyGrSqSQY6vU7wk
VCTXt9MLvpJNob+7OD3vUuHSd6i9s+egmNgUMsqF1/MyalilqFSrSit9nK6PKgLmxRYzpIMXauv8
FwbQU5frruE/7bsh0QvGF7oGnyAFo2SJghzlivtmskoQhlKFPd0I4sxBsyPZsuXHWxSNW6rkKnCp
DMZo4zwHUrWtKOjEEhlvgQ5ZAENLxD93LlLn0zZUmZvPqRJjhwOkad00F+JtwLyXYVhUGNJAZ8tC
U4/znhtkoOCaLAxfUO8hqpt7SVu1WzdaPfVhPttGE3Xomj++Ek46ZsYmX/tysRx3WBZbEJeBycwA
hG0z3XmGih2lphszX/UT3mM9QG4md6wyWbinfW4/JbzIC1gv04EDXDkidBdupGmlX3uYqC+RMUuj
V7R35XBJ058M98saulEZ6dVjv0uniJp9ARLdSPsRL+5jhY0C/pyVlekKIth7X0xfOcaFWqCVDcM1
1ObGdk5DL1gYv9liUn0nUw2MM998A72BxvM/AS9hCWIHKn3OIeft1tPbfvQEf7UwJBpk8EU2sTuX
1tPGS8prtC4YVyhvzo1OApYG8EtPOseeNHiQggHUrLP4HrPKOZlQUey0mBXEVF5yQzyO9+TqlFcV
lN9+FwZ1XZ36r30ZBsEJg08l/AFVIydscQI1lYFq5eqa1R0zNt/8lIULz319czAz+EUM5cZBbb9n
8VhCVUCTzxs0OBiYoGkrSWJgcdI6/1+hWaXK1yH84xMtjFNtoep5MwRnSgsHIDhOdfLCenYB0l8g
tvUVmd84ESCKhM7g6suCYRb9dAxFfyKHZDcmCoFO7tkMftPDBSSxExZoLses7lAvS+VBC/M1En0R
BBmhYv/nEfR2ChSFHoS2CQAb9bVWrNFIfZZtXser+kYp1TrULGlbC6Nb48eGaxR2lvrsvdG8WhRo
LJ9CSbucnxRJ3t3ZShUt9Y2w4OlO4BkaY/aZOkMk5rr83sdxGw2POOs7OI0Wn9wSyNIy61JUHoZk
7BKJczcKqV3lwMvmvhVHHx1W4ZbDlwf1oUc4nn91yZ+RU78L5n3HW4bFe0CwCO+Sn0+gMZoXVtn/
nfPn2eDqFzy9zK4bgriEubKClxiskZJ79S2w5hXuMJCcAaKB3gv3nLl5YVLkbRygLox4u0JpYmO3
pFZ2USDpFeL0QLteS30RSi2ruX/2NPJ1hsFvZqFFp2JY18SW7PgYZRGJZvoBf0OM4EkhiAh+44hC
6KUEgs+YyBcg6tprB0B3qqqEmL4NYzn4VcB8IGEWd78ddBKPQa7YJzMdKKIZu2KsgGc+MKdY4rgb
t1P8Q6W9rsElNPXJlH97KP/S3cWHozZNEnRB/E1lwVuZi/YjetcLY7Ma9ZlKzRlKHNeR8ks40ZwR
7wKSrJ1dYvsay4xMWUeJEI7yyDtWsa1+h55Z/OpC1rWO6HlIyT5RHxPipe0H8HbpgNEYY7vF3cYv
hiZ1kjKE4kINveTzbM0G4vWhx9WfJlMBK7oX97x6G7cDy5ABW0ElVBaaTnrl1wInJz6rQ9p1SVXZ
a40AqK7bIoLlejoGlFDRfP4S/SU2egwBB4dyQomQfIrWfyxg10ETF3NUWBnP8RKKP7rMb7XfFdpY
rZP0balTz+g+XRFiKjkNGxE+4tmQDQVMvUy3bhLHWqNuQQxNysszLtALBF8shtDGV0saua6Bx+cU
GEAKDHH4W9GVWD3GxDcice+hgCyzdW2hWYxOsgD/+myZLDKmXMoYz+PqTMBb2eFQD58PfPR58CJi
dtEN9Q1LpnKu/yQFGwE8Q/So8BrWMXifYocnK63jSq8PWTjwBFWl7tLwVvYA/u072dznjcTCD04p
PP/N1WSKG57Mgji8x6sdwEc+fIj0wj39BF0VyKuTnnGfB05l2hDZ0igSz+yU9vgZJcpTGhl0jvOH
FcUgjsFKw+XC8JqSCDsa2u5mPTh4hDh0ZayHz52ywXBqJ4FTPGdzJIMwUss76jwRkVl9keFVLJN3
YE4gnbo4hIkB0b7T9UOKHPkvfkhsydCms2HN6NUKwox6cp6D2FxH/pJ9q7XL26OJQ4QPjadOzyBm
VjLrY1sPnJqY+LQ2rvxZ1TMfaIyxcQKs/Qu6OUWBU1gz0JiV/PeZS9SoeQyT0RPctz4NpHWDohRM
dNOqcnRlGv0BR0kMy9TgnLKegueAYfx+JxRkXv4JxUXBldgxlmgbJmUlWWRiO3gjRrBbqQsB2uAy
Tlo418Oyj1wazC+6R68nFtgAQPK9cBjBk6Oh9lPX+psEm4qDvQVdFjbVNO4QgjxLjwSKMV1YiUH1
xBJVZnIt+i1aPgYnJQQpuv65KiixkO3UHvxmR+lkpH7WncJcp6zjjQI75s2cm0UmbwvjsQdz2j8Y
CKAJvdjjS46B7s8SRtnLo/YgORFtpe9aPdTb+WCu2QSMIJvxa3vd8uc2sIiM3lNo5j+q/mpqUpwF
CGDM5IJvIbPwFHsnTjXg/pO2cfzaNWcVQ8SDTK/Ni/DXe1RL+eAGsPfNdhu1ujjBv1W2btEbRoDK
PnxHSGBGK6adi5OZJt7TiCfcAzPJ770uRDuC4qT1KloLy3HKyFfyHa4W56GzJfWoINh++igf1y5p
7Ezycfhbpm1m6uz6pvnI7iCb9Hwnlb2fP6bT6MCamDjwTr27K/yA0wN4VBBG+pXGbXB37VdSq82K
fLRLiUTd5/ruuh+EnVMsPG/sA/fqlS3yQnAh+ai1PrclC2CEaUbPk/cmYJAb0MewdwCP1f40IB5q
cmkcHL0E3aA9RDZLPveC2wkv6yCgyF0II/dkYYwgecNt49VwYYyocxzbKY9892JYnoBlG+Bs4yYx
Az/Lh4uJ/cnzXaFHGbgezGO/XQb5S0TO5qy6FovPcX+KAcFncy0Dl/t83jk11snaxPx1TtEv6f/o
PbHlSKVJfKJzvg+Uo1YvL7zRfxruWXj3WLLdhdL3sazOtSe7CiWg6ZOPZ1r5fJhC2kjjrsYPIr2k
Qtinz+bed4IFJ2+E42pXfJMoKyOr71JLjF1+Yf16J2LN2ihe8kGQYHhbOaeDOvXOa/IcNel9L5iI
hb1q/k8oR8pNTBRWKkyVVXO/e0luRs8UKHcEnGrzDzoRkK8h2R6F8/xebBiY9i5W05OlkN1L8oRK
ZV8eomvBduKAx6SiAFV8E5LDgIrfqvTiAKvLZG2UaufBCkendUL1sXBDwhDQ1jTQ1ZBeI7lvtPWl
55RfhS75+VM5DCr4cyzmbLIeS0fEqvdTkXC5hQK9XypTFBo4WtI/LAk1fHiOixe5yM4yDJbqcDFa
np+C7b8+MjG58rPa7fz+ujg6CIleiEv8lrMiwBTS9QB+4stz3yCQHIfP+2TuQobMGGS7IJ+Hab1X
wVxQOveNnXb6qqS3BwInaa7TBysHc5/7JHRH821l7BHEnyhxlrCvwxBHxFRTJ13cuFYs36zkVpDi
w5MGTzl39TYYMr4Jqkg7QNOKFD5+UzlXs79GAzcI2XIxLyhKX3k1EpsZwOaI1Z5mOsbHQSOtD+Rx
Hj0/zC+If3mWaQQCmod7juBC6GO6RlbuK6Ew+yYsNtJIH9bK8IQVdT4RQU9phzTSYsatu1gbqsjI
z2AyEbLNv1AuB0zfwJMshqUNVbIO+fxtL1spxEdBu4ukYyBq0IjRl7T4d/nzlTcNgjv6KQ55OZJD
W/Kt+Wwr+BE0SrNdKJdoVziEMjLO2c4oYlwNzwDJx0yDXvzgc2ea/S/ESU/latN6Zy97dLARO2QN
dvk+xzY0f/A41TMrbacapmrIZ7yAIMZ3SS5/wdx4F6MsS2i+dBSI49ITPKflGjpvrnKoZATcYwcd
MRZWY+spGnRI6STjgK0IzXLvaYjnh6NMPwzdRTiWaCJ9AqD21rnaJ/Lgu88MkrUs7Jqz/Rq/FSnM
BK5lHdcjOUIHg6CdAyaEQ7fVB+wJLMoCBA5CxnmOQkFbbdQcQhZcrYDxZ6D34FWftGCgwISv2MYB
9soX372UY5P+8b9tceMgun+CTo3vnZkIDKYND0Eov7ci+jvC8ruebnrECCOdFxWRIWf7zMXIhc8b
tW8mh5IQ0IVXvE76BOfQ065H/cCPb7LOPqF7ilEg2cwdmhHI+vPePYeeu+OpJP7djGBdqIWnNgCl
y9KobGybjeXkASzyNaemczRtDCNtIyal2sC65s9Z1PEpng9Z3sW0+7x6smYbna3/ymu6QiGPQDqp
nfVVo4LAck0GsLjzrWJz1ApRQ8/G5cwiZWPwS9+fDCFC8NT2n4RVbhU6Dix8hY22VgjjTMhN2hc8
Dds3Zt/YDobBUKI4/dTIVwpseYj3dgLCkA55U3gphBbvRbwdNvl2uS/WdoQq0dm4b/bs1UY7AGHX
fy9ijmRitg+u/B8HssUlssBjrpjCriw+CrCRZRrdTgQ+n7CuLPvOGaHA/lCqjK8z1Gxxiq46Q4Ir
3dUVJ+571Kov7xy8N648TWPmf6DVV2SXPCqQEJ1p9ps92hjl2PBJzLzvETk4T0R1DFG8eGGjhkzK
Syd0U1vPIhseuCmzqRW6IaLUNCRLh+bPFYEPLFfbw3gz+0TwbXMd7EZva4aEf9WZh6ZZkzuFM6yH
H8ascp1RdQEVskGjZsGrEv4mlRHMxW5oEeHbZ1aGfGd7HLrkXE/xkipFMHjGqDBZZBHjBRTjyG8t
4P7/KLN2z3Gm67STEbgjAlTwYBpsUkkFwid3Go8M3bsu0c4IVAkmf9wO7x8uRXd6JG4u61SCanIH
sGbuCbrkIHoG2B0hRUQ2xjy9zCupKN74LJ+htPhnoalklSFe9bQ5V6dTtM2Au+a+66QWnAgp0gYB
GuNjGV1HAn4MCwujYvQ5E5IJhYFbkzpQCSJZ8AXCtPl+d9wUpQzucSlb4fewqlylHVL6p9EZTnOM
DT+5+LL1eAhSm31RZmgsu+jFs+GnBsr8OKv7IjfETqburzgU6MiAohlGDyMSJRAZz0Z0Bi24r1mD
FcBTnOey9OWL2ZLrS0fT+5ZeyVDHI1z3RpxhWsPzLVhBKhYDE9q/qgQJ76njV5NZ6ef467zuVoIR
yBXmFsEaQguGuL6u/NmEclgVavDAs1PotGByMg8hO7rWjI9q8KaeiWti8dJdrShiWLrC5is2D9tj
ulOW+Y3papL2tJFf7Xp4SwGjOVds2RDcEw0K88Gjsqlp5wkmJ+J+dIHSNobI7dwuQKNY7zGt7E3w
ZKj7OXQfs1u9V9cAUfiY5r6ci1vpNV3x/VmHoAOkbJS4Peje5dK2gyk4+N+sLG7f4rRWQ5wAXsOz
b6CEFQiAQA05hIbvT9lMU5ZuDREquSetZPNBl2bvuQ1qWQutDeXyNQAl8Y6oMJTmfxBBGiV0uPoS
nDGyhik3C8nH+WOVgoSzNBi2ghJONkkysXQPjUJeBPjcWO9yeM9x+eFAvNw4SwbooK2lKbFDCQ5L
v0dA+FT7TwJvG8o9JPUWfZzerWvQG3nZ4R2QsRaT8wcH9n+/GmpSV4F+h+rKyY41QpDvJptt+Pgf
PNGX9rVi3g4OaZvsrqyNipNSxCOO0JJgEbB/edOrQw6NBjfH9rZMY8k8A4c2fz30WscB1WG5PCCZ
O57pLvFsh6zoV4sECV1oF5V2ASz8A0mFlHz0iKB+72emmAlRMqQJWBAwzZXNnwnYspCUyVRTKpOx
EtPZtExLUf2IOEcEhIzHv8TCG64bmzN1UsouVmfPJJupW+hbRN5hWJwRMY3Ya3g2PZhDg2DeMG7o
l0VDf3NzATnz+qBG46lmcdGZM4rNxy3aqo4xSPZ1yXOkx3V2EsavLQpM9HvQc6nFiLvItPTd4w3u
nRl961HdQ0tO0jlpj8mipT0j6FH6gBoIvZBXdT+e3BqZUrs8pkGawnmsXzEZSZTAh3V0U8MWlS1S
w1cu88Sjdu1JurW6khHHo4qjcWY3EY0q3GLzFEgjqrVwgUbXKnkOqtkAv8Ltih//4I1G36FVZ29x
VpJuQ8nz9HbEgIpBunnDyYRfDpLbjIJkBPghXuYGssqNuoxkFdNROytOStfYk6ldOuSau/jNuZIg
R2sONax1PYpVmYZayB4LqaKPUJjm2jgK1s1wEsunWxfr90YfJB8Ht/lFNDX9VGWrtL0cYACkGUII
1LYDQ6WIGl1+G2DtA0K1P5hKxJ1ltuaOiX4XiQIH84xOmZLEJaPqSRqDasTz+qKNj4auFyy3YOrD
4sJ79S10jsg3yH6is/PsKR6y8M+Uks6rqgdLZMTpDwF2gJuusuGo9yw05CIyoH+416yBnYQhRICa
1tG6BMlezkZvqWNGtGTxwePmdxxMnd9RcCqscJIU7LSO0zkrlJ+6v58DfTaVTGqXdNBmZQ1GTljX
buCx41VxcXglUq0jLqUtmccIut5rrCjRw2mnZv+yaOTnooZJGPWDoB3xYBrlFPMelZYQSrKMnFcK
4JzmArGvnlcmK72pStVaNguh3PnNhlaHsCC30WpEjpuvSg5Iz+f7uxQLlJ4hlmCnPNib/ABD1P5r
vFe0wnoSypsf8sEusA4QC613s3uKY2/NHHThwXnD4yszoX5TPlS80XgiVB6Kng1wBxqCuxZkBNq0
9SDTlYdB7a68RCw8lU6mBAjjY6vCL9gfXVlMoTcGbCWSRtiMZ433Vb23KErdGokRC+mhyBjWEs3B
QTWhwMXgUgwwmpAG79eFsIus+0ZZvSyzdvtgkcRd2mCvYXuzOW7jq5TZm/iiF43VLJN1jYh8TXlI
xd1Xio9yZVn2Vzynj3mQBqPPRMxpnMIiujP/vux7h68KTId9wtmWBHfd97qgJ3gfUF+yMmkb556D
f0H99CqnzQFxSD9jZg7/qFgcKbaiqtGoxMMsXvCaSpHAYaLyzU59QHOByWLPb37xKs8ChVHzK0nN
hNrOPjXgBdHfl4uubl4uuC/BqgAokzQMsONsFvb2r9Nc5n89LmofOdeX6Yl8n+BYYr5OyJsSPsWa
jGREiZ8Gt4E736PsfK5TNaMjnyf7oE2tscJgnnKmafzT0l6iQgxzsRYdkTL+BhQwOjET/Qvv2rZp
cED3bJDJac9OspuYWzYQWLheQwp3UiqCPnSNmbcNnDUd3ev0eBOqx8N6h1jkQPqh+Rotj5qpkbT5
XN2qxBHh8hihStmjiWi84PG9WPreHz0vXl6558upMOLi9QlbpP5uXfN8ovxeEmK1d+O/1yhmMarX
vVqmKo8MJtBmgznoON63BKgB9oQRfvafzmuReX7M5lf6y00vYLxEUIl7IJU+zXkoGTI720R9jcbV
y8SldwzgK8dYMHCDjQHPOqeuI7/Blyj9LDZmSycIti/pPLzZoUqwYq06FAk8i7IugEBgOnkwKng3
DYeUp/3WcJ4Zu/Z0to+osHrotLK/WN80ZIenpE15y/IczeObb97t3GQggTTANSKzTxVpP6ywQpD6
WOEWWfqpOO2mI849kbitnD8jJZFKCcrJaFrBNDY0VzqMR2eYUVJD+uzCl46lPRNvwFN8Syp2c7F+
ZBvgURV52n6nxhDf+k/MA6RN1nmbQiUPUxUROb6gIvdPf723eUWoz38BXh6AA/N9qifmhswziRAn
XrmRkmn3vddA7/c/o9EltHC0HcBl9oqgp3uuwkpMvep0lXtoczS3Aw9c0sUvOQdq9jjmzTVFOCtd
k2xpRj+g+4dWdDt95pb86d4sYh4qP4fGWtye1kqd6Yq4GGpCS92cNyAGASUYPeGXO+pPSy9iI2aN
cuyzR1tmfN8KhilN37ZjfuAurucv249uwICmwjG2zbmBh/fscrWm/o1sEXNsGAtn2P2fwFvLygY6
VDBiWWxSmSV1Zq0VdnxWPa6gujnrKMG+YS+sDxtXAPtzOivJM2UXS3KPWvFKKt7Ny1ftWvcV3TU7
fV0652f2K4Vmfpzs09T88pJKDmybQi89woak3MloijM6T1YfYjwP4yLoz6OfBomU3Fz0ZVfp3f8J
OadLdwV3qUB7BJSS37cLFP+ZWAUf/vK6ss114pZCPVLdvcZIPLG3jFVGPV9Eh0Tduc+QHSqmdcH4
uzYwA0gMxrRZxJwC8KxxM/vcNcTLXp47IFYJUPI9WF5oxRa/6rsTYgkSQinJeAJQToiVj1LqFtFl
GXh74XiAZrdAcsf1YXM3ag1ByRbrR/ShJK0h342E3UoU2+aRJhlf0UFpl6w2pKB9x66uvNGUP0PA
aiSpT+H6kX+6Pk4I1Fec7Q0lxr9pOYY9VZbZewvVMAtgNmz7vMowA6h8G4RaJRxPBsf6LXOEB/xk
WW/M9r/TKi1OqwPGHgfwKCDDMhpxO5EX48r9Qk5ojEdJXo46lzensTR+W6gZ54ZShvilyulb9gkQ
2faqFMiR2+QKeT9l/0d0tPWtz5zkeed9vn1XDfkLpKOlQ/6tIgVYl7ZeCPTLYQbRVsnTUoiDsgST
zOhG+tNnqr3vw26qMwL7lqkQQaklB8LExVL5y+9tOFN3qk55vMIiBrZPnOyNCAFLTJ0mi29saQTR
OGe1xE/JvS2+kwyrbNhQ5A27Y6n5gkOt+v/fv5bBVgtR8F3xLa/8ePpisIDJ3+gWZYRIrDXMty1p
gFKia4Cmmn8DNt9zzVdabFD5kWOync9oJeXzql9/kgEIeLYBMeTesCLTtiiKbJEYJzUvPUmQ8/eU
1zwxmS5brp+RLOVd1+zXvC6O7RFEbZiRpIsdJ2APTuGKRXpf/Dl1oD5xIIH5uuIMPzw0ylfBGK/F
RzxslHnTZNsP1fmqB/CSBThlGhKA8cfbD1O4LHEXtCxCyuO2Iu3kkXXhi1EfkubciFhvyKgXCkeP
c1f5Y9bfO1Pp/y4spaOSQJ+X8Bc4Vyv34KGEivfNV+1ux/15CPJlvZFhM7xmdRsUva3zfwb6z/uZ
VsY3GVrPA1s78LFdX9+K99/0u7yi8L/qrsLPag9CuH9SfJ1+vvl+t45Q+3AiWVHIEZGcl+lx6tx2
o1jwQsymeHwzwvaee4dnbzhe9V8Oe8KQhiVLFcb/H2XERTT1Dr1G47H8bjCRvRvew0Ibt0PnSkk1
7PJmfgFUUb1X6b3wEgJ0yDv+Dot1FuMQgHBjxWBLkiArA4Yr46pvXs4TLEfR0MhpgxV5jYQdi+OW
mt5GibMSGPFj0TgV2a7cqwgwPktmxbsuJUyp66YIykkQXmbRO+GT6tVK7/kutoOckg4CbyLa9d/J
swnZLZd2h62XcuJ4jjrSZcK8m8zJI1+V0SkdIw5DD7ms3l7+El/MUxPtPQ+PTvAMoihdwKe0fALg
sWuDzFLDuyNwobuuEkrsYrBmYjqaEiYl/mIrG7vxD5FlMRMVW8MnDOYkq+4cwOsLEfHmJI3ZCjqb
KShsD4x3409KlXIe5mOJsjX4SwAbcOM2h78BHqheDfjwFM8rqxrudcUsBHV7SnpEczWda+0fTxSs
F7MSszly0Wxw5HowPEeutre8YmMffO4R3oSyPHxoubLV43FdtyGxMOaC08Uz9E5OHKStrXaYlTG/
XImy4WXjWzAbSBEiR3nlfIOog8mB71YP3mLdzO6h9qFSUySoqyVvL3MLFce2LWy+osczUeZGGZwI
pMAFMzmU4z60swfLqtCRvE554IEVQiQ6GmrKhRkaK4m0BtmTEszn1bIyuqBesW6UQA4STZzMZJtQ
uwuE5qohSk6gCVuMmegd9NEdQq3gEzEbFIsfkMEBdYttovYL04KS2J52mH4/Azs1GaZQUYEqkU+5
+t1mNvDPWEYYbH6AeZO1QRIjgM26sEMXBqHOoYVRW6mcknqNgSENkzZf3mLDg4NTygaHjsqGG5cZ
ydbByDAK7upmMCMwyOqgaanbXGu0C65rPOe/TdaTubYebrIrbxy3MEls7fd0cQrQh8Sa7gTFX+ey
d0iCo/REFMlrnqMrHALsBI3qFZUYEB8EXypkIrs+M5mNN4eRZzgcx5uBj7fVCfXqPfCntgb2d8+V
M69jtawU9nFN55AN5FTrcwGkmBbs/I0wGfTUGfv25758TR+qUmPJzjSONGmo3gzGAzZZTCeQC8GL
IjjbTr4evpTK1gJM9qeF9VjirTfkETc9WUtrW2+d9VwM3UbLwwvD6vYqNCa7pKSZAttLmycenzbC
E3+Um0inNP/5GjacmQCpjpuHSUgKVfpj0Kr3iXd39WBJazD4wc/UUEEExk8KGBAiguqIU9RNvZG9
mWh5tP0lD4KszpTlGbxzqyc+szLT9lW2FT1NWLKeGVM+Niwszgj0O5EEW9euC6Pv/CCpnxFeHKhk
pSlbAUxTONwG643FmvctFMkmc/Jez4efzIw3e2s9nFIkgSP4B1PfjfOv8eDpaXzRhv1CEuC+XdJ6
pVoq1hMakHb7thpeiWJlOiNWfsVwQxEZzdymjLFa3B5+L7TSvTqA+uvNimoAbyAV2nKgBinOO7om
VdSSGdE8I6mA2xZHQbf912AUWmU/ljOn+FJ6mqoGIRwl3KXK3IdXfamZlaGWZlHh5hUWcRHqU9yN
6gjFIDczyCBf4H9QLc6jXr3Hp1xfXMDh2pfiKc1WMyFcwGtWNgmcYXM2b4T+caLQteJx+l0VWUZM
AiZ+zHQpTOgDYpJK6cpripyNaoY5o/W2hpVqJSYsjCrolpTGWUpYjNqnhEhZEp7MZmp67qR3rnGj
pIRyUDF3nYybPHt2Z8bkzjWn6P3SunoyEYooC/IwKuakgjvg66EjBB8knOb4yt3U921knwHoPTnu
g0dZEf3sZhv6jwd5KVY11p89/++cVQMByaKBTL2egCxtGUQXqQ0ApZBU3pMWwjwDOMPeACLr/lEd
EdUBtCSxidC0nq8KenjoXMggFV0aX11auhJdRDP4mcOiXHDsSQwl9+UdkMVRniLv2FwMUkLkHIOo
ep/K+8CFdBZJldtp7IRMO/f/LqFudE5rX1rxsRUsC8t4BJWNpOYVz4LnfLvE0L1e05Y8dX6Uezrl
wIjOypIRD34sSuucylHQLTricTun3tCr/vAGA95iAhm7uhjOBK+omQe6KGW/x14txj0SkYty2TJa
X8HFDKz+hxDNA8taZqQoT+Yluodd8ewxsz8K4DRthVjOj++LkM3VmS6rCTuFxLyn3AFBAK11/n72
Y5NdDQwxlew3VRiCeeS+JOqZPnD8uijZc9XAnmlpY1hCuKaL24nvuSiO5ZlOiNwC9or8VhAfEl17
4IS9fbf+NAHoFhAeZ3r6T45y/u1a8K3Gj7YAIIyNoJKua3X2yH04gPqBXbG0yNRA+iKm04dMSI9y
o8mjgsfNMFnTRYnBIYC4vb2/BCjOAVkFUvXpb0z9QJdNXL2X+Vi5FyAmUI1dqqULzd08Q2qs6xeU
+9L0fq4EY6bpQ04xopRtBGjbqLL65xpZi2y++Y9GyE3M9WWcAcvmf9hqjBjTJULc0Igvk3UmP94T
LkZ5W5q7JkuEdHF1s+Oro0wtbmCmaq2o+g2aM4cfmZnQwn7FtF5+pIunr9eFCM+5IQf4UR+R3P4g
Rs/zQKDZVeCtTcTs1LvBeu4t9cU46x6nI2vgG9eVxAydLAhip6VQNobru8gnk6vf0LODI+Wcfc/P
9YZvYyEDT47SUbuV1z/WdZd3Id7kgQ4vT0fdPJv8ZHYGjX9csVTHLy7S61KQXUC5baq2+yjFkmMI
4uX5yodqX2jueh2TNdADg50zpej2tgGz+jta0tqut0u5XAFR9g23SfFuWtG9Mm5+al76Siqe9kGz
jIeDX9H+VRdLzLFYU+CZvILISAL067x+DNTUPw8l1oM6UcAIKV38eSeo2c7qQlM3OmXzhJv+aMld
zp6Fi1RTVn4BAgDc8YORzfRU8J67tlP8iOufWKQOUYGWbuiX2IHJ0IVIDgB99coaoKDmNdPsml1k
ahzGwTWIo/GYEgvYv8Z4N/SvhyYssaE+5feg/pM5hK7L2QnA9TVxsKpDTCttn8+j02FNsQ1qu00V
JMqFYL1TKE/UBdsbwbb5UlyDjH1GdSHIxm7j2HNzGX2bWJqf603HNPrrGhGyHqpiS/nEWAqmquW4
CqPTHN0xcWgw+D/zYvlSt0y8v18hNh3tld9dlmYwMBx/QXuUlBCm6EzpuZ1T+LpG9YCWrH5L1q0G
qvwxkywbMZ3pjZSmU5ZvIbItkfAIgXuAvexdwA4iYcjCu0W8w2vqvdZKmMdfegRvF8l3KhofeEDK
eYnxMwALxse9/r3Rq6kk3r7xYdb4x/5rlbftufSIwE2kIZ6pAAAAAExk9dH9jyv0AAHtTPGHAgCM
dxocscRn+wIAAAAABFla
--Apple-Mail=_FD7D67CE-9F80-4BA6-956C-9CA5413DED19--


From MAILER-DAEMON Fri Jul 22 04:47:04 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bQW6q-0003bx-DI
	for mharc-coreutils@gnu.org; Fri, 22 Jul 2016 04:47:04 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33294)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1bQW6n-0003Zc-5j
	for coreutils@gnu.org; Fri, 22 Jul 2016 04:47:03 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <pth@suse.de>) id 1bQW6j-0007pu-0E
	for coreutils@gnu.org; Fri, 22 Jul 2016 04:47:00 -0400
Received: from mx2.suse.de ([195.135.220.15]:44470)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1bQW6i-0007p8-PE
	for coreutils@gnu.org; Fri, 22 Jul 2016 04:46:56 -0400
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 26EA4AAAD
	for <coreutils@gnu.org>; Fri, 22 Jul 2016 08:46:55 +0000 (UTC)
Date: Fri, 22 Jul 2016 10:46:54 +0200
From: Philipp Thomas <pth@suse.de>
To: coreutils@gnu.org
Subject: Speeding up df
Message-ID: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
X-Operating-System: openSUSE - Kernel Linux 4.6.3-1-default x86_64
Organization: SUSE Linux GmbH, GF: =?iso-8859-1?Q?Felix_?=
	=?iso-8859-1?Q?Imend=F6rffer=2C_Jane_Smithard=2C_Graham_Norton=2C_HRB_212?=
	=?iso-8859-1?Q?84_=28AG_N=FCrnberg=29?=
User-Agent: Mutt/1.6.1-neo (2016-05-23)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x (no
	timestamps) [generic]
X-Received-From: 195.135.220.15
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Jul 2016 08:47:03 -0000

To help a customer that complained about the time it takes for df to check
one of their machines with nearly 22000 lines in /proc/self/mountinfo, my
collegue came up with this patch that uses a hash table for seaching in 
filter_mount_list() and get_dev(). Data from said machine:

Without patch:
real    0m2.273s
user    0m0.768s
sys     0m1.500s

real    0m1.731s
user    0m0.532s
sys     0m1.188s

With patch:
real    0m1.286s
user    0m0.052s
sys     0m1.232s

real    0m1.066s
user    0m0.028s
sys     0m1.032s

Is this patch acceptable?

Philipp


From MAILER-DAEMON Fri Jul 22 07:48:31 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bQYwR-0002T3-BF
	for mharc-coreutils@gnu.org; Fri, 22 Jul 2016 07:48:31 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40509)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQYwO-0002Rw-Pn
	for coreutils@gnu.org; Fri, 22 Jul 2016 07:48:29 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQYwL-0003Um-MG
	for coreutils@gnu.org; Fri, 22 Jul 2016 07:48:28 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:38092)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQYwL-0003UZ-GJ
	for coreutils@gnu.org; Fri, 22 Jul 2016 07:48:25 -0400
Received: from [192.168.1.80] (unknown [109.77.96.86])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id E1E3E4AD3;
	Fri, 22 Jul 2016 12:48:21 +0100 (IST)
Subject: Re: multibyte processing - handling invalid sequences (long)
To: Assaf Gordon <assafgordon@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
	<654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
Cc: Eric Blake <eblake@redhat.com>, Coreutils <coreutils@gnu.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <57920805.6010806@draigBrady.com>
Date: Fri, 22 Jul 2016 12:48:21 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Jul 2016 11:48:30 -0000

On 22/07/16 04:23, Assaf Gordon wrote:
> Hello,
>=20
>> On Jul 21, 2016, at 06:08, P=C3=A1draig Brady <P@draigBrady.com> wrote=
:
>> [...]
>> It seems like --normalization=3D{NFKD,NFKD,NFC,NFD} functionality woul=
d
>> also be quite cohesive in such a util.
>=20
> Attached an improved version with unicode normalization support.

Wow, very nice.

> Before continuing with other stuff (e.g. more tests, documentation, new=
s, etc.),
> it's worth discussing if this is the path to take (or if we want to add=
 this to each individual utility).

I'm not sure, but it would be nice as I said if we could get away with "r=
eplace" mode in other utils.
By having a separate util, it follows the idea of validating/transforming=
 input as early as possible
so as to simplify the rest of the system.  Also it follows the idea that =
if something can be
done separately it should be done so.

> Also, do we keep these options or modify them?
> e.g. 'uconv' uses different terminology for handling invalid sequences:=
 stop, skip, substitute, escape (corresponding to abort, discard, replace=
, recode below).

Doesn't really matter.
I find your naming slightly more descriptive.

> To keep the implementation simple, unicode normalization requires UTF-8=
 locales - is this a valid requirement?

Given how prevalent utf8 is I think this is fine.
It other tools if there is an option we should also tune for utf-8 input.

> And of course, what about the name?

I've a slight preference for unorm

thanks!
P=C3=A1draig


From MAILER-DAEMON Sat Jul 23 06:51:42 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bQuX0-0007Bd-MV
	for mharc-coreutils@gnu.org; Sat, 23 Jul 2016 06:51:42 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:41060)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQuWy-0007BV-2G
	for coreutils@gnu.org; Sat, 23 Jul 2016 06:51:41 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQuWt-00045P-Ro
	for coreutils@gnu.org; Sat, 23 Jul 2016 06:51:38 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:41436)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bQuWt-00045E-Lj
	for coreutils@gnu.org; Sat, 23 Jul 2016 06:51:35 -0400
Received: from [192.168.1.80] (unknown [109.79.84.146])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 1654994A7;
	Sat, 23 Jul 2016 11:51:32 +0100 (IST)
Subject: Re: multibyte processing - handling invalid sequences (long)
To: Assaf Gordon <assafgordon@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
	<654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
Cc: Eric Blake <eblake@redhat.com>, Coreutils <coreutils@gnu.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <57934C33.30308@draigBrady.com>
Date: Sat, 23 Jul 2016 11:51:31 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Jul 2016 10:51:41 -0000

On 22/07/16 04:23, Assaf Gordon wrote:
> Hello,
>=20
>> On Jul 21, 2016, at 06:08, P=C3=A1draig Brady <P@draigBrady.com> wrote=
:
>> [...]
>> It seems like --normalization=3D{NFKD,NFKD,NFC,NFD} functionality woul=
d
>> also be quite cohesive in such a util.
>=20
> Attached an improved version with unicode normalization support.
>=20
> Before continuing with other stuff (e.g. more tests, documentation, new=
s, etc.),
> it's worth discussing if this is the path to take (or if we want to add=
 this to each individual utility).
> Also, do we keep these options or modify them?
> e.g. 'uconv' uses different terminology for handling invalid sequences:=
 stop, skip, substitute, escape (corresponding to abort, discard, replace=
, recode below).
>=20
> To keep the implementation simple, unicode normalization requires UTF-8=
 locales - is this a valid requirement?
>=20
> And of course, what about the name?
>=20
> Comments welcomed,
>  - assaf
>=20
>=20
>=20
>=20
> Example (from 'Unicode Explained' book):
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> $ printf '\uFB01anc\u00E9\n'
> =EF=AC=81anc=C3=A9
>=20
> $ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfd | od -An -tx1c
>   ef  ac  81  61  6e  63  65  cc  81  0a
>    ?   ? 201   a   n   c   e   ? 201  \n
>=20
> $ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfc | od -An -tx1c
>   ef  ac  81  61  6e  63  c3  a9  0a
>    ?   ? 201   a   n   c   ?   ?  \n
>=20
> $ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfkd | od -An -tx1c
>   66  69  61  6e  63  65  cc  81  0a
>    f   i   a   n   c   e   ? 201  \n
>=20
> $ printf '\uFB01anc\u00E9\n' | ./src/mbfix -n nfkc | od -An -tx1c
>   66  69  61  6e  63  c3  a9  0a
>    f   i   a   n   c   ?   ?  \n
>=20
> $ ./src/mbfix --help
> Usage: ./src/mbfix [OPTION]... [FILE]...
> Fix and adjust multibyte character in files
>=20
> Mandatory arguments to long options are mandatory for short options too=
.
>   -A, --abort          same as --policy=3Dabort
>   -C, --recode         same as --policy=3Drecode
>   -c, --check          validate input, no output
>   -D, --discard        same as --policy=3Ddiscard
>   -n, --normalization=3DNORM
>                        apply unicode normalization NORM:, one of:
>                        nfd, nfc, nfkd, nfkc. Normalization requires
>                        UTF-8 locales.
>   -p, --policy=3DPOLICY  invalid-input policy: discard, abort
>                        replace (default), recode
>   -R, --replace        same as --policy=3Dreplace
>       --replace-char=3DN
>                        with 'replace' policy, use unicode character N
>                        (default: 0xFFFD 'REPLACEMENT CHARACTER')
>       --recode-format=3DFMT
>                        with 'recode' policy, recode invalid octets
>                        using FMT printf-format (default: '<0x%02x>')
>   -v, --verbose        report location of invalid input
>   -z, --zero-terminated    line delimiter is NUL, not newline
>       --help     display this help and exit
>       --version  output version information and exit

I was wondering about the tool being line/record oriented.

Disadvantages are:
  requires arbitrary large buffers for arbitrary long lines
  relatively slow in the presence of short/normal lines
  sensitive to the current stdio buffering mode
  requires -z option to support NUL termination

Processing instead a block at a time avoid such issues.
UTF-8 at least is self synchronising, so after reading a block
you just have to look at the last 3 bytes to know
how many to append to the start of the next block.

P=C3=A1draig.


From MAILER-DAEMON Sat Jul 23 14:05:50 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bR1J7-00014j-Sp
	for mharc-coreutils@gnu.org; Sat, 23 Jul 2016 14:05:49 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:50342)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bR1J5-00012o-MC
	for coreutils@gnu.org; Sat, 23 Jul 2016 14:05:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bR1J2-00049t-FX
	for coreutils@gnu.org; Sat, 23 Jul 2016 14:05:47 -0400
Received: from mail-qk0-x22f.google.com ([2607:f8b0:400d:c09::22f]:35184)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bR1J2-00049p-As
	for coreutils@gnu.org; Sat, 23 Jul 2016 14:05:44 -0400
Received: by mail-qk0-x22f.google.com with SMTP id s63so126536843qkb.2
	for <coreutils@gnu.org>; Sat, 23 Jul 2016 11:05:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=g7orWrYYEErlMS8hr9b3+5dKoImeVP4JulVhkeghGMc=;
	b=wfDHK0KRPbpbBCL6KoUIVFzu+cKdEewvveoxFzHn9yjbxkhaO1u2zeAm4ELLBvthLj
	DgPdIMW84QD3hBhdzcoQ2011dU+r4MQXx2+BjJGMu4tEQ2aWakj5MQJw/RVE+yIGW7ea
	P+zEqSMnlcyzPhQe/YmI6J1GO2uSg0jSOiGUuxiH49xlRnTdDuSBb6YvBre0o0jkhokV
	fxgQBALt/EFLAja4Sa5wcdMlXV3pLTOiiBcRIgrW8/sHAPYDXWfCcn4F8DstsYwpSV5t
	C4ir0iM8G1NVfl34DqZvWFOaU6B1WZGv6cRFAGf+SjPsGk2D/astCicrhjmzylxYOn6L
	ch8A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=g7orWrYYEErlMS8hr9b3+5dKoImeVP4JulVhkeghGMc=;
	b=Q56KAIfIWVIavL7sbT5GrbzESl9+5bxB8nYYpbhv/zskAQ2DXY1izXHdUUrnORv3bP
	/cDH5pVzbrXW3+duid7vgosMKJu6WqfabiHQhTI4Mu9E/ZGmIVJpdoHkQ5D9omGL/xV4
	QaVYr55xhPIH4nZF7NJxBFvDs2YagXOsYUIxPjgaSJMgv8OzAOgCphTBNmgxOIFDf8Ew
	8XRHj+DNNv6t3y4vcsX7NPUO4/XF3ne0+Hz0ddPgEjGpm2gRIRWsZqYTwLlWtdCi11Lz
	s/t82BVdWUAXKQ+9aXsgRRxHQJM+FzQIzT8Yyo2RK2SkspHVejimTdN1rBl26r/26U72
	q3/g==
X-Gm-Message-State: AEkooutToq55Va1FnpgTSsUAKuQFoeTUdtzQZR51OP3ZtRNTuLrDjg2myFockgUWO68kAw==
X-Received: by 10.55.96.133 with SMTP id u127mr12956392qkb.55.1469297127139;
	Sat, 23 Jul 2016 11:05:27 -0700 (PDT)
Received: from ix.home (pool-100-38-105-55.nycmny.fios.verizon.net.
	[100.38.105.55]) by smtp.gmail.com with ESMTPSA id
	o84sm10566008qke.38.2016.07.23.11.05.25
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Sat, 23 Jul 2016 11:05:26 -0700 (PDT)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: multibyte processing - handling invalid sequences (long)
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <57934C33.30308@draigBrady.com>
Date: Sat, 23 Jul 2016 14:05:25 -0400
Cc: Eric Blake <eblake@redhat.com>,
 Coreutils <coreutils@gnu.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <D92CD138-0E60-46C4-B54C-96C44DA7784E@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
	<654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
	<57934C33.30308@draigBrady.com>
To: =?utf-8?Q?P=C3=A1draig_Brady?= <P@draigBrady.com>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c09::22f
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Jul 2016 18:05:48 -0000


> On Jul 23, 2016, at 06:51, P=C3=A1draig Brady <P@draigBrady.com> =
wrote:
> I was wondering about the tool being line/record oriented.
>=20
> Disadvantages are:
>  requires arbitrary large buffers for arbitrary long lines
>  relatively slow in the presence of short/normal lines
>  sensitive to the current stdio buffering mode
>  requires -z option to support NUL termination
>=20
> Processing instead a block at a time avoid such issues.
> UTF-8 at least is self synchronising, so after reading a block
> you just have to look at the last 3 bytes to know
> how many to append to the start of the next block.

block-at-a-time would work well for detecting/fixing invalid multibyte =
sequences, especially in UTF-8.
But I'm not sure about other multibyte encodings (I'll have to =
investigate).

However, for unicode normalization, I am not sure there's a stream =
interface to it (gnu lib's uniform takes a whole string to normalize). =
IIUC, normalization requires being able to examine some unicode =
characters ahead.

-assaf



From MAILER-DAEMON Sat Jul 23 16:30:20 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bR3Yy-0005uh-OF
	for mharc-coreutils@gnu.org; Sat, 23 Jul 2016 16:30:20 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:43344)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bR3Yw-0005ua-F9
	for coreutils@gnu.org; Sat, 23 Jul 2016 16:30:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bR3Yr-0003vw-0R
	for coreutils@gnu.org; Sat, 23 Jul 2016 16:30:17 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:42754)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bR3Yq-0003vG-QL
	for coreutils@gnu.org; Sat, 23 Jul 2016 16:30:12 -0400
Received: from [192.168.1.80] (unknown [109.79.84.146])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id CDC2B94A7;
	Sat, 23 Jul 2016 21:30:08 +0100 (IST)
Subject: Re: multibyte processing - handling invalid sequences (long)
To: Assaf Gordon <assafgordon@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
	<654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
	<57934C33.30308@draigBrady.com>
	<D92CD138-0E60-46C4-B54C-96C44DA7784E@gmail.com>
Cc: Eric Blake <eblake@redhat.com>, Coreutils <coreutils@gnu.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <5793D3CF.4010707@draigBrady.com>
Date: Sat, 23 Jul 2016 21:30:07 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <D92CD138-0E60-46C4-B54C-96C44DA7784E@gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Jul 2016 20:30:19 -0000

On 23/07/16 19:05, Assaf Gordon wrote:
>=20
>> On Jul 23, 2016, at 06:51, P=C3=A1draig Brady <P@draigBrady.com> wrote=
:
>> I was wondering about the tool being line/record oriented.
>>
>> Disadvantages are:
>>  requires arbitrary large buffers for arbitrary long lines
>>  relatively slow in the presence of short/normal lines
>>  sensitive to the current stdio buffering mode
>>  requires -z option to support NUL termination
>>
>> Processing instead a block at a time avoid such issues.
>> UTF-8 at least is self synchronising, so after reading a block
>> you just have to look at the last 3 bytes to know
>> how many to append to the start of the next block.
>=20
> block-at-a-time would work well for detecting/fixing invalid multibyte =
sequences, especially in UTF-8.
> But I'm not sure about other multibyte encodings (I'll have to investig=
ate).
>=20
> However, for unicode normalization, I am not sure there's a stream inte=
rface to it (gnu lib's uniform takes a whole string to normalize). IIUC, =
normalization requires being able to examine some unicode characters ahea=
d.

Oh right I see.

You're saying that splitting per line is a natural way to ensure
you don't split processing in the middle of a decomposed character,
which is significant in normalization processing.

To support that you'd have to do something like:

  filter =3D uninorm_filter_create()
  while (read(fd, buf, BUFSIZE))
    for each mbchar;
      uchar =3D mbtowchar(mbchar);
      if (!uchar) //fix
        uninorm_filter_write(filter, uchar);
    uninorm_filter_flush(filter)

I don't know how that would perform compared to u8_normalize().
It might be faster since we're already processing each char?
Or it might be slower if u8_normalize() has some utf8 specific optimizati=
ons.

cheers,
P=C3=A1draig


From MAILER-DAEMON Sun Jul 24 17:33:59 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bRR27-0003hx-BT
	for mharc-coreutils@gnu.org; Sun, 24 Jul 2016 17:33:59 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:52510)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bRR24-0003hp-Rr
	for coreutils@gnu.org; Sun, 24 Jul 2016 17:33:58 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bRR1z-0002SO-PP
	for coreutils@gnu.org; Sun, 24 Jul 2016 17:33:56 -0400
Received: from mout.kundenserver.de ([212.227.126.135]:62111)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bRR1z-0002SH-FD
	for coreutils@gnu.org; Sun, 24 Jul 2016 17:33:51 -0400
Received: from [192.168.101.10] ([84.175.95.115]) by mrelayeu.kundenserver.de
	(mreue001) with ESMTPSA (Nemesis) id 0MApaN-1bcgcr0PNr-00BpBK;
	Sun, 24 Jul 2016 23:33:46 +0200
Subject: Re: Speeding up df
To: Philipp Thomas <pth@suse.de>, coreutils@gnu.org
References: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
From: Bernhard Voelker <mail@bernhard-voelker.de>
Message-ID: <644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
Date: Sun, 24 Jul 2016 23:33:43 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
	Thunderbird/45.2
MIME-Version: 1.0
In-Reply-To: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
Content-Type: multipart/mixed; boundary="------------9FECF7EECC03DCCED0187705"
X-Provags-ID: V03:K0:XFB7Um05Rz8+9nk73okbgWUXtQO4S5FRCAaRlvJ0p0BbEsgS2NG
	Lpjc/HmfTDTShB94VzQ9O/LvF7MvVbT05jgRmo5ki1dJUhrGYtN0RZihjihLEWoumSU4LTY
	OLa8AyPKt9RXDzvLHnJrb2bsvaUiyGK8Pc9uzqx124SZrSPMnATQYt1rdXAzun9opDDqPAg
	+s3D18If9rZ+VzzRVc9ig==
X-UI-Out-Filterresults: notjunk:1;V01:K0:6Sz8ZWVMm3g=:p+k0D9oY9N2qQb5JXegzlo
	G5RUlg4a8PZq2MsNdZUlrJLqnHlM6GXB6Z4/tMqKrDYr5bD/66RoofFq6Y0vF02CE5JKKx5Q8
	C1WBnJzUtkvg3j879Zy/PztvxTRYsQl2uguXeh/Q5EurWP2hmOGA4vRqN0DDa0NhGdizwxYM5
	Fhhj/TbF/gVE3wmTOOVagAELCZmu6PMtkt46xcphFNeuto12GhRckoxm++j1bI3SPB9VjMIvd
	ETrxsmKJ9wfbLRtAhOaJkrxp+DD2DSiBL/aZdjVCFsKEHF9YZJv0TiRj8UFL+rT6Qnk2q84sS
	XbPltPvBd3BePDZo6EQfqkd9bLTrDAAnmiPIs4K9qulxRn2O1qDsKZx6zVGsAcSKe4WVE2heF
	cNJZCIkjWOQHN/6MGuR1qDxazn9iGVOGdYUCMZ20YvWVy2y0OZYkeJFPW7zxBW4BnQ/PkLHQa
	SfxrhCVJ5bBig3ex39MacPp/lIWyZWNLb26tyskkUo9lCQEM7akiDErd9XZYDuFze/Jh0XUHX
	l7TaP3p1VnuNqxruJAz6sL7uVjFEuXxKKYJ96U+16tgq4zOGpfwHlkqeNeRsY6AbPficT3iRr
	GLdJbPvQYPpdkZsTH+mCzujkTjerHr9SD0LOI0T8d6RRyQ0fKg1I9xDnXeeTs+WX3uJkmaK2P
	dZDhTSWl+11G1Nud3+f1nS+GBts5LpKqjO/DTmma3ytsFbYWfb4cTe1u24FlpZLq9bAA=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.126.135
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 24 Jul 2016 21:33:58 -0000

This is a multi-part message in MIME format.
--------------9FECF7EECC03DCCED0187705
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit

On 07/22/2016 10:46 AM, Philipp Thomas wrote:
> To help a customer that complained about the time it takes for df to check
> one of their machines with nearly 22000 lines in /proc/self/mountinfo, my
> collegue came up with this patch that uses a hash table for seaching in 
> filter_mount_list() and get_dev(). Data from said machine:
> 
> Without patch:
> real    0m2.273s
> user    0m0.768s
> sys     0m1.500s
> 
> real    0m1.731s
> user    0m0.532s
> sys     0m1.188s
> 
> With patch:
> real    0m1.286s
> user    0m0.052s
> sys     0m1.232s
> 
> real    0m1.066s
> user    0m0.028s
> sys     0m1.032s
> 
> Is this patch acceptable?

Hi Philipp,

the patch was missing, so I attached it from here:

https://build.opensuse.org/package/view_file/Base:System/coreutils/coreutils-df-hash-in-filter.patch?expand=1

Maybe adding an IF_LINT'ed hash_free call would be worth;
otherwise it looks good to me.

Have a nice day,
Berny

--------------9FECF7EECC03DCCED0187705
Content-Type: text/x-patch;
 name="coreutils-df-hash-in-filter.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="coreutils-df-hash-in-filter.patch"

From: Philipp Thomas <pth@suse.de>
Date: 2016-07-22 10:34:59+02:00
Subject: speed up df
References: 
Upstream: 


Use hash table for seaching in filter_mount_list() and get_dev().
Patch was done by Josef Cejka <jcejka@suse.com>.

---
 src/df.c |   80 ++++++++++++++++++++++++++++++++++++++++++++++++---------------
 1 file changed, 62 insertions(+), 18 deletions(-)

Index: coreutils-8.25/src/df.c
===================================================================
--- coreutils-8.25.orig/src/df.c	2016-07-22 10:31:00.838791851 +0200
+++ coreutils-8.25/src/df.c	2016-07-22 10:33:37.285989052 +0200
@@ -34,6 +34,7 @@
 #include "mountlist.h"
 #include "quote.h"
 #include "find-mount-point.h"
+#include "hash.h"
 
 /* The official name of this program (e.g., no 'g' prefix).  */
 #define PROGRAM_NAME "df"
@@ -43,14 +44,16 @@
   proper_name ("David MacKenzie"), \
   proper_name ("Paul Eggert")
 
-/* Filled with device numbers of examined file systems to avoid
-   duplicates in output.  */
-static struct devlist
+struct devlist
 {
   dev_t dev_num;
   struct mount_entry *me;
   struct devlist *next;
-} *device_list;
+};
+
+/* Filled with device numbers of examined file systems to avoid
+   duplicates in output.  */
+static Hash_table *devlist_table;
 
 /* If true, show even file systems with zero size or
    uninteresting types.  */
@@ -603,6 +606,37 @@ excluded_fstype (const char *fstype)
   return false;
 }
 
+static size_t
+devlist_hash (void const *x, size_t table_size)
+{
+  struct devlist const *p = x;
+  return (uintmax_t) p->dev_num % table_size;
+}
+
+static bool
+devlist_compare (void const *x, void const *y)
+{
+  struct devlist const *a = x;
+  struct devlist const *b = y;
+  return a->dev_num == b->dev_num;
+}
+
+static struct devlist *
+devlist_for_dev (dev_t dev)
+{
+  struct devlist dev_entry;
+  dev_entry.dev_num = dev;
+  if (devlist_table == NULL)
+    return NULL;
+  return hash_lookup(devlist_table, &dev_entry);
+}
+
+static void
+devlist_free(void *p)
+{
+  free(p);
+}
+
 /* Filter mount list by skipping duplicate entries.
    In the case of duplicates - based on the device number - the mount entry
    with a '/' in its me_devname (i.e., not pseudo name like tmpfs) wins.
@@ -615,6 +649,20 @@ filter_mount_list (bool devices_only)
 {
   struct mount_entry *me;
 
+  /* temp list to keep entries ordered */
+  struct devlist *device_list = NULL;
+  int mount_list_size = 0;
+
+  for (me = mount_list; me; me = me->me_next)
+    mount_list_size++;
+
+  devlist_table = hash_initialize (mount_list_size, NULL,
+                                 devlist_hash,
+                                 devlist_compare,
+                                 devlist_free);
+  if (devlist_table == NULL)
+    xalloc_die ();
+
   /* Sort all 'wanted' entries into the list device_list.  */
   for (me = mount_list; me;)
     {
@@ -635,9 +683,7 @@ filter_mount_list (bool devices_only)
       else
         {
           /* If we've already seen this device...  */
-          for (devlist = device_list; devlist; devlist = devlist->next)
-            if (devlist->dev_num == buf.st_dev)
-              break;
+          devlist = devlist_for_dev(buf.st_dev);
 
           if (devlist)
             {
@@ -697,6 +743,8 @@ filter_mount_list (bool devices_only)
           devlist->dev_num = buf.st_dev;
           devlist->next = device_list;
           device_list = devlist;
+          if (hash_insert (devlist_table, devlist) == NULL)
+            xalloc_die ();
 
           me = me->me_next;
         }
@@ -711,28 +759,24 @@ filter_mount_list (bool devices_only)
         me = device_list->me;
         me->me_next = mount_list;
         mount_list = me;
-        /* Free devlist entry and advance.  */
-        struct devlist *devlist = device_list->next;
-        free (device_list);
-        device_list = devlist;
+        device_list = device_list->next;
       }
+
+      hash_free(devlist_table);
+      devlist_table = NULL;
   }
 }
 
+
 /* Search a mount entry list for device id DEV.
    Return the corresponding mount entry if found or NULL if not.  */
 
 static struct mount_entry const * _GL_ATTRIBUTE_PURE
 me_for_dev (dev_t dev)
 {
-  struct devlist *dl = device_list;
-
-  while (dl)
-    {
-      if (dl->dev_num == dev)
+  struct devlist *dl = devlist_for_dev(dev);
+  if (dl)
         return dl->me;
-      dl = dl->next;
-    }
 
   return NULL;
 }

--------------9FECF7EECC03DCCED0187705--


From MAILER-DAEMON Sun Jul 24 19:20:49 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bRShV-00025d-Mg
	for mharc-coreutils@gnu.org; Sun, 24 Jul 2016 19:20:49 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:42997)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bRShS-00025W-V1
	for coreutils@gnu.org; Sun, 24 Jul 2016 19:20:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bRShO-00068a-N0
	for coreutils@gnu.org; Sun, 24 Jul 2016 19:20:45 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:46646)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bRShO-00068S-GU
	for coreutils@gnu.org; Sun, 24 Jul 2016 19:20:42 -0400
Received: from [192.168.1.80] (unknown [109.79.115.81])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 4A9D9244;
	Mon, 25 Jul 2016 00:20:38 +0100 (IST)
Subject: Re: Speeding up df
To: Bernhard Voelker <mail@bernhard-voelker.de>, Philipp Thomas
	<pth@suse.de>, coreutils@gnu.org
References: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
	<644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <57954D45.6010408@draigBrady.com>
Date: Mon, 25 Jul 2016 00:20:37 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 24 Jul 2016 23:20:48 -0000

On 24/07/16 22:33, Bernhard Voelker wrote:
> On 07/22/2016 10:46 AM, Philipp Thomas wrote:
>> To help a customer that complained about the time it takes for df to c=
heck
>> one of their machines with nearly 22000 lines in /proc/self/mountinfo,=
 my
>> collegue came up with this patch that uses a hash table for seaching i=
n=20
>> filter_mount_list() and get_dev(). Data from said machine:
>>
>> Without patch:
>> real    0m2.273s
>> user    0m0.768s
>> sys     0m1.500s
>>
>> real    0m1.731s
>> user    0m0.532s
>> sys     0m1.188s
>>
>> With patch:
>> real    0m1.286s
>> user    0m0.052s
>> sys     0m1.232s
>>
>> real    0m1.066s
>> user    0m0.028s
>> sys     0m1.032s
>>
>> Is this patch acceptable?
>=20
> Hi Philipp,
>=20
> the patch was missing, so I attached it from here:
>=20
> https://build.opensuse.org/package/view_file/Base:System/coreutils/core=
utils-df-hash-in-filter.patch?expand=3D1
>=20
> Maybe adding an IF_LINT'ed hash_free call would be worth;
> otherwise it looks good to me.

Excellent. +1
Worth a mention in improvements in NEWS too.

thanks,
P=E1draig



From MAILER-DAEMON Mon Jul 25 07:16:22 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bRdry-0005F4-QQ
	for mharc-coreutils@gnu.org; Mon, 25 Jul 2016 07:16:22 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40343)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1bRdrv-0005Cv-MV
	for coreutils@gnu.org; Mon, 25 Jul 2016 07:16:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <pth@suse.de>) id 1bRdrr-0004i2-Gr
	for coreutils@gnu.org; Mon, 25 Jul 2016 07:16:18 -0400
Received: from mx2.suse.de ([195.135.220.15]:52652)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1bRdrr-0004gi-A9
	for coreutils@gnu.org; Mon, 25 Jul 2016 07:16:15 -0400
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id AD092ACD8
	for <coreutils@gnu.org>; Mon, 25 Jul 2016 11:16:12 +0000 (UTC)
Date: Mon, 25 Jul 2016 13:16:12 +0200
From: Philipp Thomas <pth@suse.de>
To: coreutils@gnu.org
Subject: Re: Speeding up df
Message-ID: <20160725111612.txlr7qwzfeozecno@paradies.suse.de>
References: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
	<644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
X-Operating-System: openSUSE - Kernel Linux 4.6.3-1-default x86_64
Organization: SUSE Linux GmbH, GF: =?iso-8859-1?Q?Felix_?=
	=?iso-8859-1?Q?Imend=F6rffer=2C_Jane_Smithard=2C_Graham_Norton=2C_HRB_212?=
	=?iso-8859-1?Q?84_=28AG_N=FCrnberg=29?=
User-Agent: Mutt/1.6.1-neo (2016-05-23)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x (no
	timestamps) [generic]
X-Received-From: 195.135.220.15
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Jul 2016 11:16:20 -0000

Hi Bernie,

* Bernhard Voelker (mail@bernhard-voelker.de) [20160724 23:33]:

> Maybe adding an IF_LINT'ed hash_free call would be worth;
> otherwise it looks good to me.

I don't quite understand as filter_mount_list already does call hash_free at
the end. So please explain.

Philipp


From MAILER-DAEMON Mon Jul 25 07:21:08 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bRdwa-0006EM-MP
	for mharc-coreutils@gnu.org; Mon, 25 Jul 2016 07:21:08 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40997)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1bRdwY-0006DO-UH
	for coreutils@gnu.org; Mon, 25 Jul 2016 07:21:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <pth@suse.de>) id 1bRdwU-0005Ke-NO
	for coreutils@gnu.org; Mon, 25 Jul 2016 07:21:05 -0400
Received: from mx2.suse.de ([195.135.220.15]:53117)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1bRdwU-0005KJ-Gf
	for coreutils@gnu.org; Mon, 25 Jul 2016 07:21:02 -0400
X-Virus-Scanned: by amavisd-new at test-mx.suse.de
Received: from relay2.suse.de (charybdis-ext.suse.de [195.135.220.254])
	by mx2.suse.de (Postfix) with ESMTP id 042F5ACAB
	for <coreutils@gnu.org>; Mon, 25 Jul 2016 11:21:02 +0000 (UTC)
Date: Mon, 25 Jul 2016 13:21:01 +0200
From: Philipp Thomas <pth@suse.de>
To: coreutils@gnu.org
Subject: Re: Speeding up df
Message-ID: <20160725112101.wnhlxhwiqahumwx6@paradies.suse.de>
References: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
	<644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
	<57954D45.6010408@draigBrady.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <57954D45.6010408@draigBrady.com>
X-Operating-System: openSUSE - Kernel Linux 4.6.3-1-default x86_64
Organization: SUSE Linux GmbH, GF: =?iso-8859-1?Q?Felix_?=
	=?iso-8859-1?Q?Imend=F6rffer=2C_Jane_Smithard=2C_Graham_Norton=2C_HRB_212?=
	=?iso-8859-1?Q?84_=28AG_N=FCrnberg=29?=
User-Agent: Mutt/1.6.1-neo (2016-05-23)
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x (no
	timestamps) [generic]
X-Received-From: 195.135.220.15
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Jul 2016 11:21:07 -0000

* P=E1draig Brady (P@draigBrady.com) [20160725 01:20]:

> > otherwise it looks good to me.
>=20
> Excellent. +1

I'll pass on your kind words to my collegue.

> Worth a mention in improvements in NEWS too.

Like so?

+  Df now uses a hash table to filter the mount list. This sped up
+  processing of a 22000 lines mount list by nearly 60%.

If yes and Bernies explains me why we'd need an additional hash_free call
I'll post the changed patch.

Philipp


From MAILER-DAEMON Mon Jul 25 11:48:44 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bRi7Y-0004Xt-NV
	for mharc-coreutils@gnu.org; Mon, 25 Jul 2016 11:48:44 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40064)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@gmail.com>) id 1bRi7W-0004Vr-1w
	for coreutils@gnu.org; Mon, 25 Jul 2016 11:48:43 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@gmail.com>) id 1bRi7T-0002UO-TC
	for coreutils@gnu.org; Mon, 25 Jul 2016 11:48:41 -0400
Received: from mail-oi0-x233.google.com ([2607:f8b0:4003:c06::233]:35735)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@gmail.com>) id 1bRi7T-0002UE-MK
	for coreutils@gnu.org; Mon, 25 Jul 2016 11:48:39 -0400
Received: by mail-oi0-x233.google.com with SMTP id l72so257994773oig.2
	for <coreutils@gnu.org>; Mon, 25 Jul 2016 08:48:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:sender:from:date:message-id:subject:to;
	bh=+AufyENt/9a4Ff/wqutSUxaLUbHMgmMOO/RtGSMW0Ig=;
	b=tBbJe9LqB0e9PdfhKpJPZvdgVnRQxOkvr1074c0DDepu2oeYb/WdZ6lNYhZtr/6XVj
	zunaKXPhdvA3G8HnLdbqXzYFw0K2O1gZS7nys4UC0QoW9zwMXKZrpm8wlDEp/3AqxqdT
	WC4NUJtQaIHhjOieZR2wkdUmJi/sgHRYxlfXyS2jGY5T7YOT+or3ZbGkL5/1qhQvuohp
	Hl8FKxJtMEDVhM61bpw7Yl+/YS3lPZT1aXFtVkhNaUn4jVF3WyC5x1FvEoZ3kd3H0+QI
	IXAZV6EcSUOhwW8QiZPvhWARgOp5nt92kKNTvCP21trU1JTnOMfTQTV3+vWKv2K6DulG
	prNg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:sender:from:date:message-id:subject
	:to; bh=+AufyENt/9a4Ff/wqutSUxaLUbHMgmMOO/RtGSMW0Ig=;
	b=CXXIiS5UOKLVXEpqqjZZGk6+fnTBsiuYdGNvCgG+A6WgBO8YMUSJOrha2b2xY8r0bo
	LpQhhC+Ho6IXWpWJVvLK/UHACfEE1lWYoe5kj81W//rCL3ptH23e2ZLqjuVDTjRzz352
	zfMY8ym1FM+ljLxwBgEw9HRewjlKYDwkTzg0PQ6uiX07vjBZC465DpvoMpm7L/9G4TZe
	j2NHIWmK7it3c6ns1ZJ0aRcw/RiGgXH1XIHE5se/IuioRaC2CzxmEtZZaujfR8ZBczFM
	4UYOjjOSGo+atZQ9+nC24YaygsYSkXJgtTqMMqOkbZVavEPIXNlHDrz44rXUJWgGXyor
	tSgA==
X-Gm-Message-State: AEkoousXquaBjnySijbROJ26tfb3A748IiTsw2x+fyGfsP2I1cejFHyx4xHdvAAM30Vn9I2Gn1MFlfkFY3kqHQ==
X-Received: by 10.157.60.228 with SMTP id t33mr9043350otf.163.1469461718259;
	Mon, 25 Jul 2016 08:48:38 -0700 (PDT)
MIME-Version: 1.0
Sender: meyering@gmail.com
Received: by 10.202.3.130 with HTTP; Mon, 25 Jul 2016 08:48:18 -0700 (PDT)
From: Jim Meyering <jim@meyering.net>
Date: Mon, 25 Jul 2016 08:48:18 -0700
X-Google-Sender-Auth: aiu5-5Xp8Rmb9v6U9r7byFQiAEs
Message-ID: <CA+8g5KFXEyojnf-885Sj9GvgnZEAdKOUPD9Bi3DjwWV7=wf9oQ@mail.gmail.com>
Subject: maint: skip a check when en_US.UTF-8 collation rules are broken
To: Coreutils <coreutils@gnu.org>
Content-Type: multipart/mixed; boundary=94eb2c18fe84ba7c51053877b719
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:4003:c06::233
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Jul 2016 15:48:43 -0000

--94eb2c18fe84ba7c51053877b719
Content-Type: text/plain; charset=UTF-8

On OS X, *.UTF-8 locales use ASCII collating rules(!?):

  $ readlink /usr/share/locale/*.UTF-8/LC_COLLATE|sort -u
  ../la_LN.US-ASCII/LC_COLLATE

This means that sort, and any other program that relies on strcoll,
cannot be expected to work consistently on OS X in any UTF-8 locale.

I noticed this when sed's THANKS.in file sorted differently on OS X
than everywhere else. Here's a small C program to demonstrate the
problem. It prints -51 on OS X, yet 1 (indicating "J.b" is greater
than "Ja") on linux:

$ cat k.c
#include <string.h>
#include <stdio.h>
#include <locale.h>

int
main() {
  setlocale (LC_ALL, "");
  int d = strcoll("J.b", "Ja");
  printf ("%d\n", d);
  return 0;
}
$ gcc -Wall -W k.c && ./a.out
-51

The "-51" comes from OS X's computation of '.' - 'a'.

--94eb2c18fe84ba7c51053877b719
Content-Type: text/plain; charset=US-ASCII; name="sort-vs-OSX.diff"
Content-Disposition: attachment; filename="sort-vs-OSX.diff"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ir27j6yr0

RnJvbSBmMGYyMTViNGYyNjZiYzM4YzVhOWZjNDY3MjU4MDRiMzM4OWI1MTNmIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBKaW0gTWV5ZXJpbmcgPG1leWVyaW5nQGZiLmNvbT4KRGF0ZTog
TW9uLCAyNSBKdWwgMjAxNiAwODozNzoyOCAtMDcwMApTdWJqZWN0OiBbUEFUQ0hdIG1haW50OiBz
a2lwIGEgY2hlY2sgd2hlbiBlbl9VUy5VVEYtOCBjb2xsYXRpb24gcnVsZXMgYXJlCiBicm9rZW4K
CiogY2ZnLm1rIChzY19USEFOS1NfaW5fc29ydGVkKTogVGhpcyBjaGVjayB3b3VsZCBmYWlsIG9u
IHN5c3RlbXMKZm9yIHdoaWNoICIuIiBpcyBub3QgaWdub3JlZC4gIEFkZCBhIHF1aWNrIHNvcnQt
YmFzZWQgY2hlY2sgZm9yCnRoYXQgZXJyb3IsIGFuZCBza2lwIHRoZSBjaGVjayBvbiBhbnkgYnJv
a2VuIHN5c3RlbS4KLS0tCiBjZmcubWsgfCAyNSArKysrKysrKysrKysrKysrLS0tLS0tLS0tCiAx
IGZpbGUgY2hhbmdlZCwgMTYgaW5zZXJ0aW9ucygrKSwgOSBkZWxldGlvbnMoLSkKCmRpZmYgLS1n
aXQgYS9jZmcubWsgYi9jZmcubWsKaW5kZXggYzkxN2M4Ny4uMDhkOWU5ZCAxMDA2NDQKLS0tIGEv
Y2ZnLm1rCisrKyBiL2NmZy5tawpAQCAtNzIyLDE1ICs3MjIsMjIgQEAgc2NfVEhBTktTX2luX2R1
cGxpY2F0ZXM6CiAJICAgICYmIHsgZWNobyAnJChNRSk6IHJlbW92ZSB0aGUgYWJvdmUgbmFtZXMg
ZnJvbSBUSEFOS1MuaW4nCVwKIAkJICAxPiYyOyBleGl0IDE7IH0gfHwgOgoKLSMgRW5zdXJlIHRo
ZSBjb250cmlidXRvciBsaXN0IHN0YXlzIHNvcnRlZC4gIFVzZSBvdXIgc29ydCBhcyBvdGhlcgot
IyBpbXBsZW1lbnRhdGlvbnMgbWF5IHJlc3VsdCBpbiBhIGRpZmZlcmVudCBvcmRlci4KLXNjX1RI
QU5LU19pbl9zb3J0ZWQ6ICBzcmMvc29ydAotCUBzZWQgJy9eJCQvLC9eJCQvIWQ7L14kJC9kJyAk
KHNyY2RpcikvVEhBTktTLmluID4gJEAuMTsJCVwKLQlMQ19BTEw9ZW5fVVMuVVRGLTggc3JjL3Nv
cnQgLWYgLWsxLDEgJEAuMSA+ICRALjIKLQlAZGlmZiAtdSAkQC4xICRALjI7IGRpZmY9JCQ/OwkJ
CQkJXAotCXJtIC1mICRALjEgJEAuMjsJCQkJCQlcCi0JdGVzdCAiJCRkaWZmIiA9IDAJCQkJCQlc
Ci0JICB8fCB7IGVjaG8gJyQoTUUpOiBUSEFOS1MuaW4gaXMgdW5zb3J0ZWQnIDE+JjI7IGV4aXQg
MTsgfQorIyBFbnN1cmUgdGhlIGNvbnRyaWJ1dG9yIGxpc3Qgc3RheXMgc29ydGVkLiAgSG93ZXZl
ciwgaWYgdGhlIHN5c3RlbSdzCisjIGVuX1VTLlVURi04IGxvY2FsZSBkYXRhIGlzIGVycm9uZW91
cywgZ2l2ZSBhIGRpYWdub3N0aWMgYW5kIHNraXAKKyMgdGhpcyB0ZXN0LiAgVGhpcyBhZmZlY3Rz
IE9TIFgsIHVwIHRvIGF0IGxlYXN0IDEwLjExLjYuCisjIFVzZSBvdXIgc29ydCBhcyBvdGhlciBp
bXBsZW1lbnRhdGlvbnMgbWF5IHJlc3VsdCBpbiBhIGRpZmZlcmVudCBvcmRlci4KK3NjX1RIQU5L
U19pbl9zb3J0ZWQ6CisJQHByaW50ZiAnYVxuLmJcbid8TENfQUxMPWVuX1VTLlVURi04IHNyYy9z
b3J0IC1jIDI+IC9kZXYvbnVsbAlcCisJICAmJiB7CQkJCQkJCQlcCisJICAgIHNlZCAnL14kJC8s
L14kJC8hZDsvXiQkL2QnICQoc3JjZGlyKS9USEFOS1MuaW4gPiAkQC4xICYmCVwKKwkgICAgTENf
QUxMPWVuX1VTLlVURi04IHNyYy9zb3J0IC1mIC1rMSwxICRALjEgPiAkQC4yICYmCQlcCisJICAg
IGRpZmYgLXUgJEAuMSAkQC4yOyBkaWZmPSQkPzsJCQkJXAorCSAgICBybSAtZiAkQC4xICRALjI7
CQkJCQkJXAorCSAgICB0ZXN0ICIkJGRpZmYiID0gMAkJCQkJCVwKKwkgICAgICB8fCB7IGVjaG8g
JyQoTUUpOiBUSEFOS1MuaW4gaXMgdW5zb3J0ZWQnIDE+JjI7IGV4aXQgMTsgfTsJXAorCSAgICB9
CQkJCQkJCQlcCisJICB8fCB7IGVjaG8gJyQoTUUpOiB0aGlzIHN5c3RlbSBoYXMgZXJyb25lb3Vz
IGxvY2FsZSBkYXRhOycJXAorCQkgICAgJ3NraXBwaW5nICRAJyAxPiYyOyB9CgogIyBMb29rIGZv
ciBkZXZlbG9wZXIgZGlhZ25vc3RpY3MgdGhhdCBhcmUgbWFya2VkIGZvciB0cmFuc2xhdGlvbi4K
ICMgVGhpcyB3b24ndCBmaW5kIGFueSBmb3Igd2hpY2ggZGV2bXNnJ3MgZm9ybWF0IHN0cmluZyBp
cyBvbiBhIHNlcGFyYXRlIGxpbmUuCi0tIAoyLjguMC1yYzIKCg==
--94eb2c18fe84ba7c51053877b719--


From MAILER-DAEMON Tue Jul 26 14:58:41 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bS7Yv-0004tP-M7
	for mharc-coreutils@gnu.org; Tue, 26 Jul 2016 14:58:41 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:51973)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bS7Yt-0004sH-IC
	for coreutils@gnu.org; Tue, 26 Jul 2016 14:58:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bS7Yp-0004qH-Hw
	for coreutils@gnu.org; Tue, 26 Jul 2016 14:58:39 -0400
Received: from mout.kundenserver.de ([212.227.17.13]:52383)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bS7Yp-0004pc-7V
	for coreutils@gnu.org; Tue, 26 Jul 2016 14:58:35 -0400
Received: from blackice.site ([84.175.95.115]) by mrelayeu.kundenserver.de
	(mreue103) with ESMTPSA (Nemesis) id 0M03PY-1bBZ782T9w-00uMIU;
	Tue, 26 Jul 2016 20:58:26 +0200
From: Bernhard Voelker <mail@bernhard-voelker.de>
To: coreutils@gnu.org
Cc: Rodrigues Goldwyn <rgoldwyn@suse.com>, Ludwig Nussel <lnussel@suse.com>,
	Bernhard Voelker <mail@bernhard-voelker.de>
Subject: [PATCH] maint: fix dependency of man/arch.1
Date: Tue, 26 Jul 2016 20:58:08 +0200
Message-Id: <1469559488-700-1-git-send-email-mail@bernhard-voelker.de>
X-Mailer: git-send-email 2.1.4
X-Provags-ID: V03:K0:fdBbsqtl8F0nD9EyI9a2lIv21/5tGwDLhwHDS/zrgrFNCkTwwse
	S9krTDOmDaOWYhQnSiJRiJMEXtuhTdgAVI83UCrqJ57MUuJPw+vzr3sBkHZYPBzbMwBPo0I
	UGyjc+ey6yDPDPjUauiVSTU6z9FgFkiFDbg/btoZXmCxZrYw/ecb/7aTScCdgZuX/E6K1Hc
	eXohmZ7ayz2KprnFe1m2Q==
X-UI-Out-Filterresults: notjunk:1;V01:K0:jfZCiUtNf1o=:ESLGfPUUIJdsnpnnoVo1O2
	ZI4ORBDBk2tJKcRUsEUNMNPuI0yNswJZGaFjlnsJMVMwQakKL9ReZKyAcW2dlSoXHgfuALafn
	zCspouOsWIrMs1EEvH0U2fTUAW73PupD1WWhiEhUtxFmjD/DugLVVyDtXD7crWtpw4gLazKgV
	x4LYs45SlFQrw/Qlsc1/OU07Tp6SVsPFEQE7x08s+mCpHaKmTjF5/5nE+/Tfm7mp0CW2g32fw
	mZRxX6nEDiaBg58qQAIGRj+xvHRndFfVQccWvuoz3lcbntz0Zq7GDu3AkCnduXGPNVJuy4DXU
	BaPBkrWBKIvbQmGIe/Si/07KU1fkTKKHpvGR46gVW3K/YpAtj/2Ogu004rGNEtQ11nRSqcAx5
	eD97Eics/stR5YaFTjdYG9AN7Sww6DRVdqU3bBip8dcN9ldDDN44cda9UHvi1bMQl6wBkxMH6
	Nu09yULpM/ju7Uxg9Tbz4oxp0aWgchMU7icbMKCJWbpvDQTWdNEZt0BF9IaUmPHArskmw1mY4
	jNGtsKTzzixguGMJGDpfJp45glyWdBGqLylc0NEDxcJpFU2oKbbv3Pm3j7CXi+Ff/6LAxFmR9
	QK0Bhb8Vlq39Mq0ws4Nu33jIFRcp31ePTXNHLDtDhqB6g0fGWgkZ741oOSlDsBaPQg352PnGb
	1wqc3y5hW5HaCmSSysKxy3vg2KueazxWLMkKNZgjMkqitK7NUw/p4DCz5MdVnR1i2a3I=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.17.13
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Jul 2016 18:58:40 -0000

The following is a reproducer for the wrong dependency:

  $ ./configure --enable-install-program=arch
  $ make
  $ rm -f src/arch man/arch.1
  $ make man/arch.1
    GEN      man/arch.1
  help2man: can't get `--help' info from man/arch.td/arch
  Try `--no-discard-stderr' if option outputs to stderr
  Makefile:14378: recipe for target 'man/arch.1' failed
  make: *** [man/arch.1] Error 127

* man/local.mk (man/arch.1): Change to depend on src/arch rather than
src/uname: while the arch binary depends on uname.c and uname-arch.c,
its man page depends on the arch binary.

Reported downstream by Rodrigues Goldwyn <rgoldwyn@suse.com> in
https://build.opensuse.org/request/show/415172
---
 man/local.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/man/local.mk b/man/local.mk
index f41d969..7b72d3b 100644
--- a/man/local.mk
+++ b/man/local.mk
@@ -56,10 +56,10 @@ if SINGLE_BINARY
 mandeps += src/coreutils$(EXEEXT)
 else
 # Most prog.1 man pages depend on src/prog.  List the exceptions:
-man/arch.1:      src/uname$(EXEEXT)
 man/install.1:   src/ginstall$(EXEEXT)
 man/test.1:      src/[$(EXEEXT)
 
+man/arch.1:      src/arch$(EXEEXT)
 man/base32.1:    src/base32$(EXEEXT)
 man/base64.1:    src/base64$(EXEEXT)
 man/basename.1:  src/basename$(EXEEXT)
-- 
2.1.4



From MAILER-DAEMON Tue Jul 26 22:48:10 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSEtG-0003Mj-Ey
	for mharc-coreutils@gnu.org; Tue, 26 Jul 2016 22:48:10 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40055)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bSEtC-0003KR-Ot
	for coreutils@gnu.org; Tue, 26 Jul 2016 22:48:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bSEt7-00025J-Jg
	for coreutils@gnu.org; Tue, 26 Jul 2016 22:48:06 -0400
Received: from mail-qk0-x231.google.com ([2607:f8b0:400d:c09::231]:35573)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bSEt7-00025B-Bc
	for coreutils@gnu.org; Tue, 26 Jul 2016 22:48:01 -0400
Received: by mail-qk0-x231.google.com with SMTP id s63so23378813qkb.2
	for <coreutils@gnu.org>; Tue, 26 Jul 2016 19:48:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc:message-id:references
	:to; bh=pB21L/4HPGjYjurzle4dy98nFxF029xK1QYinLwLLj0=;
	b=hl92ybfwD8QqF/JeOM8tenXcHjiVNnson5ZJzekrJSdA29A3DVN9uKiAXX9a1L1jWv
	9pIhFzOJ/afd/ffEBBk/2A+0ykb8ZazwANvR8UGO//oPsIy0Bep4Hc38FUs/gW4KsYzu
	m40qxG1REYOnpizqgv5VR3mSk0MExrg0XEEH5n1bSxfL/I/95qNfQjHTctIQpHamB/dp
	XV/v9ywZMyQA86I3pU5WdJrRuvZN29qH1YT+q5dlnP6Uu1tZH3GEnSPyWlDetg6hNJax
	SCIwyzZUWUuh0T+DYS77ly0TtFskm8SEYMSNEpPSElm2bFguKNdA7HjmdrhWLGg5i/VI
	lJOA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:message-id:references:to;
	bh=pB21L/4HPGjYjurzle4dy98nFxF029xK1QYinLwLLj0=;
	b=dG67KcRa7V+yDqNPK36yrIdYDQGnJrrCISa3pUjzCIAtnMS+A+QvId5k67Re2UYWWZ
	AFT+o0FCiTFHe5uVd7UhngMBzi8PLfNBj+ZBTF0CWkKj1IB8psSi88ifJ7SNDEL3cXgu
	azCt9W0rZRrl/sv/w6PwC8LPNCdRsfDnw1ekHFN5WwvRcZPmZ0YrilwqnPY0KespzPe/
	3cd+POQGFW0/kEivomOC5oLDv7MJHPUVbzz5TsrVZahs5ijEtRgxGukE53zJKKNKerNf
	Wvup6poT1jP4lgUzBBl5J90ErK+RONM4fWwj7YHg+ePJeroMgGFHW9zCsTQ1zVE7Cg7S
	tLFQ==
X-Gm-Message-State: AEkoousf9MpgkkGzLZxA1ePz5WIgnT/QtrmqWFhvSbfxqDi0wD0SD0+Onux8eXU+nYV3eQ==
X-Received: by 10.55.143.198 with SMTP id r189mr34080997qkd.1.1469587680853;
	Tue, 26 Jul 2016 19:48:00 -0700 (PDT)
Received: from ix.home (pool-100-38-105-55.nycmny.fios.verizon.net.
	[100.38.105.55])
	by smtp.gmail.com with ESMTPSA id x3sm2650704qkb.32.2016.07.26.19.47.59
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Tue, 26 Jul 2016 19:47:59 -0700 (PDT)
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_BEB00073-75F8-45DE-8EA4-B2F713F68EA3"
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: multibyte processing - handling invalid sequences (long)
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <5793D3CF.4010707@draigBrady.com>
Date: Tue, 26 Jul 2016 22:47:58 -0400
Cc: Eric Blake <eblake@redhat.com>,
 Coreutils <coreutils@gnu.org>
Message-Id: <24670EEA-3D00-43D1-B039-C345F827B1D2@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
	<654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
	<57934C33.30308@draigBrady.com>
	<D92CD138-0E60-46C4-B54C-96C44DA7784E@gmail.com>
	<5793D3CF.4010707@draigBrady.com>
To: =?utf-8?Q?P=C3=A1draig_Brady?= <P@draigBrady.com>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c09::231
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 02:48:09 -0000


--Apple-Mail=_BEB00073-75F8-45DE-8EA4-B2F713F68EA3
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8


> On Jul 23, 2016, at 16:30, P=C3=A1draig Brady <P@draigBrady.com> =
wrote:
>=20
> On 23/07/16 19:05, Assaf Gordon wrote:
>>=20
>>> On Jul 23, 2016, at 06:51, P=C3=A1draig Brady <P@draigBrady.com> =
wrote:
>>> I was wondering about the tool being line/record oriented.
>>>=20
>>> Disadvantages are:
>>> requires arbitrary large buffers for arbitrary long lines
>>> relatively slow in the presence of short/normal lines
>>> sensitive to the current stdio buffering mode
>>> requires -z option to support NUL termination
>>>=20
>>> Processing instead a block at a time avoid such issues.
>>> UTF-8 at least is self synchronising, so after reading a block
>>> you just have to look at the last 3 bytes to know
>>> how many to append to the start of the next block.

Attached is a partial, crude implementation of stream-based processing.
It currently only handles fixing invalid sequences, no unicode =
normalization yet.

It contains both implementation, to ease comparison (use "-S/--stream" =
to use the new implementation, or without to use the previous line-based =
implementation).

The main functions are (to facilitate discussion):
mbbuf_read - reads more data from the input, moves =
'incomplete/left-over' octets from previous read to the beginning of the =
buffer (somewhat like grep's fillbuf() but not as sophisticated).
STRM_unorm_buf - iterates over the octets in the current buffer
STRM_unorm_fd - repeatedly reads the file and calls STRM_unorm_buf.

The tests use both methods and the results are identical (except unicode =
normalization with is currently skipped for --stream).

Few issues are emerging:
1. If only validation is requires (i.e. no unicode normalization), it'll =
be wasteful to convert the input to wchar_t then back again. It'll be =
better to write the output as-is.  If unicode normalization is =
requested, then going through wchar_t and uniform's filter is needed. =
Perhaps two separate dedicated functions would be more efficient.

2. Regarding skipping STDIO buffering: I assume you referred to dealing =
with input. The code now uses file-descriptors and 'safe_read', thus =
bypassing stdio buffering on input. But it still uses stdio for output =
(this seems in line with tac, split, tr, etc.). If we want to bypass =
stdio as well, some extra code for internal buffering might be needed.

3. I believe that for this tool to be really useful, it should report =
the line number and column of offending/invalid octets. In that case, =
the code needs to count lines / columns, and will need to be aware of =
which line-terminator is used - meaning "-z" is still needed.
The attached code does count lines/columns (see struct mbbuffer), and =
thus is a bit cumbersome.
=20
Currently it seems this optimization leads to somewhat more complicated =
code.
Once I'll have the unicode normalization implemented we could compare =
speeds and see which method is preferred.

Comments very welcomed,
 - assaf



--Apple-Mail=_BEB00073-75F8-45DE-8EA4-B2F713F68EA3
Content-Disposition: attachment;
	filename*0=0001-unorm-a-new-program-to-fix-and-normalize-multibyte-f.patch.;
	filename*1=xz
Content-Type: application/octet-stream;
	name="0001-unorm-a-new-program-to-fix-and-normalize-multibyte-f.patch.xz"
Content-Transfer-Encoding: base64

/Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4Lr6MtFdACMcieb24AZFLOszufoyWbSGdpN+YDSEdwT8
d2ZxxNMl7M1CRun+04gtLlvRxMCUJLV+J0yfU59ToV3LWWsA05fMneLe7HUfs5Z2J3I29J3qObPn
2BKqC2xuy69qMqKXLTYkqWXAlvbP9KlVifzd8y/CtEg+3RgIlEe324sHd8QwM/yjZHgjAr+lV6SX
4559jgg/f0VqqdfYgh2ChHSOKPYt/3E4PjiPPiYyN9NuKPx0BtHi6XahObWlEkbfF13Yok8wwwNp
Icp8Nfw0TSvCKBxhVvXVTZblIkshS5n/WYqCoWLB5b325B6yUHaiOPD3dxjFHpwPMYcDzFqhCAXO
YcRkv85Ty/BW0YFnn+Kj1B9GTzfcnvWr5HzpRbn6R5Ef1vyInU6/27m6fEf+JVqYO8cfai4/F9G2
8jxUmZ8i8yENW9QiVnPb5LFuR+CDH8HObDH7fRjya0UfpxTzLKM3ZDh8I3H6BBN5FXmgmrevNMAC
GB802ndU4TLqcakePvxEijrv6SXGgPBYmDjEuCqDptLRauIOYYKrwlkzDFrs8+VH7iuGcA2x0gvB
W77IihTGy9wZg3xDZ9799I0MW3YxErENAXjqP1vhehKJxss6svXYyYirq6kLXht7TAmXHuUtgJ0o
EKJqHsyBMbU+UDigMSJAJY9pKRXr1K+1c4ikBzy+bPKmH5/B8nTr3gl5TWQiu6Jm54LmqElIC+pd
VFLag3vbB2ctwZFHOaSdSE+koY2AiiH+HeiAYSqepu0Mq0v7lXE/eBEe0anQEcLL63KTun8glhip
TFIGrmYaQo75MD0dbEeUQAhMa05xLLNBLi0toYKnzXLA2CVxb8vrVhxNTRhIx7tJKUVZYwBAnqdU
fx5pcfcgEOIz3DrCTq8pniQgJpnzM9hJleDaX8ubS8LsVTIcmII+oWLS/gVCr62UtpwcWLA3uJfw
UWCnYnZqc7x6K/sQ2c52REulTTvTKx3WPyP2+AX/qKAkfvV8vqknGnl41VwD6OolkGKXuxAP+WCZ
b3eAf9Twqj476eSQFq6MC6p+2Q4nvjy8xo9zivJI8P2gbyUyFFT/6czbgnTMlVAtUxycn4mxhnDr
kAlX4cagNB0YvgYgbeRvmtGm1O2mJPERYPkRiUn3wu8fmlqI5Zq8BXb8iddaifc5DDxLUCbvTWB6
GRSVYnfND+oVEtRjlW/9xXEN7wTh1ziydTQZksJRYf2XO0oB18YdFLEn+5+cqBc54rqBJ7FAhJv1
V9GgG1kItiyHTbqd8ZvC5sz2G1Rlv1zAllitoeDhzCjefkolSfAu2DMchuNnlasPM2p9uXFwagAB
20YEOdUCyEo8Lq+jG+sxUtsz2CufmcRQn7XS+TV83PgF8EoKmFagvjizJGAnuiG8gn+hb0/4tccl
rKTswTG1cJ9dz8fKVmEG5FHXIBVHSmiPVlxEGygov2tQVLnJYUyB8SkkV3OTpBzH2FVNp9yEDFdG
SIuJWf26T1gsIb22uV+i36BicFO6NRoieNZbWUY4vEsCARZZ8sRkz/HoyyQWzusZOsDvynPyNrei
q7CFZpUe00EDGaqRUGs32vjprOJJfL9n+49UzCArBw+weNrwnJMBURtYB+yLy2iVxwAIzeEq+9s/
4a9/Bm0F/069msMvqk4ZP7A6eddR2I6+flCRNYlw+cfvMz5+hWU5r7AdC3va/ztGRCOSrIGr3cbZ
ajbs4i+XdcIlvtoqb2hU0gRlnPPLn1Sw5x462P0YxadoSeoV+1oqHunhVS3Yzi/p2riuZ02CbSgf
0ZzMdGYml7Xm0zdSZJlaAQiqXoQ7xP7iaWYEJh+p1HcsZIIaU90s9s5Ar1H9ZDNdbB00iQ3hdyiK
Pm7cURZR+NbOrVHR/Eik0uWQQOimzCNGh0R8uKdToT5Gbfc+sM5DQtPxTPwxL8rnN3I32aDiypFZ
k+ZVaW4HMeWe98i1fJI2pUEQuJGTquo/F7APjf3fcrAn0RdwH9RIcqwR3KoUSDkvl7f4N5DhLcyX
Wd7mbroe7rIr2ZeYiLsHL79wKksb+K7+990Paxx6LxUxHNalgZ9X7VFBAvJ0KAwIimXIbFgnFRqB
XgxGNl2jvQy00vLyHipiyZ95OgZreg0tUjWVW/VaLi6J+x8eNohKxbbBz4gDgF5qVx2vgqDFgOer
9OLi9bRdh+hjcXKYoaso9iSqFWiTJCfft1oJxYDnQuvVt8h0/Y2o0E60FdKt9XG05KK/VNnZgxI6
0jnS0DsThjqGvv/T4ImxucBhf8i7donlzh1bvyRxt7e54QenVdKCxqtYzi7U+4yjgjbQS/5CtSNr
GN9x7seoQiGq0M4bXdnrNgZ0GPzG547LVhLadj8vdLXGL6UcOlLceNGnUNmIhsaEWcfIJJq3q75m
DhxnvWudBL/8ENAMEG34GhHYE708ueuvaxaXoz30pJToWqiRkU46aSSrp4pBgb0DM9cUTnMO5bkJ
6SUaec3+7hEv6cQPonUKh4JOtrxG45TVRbqHBZhvJyUKfi7GzqGh5hoaPfsxq/Q/7fJ711+zbusU
raqKYNhvQGqxI6K34c+QRyiQBuylEnFQG16jHDFQhGdtTZ5si3SG4wMd8x1HTl/JG5vRezw5Qn0v
i8GD3WT+/xWxntr+U1Cy8tJ5xfS/Ny1javoqeYmEup81LmhiTUH9CLC4kX0LFEi5ze9oQtSi7LlN
mIu43x5i+uAToCBMIs2+99u2wCocrv4gldECUTkZQird1LEf9cBC/5wxuIwjPSh5uFuHcDZireuB
dO8aV1BIpSM2OB5UXXvhwn+MD0SNeRmVLdhIaVjdLXdQtdUUWdiBFYA/dXZt/e0xvxid9u3UNN9b
wjPlu3vXKeQj5nM8U2bP4fAOOAwF4Hiam5h1/pWEoFg5c2nzqFoChoIofGleb9n0re0DBC62H6Nd
72oDH3NNJy6HQfFA3KZitZ+QdkTsANSRlBV9PHtH/i+KAqvdk2N9/ubkYf4rMwI37k8sooeZ/tbj
agPJUV8DRN9x3E8i8rdRwpyOQumHg9eXBwcTiqWKxOuaugwLPP2sE+L8cg03hNkT6N2J1fGg4qgZ
pAV6dMTTcQIQBEdTt3KAdy7yz69GfKb5vRhkuJpjPakKhn7bXLJdktqCoWFya3mCOuZfECRZM+u9
54aarEG7w1P7teRfeUzNuajBadmNnBlgL9wmV44+UBSseNlxgixW4HK6n+gD04Qv5VLqLBdxl44P
AY7Ynkz3EvU0BzXT7T8rTf55Vdn8p8tnuyLUsVpXWeEJacUMRqlMExkCa2ObdEbva6mtr1KkG5q6
3nMY31ciCdgJNcrMD8cqsvhD/hy5fMc8iSlKYF6UIA1hlIy4O6BOvFjujtky+ioklVpTNbfzZkEf
BHR284JeVCXZJXclClbQdnKAFTmdFm5qq/n98XwS/s1IXV7zEJlwWT+mt1jw31UU2eF9Pmm4c6Yt
tpiV3yTV515zEs3Ghf4KEFcA97/UlV5fQ1ZcW/JVV5vef2CjwRZbUJE4WhbstAzA5a4L1/O93+4K
6ZlRHjtotOoyjBUB7KADTW4q8OmTV/0nQ+vp86TG4BIT7nY/nBiL9KL53HV8aBu3XAjTCCSutv4o
OhwZ3XQ24hGaBcXVGtPfkAddKc3i6F8ViG6bpC9qbUDJJ890IEfC0wm9l0RTPIGJOod5WvnQ7j1Q
tZMns2b6L+ovZoxCfTE9n2Hd55XFW2C0TTJz0j0UzQ2mERu/uY1D7nTavvLT/XEGZ9J8WrVhoJIt
rqcMiDJdwRLNrAoMZXhnjnJsdyUwWTvn5E5JPf4XIns6gG7ISLitOOXhpG4seRCMo4+dFn9HuGNQ
h3Z3Yj5DCPnhjoWB3w5nbcu6KLCKx0hoTcnCLdwiStFRBjPw9NmzAhLg9FQK56Udyq9YLvZScUbW
LAMu6K1oFfGKPlEr8u2hQxwDh7PcloojUC+/bEvq7WLOS/+6gSHl0i5whspwwbMFOU5k+kYrxlku
RPHHfOQkH/in56rRxIyUkYMLnsN4RfxmfIZthAl8P+NobqtyHz8m1Ubzqy/cnFakjJipNlxqSh0y
c5/bHe4wHZT6ZiiapQNo78AjAncyb++R4TEhGnE7nS93t/ohABc0LV49PKgBuLia7cYCjaI+JmgE
CnChP8CiCNl5BKjpuxWjKAHGPeKcatsdtY6mbLKt7+iUwt3U4v++qpY9MDUoWQF2tapgwueU9RzA
rL5UG9OK93wVzh4VFWMkHmqrwZf8L80bVuYwKM98gLdkxPNGtIdNqfKvM89SONVXvbDFwIEiCFu0
tu3HqWkE5M2/qbjcTWD++zPaEI8x0OpEbt8Q8M3KPnyx/XcUd7t6rYRDuc5H4NsBZUOHlTGZNigK
M1rgwgC5iaPny9A1Nfy3Ze4tohVzHtQm/iHD+Z+VEu88AplfMY72ZcqJe8X9ueh49TxGGSLEh9ZE
dPP6BTl0n5RsjY0SSvK4Ji8znERC5OA7o8jqqCJ+Uk56lGNFKhgseb3AnTC0RAIgVA17PYrpTZdd
IMeBMy4vQzuGHvAhBmT/sEkFVKbPYyYzGYyKmVyq8KiN1gOKZOb3He/lTN8k9AqtUvF/RGlXRRoA
cD7lZakPSFLNbkALBsBL7x6B6jSQue/WV6VSSwVAZYqD7xfirENoNkEEXot76lFBCj+GI7MxDu/Z
qGhW7osG3CVA/YJrI9NxqAT3MA4DFJSi7VAUnfELpqjFZ4K9zhZWF0P2Ax4cQVgV/6cUh46XQLo8
3ln/zxyFpW9Bm1hy5lZk4RsWzXgQ87rYvsygcqtWM2BUCdMM0DAOtFcWia7siyXxW+EPWxvhMV0z
y5DZ/U1i1v8XlG6mvVZYAyRbGMYp9Dh2AgKAtObxG57HCduCy0PLqU6qjKq5eLvEAnID89D83iIv
merzKwVfrC2cOzz1xFLaBBIiuNeT7/3LQnyL6O0Nl/Bfupwlfef1OnPTjgHOMBdjqnj1ZFPmNGFd
aXDCazgYlwKQNSOQTC57Z256Uew4KyZ4fU1mgfY+UY6Oq6ok8JNsCTQQFpWHhsgeiAM/x3Pbcdfc
+oDgokt5N9bxoWyWrI7JtfJyyPjvWYD2s70nToOUXYEW1lJTDDcX5JG6YgIwvkO6nilPngzHVA9E
mKuIR4M6cw07KsNqgxkdt60InYDE3SggS4mUCRJxvjBnI+szWssyF3qByq9zY+gK/7p2lHt9f+nS
+xoxkLZFHQ1v75jJDzw13ry/Or7adNWjgKnwpnnqOa6CeaXl2uz8IeeIb/21NGHFYWVAyC1MqxMI
wBCE+mI37b5DTGnonR/7mGXh+ytNN5AqpI5WRAFjen9h2tZlVrcTIYK7I6Pp7/KkBuqM22jYWQKe
iLK0VCMXPQjqpp22dcmjOPx/GH0lhhAOEQZuVbLq0fFMuFHoGuBYILD/DaZJc3TN6RSIuJE27Aoy
FYDaumoBI3amCV3El3yXWQR390KB5gHxkQYSuQz1khiy39TRH4Tl+yN2VZV9oUOgprW41AsTqtad
afI1pQkEx9N4r8HW62DQo2X0I1i25m2uJZsNRIqGePxB9/AgO0/Tx1ZoLKvw4IYuwFLZL/rfeve1
L4QxNgZMI77DVyiuq7uizFzmgqfTQmVIseQ9ZTz+cHHN9UR9EreWdOkKub/3h6M+RITXy2dpOx0H
h3WqS+ZXNV+CsGxWZSP1jRP0IlfUy1Ab+s1+rLObfNc0UnH1ZjTmAItoYaGSvbicmWsR1mWv7wIb
ABKbnzwmODakBHigolQ1niVnBbMoQMvbG+iLh0LGh67FwkQTbp+h2kSICQU5V09RU7SCGxJJ71hi
DdXoC4QgfhqoyIx3a9wsjjBbAdo9QgXR4CEB2o7eeu6+yZsEULFjioLCVB+x1SUJaOGIxXdndFTQ
gil8EiUfZO1VIiBfoq7V/zFxGukvyrFrGekOeXCa5p/MOPH/f7Oyudih0dWiJ5BRwRLv+2E/3GOt
cdEkKhhfRj6ZHbi1jpLCS/0TRkMOlARKI2et0YgC/WYDS04h8YHp7xnCrH1kzsqFNMzKYF6/wBLb
8Kl0PVOrEMDja4NMDPPO36izBUmowW1n61d46A6Yq497ik1IcEpv+Qbgd6+Pl/mqflYJHnC9JUkT
74MdpuO0VxrTA3BTl/ILRv/BpUgOsP+UgMGMcpWOCutLqTJTCHAqpiPnfAttGvN+FwOaAx8E/F38
hnxXLvmY6a3yn3xdoIqm3VQRR8mn5L9sEbB/s1J79zUi/MOBqelYOb7YzywsZU/rCIs2vmM6VKYa
pD+rRZjBv8+BTcZzOOfiyHgh5QSiK4Had1xZQTPr3jYSF6q4nS+/kLIst4mD+oczW0acWkKNRkKk
0ImjfoaErl7IcgnPlcLVH+vf/drPH/Nqfn7bSf+13kCQ/AqP/x1OuKMeJMyN5s4C/FQFQoQNBvXz
RyN96ejhkKjl2fu3A9Ww3XaW37RTGQLKbzoDLDzuk6H0UX4a8i6A23l8b4/BAJdJUqDyoGHUfeJs
/h7jqe2WJP/C9rgiS872aJoePtT+zQspkKTycsIri6GR32/1+ksuu/LOJIEakLRE/u6l0E+jMxiQ
oPt3apYbPEzSW/eJ5tRr1gnja0q6MEVh1Yf8Hklxtk5PueyfJ7mJahsKqEtNEF3vOBHpXVW7VXLH
KbcB2cv4Hx7njA/tg3//Y/+9fSL+OuzxWewxkNtJwyvX7Cx/3NYgk95V2v0DCEXPNhJidnwNhMpV
c2ZAWxANZFsmqp3lMP67NJNqfCcKC9xtK1+WCAYbI4uli9a7zYgeHkEMr6rwjkynNkSD6EfKvaC8
vlzussRnsRLlNotAM4bnWOKW2N2vwSaJ2+ZYtrhRCZjSFg0pZ/eMUBMICPdWV6TCOM5Tc+Vr4udl
Be694s4T/CACCmQpp3QRg36w5MQCyLJLor1E31o8ExUZGiUTpLm3Lnvnqk2BBs8nIzzDxt8BuBJg
8KMZ90E407jGZkTEnTOtu8B6Vy4ershJTGWcZNqYooklR1v95jZGDvPrAwEuD7WyrdtM19yW0S05
7gaxTwBWgux/Iot9hgWSBIi5OemL6G2NQ7Wv3a8uDAWQPTAVh+XGXFWoc3USU/eEa+KsUR+YCvsk
/VwbuZ4Tipm3dHHxDYwej1dbIiF5PYXqYI/lIdoJN6zsro00eY4ekmqkq/Z7OFtZG2Ycv3A1tgj1
BllxPsxOksPOOaDKpIl4FV8Cw9U+r76GJBcggqsGVzsXd83nspsVuEqop2arPA5cNuBw3av6lnR1
ijH1SEDpxUGyX6QPU53m+mUPmAEFj6wgqRspDZUVtx5J9Y8Vy3lhjLxA5TWy1LtzCba4JFKOR3oy
HXE9vLRDuHbSh6qRsqBoJgscq4CK/v2hOzstvKIurUC79nr//bhAux0yDbYbvJ2QQCHLV9qBHL5F
EEjQtsWOQRZEamkvkkGIAICQn0TmEEVPrDiBKv6C1yZXvlft4zZHbi6dRGFRvf0jh6Tf/i5XJqBU
qPD/+4MHibc5dXFyQGKEytGxPRQTG3jLwrdKGqaf/54u5VUe8UIydg8W0NhESoxayVBPbQFSDkYR
cTmPAvNBdzccLgOhHUC87CMdSd9zxCm8PuRGitmDiAeqYG/zXm9LSr9NcqskYi5SotGdzyEEvVUt
e2MqomsGq4VLDDBIUIeC2HGkf3eh7bmQj8S5Mv633qMFGL1BZjAsHP2sKjK32QoKnmPLlcyl5gYW
cVpW/h7ysNRAxk1tRpDXGxMLXPUpMtdI7zUsgmqMcIp21du1fKCI5zVU6b3V1PtOL+nn8EYdr+Wa
L/AFZST0OVjUidZ02SlqiFu2iI5XpgkioNyB/0CHiwnEMqpSJvzFhysWMZDXDMYzZnZnLfeukGmn
7gBDFKgdF+QkuxuH3ISU9KQleYYFZaAGj/jaD1yQntFMqZxFUOPoGSt8AmeWMxFRfSW2g9UYkd1C
Na8SAOAuRfnNGRFDmon71LSnryPw7+lIpq/ASjiSKtgivfLs5Laig33BTRqKvqa9sqRRLAqG0x3Q
Tdf5XSBHdgV0NudtO+VtGBvJ/Pcv9VbNDxJksJxXXoPHbf2kRPLgLM8Y0eztin0bdQdIhU7Aaj7W
/lmXUpnNdLnkUrgDAXlS+8PExtACjM+F1pPGZ1eN4eR7Tih5EHvtEHtAy2DoNXLTOg8emn90N01S
LecS02d1O6i6gff//ByW4fiK12fIhqhTxw7XIbyPkfEfa8erySi7Sl6mQekF9b5gyRqdrWgKQhTq
LHsMLiYdXBt3IlvD3Snq4MMcwueYjiwbZhzVs6QVl85cEKY8ZHPdbBbkoZJUuAkgBsS1jbrrLyrC
1UPah0B6da8oZ1B5vZlz2Rb9FfbjSNHfX2DeOOrv+Wk04Q31W68aa6EzHaAtIUsRJuTLBqknGRf5
rASbnaoMEMeFe9yJ76Q8EkIOMCQ18JRyvQ9obvFpPHPChJ/6atXVVeM3joDrUZEugGk1dknRbMx/
h+0A6ricGZkmDI3fBBWvl7QoyGXIk6SjEqiSOuCXQk3qlWMCxxPqIQe9OQ+4leK31OmNxVHWOB8H
FDJVsF0/JkLazrdMQp36jTaRxbOnYipXiD7QcNXynNZCOOIXSPbE68fNef9VEkzWcHrRZ/R18OVz
e4cFkYFwBzW5IaqSdmDa49dzLWKSTdEFTYFDhzZC/aJhgW3OKRIbGjeswbib7X7D5A3BZ/Ty5T4A
vj2WrCkGNVSb18wUWPgLIc+TobGFHe/yhCctVQBFuKD0jn1hzhH+iQsVbmORiOf9bzN0CGhpytOr
kc3wBSbw2BNQnipb5fknAwNxxo6rFW/XCWEQgjPR9x3tSeiOFKwCS3vr51FE4B+jgwUwSEDPLClN
PGR+8Y4EYA/4cgGci3qV0wumN/Ee7YHHmbvnpO78oIN7SxR4DESKQJygTvsZdp02HkDU/pw8552e
XydnAtExS19Y52Sh4fLKrJpIQyT3QrnupcK/lve60BnEAYsEda5T1rbXjF06kjzLG16k/diWSu3N
VMBCPWwuzVt4BDIX6dbgmSMjbn4BeAWGHz/lHs/ZbxRTgbnbRNrbBwdYzJhB/7kWcIpkkJ/+gDtX
lRtR2YOgfFNk5Cj4QTgDQ+ouSQecRzueFmke0OFkSSKCNBU+Uh6brJ6hCiVaPbA10CGtkZDi2LEG
WWIa8Rjfhn41yqiIEhCf3dWovql6u/kzhluc33VeiHh4ycNhj2cbQ6AwvNQ7JU1URrJH6kjVUVzX
lVQm/oiBoYJmeLVGrtnFhmj/L9KXHnC9da26dQBjjPgMBjoGqzbu4HC2l8yNgdXXVe4Zx06+gflN
68N6nT1X8Wg6JqadBW3wl2gJjm/DY55E1UvYkM3utg7uHOxY1Fp/6a5MFEnEkR+vlZSBsYxJvK6F
8XkFJjGjLkREV3Y7xbnlGbuaK/oVBbTjPh4V/eOYwzSsFsYF2pOiAt8KVADKSQUPWxBkbpTKww3m
kExYWBB9wolAE/PkbSUv071BTtPClhNm2i+3jE9PYxp0pYUPHAsWfjfsSGIJjInYe1HokUmLuaFT
L3O4alsqZ9K8njze8bDshYkzcymKaH6oghg5ByrB3QNodWcAPQWVnAB+dy2ZwtfeUCv9PlCOQfXD
ltHb6AT675SrXBTuXHsWq0jiUO8C1FkZoNZ447/UITDecvU/Yj1eZSzthizbsSbVPknhXwvgg5hL
iO6DzsmlOb+DPPaNr9rtDoLsJC6K1Q2Hob4Qkf9kBNbOemFAtO5fORnzE6JLqd8eL0+LlTQSl1TT
AwYy0Vuz015TxQxw5LMTQQgCuteHFRihogQ98pqBxBwshEjAkpc/ntUmpgFZRhTnD8SzYlCcJ/VM
0JXq/pIjLTY4ZW0FSDC/r8P1J/G7kRfSAbW9kQkRyUrmapG+Gu+b4nAKoE1UNxFo2mJAoSygDRa8
nkTV/Pl+4973vySjH2v6z8/auRZTyV0ziwb3KB5TNFLcoeBWaXcZafE/gBuAhlwBOBdbqUL53KH3
9g4MEZR4bdyLR15vziz/SyCcg2bhEH4NIrnK41r7vyPIMte9tmJwBj3Vv1UdzT7SdWNzPQHn6Lmj
NW5+K6ghgZw9NwD3ez2qKxK3/mUPCM3vaZzegYIIiCVIu7IpoR8h7FooJoPpuQxasM49cJ4t/sjh
o9DcxjF72s5hgIeuhGyHOMICYgf3clNvByAlh+Qj5YUEzwCl4nQ2QIzkpumkG4gavBGbG8hLSnbg
tXcpfgdvFqAkDvOhOA5lZpRjd3Hq2ZR3aSfywo+Ek1uQfeXMJpKTsHkvDUFKjuAuIdYQmTBbvgdk
Pp2QFBdhcJeVxgZ3VoIOhve5SLOiossXOfH7Xckiwv1PbVVoNgHGVmBnVwVE/UC9Z6ZwBCpfayEp
GE9ZQUtH/Uh7+VnyK2dOr8r98F6XEsTRUtX3zYgm6vm5tGEXhf4txIcApMVJXeVkExWxQ5qEBCAl
/Uctez47og8EFAv63YCp4K5vjnr1pq3lw4P6lZQDyJBMYaORBSDoI4inlvjfXrsdEjRa+AYyLyv1
E5+ozOz6n0vCAlc+pCQoUI4bVfcQxo5m2OIUgGkdAQ3o3exTI2A38DR3kfbM5/CtuHuDyUeIlRvw
UXKwyBZWODtGfmxv/xDywnIgvAQCMFnK5zd5Vij6jxtMbKbmHtq4evLiFyAnuz0zodkKPphYdnYA
GwHrkVVsY7KWmPAMfkz8f0y1PuYV1DKt7ZmUbo9cU7iM5jrTaQ1WJdY6+L63kPQyviV2c2oDf4iH
9/70Duzdx6Uuw1/KfdD3+NUU3foT4oGJ/vX0pnlf34E0ptwQMeSiPBRu1+6lOYoITSf0EzB3w3Rv
aW0Idk/+tkcH63ngMk+QuIbYrpvia0odES751ubEDlDtIehNhy1N/m+6luTnUJUbuMyYKYaJ9qc6
XxdfG6vrhcsGXn9E03YkLmyVkEA7tPoW0qQ1czWUbongTnJsKFrK/gT6jkt5OBiTd7eE6jLfMch8
uDWKutk7ZAjXMFpaotM+1JyymZKb/MRj9oVOMqSjaJQpKccxZJho391rcCCTdb+jCraoi66gDKiP
flbCY9J5fSlugRr/gm+9mlspbt0HARI8h1CQ+heVINXSZWoO8Z8X24VuYFCB8z4eKG/88eH+Kblk
Vx7uhO2X/h0LDRyXuiswFZtvJed4NFNsaYfv4WYl+VbL/qSjBX2G7+pGdrFfsb5zUYyn4BS3B16l
/qAXgOJ7Dp89TLm6FSQWteh2wZ1euqL3ATnjULK5j9HKgDscOyY87qm2r7uQFq7Qca8l2NZmPNEO
4z909L03jBfhnFQqbvBX/EGtc1JKhvH04Tvyf0EgWUYfLP/gbEi0p1lNjbBN3oDdeW+24yVyWpQm
I9q93/SRlH3s511dZ9vsZ3+PRAT5Dmu6mkNSFh4alFkxkFcUYWqPXhyavMtWkkvAAt+Z1xb/nEf6
0AjvMK8iTsBNnrzkGf/ut/t0DXaHqy6DsVNLkqMEVJ7L0ayqbceLVFOODmFIfXF0NG02RsCFaFip
gWscXemfNhZT55ZT6Z1jsP1rJgRW7tHuV2l50Ju+jV8sJqEt34DMhekpvSVGHXT78ZJoBYQ31fLG
kFgWUh2+jQAkeR4I+tTalaiZkiGIQdvYIH8UUOD++DLg+LSHv0pMGF/vaBfmxzssQBW6IPby8/Vb
iinTSoHcdFlW1g7Hs5CaoYKmj75ZNdf3Ds2xNCPJawpsrlmgLpzUNX2lrJlWlpAmW0KtvAy8Scfs
i2cK45WgyC8IIx36TAgSw82BEw6o4S6FfxKqo9275PXxIHZBqB3KouL0Uc2u20d7MLm0qBN1iV1z
CsoO1XFs5nb/3NDIBEn+pEvVzqVsFfpO2k85opQgRR4djGpYURaJwvudvJb91rR5wACnp6Ix3bCL
nKQ+CQKmN8/2GKFXrbbIYUJ5aMfpZuDA/YPDjExKSQsgOGN3kauTfeLAp213GF9ay9u0SnytjnVA
ybw0f8ADhnyb4PMsYBE1/MB51JX7K5Haa8ASHrAHLdZQ3HGSUWKM431AGf3uATH2BtENeAi1v0g2
3P7Tvb/dh8TDKCPY2AXkSnQaUp/40fE/Q8VzyeYJZ506ItjwdU+vbrekhE9UArPNJZhz2ENMPWVN
0x0kei6PXa+tfjsTIKP33xwAx2vAU1iJ3y2rlfNONhAmW1emwhjeFl1+zDO4hQBijxXgcWqbO1pv
KFRlfy6Kcpj+OBktqb3T28ERqEdUwTWaR+MvbVJ8lMlPuXKI9gZpSJeMzB/Hw5fQPTA1HddwvW7b
PWYxW08sEwZAKU0szy6CJEPCEvRAp5UXM6L5Az40d/Ep9G1iHy3C1kz2MXQaCeAT0ffRgvfdlhYI
zxnEUBhk4Mlff8TZGSomR8qki2RH8EvlF9eWNbt/Oze/q1b1jZscIQqGmWfJ+dDGNZj6GQC1QTkg
vtQet3NpYJzxOCDLfASOfXcBDwMYeHanFA6B/FnvKb9j6+60SxNw2lNaa6nsmTJkTtgIVTwTY/9Y
65HxexOQMrtt/JXfRRdV8zdal38GSrhQwflNyTT/Aso/tsxLyYEk2wS0rrF6N+oL5i0MSFVTTflU
L2mkFc8L48Sk4GsC7VSddye0gM2mt3P3K7qsj73QHqlyMA+J16pk32wgmttSu9rj88gPWbEoRcrT
h9qmcqMyrTwJ5dhGSHP8U9KCNi+eQWJINStjrkBDCZuHPUwBaOty+pqLHDL8I6Tzr4kqJw0RggCM
ljUJqir3Ryq0/38kNkMM3rCsc/sj1W6gNmdHCAZie05Fe9/z/jO5oM3//9RddaIbQeeFpFSquAX1
a1HyN3ykP1/seP8iHACKXfURK4QAorQ9jYhOZ4ebdoWs9e0ScIAjcKsBf17hZitwi/SiYWWjgepX
Mb3Tzm5UbgXbOp1IQpx8vad7wdPTxE04mh6xlUUXthVTSV6Nf5d0Xsp4VdvVy7avTEfHltS51b/B
Y9W6GadQerPKq9+QNSuo+yVS06Uhed33tHkt/y/R1av30xdIuKFUoMOQdcQiOYw31UmuBFqzNmIR
xUSDS6ejsIH8gD9eh+DvEtqWD+zfHMekO98GHGZ5NwFTQl8k6TvRSR2X4VUoYC8jieuPl4Z69i6h
gX2bKyhaWw71i+6f7qCB341GQvH37w8f60Jedh1vjnPlAT477LMUSJ9CdjXvhWwlVZd2D/9Xv/k9
k6K0nrftPUAMA8xYfHv5kJnzdbFXazaTdbDo2fcHKNc8pQ3nHOlg6BDu5O5RICcCPrkoRWNPWHro
BO+Lw6/IdkWJQyK5gt5FHCQtvtrEAtxcsG5YAKlKgEDICCJbDdqrf3jFlUuAx9BKLNAMG1QENgMK
sQeVLwxWoNSs4GrFfGQbtEl5AM0XiTzB9OfCbJt2ieC3lw050VBIfKxtYx5/HYk1bFRzJoufb/KK
NldH+cIdx588EfyM3nHz+Om4b+5/CBnyoLjN9OviQU9vUWIUFcEvYNZgZBs7nc9dHwWrHVOCRgtx
8pKQk80ORjCzAXADwn/bB/T3hBlK6rPA1H3fIc27Mkr6R77jGFnG/pfS00qAoyKVCEKE+GLgtIk7
NHfTE6/wTWtYv5X8OkvZRcPER9GXMYQQnQd5T4FHTaziA/FBzeZVpDHcL4UAPxSEnYsCpN/F+Ybw
E/dbt4CgbhyKjAn8RsOBY4fBFrFMvlxmdjZ9PXPR78NzzucJmFO6zzbkfsP6U8448vCGTX6l06ZP
vpO26jT+xBkyqk8XOIH/LmiEL4uhywdu5C27XtrpmPNNdxNgBfw34CRSeCA1TDfoLqhxLj3//wT7
J3rRrUPn+BN5HHNAH+2OgdMg+iIUfAbqkYyvpP6XpOpuXeT1TbA+NJ2hnrXjbeBJTN6FtkOG8RSZ
jp1ejCaU1I0li6MaxVLur6Ju+PrJFgh6ynKj6NPsJdLk6kqpuI0nl1MRe/BamWIAFAgzRlK77f7z
UdcRTOUbAtbDn3OZjUsw1GxsmWJHwSAj3cj0XaBVUmslopDjWpadLBr/A53PGRoh90eJz1Ir2Sf0
eLG1otwmbBYFdf5BZGJg08V67UtbFvKSd0ovfsJ9I0xlqPTpe4SnHGC1/PtP4i/qd3WuqvF5mPsv
Kt7jmB6UZDbx4UULKbR260L9BvOI/gB5XP8De6ZFEjvNgafQuoM8cM5ICKlPOFkGXtMLUd9b8kkA
+1lmDMB/mHRKagpT2BHnfMlMEMsQbGPY8vkwCWWCR9awU3oSybo2lggqCcPIWJh6D57Bp5Q1hUgm
ynMvAuI4wcR2pjMB9IdftL0BxrAgkFhfs6DMokiVcNYKKYGigM4zoSniIstP29fo/1Rq+S90dqgl
kj7MWZGrlQzf5Z7SS+rt8Zme5oJ/+ja+o+oSSifOMzD6VRHmGsh0i6jd/VZVDj4uKGw1A1thp7os
qBGUqIBi057aUpaQd2FY3EqCqlYPlHViuBuEJYguWiV59QWYSvTBMmFmcWFzB8AJ9kRbnmBU1ILT
F2IgghS74zOJcD9YGSqqvnZlhyJg6ePPifah8X7IVFsCa8b6svf7QLWNFo/jyLkVIw9ToCFxMr8L
kWQDL+g+dbOD9oz8mynjR9M7IegquK5GBQL9fXtj03jq5ov2P6MXiKMUN7PgbeupA1S++zALAoCF
+H2kFjU7pavq3Tm2dLuk21hPxTuxLVygYM+LMPH21H67XVps5AwGkBVr1qd+9d/VEbX7db61twv2
IIpiQdVtadYUpkV8YFqsi8LH5R5b9DeVODJ4Esz10hS/V+sWLlmlkoDIXu+pOtpHNQiz24es/r9g
T5ieklJNwzRq1oOlvQMlPeuNKKYi6wQPFZSlu3DgJun/PFrK0V6L52qMpnVcCUsg4YXiP238Bxov
6WrVRwVvSgu5C2UdRX8g0readWAYxVIiYxeXxPxAqVx4LOIA97FsvZO3HWfHOcHlqwnlvhpcbAGZ
jZd1tn6UWfgICk5Cd4JP07hAviNyHKFV8dfytYeaYaxp4ylgKma5crz2PdDe5jnx69DBYl+Nwea2
bpoODlgSuMtn7D99W3RVeVjgDlMG1ptKix1lC9GV+K8BW7Drb5evAcmN37VxLexZkQb0WfyHqXi+
amh3NQSY7lGsNJDVmAjFmv8kua7nhQks4fXi7aOa8fKAKZs8pbfC/WA4bmnyaD3Y7janvJmAdSoj
1HI8LPg+z9frxQvTc7GC3rqxO3wFOI5w8ZsHB8qro1NAOYNcqxfZz/Pr06GrgedSCYIippDEUm9N
Uq5WBVLJEdh2Pmtx/7iMtKnfRESxeWq4ITgiDlbX9IfT5SvpdlrihcX+8zGLnEsh9Q5l753pJOrx
mO+pVzuTw6GGjBx97Z7BTQCf6xirRTDtj1xYKGzin/LgZSpffNmSDTS+KSRIAahADjkbBnDhdMiC
pQZYG8N3YhJZ/+Jo79JS3UY1c8FCqhgqzWQfXZRztCOzaQfKm4wpDe7WZF3XLtqE0OROpgiH0qXD
bhm95fWEXs1P6fhiTEMMYxwMp75sEeSmUEx1UovoR4SST6PWt/4EZadQa1abicSfJ+TpO+RklpYy
/mx65iBrk5KL1b14DrhtvL0/mEgkqbBiey++mALdmtzDYnk9fqfq7m4xf5kZYgJn/GQ6OOaMWIF/
nqj63qnDPzwYmuUSQsOORSIrhie31PpayJpNquSZUSkLyGD1I1nFPc+i7N5OVCpjfSWAx6iMcuIE
5G+G71gYYK0PmOvCfZ13yP+/yQUPqJFbV6XDeoiXWE/WkbPv41/239AbkIgGVBv6ctD1lMj4KJ4+
cCv0Lx2rFj7InQu69rOjJJSDzFVU3M9N1bY4fccMz+WkDKDx4Y5BgVvXYxlKUyysgLyGyHHgbDj9
Ys3Gsmnl01ep6ACHKW0PuwG4K7oiFbgQHqyQMUGKSYNGVurR4wqEv0MqPsKccpnF4l04nBvtEdnc
S9Q3GVN0AcoeOjk3Asao2UOgiQx6wxXnAzG0wFnsV5elx9UiEv7TENx9JE7G556wESbsb+JtrRJL
f1PXqKdRhB5B6n8GwCVyoQoEkJg4pOXnHSPcW9tqAP2hDKVaztkKmF/1+XENqr+jyazESKVrBQ5Z
6wYSa9oI4Gb6Nlm8IPwXUmoFqFAjjON9gvKK7ET2DSmzgJY4uB3P55axP97cHKwtKJ/FqoVBp51P
02Y03fTy1v1JPfWDGg1SgegtrKBBWHxylIb9OlRpZHx0w/EQ6t4btzMg1Aabp9FNVSTxQet4LQZP
6/lFk6h4lOX1GUVQUgI+arjB5WZvrJKFDNsb/QQ16F85MyMXBle9ax0zporaxit43XqZ2kqJVOoi
LtrAWffJrUoW9Cx0A/pBLyeD6rAHa9F1Ukblkvkf3SJNf1v6cJ77PL0INUxdgBFSPihuKa6V6Nwb
FQ62994zT2hcrk6fY5XXU0ynQ5B+BThGlc9VEdQAlWI9zG6bq35ps9nkVJzD6X4ojHWDTnjwV95K
br4ZlFE8AuNQKWvwrPLrg0QfbyNY/i6Cn4nhEVs15RPfRzdluGS9oLx+E/yflqXz0enoI5qmcFMp
RDO1b4U/P1LSGHCZTrN6+Li+2/LWpUE0exfpNAJ/AKJF/3ei8IODhUKJmWvA0oHG0e+FLvEY99mp
afSJT0GGO+nICWEiNYLsfeKyFN4wfnQxGklwUWt17v2az1hIBApRPX+33AenZZcHHn445hR6nd89
S7I1Frer6C0mWY1DPVLAkS9gneaotK/a12+kYDXKMIkXH8Z3cIS7qSnhLuAfDelsHpD/SB6Y9DWY
zq1EVTBBrt6JbGpMtl2DzFDiWFeShAMRqn6RUYbnTLqJSQ/M5qVP8qkQBe4FAcQc3AYrQAPmwoKb
FfbRNF+5NrhEcBwDJZvpzH/tHCuRSs5mf74tM14X8JEOL2VliV1g7lMhjQpwqsVkRsueqb1UN/yQ
mQzWpyCJeOgWyRLm4WxTniEFFtSZtd91Ln01+xCSrtWSCj03cSztq8Y+vZhjnMB5kyagqPkj+NM6
840GMaQlbsu4q90A2VW5AcYJLRs3GDxedaP8grYxsQTBqKJgxPbS8C7Vx+kDsHjm+Eh9sZ20AjF5
MeSl6StxQoC1g/SspN0j2rtRwnAx5+Zu9AG0Zymn3A4aiCC7vPwpfXfcMzYdvEvQebi1FxXzWnxy
h+MDNLGJt2LAuLuDS2RoAISA5232D6ARsqn1jssYOYNLLDjOmU/JT7PlOqD1oHHDVmXjMy4TzEzt
bkDkwD3rDHHx6tWuZIRWyFCXxGB+BCZQkFZ090zWIz4N3JFkAGmPZ9RAwQX3L41LunW1JHpLbNa4
kHpWMKQ63rvpbLe4kkUewRHw4scQ5HzVtemTT7VUsxlir3/0fB0kAmhtt4IAAAAAEwuW0dsSWiEA
Ae1l+/UCAKNmA+ixxGf7AgAAAAAEWVo=
--Apple-Mail=_BEB00073-75F8-45DE-8EA4-B2F713F68EA3
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii





--Apple-Mail=_BEB00073-75F8-45DE-8EA4-B2F713F68EA3--


From MAILER-DAEMON Tue Jul 26 23:41:06 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSFiT-0000T7-Rw
	for mharc-coreutils@gnu.org; Tue, 26 Jul 2016 23:41:05 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55709)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <calestyo@scientia.net>) id 1bSFEc-0003gf-R5
	for coreutils@gnu.org; Tue, 26 Jul 2016 23:10:15 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <calestyo@scientia.net>) id 1bSFEX-0003GY-P2
	for coreutils@gnu.org; Tue, 26 Jul 2016 23:10:13 -0400
Received: from mailgw-02.dd24.net ([193.46.215.43]:37775)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <calestyo@scientia.net>) id 1bSFEX-00037p-Fg
	for coreutils@gnu.org; Tue, 26 Jul 2016 23:10:09 -0400
Received: from mailpolicy-01.live.igb.homer.key-systems.net
	(mailpolicy-01.live.igb.homer.key-systems.net [192.168.1.26])
	by mailgw-02.dd24.net (Postfix) with ESMTP id C72E25FC9A
	for <coreutils@gnu.org>; Wed, 27 Jul 2016 03:10:00 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at
	mailpolicy-01.live.igb.homer.key-systems.net
Received: from mailgw-02.dd24.net ([192.168.1.36])
	by mailpolicy-01.live.igb.homer.key-systems.net
	(mailpolicy-01.live.igb.homer.key-systems.net [192.168.1.25])
	(amavisd-new, port 10236)
	with ESMTP id xJYx3AXPMPvC for <coreutils@gnu.org>;
	Wed, 27 Jul 2016 03:09:58 +0000 (UTC)
Received: from heisenberg.fritz.box (ppp-46-244-216-201.dynamic.mnet-online.de
	[46.244.216.201])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mailgw-02.dd24.net (Postfix) with ESMTPSA
	for <coreutils@gnu.org>; Wed, 27 Jul 2016 03:09:58 +0000 (UTC)
Message-ID: <1469588998.8048.18.camel@scientia.net>
Subject: grep/sed and some strange patterns/inputs
From: Christoph Anton Mitterer <calestyo@scientia.net>
To: coreutils@gnu.org
Date: Wed, 27 Jul 2016 05:09:58 +0200
Content-Type: multipart/signed; micalg="sha-512";
	protocol="application/x-pkcs7-signature";
	boundary="=-HA7mUBOd7UTX3IX+Ztga"
X-Mailer: Evolution 3.20.4-1 
Mime-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 193.46.215.43
X-Mailman-Approved-At: Tue, 26 Jul 2016 23:41:04 -0400
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 03:10:15 -0000


--=-HA7mUBOd7UTX3IX+Ztga
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey.

I've always had the impression that ^ and $ were the end/begin anchor
of the current pattern, and since e.g. grep/sed work normally in terms
of lines the start/end of lines.

What I found a bit strange is that e.g.:
printf '' | sed 's/^/foo/'
printf '' | sed 's/$/foo/'
printf '' | sed 's/^$/foo/'
doesn't produce foo and that e.g.
printf '' | grep '^'
printf '' | grep '$'
printf '' | grep '^$'
printf '' | grep '*'
don't match.

Why? Or better said, which part of POSIX mandates this? Or is it simply
"no stdin, nothing happens"?
OTOH, other=C2=A0tools do operate on that:
$ printf '' =C2=A0| wc
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A00=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00


What looks IMO also a bit strange:
$ printf 'f' | sed "s/$/foo/"
foof
$printf 'f' | sed "s/$/foo/"
ffoo
(with no newlines)
So they do match,
even though there is no \n character.
OTOH
$ printf 'f\n' | sed
"s/$/foo/"
foof
$printf 'f\n' | sed "s/$/foo/"
ffoo
give the same, so while
there clearly is a \n now (and thus I'd assume a
new line) it's not
matched.
However:
$ printf 'f\n\n' | sed "s/^/foo/"
foof
foo
here it
matches,... so it seems it would just ignore trailing non-\n-ended
and
empty lines, but NOT trailing (or only) non-\n-ended but non-empty
lines
.
Seems a bit inconsequential.


Any ideas? Thanks,
Chris.
--=-HA7mUBOd7UTX3IX+Ztga
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCCEmow
ggXiMIIDyqADAgECAhBctkhuRwyYxn/2gNtkSuKNMA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNVBAYT
AklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0
aWZpY2F0ZSBTaWduaW5nMSkwJwYDVQQDEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0xNTEyMTYwMTAwMDVaFw0zMDEyMTYwMTAwMDVaMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQK
Ew1TdGFydENvbSBMdGQuMSkwJwYDVQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTEjMCEGA1UEAxMaU3RhcnRDb20gQ2xhc3MgMiBDbGllbnQgQ0EwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDuD1CMlQkjhKz1UGqP1jeiTiH9MgllRz6vOPrVG/eE0H/J4QQLV/PeL8RT
4xc44bEzsoJu0IhwnEchb+TxE/qw88w7hxODuw3N8Faxix6a1jp83+RWvZHZf78+O+3GYBpekZfT
Oe9A/FoTXbcgwZfLTMQodn+ckNnX31M/1M2f2/7VA7QBlvihontyHQOlIlryQXnGI0UMCD21oopK
tW48ckv0wUVg8irBKGMeD65gTON/Fsw/ZBbBqadoD1jt85FIM1ql24WUBEBwO1d0ykCKOIbgcqes
3fbcjQpruUNMBbIu1MMIRMqwjx/M7IvSKcS7VYRWl0/K2byzWvBAHh/1AgMBAAGjggFkMIIBYDAO
BgNVHQ8BAf8EBAMCAQYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMBIGA1UdEwEB/wQI
MAYBAf8CAQAwMgYDVR0fBCswKTAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2Eu
Y3JsMGYGCCsGAQUFBwEBBFowWDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Auc3RhcnRzc2wuY29t
MDAGCCsGAQUFBzAChiRodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9jYS5jcnQwHQYDVR0O
BBYEFJmXqxg1OotZRUOYsnJxyPT7Cc8WMB8GA1UdIwQYMBaAFE4L7xqkQFulF2mHMMo0aEPQQa7y
MD8GA1UdIAQ4MDYwNAYEVR0gADAsMCoGCCsGAQUFBwIBFh5odHRwOi8vd3d3LnN0YXJ0c3NsLmNv
bS9wb2xpY3kwDQYJKoZIhvcNAQELBQADggIBAJlBQQTO9iT1TjA0eEO7V/1AbBvMAfibjAgofxmJ
01jBmHKg0pbTjWInTTYoxb3LBgz3mfjvvS1PjnIfb29MyVm0G/PSHjgq7Ews1dEJMPC9XTuxPf2c
+MWLkynBlotW542JprW+iTWfZafyUtzIKW1hk0YASJ8zSSj8D++9yR+0UhkbvlECJkdi1+et0EaI
7HIX6ccj1rfcFFflWX/fPT64dn9jpg9s0nuJug4WsVkEK236WndZoMHrZmgF7CIyZ3T0muqYwkAS
DDcRt9A21o/Mc+D8Q6GVmKRGB3gEKvOtsioHZEqJv6CdAm3a6gloo5pX3RL1eCzc8Lzfs4T0ISZj
r8xNMbTGlsuHaFH+stDewKsfnpo4N64OtAGrzmVfFFsMIRENRHsVlSEe/6LVpBpjn00+7bqEN3qe
qSxIOraYJ12mJ08G4YnP2U1fadHIaS50O5ZXqAivoBl9pi/6CBNc5wIMlkXMyFZ0sLsI+9ErFDu9
OJhX7iWCo69X0ydlzXj08+2K1PKyr+g6/vTPHur63JotNrhyoWEJyWEjBLA4QmJXfGpB5u+bCwNf
sFpcYAlkiFt1Rs2vemgSBy3q4DHDbPKvr1YImZNRSHP+TX9NP94JRkmqcyD/qaN7u2JV1pRAlwG0
npyrn6ZQM1QHV+iFbwQ4VqLMWk//QVo/rZy1MIIGPjCCBSagAwIBAgIQPZ6tDP/RHLwcz1F+2kRA
DzANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjEp
MCcGA1UECxMgU3RhcnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxIzAhBgNVBAMTGlN0YXJ0
Q29tIENsYXNzIDIgQ2xpZW50IENBMB4XDTE2MDEwMjAyMDIzNloXDTE4MDEwMjAyMDIzNlowejEL
MAkGA1UEBhMCREUxDzANBgNVBAgMBkJheWVybjERMA8GA1UEBwwITcO8bmNoZW4xITAfBgNVBAMM
GENocmlzdG9waCBBbnRvbiBNaXR0ZXJlcjEkMCIGCSqGSIb3DQEJARYVY2FsZXN0eW9Ac2NpZW50
aWEubmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsoxs789VtR81tlqeP2vwE3Cp
YosZYRfrovj+Wb91paoV5OQcFuXA7yONnLsWxOnJ3AGgy1/wG79Ko+yHSmF2K88pEWjrFe0zmZWZ
0b2OtMoKLEjxz7Nz2N7/lmgss8XinJ083LwLKSe4f+JLjnF5hX/g6wG4NkFXVOIf2YFZ0+c1NYFQ
T6Vy42EWdk+JLDoXjyd2hEperLCz9rQr2k3wh0cn8R1FDUGerBGpMnvyKV6JXj5vZsqJnmiiId9e
0IdBj4Wcs3hT2usk6dJYwHrZ8b7Fkv6BRq90DUI7i8+ukTdp5hnAb8TPSbe1JS4h3Jc7r2NVLkpo
Gbuw98er9s2PEAvpECD3toO3ojBevL56vEDbGa5tXpvvcWCz6t3QyEKJ2E54hWw1fbUA0BtydQC0
mhKBtrENVnnCDrSgiyBMnd8y+kw6iPKeJEjNObGubc4BZp47zPn/ZiDBaHue6S0MXsoS7XEGOuCU
09S/8kD8wTecBg+KdlLSNkLm+xqx6Cy7zwcj3IuQv0PeWi4kfEkAJfl8IeAP/4049iA6a1rkhJl7
1rDjpEGBCG+i9BAgBeNdHLtQ7bugdvU+GfHjFXEU1emUk91E2liML8kpCvhUAXcLhhL8bxPJiX2/
VQ2esrpQVO56OzpivNauonLaWmHEKOSfPC3s5AKMytMTiUiszEUCAwEAAaOCAcMwggG/MAsGA1Ud
DwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwCQYDVR0TBAIwADAdBgNVHQ4E
FgQUwRklxF1ZV2ooSMkFwLiK13apkK0wHwYDVR0jBBgwFoAUmZerGDU6i1lFQ5iycnHI9PsJzxYw
bwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5jb20wOQYI
KwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5jbGllbnQyLmNydDA4
BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2EtY2xpZW50Mi5jcmww
IAYDVR0RBBkwF4EVY2FsZXN0eW9Ac2NpZW50aWEubmV0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cu
c3RhcnRzc2wuY29tLzBUBgNVHSAETTBLMAwGCisGAQQBgbU3BgEwOwYLKwYBBAGBtTcBAgQwLDAq
BggrBgEFBQcCARYeaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5MA0GCSqGSIb3DQEBCwUA
A4IBAQBb8EIsoUN/tgUgQqrNXNtlksPep84kx5yRBgr71uf3ITLddGtzfDBj6KdZUoy7SG3MQkyO
kvYmXBClj23rv8Iol48/3oi9XWZw5EV3uHrRse1TzQgMPZE0hZsDgkqXVoxMQfk55ndjZIHMfSkn
hdnqSP5zZ4TCmDEKLppPMDcQSSrjilnbthpxlIJzeGeEFtrh6ssh/oF6mUaEGFcd8kx9RS51K1gt
H4J36y4E6pKB7EdxG2+0yVzAIta8dkD/BiMCKRWhp1EmzQ2uIh2nX5y8t4e2xHHuiy11Yeq6UTW/
JmfxF4xcbkNK/rv9ISBg9K+mZtg2QXr+P/CJBtej2RLUMIIGPjCCBSagAwIBAgIQPZ6tDP/RHLwc
z1F+2kRADzANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20g
THRkLjEpMCcGA1UECxMgU3RhcnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxIzAhBgNVBAMT
GlN0YXJ0Q29tIENsYXNzIDIgQ2xpZW50IENBMB4XDTE2MDEwMjAyMDIzNloXDTE4MDEwMjAyMDIz
NlowejELMAkGA1UEBhMCREUxDzANBgNVBAgMBkJheWVybjERMA8GA1UEBwwITcO8bmNoZW4xITAf
BgNVBAMMGENocmlzdG9waCBBbnRvbiBNaXR0ZXJlcjEkMCIGCSqGSIb3DQEJARYVY2FsZXN0eW9A
c2NpZW50aWEubmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsoxs789VtR81tlqe
P2vwE3CpYosZYRfrovj+Wb91paoV5OQcFuXA7yONnLsWxOnJ3AGgy1/wG79Ko+yHSmF2K88pEWjr
Fe0zmZWZ0b2OtMoKLEjxz7Nz2N7/lmgss8XinJ083LwLKSe4f+JLjnF5hX/g6wG4NkFXVOIf2YFZ
0+c1NYFQT6Vy42EWdk+JLDoXjyd2hEperLCz9rQr2k3wh0cn8R1FDUGerBGpMnvyKV6JXj5vZsqJ
nmiiId9e0IdBj4Wcs3hT2usk6dJYwHrZ8b7Fkv6BRq90DUI7i8+ukTdp5hnAb8TPSbe1JS4h3Jc7
r2NVLkpoGbuw98er9s2PEAvpECD3toO3ojBevL56vEDbGa5tXpvvcWCz6t3QyEKJ2E54hWw1fbUA
0BtydQC0mhKBtrENVnnCDrSgiyBMnd8y+kw6iPKeJEjNObGubc4BZp47zPn/ZiDBaHue6S0MXsoS
7XEGOuCU09S/8kD8wTecBg+KdlLSNkLm+xqx6Cy7zwcj3IuQv0PeWi4kfEkAJfl8IeAP/4049iA6
a1rkhJl71rDjpEGBCG+i9BAgBeNdHLtQ7bugdvU+GfHjFXEU1emUk91E2liML8kpCvhUAXcLhhL8
bxPJiX2/VQ2esrpQVO56OzpivNauonLaWmHEKOSfPC3s5AKMytMTiUiszEUCAwEAAaOCAcMwggG/
MAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwCQYDVR0TBAIwADAd
BgNVHQ4EFgQUwRklxF1ZV2ooSMkFwLiK13apkK0wHwYDVR0jBBgwFoAUmZerGDU6i1lFQ5iycnHI
9PsJzxYwbwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5j
b20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5jbGllbnQy
LmNydDA4BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2EtY2xpZW50
Mi5jcmwwIAYDVR0RBBkwF4EVY2FsZXN0eW9Ac2NpZW50aWEubmV0MCMGA1UdEgQcMBqGGGh0dHA6
Ly93d3cuc3RhcnRzc2wuY29tLzBUBgNVHSAETTBLMAwGCisGAQQBgbU3BgEwOwYLKwYBBAGBtTcB
AgQwLDAqBggrBgEFBQcCARYeaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5MA0GCSqGSIb3
DQEBCwUAA4IBAQBb8EIsoUN/tgUgQqrNXNtlksPep84kx5yRBgr71uf3ITLddGtzfDBj6KdZUoy7
SG3MQkyOkvYmXBClj23rv8Iol48/3oi9XWZw5EV3uHrRse1TzQgMPZE0hZsDgkqXVoxMQfk55ndj
ZIHMfSknhdnqSP5zZ4TCmDEKLppPMDcQSSrjilnbthpxlIJzeGeEFtrh6ssh/oF6mUaEGFcd8kx9
RS51K1gtH4J36y4E6pKB7EdxG2+0yVzAIta8dkD/BiMCKRWhp1EmzQ2uIh2nX5y8t4e2xHHuiy11
Yeq6UTW/JmfxF4xcbkNK/rv9ISBg9K+mZtg2QXr+P/CJBtej2RLUMYIEfjCCBHoCAQEwgYkwdTEL
MAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENl
cnRpZmljYXRpb24gQXV0aG9yaXR5MSMwIQYDVQQDExpTdGFydENvbSBDbGFzcyAyIENsaWVudCBD
QQIQPZ6tDP/RHLwcz1F+2kRADzANBglghkgBZQMEAgMFAKCCAcUwGAYJKoZIhvcNAQkDMQsGCSqG
SIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTYwNzI3MDMwOTU4WjBPBgkqhkiG9w0BCQQxQgRAFoS1
Sxl1XRrC0wIwP4kaOSR1X2bQvZuvllUDYxWfmbxZ9Xtte9o1adzuW5PpTq0jHW08AaNnPR7G0psN
jCJR9jCBmgYJKwYBBAGCNxAEMYGMMIGJMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENv
bSBMdGQuMSkwJwYDVQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEjMCEGA1UE
AxMaU3RhcnRDb20gQ2xhc3MgMiBDbGllbnQgQ0ECED2erQz/0Ry8HM9RftpEQA8wgZwGCyqGSIb3
DQEJEAILMYGMoIGJMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSkwJwYD
VQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEjMCEGA1UEAxMaU3RhcnRDb20g
Q2xhc3MgMiBDbGllbnQgQ0ECED2erQz/0Ry8HM9RftpEQA8wDQYJKoZIhvcNAQEBBQAEggIAG3FK
j/WJ9ZP3U8ssgVn//IrZZ4Rredb5AWv6OOJJy08aYFWgSf/hMJr9mpnw3mLr3GtjrHlGZfAcGzXX
VFX2iUd7zPbp5nHw90Uqn2LfA1hWcJcuhWBsv7xzidksOjoRnXUpeyx+KD/DAhPgv8LrnrOpW5o2
yt4HJdey69I8VFSWDt4UPqcDHAPD58TP3MRdrtc0D0kk5JTz+IqxexX2s/AuZBQktQph9n3AH0Vk
ZgpDqfJ6pZNUoltvT3r9NakgXAa0G4uFFD6ok/6JHTwbt4FZiojiW1G04zr0kHRMehwZTZql3QL1
dX0DPEuoLIG92cYxGOXZcqGRS4f+CSRBUUJRMpWdBSZMgERuNCaNV4mmrgAVwwuriJ8/fqz96IPT
oM9KYZUMoajU4o4ozUMC2JXdUPAbkkPFaGrpyJr8y2BpJ6qqBvvubi3Kq4CFiTTm4VzmTyZaNJm5
uwtJwNFwQVB40otlLRv2Szjm31XgPVd6Yl7YL3EGToqV/YMdjEFDxmYy/LJOpS9ihhksgX3KGIO2
aqw4/7aDMblXfDihGmSbl1zpGzXpjcj+Ifh/TYGLgnlVtqnJUYMy+Oy0YBASIs4QyIE/HGbxYeO7
nCzdxj8sGeNEcPeziGOMrgxBUuH/mc+ov3VULjFqDS4lmIptc1M8JawEtvx0PwOQt5RrFQMAAAAA
AAA=


--=-HA7mUBOd7UTX3IX+Ztga--



From MAILER-DAEMON Wed Jul 27 00:55:06 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSGs6-0006vO-19
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 00:55:06 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:47981)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bSGs3-0006uw-Gg
	for coreutils@gnu.org; Wed, 27 Jul 2016 00:55:04 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bSGs0-0002ee-Al
	for coreutils@gnu.org; Wed, 27 Jul 2016 00:55:03 -0400
Received: from mail-qt0-x231.google.com ([2607:f8b0:400d:c0d::231]:33301)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1bSGs0-0002eL-4w
	for coreutils@gnu.org; Wed, 27 Jul 2016 00:55:00 -0400
Received: by mail-qt0-x231.google.com with SMTP id w38so23787463qtb.0
	for <coreutils@gnu.org>; Tue, 26 Jul 2016 21:54:58 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=Smtf43WZPefVYS7NG+7a4uNZpFdsCWDS4l6ncI/Oz3c=;
	b=JSc4qF2gn7wY8aCnbY3nA4Yd7acQam1SZUwGs0xSZJw/dImZmuF4flKZUPKX7+sapB
	3ickXYDjTSc+AFKgLkNSOPiX/MLM8NsLRLumxibxqEMyKGMtP2r8ukTsCCbO/DxQhu43
	TQ6++Ddt9vmJM7/rEiwkGdJNHrz2zdUlAI/lHVl6H3zf4fID4rXtfOmICLyjzd0lVWMu
	XtnDQOe1PW4fVjtms6AyXgy2IrcqUwuHEud2A0JmeFF9oSU5Y9tPaiTBxPx5keY9aDdY
	WjOcRIukPL/aFWURB8+gST3YUNJz8oK73ALW1R8UN3z5SlkhNpvxGAnfhtHosO6tucs8
	0LGA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
	d=1e100.net; s=20130820;
	h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc
	:content-transfer-encoding:message-id:references:to;
	bh=Smtf43WZPefVYS7NG+7a4uNZpFdsCWDS4l6ncI/Oz3c=;
	b=XxlM+QEWiPHFsqje9ZzmOmuhpbO6iJ5Vri6j0yyk+/Sj7oqdtTGADo6dxp7S6IFmIk
	A4EeqbY84BtVcTHK9RbODxxEPIbpwjZwlb9hUlJ3Z6qhjLipOOeYsf//I0izH3pVrAGY
	DSr7Udgl0sPdxuHzN5FnMywhsZLLftUnoNNsPz7wpuayIbnKHy6ZooJ/G1NhSVHiLx74
	toFB0SNeJM3T6cgCWiqoWUWRX7m1QnoKGqDkkSwGzNy2h1RyFRZ1+4zFYH+yVyV0TYib
	v0RigEF1CL5IHuFX5ICxUCGBUOcIdsSUr2nx6rPOEbeSLLu/oL2EYMI87AJ603263YV9
	2h7A==
X-Gm-Message-State: AEkoouv4tg2bsD1G6QN1DitNL3m3KXSMPURCK5iMKTHeMz6kHQyq4GOUGIjQPalaeO180Q==
X-Received: by 10.200.45.108 with SMTP id o41mr44337120qta.100.1469595297992; 
	Tue, 26 Jul 2016 21:54:57 -0700 (PDT)
Received: from ix.home (pool-100-38-105-55.nycmny.fios.verizon.net.
	[100.38.105.55]) by smtp.gmail.com with ESMTPSA id
	b206sm2938684qkc.43.2016.07.26.21.54.57
	(version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
	Tue, 26 Jul 2016 21:54:57 -0700 (PDT)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\))
Subject: Re: grep/sed and some strange patterns/inputs
From: Assaf Gordon <assafgordon@gmail.com>
In-Reply-To: <1469588998.8048.18.camel@scientia.net>
Date: Wed, 27 Jul 2016 00:54:56 -0400
Cc: Coreutils <coreutils@gnu.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <18DAEF49-ED1E-4218-B0F1-E5C8F11C96E0@gmail.com>
References: <1469588998.8048.18.camel@scientia.net>
To: Christoph Anton Mitterer <calestyo@scientia.net>
X-Mailer: Apple Mail (2.2102)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 2607:f8b0:400d:c0d::231
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 04:55:04 -0000

Hello,

> On Jul 26, 2016, at 23:09, Christoph Anton Mitterer =
<calestyo@scientia.net> wrote:
>=20
> I've always had the impression that ^ and $ were the end/begin anchor
> of the current pattern, and since e.g. grep/sed work normally in terms
> of lines the start/end of lines.
[...]
> What I found a bit strange is that e.g.:
> printf '' | sed 's/^/foo/'
>=20
> doesn't produce foo and that e.g.
> printf '' | grep '^'
> don't match.
>=20
> Why? Or better said, which part of POSIX mandates this? Or is it =
simply
> "no stdin, nothing happens"?

Exactly!

The command  "printf '' " sends no output, and it is equivalent to =
redirection from /dev/null,
which means sed immediately receives an end-of-file marker and does not =
try to execute any command.

printf with *any* output (with newlines or not) will cause 'sed' and =
'grep' to read some characters, and then to try to execute commands or =
match patterns on the input.

This can be demonstrated using 'strace' on GNU/Linux machines.
The commands below run 'sed' with both printf and /dev/null, and =
'strace' will report the 'read' system-call.
The first 'read(3,...)' can be ignored, it is the OS reading a shared =
library.

The second 'read(0,...)' is the interesting one:
The first "0" indicates reading from STDIN.
sed tries to read upto 4096 bytes from STDIN, and the returned value is =
zero (following the equal sign).
Zero value indicates an end-of-file - meaning there is no input at all,
and sed will not try to execute any commands.
Notice that printf with an empty string and /dev/null result in the same =
behavior:=20

  $ strace -e read sed 's/^/foo/' < /dev/null
  read(3, =
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., =
832) =3D 832
  read(0, "", 4096)                       =3D 0
  +++ exited with 0 +++

  $ printf '' | strace -e read sed 's/^/foo/'
  read(3, =
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., =
832) =3D 832
  read(0, "", 4096)                       =3D 0
  +++ exited with 0 +++


However, if even one character is provided in STDIN,
The read() function will return it, and sed will try to execute the =
pattern/command on the input:

  $ printf 'a' | strace -e read sed 's/^/foo/'
  read(3, =
"\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., =
832) =3D 832
  read(0, "a", 4096)                      =3D 1
  read(0, "", 4096)                       =3D 0
  read(0, "", 4096)                       =3D 0
  fooa+++ exited with 0 +


One possible source of confusion is 'echo' vs 'printf': echo by default =
automatically adds a newline.
Thus, the command:
   echo '' | sed 's/^/foo/'
does work as expected because there is some input (one byte: a newline).
Where as this command does not, since there is no input at all:
   printf '' | sed 's/^/foo/'


'grep' follows the same principle, and can be examined using:
   printf '' | strace -e read grep -q '.' && echo match || echo no-match
   strace -e read grep -q '.' </dev/null  && echo match || echo no-match
   printf 'a' | strace -e read grep -q '.' && echo match || echo =
no-match


Others can perhaps elaborate regarding POSIX standard.
=46rom a cursory look, it seems the wording for 'grep' and 'sed' imply =
the output is tied to having input,
while there is mandatory default output for 'wc' regardless of input ( =
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/wc.html#tag_20_1=
54_10 ).



Hope this helps,
  - assaf


P.S.
A minor nitpick: coreutils is a separate project from grep or sed.
grep questions should be sent to bug-grep@gnu.org , and
sed questions should be sent to sed-devel@gnu.org .





From MAILER-DAEMON Wed Jul 27 03:33:46 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSJLe-0002vi-3p
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 03:33:46 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:45834)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSJLc-0002rq-76
	for coreutils@gnu.org; Wed, 27 Jul 2016 03:33:45 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSJLX-0005wS-AQ
	for coreutils@gnu.org; Wed, 27 Jul 2016 03:33:43 -0400
Received: from mout.kundenserver.de ([212.227.17.10]:51367)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSJLX-0005wG-0V
	for coreutils@gnu.org; Wed, 27 Jul 2016 03:33:39 -0400
Received: from tweedy.camline.com ([62.153.148.194]) by
	mrelayeu.kundenserver.de (mreue103) with ESMTPSA (Nemesis) id
	0LiWYm-1asABR2v76-00chKV; Wed, 27 Jul 2016 09:33:37 +0200
From: Bernhard Voelker <mail@bernhard-voelker.de>
To: coreutils@gnu.org
Cc: Bernhard Voelker <mail@bernhard-voelker.de>
Subject: [PATCH 1/2] build: update gnulib submodule to latest
Date: Wed, 27 Jul 2016 09:33:18 +0200
Message-Id: <20160727073319.30422-1-mail@bernhard-voelker.de>
X-Mailer: git-send-email 2.9.2
X-Provags-ID: V03:K0:vvDxnMwDkLiXAFICS9z2GSNZsiYSDit6iIM6ALKs3JhYqndoreN
	mQraxTc+cJu+ETf2woyRQgXlAaxPZVxIKsEWhlo+Zmb0Zc8Bww3tqvki5iiz4rUTHDQ/bZr
	ghsxumEIO6mZ8ypB9SkdvoydXZJgaBnVB4s4kZ9Me4AiWSmHleL08M6Be7cMIiJeiHT06qt
	8xI+HWeIZZMQDCkYgckRA==
X-UI-Out-Filterresults: notjunk:1;V01:K0:SBJGSaHHleg=:B4+luum48JZs0ARy/z9KYg
	fTpO5an9ExU/M4VInovkTFTGTRImmqnYz0ZXPH7+gN0wYQEGlRpQdBG432hLr3Penqx8RX8I0
	lc6UN+rDj9dLf1X8BVJ5V4P577/dVP7NP3JE5MyicrqjUBPS4LPi0WCsVJn6om5viKZE025+R
	dzBdaBQO/zU0uQTDt0TlTv89clxmsLVMfedM31pszlTbgLfgi+yulI/QXWGRXbEJ6o1iOedts
	tYbH3JmenBSPle1z3SeLpPBmUrZXJLfov1YgOa8cRKY9mH3wIE/eevlfLjix2SWjG8+8myUY3
	4fW0gZDGHpXmKTvzuHy/Wk9UHKQPJ4UHZytYJTYf4RPopOZqI1kwEiZZz69SS9DtwCS4jGShs
	RfxbGlKpawZPUc4c7J1fBj1v+OTvM1K24xOizdrMtD3L9UtUDbIjMczPR3IhNLREufcDmP5Tv
	xRlIbJbg2YearhIcWyrA6nyvpBpLReTECy93VNI8jH42ShKAAI2/KAmPvr6yc2AFQPfv4HWFR
	02cno99kvauqC4iAvttJ6rQ2ceSZjrhPOHSMhgoj64UPvCka+9ty2274lhjvidicJy1IiI5z0
	9UuCNIIWWn8MrjR8V7wzUwBcaS8cgeURztUlw8VeBbQvQKZOEJ+bdpFsNqwrbL/Y9FxwW1vL8
	DCmYeR1BnF91Cpo2x2yLcJHBtE4/OJoHq2/SSMz+acPIHwZlb8sbGQx7QBHgsvoOrZoM=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.17.10
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 07:33:45 -0000

Mainly for
* 77d36b5 obstack: pacify GCC 6 with -Wnull-dereference
---
 gnulib | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnulib b/gnulib
index 6835fc4..348402f 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit 6835fc458f30b94f15d69c35a79cbc2dfabe2d06
+Subproject commit 348402f2aac342bc925b7aaea9ee3cc353f427a9
-- 
2.9.2



From MAILER-DAEMON Wed Jul 27 03:33:49 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSJLh-0002yO-3t
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 03:33:49 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:45847)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSJLe-0002uP-JR
	for coreutils@gnu.org; Wed, 27 Jul 2016 03:33:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSJLZ-0005ws-I9
	for coreutils@gnu.org; Wed, 27 Jul 2016 03:33:45 -0400
Received: from mout.kundenserver.de ([217.72.192.75]:50758)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSJLZ-0005wi-8A
	for coreutils@gnu.org; Wed, 27 Jul 2016 03:33:41 -0400
Received: from tweedy.camline.com ([62.153.148.194]) by
	mrelayeu.kundenserver.de (mreue103) with ESMTPSA (Nemesis) id
	0MM06e-1bMqER3eyC-007kI7; Wed, 27 Jul 2016 09:33:39 +0200
From: Bernhard Voelker <mail@bernhard-voelker.de>
To: coreutils@gnu.org
Cc: Bernhard Voelker <mail@bernhard-voelker.de>
Subject: [PATCH 2/2] maint: pacify GCC 6 with -Wnull-dereference
Date: Wed, 27 Jul 2016 09:33:19 +0200
Message-Id: <20160727073319.30422-2-mail@bernhard-voelker.de>
X-Mailer: git-send-email 2.9.2
In-Reply-To: <20160727073319.30422-1-mail@bernhard-voelker.de>
References: <20160727073319.30422-1-mail@bernhard-voelker.de>
X-Provags-ID: V03:K0:GpugKihc5wHfzBiyJm2SAvYqoR9mIhjf8m13qNEPeUwQhrPx4GB
	HRCJo4GIiwfivVY3xrudN9f96uFEDWuB+aX6MzhLGWk4/8koIbujJRXkRfW8++GTa4w05Rh
	xXvPCtXcw/6y9aMBYUoOYQG3g6aUa1jyQONf5Z/jqL0cYLXfAXgy9Yt6qfFsiKQoQupZRPl
	I4qo7Zgvq32RmX65Ptb5Q==
X-UI-Out-Filterresults: notjunk:1;V01:K0:8VGTNo2sie4=:SZOATFBPCJJKiCm+GCnT2C
	bSaI4yGxLzwvBrrpvh6+tvN96cmO6I5HAfE4aHE4g7+qja8N20HdvsCMYJp1QR2W01xV9azSu
	3v9R3hk1zrONzkTnqaATXNdIWshPC7JzxTuBI0Y9zy1jf6dNouPa4L7CxScqRs0J/+KkLwhjI
	Nd1xHzq4neP9FcfxSlhYWDDCrAxWx6U+ZD+384pA6+XC+8I4+YybyAfIGQdJr6fV1bv/DRwQ5
	P3WFtsSoWf2iC+Uz93gN5SzeLCoatfSMRNMyuABWM8n3/y2CN84csP7tooK+ehkoLtpTzSgsL
	018LESxjXDmTmYM8zSXiwyAzPw3flcLjljhBeMOi+9oV9rvvrxdRv2QPUMjv2IYrFAAFxPoaS
	RzePZdK/tl7XeFBBsxwmKVC81RBC+vN5d1+AMofsU96jQNlqOVnZAReDRajylkmXvKutzA74G
	GUMxtMaesLr7dldPy409BSKvySJ0hr4zLmn/riQhxF6hpf/ILHFtz9hnwIR66OTpSR/c2xotM
	vG1LJFCHRzU/nB+tmQYVRIAdJ17N59Wg309ja9hixhHFQw8LYGqra9gmw07WVuZ8VGEs3gQhO
	3rFfHbIxYuj9bJ8dvaivP5e2yK8oIt7lVJOvyhh+JrJJv8bcKsXKOyrZiUiFFiQrA3a8+nV2Y
	DHe1SyUTSwztA9a1pVBh1wi48FQOfbaf30mv85mB//0u0BYzSBd7KP2MH+leA5vHUHbM=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 217.72.192.75
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 07:33:47 -0000

src/id.c:249:29: error: potential null pointer dereference \
[-Werror=null-dereference]
       pw_name = xstrdup (pwd->pw_name);
                          ~~~^~~~~~~~~
src/whoami.c:89:11: error: potential null pointer dereference \
[-Werror=null-dereference]
   puts (pw->pw_name);
         ~~^~~~~~~~~

* src/id.c (main): Explicitly exit with EXIT_FAILURE after an eror to
help gcc-6 to detect that the dereferenced pointer is valid.
* src/whoami.c (main): Likewise.
---
 src/id.c     | 5 ++++-
 src/whoami.c | 7 +++++--
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/src/id.c b/src/id.c
index 218ee5a..35cbeb5 100644
--- a/src/id.c
+++ b/src/id.c
@@ -245,7 +245,10 @@ main (int argc, char **argv)
             }
         }
       if (pwd == NULL)
-        error (EXIT_FAILURE, 0, _("%s: no such user"), quote (spec));
+        {
+          error (0, 0, _("%s: no such user"), quote (spec));
+          exit (EXIT_FAILURE);
+        }
       pw_name = xstrdup (pwd->pw_name);
       ruid = euid = pwd->pw_uid;
       rgid = egid = pwd->pw_gid;
diff --git a/src/whoami.c b/src/whoami.c
index e58c575..972cd55 100644
--- a/src/whoami.c
+++ b/src/whoami.c
@@ -84,8 +84,11 @@ main (int argc, char **argv)
   uid = geteuid ();
   pw = (uid == NO_UID && errno ? NULL : getpwuid (uid));
   if (!pw)
-    error (EXIT_FAILURE, errno, _("cannot find name for user ID %lu"),
-           (unsigned long int) uid);
+    {
+      error (0, errno, _("cannot find name for user ID %lu"),
+             (unsigned long int) uid);
+      exit (EXIT_FAILURE);
+    }
   puts (pw->pw_name);
   return EXIT_SUCCESS;
 }
-- 
2.9.2



From MAILER-DAEMON Wed Jul 27 04:39:17 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSKN3-0003GP-2c
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 04:39:17 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33345)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKN0-0003GG-IT
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:39:15 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKMw-0006ES-Aa
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:39:13 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:54830)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKMw-0006EM-50
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:39:10 -0400
Received: from [192.168.1.80] (unknown [109.76.143.177])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 0D80B949C;
	Wed, 27 Jul 2016 09:39:08 +0100 (IST)
Subject: Re: multibyte processing - handling invalid sequences (long)
To: Assaf Gordon <assafgordon@gmail.com>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
	<E56651FF-B445-4663-87D0-593EDD55BC61@gmail.com>
	<57909F2C.7060403@draigBrady.com>
	<654AF25C-6BC4-4865-BD6C-41482B077BD2@gmail.com>
	<57934C33.30308@draigBrady.com>
	<D92CD138-0E60-46C4-B54C-96C44DA7784E@gmail.com>
	<5793D3CF.4010707@draigBrady.com>
	<24670EEA-3D00-43D1-B039-C345F827B1D2@gmail.com>
Cc: Eric Blake <eblake@redhat.com>, Coreutils <coreutils@gnu.org>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <5798732B.8010500@draigBrady.com>
Date: Wed, 27 Jul 2016 09:39:07 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <24670EEA-3D00-43D1-B039-C345F827B1D2@gmail.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 08:39:15 -0000

On 27/07/16 03:47, Assaf Gordon wrote:
>=20
>> On Jul 23, 2016, at 16:30, P=E1draig Brady <P@draigBrady.com> wrote:
>>
>> On 23/07/16 19:05, Assaf Gordon wrote:
>>>
>>>> On Jul 23, 2016, at 06:51, P=E1draig Brady <P@draigBrady.com> wrote:
>>>> I was wondering about the tool being line/record oriented.
>>>>
>>>> Disadvantages are:
>>>> requires arbitrary large buffers for arbitrary long lines
>>>> relatively slow in the presence of short/normal lines
>>>> sensitive to the current stdio buffering mode
>>>> requires -z option to support NUL termination
>>>>
>>>> Processing instead a block at a time avoid such issues.
>>>> UTF-8 at least is self synchronising, so after reading a block
>>>> you just have to look at the last 3 bytes to know
>>>> how many to append to the start of the next block.
>=20
> Attached is a partial, crude implementation of stream-based processing.
> It currently only handles fixing invalid sequences, no unicode normaliz=
ation yet.
>=20
> It contains both implementation, to ease comparison (use "-S/--stream" =
to use the new implementation, or without to use the previous line-based =
implementation).
>=20
> The main functions are (to facilitate discussion):
> mbbuf_read - reads more data from the input, moves 'incomplete/left-ove=
r' octets from previous read to the beginning of the buffer (somewhat lik=
e grep's fillbuf() but not as sophisticated).
> STRM_unorm_buf - iterates over the octets in the current buffer
> STRM_unorm_fd - repeatedly reads the file and calls STRM_unorm_buf.
>=20
> The tests use both methods and the results are identical (except unicod=
e normalization with is currently skipped for --stream).
>=20
> Few issues are emerging:
> 1. If only validation is requires (i.e. no unicode normalization), it'l=
l be wasteful to convert the input to wchar_t then back again. It'll be b=
etter to write the output as-is.  If unicode normalization is requested, =
then going through wchar_t and uniform's filter is needed. Perhaps two se=
parate dedicated functions would be more efficient.

Yes that makes sense.
You might simplify by not converting only in the utf8 case if that helps,
as that's going to be by far the most common case.

> 2. Regarding skipping STDIO buffering: I assume you referred to dealing=
 with input. The code now uses file-descriptors and 'safe_read', thus byp=
assing stdio buffering on input. But it still uses stdio for output (this=
 seems in line with tac, split, tr, etc.). If we want to bypass stdio as =
well, some extra code for internal buffering might be needed.

Well if reading and writing blocks then stdio buffering is only problemat=
ic overhead.

>=20
> 3. I believe that for this tool to be really useful, it should report t=
he line number and column of offending/invalid octets. In that case, the =
code needs to count lines / columns, and will need to be aware of which l=
ine-terminator is used - meaning "-z" is still needed.
> The attached code does count lines/columns (see struct mbbuffer), and t=
hus is a bit cumbersome.

Interesting. Fundamentally the processing doesn't need to be line oriente=
d,
hence avoiding buffering issues and possibly better perf, but I do see it=
 useful
to report the offsets of errors. If iterating over the buffer we can stil=
l
record "lines" seen.

> Currently it seems this optimization leads to somewhat more complicated=
 code.
> Once I'll have the unicode normalization implemented we could compare s=
peeds and see which method is preferred.

thanks!
P=E1draig



From MAILER-DAEMON Wed Jul 27 04:41:06 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSKOo-0003Ti-Pz
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 04:41:06 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33635)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKOm-0003Tc-MA
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:41:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKOj-0006dT-G9
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:41:04 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:54842)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKOj-0006dM-AC
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:41:01 -0400
Received: from [192.168.1.80] (unknown [109.76.143.177])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 93812949C;
	Wed, 27 Jul 2016 09:41:00 +0100 (IST)
Subject: Re: maint: skip a check when en_US.UTF-8 collation rules are broken
To: Jim Meyering <jim@meyering.net>, Coreutils <coreutils@gnu.org>
References: <CA+8g5KFXEyojnf-885Sj9GvgnZEAdKOUPD9Bi3DjwWV7=wf9oQ@mail.gmail.com>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <5798739C.1050801@draigBrady.com>
Date: Wed, 27 Jul 2016 09:41:00 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <CA+8g5KFXEyojnf-885Sj9GvgnZEAdKOUPD9Bi3DjwWV7=wf9oQ@mail.gmail.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 08:41:05 -0000

On 25/07/16 16:48, Jim Meyering wrote:
> On OS X, *.UTF-8 locales use ASCII collating rules(!?):
> 
>   $ readlink /usr/share/locale/*.UTF-8/LC_COLLATE|sort -u
>   ../la_LN.US-ASCII/LC_COLLATE
> 
> This means that sort, and any other program that relies on strcoll,
> cannot be expected to work consistently on OS X in any UTF-8 locale.

Wow.
The change looks good.

thanks



From MAILER-DAEMON Wed Jul 27 04:56:55 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSKe7-0001fI-Ox
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 04:56:55 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:36242)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKe5-0001f9-DX
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:56:54 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKdz-00026J-EJ
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:56:52 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:54904)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSKdz-00025h-7h
	for coreutils@gnu.org; Wed, 27 Jul 2016 04:56:47 -0400
Received: from [192.168.1.80] (unknown [109.76.143.177])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id BC2A3949C;
	Wed, 27 Jul 2016 09:56:44 +0100 (IST)
Subject: Re: [PATCH] maint: fix dependency of man/arch.1
To: Bernhard Voelker <mail@bernhard-voelker.de>, coreutils@gnu.org
References: <1469559488-700-1-git-send-email-mail@bernhard-voelker.de>
Cc: Ludwig Nussel <lnussel@suse.com>, Rodrigues Goldwyn <rgoldwyn@suse.com>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <5798774C.9030101@draigBrady.com>
Date: Wed, 27 Jul 2016 09:56:44 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <1469559488-700-1-git-send-email-mail@bernhard-voelker.de>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 08:56:54 -0000

On 26/07/16 19:58, Bernhard Voelker wrote:
> The following is a reproducer for the wrong dependency:
> 
>   $ ./configure --enable-install-program=arch
>   $ make
>   $ rm -f src/arch man/arch.1
>   $ make man/arch.1
>     GEN      man/arch.1
>   help2man: can't get `--help' info from man/arch.td/arch
>   Try `--no-discard-stderr' if option outputs to stderr
>   Makefile:14378: recipe for target 'man/arch.1' failed
>   make: *** [man/arch.1] Error 127
> 
> * man/local.mk (man/arch.1): Change to depend on src/arch rather than
> src/uname: while the arch binary depends on uname.c and uname-arch.c,
> its man page depends on the arch binary.
> 
> Reported downstream by Rodrigues Goldwyn <rgoldwyn@suse.com> in
> https://build.opensuse.org/request/show/415172
> ---
>  man/local.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/man/local.mk b/man/local.mk
> index f41d969..7b72d3b 100644
> --- a/man/local.mk
> +++ b/man/local.mk
> @@ -56,10 +56,10 @@ if SINGLE_BINARY
>  mandeps += src/coreutils$(EXEEXT)
>  else
>  # Most prog.1 man pages depend on src/prog.  List the exceptions:
> -man/arch.1:      src/uname$(EXEEXT)
>  man/install.1:   src/ginstall$(EXEEXT)
>  man/test.1:      src/[$(EXEEXT)
>  
> +man/arch.1:      src/arch$(EXEEXT)
>  man/base32.1:    src/base32$(EXEEXT)
>  man/base64.1:    src/base64$(EXEEXT)
>  man/basename.1:  src/basename$(EXEEXT)

+1 thanks!



From MAILER-DAEMON Wed Jul 27 05:35:46 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSLFi-0007A5-Iq
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 05:35:46 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:43080)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSLFg-00079z-Dw
	for coreutils@gnu.org; Wed, 27 Jul 2016 05:35:45 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSLFb-0003qm-D6
	for coreutils@gnu.org; Wed, 27 Jul 2016 05:35:44 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:55032)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSLFb-0003qZ-7C
	for coreutils@gnu.org; Wed, 27 Jul 2016 05:35:39 -0400
Received: from [192.168.1.80] (unknown [109.76.143.177])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 83D83949C;
	Wed, 27 Jul 2016 10:35:37 +0100 (IST)
Subject: Re: [PATCH 2/2] maint: pacify GCC 6 with -Wnull-dereference
To: Bernhard Voelker <mail@bernhard-voelker.de>, coreutils@gnu.org
References: <20160727073319.30422-1-mail@bernhard-voelker.de>
	<20160727073319.30422-2-mail@bernhard-voelker.de>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <57988069.8090401@draigBrady.com>
Date: Wed, 27 Jul 2016 10:35:37 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <20160727073319.30422-2-mail@bernhard-voelker.de>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 09:35:45 -0000

On 27/07/16 08:33, Bernhard Voelker wrote:
> src/id.c:249:29: error: potential null pointer dereference \
> [-Werror=3Dnull-dereference]
>        pw_name =3D xstrdup (pwd->pw_name);
>                           ~~~^~~~~~~~~
> src/whoami.c:89:11: error: potential null pointer dereference \
> [-Werror=3Dnull-dereference]
>    puts (pw->pw_name);
>          ~~^~~~~~~~~
>=20
> * src/id.c (main): Explicitly exit with EXIT_FAILURE after an eror to
> help gcc-6 to detect that the dereferenced pointer is valid.
> * src/whoami.c (main): Likewise.
> ---
>  src/id.c     | 5 ++++-
>  src/whoami.c | 7 +++++--
>  2 files changed, 9 insertions(+), 3 deletions(-)
>=20
> diff --git a/src/id.c b/src/id.c
> index 218ee5a..35cbeb5 100644
> --- a/src/id.c
> +++ b/src/id.c
> @@ -245,7 +245,10 @@ main (int argc, char **argv)
>              }
>          }
>        if (pwd =3D=3D NULL)
> -        error (EXIT_FAILURE, 0, _("%s: no such user"), quote (spec));
> +        {
> +          error (0, 0, _("%s: no such user"), quote (spec));
> +          exit (EXIT_FAILURE);
> +        }
>        pw_name =3D xstrdup (pwd->pw_name);
>        ruid =3D euid =3D pwd->pw_uid;
>        rgid =3D egid =3D pwd->pw_gid;
> diff --git a/src/whoami.c b/src/whoami.c
> index e58c575..972cd55 100644
> --- a/src/whoami.c
> +++ b/src/whoami.c
> @@ -84,8 +84,11 @@ main (int argc, char **argv)
>    uid =3D geteuid ();
>    pw =3D (uid =3D=3D NO_UID && errno ? NULL : getpwuid (uid));
>    if (!pw)
> -    error (EXIT_FAILURE, errno, _("cannot find name for user ID %lu"),
> -           (unsigned long int) uid);
> +    {
> +      error (0, errno, _("cannot find name for user ID %lu"),
> +             (unsigned long int) uid);
> +      exit (EXIT_FAILURE);
> +    }
>    puts (pw->pw_name);
>    return EXIT_SUCCESS;
>  }
>=20

Hrm, I wonder should we have an errorx to wrap error (EXIT_FAILURE, ...)
and that can be marked as noreturn? I.E. stick this somewhere?

#include <assert.h>
#include "verror.h"
static inline void
_GL_ATTRIBUTE_FORMAT ((__printf__, 3, 4))
ATTRIBUTE_NORETURN
errorx (int status, int errnum, const char *fmt, ...)
{
  assert (status);
  va_list ap;
  va_start (ap, fmt);
  verror (status, errnum, fmt, ap);
  va_end (ap);
}



From MAILER-DAEMON Wed Jul 27 06:40:19 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSMGB-00044J-S2
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 06:40:19 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:54996)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSMGA-00043f-KG
	for coreutils@gnu.org; Wed, 27 Jul 2016 06:40:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSMG5-00045m-M4
	for coreutils@gnu.org; Wed, 27 Jul 2016 06:40:17 -0400
Received: from mout.kundenserver.de ([212.227.17.13]:61179)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bSMG5-00045a-Bv
	for coreutils@gnu.org; Wed, 27 Jul 2016 06:40:13 -0400
Received: from [10.0.2.15] ([62.153.148.194]) by mrelayeu.kundenserver.de
	(mreue103) with ESMTPSA (Nemesis) id 0MVasZ-1boBbq1Zvj-00Z02Q;
	Wed, 27 Jul 2016 12:40:09 +0200
Subject: Re: [PATCH 2/2] maint: pacify GCC 6 with -Wnull-dereference
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>, coreutils@gnu.org
References: <20160727073319.30422-1-mail@bernhard-voelker.de>
	<20160727073319.30422-2-mail@bernhard-voelker.de>
	<57988069.8090401@draigBrady.com>
From: Bernhard Voelker <mail@bernhard-voelker.de>
Message-ID: <a6260364-c3f7-1e9a-77bf-610b6506a1a9@bernhard-voelker.de>
Date: Wed, 27 Jul 2016 12:40:08 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
	Thunderbird/45.2
MIME-Version: 1.0
In-Reply-To: <57988069.8090401@draigBrady.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit
X-Provags-ID: V03:K0:gF9tgc0fLiX9Fqzn7hWknmuAeqoPX/re6CbMoGjl6ZvdOcc73G/
	25esBqQKguXcXpraoXBXV7N216N7pqRCI6vriq5RIrJs1VG1WNVNC4sczlvBGF6oIeK+6LM
	0d0Zs5qhnPDVfWLHMKJkicqn5PAxZ+zsbpmmUbh+6fTIYvaqfvZx2C8J0P+/HmZP1FFGrIp
	lbF6kYhmlKe2i/7NYrNpg==
X-UI-Out-Filterresults: notjunk:1;V01:K0:ncq4HYQSgvk=:fdVqv1kREvmskv4oSUv3VW
	PiHC94uRK2+qsd1CuJ1oDSt/o8KeMnGUy6xwOy6rRU6tTsv/CfxQjGfmEKE0zv5AllCy608JN
	PkOk8PZ1/seghdOATJUYf1xstj7mjZpWPLHvtgXvvaulAXNIhKu9V1/1Dyt5acN48HoYw52vH
	25uGvdwIrTt9jxPGzK6Xvwm2f6GCant0ivBvNc3gAyixJS+ESF1ZtbJvB3hKfUQsKHMgv4tAB
	vnD9LvSISBB38EKx1Gnu/Mv7l1QFqAGtGjAK3/6xvlfxKhKeq413wdTYQWfjN3L0nEK4GjV2v
	hRIbwhZUMBq7CJCExc2O4WMAXwBm04US50zee5b5A8ilHAiTGw8V/hL/Ixe65nTTqbzRPQ4jW
	zhxx/DQ8RHXpDvAMeDq7wt3Fd/5IFNsxvXIq/xZeHsOJkYLXY+y86Z2TF3b/yVZuFe2FE39bz
	UaFkrzkaJdmUxcUA4MnJag05p2W28Nx3RrXED6CX8HRqJ2LwRUGI1R/z7I0cB030xyVduG1GH
	MftBQ3h03vvsTaxIKSh6rbDFFvqby97Old1oAdxmPkHwOTmBg6uy+7N/0mlsUySm7V1ouSb+m
	RGphpfBqcoRvNWYgj+F9dPZPibHA7ZzgynJcFXq1sto9FR54biQCtR3E6dBDCEMLMc7/0O8BL
	wyvYc6muQE4z/MtvFnhQ4Q9K5dJfSoShsyDbeXnfpauQ0NSkNY/aGRd4xT0rTGSfC+NooVH0H
	Anu9x036id8cnCrc
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.17.13
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 10:40:19 -0000

On 07/27/2016 11:35 AM, Pádraig Brady wrote:
> Hrm, I wonder should we have an errorx to wrap error (EXIT_FAILURE, ...)
> and that can be marked as noreturn? I.E. stick this somewhere?

Well, it seems there are only a couple of places where error() is followed
by exit():

  $ git grep -FA2 'error (' | grep -F 'exit ('
  src/id.c-          exit (EXIT_FAILURE);
  src/ptx.c-  exit (EXIT_FAILURE);
  src/seq.c-  exit (EXIT_FAILURE);
  src/shred.c-  atexit (clear_random_data);
  src/sort.c-  exit (SORT_FAILURE);
  src/test.c-  test_exit (TEST_FAILURE);
  src/whoami.c-      exit (EXIT_FAILURE);

Therefore, I'd leave it more explicitly and readable rather than
inventing another wrapper to remember.

Have a nice day,
Berny


From MAILER-DAEMON Wed Jul 27 10:28:08 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSPoe-0007e1-IA
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 10:28:08 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:52359)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSPoc-0007bv-9U
	for coreutils@gnu.org; Wed, 27 Jul 2016 10:28:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSPoX-0004d8-UP
	for coreutils@gnu.org; Wed, 27 Jul 2016 10:28:05 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:55844)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bSPoX-0004cN-OL
	for coreutils@gnu.org; Wed, 27 Jul 2016 10:28:01 -0400
Received: from [192.168.1.80] (unknown [109.76.143.177])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id D5579949C;
	Wed, 27 Jul 2016 15:27:57 +0100 (IST)
Subject: Re: [PATCH 2/2] maint: pacify GCC 6 with -Wnull-dereference
To: Bernhard Voelker <mail@bernhard-voelker.de>, coreutils@gnu.org
References: <20160727073319.30422-1-mail@bernhard-voelker.de>
	<20160727073319.30422-2-mail@bernhard-voelker.de>
	<57988069.8090401@draigBrady.com>
	<a6260364-c3f7-1e9a-77bf-610b6506a1a9@bernhard-voelker.de>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <5798C4ED.2060000@draigBrady.com>
Date: Wed, 27 Jul 2016 15:27:57 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <a6260364-c3f7-1e9a-77bf-610b6506a1a9@bernhard-voelker.de>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 14:28:07 -0000

On 27/07/16 11:40, Bernhard Voelker wrote:
> On 07/27/2016 11:35 AM, P=E1draig Brady wrote:
>> Hrm, I wonder should we have an errorx to wrap error (EXIT_FAILURE, ..=
.)
>> and that can be marked as noreturn? I.E. stick this somewhere?
>=20
> Well, it seems there are only a couple of places where error() is follo=
wed
> by exit():
>=20
>   $ git grep -FA2 'error (' | grep -F 'exit ('
>   src/id.c-          exit (EXIT_FAILURE);
>   src/ptx.c-  exit (EXIT_FAILURE);
>   src/seq.c-  exit (EXIT_FAILURE);
>   src/shred.c-  atexit (clear_random_data);
>   src/sort.c-  exit (SORT_FAILURE);
>   src/test.c-  test_exit (TEST_FAILURE);
>   src/whoami.c-      exit (EXIT_FAILURE);
>=20
> Therefore, I'd leave it more explicitly and readable rather than
> inventing another wrapper to remember.
>=20
> Have a nice day,
> Berny
>=20
+1


From MAILER-DAEMON Wed Jul 27 11:11:11 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSQUJ-00043l-4R
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 11:11:11 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:34879)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bSQUF-00040e-6f
	for coreutils@gnu.org; Wed, 27 Jul 2016 11:11:08 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bSQU9-00009A-PK
	for coreutils@gnu.org; Wed, 27 Jul 2016 11:11:06 -0400
Received: from mx1.redhat.com ([209.132.183.28]:55084)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bSQU9-00008k-Fk
	for coreutils@gnu.org; Wed, 27 Jul 2016 11:11:01 -0400
Received: from int-mx10.intmail.prod.int.phx2.redhat.com
	(int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.redhat.com (Postfix) with ESMTPS id 78ADC8B137;
	Wed, 27 Jul 2016 15:11:00 +0000 (UTC)
Received: from [10.3.116.185] (ovpn-116-185.phx2.redhat.com [10.3.116.185])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id u6RFAxB1019685; Wed, 27 Jul 2016 11:10:59 -0400
Subject: Re: grep/sed and some strange patterns/inputs
To: Christoph Anton Mitterer <calestyo@scientia.net>, coreutils@gnu.org
References: <1469588998.8048.18.camel@scientia.net>
From: Eric Blake <eblake@redhat.com>
Openpgp: url=http://people.redhat.com/eblake/eblake.gpg
Organization: Red Hat, Inc.
Message-ID: <5798CECB.3070808@redhat.com>
Date: Wed, 27 Jul 2016 09:10:03 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <1469588998.8048.18.camel@scientia.net>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="iUUP2JsBb4GfMr877PWu2kVwMWUOiVe1P"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(mx1.redhat.com [10.5.110.28]);
	Wed, 27 Jul 2016 15:11:00 +0000 (UTC)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 209.132.183.28
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 15:11:08 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--iUUP2JsBb4GfMr877PWu2kVwMWUOiVe1P
From: Eric Blake <eblake@redhat.com>
To: Christoph Anton Mitterer <calestyo@scientia.net>, coreutils@gnu.org
Message-ID: <5798CECB.3070808@redhat.com>
Subject: Re: grep/sed and some strange patterns/inputs
References: <1469588998.8048.18.camel@scientia.net>
In-Reply-To: <1469588998.8048.18.camel@scientia.net>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 07/26/2016 09:09 PM, Christoph Anton Mitterer wrote:
> Hey.
>=20
> I've always had the impression that ^ and $ were the end/begin anchor
> of the current pattern, and since e.g. grep/sed work normally in terms
> of lines the start/end of lines.
>=20
> What I found a bit strange is that e.g.:
> printf '' | sed 's/^/foo/'
> printf '' | sed 's/$/foo/'
> printf '' | sed 's/^$/foo/'
> doesn't produce foo and that e.g.
> printf '' | grep '^'
> printf '' | grep '$'
> printf '' | grep '^$'
> printf '' | grep '*'
> don't match.

Line-oriented programs, when presented with 0 lines on input, have 0
lines to check for and therefore nothing to output.

>=20
> Why? Or better said, which part of POSIX mandates this? Or is it simply=

> "no stdin, nothing happens"?

There IS stdin, just that stdin had 0 lines (it reached end-of-file
immediately).

> OTOH, other tools do operate on that:
> $ printf ''  | wc
>       0       0       0
>=20
>=20
> What looks IMO also a bit strange:
> $ printf 'f' | sed "s/$/foo/"
> foof

This is unspecified by POSIX.  POSIX says that sed is only required to
operate on text files, but it also states that a text file is REQUIRED
to end in a newline if it is larger than 0 bytes.  Your input doesn't
end in a newline, therefore it is not a text file, therefore the output
of sed is unspecified.  Perhaps this is a bug in quality of
implementation of the sed implementation you are using that it could do
better by outputting 'ffoo' instead of 'foof', but that's a bug report
for the sed list, not coreutils.

> $printf 'f' | sed "s/$/foo/"
> ffoo
> (with no newlines)

I'm not sure I follow your example here; you have two identical command
lines with different results.  Did you copy-and-paste one incorrectly
from what you were trying to ask?

> So they do match,
> even though there is no \n character.
> OTOH
> $ printf 'f\n' | sed
> "s/$/foo/"
> foof
> $printf 'f\n' | sed "s/$/foo/"
> ffoo

Again, your example is not clear on what you were intending to report.
But once there is a newline, then the behavior is no longer unspecified,
and you correctly get:

$ printf 'f\n' | sed "s/$/foo/"
ffoo

> give the same, so while
> there clearly is a \n now (and thus I'd assume a
> new line) it's not
> matched.
> However:
> $ printf 'f\n\n' | sed "s/^/foo/"
> foof
> foo
> here it
> matches,... so it seems it would just ignore trailing non-\n-ended
> and
> empty lines, but NOT trailing (or only) non-\n-ended but non-empty
> lines
> .

I'm not sure what you're trying to ask here, but the point is that sed
tries to match all lines, and with two \n characters in the input, you
have two lines where it can match.

--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--iUUP2JsBb4GfMr877PWu2kVwMWUOiVe1P
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJXmM7LAAoJEKeha0olJ0Nq6P0IAK8bh0osGvdlmIM9hhS+u6dA
7IPD0s7mT7EoCQVhx/0i/dXbfE2UBgG4YmCRtzhPVbHxxITqpp8k+vzO6tV15iy/
/RuuxhFuzoDEmYj9zePrFbK/bxum10fJSxkTvQb0IYHYs3Wr29BRo4BS8I70Q51F
sXoGGI3gQv/EB83AUbgRa0JkDPeGfqiU9QEAa8sIDemH01Kl/ziGsf1dus+eT3uP
9SpJzF1tlaDWL+/x7s9Lqw1GEVOWvoawwUvXBnCT9rZiHfTdOWB0OCQt6jIUxiDq
IoashXkqoxpUHugibFG9Elb617szuquPG3j8h4sgPddtaG4GMvPIijfSgIMaPic=
=9czC
-----END PGP SIGNATURE-----

--iUUP2JsBb4GfMr877PWu2kVwMWUOiVe1P--


From MAILER-DAEMON Wed Jul 27 12:16:12 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSRVD-0002gJ-PL
	for mharc-coreutils@gnu.org; Wed, 27 Jul 2016 12:16:11 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:33296)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <calestyo@scientia.net>) id 1bSQL0-00012q-Kl
	for coreutils@gnu.org; Wed, 27 Jul 2016 11:01:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <calestyo@scientia.net>) id 1bSQKw-0006aq-8Y
	for coreutils@gnu.org; Wed, 27 Jul 2016 11:01:33 -0400
Received: from mailgw-02.dd24.net ([193.46.215.43]:41499)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <calestyo@scientia.net>) id 1bSQKv-0006aQ-Ui
	for coreutils@gnu.org; Wed, 27 Jul 2016 11:01:30 -0400
Received: from mailpolicy-01.live.igb.homer.key-systems.net
	(mailpolicy-02.live.igb.homer.key-systems.net [192.168.1.27])
	by mailgw-02.dd24.net (Postfix) with ESMTP id 79B175FC9A
	for <coreutils@gnu.org>; Wed, 27 Jul 2016 15:01:28 +0000 (UTC)
X-Virus-Scanned: Debian amavisd-new at
	mailpolicy-02.live.igb.homer.key-systems.net
Received: from mailgw-02.dd24.net ([192.168.1.36])
	by mailpolicy-01.live.igb.homer.key-systems.net
	(mailpolicy-02.live.igb.homer.key-systems.net [192.168.1.25])
	(amavisd-new, port 10236)
	with ESMTP id cxgSysI11o4S for <coreutils@gnu.org>;
	Wed, 27 Jul 2016 15:01:22 +0000 (UTC)
Received: from heisenberg.fritz.box (ppp-46-244-216-201.dynamic.mnet-online.de
	[46.244.216.201])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mailgw-02.dd24.net (Postfix) with ESMTPSA
	for <coreutils@gnu.org>; Wed, 27 Jul 2016 15:01:22 +0000 (UTC)
Message-ID: <1469631682.11993.33.camel@scientia.net>
Subject: grep/sed and some strange patterns/inputs
From: Christoph Anton Mitterer <calestyo@scientia.net>
To: coreutils@gnu.org
Date: Wed, 27 Jul 2016 17:01:22 +0200
Content-Type: multipart/signed; micalg="sha-512";
	protocol="application/x-pkcs7-signature";
	boundary="=-GG6ckaF4cg6SzDQIRD/y"
X-Mailer: Evolution 3.20.4-1 
Mime-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 193.46.215.43
X-Mailman-Approved-At: Wed, 27 Jul 2016 12:16:11 -0400
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Jul 2016 15:01:40 -0000


--=-GG6ckaF4cg6SzDQIRD/y
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hey.

I've always had the impression that ^ and $ were the end/begin anchor
of the current pattern, and since e.g. grep/sed work normally in terms
of lines the start/end of lines.

What I found a bit strange is that e.g.:
printf '' | sed 's/^/foo/'
printf '' | sed 's/$/foo/'
printf '' | sed 's/^$/foo/'
doesn't produce foo and that e.g.
printf '' | grep '^'
printf '' | grep '$'
printf '' | grep '^$'
printf '' | grep '*'
don't match.

Why? Or better said, which part of POSIX mandates this? Or is it simply
"no stdin, nothing happens"?
OTOH, other=C2=A0tools do operate on that:
$ printf '' =C2=A0| wc
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A00=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00


What looks IMO also a bit strange:
$ printf 'f' | sed "s/$/foo/"
foof
$printf 'f' | sed "s/$/foo/"
ffoo
(with no newlines)
So they do match,
even though there is no \n character.
OTOH
$ printf 'f\n' | sed
"s/$/foo/"
foof
$printf 'f\n' | sed "s/$/foo/"
ffoo
give the same, so while
there clearly is a \n now (and thus I'd assume a
new line) it's not
matched.
However:
$ printf 'f\n\n' | sed "s/^/foo/"
foof
foo
here it
matches,... so it seems it would just ignore trailing non-\n-ended
and
empty lines, but NOT trailing (or only) non-\n-ended but non-empty
lines
.
Seems a bit inconsequential.


Any ideas? Thanks,
Chris.
--=-GG6ckaF4cg6SzDQIRD/y
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Disposition: attachment; filename="smime.p7s"
Content-Transfer-Encoding: base64

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCCEmow
ggXiMIIDyqADAgECAhBctkhuRwyYxn/2gNtkSuKNMA0GCSqGSIb3DQEBCwUAMH0xCzAJBgNVBAYT
AklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0
aWZpY2F0ZSBTaWduaW5nMSkwJwYDVQQDEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTAeFw0xNTEyMTYwMTAwMDVaFw0zMDEyMTYwMTAwMDVaMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQK
Ew1TdGFydENvbSBMdGQuMSkwJwYDVQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0
eTEjMCEGA1UEAxMaU3RhcnRDb20gQ2xhc3MgMiBDbGllbnQgQ0EwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDuD1CMlQkjhKz1UGqP1jeiTiH9MgllRz6vOPrVG/eE0H/J4QQLV/PeL8RT
4xc44bEzsoJu0IhwnEchb+TxE/qw88w7hxODuw3N8Faxix6a1jp83+RWvZHZf78+O+3GYBpekZfT
Oe9A/FoTXbcgwZfLTMQodn+ckNnX31M/1M2f2/7VA7QBlvihontyHQOlIlryQXnGI0UMCD21oopK
tW48ckv0wUVg8irBKGMeD65gTON/Fsw/ZBbBqadoD1jt85FIM1ql24WUBEBwO1d0ykCKOIbgcqes
3fbcjQpruUNMBbIu1MMIRMqwjx/M7IvSKcS7VYRWl0/K2byzWvBAHh/1AgMBAAGjggFkMIIBYDAO
BgNVHQ8BAf8EBAMCAQYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMBIGA1UdEwEB/wQI
MAYBAf8CAQAwMgYDVR0fBCswKTAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2Eu
Y3JsMGYGCCsGAQUFBwEBBFowWDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3Auc3RhcnRzc2wuY29t
MDAGCCsGAQUFBzAChiRodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9jYS5jcnQwHQYDVR0O
BBYEFJmXqxg1OotZRUOYsnJxyPT7Cc8WMB8GA1UdIwQYMBaAFE4L7xqkQFulF2mHMMo0aEPQQa7y
MD8GA1UdIAQ4MDYwNAYEVR0gADAsMCoGCCsGAQUFBwIBFh5odHRwOi8vd3d3LnN0YXJ0c3NsLmNv
bS9wb2xpY3kwDQYJKoZIhvcNAQELBQADggIBAJlBQQTO9iT1TjA0eEO7V/1AbBvMAfibjAgofxmJ
01jBmHKg0pbTjWInTTYoxb3LBgz3mfjvvS1PjnIfb29MyVm0G/PSHjgq7Ews1dEJMPC9XTuxPf2c
+MWLkynBlotW542JprW+iTWfZafyUtzIKW1hk0YASJ8zSSj8D++9yR+0UhkbvlECJkdi1+et0EaI
7HIX6ccj1rfcFFflWX/fPT64dn9jpg9s0nuJug4WsVkEK236WndZoMHrZmgF7CIyZ3T0muqYwkAS
DDcRt9A21o/Mc+D8Q6GVmKRGB3gEKvOtsioHZEqJv6CdAm3a6gloo5pX3RL1eCzc8Lzfs4T0ISZj
r8xNMbTGlsuHaFH+stDewKsfnpo4N64OtAGrzmVfFFsMIRENRHsVlSEe/6LVpBpjn00+7bqEN3qe
qSxIOraYJ12mJ08G4YnP2U1fadHIaS50O5ZXqAivoBl9pi/6CBNc5wIMlkXMyFZ0sLsI+9ErFDu9
OJhX7iWCo69X0ydlzXj08+2K1PKyr+g6/vTPHur63JotNrhyoWEJyWEjBLA4QmJXfGpB5u+bCwNf
sFpcYAlkiFt1Rs2vemgSBy3q4DHDbPKvr1YImZNRSHP+TX9NP94JRkmqcyD/qaN7u2JV1pRAlwG0
npyrn6ZQM1QHV+iFbwQ4VqLMWk//QVo/rZy1MIIGPjCCBSagAwIBAgIQPZ6tDP/RHLwcz1F+2kRA
DzANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjEp
MCcGA1UECxMgU3RhcnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxIzAhBgNVBAMTGlN0YXJ0
Q29tIENsYXNzIDIgQ2xpZW50IENBMB4XDTE2MDEwMjAyMDIzNloXDTE4MDEwMjAyMDIzNlowejEL
MAkGA1UEBhMCREUxDzANBgNVBAgMBkJheWVybjERMA8GA1UEBwwITcO8bmNoZW4xITAfBgNVBAMM
GENocmlzdG9waCBBbnRvbiBNaXR0ZXJlcjEkMCIGCSqGSIb3DQEJARYVY2FsZXN0eW9Ac2NpZW50
aWEubmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsoxs789VtR81tlqeP2vwE3Cp
YosZYRfrovj+Wb91paoV5OQcFuXA7yONnLsWxOnJ3AGgy1/wG79Ko+yHSmF2K88pEWjrFe0zmZWZ
0b2OtMoKLEjxz7Nz2N7/lmgss8XinJ083LwLKSe4f+JLjnF5hX/g6wG4NkFXVOIf2YFZ0+c1NYFQ
T6Vy42EWdk+JLDoXjyd2hEperLCz9rQr2k3wh0cn8R1FDUGerBGpMnvyKV6JXj5vZsqJnmiiId9e
0IdBj4Wcs3hT2usk6dJYwHrZ8b7Fkv6BRq90DUI7i8+ukTdp5hnAb8TPSbe1JS4h3Jc7r2NVLkpo
Gbuw98er9s2PEAvpECD3toO3ojBevL56vEDbGa5tXpvvcWCz6t3QyEKJ2E54hWw1fbUA0BtydQC0
mhKBtrENVnnCDrSgiyBMnd8y+kw6iPKeJEjNObGubc4BZp47zPn/ZiDBaHue6S0MXsoS7XEGOuCU
09S/8kD8wTecBg+KdlLSNkLm+xqx6Cy7zwcj3IuQv0PeWi4kfEkAJfl8IeAP/4049iA6a1rkhJl7
1rDjpEGBCG+i9BAgBeNdHLtQ7bugdvU+GfHjFXEU1emUk91E2liML8kpCvhUAXcLhhL8bxPJiX2/
VQ2esrpQVO56OzpivNauonLaWmHEKOSfPC3s5AKMytMTiUiszEUCAwEAAaOCAcMwggG/MAsGA1Ud
DwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwCQYDVR0TBAIwADAdBgNVHQ4E
FgQUwRklxF1ZV2ooSMkFwLiK13apkK0wHwYDVR0jBBgwFoAUmZerGDU6i1lFQ5iycnHI9PsJzxYw
bwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5jb20wOQYI
KwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5jbGllbnQyLmNydDA4
BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2EtY2xpZW50Mi5jcmww
IAYDVR0RBBkwF4EVY2FsZXN0eW9Ac2NpZW50aWEubmV0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cu
c3RhcnRzc2wuY29tLzBUBgNVHSAETTBLMAwGCisGAQQBgbU3BgEwOwYLKwYBBAGBtTcBAgQwLDAq
BggrBgEFBQcCARYeaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5MA0GCSqGSIb3DQEBCwUA
A4IBAQBb8EIsoUN/tgUgQqrNXNtlksPep84kx5yRBgr71uf3ITLddGtzfDBj6KdZUoy7SG3MQkyO
kvYmXBClj23rv8Iol48/3oi9XWZw5EV3uHrRse1TzQgMPZE0hZsDgkqXVoxMQfk55ndjZIHMfSkn
hdnqSP5zZ4TCmDEKLppPMDcQSSrjilnbthpxlIJzeGeEFtrh6ssh/oF6mUaEGFcd8kx9RS51K1gt
H4J36y4E6pKB7EdxG2+0yVzAIta8dkD/BiMCKRWhp1EmzQ2uIh2nX5y8t4e2xHHuiy11Yeq6UTW/
JmfxF4xcbkNK/rv9ISBg9K+mZtg2QXr+P/CJBtej2RLUMIIGPjCCBSagAwIBAgIQPZ6tDP/RHLwc
z1F+2kRADzANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20g
THRkLjEpMCcGA1UECxMgU3RhcnRDb20gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxIzAhBgNVBAMT
GlN0YXJ0Q29tIENsYXNzIDIgQ2xpZW50IENBMB4XDTE2MDEwMjAyMDIzNloXDTE4MDEwMjAyMDIz
NlowejELMAkGA1UEBhMCREUxDzANBgNVBAgMBkJheWVybjERMA8GA1UEBwwITcO8bmNoZW4xITAf
BgNVBAMMGENocmlzdG9waCBBbnRvbiBNaXR0ZXJlcjEkMCIGCSqGSIb3DQEJARYVY2FsZXN0eW9A
c2NpZW50aWEubmV0MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAsoxs789VtR81tlqe
P2vwE3CpYosZYRfrovj+Wb91paoV5OQcFuXA7yONnLsWxOnJ3AGgy1/wG79Ko+yHSmF2K88pEWjr
Fe0zmZWZ0b2OtMoKLEjxz7Nz2N7/lmgss8XinJ083LwLKSe4f+JLjnF5hX/g6wG4NkFXVOIf2YFZ
0+c1NYFQT6Vy42EWdk+JLDoXjyd2hEperLCz9rQr2k3wh0cn8R1FDUGerBGpMnvyKV6JXj5vZsqJ
nmiiId9e0IdBj4Wcs3hT2usk6dJYwHrZ8b7Fkv6BRq90DUI7i8+ukTdp5hnAb8TPSbe1JS4h3Jc7
r2NVLkpoGbuw98er9s2PEAvpECD3toO3ojBevL56vEDbGa5tXpvvcWCz6t3QyEKJ2E54hWw1fbUA
0BtydQC0mhKBtrENVnnCDrSgiyBMnd8y+kw6iPKeJEjNObGubc4BZp47zPn/ZiDBaHue6S0MXsoS
7XEGOuCU09S/8kD8wTecBg+KdlLSNkLm+xqx6Cy7zwcj3IuQv0PeWi4kfEkAJfl8IeAP/4049iA6
a1rkhJl71rDjpEGBCG+i9BAgBeNdHLtQ7bugdvU+GfHjFXEU1emUk91E2liML8kpCvhUAXcLhhL8
bxPJiX2/VQ2esrpQVO56OzpivNauonLaWmHEKOSfPC3s5AKMytMTiUiszEUCAwEAAaOCAcMwggG/
MAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwQwCQYDVR0TBAIwADAd
BgNVHQ4EFgQUwRklxF1ZV2ooSMkFwLiK13apkK0wHwYDVR0jBBgwFoAUmZerGDU6i1lFQ5iycnHI
9PsJzxYwbwYIKwYBBQUHAQEEYzBhMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5zdGFydHNzbC5j
b20wOQYIKwYBBQUHMAKGLWh0dHA6Ly9haWEuc3RhcnRzc2wuY29tL2NlcnRzL3NjYS5jbGllbnQy
LmNydDA4BgNVHR8EMTAvMC2gK6AphidodHRwOi8vY3JsLnN0YXJ0c3NsLmNvbS9zY2EtY2xpZW50
Mi5jcmwwIAYDVR0RBBkwF4EVY2FsZXN0eW9Ac2NpZW50aWEubmV0MCMGA1UdEgQcMBqGGGh0dHA6
Ly93d3cuc3RhcnRzc2wuY29tLzBUBgNVHSAETTBLMAwGCisGAQQBgbU3BgEwOwYLKwYBBAGBtTcB
AgQwLDAqBggrBgEFBQcCARYeaHR0cDovL3d3dy5zdGFydHNzbC5jb20vcG9saWN5MA0GCSqGSIb3
DQEBCwUAA4IBAQBb8EIsoUN/tgUgQqrNXNtlksPep84kx5yRBgr71uf3ITLddGtzfDBj6KdZUoy7
SG3MQkyOkvYmXBClj23rv8Iol48/3oi9XWZw5EV3uHrRse1TzQgMPZE0hZsDgkqXVoxMQfk55ndj
ZIHMfSknhdnqSP5zZ4TCmDEKLppPMDcQSSrjilnbthpxlIJzeGeEFtrh6ssh/oF6mUaEGFcd8kx9
RS51K1gtH4J36y4E6pKB7EdxG2+0yVzAIta8dkD/BiMCKRWhp1EmzQ2uIh2nX5y8t4e2xHHuiy11
Yeq6UTW/JmfxF4xcbkNK/rv9ISBg9K+mZtg2QXr+P/CJBtej2RLUMYIEfjCCBHoCAQEwgYkwdTEL
MAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKTAnBgNVBAsTIFN0YXJ0Q29tIENl
cnRpZmljYXRpb24gQXV0aG9yaXR5MSMwIQYDVQQDExpTdGFydENvbSBDbGFzcyAyIENsaWVudCBD
QQIQPZ6tDP/RHLwcz1F+2kRADzANBglghkgBZQMEAgMFAKCCAcUwGAYJKoZIhvcNAQkDMQsGCSqG
SIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTYwNzI3MTUwMTIyWjBPBgkqhkiG9w0BCQQxQgRAFoS1
Sxl1XRrC0wIwP4kaOSR1X2bQvZuvllUDYxWfmbxZ9Xtte9o1adzuW5PpTq0jHW08AaNnPR7G0psN
jCJR9jCBmgYJKwYBBAGCNxAEMYGMMIGJMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENv
bSBMdGQuMSkwJwYDVQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEjMCEGA1UE
AxMaU3RhcnRDb20gQ2xhc3MgMiBDbGllbnQgQ0ECED2erQz/0Ry8HM9RftpEQA8wgZwGCyqGSIb3
DQEJEAILMYGMoIGJMHUxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSkwJwYD
VQQLEyBTdGFydENvbSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTEjMCEGA1UEAxMaU3RhcnRDb20g
Q2xhc3MgMiBDbGllbnQgQ0ECED2erQz/0Ry8HM9RftpEQA8wDQYJKoZIhvcNAQEBBQAEggIAC+WX
Cwr61Wcf+h2KyYFCC4qouYPPdaqmnbTQIIrbG5C34F4f6s+tzAux7ItSZosb09O/YcwmvqV0m0Eo
pSMgAmf0fp5uRemW0mAGn7wq7ucbdQ4ClsSGbl95pJMRPiWnvG5xfO45Cwkym4yATcOXQXPGqxbF
W9bGVUaf9JiH7BEpvpTegGUsIk2G2gJc9hziHmBpgBkoK6s9bh4ClenHOLkGm5y9XW+O+bQGq3qf
gQEubyonBM3orsAnk635sNSCSQNbHKwMAzji5r8Y/pFRbtkTw7fOJI5sYcGW2ZGb6AV389pSeg7G
P0WmWL0GTgpTrJ9JBn6guCkDymgXn01INc3OiwwdooufTnmkDx41Tkeyy+ZW4ocJ9FHL21ZjROt2
J+l0re8BSmC8Bzw3rixohj+AURWC/eMURiGxo68CEu4Rk9F4Mnc2I3GQZshKLgbpB8zwxFiLuNVJ
XUnE3YN6kxifCYmEuVEqoP5WmRrHrMdRMa3g27oNwpAOhWkV2l1X+Uby+VNm3CXAi0j0+h8HOlj/
+j+eku0aSqHF6Z3DyLn/DJaSl+bcUR3ZOtf9a2r1w8JCHUKFoP/oqi1GBlGALCSSwljzX8eewskC
NWE/nz23JMFnLAL/13ZwMk8HEhLoMrxElS7IkYeWSbB1+x4TjaDD6VPheWbkXxraVV/525kAAAAA
AAA=


--=-GG6ckaF4cg6SzDQIRD/y--



From MAILER-DAEMON Thu Jul 28 13:18:56 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bSoxU-0006Sv-Oe
	for mharc-coreutils@gnu.org; Thu, 28 Jul 2016 13:18:56 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55941)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bSoxR-0006Sg-T8
	for coreutils@gnu.org; Thu, 28 Jul 2016 13:18:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bSoxM-00060k-Vo
	for coreutils@gnu.org; Thu, 28 Jul 2016 13:18:52 -0400
Received: from mx1.redhat.com ([209.132.183.28]:55564)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1bSoxM-00060N-MN
	for coreutils@gnu.org; Thu, 28 Jul 2016 13:18:48 -0400
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mx1.redhat.com (Postfix) with ESMTPS id 5F2CF3B703;
	Thu, 28 Jul 2016 17:18:47 +0000 (UTC)
Received: from [10.3.116.90] (ovpn-116-90.phx2.redhat.com [10.3.116.90])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id u6SHIkMh019137; Thu, 28 Jul 2016 13:18:46 -0400
Subject: Re: multibyte processing - handling invalid sequences (long)
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>,
	Assaf Gordon <assafgordon@gmail.com>, Coreutils <coreutils@gnu.org>
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
	<578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
From: Eric Blake <eblake@redhat.com>
Openpgp: url=http://people.redhat.com/eblake/eblake.gpg
Organization: Red Hat, Inc.
Message-ID: <579A3E75.5080400@redhat.com>
Date: Thu, 28 Jul 2016 11:18:45 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <578F76F3.1070206@redhat.com>
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="J974IWueROCmUL0KlG2oDx5w3If0p3dfw"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16
	(mx1.redhat.com [10.5.110.30]);
	Thu, 28 Jul 2016 17:18:47 +0000 (UTC)
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 209.132.183.28
X-Content-Filtered-By: Mailman/MimeDel 2.1.21
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 28 Jul 2016 17:18:55 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--J974IWueROCmUL0KlG2oDx5w3If0p3dfw
From: Eric Blake <eblake@redhat.com>
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>,
 Assaf Gordon <assafgordon@gmail.com>, Coreutils <coreutils@gnu.org>
Message-ID: <579A3E75.5080400@redhat.com>
Subject: Re: multibyte processing - handling invalid sequences (long)
References: <D9117DC1-6E34-48B6-9FED-B958B21938B0@gmail.com>
 <578F6CAD.8080205@draigBrady.com> <578F76F3.1070206@redhat.com>
In-Reply-To: <578F76F3.1070206@redhat.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 07/20/2016 07:04 AM, Eric Blake wrote:
> On 07/20/2016 06:21 AM, P=C3=A1draig Brady wrote:
>=20
>> It's worth considering having a separate (already existing?) util
>> to fix data before processing. That could have options to:
>>   drop invalid chars, replace with replacement char,
>>   apply various http://unicode.org/reports/tr15/#Norm_Forms,
>>   convert enclosed forms like =E3=8A=B7 to 42 etc.
>> I.E. we should avoid complicating each util where possible,
>> and at least avoid having options on each util that could be
>> hoisted to a more general util like above.
>>
>> Silently dropping invalid characters probably isn't a great idea,
>> and warnings to stderr is a bit messy and could be seen to contradict
>> POSIX which suggests exiting with failure if anything output to stderr=
=2E
>> A compromise might be to just replace invalid chars with
>> the replacement character =EF=BF=BD and then include that in
>> normal character processing, to make issues in input apparent.
>=20
> Since there are several plausible error-handling methods (silently
> discard invalid input, flag input as invalid with an error and no
> further output, convert invalid input into replacement character and
> proceed with output), all of which can be considered desirable in some
> circumstances, I wonder if we should give ALL utilities a common
> --encoding-error=3DPOLICY option that allows runtime selection between =
the
> three policies, and/or an environment variable that selects the default=

> policy in absence of a command line choice.

Interestingly enough, today's POSIX phone call started discussions on
how iconv() needs to be enhanced to support multiple error handling modes=
:

http://austingroupbugs.net/bug_view_page.php?bug_id=3D1007


--=20
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


--J974IWueROCmUL0KlG2oDx5w3If0p3dfw
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCAAGBQJXmj52AAoJEKeha0olJ0NqMQYH/jkLJBd3KwQZuAISFYs+gtuq
N+uSnHfLTVaEL3Kk5euLm8W7w/WwvF5ObuMdm2raM6ivSKSuO98hXDCOLEXLWT3d
lcyf1bqkfRLkQ0WkG2aA4fX+TbXBLOS3Ta8WwU4s7DVUaU4wS6xpEMJYzu7JtjqC
v0O6R1fQ8q6gnD0zeUSSwYLexUIpxs6TW+lwyYqZ2w+IiV1LIdHtAbvoaxyLlvNj
XDpb+dBjVP4R8x1paKIfb9y7/wWSAd6JNXoQ5HCRGogGoHBKiiHDjzGGRE7pFzzA
37BlR5FBVpSBbNeuZ5iXdt28OYsIQhUoGG0uJE7KBJguuJnpRpcGN2/wYgdIj8I=
=uCoP
-----END PGP SIGNATURE-----

--J974IWueROCmUL0KlG2oDx5w3If0p3dfw--


From MAILER-DAEMON Fri Jul 29 04:34:46 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bT3Fm-0001SD-FE
	for mharc-coreutils@gnu.org; Fri, 29 Jul 2016 04:34:46 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:40091)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assirati@if.usp.br>) id 1bSvlH-0005Ug-03
	for coreutils@gnu.org; Thu, 28 Jul 2016 20:34:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assirati@if.usp.br>) id 1bSvlD-0007bJ-RH
	for coreutils@gnu.org; Thu, 28 Jul 2016 20:34:47 -0400
Received: from gaivota.if.usp.br ([143.107.128.55]:39496 helo=mail.if.usp.br)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assirati@if.usp.br>) id 1bSvlD-0007b0-7H
	for coreutils@gnu.org; Thu, 28 Jul 2016 20:34:43 -0400
Received: by mail.if.usp.br (Postfix, from userid 33)
	id 26B5610C3B5A; Thu, 28 Jul 2016 21:34:42 -0300 (BRT)
Received: from nonada.if.usp.br (nonada.if.usp.br [143.107.131.187]) by
	correio2.if.usp.br (Horde Framework) with HTTP;
	Thu, 28 Jul 2016 21:34:42 -0300
Date: Thu, 28 Jul 2016 21:34:42 -0300
Message-ID: <20160728213442.Horde.LWjUgXabrQWA545QDq3CvA1@correio2.if.usp.br>
From: assirati@if.usp.br
To: coreutils@gnu.org
Subject: Patch: add basic package management  for 'install'
User-Agent: Internet Messaging Program (IMP) H5 (6.2.1)
Content-Type: multipart/mixed; boundary="=_o_YrhOGPIKigUaPvLO2wfQ1"
MIME-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x
X-Received-From: 143.107.128.55
X-Mailman-Approved-At: Fri, 29 Jul 2016 04:34:44 -0400
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Jul 2016 00:34:48 -0000

This message is in MIME format.

--=_o_YrhOGPIKigUaPvLO2wfQ1
Content-Type: text/plain; charset=UTF-8; format=flowed; DelSp=Yes
Content-Disposition: inline


Hello,

Attached is a patch that extends the functionality of Coreutils  
install creating a basic package management capability. If it is to be  
considered useful or viable, I would like very much to receive  
reveiews, suggestions for improvements or bug reports. My purpose is  
to get the code integrated to Coreutils 'install'.

Thank you,
Joao Luis.

** RATIONALE **

When installing programs from upstream source code, 'install' is the  
traditional way makefiles install files to /usr/local or other  
locations. After that, it is difficult to get a full listing of what  
was installed where, and usually one must keep the source code so it  
is possible to 'make uninstall' in the future, in case the makefile  
provides this possibility.

This patch allows 'install' to make a list of all installed files and  
created directories, making it easy to track all files and uninstall  
the "package".

** USAGE **

With this patch, when 'install' is executed with the command line options

-l FILE

or

--list=FILE

it will append to FILE the canonicalized name of the installed file or  
directory.

For example

install -l file_list -D my_file /usr/local/my_dir/my_file

or

install --list=file_list -D my_file /usr/local/my_dir/my_file

or

export INSTALL_LIST=file_list
install -D my_file /usr/local/my_dir/my_file

will append to file_list (or create it if it doesn't exist) containing

/usr/local/my_dir
/usr/local/my_dir/my_file

After that, files and directories may be easily uninstalled with

tac file_list | while read f; do test -d "$f" && \rmdir "$f" || \rm  
"$f"; done; \rm file_list

** USE CASES **

Install may be called in new makefiles with the option -l to generate  
a list of installed files and direcory. For legacy makefiles, one can  
execute

INSTALL_LIST=somefile make install

or

export INSTALL_LIST=somefile
make install

or the like.

** DETAILS **

A global variable

static FILE *install_list = NULL;

was created to hold the stream for the file list. This stream is  
fopen()ed for append in main() and closed by the atexit() handler  
close_install_list().

A function

static void install_list_append (const char *fname)

was created to write to the file list. This function is called by:

make_ancestor(), to record all the created parent directories (case of  
option -D)

process_dir(), to record a created dir (case of option -d)

install_file_in_file(), to record installed files.



--=_o_YrhOGPIKigUaPvLO2wfQ1
Content-Type: text/x-diff; name=install-list.diff; charset=us-ascii
Content-Disposition: attachment; size=4702; filename=install-list.diff

--- src/install.c.ORIG	2016-07-28 15:01:31.115348097 -0300
+++ src/install.c	2016-07-28 20:52:18.193822263 -0300
@@ -18,6 +18,7 @@
 
 #include <config.h>
 #include <stdio.h>
+#include <stdlib.h>
 #include <getopt.h>
 #include <sys/types.h>
 #include <signal.h>
@@ -28,6 +29,7 @@
 
 #include "system.h"
 #include "backupfile.h"
+#include "canonicalize.h"
 #include "error.h"
 #include "cp-hash.h"
 #include "copy.h"
@@ -109,6 +111,9 @@
 /* Program used to strip binaries, "strip" is default */
 static char const *strip_program = "strip";
 
+/* Stream to record installed files and directories */
+static FILE *install_list = NULL;
+
 /* For long options that have no equivalent short option, use a
    non-character as a pseudo short option, starting with CHAR_MAX + 1.  */
 enum
@@ -124,6 +129,7 @@
   {GETOPT_SELINUX_CONTEXT_OPTION_DECL},
   {"directory", no_argument, NULL, 'd'},
   {"group", required_argument, NULL, 'g'},
+  {"list", required_argument, NULL, 'l'},
   {"mode", required_argument, NULL, 'm'},
   {"no-target-directory", no_argument, NULL, 'T'},
   {"owner", required_argument, NULL, 'o'},
@@ -409,6 +415,20 @@
   return is_a_dir;
 }
 
+/* Append fname to the install list */
+
+static void
+install_list_append (const char *fname)
+{
+  char *fname_canon;
+  if (install_list == NULL)
+    return;
+  fname_canon = canonicalize_file_name (fname);
+  if (fprintf (install_list, "%s\n", fname_canon) < 0)
+    error (EXIT_FAILURE, errno, _("cannot write to install list"));
+  free (fname_canon);
+}
+
 /* Report that directory DIR was made, if OPTIONS requests this.  */
 static void
 announce_mkdir (char const *dir, void *options)
@@ -432,7 +452,10 @@
 
   int r = mkdir (component, DEFAULT_MODE);
   if (r == 0)
-    announce_mkdir (dir, options);
+    {
+      install_list_append (component);
+      announce_mkdir (dir, options);
+    }
   return r;
 }
 
@@ -448,6 +471,10 @@
           ? EXIT_SUCCESS
           : EXIT_FAILURE);
 
+  /* After patch is reviewed and accepted, integrate with if below */
+  if (ret == EXIT_SUCCESS)
+    install_list_append (dir);
+
   /* FIXME: Due to the current structure of make_dir_parents()
      we don't have the facility to call defaultcon() before the
      final component of DIR is created.  So for now, create the
@@ -658,6 +685,9 @@
                         or all components of --target-directory,\n\
                         then copy SOURCE to DEST\n\
   -g, --group=GROUP   set group ownership, instead of process' current group\n\
+  -l, --list=FILE     append to FILE the installed file or directory names;\n\
+                        FILE may be passed also through the environment\n\
+                        variable INSTALL_LIST\n\
   -m, --mode=MODE     set permission mode (as in chmod), instead of rwxr-xr-x\n\
   -o, --owner=OWNER   set ownership (super-user only)\n\
 "), stdout);
@@ -715,6 +745,7 @@
     }
   if (! copy_file (from, to, x))
     return false;
+  install_list_append (to);
   if (strip_files)
     if (! strip (to))
       {
@@ -800,6 +831,12 @@
   return ret;
 }
 
+static void close_install_list (void)
+{
+  if (install_list && fclose (install_list) == EOF)
+    error (0, errno, _("cannot write to install list"));
+}
+
 int
 main (int argc, char **argv)
 {
@@ -817,6 +854,7 @@
   char **file;
   bool strip_program_specified = false;
   char const *scontext = NULL;
+  char *install_list_fname = NULL;
   /* set iff kernel has extra selinux system calls */
   selinux_enabled = (0 < is_selinux_enabled ());
 
@@ -827,6 +865,7 @@
   textdomain (PACKAGE);
 
   atexit (close_stdin);
+  atexit (close_install_list);
 
   cp_option_init (&x);
 
@@ -840,8 +879,8 @@
      we'll actually use backup_suffix_string.  */
   backup_suffix_string = getenv ("SIMPLE_BACKUP_SUFFIX");
 
-  while ((optc = getopt_long (argc, argv, "bcCsDdg:m:o:pt:TvS:Z", long_options,
-                              NULL)) != -1)
+  while ((optc = getopt_long (argc, argv, "bcCsDdg:l:m:o:pt:TvS:Z",
+			      long_options, NULL)) != -1)
     {
       switch (optc)
         {
@@ -878,6 +917,9 @@
         case 'g':
           group_name = optarg;
           break;
+	case 'l':
+	  install_list_fname = optarg;
+	  break;
         case 'm':
           specified_mode = optarg;
           break;
@@ -1012,6 +1054,13 @@
                quoteaf (file[n_files - 1]));
     }
 
+  if (install_list_fname || (install_list_fname = getenv ("INSTALL_LIST")))
+    {
+      install_list = fopen (install_list_fname, "a");
+      if (install_list == NULL)
+	error (EXIT_FAILURE, errno, _("failed to open %s"), install_list_fname);
+    }
+
   if (specified_mode)
     {
       struct mode_change *change = mode_compile (specified_mode);

--=_o_YrhOGPIKigUaPvLO2wfQ1--


From MAILER-DAEMON Fri Jul 29 06:31:28 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bT54h-0002I1-W1
	for mharc-coreutils@gnu.org; Fri, 29 Jul 2016 06:31:28 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:51792)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bT54Y-0002D4-S8
	for coreutils@gnu.org; Fri, 29 Jul 2016 06:31:25 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bT54S-000321-7c
	for coreutils@gnu.org; Fri, 29 Jul 2016 06:31:18 -0400
Received: from mail.magicbluesmoke.com ([82.195.144.49]:34648)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1bT54S-00030i-1E
	for coreutils@gnu.org; Fri, 29 Jul 2016 06:31:12 -0400
Received: from [192.168.1.80] (unknown [109.79.69.113])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 792AA4AF1;
	Fri, 29 Jul 2016 11:31:07 +0100 (IST)
Subject: Re: Patch: add basic package management for 'install'
To: assirati@if.usp.br, coreutils@gnu.org
References: <20160728213442.Horde.LWjUgXabrQWA545QDq3CvA1@correio2.if.usp.br>
From: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>
Message-ID: <579B306A.7000309@draigBrady.com>
Date: Fri, 29 Jul 2016 11:31:06 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
	Thunderbird/38.3.0
MIME-Version: 1.0
In-Reply-To: <20160728213442.Horde.LWjUgXabrQWA545QDq3CvA1@correio2.if.usp.br>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 82.195.144.49
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Jul 2016 10:31:25 -0000

On 29/07/16 01:34, assirati@if.usp.br wrote:
>=20
> Hello,
>=20
> Attached is a patch that extends the functionality of Coreutils =20
> install creating a basic package management capability. If it is to be =
=20
> considered useful or viable, I would like very much to receive =20
> reveiews, suggestions for improvements or bug reports. My purpose is =20
> to get the code integrated to Coreutils 'install'.
>=20
> Thank you,
> Joao Luis.
>=20
> ** RATIONALE **
>=20
> When installing programs from upstream source code, 'install' is the =20
> traditional way makefiles install files to /usr/local or other =20
> locations. After that, it is difficult to get a full listing of what =20
> was installed where, and usually one must keep the source code so it =20
> is possible to 'make uninstall' in the future, in case the makefile =20
> provides this possibility.
>=20
> This patch allows 'install' to make a list of all installed files and =20
> created directories, making it easy to track all files and uninstall =20
> the "package".
>=20
> ** USAGE **
>=20
> With this patch, when 'install' is executed with the command line optio=
ns
>=20
> -l FILE
>=20
> or
>=20
> --list=3DFILE
>=20
> it will append to FILE the canonicalized name of the installed file or =
=20
> directory.
>=20
> For example
>=20
> install -l file_list -D my_file /usr/local/my_dir/my_file
>=20
> or
>=20
> install --list=3Dfile_list -D my_file /usr/local/my_dir/my_file
>=20
> or
>=20
> export INSTALL_LIST=3Dfile_list
> install -D my_file /usr/local/my_dir/my_file
>=20
> will append to file_list (or create it if it doesn't exist) containing
>=20
> /usr/local/my_dir
> /usr/local/my_dir/my_file
>=20
> After that, files and directories may be easily uninstalled with
>=20
> tac file_list | while read f; do test -d "$f" && \rmdir "$f" || \rm =20
> "$f"; done; \rm file_list
>=20
> ** USE CASES **
>=20
> Install may be called in new makefiles with the option -l to generate =20
> a list of installed files and direcory. For legacy makefiles, one can =20
> execute
>=20
> INSTALL_LIST=3Dsomefile make install
>=20
> or
>=20
> export INSTALL_LIST=3Dsomefile
> make install
>=20
> or the like.
>=20
> ** DETAILS **
>=20
> A global variable
>=20
> static FILE *install_list =3D NULL;
>=20
> was created to hold the stream for the file list. This stream is =20
> fopen()ed for append in main() and closed by the atexit() handler =20
> close_install_list().
>=20
> A function
>=20
> static void install_list_append (const char *fname)
>=20
> was created to write to the file list. This function is called by:
>=20
> make_ancestor(), to record all the created parent directories (case of =
=20
> option -D)
>=20
> process_dir(), to record a created dir (case of option -d)
>=20
> install_file_in_file(), to record installed files.

Interesting. Thanks for the careful proposal.

My first though was that since install(1) needs to be given
a list of files or directories to create and doesn't recurse into
source directories, that all info required to uninstall is available
and therefore `make uninstall` is the correct way to do it
(which may also do other things than just remove(2)).

I also thought that one could leverage --verbose to get
the same functionality, though that is awkward and
one would have to deal with quoting etc.

wrt quoting you've have to be careful with '\n' in filenames
(maybe from a malicious `make install`) that could induce
an unintended unlink() in a subsequent processing of the list file.
You'd probably still quote in that case.

Also I don't like the list file management within install(1).
If we were to do this I'd just output the list of created files/dirs.
Perhaps with an interface like:
  install --verbose=3Dmanifest > install.list

With the above caveats I'm 50:50 on supporting this
due to lack of generality.

Have you considered managing the installed list in a more
abstract way, like with `checkinstall` to auto create packages,
or using GNU stow to manage the list with symlinks.

thanks,
P=E1draig


From MAILER-DAEMON Fri Jul 29 14:00:12 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bTC4x-0000q0-Tm
	for mharc-coreutils@gnu.org; Fri, 29 Jul 2016 14:00:11 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:55837)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assirati@if.usp.br>) id 1bTC4v-0000dA-Rx
	for coreutils@gnu.org; Fri, 29 Jul 2016 14:00:10 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assirati@if.usp.br>) id 1bTC4r-00063U-Db
	for coreutils@gnu.org; Fri, 29 Jul 2016 14:00:09 -0400
Received: from fragata.if.usp.br ([143.107.128.56]:45897 helo=mail.if.usp.br)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assirati@if.usp.br>) id 1bTC4r-0005zm-1l
	for coreutils@gnu.org; Fri, 29 Jul 2016 14:00:05 -0400
Received: from [192.168.1.2] (unknown [191.254.165.28])
	by mail.if.usp.br (Postfix) with ESMTPSA id DA678FF38;
	Fri, 29 Jul 2016 14:59:58 -0300 (EST)
Subject: Re: Patch: add basic package management for 'install'
To: =?UTF-8?Q?P=c3=a1draig_Brady?= <P@draigBrady.com>, coreutils@gnu.org
References: <20160728213442.Horde.LWjUgXabrQWA545QDq3CvA1@correio2.if.usp.br>
	<579B306A.7000309@draigBrady.com>
From: Joao Luis Meloni Assirati <assirati@if.usp.br>
Message-ID: <ba68bef3-2dac-02a4-ad97-21a6ebb67b8c@if.usp.br>
Date: Fri, 29 Jul 2016 15:00:01 -0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
	Icedove/45.1.0
MIME-Version: 1.0
In-Reply-To: <579B306A.7000309@draigBrady.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
X-Virus-Scanned: clamav-milter 0.98.1 at fragata
X-Virus-Status: Clean
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 143.107.128.56
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Jul 2016 18:00:11 -0000

Hi P=E1draig,

thank you for your considerations.

Em 29-07-2016 07:31, P=E1draig Brady escreveu:

> Interesting. Thanks for the careful proposal.
>
> My first though was that since install(1) needs to be given
> a list of files or directories to create and doesn't recurse into
> source directories, that all info required to uninstall is available
> and therefore `make uninstall` is the correct way to do it
> (which may also do other things than just remove(2)).

The problem with `make uninstall` is twofold: the provider of the=20
software must implement it and one must keep the sources to be able to=20
use it.

Also, by creating lists of installed files, one can at any moment decide=20
to which source some file file belongs, which is useful for many other=20
thins besides uninstallalation.

> I also thought that one could leverage --verbose to get
> the same functionality, though that is awkward and
> one would have to deal with quoting etc.
>
> wrt quoting you've have to be careful with '\n' in filenames
> (maybe from a malicious `make install`) that could induce
> an unintended unlink() in a subsequent processing of the list file.
> You'd probably still quote in that case.
>
> Also I don't like the list file management within install(1).
> If we were to do this I'd just output the list of created files/dirs.
> Perhaps with an interface like:
>    install --verbose=3Dmanifest > install.list

The implementation of this would not differ very much to the code I=20
proposed, the only difference being not opening a file but writing to=20
stdout. But to use a feature like this in legacy makefiles, or makefiles=20
whose creator did no bother to create the install.list, one would have=20
to edit the makefile. This is troublesome and error prone.

With my proposal, if the makefile uses install(1), one just `export=20
INSTALL_LIST=3Dinstall.list` before `make install` with no modification o=
f=20
makefiles or installation scripts even if install is called with no=20
special options inside the makefile.

> With the above caveats I'm 50:50 on supporting this
> due to lack of generality.
>
> Have you considered managing the installed list in a more
> abstract way, like with `checkinstall` to auto create packages,
> or using GNU stow to manage the list with symlinks.

My concern is not about makefiles I create, where I can use such ideas,=20
but others. In short, I wish to be able do administer /usr/local with a=20
multitude of third party software installed by just keeping a collection=20
of standard lists of installed files in, say, /usr/local/install.

> thanks,
> P=E1draig

Thank you,
Jo=E3o Luis.



From MAILER-DAEMON Sun Jul 31 15:33:43 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bTwUZ-0006BY-NE
	for mharc-coreutils@gnu.org; Sun, 31 Jul 2016 15:33:43 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:41680)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bTwUW-0006B2-66
	for coreutils@gnu.org; Sun, 31 Jul 2016 15:33:41 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bTwUR-0001Gt-3T
	for coreutils@gnu.org; Sun, 31 Jul 2016 15:33:39 -0400
Received: from mout.kundenserver.de ([212.227.17.10]:57029)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bTwUQ-0001GH-P2
	for coreutils@gnu.org; Sun, 31 Jul 2016 15:33:35 -0400
Received: from [192.168.101.10] ([84.175.95.115]) by mrelayeu.kundenserver.de
	(mreue104) with ESMTPSA (Nemesis) id 0MfZIB-1beaYT0lRL-00P74K;
	Sun, 31 Jul 2016 21:33:22 +0200
Subject: Re: Speeding up df
To: Philipp Thomas <pth@suse.de>, coreutils@gnu.org,
	Josef Cejka <jcejka@suse.com>, =?UTF-8?Q?P=c3=a1draig_Brady?=
	<P@draigbrady.com>
References: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
	<644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
	<57954D45.6010408@draigBrady.com>
	<20160725112101.wnhlxhwiqahumwx6@paradies.suse.de>
From: Bernhard Voelker <mail@bernhard-voelker.de>
Message-ID: <d4f8f144-2d70-15c7-a9d0-3f1f2b2ad6a0@bernhard-voelker.de>
Date: Sun, 31 Jul 2016 21:33:19 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
	Thunderbird/45.2
MIME-Version: 1.0
In-Reply-To: <20160725112101.wnhlxhwiqahumwx6@paradies.suse.de>
Content-Type: multipart/mixed; boundary="------------F7F786C2CC27F00CEDCC01A2"
X-Provags-ID: V03:K0:hO0VN/MXk60YJxV5OjvqODisBgrk8rimF8vIklOC+ljblLP3t3Z
	HHFWiQ9iSTdfvHQSm6scj+nWR/6lVRf+KiiY2xNMsq+ZB6d6bg+h2JeSB8K8KZ+fdlHpu6P
	Kheyq47+GIM1Nys7jFv/mzyHq0S2WGKQwnCdIl/lAdQMGym1Aygyhyp0WPxre5n4chrpLC/
	LK6/ndUOhBfjSgK1fbIvg==
X-UI-Out-Filterresults: notjunk:1;V01:K0:Yn7xkaSXnNk=:qSIx82bB6zpmj7mlTvxH2H
	Z3xhBmhobKIREhBgOY4AuzVmX5mF4bPP5ed71A9PP9HrgfUrUKnpc/oWXeNH/0d4TkpSMtibz
	hGWS9MO9URNp9nn6ga59paMyS9A0yjpnEbeqc2iWuMUZ4Q933PqZB64AaArSBMzV277/aLSLQ
	SmTMWeG9E+EAqeiJ66EuFSlNqeVxhAJRfUUAQApAtoD98B1L6QAujeBPLY6tMimMrK7MyT/jh
	vwA0GloLtXpcMWxP9ZGmm/zqZTTRakC7trV3/X0nBXp9iXKMPtAAMtiFtf2olaUeDCHSYOm96
	a8QzN/75iNAGNeGhbp5EVaJ2PFbPuoTbtcLkiQl1xoSiKN8GWebISJD0J/nNPaqy8+ei2hjBQ
	MYRPNzUAlm12I9FKtM+Y9OBkT9za1xTWVvMeJ9WqT/SFkijZJoPJPKt1zm9RbMfnrmyfwG0BD
	WMZNpR6xEIx94uvUXkkQVrHpArZHBEDObqufItCpwfqmuI+piK+qedb/hVtHjzqWHTK5JDBeV
	TmlD+ZB7AhhM4kzj2BBDIJsmwhGhnWXhh2SGTFkfFOO6tlO4Tmy+qfgzOev3QHveC2qwYBRIL
	a69zG7qNzS5e2AmQ+AmNutfdE2kFqwW9CS+SnzilHKJEzXomqxW/FFKiQMveQkPTmqwgDgfAT
	Sk6Cub7P8cKQ2SkS3MXZpy7xHySVawFX+E6b/GqRabBIXCK5Gx3f8KLDj8l59P5bxVfE=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 212.227.17.10
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 31 Jul 2016 19:33:41 -0000

This is a multi-part message in MIME format.
--------------F7F786C2CC27F00CEDCC01A2
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit

@Padraig:
I'll be away from keyboard the next 3 weeks, so please continue
with this one to finally push.


On 07/25/2016 01:21 PM, Philipp Thomas wrote:
> * Pádraig Brady (P@draigBrady.com) [20160725 01:20]:
> 
>>> otherwise it looks good to me.
>>
>> Excellent. +1
> 
> I'll pass on your kind words to my collegue.
> 
>> Worth a mention in improvements in NEWS too.
> 
> Like so?
> 
> +  Df now uses a hash table to filter the mount list. This sped up
> +  processing of a 22000 lines mount list by nearly 60%.
> 
> If yes and Bernies explains me why we'd need an additional hash_free call
> I'll post the changed patch.

Thanks.  I wrapped this into a patch with a nice commit message.
I also squashed in the renaming of the 'devlist' variable, as that
many "devlists" were too confusing.

I see this hashing as a starting point, as other functions also need
a better lookup ... and to avoid superfluous stat/statfs calls (see
e.g. "strace -e stat,statfs df -a").

BTW: surprisingly, one can get a HUGE /proc/self/mountinfo very easily:

  # !!! DO NOT DO THIS (with "seq 25") !!!
  mkdir /root/tmp &&
  for f in $(seq 25); do \
    mkdir /root/tmp/$f \
      && mount --bind / /root/tmp/$f
  done

To get the 20.000-30.000 entries, you only need ~"seq 16".

Have a nice day,
Berny

--------------F7F786C2CC27F00CEDCC01A2
Content-Type: text/x-patch;
 name="0001-df-improve-performance-with-many-mount-points.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="0001-df-improve-performance-with-many-mount-points.patch"

>From 93ad559870eec2d827c8f1d34b53c4f2a512b245 Mon Sep 17 00:00:00 2001
From: Philipp Thomas <pth@suse.de>
Date: Sun, 31 Jul 2016 21:24:18 +0200
Subject: [PATCH] df: improve performance with many mount points

Use hash table for seaching in filter_mount_list()
and get_dev().

* src/df.c (devlist_table): Define hash table.
(devlist_hash): Add hash function.
(devlist_compare): Add hash comparison function.
(devlist_for_dev): Add lookup function.
(devlist_free): Add cleanup function.
(filter_mount_list): Use the above hash table.
While at it, rename the variable 'devlist' to 'seen_dev' for
better readability.
(me_for_dev): Use the above lookup function.

Original version of the patch by Josef Cejka <jcejka@suse.com>.
---
 NEWS     |   3 ++
 src/df.c | 109 ++++++++++++++++++++++++++++++++++++++++++++-------------------
 2 files changed, 79 insertions(+), 33 deletions(-)

diff --git a/NEWS b/NEWS
index 736b95e..4c28fc1 100644
--- a/NEWS
+++ b/NEWS
@@ -50,6 +50,9 @@ GNU coreutils NEWS                                    -*- outline -*-
 
 ** Improvements
 
+  df now uses a hash table to filter the mount list.  This speeds up
+  processing of a 22000 lines mount list by nearly 60%.
+
   install -Z now also sets the default SELinux context for created directories.
 
   stat and tail now know about "prl_fs" (a parallels file system),
diff --git a/src/df.c b/src/df.c
index cbd8ef5..3813346 100644
--- a/src/df.c
+++ b/src/df.c
@@ -34,6 +34,7 @@
 #include "mountlist.h"
 #include "quote.h"
 #include "find-mount-point.h"
+#include "hash.h"
 
 /* The official name of this program (e.g., no 'g' prefix).  */
 #define PROGRAM_NAME "df"
@@ -43,14 +44,16 @@
   proper_name ("David MacKenzie"), \
   proper_name ("Paul Eggert")
 
-/* Filled with device numbers of examined file systems to avoid
-   duplicates in output.  */
-static struct devlist
+struct devlist
 {
   dev_t dev_num;
   struct mount_entry *me;
   struct devlist *next;
-} *device_list;
+};
+
+/* Filled with device numbers of examined file systems to avoid
+   duplicates in output.  */
+static Hash_table *devlist_table;
 
 /* If true, show even file systems with zero size or
    uninteresting types.  */
@@ -603,23 +606,67 @@ excluded_fstype (const char *fstype)
   return false;
 }
 
+static size_t
+devlist_hash (void const *x, size_t table_size)
+{
+  struct devlist const *p = x;
+  return (uintmax_t) p->dev_num % table_size;
+}
+
+static bool
+devlist_compare (void const *x, void const *y)
+{
+  struct devlist const *a = x;
+  struct devlist const *b = y;
+  return a->dev_num == b->dev_num;
+}
+
+static struct devlist *
+devlist_for_dev (dev_t dev)
+{
+  if (devlist_table == NULL)
+    return NULL;
+  struct devlist dev_entry;
+  dev_entry.dev_num = dev;
+  return hash_lookup(devlist_table, &dev_entry);
+}
+
+static void
+devlist_free(void *p)
+{
+  free(p);
+}
+
 /* Filter mount list by skipping duplicate entries.
    In the case of duplicates - based on the device number - the mount entry
    with a '/' in its me_devname (i.e., not pseudo name like tmpfs) wins.
    If both have a real devname (e.g. bind mounts), then that with the shorter
    me_mountdir wins.  With DEVICES_ONLY == true (set with df -a), only update
-   the global device_list, rather than filtering the global mount_list.  */
+   the global devlist_table, rather than filtering the global mount_list.  */
 
 static void
 filter_mount_list (bool devices_only)
 {
   struct mount_entry *me;
 
+  /* Temporary list to keep entries ordered.  */
+  struct devlist *device_list = NULL;
+  int mount_list_size = 0;
+
+  for (me = mount_list; me; me = me->me_next)
+    mount_list_size++;
+
+  devlist_table = hash_initialize (mount_list_size, NULL,
+                                 devlist_hash,
+                                 devlist_compare,
+                                 devlist_free);
+  if (devlist_table == NULL)
+    xalloc_die ();
+
   /* Sort all 'wanted' entries into the list device_list.  */
   for (me = mount_list; me;)
     {
       struct stat buf;
-      struct devlist *devlist;
       struct mount_entry *discard_me = NULL;
 
       /* Avoid stating remote file systems as that may hang.
@@ -635,21 +682,19 @@ filter_mount_list (bool devices_only)
       else
         {
           /* If we've already seen this device...  */
-          for (devlist = device_list; devlist; devlist = devlist->next)
-            if (devlist->dev_num == buf.st_dev)
-              break;
+          struct devlist *seen_dev = devlist_for_dev(buf.st_dev);
 
-          if (devlist)
+          if (seen_dev)
             {
-              bool target_nearer_root = strlen (devlist->me->me_mountdir)
+              bool target_nearer_root = strlen (seen_dev->me->me_mountdir)
                                         > strlen (me->me_mountdir);
               /* With bind mounts, prefer items nearer the root of the source */
-              bool source_below_root = devlist->me->me_mntroot != NULL
+              bool source_below_root = seen_dev->me->me_mntroot != NULL
                                        && me->me_mntroot != NULL
-                                       && (strlen (devlist->me->me_mntroot)
+                                       && (strlen (seen_dev->me->me_mntroot)
                                            < strlen (me->me_mntroot));
-              if (! print_grand_total && me->me_remote && devlist->me->me_remote
-                  && ! STREQ (devlist->me->me_devname, me->me_devname))
+              if (! print_grand_total && me->me_remote && seen_dev->me->me_remote
+                  && ! STREQ (seen_dev->me->me_devname, me->me_devname))
                 {
                   /* Don't discard remote entries with different locations,
                      as these are more likely to be explicitly mounted.
@@ -658,21 +703,21 @@ filter_mount_list (bool devices_only)
                 }
               else if ((strchr (me->me_devname, '/')
                        /* let "real" devices with '/' in the name win.  */
-                        && ! strchr (devlist->me->me_devname, '/'))
+                        && ! strchr (seen_dev->me->me_devname, '/'))
                        /* let points towards the root of the device win.  */
                        || (target_nearer_root && ! source_below_root)
                        /* let an entry overmounted on a new device win...  */
-                       || (! STREQ (devlist->me->me_devname, me->me_devname)
+                       || (! STREQ (seen_dev->me->me_devname, me->me_devname)
                            /* ... but only when matching an existing mnt point,
                               to avoid problematic replacement when given
                               inaccurate mount lists, seen with some chroot
                               environments for example.  */
                            && STREQ (me->me_mountdir,
-                                     devlist->me->me_mountdir)))
+                                     seen_dev->me->me_mountdir)))
                 {
                   /* Discard mount entry for existing device.  */
-                  discard_me = devlist->me;
-                  devlist->me = me;
+                  discard_me = seen_dev->me;
+                  seen_dev->me = me;
                 }
               else
                 {
@@ -691,12 +736,14 @@ filter_mount_list (bool devices_only)
         }
       else
         {
-          /* Add the device number to the global list devlist.  */
-          devlist = xmalloc (sizeof *devlist);
+          /* Add the device number to the device_table.  */
+          struct devlist *devlist = xmalloc (sizeof *devlist);
           devlist->me = me;
           devlist->dev_num = buf.st_dev;
           devlist->next = device_list;
           device_list = devlist;
+          if (hash_insert (devlist_table, devlist) == NULL)
+            xalloc_die ();
 
           me = me->me_next;
         }
@@ -711,28 +758,24 @@ filter_mount_list (bool devices_only)
         me = device_list->me;
         me->me_next = mount_list;
         mount_list = me;
-        /* Free devlist entry and advance.  */
-        struct devlist *devlist = device_list->next;
-        free (device_list);
-        device_list = devlist;
+        device_list = device_list->next;
       }
+
+      hash_free(devlist_table);
+      devlist_table = NULL;
   }
 }
 
+
 /* Search a mount entry list for device id DEV.
    Return the corresponding mount entry if found or NULL if not.  */
 
 static struct mount_entry const * _GL_ATTRIBUTE_PURE
 me_for_dev (dev_t dev)
 {
-  struct devlist *dl = device_list;
-
-  while (dl)
-    {
-      if (dl->dev_num == dev)
+  struct devlist *dl = devlist_for_dev(dev);
+  if (dl)
         return dl->me;
-      dl = dl->next;
-    }
 
   return NULL;
 }
-- 
2.1.4


--------------F7F786C2CC27F00CEDCC01A2--


From MAILER-DAEMON Sun Jul 31 15:42:10 2016
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1bTwck-0008Db-1K
	for mharc-coreutils@gnu.org; Sun, 31 Jul 2016 15:42:10 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:43181)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bTwci-0008CF-76
	for coreutils@gnu.org; Sun, 31 Jul 2016 15:42:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bTwcd-0002jW-6s
	for coreutils@gnu.org; Sun, 31 Jul 2016 15:42:07 -0400
Received: from mout.kundenserver.de ([217.72.192.74]:56168)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <mail@bernhard-voelker.de>) id 1bTwcc-0002jJ-Sj
	for coreutils@gnu.org; Sun, 31 Jul 2016 15:42:03 -0400
Received: from [192.168.101.10] ([84.175.95.115]) by mrelayeu.kundenserver.de
	(mreue102) with ESMTPSA (Nemesis) id 0MX0eS-1bomuC23Qi-00VvFH;
	Sun, 31 Jul 2016 21:42:00 +0200
Subject: Re: Speeding up df
To: Philipp Thomas <pth@suse.de>, coreutils@gnu.org
References: <20160722084654.wi56p2vjcgbt2ghx@paradies.suse.de>
	<644de882-8d96-2e41-37ef-86a623d33307@bernhard-voelker.de>
	<20160725111612.txlr7qwzfeozecno@paradies.suse.de>
From: Bernhard Voelker <mail@bernhard-voelker.de>
Message-ID: <57f2b1a4-6594-3512-f5e5-3b5eba5706e3@bernhard-voelker.de>
Date: Sun, 31 Jul 2016 21:41:59 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101
	Thunderbird/45.2
MIME-Version: 1.0
In-Reply-To: <20160725111612.txlr7qwzfeozecno@paradies.suse.de>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
X-Provags-ID: V03:K0:VFGYH6hbOOUfGmUetvepogRnScRjykMXz7/txjqAbQcsJ8ol5TF
	uAlunF9WpHDZTnwQvaUgUqppqkj3JsAii5JZtmajwytAbDYU7h/WVSbYRu7/aBqDWlV7Rd6
	Pdxr0N6g/QyflG3vcS1CKa8yQr4xBTtAfu0EyFI+M8xt94qhNRG2JXNqubsGq426PVszb0r
	c8/2deTrHi/RKjVJJmd9A==
X-UI-Out-Filterresults: notjunk:1;V01:K0:dJgnJbSd/nA=:o1p3wEwHWOiLvdFjoZa8S4
	UnWPVL7YGk3WA0pHNVWzBi8jDQqU82+CHUjKuVyp7W0jFtreNmzD4cuuIJLyn2+iA4RayvdgA
	Z/Cm9yvrsrOp82oYYBlV5SFCqGEYFWtaJyZwY9KAMFpd2JVOHMIrwKGjG+V0ubQP1+MWIEJSy
	kMODd7f9UcYPOhV6LNHVCIvHPzKQfqqmjSSkj/Gn8O76CyLCSe/cYA8L41ZIueCEz3H9sGw5c
	u6Oj/9WQPKtXPWbR0k07d8nVZJpp53EmnxvnRBPiiI1ygL4Vbw1Hhfnd+KA8igjm6sDtOWZaO
	fjBBi4F3Zktsn3LL55L+p8FTOqh+LEtMKxQWqFTJ0wbo5jiS7oluuj3TKoiOv1yvhWFMD3+39
	+XA8g4aPmvs1BYB1HcxMrUFRShle6/saauW+CTtpDIHEa6IRyvtPhQsckJQhPG9aHxKjgzuxV
	i0dsU90tZZkfivSa9VfZdup5XcdoxS5vjxjlVx6t2Ki0qW43GhB4RUNiMSlnAm0AsLXyh+MaK
	DKO5rtnA3HC85NikcG2IcMZhB5fY22/EVo91baPTBG3ITqfgTyIa0ztlcRCcYOFkyOm0o/8GW
	ad9atKOZjVxzq2q35n8YyT4A8lul050AYqpEdbzp6oPm0mz5v92PFdAwjCg/oNtQiWfKpUdIg
	PaNn/eyS9qJoe2muT5VyVJQJkzKEeP7kA8nmSd0omJ7JcJUUpQXdkl4113ol+TwwtvNLzvOXM
	wQ3ywbeexCqC1a3A
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic]
X-Received-From: 217.72.192.74
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.21
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils/>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 31 Jul 2016 19:42:09 -0000

On 07/25/2016 01:16 PM, Philipp Thomas wrote:
> Hi Bernie,
> 
> * Bernhard Voelker (mail@bernhard-voelker.de) [20160724 23:33]:
> 
>> Maybe adding an IF_LINT'ed hash_free call would be worth;
>> otherwise it looks good to me.
> 
> I don't quite understand as filter_mount_list already does call hash_free at
> the end. So please explain.

hash_free is only called if (! devices_only), i.e., so not if the -a option
is passed.  There are also other allocations not free()d in df.c.  Maybe it'd
be good to guard them all with IF_LINT(...) for better maintainability: to
see what are real leaks and what is freed at exit anyway.

Have a nice day,
Berny


