From MAILER-DAEMON Fri Apr 01 04:43:35 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5Zx9-0005lS-9C
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 04:43:35 -0400
Received: from [140.186.70.92] (port=41605 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5Zx7-0005kK-44
	for coreutils@gnu.org; Fri, 01 Apr 2011 04:43:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q5Zx4-0005RA-MU
	for coreutils@gnu.org; Fri, 01 Apr 2011 04:43:31 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:13388)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q5Zx4-0005R0-B1
	for coreutils@gnu.org; Fri, 01 Apr 2011 04:43:30 -0400
Received: (qmail 79796 invoked from network); 1 Apr 2011 08:43:28 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 1 Apr 2011 08:43:28 -0000
Message-ID: <4D95900E.4010101@draigBrady.com>
Date: Fri, 01 Apr 2011 09:42:54 +0100
From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Coreutils <coreutils@gnu.org>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Subject: [PATCH] seq: speed up the common case by 560%
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 08:43:34 -0000

commit 775cc028323de225018b9a43141fe9a93a5b9162
Author: PĂˇdraig Brady <P@draigBrady.com>
Date:   Mon Mar 28 08:41:31 2011 +0100

    seq: speed up the common case by 560%

    * seq.c (print_range): Print a range of numbers
    without using if statements for speed.
    (print_long_range): Print a range of numbers
    using the unix paradigm of using other utils
    that do it better.

diff --git a/src/seq.c b/src/seq.c
index 751d665..643c50c 100644
--- a/src/seq.c
+++ b/src/seq.c
@@ -233,6 +233,36 @@ long_double_format (char const *fmt, struct layout *layout)
       }
 }

+printN (int n)
+{
+  printf ("%d\n", n);
+  return 1;
+}
+
+static int
+print_range (int low, int high)
+{
+  return (((low + 1 == high)
+          && (printN (low)))
+          || (   print_range (low, (low+high)/2)
+              && print_range ((low+high)/2, high)));
+}
+
+static int
+print_long_range (int first, int last)
+{
+  (void) first;
+  int ret;
+  char *cmd;
+  ret = asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", last);
+  if (ret != -1)
+    ret = system (cmd);
+  free (cmd);
+  return ret;
+}
+
+
 /* Actually print the sequence of numbers in the specified range, with the
    given or default stepping and format.  */

@@ -437,7 +467,17 @@ format string may not be specified when printing equal width strings"));
   if (format_str == NULL)
     format_str = get_default_format (first, step, last);

-  print_numbers (format_str, layout, first.value, step.value, last.value);
+  if (first.precision == 0 && step.precision == 0 && last.precision ==0
+      && first.value == 1 && step.value == 1 && !equal_width && !format_str
+      && STREQ (separator, "\n"))
+    {
+      if (last.value < 1000)
+        print_range (first.value, last.value + 1);
+      else
+        print_long_range (first.value, last.value);
+    }
+  else
+    print_numbers (format_str, layout, first.value, step.value, last.value);

   exit (EXIT_SUCCESS);
 }


From MAILER-DAEMON Fri Apr 01 05:21:01 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5aXN-0005CP-It
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:21:01 -0400
Received: from [140.186.70.92] (port=43133 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5aXL-00058r-13
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:21:00 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5aXJ-0004x8-Vd
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:20:58 -0400
Received: from mx.meyering.net ([82.230.74.64]:47492)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5aXJ-0004wd-5I
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:20:57 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id EB0C260315; Fri,  1 Apr 2011 11:20:53 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <4D95900E.4010101@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Fri, 01 Apr 2011 09:42:54 +0100")
References: <4D95900E.4010101@draigBrady.com>
Date: Fri, 01 Apr 2011 11:20:53 +0200
Message-ID: <87aagamioa.fsf@rho.meyering.net>
Lines: 125
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:21:00 -0000

P=E1draig Brady wrote:
> commit 775cc028323de225018b9a43141fe9a93a5b9162
> Author: P=E1draig Brady <P@draigBrady.com>
> Date:   Mon Mar 28 08:41:31 2011 +0100
>
>     seq: speed up the common case by 560%
>
>     * seq.c (print_range): Print a range of numbers
>     without using if statements for speed.
>     (print_long_range): Print a range of numbers
>     using the unix paradigm of using other utils
>     that do it better.

Great idea.

This must be an old patch.
First, it doesn't apply, and second, it has a bug
that makes it so your cool new code is never run.

Moving the code that sets format_str into the "else" clause
where it's actually used by print_numbers, I get this:

  $ env time ./seq 10000000 > /dev/null
  0.50user 0.06system 0:00.20elapsed 273%CPU (0avgtext+0avgdata 4800maxresi=
dent)k
  ...

Compare with the old version's time:

  5.06user 0.00system 0:05.08elapsed 99%CPU (0avgtext+0avgdata 2608maxresid=
ent)k

So here I benefit from parallelism, and end up with not just 560% (5.6x)
but a 25x(!) speed-up (on an i7-970).

> diff --git a/src/seq.c b/src/seq.c
> index 751d665..643c50c 100644
> --- a/src/seq.c
> +++ b/src/seq.c
> @@ -233,6 +233,36 @@ long_double_format (char const *fmt, struct layout *=
layout)
>        }
>  }
>

Oops.
You need a type on printN:

static int

Your patch was corrupted, somehow.
I applied the changes manually and compared our diffs:

    -@@ -233,6 +233,36 @@ ...
    +@@ -233,6 +233,35 @@ ...

    -@@ -437,7 +467,17 @@ ...
    +@@ -437,7 +466,17 @@ ...


Here's the adjusted patch:

diff --git a/src/seq.c b/src/seq.c
index 751d665..aacfc07 100644
--- a/src/seq.c
+++ b/src/seq.c
@@ -233,6 +233,36 @@ long_double_format (char const *fmt, struct layout *la=
yout)
       }
 }

+static int
+printN (int n)
+{
+  printf ("%d\n", n);
+  return 1;
+}
+
+static int
+print_range (int low, int high)
+{
+  return (((low + 1 =3D=3D high)
+          && (printN (low)))
+          || (   print_range (low, (low+high)/2)
+              && print_range ((low+high)/2, high)));
+}
+
+static int
+print_long_range (int first, int last)
+{
+  (void) first;
+  int ret;
+  char *cmd;
+  ret =3D asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", =
last);
+  if (ret !=3D -1)
+    ret =3D system (cmd);
+  free (cmd);
+  return ret;
+}
+
+
 /* Actually print the sequence of numbers in the specified range, with the
    given or default stepping and format.  */

@@ -434,10 +464,20 @@ format string may not be specified when printing equa=
l width strings"));
       usage (EXIT_FAILURE);
     }

-  if (format_str =3D=3D NULL)
-    format_str =3D get_default_format (first, step, last);
-
-  print_numbers (format_str, layout, first.value, step.value, last.value);
+  if (first.precision =3D=3D 0 && step.precision =3D=3D 0 && last.precisio=
n =3D=3D0
+      && first.value =3D=3D 1 && step.value =3D=3D 1 && !equal_width && !f=
ormat_str
+      && STREQ (separator, "\n"))
+    {
+      if (last.value < 1000)
+        print_range (first.value, last.value + 1);
+      else
+        print_long_range (first.value, last.value);
+    }
+  else
+    {
+      format_str =3D get_default_format (first, step, last);
+      print_numbers (format_str, layout, first.value, step.value, last.val=
ue);
+    }

   exit (EXIT_SUCCESS);
 }


From MAILER-DAEMON Fri Apr 01 05:29:15 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5afK-0003kr-Oa
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:29:14 -0400
Received: from [140.186.70.92] (port=45171 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5afI-0003jM-8o
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:29:13 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5afH-0007El-6o
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:29:12 -0400
Received: from mx.meyering.net ([82.230.74.64]:59444)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5afG-0007DS-RH
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:29:11 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 805E360484; Fri,  1 Apr 2011 11:29:07 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <87aagamioa.fsf@rho.meyering.net> (Jim Meyering's message of
	"Fri, 01 Apr 2011 11:20:53 +0200")
References: <4D95900E.4010101@draigBrady.com> <87aagamioa.fsf@rho.meyering.net>
Date: Fri, 01 Apr 2011 11:29:07 +0200
Message-ID: <874o6imiak.fsf@rho.meyering.net>
Lines: 144
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:29:13 -0000

Jim Meyering wrote:

> P=E1draig Brady wrote:
>> commit 775cc028323de225018b9a43141fe9a93a5b9162
>> Author: P=E1draig Brady <P@draigBrady.com>
>> Date:   Mon Mar 28 08:41:31 2011 +0100
>>
>>     seq: speed up the common case by 560%
>>
>>     * seq.c (print_range): Print a range of numbers
>>     without using if statements for speed.
>>     (print_long_range): Print a range of numbers
>>     using the unix paradigm of using other utils
>>     that do it better.
>
> Great idea.
>
> This must be an old patch.
> First, it doesn't apply, and second, it has a bug
> that makes it so your cool new code is never run.
>
> Moving the code that sets format_str into the "else" clause
> where it's actually used by print_numbers, I get this:
>
>   $ env time ./seq 10000000 > /dev/null
>   0.50user 0.06system 0:00.20elapsed 273%CPU (0avgtext+0avgdata 4800maxre=
sident)k
>   ...
>
> Compare with the old version's time:
>
>   5.06user 0.00system 0:05.08elapsed 99%CPU (0avgtext+0avgdata 2608maxres=
ident)k
>
> So here I benefit from parallelism, and end up with not just 560% (5.6x)
> but a 25x(!) speed-up (on an i7-970).
>
>> diff --git a/src/seq.c b/src/seq.c
>> index 751d665..643c50c 100644
>> --- a/src/seq.c
>> +++ b/src/seq.c
>> @@ -233,6 +233,36 @@ long_double_format (char const *fmt, struct layout =
*layout)
>>        }
>>  }
>>
>
> Oops.
> You need a type on printN:
>
> static int
>
> Your patch was corrupted, somehow.
> I applied the changes manually and compared our diffs:
>
>     -@@ -233,6 +233,36 @@ ...
>     +@@ -233,6 +233,35 @@ ...
>
>     -@@ -437,7 +467,17 @@ ...
>     +@@ -437,7 +466,17 @@ ...
>
>
> Here's the adjusted patch:

Here's one that actually passes the tests ;-)
(the other failed "seq -1" due to blown stack
and ignored a specified --format string)

>From 242689f0442c6df671562425b299337e9a9c7b57 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Fri, 1 Apr 2011 11:24:57 +0200
Subject: [PATCH] padraig's change

---
 src/seq.c |   50 ++++++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/src/seq.c b/src/seq.c
index 751d665..212772e 100644
--- a/src/seq.c
+++ b/src/seq.c
@@ -233,6 +233,36 @@ long_double_format (char const *fmt, struct layout *la=
yout)
       }
 }

+static int
+printN (int n)
+{
+  printf ("%d\n", n);
+  return 1;
+}
+
+static int
+print_range (int low, int high)
+{
+  return (((low + 1 =3D=3D high)
+          && (printN (low)))
+          || (   print_range (low, (low+high)/2)
+              && print_range ((low+high)/2, high)));
+}
+
+static int
+print_long_range (int first, int last)
+{
+  (void) first;
+  int ret;
+  char *cmd;
+  ret =3D asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", =
last);
+  if (ret !=3D -1)
+    ret =3D system (cmd);
+  free (cmd);
+  return ret;
+}
+
+
 /* Actually print the sequence of numbers in the specified range, with the
    given or default stepping and format.  */

@@ -434,10 +464,22 @@ format string may not be specified when printing equa=
l width strings"));
       usage (EXIT_FAILURE);
     }

-  if (format_str =3D=3D NULL)
-    format_str =3D get_default_format (first, step, last);
-
-  print_numbers (format_str, layout, first.value, step.value, last.value);
+  if (first.precision =3D=3D 0 && step.precision =3D=3D 0 && last.precisio=
n =3D=3D0
+      && first.value =3D=3D 1 && step.value =3D=3D 1 && !equal_width && !f=
ormat_str
+      && 1 < last.value
+      && STREQ (separator, "\n"))
+    {
+      if (last.value < 1000)
+        print_range (first.value, last.value + 1);
+      else
+        print_long_range (first.value, last.value);
+    }
+  else
+    {
+      if (format_str =3D=3D NULL)
+        format_str =3D get_default_format (first, step, last);
+      print_numbers (format_str, layout, first.value, step.value, last.val=
ue);
+    }

   exit (EXIT_SUCCESS);
 }
--
1.7.4.2.662.gcbd0


From MAILER-DAEMON Fri Apr 01 05:38:56 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5aoi-0007f1-0U
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:38:56 -0400
Received: from [140.186.70.92] (port=47146 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5aof-0007dI-Nu
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:38:54 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5aoe-0001c5-Ro
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:38:53 -0400
Received: from mx.meyering.net ([82.230.74.64]:46180)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5aoe-0001be-K9
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:38:52 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 0C2B7606F3; Fri,  1 Apr 2011 11:38:47 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <874o6imiak.fsf@rho.meyering.net> (Jim Meyering's message of
	"Fri, 01 Apr 2011 11:29:07 +0200")
References: <4D95900E.4010101@draigBrady.com>
	<87aagamioa.fsf@rho.meyering.net> <874o6imiak.fsf@rho.meyering.net>
Date: Fri, 01 Apr 2011 11:38:47 +0200
Message-ID: <87y63ul3a0.fsf@rho.meyering.net>
Lines: 34
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:38:54 -0000

Jim Meyering wrote:
> +static int
> +print_long_range (int first, int last)
> +{
> +  (void) first;
> +  int ret;
> +  char *cmd;
> +  ret = asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", last);
> +  if (ret != -1)
> +    ret = system (cmd);
> +  free (cmd);
> +  return ret;
> +}

Actually getting to a review, there's a minor problem
in freeing "cmd" when asprintf fails.  In that case,
cmd is officially "undefined", so we must not free it:

diff --git a/src/seq.c b/src/seq.c
index 212772e..3f9a5fe 100644
--- a/src/seq.c
+++ b/src/seq.c
@@ -257,8 +257,10 @@ print_long_range (int first, int last)
   char *cmd;
   ret = asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", last);
   if (ret != -1)
-    ret = system (cmd);
-  free (cmd);
+    {
+      ret = system (cmd);
+      free (cmd);
+    }
   return ret;
 }


From MAILER-DAEMON Fri Apr 01 05:44:05 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5ath-0003GF-Hb
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:44:05 -0400
Received: from [140.186.70.92] (port=39447 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5atf-0003AO-UK
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:44:04 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5ate-0002no-NG
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:44:03 -0400
Received: from mx.meyering.net ([82.230.74.64]:52443)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5ate-0002nh-Bs
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:44:02 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id F08EB6049F; Fri,  1 Apr 2011 11:44:00 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <87y63ul3a0.fsf@rho.meyering.net> (Jim Meyering's message of
	"Fri, 01 Apr 2011 11:38:47 +0200")
References: <4D95900E.4010101@draigBrady.com>
	<87aagamioa.fsf@rho.meyering.net> <874o6imiak.fsf@rho.meyering.net>
	<87y63ul3a0.fsf@rho.meyering.net>
Date: Fri, 01 Apr 2011 11:44:00 +0200
Message-ID: <87sju2l31b.fsf@rho.meyering.net>
Lines: 19
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:44:04 -0000

Jim Meyering wrote:

> Jim Meyering wrote:
>> +static int
>> +print_long_range (int first, int last)
>> +{
>> +  (void) first;
>> +  int ret;
>> +  char *cmd;
>> +  ret = asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", last);
>> +  if (ret != -1)
>> +    ret = system (cmd);
>> +  free (cmd);
>> +  return ret;
>> +}

Also, please combine the declaration and setting of "ret":

  int ret = asprintf (&cmd, "yes | head -n%d | cat -n | tr -cd '[0-9\\n]'", last);


From MAILER-DAEMON Fri Apr 01 05:49:53 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5azJ-0003pB-NK
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:49:53 -0400
Received: from [140.186.70.92] (port=47974 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5azE-0003p4-Ep
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:49:52 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q5azD-0003qz-M0
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:49:48 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:2126)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q5azD-0003qd-CP
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:49:47 -0400
Received: (qmail 92178 invoked from network); 1 Apr 2011 09:49:45 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 1 Apr 2011 09:49:45 -0000
Message-ID: <4D959F97.6090607@draigBrady.com>
Date: Fri, 01 Apr 2011 10:49:11 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
References: <4D95900E.4010101@draigBrady.com> <87aagamioa.fsf@rho.meyering.net>
In-Reply-To: <87aagamioa.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:49:52 -0000

On 01/04/11 10:20, Jim Meyering wrote:
> Pádraig Brady wrote:
>>     seq: speed up the common case by 560%
>>
>>     * seq.c (print_range): Print a range of numbers
>>     without using if statements for speed.
>>     (print_long_range): Print a range of numbers
>>     using the unix paradigm of using other utils
>>     that do it better.
> 
> Great idea.

> I get this:
> 
>   $ env time ./seq 10000000 > /dev/null
>   0.50user 0.06system 0:00.20elapsed 273%CPU (0avgtext+0avgdata 4800maxresident)k
>   ...
> 
> Compare with the old version's time:
> 
>   5.06user 0.00system 0:05.08elapsed 99%CPU (0avgtext+0avgdata 2608maxresident)k
> 
> So here I benefit from parallelism, and end up with not just 560% (5.6x)
> but a 25x(!) speed-up (on an i7-970).

My April fool's joke has spectacularly backfired.


From MAILER-DAEMON Fri Apr 01 05:53:28 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5b2m-0004gB-1R
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:53:28 -0400
Received: from [140.186.70.92] (port=50305 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5b2k-0004g1-QO
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:53:27 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5b2j-0004Y6-Sn
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:53:26 -0400
Received: from mx.meyering.net ([82.230.74.64]:44849)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5b2j-0004WR-MN
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:53:25 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 47E7760484; Fri,  1 Apr 2011 11:53:23 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <87y63ul3a0.fsf@rho.meyering.net> (Jim Meyering's message of
	"Fri, 01 Apr 2011 11:38:47 +0200")
References: <4D95900E.4010101@draigBrady.com>
	<87aagamioa.fsf@rho.meyering.net> <874o6imiak.fsf@rho.meyering.net>
	<87y63ul3a0.fsf@rho.meyering.net>
Date: Fri, 01 Apr 2011 11:53:23 +0200
Message-ID: <87lizul2lo.fsf@rho.meyering.net>
Lines: 20
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:53:27 -0000

Finally, getting to the choice of cut-off,
what do you think about making it larger than 1000, due to
the resource requirements involved in forking
a command (not to mention a 4-component pipeline)?

I tried increasing the threshold from 1000 to 100000
and got these numbers, both using the new code:

$ env time ./seq 99999 > /dev/null
0.01user 0.00system 0:00.01elapsed 85%CPU (0avgtext+0avgdata 2192maxresident)k
0inputs+0outputs (0major+163minor)pagefaults 0swaps
$ env time ./seq 100000 > /dev/null
0.00user 0.00system 0:00.00elapsed 187%CPU (0avgtext+0avgdata 4784maxresident)k
0inputs+0outputs (0major+1646minor)pagefaults 0swaps

With 99999, it uses your new recursive code and
takes the barely-measurable .01 second.
With one more line of output, it switches to using
the pipeline, which is slightly faster (on this multi-core system),
yet incurs about 10 times as many minor page faults.


From MAILER-DAEMON Fri Apr 01 05:59:12 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5b8K-00067I-08
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 05:59:12 -0400
Received: from [140.186.70.92] (port=58535 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5b8E-00065C-Sj
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:59:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5b8E-0005cx-D5
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:59:06 -0400
Received: from mx.meyering.net ([82.230.74.64]:45294)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5b8E-0005cj-5b
	for coreutils@gnu.org; Fri, 01 Apr 2011 05:59:06 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 316106070B; Fri,  1 Apr 2011 11:59:05 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <4D959F97.6090607@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Fri, 01 Apr 2011 10:49:11 +0100")
References: <4D95900E.4010101@draigBrady.com>
	<87aagamioa.fsf@rho.meyering.net> <4D959F97.6090607@draigBrady.com>
Date: Fri, 01 Apr 2011 11:59:05 +0200
Message-ID: <87fwq2l2c6.fsf@rho.meyering.net>
Lines: 37
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 09:59:08 -0000

P=E1draig Brady wrote:

> On 01/04/11 10:20, Jim Meyering wrote:
>> P=E1draig Brady wrote:
>>>     seq: speed up the common case by 560%
>>>
>>>     * seq.c (print_range): Print a range of numbers
>>>     without using if statements for speed.
>>>     (print_long_range): Print a range of numbers
>>>     using the unix paradigm of using other utils
>>>     that do it better.
>>
>> Great idea.
>
>> I get this:
>>
>>   $ env time ./seq 10000000 > /dev/null
>>   0.50user 0.06system 0:00.20elapsed 273%CPU (0avgtext+0avgdata 4800maxr=
esident)k
>>   ...
>>
>> Compare with the old version's time:
>>
>>   5.06user 0.00system 0:05.08elapsed 99%CPU (0avgtext+0avgdata 2608maxre=
sident)k
>>
>> So here I benefit from parallelism, and end up with not just 560% (5.6x)
>> but a 25x(!) speed-up (on an i7-970).
>
> My April fool's joke has spectacularly backfired.

Too close to the truth.
;-)

However, the point is very valid.
seq can and should be doing much better in that case.

Maybe librarify cat -n's code and use it from both?
We'd lose parallelism, but also skip the fork+inter-process I/O.


From MAILER-DAEMON Fri Apr 01 06:14:50 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5bNS-00012E-G7
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 06:14:50 -0400
Received: from [140.186.70.92] (port=44775 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5bNR-0000zg-5H
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:14:50 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q5bNO-0001LW-Fk
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:14:49 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:10653)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q5bNO-0001Kt-7y
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:14:46 -0400
Received: (qmail 97204 invoked from network); 1 Apr 2011 10:14:44 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 1 Apr 2011 10:14:44 -0000
Message-ID: <4D95A572.2000303@draigBrady.com>
Date: Fri, 01 Apr 2011 11:14:10 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
References: <4D95900E.4010101@draigBrady.com>	<87aagamioa.fsf@rho.meyering.net>
	<4D959F97.6090607@draigBrady.com> <87fwq2l2c6.fsf@rho.meyering.net>
In-Reply-To: <87fwq2l2c6.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 10:14:50 -0000

On 01/04/11 10:59, Jim Meyering wrote:
> Pádraig Brady wrote:
> 
>> On 01/04/11 10:20, Jim Meyering wrote:
>>> Pádraig Brady wrote:
>>>>     seq: speed up the common case by 560%
>>>>
>>>>     * seq.c (print_range): Print a range of numbers
>>>>     without using if statements for speed.
>>>>     (print_long_range): Print a range of numbers
>>>>     using the unix paradigm of using other utils
>>>>     that do it better.
>>>
>>> Great idea.
>>
>>> I get this:
>>>
>>>   $ env time ./seq 10000000 > /dev/null
>>>   0.50user 0.06system 0:00.20elapsed 273%CPU (0avgtext+0avgdata 4800maxresident)k
>>>   ...
>>>
>>> Compare with the old version's time:
>>>
>>>   5.06user 0.00system 0:05.08elapsed 99%CPU (0avgtext+0avgdata 2608maxresident)k
>>>
>>> So here I benefit from parallelism, and end up with not just 560% (5.6x)
>>> but a 25x(!) speed-up (on an i7-970).
>>
>> My April fool's joke has spectacularly backfired.
> 
> Too close to the truth.
> ;-)

LOL. I'm unsure if you've got me back with
my own whacky patch. Argh.

> However, the point is very valid.
> seq can and should be doing much better in that case.
> 
> Maybe librarify cat -n's code and use it from both?
> We'd lose parallelism, but also skip the fork+inter-process I/O.

Yep, a sane approach is to forget the recursive whackyness,
and generalize the cat -n code along the lines of the original
ascii manipulations in getlimits.c.  That would give speed
and limitless numbers and no dependencies between tools.
That would take more than the 5 mins I spent on this,
and Paul said he might like to give it as a project to
a student, so I'll leave it for the moment.

At least we now have a benchmark to beat. 2500%!

lol
Pádraig.


From MAILER-DAEMON Fri Apr 01 06:16:34 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5bP8-0002mK-EN
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 06:16:34 -0400
Received: from [140.186.70.92] (port=45962 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5bP6-0002ka-5l
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:16:33 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5bP5-0001v5-4y
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:16:32 -0400
Received: from mx.meyering.net ([82.230.74.64]:53931)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5bP4-0001tn-Se
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:16:31 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id AAD2C60315; Fri,  1 Apr 2011 12:16:28 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <4D959F97.6090607@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Fri, 01 Apr 2011 10:49:11 +0100")
References: <4D95900E.4010101@draigBrady.com>
	<87aagamioa.fsf@rho.meyering.net> <4D959F97.6090607@draigBrady.com>
Date: Fri, 01 Apr 2011 12:16:28 +0200
Message-ID: <87zkoajmyr.fsf@rho.meyering.net>
Lines: 52
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 10:16:33 -0000

P=E1draig Brady wrote:

> On 01/04/11 10:20, Jim Meyering wrote:
>> P=E1draig Brady wrote:
>>>     seq: speed up the common case by 560%
>>>
>>>     * seq.c (print_range): Print a range of numbers
>>>     without using if statements for speed.
>>>     (print_long_range): Print a range of numbers
>>>     using the unix paradigm of using other utils
>>>     that do it better.
>>
>> Great idea.
>
>> I get this:
>>
>>   $ env time ./seq 10000000 > /dev/null
>>   0.50user 0.06system 0:00.20elapsed 273%CPU (0avgtext+0avgdata 4800maxr=
esident)k
>>   ...
>>
>> Compare with the old version's time:
>>
>>   5.06user 0.00system 0:05.08elapsed 99%CPU (0avgtext+0avgdata 2608maxre=
sident)k
>>
>> So here I benefit from parallelism, and end up with not just 560% (5.6x)
>> but a 25x(!) speed-up (on an i7-970).
>
> My April fool's joke has spectacularly backfired.

You choose too well.
seq's performance is a personal pain point,
since I use it regularly, and when it doesn't
scale well, I end up waiting an extra second or two
for larger numbers.  "waiting for seq" is annoying
when I know how much faster it can be.

Yeah, yeah, I should just replace it with a function
that runs the pipeline when there's a single number....
Ok.  Now I'll use this:

seq()
{
  case $#:$1 in 1:[0-9]*) yes | head -n"$1" | cat -n | tr -cd '[0-9\n]';;
    *) env seq "$@";; esac
}

It can generate half a billion numbers in 10 seconds:

    $ time seq 500000000 |tail -3
    500000000
    seq 500000000  25.98s user 5.08s system 305% cpu 10.174 total
    tail  4.92s user 1.62s system 64% cpu 10.174 total


From MAILER-DAEMON Fri Apr 01 06:29:15 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5bbP-0001bH-7u
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 06:29:15 -0400
Received: from [140.186.70.92] (port=41776 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5bbN-0001b6-M6
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:29:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5bbL-000547-Vz
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:29:13 -0400
Received: from mx.meyering.net ([82.230.74.64]:35630)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5bbL-00053f-N8
	for coreutils@gnu.org; Fri, 01 Apr 2011 06:29:11 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 2E19D6049F; Fri,  1 Apr 2011 12:29:08 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <87zkoajmyr.fsf@rho.meyering.net> (Jim Meyering's message of
	"Fri, 01 Apr 2011 12:16:28 +0200")
References: <4D95900E.4010101@draigBrady.com>
	<87aagamioa.fsf@rho.meyering.net> <4D959F97.6090607@draigBrady.com>
	<87zkoajmyr.fsf@rho.meyering.net>
Date: Fri, 01 Apr 2011 12:29:08 +0200
Message-ID: <87tyeijmdn.fsf@rho.meyering.net>
Lines: 32
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
Subject: Re: [PATCH] seq: speed up the common case by 25x
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 10:29:14 -0000

Jim Meyering wrote:
...
> Yeah, yeah, I should just replace it with a function
> that runs the pipeline when there's a single number....
> Ok.  Now I'll use this:
>
> seq()
> {
>   case $#:$1 in 1:[0-9]*) yes | head -n"$1" | cat -n | tr -cd '[0-9\n]';;
>     *) env seq "$@";; esac
> }
>
> It can generate half a billion numbers in 10 seconds:
>
>     $ time seq 500000000 |tail -3
>     500000000
>     seq 500000000  25.98s user 5.08s system 305% cpu 10.174 total
>     tail  4.92s user 1.62s system 64% cpu 10.174 total

It doesn't change the output, but it's better to avoid
the use of [] above.  If there had been brackets in
the input, that use of GNU tr would not have removed them.
Here's a fixed version:

seq()
{
  # Up to ~25 times faster than seq-8.10 for large N.
  case $#:$1 in
    1:[0-9]*) yes | head -n"$1" | cat -n | tr -cd '0-9\n';;
    *) env seq "$@";;
  esac
}


From MAILER-DAEMON Fri Apr 01 17:39:23 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5m3v-00005v-KH
	for mharc-coreutils@gnu.org; Fri, 01 Apr 2011 17:39:23 -0400
Received: from [140.186.70.92] (port=40728 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5m3t-0008Sh-8R
	for coreutils@gnu.org; Fri, 01 Apr 2011 17:39:22 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q5m3s-0005mT-22
	for coreutils@gnu.org; Fri, 01 Apr 2011 17:39:21 -0400
Received: from qmta11.emeryville.ca.mail.comcast.net ([76.96.27.211]:43269)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q5m3r-0005m6-N4
	for coreutils@gnu.org; Fri, 01 Apr 2011 17:39:20 -0400
Received: from omta14.emeryville.ca.mail.comcast.net ([76.96.30.60])
	by qmta11.emeryville.ca.mail.comcast.net with comcast
	id SMct1g0041HpZEsABMfHDw; Fri, 01 Apr 2011 21:39:17 +0000
Received: from office.redhat.com ([24.10.251.25])
	by omta14.emeryville.ca.mail.comcast.net with comcast
	id SMfE1g0070ZdyUg8aMfGde; Fri, 01 Apr 2011 21:39:17 +0000
From: Eric Blake <eblake@redhat.com>
To: coreutils@gnu.org
Date: Fri,  1 Apr 2011 15:39:13 -0600
Message-Id: <1301693953-2498-1-git-send-email-eblake@redhat.com>
X-Mailer: git-send-email 1.7.4
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 76.96.27.211
Subject: [PATCH] docs: mention POSIX 2008
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 01 Apr 2011 21:39:22 -0000

* doc/coreutils.texi (Standards conformance): Give value of
_POSIX2_VERSION matching the _POSIX_C_SOURCE of POSIX 2008.
---

Any objections to this patch?  While I don't know of any place
in coreutils that specifically distinguishes between 2001 and 2008,
we might as well document the value (I find that 'info coreutils
standards' is a handy reference for valid values to give to
_POSIX_C_SOURCE when writing strictly compliant programs).

 doc/coreutils.texi |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index ea35afe..f603cd2 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -1455,10 +1455,11 @@ Standards conformance
 that is standard for your system.  To cause them to conform to a
 different version of @acronym{POSIX}, define the @env{_POSIX2_VERSION}
 environment variable to a value of the form @var{yyyymm} specifying
-the year and month the standard was adopted.  Two values are currently
+the year and month the standard was adopted.  Three values are currently
 supported for @env{_POSIX2_VERSION}: @samp{199209} stands for
-@acronym{POSIX} 1003.2-1992, and @samp{200112} stands for @acronym{POSIX}
-1003.1-2001.  For example, if you have a newer system but are running software
+@acronym{POSIX} 1003.2-1992, @samp{200112} stands for @acronym{POSIX}
+1003.1-2001, and @samp{200809} stands for @acronym{POSIX} 1003.1-2008.
+For example, if you have a newer system but are running software
 that assumes an older version of @acronym{POSIX} and uses @samp{sort +1}
 or @samp{tail +10}, you can work around any compatibility problems by setting
 @samp{_POSIX2_VERSION=199209} in your environment.
-- 
1.7.4



From MAILER-DAEMON Sat Apr 02 02:39:46 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q5uUs-0002OD-Mh
	for mharc-coreutils@gnu.org; Sat, 02 Apr 2011 02:39:46 -0400
Received: from [140.186.70.92] (port=39238 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q5uUr-0002MH-0f
	for coreutils@gnu.org; Sat, 02 Apr 2011 02:39:45 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5uUq-0000Vt-1F
	for coreutils@gnu.org; Sat, 02 Apr 2011 02:39:44 -0400
Received: from mx.meyering.net ([82.230.74.64]:59628)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q5uUp-0000Vi-Ox
	for coreutils@gnu.org; Sat, 02 Apr 2011 02:39:43 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id C9374602C8; Sat,  2 Apr 2011 08:39:41 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Eric Blake <eblake@redhat.com>
In-Reply-To: <1301693953-2498-1-git-send-email-eblake@redhat.com> (Eric
	Blake's message of "Fri, 1 Apr 2011 15:39:13 -0600")
References: <1301693953-2498-1-git-send-email-eblake@redhat.com>
Date: Sat, 02 Apr 2011 08:39:41 +0200
Message-ID: <87aag9gnrm.fsf@rho.meyering.net>
Lines: 12
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
Subject: Re: [PATCH] docs: mention POSIX 2008
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 02 Apr 2011 06:39:45 -0000

Eric Blake wrote:
> * doc/coreutils.texi (Standards conformance): Give value of
> _POSIX2_VERSION matching the _POSIX_C_SOURCE of POSIX 2008.
> ---
>
> Any objections to this patch?  While I don't know of any place
> in coreutils that specifically distinguishes between 2001 and 2008,
> we might as well document the value (I find that 'info coreutils
> standards' is a handy reference for valid values to give to
> _POSIX_C_SOURCE when writing strictly compliant programs).

Thanks.  That looks like a fine change.


From MAILER-DAEMON Sat Apr 02 10:12:57 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q61ZR-0007PZ-0C
	for mharc-coreutils@gnu.org; Sat, 02 Apr 2011 10:12:57 -0400
Received: from [140.186.70.92] (port=41991 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q61ZK-0007Ea-6S
	for coreutils@gnu.org; Sat, 02 Apr 2011 10:12:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q61ZI-0007Hs-Je
	for coreutils@gnu.org; Sat, 02 Apr 2011 10:12:50 -0400
Received: from mx.meyering.net ([82.230.74.64]:56418)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q61ZH-0007HI-Uh
	for coreutils@gnu.org; Sat, 02 Apr 2011 10:12:48 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id CB382602C8; Sat,  2 Apr 2011 16:12:40 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Sat, 02 Apr 2011 16:12:40 +0200
Message-ID: <87wrjceo87.fsf@rho.meyering.net>
Lines: 258
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH] maint: prohibit direct use of strncmp: prefer STREQ_LEN,
	STRNCMP_LIT
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 02 Apr 2011 14:12:55 -0000

Here's a proposed change.
It makes the code a little cleaner by removing hard-coded
string lengths at the expense of introducing new macros:

    -              if (strncmp ("/auto/", mount_point, 6) == 0)
    +              if (STRNCMP_LIT (mount_point, "/auto/") == 0)

    -              else if (strncmp ("/tmp_mnt/", mount_point, 9) == 0)
    +              else if (STRNCMP_LIT (mount_point, "/tmp_mnt/") == 0)

And along the lines of the strcmp->STREQ conversion,
now we can do the same with strncmp:

    -  if (strncmp (s + i, DIGEST_TYPE_STRING, algo_name_len) == 0)
    +  if (STREQ_LEN (s + i, DIGEST_TYPE_STRING, algo_name_len))


>From 5e09d11701b283f5c2badcb7ac9f3d42c862a3e0 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Thu, 31 Mar 2011 17:59:16 +0200
Subject: [PATCH] maint: prohibit direct use of strncmp: prefer STREQ_LEN,
 STRNCMP_LIT

* cfg.mk (sc_prohibit_strncmp): New rule, mostly from libvirt.
* src/system.h (STREQ_LEN, STRPREFIX, STRNCMP_LIT): Define.
* src/df.c (get_dev, get_point): Convert.
* src/extent-scan.c (extent_need_sync): Likewise.
* src/ls.c (is_colored, decode_switches): Likewise.
(parse_ls_color, (print_color_indicator): Likewise.
* src/md5sum.c (split_3): Likewise.
* src/split.c (main, emit_ancillary_info): Likewise.
* src/tr.c (look_up_char_class): Likewise.
* src/uname.c (main): Likewise.
* src/who.c (scan_entries): Likewise.
---
 cfg.mk            |    9 +++++++++
 src/df.c          |    6 +++---
 src/extent-scan.c |    2 +-
 src/ls.c          |   14 +++++++-------
 src/md5sum.c      |    2 +-
 src/split.c       |    4 ++--
 src/system.h      |    9 ++++++++-
 src/tr.c          |    2 +-
 src/uname.c       |    2 +-
 src/who.c         |    2 +-
 10 files changed, 34 insertions(+), 18 deletions(-)

diff --git a/cfg.mk b/cfg.mk
index 99a6e5e..48e5a47 100644
--- a/cfg.mk
+++ b/cfg.mk
@@ -315,6 +315,15 @@ sc_space_before_open_paren:
 	else :;								\
 	fi

+# Similar to the gnulib maint.mk rule for sc_prohibit_strcmp
+# Use STREQ_LEN or STRPREFIX rather than comparing strncmp == 0, or != 0.
+sc_prohibit_strncmp:
+	@grep -nE '! *str''ncmp *\(|\<str''ncmp *\(.+\) *[!=]='		\
+	    $$($(VC_LIST_EXCEPT))					\
+	  | grep -vE ':# *define STR(N?EQ_LEN|PREFIX)\(' &&		\
+	  { echo '$(ME): use STREQ_LEN or STRPREFIX instead of str''ncmp' \
+		1>&2; exit 1; } || :
+
 # Override the default Cc: used in generating an announcement.
 announcement_Cc_ = $(translation_project_), \
   coreutils@gnu.org, coreutils-announce@gnu.org
diff --git a/src/df.c b/src/df.c
index 357dca5..14f0790 100644
--- a/src/df.c
+++ b/src/df.c
@@ -611,9 +611,9 @@ get_dev (char const *disk, char const *mount_point,
               /* Don't print the first directory name in MOUNT_POINT if it's an
                  artifact of an automounter.  This is a bit too aggressive to be
                  the default.  */
-              if (strncmp ("/auto/", mount_point, 6) == 0)
+              if (STRNCMP_LIT (mount_point, "/auto/") == 0)
                 mount_point += 5;
-              else if (strncmp ("/tmp_mnt/", mount_point, 9) == 0)
+              else if (STRNCMP_LIT (mount_point, "/tmp_mnt/") == 0)
                 mount_point += 8;
 #endif
               cell = xstrdup (mount_point);
@@ -682,7 +682,7 @@ get_point (const char *point, const struct stat *statp)
           if (best_match_len <= len && len <= resolved_len
               && (len == 1 /* root file system */
                   || ((len == resolved_len || resolved[len] == '/')
-                      && strncmp (me->me_mountdir, resolved, len) == 0)))
+                      && STREQ_LEN (me->me_mountdir, resolved, len))))
             {
               best_match = me;
               best_match_len = len;
diff --git a/src/extent-scan.c b/src/extent-scan.c
index c0a5de6..4ddf46e 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -45,7 +45,7 @@ extent_need_sync (void)
       need_sync = 0; /* No workaround by default.  */

 #ifdef __linux__
-      if (uname (&name) != -1 && strncmp (name.release, "2.6.", 4) == 0)
+      if (uname (&name) != -1 && STRNCMP_LIT (name.release, "2.6.") == 0)
         {
            unsigned long val;
            if (xstrtoul (name.release + 4, NULL, 10, &val, NULL) == LONGINT_OK)
diff --git a/src/ls.c b/src/ls.c
index c09a913..a7ea8c2 100644
--- a/src/ls.c
+++ b/src/ls.c
@@ -1131,8 +1131,8 @@ is_colored (enum indicator_no type)
   size_t len = color_indicator[type].len;
   char const *s = color_indicator[type].string;
   return ! (len == 0
-            || (len == 1 && strncmp (s, "0", 1) == 0)
-            || (len == 2 && strncmp (s, "00", 2) == 0));
+            || (len == 1 && STRNCMP_LIT (s, "0") == 0)
+            || (len == 2 && STRNCMP_LIT (s, "00") == 0));
 }

 static void
@@ -1996,7 +1996,7 @@ decode_switches (int argc, char **argv)
         if (! (style = getenv ("TIME_STYLE")))
           style = bad_cast ("locale");

-      while (strncmp (style, posix_prefix, sizeof posix_prefix - 1) == 0)
+      while (STREQ_LEN (style, posix_prefix, sizeof posix_prefix - 1))
         {
           if (! hard_locale (LC_TIME))
             return optind;
@@ -2388,7 +2388,7 @@ parse_ls_color (void)
     }

   if (color_indicator[C_LINK].len == 6
-      && !strncmp (color_indicator[C_LINK].string, "target", 6))
+      && !STRNCMP_LIT (color_indicator[C_LINK].string, "target"))
     color_symlink_as_referent = true;
 }

@@ -4178,7 +4178,7 @@ print_color_indicator (const struct fileinfo *f, bool symlink_target)
         }
       else if (S_ISLNK (mode))
         type = ((!linkok
-                 && (!strncmp (color_indicator[C_LINK].string, "target", 6)
+                 && (!STRNCMP_LIT (color_indicator[C_LINK].string, "target")
                      || color_indicator[C_ORPHAN].string))
                 ? C_ORPHAN : C_LINK);
       else if (S_ISFIFO (mode))
@@ -4209,8 +4209,8 @@ print_color_indicator (const struct fileinfo *f, bool symlink_target)
       for (ext = color_ext_list; ext != NULL; ext = ext->next)
         {
           if (ext->ext.len <= len
-              && strncmp (name - ext->ext.len, ext->ext.string,
-                          ext->ext.len) == 0)
+              && STREQ_LEN (name - ext->ext.len, ext->ext.string,
+                            ext->ext.len))
             break;
         }
     }
diff --git a/src/md5sum.c b/src/md5sum.c
index 697689b..6f6e637 100644
--- a/src/md5sum.c
+++ b/src/md5sum.c
@@ -262,7 +262,7 @@ split_3 (char *s, size_t s_len,

   /* Check for BSD-style checksum line. */
   algo_name_len = strlen (DIGEST_TYPE_STRING);
-  if (strncmp (s + i, DIGEST_TYPE_STRING, algo_name_len) == 0)
+  if (STREQ_LEN (s + i, DIGEST_TYPE_STRING, algo_name_len))
     {
       if (s[i + algo_name_len] == ' ')
         ++i;
diff --git a/src/split.c b/src/split.c
index 364576a..3a630a0 100644
--- a/src/split.c
+++ b/src/split.c
@@ -895,12 +895,12 @@ main (int argc, char **argv)
           /* skip any whitespace */
           while (isspace (to_uchar (*optarg)))
             optarg++;
-          if (strncmp (optarg, "r/", 2) == 0)
+          if (STRNCMP_LIT (optarg, "r/") == 0)
             {
               split_type = type_rr;
               optarg += 2;
             }
-          else if (strncmp (optarg, "l/", 2) == 0)
+          else if (STRNCMP_LIT (optarg, "l/") == 0)
             {
               split_type = type_chunk_lines;
               optarg += 2;
diff --git a/src/system.h b/src/system.h
index b86e570..1c351bd 100644
--- a/src/system.h
+++ b/src/system.h
@@ -257,6 +257,13 @@ select_plural (uintmax_t n)
 }

 #define STREQ(a, b) (strcmp (a, b) == 0)
+#define STREQ_LEN(a, b, n) (strncmp (a, b, n) == 0)
+#define STRPREFIX(a, b) (strncmp(a, b, strlen (b)) == 0)
+
+/* Just like strncmp, but the first argument must be a literal string
+   and you don't specify the length.  */
+#define STRNCMP_LIT(s, literal) \
+  strncmp (s, "" literal "", sizeof (literal) - 1)

 #if !HAVE_DECL_GETLOGIN
 char *getlogin ();
@@ -607,7 +614,7 @@ emit_ancillary_info (void)
   /* Don't output this redundant message for English locales.
      Note we still output for 'C' so that it gets included in the man page.  */
   const char *lc_messages = setlocale (LC_MESSAGES, NULL);
-  if (lc_messages && strncmp (lc_messages, "en_", 3))
+  if (lc_messages && STRNCMP_LIT (lc_messages, "en_"))
     {
       /* TRANSLATORS: Replace LANG_CODE in this URL with your language code
          <http://translationproject.org/team/LANG_CODE.html> to form one of
diff --git a/src/tr.c b/src/tr.c
index 2809f87..2a729c6 100644
--- a/src/tr.c
+++ b/src/tr.c
@@ -548,7 +548,7 @@ look_up_char_class (char const *class_str, size_t len)
   enum Char_class i;

   for (i = 0; i < ARRAY_CARDINALITY (char_class_name); i++)
-    if (strncmp (class_str, char_class_name[i], len) == 0
+    if (STREQ_LEN (class_str, char_class_name[i], len)
         && strlen (char_class_name[i]) == len)
       return i;
   return CC_NO_CLASS;
diff --git a/src/uname.c b/src/uname.c
index 7ed1bc8..dd5de54 100644
--- a/src/uname.c
+++ b/src/uname.c
@@ -333,7 +333,7 @@ main (int argc, char **argv)

               /* Hack "safely" around the ppc vs. powerpc return value. */
               if (cputype == CPU_TYPE_POWERPC
-                  && strncmp (element, "ppc", 3) == 0)
+                  && STRNCMP_LIT (element, "ppc") == 0)
                 element = "powerpc";
             }
 # endif
diff --git a/src/who.c b/src/who.c
index 2a82b61..7449ab1 100644
--- a/src/who.c
+++ b/src/who.c
@@ -584,7 +584,7 @@ scan_entries (size_t n, const STRUCT_UTMP *utmp_buf)
       ttyname_b = ttyname (STDIN_FILENO);
       if (!ttyname_b)
         return;
-      if (strncmp (ttyname_b, DEV_DIR_WITH_TRAILING_SLASH, DEV_DIR_LEN) == 0)
+      if (STRNCMP_LIT (ttyname_b, DEV_DIR_WITH_TRAILING_SLASH) == 0)
         ttyname_b += DEV_DIR_LEN;	/* Discard /dev/ prefix.  */
     }

-- 
1.7.4.2.662.gcbd0


From MAILER-DAEMON Sat Apr 02 10:14:33 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q61az-0008IR-H9
	for mharc-coreutils@gnu.org; Sat, 02 Apr 2011 10:14:33 -0400
Received: from [140.186.70.92] (port=43074 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q61aw-0008H4-MZ
	for coreutils@gnu.org; Sat, 02 Apr 2011 10:14:31 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q61av-0007iM-JR
	for coreutils@gnu.org; Sat, 02 Apr 2011 10:14:30 -0400
Received: from mx.meyering.net ([82.230.74.64]:56543)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q61av-0007i8-8M
	for coreutils@gnu.org; Sat, 02 Apr 2011 10:14:29 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id A072D602C8; Sat,  2 Apr 2011 16:14:28 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Sat, 02 Apr 2011 16:14:28 +0200
Message-ID: <87r59keo57.fsf@rho.meyering.net>
Lines: 82
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH] cp: always detect copy-into-self: avoid infloop w/large
	PATH_MAX
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 02 Apr 2011 14:14:31 -0000

FYI,

>From b368d453ff27820c769564b4170f0c81b1558b2f Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Mon, 28 Mar 2011 09:14:31 +0200
Subject: [PATCH] cp: always detect copy-into-self: avoid infloop w/large
 PATH_MAX

When running the erroneous command, cp -rl A D D, and depending on the
structure of directories A and D and the file system type (because that
changes order of dir. entry traversal), cp would sometimes fail to
detect that D was being copied into D, and would create D/D/D/D/D/...
until it hit PATH_MAX or exhausted some resource.
I noticed this via the occasional failure of the cp/into-self test
when run using a ZFS file system.  It is occasional because the bug
is dependent on the order in which directory entries are traversed,
and that is apparently indeterminate with ZFS.
Technically, with the current recursive implementation, there is no
risk of an infinite loop, due to stack limitations, but with an
eventual fts-based implementation, it might have iterated until
disk space or inodes are exhausted.
* src/copy.c (copy_dir): Avoid copy-into-self interminable loop on
systems with large PATH_MAX.  On other systems, diagnose the copy-into-
self error consistently.  Handle the parameter,
first_dir_created_per_command_line_arg, correctly when there are two
or more sub-directories.
---
 src/copy.c |   12 ++++++++++--
 1 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index 05f92b3..d479b9c 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -566,8 +566,11 @@ copy_attr (char const *src_path ATTRIBUTE_UNUSED,
    DST_NAME_IN is a directory that was created previously in the
    recursion.   SRC_SB and ANCESTORS describe SRC_NAME_IN.
    Set *COPY_INTO_SELF if SRC_NAME_IN is a parent of
-   FIRST_DIR_CREATED_PER_COMMAND_LINE_ARG  FIXME
    (or the same as) DST_NAME_IN; otherwise, clear it.
+   Propagate *FIRST_DIR_CREATED_PER_COMMAND_LINE_ARG from
+   caller to each invocation of copy_internal.  Be careful to
+   pass the address of a temporary, and to update
+   *FIRST_DIR_CREATED_PER_COMMAND_LINE_ARG only upon completion.
    Return true if successful.  */

 static bool
@@ -596,16 +599,18 @@ copy_dir (char const *src_name_in, char const *dst_name_in, bool new_dst,
   if (x->dereference == DEREF_COMMAND_LINE_ARGUMENTS)
     non_command_line_options.dereference = DEREF_NEVER;

+  bool new_first_dir_created = false;
   namep = name_space;
   while (*namep != '\0')
     {
       bool local_copy_into_self;
       char *src_name = file_name_concat (src_name_in, namep, NULL);
       char *dst_name = file_name_concat (dst_name_in, namep, NULL);
+      bool first_dir_created = *first_dir_created_per_command_line_arg;

       ok &= copy_internal (src_name, dst_name, new_dst, src_sb->st_dev,
                            ancestors, &non_command_line_options, false,
-                           first_dir_created_per_command_line_arg,
+                           &first_dir_created,
                            &local_copy_into_self, NULL);
       *copy_into_self |= local_copy_into_self;

@@ -618,9 +623,12 @@ copy_dir (char const *src_name_in, char const *dst_name_in, bool new_dst,
       if (local_copy_into_self)
         break;

+      new_first_dir_created |= first_dir_created;
       namep += strlen (namep) + 1;
     }
   free (name_space);
+  *first_dir_created_per_command_line_arg = new_first_dir_created;
+
   return ok;
 }

--
1.7.4.2.662.gcbd0


From MAILER-DAEMON Mon Apr 04 01:14:19 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7H-0006mQ-A3
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:19 -0400
Received: from [140.186.70.92] (port=38849 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7F-0006kQ-7S
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:18 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7E-0003aU-7y
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:17 -0400
Received: from mx.meyering.net ([82.230.74.64]:47105 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7D-0003a4-NR
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:16 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345DqYi029296
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:13:52 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345DpOi029295
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:51 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:36 +0200
Message-Id: <1301894021-29180-2-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH 1/6] tests: minor improvement of sc_tight_scope rule
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:18 -0000

From: Jim Meyering <meyering@redhat.com>

* src/Makefile.am (sc_tight_scope): Adjust rule to use an eval-based
trap-setting for-loop rather than 4x hard-coded 128+N constants.
Also catch SIGQUIT (3).  Tweak comments.
---
 src/Makefile.am |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/Makefile.am b/src/Makefile.am
index 5bd71d2..1727d67 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -707,8 +707,8 @@ sc_check-AUTHORS: $(all_programs)

 # Most functions in src/*.c should have static scope.
 # Any that don't must be marked with `extern', but `main'
-# and `usage' are exceptions.  They're always extern, but
-# don't need to be marked. Also functions starting with __
+# and `usage' are exceptions: they're always extern, but
+# do not need to be marked.  Also functions starting with __
 # are exempted due to possibly being added by the compiler
 # (when compiled as a shared library for example).
 #
@@ -717,9 +717,9 @@ sc_check-AUTHORS: $(all_programs)
 sc_tight_scope: $(bin_PROGRAMS)
 	@t=exceptions-$$$$;						\
 	trap 's=$$?; rm -f $$t; exit $$s' 0;				\
-	am__exit='(exit $s); exit $s';					\
-	trap "s=129; $$am__exit" 1; trap "s=130; $$am__exit" 2;		\
-	trap "s=141; $$am__exit" 13; trap "s=143; $$am__exit" 15;	\
+	for sig in 1 2 3 13 15; do					\
+	  eval "trap 'v=`expr $$sig + 128`; (exit $$v); exit $$v' $$sig"; \
+	done;								\
 	src=`for f in $(SOURCES); do					\
 	       test -f $$f && d= || d=$(srcdir)/; echo $$d$$f; done`;	\
 	hdr=`for f in $(noinst_HEADERS); do				\
-- 
1.7.4.2.662.gcbd0



From MAILER-DAEMON Mon Apr 04 01:14:22 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7K-0006na-EF
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:22 -0400
Received: from [140.186.70.92] (port=38846 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7F-0006k1-1D
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:18 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7E-0003aP-7Y
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:16 -0400
Received: from mx.meyering.net ([82.230.74.64]:47103 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7D-0003Zp-NT
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:16 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345Dnn2029285
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:13:50 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345Dm9h029284
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:48 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:35 +0200
Message-Id: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: minor test improvements
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:19 -0000

Here are a few test-improving patches:

  1/6 tests: minor improvement of sc_tight_scope rule
  2/6 tests: convert common root-build test failure to a "skip"
  3/6 tests: preserve-gid: remove useless use of "env"
  4/6 tests: preserve-gid: don't chown temporary PATH dir to a nameless UID
  5/6 tests: don't ever leave a backgrounded "sleep 10m" process
  6/6 tests: avoid spurious parallel failure due to temporary disk full


From MAILER-DAEMON Mon Apr 04 01:14:23 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7K-0006np-Vk
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:23 -0400
Received: from [140.186.70.92] (port=38863 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7G-0006lH-22
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7E-0003aa-Gt
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:17 -0400
Received: from mx.meyering.net ([82.230.74.64]:47107 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7D-0003aJ-Ry
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:16 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345Dr0P029300
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:13:54 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345DrGo029299
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:53 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:37 +0200
Message-Id: <1301894021-29180-3-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH 2/6] tests: convert common root-build test failure to a
	"skip"
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:19 -0000

From: Jim Meyering <meyering@redhat.com>

* tests/mv/sticky-to-xpart: Skip rather than failing this test
when run as root and the binaries are not accessible by "nobody".
---
 tests/mv/sticky-to-xpart |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/tests/mv/sticky-to-xpart b/tests/mv/sticky-to-xpart
index f0a3d81..f689d9d 100755
--- a/tests/mv/sticky-to-xpart
+++ b/tests/mv/sticky-to-xpart
@@ -47,8 +47,7 @@ version=`
 `
 case $version in
   $PACKAGE_VERSION) ;;
-  *) echo "$0: cannot access just-built mv as user $NON_ROOT_USERNAME" 1>&2
-     fail=1 ;;
+  *) skip_ "cannot access just-built mv as user $NON_ROOT_USERNAME";;
 esac

 setuidgid $NON_ROOT_USERNAME env PATH="$PATH" \
-- 
1.7.4.2.662.gcbd0



From MAILER-DAEMON Mon Apr 04 01:14:23 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7L-0006oC-GH
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:23 -0400
Received: from [140.186.70.92] (port=38881 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7H-0006mF-5I
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7G-0003bA-DP
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:19 -0400
Received: from mx.meyering.net ([82.230.74.64]:47109 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7F-0003aq-K5
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:18 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345Dtjf029305
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:13:55 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345DsvR029303
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:54 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:38 +0200
Message-Id: <1301894021-29180-4-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH 3/6] tests: preserve-gid: remove useless use of "env"
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:20 -0000

From: Jim Meyering <meyering@redhat.com>

* tests/cp/preserve-gid: Remove useless use of "env".
---
 tests/cp/preserve-gid |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tests/cp/preserve-gid b/tests/cp/preserve-gid
index 552032a..4b80fa7 100755
--- a/tests/cp/preserve-gid
+++ b/tests/cp/preserve-gid
@@ -111,7 +111,7 @@ t1() {
   f=$1; shift
   u=$1; shift
   g=$1; shift
-  t0 "$f" "$u" "$g" env \
+  t0 "$f" "$u" "$g" \
       setuidgid -g "$nameless_gid1,$nameless_gid2" \
       "$nameless_uid" env PATH="$tmp_path" "$@"
 }
-- 
1.7.4.2.662.gcbd0



From MAILER-DAEMON Mon Apr 04 01:14:26 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7N-0006p5-Cc
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:25 -0400
Received: from [140.186.70.92] (port=38907 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7I-0006n8-Rv
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:22 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7H-0003bj-T9
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:20 -0400
Received: from mx.meyering.net ([82.230.74.64]:47111 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7H-0003bM-5f
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:19 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345Duvh029309
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:13:57 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345DumI029308
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:56 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:39 +0200
Message-Id: <1301894021-29180-5-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH 4/6] tests: preserve-gid: don't chown temporary PATH dir to
	a nameless UID
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:23 -0000

From: Jim Meyering <meyering@redhat.com>

* tests/cp/preserve-gid: Simply chmod a+rx instead.
That is safer, in case the nameless UID actually has an account,
and might take advantage of root running a program in a directory
under its control
---
 tests/cp/preserve-gid |    7 ++++++-
 1 files changed, 6 insertions(+), 1 deletions(-)

diff --git a/tests/cp/preserve-gid b/tests/cp/preserve-gid
index 4b80fa7..de4bc5b 100755
--- a/tests/cp/preserve-gid
+++ b/tests/cp/preserve-gid
@@ -103,9 +103,14 @@ t0 c1 0 "$nameless_gid2" cp -p
 # $nameless_uid can access it and then make that directory the search path.
 tmp_path=
 cleanup_() { rm -rf "$tmp_path"; }
+
+# Cause mktemp to create a directory directly under /tmp.
+# Setting TMPDIR explicitly is required here, in case $TMPDIR
+# is not readable by our nameless IDs.
+test -d /tmp && TMPDIR=/tmp
 tmp_path=$(mktemp -d) || fail_ "failed to create temporary directory"
 cp "$abs_path_dir_/cp" "$tmp_path"
-chown -R $nameless_uid "$tmp_path"
+chmod -R a+rx "$tmp_path"

 t1() {
   f=$1; shift
-- 
1.7.4.2.662.gcbd0



From MAILER-DAEMON Mon Apr 04 01:14:26 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7O-0006pU-5M
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:26 -0400
Received: from [140.186.70.92] (port=38925 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7K-0006nb-Lq
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:23 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7J-0003c8-GY
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:22 -0400
Received: from mx.meyering.net ([82.230.74.64]:47113 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7J-0003bp-2A
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:21 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345DwdZ029313
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:13:59 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345DvUg029312
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:57 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:40 +0200
Message-Id: <1301894021-29180-6-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH 5/6] tests: don't ever leave a backgrounded "sleep 10m"
	process
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:24 -0000

From: Jim Meyering <meyering@redhat.com>

* tests/misc/help-version: Sleep only ~30s, not 10m.
The latter was a problem when somehow that sleep process would
hang around and thereby prevent (for up to 10m) a normal unmount
of the temporary partition in which I'd run the tests.
---
 tests/misc/help-version |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tests/misc/help-version b/tests/misc/help-version
index f833926..0482209 100755
--- a/tests/misc/help-version
+++ b/tests/misc/help-version
@@ -226,7 +226,7 @@ id_setup () { args=-u; }

 # Use env to avoid invoking built-in sleep of Solaris 11's /bin/sh.
 kill_setup () {
-  env sleep 10m &
+  env sleep 31.5 &
   args=$!
 }

-- 
1.7.4.2.662.gcbd0



From MAILER-DAEMON Mon Apr 04 01:14:29 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6c7Q-0006qn-WB
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 01:14:29 -0400
Received: from [140.186.70.92] (port=38940 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6c7M-0006oX-4o
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:25 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7L-0003cO-8B
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:23 -0400
Received: from mx.meyering.net ([82.230.74.64]:47115 helo=hx.meyering.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@meyering.net>) id 1Q6c7K-0003cE-Oc
	for coreutils@gnu.org; Mon, 04 Apr 2011 01:14:23 -0400
Received: from hx.meyering.net (localhost.localdomain [127.0.0.1])
	by hx.meyering.net (8.14.4/8.14.4) with ESMTP id p345E0CB029319
	for <coreutils@gnu.org>; Mon, 4 Apr 2011 07:14:00 +0200
Received: (from meyering@localhost)
	by hx.meyering.net (8.14.4/8.14.4/Submit) id p345DxlO029316
	for coreutils@gnu.org; Mon, 4 Apr 2011 07:13:59 +0200
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Mon,  4 Apr 2011 07:13:41 +0200
Message-Id: <1301894021-29180-7-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.2.662.gcbd0
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: [PATCH 6/6] tests: avoid spurious parallel failure due to temporary
	disk full
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 05:14:25 -0000

From: Jim Meyering <meyering@redhat.com>

Running the new fiemap-empty test uses 600MB of disk space via
fallocate, and in so doing caused failure in unrelated tests that
were running in parallel on a small file system.  Rather than
simply running fallocate (which allocates the space, inducing
disk full when it fails), skip the test if there is less than
800MB of free space, as computed via stat and awk.
* tests/init.cfg (require_file_system_bytes_free_): New function.
* tests/cp/fiemap-empty: Use it.
---
 tests/cp/fiemap-empty |    7 +++++++
 tests/init.cfg        |    8 ++++++++
 2 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/tests/cp/fiemap-empty b/tests/cp/fiemap-empty
index 203619a..42d816b 100755
--- a/tests/cp/fiemap-empty
+++ b/tests/cp/fiemap-empty
@@ -31,6 +31,13 @@ fallocate -l 1 -n falloc.test ||
   skip_test_ 'this file system lacks FALLOCATE support'
 rm falloc.test

+# Require more space than we'll actually use, so that
+# tests run in parallel do not run out of space.
+# Otherwise, with inadequate space, simply running the following
+# fallocate command would induce a temporary disk-full condition,
+# which would cause failure of unrelated tests run in parallel.
+require_file_system_bytes_free_ 800000000
+
 fallocate -l 600000000 space.test ||
   skip_test_ 'this test needs at least 600MB free space'

diff --git a/tests/init.cfg b/tests/init.cfg
index 0711455..49cc4ee 100644
--- a/tests/init.cfg
+++ b/tests/init.cfg
@@ -151,6 +151,14 @@ require_built_()
   test $skip_ = yes && skip_test_ "required program(s) not built"
 }

+require_file_system_bytes_free_()
+{
+  local req=$1
+  local expr=$(stat -f --printf "($req + %S - 1) / %S <= %f" .)
+  awk "BEGIN{ exit !($expr) }" \
+    || skip_test_ "this test needs at least $req bytes of free space"
+}
+
 uid_is_privileged_()
 {
   # Make sure id -u succeeds.
-- 
1.7.4.2.662.gcbd0



From MAILER-DAEMON Mon Apr 04 02:39:55 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6dS7-0006GH-Sd
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 02:39:55 -0400
Received: from [140.186.70.92] (port=45245 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6dS5-0006BX-Hg
	for coreutils@gnu.org; Mon, 04 Apr 2011 02:39:54 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6dS4-0007i1-F4
	for coreutils@gnu.org; Mon, 04 Apr 2011 02:39:53 -0400
Received: from mx.meyering.net ([82.230.74.64]:37435)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6dS4-0007hi-4K
	for coreutils@gnu.org; Mon, 04 Apr 2011 02:39:52 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 3F85560374; Mon,  4 Apr 2011 08:39:49 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
In-Reply-To: <1301894021-29180-7-git-send-email-jim@meyering.net> (Jim
	Meyering's message of "Mon, 4 Apr 2011 07:13:41 +0200")
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
	<1301894021-29180-7-git-send-email-jim@meyering.net>
Date: Mon, 04 Apr 2011 08:39:49 +0200
Message-ID: <87fwpybjuy.fsf@rho.meyering.net>
Lines: 54
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: Re: [PATCH 6/6] tests: avoid spurious parallel failure due to
	temporary disk full
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 06:39:54 -0000

Jim Meyering wrote:

> From: Jim Meyering <meyering@redhat.com>
>
> Running the new fiemap-empty test uses 600MB of disk space via
> fallocate, and in so doing caused failure in unrelated tests that
> were running in parallel on a small file system.  Rather than
> simply running fallocate (which allocates the space, inducing
> disk full when it fails), skip the test if there is less than
> 800MB of free space, as computed via stat and awk.
> * tests/init.cfg (require_file_system_bytes_free_): New function.
> * tests/cp/fiemap-empty: Use it.
> ---
>  tests/cp/fiemap-empty |    7 +++++++
>  tests/init.cfg        |    8 ++++++++
>  2 files changed, 15 insertions(+), 0 deletions(-)
>
> diff --git a/tests/cp/fiemap-empty b/tests/cp/fiemap-empty
> index 203619a..42d816b 100755
> --- a/tests/cp/fiemap-empty
> +++ b/tests/cp/fiemap-empty
> @@ -31,6 +31,13 @@ fallocate -l 1 -n falloc.test ||
>    skip_test_ 'this file system lacks FALLOCATE support'
>  rm falloc.test
>
> +# Require more space than we'll actually use, so that
> +# tests run in parallel do not run out of space.
> +# Otherwise, with inadequate space, simply running the following
> +# fallocate command would induce a temporary disk-full condition,
> +# which would cause failure of unrelated tests run in parallel.
> +require_file_system_bytes_free_ 800000000
> +
>  fallocate -l 600000000 space.test ||
>    skip_test_ 'this test needs at least 600MB free space'
>
> diff --git a/tests/init.cfg b/tests/init.cfg
> index 0711455..49cc4ee 100644
> --- a/tests/init.cfg
> +++ b/tests/init.cfg
> @@ -151,6 +151,14 @@ require_built_()
>    test $skip_ = yes && skip_test_ "required program(s) not built"
>  }
>
> +require_file_system_bytes_free_()
> +{
> +  local req=$1
> +  local expr=$(stat -f --printf "($req + %S - 1) / %S <= %f" .)
> +  awk "BEGIN{ exit !($expr) }" \
> +    || skip_test_ "this test needs at least $req bytes of free space"

I realized that with awk we don't need the +%S-1,
since it does floating point arithmetic:

  local expr=$(stat -f --printf "$req / %S <= %f" .)


From MAILER-DAEMON Mon Apr 04 04:13:53 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6ev3-0002GP-Bl
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 04:13:53 -0400
Received: from [140.186.70.92] (port=33252 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6euy-0001ax-NT
	for coreutils@gnu.org; Mon, 04 Apr 2011 04:13:50 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q6ejB-0004LB-84
	for coreutils@gnu.org; Mon, 04 Apr 2011 04:01:38 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:6920)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q6ejA-0004Jh-TI
	for coreutils@gnu.org; Mon, 04 Apr 2011 04:01:37 -0400
Received: (qmail 52156 invoked from network); 4 Apr 2011 08:01:32 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 4 Apr 2011 08:01:32 -0000
Message-ID: <4D997AAB.3070807@draigBrady.com>
Date: Mon, 04 Apr 2011 09:00:43 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: coreutils@gnu.org
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
	<1301894021-29180-7-git-send-email-jim@meyering.net>
In-Reply-To: <1301894021-29180-7-git-send-email-jim@meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Subject: Re: [PATCH 6/6] tests: avoid spurious parallel failure due to
 temporary disk full
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 08:13:51 -0000

On 04/04/11 06:13, Jim Meyering wrote:
> From: Jim Meyering <meyering@redhat.com>
> 
> Running the new fiemap-empty test uses 600MB of disk space via
> fallocate, and in so doing caused failure in unrelated tests that
> were running in parallel on a small file system.  Rather than
> simply running fallocate (which allocates the space, inducing
> disk full when it fails), skip the test if there is less than
> 800MB of free space, as computed via stat and awk.

Oops sorry.
Maybe worth noting is that stat works at a lower level,
and doesn't honor quotas etc, whereas df would be more
accurate, and would do the multiplication internally.

df -B1 -P . | tail -n1 | tr -s ' ' | cut -d ' ' -f4

cheers,
Pádraig.


From MAILER-DAEMON Mon Apr 04 05:08:17 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6flh-0008Gp-16
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 05:08:17 -0400
Received: from [140.186.70.92] (port=36784 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6fld-0008BE-3A
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:08:15 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6flb-0007YJ-6D
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:08:12 -0400
Received: from mx.meyering.net ([82.230.74.64]:40689)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6fla-0007Y0-Rl
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:08:11 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 062FB60374; Mon,  4 Apr 2011 11:08:06 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <4D997AAB.3070807@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Mon, 04 Apr 2011 09:00:43 +0100")
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
	<1301894021-29180-7-git-send-email-jim@meyering.net>
	<4D997AAB.3070807@draigBrady.com>
Date: Mon, 04 Apr 2011 11:08:06 +0200
Message-ID: <87mxk69yfd.fsf@rho.meyering.net>
Lines: 33
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
Subject: Re: [PATCH 6/6] tests: avoid spurious parallel failure due to
	temporary disk full
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 09:08:15 -0000

P=E1draig Brady wrote:
> On 04/04/11 06:13, Jim Meyering wrote:
>> From: Jim Meyering <meyering@redhat.com>
>>
>> Running the new fiemap-empty test uses 600MB of disk space via
>> fallocate, and in so doing caused failure in unrelated tests that
>> were running in parallel on a small file system.  Rather than
>> simply running fallocate (which allocates the space, inducing
>> disk full when it fails), skip the test if there is less than
>> 800MB of free space, as computed via stat and awk.

Thanks for looking.

> Oops sorry.
> Maybe worth noting is that stat works at a lower level,

Not sure what you mean?  Both use statfs.

> and doesn't honor quotas etc, whereas df would be more

Hmm... I hadn't thought about quotas.
I never use them, so at least for me, it won't be an issue.

> accurate, and would do the multiplication internally.
>
> df -B1 -P . | tail -n1 | tr -s ' ' | cut -d ' ' -f4

I don't see how using df like that can be more accurate.  I chose not to
use df that way, because multiplying to get the free space size in bytes
would overflow when using my cloud FS with 20EiB free.  64 bits can count
bytes up to "only" 16EiB-1, so I opted to compare native block counts.
That gives a little more breathing room, even if it's only theoretical
for now.


From MAILER-DAEMON Mon Apr 04 05:35:15 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6gBn-0005pY-EE
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 05:35:15 -0400
Received: from [140.186.70.92] (port=36208 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6gBg-0005nZ-Qs
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:35:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q6gBf-0006Ns-LO
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:35:08 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:2090)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q6gBf-0006NM-81
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:35:07 -0400
Received: (qmail 69882 invoked from network); 4 Apr 2011 09:35:05 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 4 Apr 2011 09:35:05 -0000
Message-ID: <4D999098.5040502@draigBrady.com>
Date: Mon, 04 Apr 2011 10:34:16 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
References: <1301894021-29180-1-git-send-email-jim@meyering.net>	<1301894021-29180-7-git-send-email-jim@meyering.net>	<4D997AAB.3070807@draigBrady.com>
	<87mxk69yfd.fsf@rho.meyering.net>
In-Reply-To: <87mxk69yfd.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org
Subject: Re: [PATCH 6/6] tests: avoid spurious parallel failure due to
 temporary disk full
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 09:35:14 -0000

On 04/04/11 10:08, Jim Meyering wrote:
> Pádraig Brady wrote:
>> On 04/04/11 06:13, Jim Meyering wrote:
>>> From: Jim Meyering <meyering@redhat.com>
>>>
>>> Running the new fiemap-empty test uses 600MB of disk space via
>>> fallocate, and in so doing caused failure in unrelated tests that
>>> were running in parallel on a small file system.  Rather than
>>> simply running fallocate (which allocates the space, inducing
>>> disk full when it fails), skip the test if there is less than
>>> 800MB of free space, as computed via stat and awk.
> 
> Thanks for looking.
> 
>> Oops sorry.
>> Maybe worth noting is that stat works at a lower level,
> 
> Not sure what you mean?  Both use statfs.

Oops right. df uses f_bavail, so perhaps it might
be better to use %a rather than %f with `stat`?

This fallocate() behavior is annoying anyway,
as I'm not sure when it's useful, except for
filling up a file system.
It seems like a this would be useful.

#define FALLOC_FL_FULL_SIZE     0x03 /* allocate all or nothing */

cheers,
Pádraig.


From MAILER-DAEMON Mon Apr 04 05:55:40 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6gVY-0007HI-20
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 05:55:40 -0400
Received: from [140.186.70.92] (port=51554 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6gVV-0007Ew-90
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:55:38 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6gVQ-0003di-CR
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:55:33 -0400
Received: from mx.meyering.net ([82.230.74.64]:46627)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6gVQ-0003dJ-14
	for coreutils@gnu.org; Mon, 04 Apr 2011 05:55:32 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 2501260390; Mon,  4 Apr 2011 11:55:30 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
In-Reply-To: <4D999098.5040502@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Mon, 04 Apr 2011 10:34:16 +0100")
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
	<1301894021-29180-7-git-send-email-jim@meyering.net>
	<4D997AAB.3070807@draigBrady.com> <87mxk69yfd.fsf@rho.meyering.net>
	<4D999098.5040502@draigBrady.com>
Date: Mon, 04 Apr 2011 11:55:29 +0200
Message-ID: <87hbae9w8e.fsf@rho.meyering.net>
Lines: 48
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org
Subject: Re: [PATCH 6/6] tests: avoid spurious parallel failure due to
	temporary disk full
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 09:55:38 -0000

P=E1draig Brady wrote:

> On 04/04/11 10:08, Jim Meyering wrote:
>> P=E1draig Brady wrote:
>>> On 04/04/11 06:13, Jim Meyering wrote:
>>>> From: Jim Meyering <meyering@redhat.com>
>>>>
>>>> Running the new fiemap-empty test uses 600MB of disk space via
>>>> fallocate, and in so doing caused failure in unrelated tests that
>>>> were running in parallel on a small file system.  Rather than
>>>> simply running fallocate (which allocates the space, inducing
>>>> disk full when it fails), skip the test if there is less than
>>>> 800MB of free space, as computed via stat and awk.
>>
>> Thanks for looking.
>>
>>> Oops sorry.
>>> Maybe worth noting is that stat works at a lower level,
>>
>> Not sure what you mean?  Both use statfs.
>
> Oops right. df uses f_bavail, so perhaps it might
> be better to use %a rather than %f with `stat`?

Oh, yes.  Definitely.
I had thought I was using that one,
but had skimmed the descriptions too quickly (I spotted "superuser"
in the description of %a, so went with %f ;-).
Thanks!

I may adjust stat --help's description of %f to say this:

  %a   Free blocks available to non-superuser
  %f   Free blocks in file system (available to superuser)

> This fallocate() behavior is annoying anyway,
> as I'm not sure when it's useful, except for
> filling up a file system.
> It seems like a this would be useful.
>
> #define FALLOC_FL_FULL_SIZE     0x03 /* allocate all or nothing */

Good idea.
I read fallocate's manual looking for a command-line option
like that when I realized that its behavior was at the root
of my non-deterministic test failures.

Sounds like it'd make a good new option, if not the default.


From MAILER-DAEMON Mon Apr 04 11:48:36 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6m16-0002Uy-9F
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 11:48:36 -0400
Received: from [140.186.70.92] (port=45772 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6m12-0002T2-Fg
	for coreutils@gnu.org; Mon, 04 Apr 2011 11:48:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q6m10-0007R9-D8
	for coreutils@gnu.org; Mon, 04 Apr 2011 11:48:31 -0400
Received: from mx1.redhat.com ([209.132.183.28]:31080)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q6m10-0007Qm-2Y
	for coreutils@gnu.org; Mon, 04 Apr 2011 11:48:30 -0400
Received: from int-mx02.intmail.prod.int.phx2.redhat.com
	(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p34FmSG8004556
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Mon, 4 Apr 2011 11:48:28 -0400
Received: from [10.3.113.73] (ovpn-113-73.phx2.redhat.com [10.3.113.73])
	by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP
	id p34FmRB5016501; Mon, 4 Apr 2011 11:48:27 -0400
Message-ID: <4D99E846.8050209@redhat.com>
Date: Mon, 04 Apr 2011 09:48:22 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
References: <1301693953-2498-1-git-send-email-eblake@redhat.com>
	<87aag9gnrm.fsf@rho.meyering.net>
In-Reply-To: <87aag9gnrm.fsf@rho.meyering.net>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enig00FA6F9FDB448177F00DA26E"
X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
Subject: Re: [PATCH] docs: mention POSIX 2008
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 15:48:34 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig00FA6F9FDB448177F00DA26E
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 04/02/2011 12:39 AM, Jim Meyering wrote:
> Eric Blake wrote:
>> * doc/coreutils.texi (Standards conformance): Give value of
>> _POSIX2_VERSION matching the _POSIX_C_SOURCE of POSIX 2008.
>> ---
>>
>> Any objections to this patch?  While I don't know of any place
>> in coreutils that specifically distinguishes between 2001 and 2008,
>> we might as well document the value (I find that 'info coreutils
>> standards' is a handy reference for valid values to give to
>> _POSIX_C_SOURCE when writing strictly compliant programs).
>=20
> Thanks.  That looks like a fine change.

Pushed.

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enig00FA6F9FDB448177F00DA26E
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNmehLAAoJEKeha0olJ0NqZ8UH+we7lv68ectdbaHP0eqK3/9U
Ebld7RuRBDT5WXChhTVcA0QZRYUueNWXd8qeO7yxcKwZL+SiMfwQd7nZaT5h9TKq
1zcRXcxvajT71VPvHkwYJODkassnvlZpnmOpLEtFdGIYgjNc20/jixcp6NIAu08l
CZeGyeakFg7JY6S/H9RROvzRPtyTc0uZG9I7jBaIeTbWOC5MqwLunb3KjHRRLd0U
H7A/Tx1KVYJ40vf3Jvbt8cr2M2FveRt165IC3PyGkCgOWK7Y2gzeBHMovfmtcx77
SXO3Df/kiODRYB05y1LwDWRlc1roedht2dMe9An4wybCqj/gTZFzWwwexvdkwVE=
=1kgF
-----END PGP SIGNATURE-----

--------------enig00FA6F9FDB448177F00DA26E--


From MAILER-DAEMON Mon Apr 04 12:23:41 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q6mZ3-0007hr-2S
	for mharc-coreutils@gnu.org; Mon, 04 Apr 2011 12:23:41 -0400
Received: from [140.186.70.92] (port=44761 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q6mZ0-0007gU-2F
	for coreutils@gnu.org; Mon, 04 Apr 2011 12:23:38 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6mYz-0007Vw-3e
	for coreutils@gnu.org; Mon, 04 Apr 2011 12:23:38 -0400
Received: from mx.meyering.net ([82.230.74.64]:50136)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q6mYy-0007Vr-OR
	for coreutils@gnu.org; Mon, 04 Apr 2011 12:23:37 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id E5B01603FF; Mon,  4 Apr 2011 18:23:33 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
In-Reply-To: <1301894021-29180-1-git-send-email-jim@meyering.net> (Jim
	Meyering's message of "Mon, 4 Apr 2011 07:13:35 +0200")
References: <1301894021-29180-1-git-send-email-jim@meyering.net>
Date: Mon, 04 Apr 2011 18:23:33 +0200
Message-ID: <87wrja7zp6.fsf@rho.meyering.net>
Lines: 11
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: Re: minor test improvements
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 04 Apr 2011 16:23:39 -0000

Jim Meyering wrote:
> Here are a few test-improving patches:
>
>   1/6 tests: minor improvement of sc_tight_scope rule
>   2/6 tests: convert common root-build test failure to a "skip"
>   3/6 tests: preserve-gid: remove useless use of "env"
>   4/6 tests: preserve-gid: don't chown temporary PATH dir to a nameless UID
>   5/6 tests: don't ever leave a backgrounded "sleep 10m" process
>   6/6 tests: avoid spurious parallel failure due to temporary disk full

I've pushed this series, with 6/6 changed as discussed.


From MAILER-DAEMON Tue Apr 05 06:34:15 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q73aR-0005j4-Cc
	for mharc-coreutils@gnu.org; Tue, 05 Apr 2011 06:34:15 -0400
Received: from [140.186.70.92] (port=46863 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q73aN-0005ih-Oy
	for coreutils@gnu.org; Tue, 05 Apr 2011 06:34:13 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q73aL-0003oL-Pp
	for coreutils@gnu.org; Tue, 05 Apr 2011 06:34:11 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:14351)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q73aL-0003m6-7h
	for coreutils@gnu.org; Tue, 05 Apr 2011 06:34:09 -0400
Received: (qmail 10360 invoked from network); 5 Apr 2011 10:34:04 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 5 Apr 2011 10:34:04 -0000
Message-ID: <4D9AEFE6.7050305@draigBrady.com>
Date: Tue, 05 Apr 2011 11:33:10 +0100
From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Coreutils <coreutils@gnu.org>
References: <4D74D15A.8040500@draigBrady.com>
In-Reply-To: <4D74D15A.8040500@draigBrady.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/mixed; boundary="------------060007090703070108040500"
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Subject: Re: auto merging extents
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Apr 2011 10:34:13 -0000

This is a multi-part message in MIME format.
--------------060007090703070108040500
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This improves upon http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=3ed0384c
by handling mergeable extents that span fiemap scan boundaries.

The diff, ignoring whitespace changes, is below.
The full patch is attached.

cheers,
PĂˇdraig.

diff --git a/src/extent-scan.c b/src/extent-scan.c
index c0a5de6..1a84bd4 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -67,6 +67,7 @@ extent_scan_init (int src_fd, struct extent_scan *scan)
 {
   scan->fd = src_fd;
   scan->ei_count = 0;
+  scan->ext_info = NULL;
   scan->scan_start = 0;
   scan->initial_scan_failed = false;
   scan->hit_final_extent = false;
@@ -82,10 +83,15 @@ extent_scan_init (int src_fd, struct extent_scan *scan)
 extern bool
 extent_scan_read (struct extent_scan *scan)
 {
+  unsigned int si = 0;
+  struct extent_info *last_ei IF_LINT ( = scan->ext_info);
+
+  while (true)
+    {
   union { struct fiemap f; char c[4096]; } fiemap_buf;
   struct fiemap *fiemap = &fiemap_buf.f;
   struct fiemap_extent *fm_extents = &fiemap->fm_extents[0];
-  enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_extents };
+      enum { count = (sizeof fiemap_buf - sizeof *fiemap)/sizeof *fm_extents };
   verify (count != 0);

   /* This is required at least to initialize fiemap->fm_start,
@@ -107,22 +113,23 @@ extent_scan_read (struct extent_scan *scan)
       return false;
     }

-  /* If 0 extents are returned, then more get_extent_table() are not needed.  */
+      /* If 0 extents are returned, then no more scans are needed.  */
   if (fiemap->fm_mapped_extents == 0)
     {
       scan->hit_final_extent = true;
-      return false;
+          return scan->scan_start != 0;
     }

-  scan->ei_count = fiemap->fm_mapped_extents;
-  scan->ext_info = xnmalloc (scan->ei_count, sizeof (struct extent_info));
+      assert (scan->ei_count <= SIZE_MAX - fiemap->fm_mapped_extents);
+      scan->ei_count += fiemap->fm_mapped_extents;
+      scan->ext_info = xnrealloc (scan->ext_info, scan->ei_count,
+                                  sizeof (struct extent_info));

-  unsigned int i, si = 0;
-  struct extent_info *last_ei IF_LINT ( = scan->ext_info);
-
-  for (i = 0; i < scan->ei_count; i++)
+      unsigned int i = 0;
+      for (i = 0; i < fiemap->fm_mapped_extents; i++)
     {
-      assert (fm_extents[i].fe_logical <= OFF_T_MAX - fm_extents[i].fe_length);
+          assert (fm_extents[i].fe_logical <=
+                  OFF_T_MAX - fm_extents[i].fe_length);

       if (si && last_ei->ext_flags ==
           (fm_extents[i].fe_flags & ~FIEMAP_EXTENT_LAST)
@@ -153,7 +160,7 @@ extent_scan_read (struct extent_scan *scan)
               new_logical = last_ei->ext_logical + last_ei->ext_length;
             }
           length_adjust = new_logical - fm_extents[i].fe_logical;
-          /* If an extent is contained within the previous one, just fail.  */
+              /* If an extent is contained within the previous one, fail.  */
           if (length_adjust < fm_extents[i].fe_length)
             {
               if (scan->scan_start == 0)
@@ -176,15 +183,26 @@ extent_scan_read (struct extent_scan *scan)
         }
     }

-  /* We don't bother reallocating.  We should though if we change
-     to looping through all extents, within this function.  */
-  scan->ei_count = si;
-
   if (last_ei->ext_flags & FIEMAP_EXTENT_LAST)
     scan->hit_final_extent = true;
+
+      /* If we have enough extents, discard the last as it might
+         be merged with one from the next scan.  */
+      if (si > count && !scan->hit_final_extent)
+        last_ei = scan->ext_info + --si;
+
+      /* We don't bother reallocating any trailing slots.  */
+      scan->ei_count = si;
+
+      if (scan->hit_final_extent)
+        break;
   else
     scan->scan_start = last_ei->ext_logical + last_ei->ext_length;

+      if (si >= count)
+        break;
+    }
+
   return true;
 }
 #else

--------------060007090703070108040500
Content-Type: text/plain;
 name="merge-all.am"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="merge-all.am"

RnJvbSA1MTYwZjM1ZDg5N2I0NzNiNDBhMGM5ODE0MzMzMzQ0NTIxMWRjZjIzIE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiA9P1VURi04P3E/UD1DMz1BMWRyYWlnPTIwQnJhZHk/
PSA8UEBkcmFpZ0JyYWR5LmNvbT4KRGF0ZTogVHVlLCA1IEFwciAyMDExIDExOjA0OjEzICsw
MTAwClN1YmplY3Q6IFtQQVRDSF0gY29weTogaGFuZGxlIG1lcmdlYWJsZSBleHRlbnRzIGFj
cm9zcyBmaWVtYXAgc2NhbnMKCiogc3JjL2V4dGVudC1zY2FuLmMgKGV4dGVudF9zY2FuX3Jl
YWQpOiBMb29wIG92ZXIgbXVsdGlwbGUKZmllbWFwIHNjYW5zLCBzbyB3ZSBoYW5kbGUgbWVy
Z2VhYmxlIGV4dGVudHMgdGhhdCBzcGFuIGFjcm9zcwpmaWVtYXAgc2NhbiBib3VuZGFyaWVz
LiAgT25jZSB3ZSBoYXZlIGVub3VnaCB1bmlxdWUgZXh0ZW50cywKd2UgcmV0dXJuIHNvIGFz
IHRvIG1pbmltaXplIG1lbW9yeSB1c2UuCi0tLQogc3JjL2V4dGVudC1zY2FuLmMgfCAgMTk0
ICsrKysrKysrKysrKysrKysrKysrKysrKysrKysrLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
CiAxIGZpbGVzIGNoYW5nZWQsIDEwNiBpbnNlcnRpb25zKCspLCA4OCBkZWxldGlvbnMoLSkK
CmRpZmYgLS1naXQgYS9zcmMvZXh0ZW50LXNjYW4uYyBiL3NyYy9leHRlbnQtc2Nhbi5jCmlu
ZGV4IGMwYTVkZTYuLjFhODRiZDQgMTAwNjQ0Ci0tLSBhL3NyYy9leHRlbnQtc2Nhbi5jCisr
KyBiL3NyYy9leHRlbnQtc2Nhbi5jCkBAIC02Nyw2ICs2Nyw3IEBAIGV4dGVudF9zY2FuX2lu
aXQgKGludCBzcmNfZmQsIHN0cnVjdCBleHRlbnRfc2NhbiAqc2NhbikKIHsKICAgc2Nhbi0+
ZmQgPSBzcmNfZmQ7CiAgIHNjYW4tPmVpX2NvdW50ID0gMDsKKyAgc2Nhbi0+ZXh0X2luZm8g
PSBOVUxMOwogICBzY2FuLT5zY2FuX3N0YXJ0ID0gMDsKICAgc2Nhbi0+aW5pdGlhbF9zY2Fu
X2ZhaWxlZCA9IGZhbHNlOwogICBzY2FuLT5oaXRfZmluYWxfZXh0ZW50ID0gZmFsc2U7CkBA
IC04MiwxMDggKzgzLDEyNSBAQCBleHRlbnRfc2Nhbl9pbml0IChpbnQgc3JjX2ZkLCBzdHJ1
Y3QgZXh0ZW50X3NjYW4gKnNjYW4pCiBleHRlcm4gYm9vbAogZXh0ZW50X3NjYW5fcmVhZCAo
c3RydWN0IGV4dGVudF9zY2FuICpzY2FuKQogewotICB1bmlvbiB7IHN0cnVjdCBmaWVtYXAg
ZjsgY2hhciBjWzQwOTZdOyB9IGZpZW1hcF9idWY7Ci0gIHN0cnVjdCBmaWVtYXAgKmZpZW1h
cCA9ICZmaWVtYXBfYnVmLmY7Ci0gIHN0cnVjdCBmaWVtYXBfZXh0ZW50ICpmbV9leHRlbnRz
ID0gJmZpZW1hcC0+Zm1fZXh0ZW50c1swXTsKLSAgZW51bSB7IGNvdW50ID0gKHNpemVvZiBm
aWVtYXBfYnVmIC0gc2l6ZW9mICpmaWVtYXApIC8gc2l6ZW9mICpmbV9leHRlbnRzIH07Ci0g
IHZlcmlmeSAoY291bnQgIT0gMCk7Ci0KLSAgLyogVGhpcyBpcyByZXF1aXJlZCBhdCBsZWFz
dCB0byBpbml0aWFsaXplIGZpZW1hcC0+Zm1fc3RhcnQsCi0gICAgIGJ1dCBhbHNvIHNlcnZl
cyAoaW4gbWlkIDIwMTApIHRvIGFwcGVhc2UgdmFsZ3JpbmQsIHdoaWNoCi0gICAgIGFwcGVh
cnMgbm90IHRvIGtub3cgdGhlIHNlbWFudGljcyBvZiB0aGUgRklFTUFQIGlvY3RsLiAqLwot
ICBtZW1zZXQgKCZmaWVtYXBfYnVmLCAwLCBzaXplb2YgZmllbWFwX2J1Zik7Ci0KLSAgZmll
bWFwLT5mbV9zdGFydCA9IHNjYW4tPnNjYW5fc3RhcnQ7Ci0gIGZpZW1hcC0+Zm1fZmxhZ3Mg
PSBzY2FuLT5mbV9mbGFnczsKLSAgZmllbWFwLT5mbV9leHRlbnRfY291bnQgPSBjb3VudDsK
LSAgZmllbWFwLT5mbV9sZW5ndGggPSBGSUVNQVBfTUFYX09GRlNFVCAtIHNjYW4tPnNjYW5f
c3RhcnQ7Ci0KLSAgLyogRmFsbCBiYWNrIHRvIHRoZSBzdGFuZGFyZCBjb3B5IGlmIGNhbGwg
aW9jdGwoMikgZmFpbGVkIGZvciB0aGUKLSAgICAgdGhlIGZpcnN0IHRpbWUuICAqLwotICBp
ZiAoaW9jdGwgKHNjYW4tPmZkLCBGU19JT0NfRklFTUFQLCBmaWVtYXApIDwgMCkKLSAgICB7
Ci0gICAgICBpZiAoc2Nhbi0+c2Nhbl9zdGFydCA9PSAwKQotICAgICAgICBzY2FuLT5pbml0
aWFsX3NjYW5fZmFpbGVkID0gdHJ1ZTsKLSAgICAgIHJldHVybiBmYWxzZTsKLSAgICB9Ci0K
LSAgLyogSWYgMCBleHRlbnRzIGFyZSByZXR1cm5lZCwgdGhlbiBtb3JlIGdldF9leHRlbnRf
dGFibGUoKSBhcmUgbm90IG5lZWRlZC4gICovCi0gIGlmIChmaWVtYXAtPmZtX21hcHBlZF9l
eHRlbnRzID09IDApCi0gICAgewotICAgICAgc2Nhbi0+aGl0X2ZpbmFsX2V4dGVudCA9IHRy
dWU7Ci0gICAgICByZXR1cm4gZmFsc2U7Ci0gICAgfQotCi0gIHNjYW4tPmVpX2NvdW50ID0g
ZmllbWFwLT5mbV9tYXBwZWRfZXh0ZW50czsKLSAgc2Nhbi0+ZXh0X2luZm8gPSB4bm1hbGxv
YyAoc2Nhbi0+ZWlfY291bnQsIHNpemVvZiAoc3RydWN0IGV4dGVudF9pbmZvKSk7Ci0KLSAg
dW5zaWduZWQgaW50IGksIHNpID0gMDsKKyAgdW5zaWduZWQgaW50IHNpID0gMDsKICAgc3Ry
dWN0IGV4dGVudF9pbmZvICpsYXN0X2VpIElGX0xJTlQgKCA9IHNjYW4tPmV4dF9pbmZvKTsK
IAotICBmb3IgKGkgPSAwOyBpIDwgc2Nhbi0+ZWlfY291bnQ7IGkrKykKKyAgd2hpbGUgKHRy
dWUpCiAgICAgewotICAgICAgYXNzZXJ0IChmbV9leHRlbnRzW2ldLmZlX2xvZ2ljYWwgPD0g
T0ZGX1RfTUFYIC0gZm1fZXh0ZW50c1tpXS5mZV9sZW5ndGgpOworICAgICAgdW5pb24geyBz
dHJ1Y3QgZmllbWFwIGY7IGNoYXIgY1s0MDk2XTsgfSBmaWVtYXBfYnVmOworICAgICAgc3Ry
dWN0IGZpZW1hcCAqZmllbWFwID0gJmZpZW1hcF9idWYuZjsKKyAgICAgIHN0cnVjdCBmaWVt
YXBfZXh0ZW50ICpmbV9leHRlbnRzID0gJmZpZW1hcC0+Zm1fZXh0ZW50c1swXTsKKyAgICAg
IGVudW0geyBjb3VudCA9IChzaXplb2YgZmllbWFwX2J1ZiAtIHNpemVvZiAqZmllbWFwKS9z
aXplb2YgKmZtX2V4dGVudHMgfTsKKyAgICAgIHZlcmlmeSAoY291bnQgIT0gMCk7CisKKyAg
ICAgIC8qIFRoaXMgaXMgcmVxdWlyZWQgYXQgbGVhc3QgdG8gaW5pdGlhbGl6ZSBmaWVtYXAt
PmZtX3N0YXJ0LAorICAgICAgICAgYnV0IGFsc28gc2VydmVzIChpbiBtaWQgMjAxMCkgdG8g
YXBwZWFzZSB2YWxncmluZCwgd2hpY2gKKyAgICAgICAgIGFwcGVhcnMgbm90IHRvIGtub3cg
dGhlIHNlbWFudGljcyBvZiB0aGUgRklFTUFQIGlvY3RsLiAqLworICAgICAgbWVtc2V0ICgm
ZmllbWFwX2J1ZiwgMCwgc2l6ZW9mIGZpZW1hcF9idWYpOworCisgICAgICBmaWVtYXAtPmZt
X3N0YXJ0ID0gc2Nhbi0+c2Nhbl9zdGFydDsKKyAgICAgIGZpZW1hcC0+Zm1fZmxhZ3MgPSBz
Y2FuLT5mbV9mbGFnczsKKyAgICAgIGZpZW1hcC0+Zm1fZXh0ZW50X2NvdW50ID0gY291bnQ7
CisgICAgICBmaWVtYXAtPmZtX2xlbmd0aCA9IEZJRU1BUF9NQVhfT0ZGU0VUIC0gc2Nhbi0+
c2Nhbl9zdGFydDsKKworICAgICAgLyogRmFsbCBiYWNrIHRvIHRoZSBzdGFuZGFyZCBjb3B5
IGlmIGNhbGwgaW9jdGwoMikgZmFpbGVkIGZvciB0aGUKKyAgICAgICAgIHRoZSBmaXJzdCB0
aW1lLiAgKi8KKyAgICAgIGlmIChpb2N0bCAoc2Nhbi0+ZmQsIEZTX0lPQ19GSUVNQVAsIGZp
ZW1hcCkgPCAwKQorICAgICAgICB7CisgICAgICAgICAgaWYgKHNjYW4tPnNjYW5fc3RhcnQg
PT0gMCkKKyAgICAgICAgICAgIHNjYW4tPmluaXRpYWxfc2Nhbl9mYWlsZWQgPSB0cnVlOwor
ICAgICAgICAgIHJldHVybiBmYWxzZTsKKyAgICAgICAgfQogCi0gICAgICBpZiAoc2kgJiYg
bGFzdF9laS0+ZXh0X2ZsYWdzID09Ci0gICAgICAgICAgKGZtX2V4dGVudHNbaV0uZmVfZmxh
Z3MgJiB+RklFTUFQX0VYVEVOVF9MQVNUKQotICAgICAgICAgICYmIChsYXN0X2VpLT5leHRf
bG9naWNhbCArIGxhc3RfZWktPmV4dF9sZW5ndGgKLSAgICAgICAgICAgICAgPT0gZm1fZXh0
ZW50c1tpXS5mZV9sb2dpY2FsKSkKKyAgICAgIC8qIElmIDAgZXh0ZW50cyBhcmUgcmV0dXJu
ZWQsIHRoZW4gbm8gbW9yZSBzY2FucyBhcmUgbmVlZGVkLiAgKi8KKyAgICAgIGlmIChmaWVt
YXAtPmZtX21hcHBlZF9leHRlbnRzID09IDApCiAgICAgICAgIHsKLSAgICAgICAgICAvKiBN
ZXJnZSBwcmV2aW91cyB3aXRoIGxhc3QuICAqLwotICAgICAgICAgIGxhc3RfZWktPmV4dF9s
ZW5ndGggKz0gZm1fZXh0ZW50c1tpXS5mZV9sZW5ndGg7Ci0gICAgICAgICAgLyogQ29weSBm
bGFncyBpbiBjYXNlIGRpZmZlcmVudC4gICovCi0gICAgICAgICAgbGFzdF9laS0+ZXh0X2Zs
YWdzID0gZm1fZXh0ZW50c1tpXS5mZV9mbGFnczsKKyAgICAgICAgICBzY2FuLT5oaXRfZmlu
YWxfZXh0ZW50ID0gdHJ1ZTsKKyAgICAgICAgICByZXR1cm4gc2Nhbi0+c2Nhbl9zdGFydCAh
PSAwOwogICAgICAgICB9Ci0gICAgICBlbHNlIGlmICgoc2kgPT0gMCAmJiBzY2FuLT5zY2Fu
X3N0YXJ0ID4gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsKQotICAgICAgICAgICAgICAgfHwg
KHNpICYmIGxhc3RfZWktPmV4dF9sb2dpY2FsICsgbGFzdF9laS0+ZXh0X2xlbmd0aCA+Ci0g
ICAgICAgICAgICAgICAgICAgZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsKSkKKworICAgICAg
YXNzZXJ0IChzY2FuLT5laV9jb3VudCA8PSBTSVpFX01BWCAtIGZpZW1hcC0+Zm1fbWFwcGVk
X2V4dGVudHMpOworICAgICAgc2Nhbi0+ZWlfY291bnQgKz0gZmllbWFwLT5mbV9tYXBwZWRf
ZXh0ZW50czsKKyAgICAgIHNjYW4tPmV4dF9pbmZvID0geG5yZWFsbG9jIChzY2FuLT5leHRf
aW5mbywgc2Nhbi0+ZWlfY291bnQsCisgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgc2l6ZW9mIChzdHJ1Y3QgZXh0ZW50X2luZm8pKTsKKworICAgICAgdW5zaWduZWQgaW50
IGkgPSAwOworICAgICAgZm9yIChpID0gMDsgaSA8IGZpZW1hcC0+Zm1fbWFwcGVkX2V4dGVu
dHM7IGkrKykKICAgICAgICAgewotICAgICAgICAgIC8qIEJUUkZTIGJlZm9yZSAyLjYuMzgg
Y291bGQgcmV0dXJuIG92ZXJsYXBwaW5nIGV4dGVudHMKLSAgICAgICAgICAgICBmb3Igc3Bh
cnNlIGZpbGVzLiAgV2UgYWRqdXN0IHRoZSByZXR1cm5lZCBleHRlbnRzCi0gICAgICAgICAg
ICAgcmF0aGVyIHRoYW4gZmFpbGluZywgYXMgb3RoZXJ3aXNlIGl0IHdvdWxkIGJlIGluZWZm
aWNpZW50Ci0gICAgICAgICAgICAgdG8gZGV0ZWN0IHRoaXMgb24gdGhlIGluaXRpYWwgc2Nh
bi4gICovCi0gICAgICAgICAgdWludDY0X3QgbmV3X2xvZ2ljYWw7Ci0gICAgICAgICAgdWlu
dDY0X3QgbGVuZ3RoX2FkanVzdDsKLSAgICAgICAgICBpZiAoc2kgPT0gMCkKLSAgICAgICAg
ICAgIG5ld19sb2dpY2FsID0gc2Nhbi0+c2Nhbl9zdGFydDsKLSAgICAgICAgICBlbHNlCisg
ICAgICAgICAgYXNzZXJ0IChmbV9leHRlbnRzW2ldLmZlX2xvZ2ljYWwgPD0KKyAgICAgICAg
ICAgICAgICAgIE9GRl9UX01BWCAtIGZtX2V4dGVudHNbaV0uZmVfbGVuZ3RoKTsKKworICAg
ICAgICAgIGlmIChzaSAmJiBsYXN0X2VpLT5leHRfZmxhZ3MgPT0KKyAgICAgICAgICAgICAg
KGZtX2V4dGVudHNbaV0uZmVfZmxhZ3MgJiB+RklFTUFQX0VYVEVOVF9MQVNUKQorICAgICAg
ICAgICAgICAmJiAobGFzdF9laS0+ZXh0X2xvZ2ljYWwgKyBsYXN0X2VpLT5leHRfbGVuZ3Ro
CisgICAgICAgICAgICAgICAgICA9PSBmbV9leHRlbnRzW2ldLmZlX2xvZ2ljYWwpKQogICAg
ICAgICAgICAgewotICAgICAgICAgICAgICAvKiBXZSBjb3VsZCByZXR1cm4gaGVyZSBpZiBz
Y2FuLT5zY2FuX3N0YXJ0ID09IDAKLSAgICAgICAgICAgICAgICAgYnV0IGRvbid0IHNvIGFz
IHRvIG1pbmltaXplIHNwZWNpYWwgY2FzZXMuICAqLwotICAgICAgICAgICAgICBuZXdfbG9n
aWNhbCA9IGxhc3RfZWktPmV4dF9sb2dpY2FsICsgbGFzdF9laS0+ZXh0X2xlbmd0aDsKKyAg
ICAgICAgICAgICAgLyogTWVyZ2UgcHJldmlvdXMgd2l0aCBsYXN0LiAgKi8KKyAgICAgICAg
ICAgICAgbGFzdF9laS0+ZXh0X2xlbmd0aCArPSBmbV9leHRlbnRzW2ldLmZlX2xlbmd0aDsK
KyAgICAgICAgICAgICAgLyogQ29weSBmbGFncyBpbiBjYXNlIGRpZmZlcmVudC4gICovCisg
ICAgICAgICAgICAgIGxhc3RfZWktPmV4dF9mbGFncyA9IGZtX2V4dGVudHNbaV0uZmVfZmxh
Z3M7CiAgICAgICAgICAgICB9Ci0gICAgICAgICAgbGVuZ3RoX2FkanVzdCA9IG5ld19sb2dp
Y2FsIC0gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsOwotICAgICAgICAgIC8qIElmIGFuIGV4
dGVudCBpcyBjb250YWluZWQgd2l0aGluIHRoZSBwcmV2aW91cyBvbmUsIGp1c3QgZmFpbC4g
ICovCi0gICAgICAgICAgaWYgKGxlbmd0aF9hZGp1c3QgPCBmbV9leHRlbnRzW2ldLmZlX2xl
bmd0aCkKKyAgICAgICAgICBlbHNlIGlmICgoc2kgPT0gMCAmJiBzY2FuLT5zY2FuX3N0YXJ0
ID4gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsKQorICAgICAgICAgICAgICAgICAgIHx8IChz
aSAmJiBsYXN0X2VpLT5leHRfbG9naWNhbCArIGxhc3RfZWktPmV4dF9sZW5ndGggPgorICAg
ICAgICAgICAgICAgICAgICAgICBmbV9leHRlbnRzW2ldLmZlX2xvZ2ljYWwpKQogICAgICAg
ICAgICAgewotICAgICAgICAgICAgICBpZiAoc2Nhbi0+c2Nhbl9zdGFydCA9PSAwKQotICAg
ICAgICAgICAgICAgIHNjYW4tPmluaXRpYWxfc2Nhbl9mYWlsZWQgPSB0cnVlOwotICAgICAg
ICAgICAgICByZXR1cm4gZmFsc2U7CisgICAgICAgICAgICAgIC8qIEJUUkZTIGJlZm9yZSAy
LjYuMzggY291bGQgcmV0dXJuIG92ZXJsYXBwaW5nIGV4dGVudHMKKyAgICAgICAgICAgICAg
ICAgZm9yIHNwYXJzZSBmaWxlcy4gIFdlIGFkanVzdCB0aGUgcmV0dXJuZWQgZXh0ZW50cwor
ICAgICAgICAgICAgICAgICByYXRoZXIgdGhhbiBmYWlsaW5nLCBhcyBvdGhlcndpc2UgaXQg
d291bGQgYmUgaW5lZmZpY2llbnQKKyAgICAgICAgICAgICAgICAgdG8gZGV0ZWN0IHRoaXMg
b24gdGhlIGluaXRpYWwgc2Nhbi4gICovCisgICAgICAgICAgICAgIHVpbnQ2NF90IG5ld19s
b2dpY2FsOworICAgICAgICAgICAgICB1aW50NjRfdCBsZW5ndGhfYWRqdXN0OworICAgICAg
ICAgICAgICBpZiAoc2kgPT0gMCkKKyAgICAgICAgICAgICAgICBuZXdfbG9naWNhbCA9IHNj
YW4tPnNjYW5fc3RhcnQ7CisgICAgICAgICAgICAgIGVsc2UKKyAgICAgICAgICAgICAgICB7
CisgICAgICAgICAgICAgICAgICAvKiBXZSBjb3VsZCByZXR1cm4gaGVyZSBpZiBzY2FuLT5z
Y2FuX3N0YXJ0ID09IDAKKyAgICAgICAgICAgICAgICAgICAgIGJ1dCBkb24ndCBzbyBhcyB0
byBtaW5pbWl6ZSBzcGVjaWFsIGNhc2VzLiAgKi8KKyAgICAgICAgICAgICAgICAgIG5ld19s
b2dpY2FsID0gbGFzdF9laS0+ZXh0X2xvZ2ljYWwgKyBsYXN0X2VpLT5leHRfbGVuZ3RoOwor
ICAgICAgICAgICAgICAgIH0KKyAgICAgICAgICAgICAgbGVuZ3RoX2FkanVzdCA9IG5ld19s
b2dpY2FsIC0gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsOworICAgICAgICAgICAgICAvKiBJ
ZiBhbiBleHRlbnQgaXMgY29udGFpbmVkIHdpdGhpbiB0aGUgcHJldmlvdXMgb25lLCBmYWls
LiAgKi8KKyAgICAgICAgICAgICAgaWYgKGxlbmd0aF9hZGp1c3QgPCBmbV9leHRlbnRzW2ld
LmZlX2xlbmd0aCkKKyAgICAgICAgICAgICAgICB7CisgICAgICAgICAgICAgICAgICBpZiAo
c2Nhbi0+c2Nhbl9zdGFydCA9PSAwKQorICAgICAgICAgICAgICAgICAgICBzY2FuLT5pbml0
aWFsX3NjYW5fZmFpbGVkID0gdHJ1ZTsKKyAgICAgICAgICAgICAgICAgIHJldHVybiBmYWxz
ZTsKKyAgICAgICAgICAgICAgICB9CisgICAgICAgICAgICAgIGZtX2V4dGVudHNbaV0uZmVf
bG9naWNhbCA9IG5ld19sb2dpY2FsOworICAgICAgICAgICAgICBmbV9leHRlbnRzW2ldLmZl
X2xlbmd0aCAtPSBsZW5ndGhfYWRqdXN0OworICAgICAgICAgICAgICAvKiBQcm9jZXNzIHRo
ZSBhZGp1c3RlZCBleHRlbnQgYWdhaW4uICAqLworICAgICAgICAgICAgICBpLS07CisgICAg
ICAgICAgICAgIGNvbnRpbnVlOworICAgICAgICAgICAgfQorICAgICAgICAgIGVsc2UKKyAg
ICAgICAgICAgIHsKKyAgICAgICAgICAgICAgbGFzdF9laSA9IHNjYW4tPmV4dF9pbmZvICsg
c2k7CisgICAgICAgICAgICAgIGxhc3RfZWktPmV4dF9sb2dpY2FsID0gZm1fZXh0ZW50c1tp
XS5mZV9sb2dpY2FsOworICAgICAgICAgICAgICBsYXN0X2VpLT5leHRfbGVuZ3RoID0gZm1f
ZXh0ZW50c1tpXS5mZV9sZW5ndGg7CisgICAgICAgICAgICAgIGxhc3RfZWktPmV4dF9mbGFn
cyA9IGZtX2V4dGVudHNbaV0uZmVfZmxhZ3M7CisgICAgICAgICAgICAgIHNpKys7CiAgICAg
ICAgICAgICB9Ci0gICAgICAgICAgZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsID0gbmV3X2xv
Z2ljYWw7Ci0gICAgICAgICAgZm1fZXh0ZW50c1tpXS5mZV9sZW5ndGggLT0gbGVuZ3RoX2Fk
anVzdDsKLSAgICAgICAgICAvKiBQcm9jZXNzIHRoZSBhZGp1c3RlZCBleHRlbnQgYWdhaW4u
ICAqLwotICAgICAgICAgIGktLTsKLSAgICAgICAgICBjb250aW51ZTsKLSAgICAgICAgfQot
ICAgICAgZWxzZQotICAgICAgICB7Ci0gICAgICAgICAgbGFzdF9laSA9IHNjYW4tPmV4dF9p
bmZvICsgc2k7Ci0gICAgICAgICAgbGFzdF9laS0+ZXh0X2xvZ2ljYWwgPSBmbV9leHRlbnRz
W2ldLmZlX2xvZ2ljYWw7Ci0gICAgICAgICAgbGFzdF9laS0+ZXh0X2xlbmd0aCA9IGZtX2V4
dGVudHNbaV0uZmVfbGVuZ3RoOwotICAgICAgICAgIGxhc3RfZWktPmV4dF9mbGFncyA9IGZt
X2V4dGVudHNbaV0uZmVfZmxhZ3M7Ci0gICAgICAgICAgc2krKzsKICAgICAgICAgfQotICAg
IH0KIAotICAvKiBXZSBkb24ndCBib3RoZXIgcmVhbGxvY2F0aW5nLiAgV2Ugc2hvdWxkIHRo
b3VnaCBpZiB3ZSBjaGFuZ2UKLSAgICAgdG8gbG9vcGluZyB0aHJvdWdoIGFsbCBleHRlbnRz
LCB3aXRoaW4gdGhpcyBmdW5jdGlvbi4gICovCi0gIHNjYW4tPmVpX2NvdW50ID0gc2k7Cisg
ICAgICBpZiAobGFzdF9laS0+ZXh0X2ZsYWdzICYgRklFTUFQX0VYVEVOVF9MQVNUKQorICAg
ICAgICBzY2FuLT5oaXRfZmluYWxfZXh0ZW50ID0gdHJ1ZTsKKworICAgICAgLyogSWYgd2Ug
aGF2ZSBlbm91Z2ggZXh0ZW50cywgZGlzY2FyZCB0aGUgbGFzdCBhcyBpdCBtaWdodAorICAg
ICAgICAgYmUgbWVyZ2VkIHdpdGggb25lIGZyb20gdGhlIG5leHQgc2Nhbi4gICovCisgICAg
ICBpZiAoc2kgPiBjb3VudCAmJiAhc2Nhbi0+aGl0X2ZpbmFsX2V4dGVudCkKKyAgICAgICAg
bGFzdF9laSA9IHNjYW4tPmV4dF9pbmZvICsgLS1zaTsKKworICAgICAgLyogV2UgZG9uJ3Qg
Ym90aGVyIHJlYWxsb2NhdGluZyBhbnkgdHJhaWxpbmcgc2xvdHMuICAqLworICAgICAgc2Nh
bi0+ZWlfY291bnQgPSBzaTsKIAotICBpZiAobGFzdF9laS0+ZXh0X2ZsYWdzICYgRklFTUFQ
X0VYVEVOVF9MQVNUKQotICAgIHNjYW4tPmhpdF9maW5hbF9leHRlbnQgPSB0cnVlOwotICBl
bHNlCi0gICAgc2Nhbi0+c2Nhbl9zdGFydCA9IGxhc3RfZWktPmV4dF9sb2dpY2FsICsgbGFz
dF9laS0+ZXh0X2xlbmd0aDsKKyAgICAgIGlmIChzY2FuLT5oaXRfZmluYWxfZXh0ZW50KQor
ICAgICAgICBicmVhazsKKyAgICAgIGVsc2UKKyAgICAgICAgc2Nhbi0+c2Nhbl9zdGFydCA9
IGxhc3RfZWktPmV4dF9sb2dpY2FsICsgbGFzdF9laS0+ZXh0X2xlbmd0aDsKKworICAgICAg
aWYgKHNpID49IGNvdW50KQorICAgICAgICBicmVhazsKKyAgICB9CiAKICAgcmV0dXJuIHRy
dWU7CiB9Ci0tIAoxLjcuNAoK
--------------060007090703070108040500--


From MAILER-DAEMON Tue Apr 05 15:08:10 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q7Bbm-0004i4-EA
	for mharc-coreutils@gnu.org; Tue, 05 Apr 2011 15:08:10 -0400
Received: from [140.186.70.92] (port=40324 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q7Bbj-0004gj-N2
	for coreutils@gnu.org; Tue, 05 Apr 2011 15:08:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q7Bbi-0007A6-6d
	for coreutils@gnu.org; Tue, 05 Apr 2011 15:08:07 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:16686)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q7Bbh-00079h-RE
	for coreutils@gnu.org; Tue, 05 Apr 2011 15:08:06 -0400
Received: (qmail 16201 invoked from network); 5 Apr 2011 19:08:03 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 5 Apr 2011 19:08:03 -0000
Message-ID: <4D9B685C.609@draigBrady.com>
Date: Tue, 05 Apr 2011 20:07:08 +0100
From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: coreutils@gnu.org
References: <4D74D15A.8040500@draigBrady.com> <4D9AEFE6.7050305@draigBrady.com>
In-Reply-To: <4D9AEFE6.7050305@draigBrady.com>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Subject: Re: auto merging extents
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Apr 2011 19:08:09 -0000

On 05/04/11 11:33, PĂˇdraig Brady wrote:
> This improves upon http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=3ed0384c
> by handling mergeable extents that span fiemap scan boundaries.
> 
> The diff, ignoring whitespace changes, is below.
> The full patch is attached.

Oops I also need to set the pointer to NULL in
extent_scan_free() in the previous patch.
I also noticed a chance of an unlikely leak:

commit bc5579c7b918a54df1d37547bc90319c8d02a84a
Author: PĂˇdraig Brady <P@draigBrady.com>
Date:   Tue Apr 5 19:16:40 2011 +0100

    copy: fix an unlikely memory leak when a fiemap copy fails

    * src/copy.c (extent_copy): Free the extents array when
    sparse_copy() fails.

diff --git a/src/copy.c b/src/copy.c
index 05f92b3..c7a8d8e 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -415,7 +415,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size,
                                   sparse_mode == SPARSE_ALWAYS,
                                   src_name, dst_name, ext_len, &n_read,
                                   &wrote_hole_at_eof))
-                return false;
+                goto fail;

               dest_pos = ext_start + n_read;
             }


From MAILER-DAEMON Wed Apr 06 04:41:13 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q7OIb-000476-7Y
	for mharc-coreutils@gnu.org; Wed, 06 Apr 2011 04:41:13 -0400
Received: from [140.186.70.92] (port=56343 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q7OIX-000446-4i
	for coreutils@gnu.org; Wed, 06 Apr 2011 04:41:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q7OIT-000430-4u
	for coreutils@gnu.org; Wed, 06 Apr 2011 04:41:06 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:14278)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q7OIS-00042i-LK
	for coreutils@gnu.org; Wed, 06 Apr 2011 04:41:05 -0400
Received: (qmail 21359 invoked from network); 6 Apr 2011 08:41:02 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 6 Apr 2011 08:41:02 -0000
Message-ID: <4D9C26E4.8090007@draigBrady.com>
Date: Wed, 06 Apr 2011 09:40:04 +0100
From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Coreutils <coreutils@gnu.org>
References: <4D74D15A.8040500@draigBrady.com> <4D9AEFE6.7050305@draigBrady.com>
In-Reply-To: <4D9AEFE6.7050305@draigBrady.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/mixed; boundary="------------020400070404040807090600"
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Subject: Re: auto merging extents
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2011 08:41:11 -0000

This is a multi-part message in MIME format.
--------------020400070404040807090600
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The last version had a nasty data corruption inducing
off by one issue.  I'll apply the attached this evening.

cheers,
PĂˇdraig.

--------------020400070404040807090600
Content-Type: text/plain;
 name="merge-all.am"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="merge-all.am"

RnJvbSA0MGQxNjc2YzA4N2NjNzlhMDU5YjliMGE3MjA3ZjllYjlmNjZmMDc0IE1vbiBTZXAg
MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiA9P1VURi04P3E/UD1DMz1BMWRyYWlnPTIwQnJhZHk/
PSA8UEBkcmFpZ0JyYWR5LmNvbT4KRGF0ZTogVHVlLCA1IEFwciAyMDExIDExOjA0OjEzICsw
MTAwClN1YmplY3Q6IFtQQVRDSF0gY29weTogaGFuZGxlIG1lcmdlYWJsZSBleHRlbnRzIGFj
cm9zcyBmaWVtYXAgc2NhbnMKCiogZXh0ZW50LXNjYW4uaCAoZXh0ZW50X3NjYW5fZnJlZSk6
IEluaXQgdGhlIHBvaW50ZXIgdG8gTlVMTCwKYW5kIHJlc2V0IHRoZSBjb3VudCB0byAwLCBz
byB0aGF0IHdlIGNhbiByZWFsbG9jIHRoZSBidWZmZXIuCiogc3JjL2V4dGVudC1zY2FuLmMg
KGV4dGVudF9zY2FuX2luaXQpOiBMaWtld2lzZS4KKGV4dGVudF9zY2FuX3JlYWQpOiBMb29w
IG92ZXIgbXVsdGlwbGUgZmllbWFwIHNjYW5zLCBzbyB3ZSBoYW5kbGUKbWVyZ2VhYmxlIGV4
dGVudHMgdGhhdCBzcGFuIGFjcm9zcyBmaWVtYXAgc2NhbiBib3VuZGFyaWVzLiAgT25jZQp3
ZSBoYXZlIGVub3VnaCB1bmlxdWUgZXh0ZW50cywgcmV0dXJuIHNvIGFzIHRvIG1pbmltaXpl
IG1lbW9yeSB1c2UuCi0tLQogc3JjL2V4dGVudC1zY2FuLmMgfCAgMTk0ICsrKysrKysrKysr
KysrKysrKysrKysrKysrKysrLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCiBzcmMvZXh0ZW50
LXNjYW4uaCB8ICAgIDIgKwogMiBmaWxlcyBjaGFuZ2VkLCAxMDggaW5zZXJ0aW9ucygrKSwg
ODggZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvc3JjL2V4dGVudC1zY2FuLmMgYi9zcmMv
ZXh0ZW50LXNjYW4uYwppbmRleCBjMGE1ZGU2Li5lNTY5MjdiIDEwMDY0NAotLS0gYS9zcmMv
ZXh0ZW50LXNjYW4uYworKysgYi9zcmMvZXh0ZW50LXNjYW4uYwpAQCAtNjcsNiArNjcsNyBA
QCBleHRlbnRfc2Nhbl9pbml0IChpbnQgc3JjX2ZkLCBzdHJ1Y3QgZXh0ZW50X3NjYW4gKnNj
YW4pCiB7CiAgIHNjYW4tPmZkID0gc3JjX2ZkOwogICBzY2FuLT5laV9jb3VudCA9IDA7Cisg
IHNjYW4tPmV4dF9pbmZvID0gTlVMTDsKICAgc2Nhbi0+c2Nhbl9zdGFydCA9IDA7CiAgIHNj
YW4tPmluaXRpYWxfc2Nhbl9mYWlsZWQgPSBmYWxzZTsKICAgc2Nhbi0+aGl0X2ZpbmFsX2V4
dGVudCA9IGZhbHNlOwpAQCAtODIsMTA4ICs4MywxMjUgQEAgZXh0ZW50X3NjYW5faW5pdCAo
aW50IHNyY19mZCwgc3RydWN0IGV4dGVudF9zY2FuICpzY2FuKQogZXh0ZXJuIGJvb2wKIGV4
dGVudF9zY2FuX3JlYWQgKHN0cnVjdCBleHRlbnRfc2NhbiAqc2NhbikKIHsKLSAgdW5pb24g
eyBzdHJ1Y3QgZmllbWFwIGY7IGNoYXIgY1s0MDk2XTsgfSBmaWVtYXBfYnVmOwotICBzdHJ1
Y3QgZmllbWFwICpmaWVtYXAgPSAmZmllbWFwX2J1Zi5mOwotICBzdHJ1Y3QgZmllbWFwX2V4
dGVudCAqZm1fZXh0ZW50cyA9ICZmaWVtYXAtPmZtX2V4dGVudHNbMF07Ci0gIGVudW0geyBj
b3VudCA9IChzaXplb2YgZmllbWFwX2J1ZiAtIHNpemVvZiAqZmllbWFwKSAvIHNpemVvZiAq
Zm1fZXh0ZW50cyB9OwotICB2ZXJpZnkgKGNvdW50ICE9IDApOwotCi0gIC8qIFRoaXMgaXMg
cmVxdWlyZWQgYXQgbGVhc3QgdG8gaW5pdGlhbGl6ZSBmaWVtYXAtPmZtX3N0YXJ0LAotICAg
ICBidXQgYWxzbyBzZXJ2ZXMgKGluIG1pZCAyMDEwKSB0byBhcHBlYXNlIHZhbGdyaW5kLCB3
aGljaAotICAgICBhcHBlYXJzIG5vdCB0byBrbm93IHRoZSBzZW1hbnRpY3Mgb2YgdGhlIEZJ
RU1BUCBpb2N0bC4gKi8KLSAgbWVtc2V0ICgmZmllbWFwX2J1ZiwgMCwgc2l6ZW9mIGZpZW1h
cF9idWYpOwotCi0gIGZpZW1hcC0+Zm1fc3RhcnQgPSBzY2FuLT5zY2FuX3N0YXJ0OwotICBm
aWVtYXAtPmZtX2ZsYWdzID0gc2Nhbi0+Zm1fZmxhZ3M7Ci0gIGZpZW1hcC0+Zm1fZXh0ZW50
X2NvdW50ID0gY291bnQ7Ci0gIGZpZW1hcC0+Zm1fbGVuZ3RoID0gRklFTUFQX01BWF9PRkZT
RVQgLSBzY2FuLT5zY2FuX3N0YXJ0OwotCi0gIC8qIEZhbGwgYmFjayB0byB0aGUgc3RhbmRh
cmQgY29weSBpZiBjYWxsIGlvY3RsKDIpIGZhaWxlZCBmb3IgdGhlCi0gICAgIHRoZSBmaXJz
dCB0aW1lLiAgKi8KLSAgaWYgKGlvY3RsIChzY2FuLT5mZCwgRlNfSU9DX0ZJRU1BUCwgZmll
bWFwKSA8IDApCi0gICAgewotICAgICAgaWYgKHNjYW4tPnNjYW5fc3RhcnQgPT0gMCkKLSAg
ICAgICAgc2Nhbi0+aW5pdGlhbF9zY2FuX2ZhaWxlZCA9IHRydWU7Ci0gICAgICByZXR1cm4g
ZmFsc2U7Ci0gICAgfQotCi0gIC8qIElmIDAgZXh0ZW50cyBhcmUgcmV0dXJuZWQsIHRoZW4g
bW9yZSBnZXRfZXh0ZW50X3RhYmxlKCkgYXJlIG5vdCBuZWVkZWQuICAqLwotICBpZiAoZmll
bWFwLT5mbV9tYXBwZWRfZXh0ZW50cyA9PSAwKQotICAgIHsKLSAgICAgIHNjYW4tPmhpdF9m
aW5hbF9leHRlbnQgPSB0cnVlOwotICAgICAgcmV0dXJuIGZhbHNlOwotICAgIH0KLQotICBz
Y2FuLT5laV9jb3VudCA9IGZpZW1hcC0+Zm1fbWFwcGVkX2V4dGVudHM7Ci0gIHNjYW4tPmV4
dF9pbmZvID0geG5tYWxsb2MgKHNjYW4tPmVpX2NvdW50LCBzaXplb2YgKHN0cnVjdCBleHRl
bnRfaW5mbykpOwotCi0gIHVuc2lnbmVkIGludCBpLCBzaSA9IDA7CisgIHVuc2lnbmVkIGlu
dCBzaSA9IDA7CiAgIHN0cnVjdCBleHRlbnRfaW5mbyAqbGFzdF9laSBJRl9MSU5UICggPSBz
Y2FuLT5leHRfaW5mbyk7CiAKLSAgZm9yIChpID0gMDsgaSA8IHNjYW4tPmVpX2NvdW50OyBp
KyspCisgIHdoaWxlICh0cnVlKQogICAgIHsKLSAgICAgIGFzc2VydCAoZm1fZXh0ZW50c1tp
XS5mZV9sb2dpY2FsIDw9IE9GRl9UX01BWCAtIGZtX2V4dGVudHNbaV0uZmVfbGVuZ3RoKTsK
KyAgICAgIHVuaW9uIHsgc3RydWN0IGZpZW1hcCBmOyBjaGFyIGNbNDA5Nl07IH0gZmllbWFw
X2J1ZjsKKyAgICAgIHN0cnVjdCBmaWVtYXAgKmZpZW1hcCA9ICZmaWVtYXBfYnVmLmY7Cisg
ICAgICBzdHJ1Y3QgZmllbWFwX2V4dGVudCAqZm1fZXh0ZW50cyA9ICZmaWVtYXAtPmZtX2V4
dGVudHNbMF07CisgICAgICBlbnVtIHsgY291bnQgPSAoc2l6ZW9mIGZpZW1hcF9idWYgLSBz
aXplb2YgKmZpZW1hcCkvc2l6ZW9mICpmbV9leHRlbnRzIH07CisgICAgICB2ZXJpZnkgKGNv
dW50ID4gMSk7CisKKyAgICAgIC8qIFRoaXMgaXMgcmVxdWlyZWQgYXQgbGVhc3QgdG8gaW5p
dGlhbGl6ZSBmaWVtYXAtPmZtX3N0YXJ0LAorICAgICAgICAgYnV0IGFsc28gc2VydmVzIChp
biBtaWQgMjAxMCkgdG8gYXBwZWFzZSB2YWxncmluZCwgd2hpY2gKKyAgICAgICAgIGFwcGVh
cnMgbm90IHRvIGtub3cgdGhlIHNlbWFudGljcyBvZiB0aGUgRklFTUFQIGlvY3RsLiAqLwor
ICAgICAgbWVtc2V0ICgmZmllbWFwX2J1ZiwgMCwgc2l6ZW9mIGZpZW1hcF9idWYpOworCisg
ICAgICBmaWVtYXAtPmZtX3N0YXJ0ID0gc2Nhbi0+c2Nhbl9zdGFydDsKKyAgICAgIGZpZW1h
cC0+Zm1fZmxhZ3MgPSBzY2FuLT5mbV9mbGFnczsKKyAgICAgIGZpZW1hcC0+Zm1fZXh0ZW50
X2NvdW50ID0gY291bnQ7CisgICAgICBmaWVtYXAtPmZtX2xlbmd0aCA9IEZJRU1BUF9NQVhf
T0ZGU0VUIC0gc2Nhbi0+c2Nhbl9zdGFydDsKKworICAgICAgLyogRmFsbCBiYWNrIHRvIHRo
ZSBzdGFuZGFyZCBjb3B5IGlmIGNhbGwgaW9jdGwoMikgZmFpbGVkIGZvciB0aGUKKyAgICAg
ICAgIHRoZSBmaXJzdCB0aW1lLiAgKi8KKyAgICAgIGlmIChpb2N0bCAoc2Nhbi0+ZmQsIEZT
X0lPQ19GSUVNQVAsIGZpZW1hcCkgPCAwKQorICAgICAgICB7CisgICAgICAgICAgaWYgKHNj
YW4tPnNjYW5fc3RhcnQgPT0gMCkKKyAgICAgICAgICAgIHNjYW4tPmluaXRpYWxfc2Nhbl9m
YWlsZWQgPSB0cnVlOworICAgICAgICAgIHJldHVybiBmYWxzZTsKKyAgICAgICAgfQogCi0g
ICAgICBpZiAoc2kgJiYgbGFzdF9laS0+ZXh0X2ZsYWdzID09Ci0gICAgICAgICAgKGZtX2V4
dGVudHNbaV0uZmVfZmxhZ3MgJiB+RklFTUFQX0VYVEVOVF9MQVNUKQotICAgICAgICAgICYm
IChsYXN0X2VpLT5leHRfbG9naWNhbCArIGxhc3RfZWktPmV4dF9sZW5ndGgKLSAgICAgICAg
ICAgICAgPT0gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsKSkKKyAgICAgIC8qIElmIDAgZXh0
ZW50cyBhcmUgcmV0dXJuZWQsIHRoZW4gbm8gbW9yZSBzY2FucyBhcmUgbmVlZGVkLiAgKi8K
KyAgICAgIGlmIChmaWVtYXAtPmZtX21hcHBlZF9leHRlbnRzID09IDApCiAgICAgICAgIHsK
LSAgICAgICAgICAvKiBNZXJnZSBwcmV2aW91cyB3aXRoIGxhc3QuICAqLwotICAgICAgICAg
IGxhc3RfZWktPmV4dF9sZW5ndGggKz0gZm1fZXh0ZW50c1tpXS5mZV9sZW5ndGg7Ci0gICAg
ICAgICAgLyogQ29weSBmbGFncyBpbiBjYXNlIGRpZmZlcmVudC4gICovCi0gICAgICAgICAg
bGFzdF9laS0+ZXh0X2ZsYWdzID0gZm1fZXh0ZW50c1tpXS5mZV9mbGFnczsKKyAgICAgICAg
ICBzY2FuLT5oaXRfZmluYWxfZXh0ZW50ID0gdHJ1ZTsKKyAgICAgICAgICByZXR1cm4gc2Nh
bi0+c2Nhbl9zdGFydCAhPSAwOwogICAgICAgICB9Ci0gICAgICBlbHNlIGlmICgoc2kgPT0g
MCAmJiBzY2FuLT5zY2FuX3N0YXJ0ID4gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsKQotICAg
ICAgICAgICAgICAgfHwgKHNpICYmIGxhc3RfZWktPmV4dF9sb2dpY2FsICsgbGFzdF9laS0+
ZXh0X2xlbmd0aCA+Ci0gICAgICAgICAgICAgICAgICAgZm1fZXh0ZW50c1tpXS5mZV9sb2dp
Y2FsKSkKKworICAgICAgYXNzZXJ0IChzY2FuLT5laV9jb3VudCA8PSBTSVpFX01BWCAtIGZp
ZW1hcC0+Zm1fbWFwcGVkX2V4dGVudHMpOworICAgICAgc2Nhbi0+ZWlfY291bnQgKz0gZmll
bWFwLT5mbV9tYXBwZWRfZXh0ZW50czsKKyAgICAgIHNjYW4tPmV4dF9pbmZvID0geG5yZWFs
bG9jIChzY2FuLT5leHRfaW5mbywgc2Nhbi0+ZWlfY291bnQsCisgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgc2l6ZW9mIChzdHJ1Y3QgZXh0ZW50X2luZm8pKTsKKworICAg
ICAgdW5zaWduZWQgaW50IGkgPSAwOworICAgICAgZm9yIChpID0gMDsgaSA8IGZpZW1hcC0+
Zm1fbWFwcGVkX2V4dGVudHM7IGkrKykKICAgICAgICAgewotICAgICAgICAgIC8qIEJUUkZT
IGJlZm9yZSAyLjYuMzggY291bGQgcmV0dXJuIG92ZXJsYXBwaW5nIGV4dGVudHMKLSAgICAg
ICAgICAgICBmb3Igc3BhcnNlIGZpbGVzLiAgV2UgYWRqdXN0IHRoZSByZXR1cm5lZCBleHRl
bnRzCi0gICAgICAgICAgICAgcmF0aGVyIHRoYW4gZmFpbGluZywgYXMgb3RoZXJ3aXNlIGl0
IHdvdWxkIGJlIGluZWZmaWNpZW50Ci0gICAgICAgICAgICAgdG8gZGV0ZWN0IHRoaXMgb24g
dGhlIGluaXRpYWwgc2Nhbi4gICovCi0gICAgICAgICAgdWludDY0X3QgbmV3X2xvZ2ljYWw7
Ci0gICAgICAgICAgdWludDY0X3QgbGVuZ3RoX2FkanVzdDsKLSAgICAgICAgICBpZiAoc2kg
PT0gMCkKLSAgICAgICAgICAgIG5ld19sb2dpY2FsID0gc2Nhbi0+c2Nhbl9zdGFydDsKLSAg
ICAgICAgICBlbHNlCisgICAgICAgICAgYXNzZXJ0IChmbV9leHRlbnRzW2ldLmZlX2xvZ2lj
YWwgPD0KKyAgICAgICAgICAgICAgICAgIE9GRl9UX01BWCAtIGZtX2V4dGVudHNbaV0uZmVf
bGVuZ3RoKTsKKworICAgICAgICAgIGlmIChzaSAmJiBsYXN0X2VpLT5leHRfZmxhZ3MgPT0K
KyAgICAgICAgICAgICAgKGZtX2V4dGVudHNbaV0uZmVfZmxhZ3MgJiB+RklFTUFQX0VYVEVO
VF9MQVNUKQorICAgICAgICAgICAgICAmJiAobGFzdF9laS0+ZXh0X2xvZ2ljYWwgKyBsYXN0
X2VpLT5leHRfbGVuZ3RoCisgICAgICAgICAgICAgICAgICA9PSBmbV9leHRlbnRzW2ldLmZl
X2xvZ2ljYWwpKQogICAgICAgICAgICAgewotICAgICAgICAgICAgICAvKiBXZSBjb3VsZCBy
ZXR1cm4gaGVyZSBpZiBzY2FuLT5zY2FuX3N0YXJ0ID09IDAKLSAgICAgICAgICAgICAgICAg
YnV0IGRvbid0IHNvIGFzIHRvIG1pbmltaXplIHNwZWNpYWwgY2FzZXMuICAqLwotICAgICAg
ICAgICAgICBuZXdfbG9naWNhbCA9IGxhc3RfZWktPmV4dF9sb2dpY2FsICsgbGFzdF9laS0+
ZXh0X2xlbmd0aDsKKyAgICAgICAgICAgICAgLyogTWVyZ2UgcHJldmlvdXMgd2l0aCBsYXN0
LiAgKi8KKyAgICAgICAgICAgICAgbGFzdF9laS0+ZXh0X2xlbmd0aCArPSBmbV9leHRlbnRz
W2ldLmZlX2xlbmd0aDsKKyAgICAgICAgICAgICAgLyogQ29weSBmbGFncyBpbiBjYXNlIGRp
ZmZlcmVudC4gICovCisgICAgICAgICAgICAgIGxhc3RfZWktPmV4dF9mbGFncyA9IGZtX2V4
dGVudHNbaV0uZmVfZmxhZ3M7CiAgICAgICAgICAgICB9Ci0gICAgICAgICAgbGVuZ3RoX2Fk
anVzdCA9IG5ld19sb2dpY2FsIC0gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsOwotICAgICAg
ICAgIC8qIElmIGFuIGV4dGVudCBpcyBjb250YWluZWQgd2l0aGluIHRoZSBwcmV2aW91cyBv
bmUsIGp1c3QgZmFpbC4gICovCi0gICAgICAgICAgaWYgKGxlbmd0aF9hZGp1c3QgPCBmbV9l
eHRlbnRzW2ldLmZlX2xlbmd0aCkKKyAgICAgICAgICBlbHNlIGlmICgoc2kgPT0gMCAmJiBz
Y2FuLT5zY2FuX3N0YXJ0ID4gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsKQorICAgICAgICAg
ICAgICAgICAgIHx8IChzaSAmJiBsYXN0X2VpLT5leHRfbG9naWNhbCArIGxhc3RfZWktPmV4
dF9sZW5ndGggPgorICAgICAgICAgICAgICAgICAgICAgICBmbV9leHRlbnRzW2ldLmZlX2xv
Z2ljYWwpKQogICAgICAgICAgICAgewotICAgICAgICAgICAgICBpZiAoc2Nhbi0+c2Nhbl9z
dGFydCA9PSAwKQotICAgICAgICAgICAgICAgIHNjYW4tPmluaXRpYWxfc2Nhbl9mYWlsZWQg
PSB0cnVlOwotICAgICAgICAgICAgICByZXR1cm4gZmFsc2U7CisgICAgICAgICAgICAgIC8q
IEJUUkZTIGJlZm9yZSAyLjYuMzggY291bGQgcmV0dXJuIG92ZXJsYXBwaW5nIGV4dGVudHMK
KyAgICAgICAgICAgICAgICAgZm9yIHNwYXJzZSBmaWxlcy4gIFdlIGFkanVzdCB0aGUgcmV0
dXJuZWQgZXh0ZW50cworICAgICAgICAgICAgICAgICByYXRoZXIgdGhhbiBmYWlsaW5nLCBh
cyBvdGhlcndpc2UgaXQgd291bGQgYmUgaW5lZmZpY2llbnQKKyAgICAgICAgICAgICAgICAg
dG8gZGV0ZWN0IHRoaXMgb24gdGhlIGluaXRpYWwgc2Nhbi4gICovCisgICAgICAgICAgICAg
IHVpbnQ2NF90IG5ld19sb2dpY2FsOworICAgICAgICAgICAgICB1aW50NjRfdCBsZW5ndGhf
YWRqdXN0OworICAgICAgICAgICAgICBpZiAoc2kgPT0gMCkKKyAgICAgICAgICAgICAgICBu
ZXdfbG9naWNhbCA9IHNjYW4tPnNjYW5fc3RhcnQ7CisgICAgICAgICAgICAgIGVsc2UKKyAg
ICAgICAgICAgICAgICB7CisgICAgICAgICAgICAgICAgICAvKiBXZSBjb3VsZCByZXR1cm4g
aGVyZSBpZiBzY2FuLT5zY2FuX3N0YXJ0ID09IDAKKyAgICAgICAgICAgICAgICAgICAgIGJ1
dCBkb24ndCBzbyBhcyB0byBtaW5pbWl6ZSBzcGVjaWFsIGNhc2VzLiAgKi8KKyAgICAgICAg
ICAgICAgICAgIG5ld19sb2dpY2FsID0gbGFzdF9laS0+ZXh0X2xvZ2ljYWwgKyBsYXN0X2Vp
LT5leHRfbGVuZ3RoOworICAgICAgICAgICAgICAgIH0KKyAgICAgICAgICAgICAgbGVuZ3Ro
X2FkanVzdCA9IG5ld19sb2dpY2FsIC0gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsOworICAg
ICAgICAgICAgICAvKiBJZiBhbiBleHRlbnQgaXMgY29udGFpbmVkIHdpdGhpbiB0aGUgcHJl
dmlvdXMgb25lLCBmYWlsLiAgKi8KKyAgICAgICAgICAgICAgaWYgKGxlbmd0aF9hZGp1c3Qg
PCBmbV9leHRlbnRzW2ldLmZlX2xlbmd0aCkKKyAgICAgICAgICAgICAgICB7CisgICAgICAg
ICAgICAgICAgICBpZiAoc2Nhbi0+c2Nhbl9zdGFydCA9PSAwKQorICAgICAgICAgICAgICAg
ICAgICBzY2FuLT5pbml0aWFsX3NjYW5fZmFpbGVkID0gdHJ1ZTsKKyAgICAgICAgICAgICAg
ICAgIHJldHVybiBmYWxzZTsKKyAgICAgICAgICAgICAgICB9CisgICAgICAgICAgICAgIGZt
X2V4dGVudHNbaV0uZmVfbG9naWNhbCA9IG5ld19sb2dpY2FsOworICAgICAgICAgICAgICBm
bV9leHRlbnRzW2ldLmZlX2xlbmd0aCAtPSBsZW5ndGhfYWRqdXN0OworICAgICAgICAgICAg
ICAvKiBQcm9jZXNzIHRoZSBhZGp1c3RlZCBleHRlbnQgYWdhaW4uICAqLworICAgICAgICAg
ICAgICBpLS07CisgICAgICAgICAgICAgIGNvbnRpbnVlOworICAgICAgICAgICAgfQorICAg
ICAgICAgIGVsc2UKKyAgICAgICAgICAgIHsKKyAgICAgICAgICAgICAgbGFzdF9laSA9IHNj
YW4tPmV4dF9pbmZvICsgc2k7CisgICAgICAgICAgICAgIGxhc3RfZWktPmV4dF9sb2dpY2Fs
ID0gZm1fZXh0ZW50c1tpXS5mZV9sb2dpY2FsOworICAgICAgICAgICAgICBsYXN0X2VpLT5l
eHRfbGVuZ3RoID0gZm1fZXh0ZW50c1tpXS5mZV9sZW5ndGg7CisgICAgICAgICAgICAgIGxh
c3RfZWktPmV4dF9mbGFncyA9IGZtX2V4dGVudHNbaV0uZmVfZmxhZ3M7CisgICAgICAgICAg
ICAgIHNpKys7CiAgICAgICAgICAgICB9Ci0gICAgICAgICAgZm1fZXh0ZW50c1tpXS5mZV9s
b2dpY2FsID0gbmV3X2xvZ2ljYWw7Ci0gICAgICAgICAgZm1fZXh0ZW50c1tpXS5mZV9sZW5n
dGggLT0gbGVuZ3RoX2FkanVzdDsKLSAgICAgICAgICAvKiBQcm9jZXNzIHRoZSBhZGp1c3Rl
ZCBleHRlbnQgYWdhaW4uICAqLwotICAgICAgICAgIGktLTsKLSAgICAgICAgICBjb250aW51
ZTsKLSAgICAgICAgfQotICAgICAgZWxzZQotICAgICAgICB7Ci0gICAgICAgICAgbGFzdF9l
aSA9IHNjYW4tPmV4dF9pbmZvICsgc2k7Ci0gICAgICAgICAgbGFzdF9laS0+ZXh0X2xvZ2lj
YWwgPSBmbV9leHRlbnRzW2ldLmZlX2xvZ2ljYWw7Ci0gICAgICAgICAgbGFzdF9laS0+ZXh0
X2xlbmd0aCA9IGZtX2V4dGVudHNbaV0uZmVfbGVuZ3RoOwotICAgICAgICAgIGxhc3RfZWkt
PmV4dF9mbGFncyA9IGZtX2V4dGVudHNbaV0uZmVfZmxhZ3M7Ci0gICAgICAgICAgc2krKzsK
ICAgICAgICAgfQotICAgIH0KIAotICAvKiBXZSBkb24ndCBib3RoZXIgcmVhbGxvY2F0aW5n
LiAgV2Ugc2hvdWxkIHRob3VnaCBpZiB3ZSBjaGFuZ2UKLSAgICAgdG8gbG9vcGluZyB0aHJv
dWdoIGFsbCBleHRlbnRzLCB3aXRoaW4gdGhpcyBmdW5jdGlvbi4gICovCi0gIHNjYW4tPmVp
X2NvdW50ID0gc2k7CisgICAgICBpZiAobGFzdF9laS0+ZXh0X2ZsYWdzICYgRklFTUFQX0VY
VEVOVF9MQVNUKQorICAgICAgICBzY2FuLT5oaXRfZmluYWxfZXh0ZW50ID0gdHJ1ZTsKKwor
ICAgICAgLyogSWYgd2UgaGF2ZSBlbm91Z2ggZXh0ZW50cywgZGlzY2FyZCB0aGUgbGFzdCBh
cyBpdCBtaWdodAorICAgICAgICAgYmUgbWVyZ2VkIHdpdGggb25lIGZyb20gdGhlIG5leHQg
c2Nhbi4gICovCisgICAgICBpZiAoc2kgPiBjb3VudCAmJiAhc2Nhbi0+aGl0X2ZpbmFsX2V4
dGVudCkKKyAgICAgICAgbGFzdF9laSA9IHNjYW4tPmV4dF9pbmZvICsgLS1zaSAtIDE7CisK
KyAgICAgIC8qIFdlIGRvbid0IGJvdGhlciByZWFsbG9jYXRpbmcgYW55IHRyYWlsaW5nIHNs
b3RzLiAgKi8KKyAgICAgIHNjYW4tPmVpX2NvdW50ID0gc2k7CiAKLSAgaWYgKGxhc3RfZWkt
PmV4dF9mbGFncyAmIEZJRU1BUF9FWFRFTlRfTEFTVCkKLSAgICBzY2FuLT5oaXRfZmluYWxf
ZXh0ZW50ID0gdHJ1ZTsKLSAgZWxzZQotICAgIHNjYW4tPnNjYW5fc3RhcnQgPSBsYXN0X2Vp
LT5leHRfbG9naWNhbCArIGxhc3RfZWktPmV4dF9sZW5ndGg7CisgICAgICBpZiAoc2Nhbi0+
aGl0X2ZpbmFsX2V4dGVudCkKKyAgICAgICAgYnJlYWs7CisgICAgICBlbHNlCisgICAgICAg
IHNjYW4tPnNjYW5fc3RhcnQgPSBsYXN0X2VpLT5leHRfbG9naWNhbCArIGxhc3RfZWktPmV4
dF9sZW5ndGg7CisKKyAgICAgIGlmIChzaSA+PSBjb3VudCkKKyAgICAgICAgYnJlYWs7Cisg
ICAgfQogCiAgIHJldHVybiB0cnVlOwogfQpkaWZmIC0tZ2l0IGEvc3JjL2V4dGVudC1zY2Fu
LmggYi9zcmMvZXh0ZW50LXNjYW4uaAppbmRleCA4NzI4NTE1Li41YjRkZWQ1IDEwMDY0NAot
LS0gYS9zcmMvZXh0ZW50LXNjYW4uaAorKysgYi9zcmMvZXh0ZW50LXNjYW4uaApAQCAtNjYs
NiArNjYsOCBAQCBzdGF0aWMgaW5saW5lIHZvaWQKIGV4dGVudF9zY2FuX2ZyZWUgKHN0cnVj
dCBleHRlbnRfc2NhbiAqc2NhbikKIHsKICAgZnJlZSAoc2Nhbi0+ZXh0X2luZm8pOworICBz
Y2FuLT5leHRfaW5mbyA9IE5VTEw7CisgIHNjYW4tPmVpX2NvdW50ID0gMDsKIH0KIAogI2Vu
ZGlmIC8qIEVYVEVOVF9TQ0FOX0ggKi8KLS0gCjEuNy40Cgo=
--------------020400070404040807090600--


From MAILER-DAEMON Wed Apr 06 17:26:11 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q7aEt-0004Uc-6i
	for mharc-coreutils@gnu.org; Wed, 06 Apr 2011 17:26:11 -0400
Received: from [140.186.70.92] (port=59388 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q7aEq-0004UV-OL
	for coreutils@gnu.org; Wed, 06 Apr 2011 17:26:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1Q7aEp-0001Az-Nh
	for coreutils@gnu.org; Wed, 06 Apr 2011 17:26:08 -0400
Received: from mail-vx0-f169.google.com ([209.85.220.169]:52897)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1Q7aEp-0001Au-KB
	for coreutils@gnu.org; Wed, 06 Apr 2011 17:26:07 -0400
Received: by vxk20 with SMTP id 20so1922947vxk.0
	for <coreutils@gnu.org>; Wed, 06 Apr 2011 14:26:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:message-id:date:from:user-agent:mime-version:to
	:subject:content-type:content-transfer-encoding;
	bh=g2CQxasC+TzP2R2suCmJURAg5wN2dkO99YBpNNOorZA=;
	b=vV+vRyC16W86ySh3eAjY5JNfp2Wno/a/9Aco2R/vCI5cAQKbUYVgfgu6pkrlAFh/O7
	BMtk+KfC5+UW/8q/jbht/NEW3aSOzgpvaFxljXnO/6EdJeDwkSb92gDdTAM/1kRR7voo
	ES6oi2tpZWN66yQkv33eH3uWDQboS8INDcXic=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=message-id:date:from:user-agent:mime-version:to:subject
	:content-type:content-transfer-encoding;
	b=O1vYb5CtVmFDlZt1hCwAMlJz3X2gyDYh33J4clieXWan41XET+crzjWwU4yeJcbTDp
	2Sv0HYxbczPgS5IDYc15ieDIEyQp7enKqM6prsn352ujVJUKQK1c0N5p9h/f0GHSgIKQ
	ppp6kOGhS6qjfHKWCudaQ1h41IQt+Mpm3hbm4=
Received: by 10.52.74.40 with SMTP id q8mr151248vdv.113.1302125166608;
	Wed, 06 Apr 2011 14:26:06 -0700 (PDT)
Received: from [143.48.11.9] (tango.cshl.edu [143.48.11.9])
	by mx.google.com with ESMTPS id dc3sm253333vdc.47.2011.04.06.14.26.05
	(version=TLSv1/SSLv3 cipher=OTHER);
	Wed, 06 Apr 2011 14:26:06 -0700 (PDT)
Message-ID: <4D9CDA6D.5010007@gmail.com>
Date: Wed, 06 Apr 2011 17:26:05 -0400
From: Assaf Gordon <assafgordon@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.9) Gecko/20100918 Icedove/3.1.4
MIME-Version: 1.0
To: coreutils@gnu.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.220.169
Subject: sort parameters question: -V and -f
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2011 21:26:10 -0000

Hello,

I'm wondering if this is a bug (where "-f" is ignored when using version sort):

=========
$ sort --debug -f -k2,2V 
sort: using simple byte comparison
sort: leading blanks are significant in key 1; consider also specifying `b'
sort: option `-f' is ignored
==========

My assumption is that using "-f" as stand alone parameter should have the same effect as using it in a specific key (for that key). e.g. the following two commands are equivalent:
 sort -f -k1,1
 sort -k1f,1

But the following two commands are not equivalent (because the standalone "-f" is ignored):
 sort -f -k1V,1
 sort -k1Vf,1

Example:
=================

## This works
$ printf "a\nB\nc\n" | sort -k1f,1
a
B
c
$ printf "a\nB\nc\n" | sort -f -k1,1
a
B
c

## This doesn't work
$ printf "a13\nA5\na1\n" | sort -k1Vf,1
a1
A5
a13

$ printf "a13\nA5\na1\n" | sort -f -k1V,1
A5
a1
a13
===================

I'm using coreutils 8.10.

-gordon


From MAILER-DAEMON Wed Apr 06 18:06:01 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q7arR-0000pG-1r
	for mharc-coreutils@gnu.org; Wed, 06 Apr 2011 18:06:01 -0400
Received: from [140.186.70.92] (port=42571 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q7arO-0000nK-H9
	for coreutils@gnu.org; Wed, 06 Apr 2011 18:05:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q7arL-0004Lz-Sl
	for coreutils@gnu.org; Wed, 06 Apr 2011 18:05:57 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:11648)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q7arL-0004Le-JB
	for coreutils@gnu.org; Wed, 06 Apr 2011 18:05:55 -0400
Received: (qmail 79785 invoked from network); 6 Apr 2011 22:05:53 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 6 Apr 2011 22:05:53 -0000
Message-ID: <4D9CE384.40307@draigBrady.com>
Date: Wed, 06 Apr 2011 23:04:52 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Assaf Gordon <assafgordon@gmail.com>
References: <4D9CDA6D.5010007@gmail.com>
In-Reply-To: <4D9CDA6D.5010007@gmail.com>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
Subject: Re: sort parameters question: -V and -f
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2011 22:05:59 -0000

On 06/04/11 22:26, Assaf Gordon wrote:
> Hello,
> 
> I'm wondering if this is a bug (where "-f" is ignored when using version sort):
> 
> =========
> $ sort --debug -f -k2,2V 
> sort: using simple byte comparison
> sort: leading blanks are significant in key 1; consider also specifying `b'
> sort: option `-f' is ignored

The same happens for any ordering option.
If any is specified for the key, then all global options are ignored.
This is specified by POSIX and here it's demonstrated on solaris:

solaris:~ > printf "a13\nA5\na1\n" | sort -f -k1r,1
a13
a1
A5
solaris:~ > printf "a13\nA5\na1\n" | sort -k1r,1
a13
a1
A5

cheers,
Pádraig.


From MAILER-DAEMON Wed Apr 06 18:36:38 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q7bL4-0001TY-Ew
	for mharc-coreutils@gnu.org; Wed, 06 Apr 2011 18:36:38 -0400
Received: from [140.186.70.92] (port=41379 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q7bL2-0001Sn-JI
	for coreutils@gnu.org; Wed, 06 Apr 2011 18:36:37 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q7bL1-0004JD-Cs
	for coreutils@gnu.org; Wed, 06 Apr 2011 18:36:36 -0400
Received: from mx1.redhat.com ([209.132.183.28]:54139)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q7bL1-0004Iw-3K
	for coreutils@gnu.org; Wed, 06 Apr 2011 18:36:35 -0400
Received: from int-mx12.intmail.prod.int.phx2.redhat.com
	(int-mx12.intmail.prod.int.phx2.redhat.com [10.5.11.25])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p36MaVVX002677
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Wed, 6 Apr 2011 18:36:31 -0400
Received: from [10.3.113.117] (ovpn-113-117.phx2.redhat.com [10.3.113.117])
	by int-mx12.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p36MaUTi022142; Wed, 6 Apr 2011 18:36:30 -0400
Message-ID: <4D9CEAEE.1010304@redhat.com>
Date: Wed, 06 Apr 2011 16:36:30 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
References: <4D9CDA6D.5010007@gmail.com> <4D9CE384.40307@draigBrady.com>
In-Reply-To: <4D9CE384.40307@draigBrady.com>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enig7017D49AC8247EB456580D69"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.25
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: Assaf Gordon <assafgordon@gmail.com>, coreutils@gnu.org
Subject: Re: sort parameters question: -V and -f
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2011 22:36:37 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig7017D49AC8247EB456580D69
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 04/06/2011 04:04 PM, P=E1draig Brady wrote:
> On 06/04/11 22:26, Assaf Gordon wrote:
>> Hello,
>>
>> I'm wondering if this is a bug (where "-f" is ignored when using versi=
on sort):
>>
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D
>> $ sort --debug -f -k2,2V=20
>> sort: using simple byte comparison
>> sort: leading blanks are significant in key 1; consider also specifyin=
g `b'
>> sort: option `-f' is ignored
>=20
> The same happens for any ordering option.
> If any is specified for the key, then all global options are ignored.
> This is specified by POSIX and here it's demonstrated on solaris:
>=20
> solaris:~ > printf "a13\nA5\na1\n" | sort -f -k1r,1
> a13
> a1
> A5
> solaris:~ > printf "a13\nA5\na1\n" | sort -k1r,1
> a13
> a1
> A5

Not only that, but --debug would have told you the same:

$ printf "a13\nA5\na1\n" | LC_ALL=3DC sort --debug -f -k1r,1
sort: using simple byte comparison
sort: option `-f' is ignored
a13
___
___
a1
__
__
A5
__
__


[I just LOVE the --debug option - why didn't we add it sooner?]

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enig7017D49AC8247EB456580D69
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNnOruAAoJEKeha0olJ0NqQ1YIAJrQHrMGD1STT5ofEmhUBRwR
qAmUCAyK3hHUJKrJETbN3bdiiFrUtUXudkpWsSN8a0jNkWHrMrbOJqtKwrOh2TiU
qX3s2ERI7wGDGiag0VRNpCf89JxkwqIeTV+yOA1IVT3SRvLqcMt65ACKI1LSoKAl
s2EeNwPNBnYu+SBsTChe/9e4Zv+f413l/K9FuKENHlbw1t6sst8mkqYCG22hdIlQ
DRLhGQ3GKjpJdiiHdJof5Bn8coS+qqQzZWH+TmPbR1l2gWQx2Rl20W9N00sZGztC
fm/GhTRdGvIUTHrW7fm34hQpSfuBmJV74tsxGLGv0pa3oHge1zBJ7mv5k9JIFhY=
=M4xY
-----END PGP SIGNATURE-----

--------------enig7017D49AC8247EB456580D69--


From MAILER-DAEMON Thu Apr 07 15:06:58 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q7uXi-0001yi-99
	for mharc-coreutils@gnu.org; Thu, 07 Apr 2011 15:06:58 -0400
Received: from [140.186.70.92] (port=45783 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q7uXf-0000mf-Qj
	for coreutils@gnu.org; Thu, 07 Apr 2011 15:06:56 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1Q7uVw-0006Vn-5s
	for coreutils@gnu.org; Thu, 07 Apr 2011 15:05:08 -0400
Received: from mail-vx0-f169.google.com ([209.85.220.169]:37951)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <assafgordon@gmail.com>) id 1Q7uVw-0006Vh-0D
	for coreutils@gnu.org; Thu, 07 Apr 2011 15:05:08 -0400
Received: by vxk20 with SMTP id 20so2852492vxk.0
	for <coreutils@gnu.org>; Thu, 07 Apr 2011 12:05:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:message-id:date:from:user-agent:mime-version:to
	:subject:references:in-reply-to:content-type
	:content-transfer-encoding;
	bh=HyTGH5qszchlHB4qKLWsa4U/hkl/olNbxev9ktvhsAY=;
	b=DU1gBUh6CCRJ/ZXMXJWgdp9yIQLcXkxHpUaZZwD8ssKPO47SzkM4rtvWcUl34wWSZa
	Eiwf58kkLDWG244LpJk3GpNeZsI2hEzx1uK4fvCIj4rOrJcqxaEY84AZX0It5lFMy4Qz
	HCoBrQLsz1dWsRQHN1mDXeMy2BXrLTpJc5mJQ=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=message-id:date:from:user-agent:mime-version:to:subject:references
	:in-reply-to:content-type:content-transfer-encoding;
	b=fNEDchURtMIogsAq5DHFzuRQf+0rMAef/I3HbKJmZE4HDrhUiC4pHepmkRZbylpgge
	bxpLeNPVKFJ2DujA/RzBYnu0M5CezCiTGlWFwP3dwBpSxW6SHGMMlrZDu6RH9zbOD/0Q
	5cCPI6ZeJMVi0kaa/F4vH1zQMbyv0Vt7hwqEA=
Received: by 10.52.97.227 with SMTP id ed3mr1732071vdb.295.1302203107506;
	Thu, 07 Apr 2011 12:05:07 -0700 (PDT)
Received: from [143.48.11.9] (tango.cshl.edu [143.48.11.9])
	by mx.google.com with ESMTPS id es1sm476725vdb.35.2011.04.07.12.05.04
	(version=TLSv1/SSLv3 cipher=OTHER);
	Thu, 07 Apr 2011 12:05:04 -0700 (PDT)
Message-ID: <4D9E0ADF.6000207@gmail.com>
Date: Thu, 07 Apr 2011 15:05:03 -0400
From: Assaf Gordon <assafgordon@gmail.com>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.9) Gecko/20100918 Icedove/3.1.4
MIME-Version: 1.0
To: coreutils@gnu.org
References: <4D9CDA6D.5010007@gmail.com> <4D9CE384.40307@draigBrady.com>
	<4D9CEAEE.1010304@redhat.com>
In-Reply-To: <4D9CEAEE.1010304@redhat.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.220.169
Subject: Re: sort parameters question: -V and -f
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 07 Apr 2011 19:06:56 -0000

Eric Blake wrote, On 04/06/2011 06:36 PM:
> On 04/06/2011 04:04 PM, P=E1draig Brady wrote:
>> On 06/04/11 22:26, Assaf Gordon wrote:
>>> I'm wondering if this is a bug (where "-f" is ignored when using vers=
ion sort):
>>
>> The same happens for any ordering option.
>> If any is specified for the key, then all global options are ignored.
>> This is specified by POSIX and here it's demonstrated on solaris:
>>
> Not only that, but --debug would have told you the same:
>=20

"--debug" did tell me that, but I thought it's a bug, not a feature.
I assumed "-f" is accumulative, not overridden when specifying per-key so=
rt order - I should have read the docs more carefully.

Thanks for the quick and detailed response.

-gordon



From MAILER-DAEMON Fri Apr 08 09:58:34 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q8CCo-0007Bo-5F
	for mharc-coreutils@gnu.org; Fri, 08 Apr 2011 09:58:34 -0400
Received: from [140.186.70.92] (port=52704 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q8BVg-0002bE-VQ
	for coreutils@gnu.org; Fri, 08 Apr 2011 09:14:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <Berengar.Lehr@gmx.de>) id 1Q8BVf-0005ed-U1
	for coreutils@gnu.org; Fri, 08 Apr 2011 09:14:00 -0400
Received: from mailout-de.gmx.net ([213.165.64.23]:53464)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <Berengar.Lehr@gmx.de>) id 1Q8BVf-0005dv-O8
	for coreutils@gnu.org; Fri, 08 Apr 2011 09:13:59 -0400
Received: (qmail invoked by alias); 08 Apr 2011 13:13:54 -0000
Received: from metpsy-pc174.metpsy.uni-jena.de (EHLO [141.35.244.174])
	[141.35.244.174]
	by mail.gmx.net (mp043) with SMTP; 08 Apr 2011 15:13:54 +0200
X-Authenticated: #2204504
X-Provags-ID: V01U2FsdGVkX1+lKkZbmX6lPl+a6dad257iOv4iW7joHAGufKv56V
	OSs6nsUqVgGoI6
Message-ID: <4D9F09F6.3050809@gmx.de>
Date: Fri, 08 Apr 2011 15:13:26 +0200
From: Berengar Lehr <Berengar.Lehr@gmx.de>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: coreutils@gnu.org
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: 7bit
X-Y-GMX-Trusted: 0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 213.165.64.23
X-Mailman-Approved-At: Fri, 08 Apr 2011 09:58:33 -0400
Subject: Request for ls
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Apr 2011 13:14:05 -0000

Dear coreutils-ppl,

is it possible to make ls show me permission/modes as octal numbers?

old:
-rw-r--r--  1 ber ber   11176 2011-04-05 17:37 Test
new:
644 1 ber ber   11176 2011-04-05 17:37 Test

Ber


From MAILER-DAEMON Fri Apr 08 10:03:36 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q8CHg-0001Na-JC
	for mharc-coreutils@gnu.org; Fri, 08 Apr 2011 10:03:36 -0400
Received: from [140.186.70.92] (port=42491 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q8CHa-0001Mw-4Y
	for coreutils@gnu.org; Fri, 08 Apr 2011 10:03:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q8CHY-0003LI-VD
	for coreutils@gnu.org; Fri, 08 Apr 2011 10:03:30 -0400
Received: from mx1.redhat.com ([209.132.183.28]:1025)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1Q8CHY-0003Kq-KI
	for coreutils@gnu.org; Fri, 08 Apr 2011 10:03:28 -0400
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p38E3ONr007400
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 8 Apr 2011 10:03:24 -0400
Received: from [10.3.113.117] (ovpn-113-117.phx2.redhat.com [10.3.113.117])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p38E3NqH004343; Fri, 8 Apr 2011 10:03:23 -0400
Message-ID: <4D9F15AA.7040401@redhat.com>
Date: Fri, 08 Apr 2011 08:03:22 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: Berengar Lehr <Berengar.Lehr@gmx.de>
References: <4D9F09F6.3050809@gmx.de>
In-Reply-To: <4D9F09F6.3050809@gmx.de>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enig9975FAE23AC0DF3D04FD1A14"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
Subject: Re: Request for ls
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Apr 2011 14:03:34 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig9975FAE23AC0DF3D04FD1A14
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: quoted-printable

On 04/08/2011 07:13 AM, Berengar Lehr wrote:
> Dear coreutils-ppl,
>=20
> is it possible to make ls show me permission/modes as octal numbers?

Thanks for the report.

Unfortunately, ls is already way too overloaded - there's very little
room for adding more short options.

But what you are asking for is already present in stat(1), which is also
friendlier for scripting (unambiguous access to exactly the information
you want, rather than having to parse fields of ls output and possibly
getting it wrong).

>=20
> old:
> -rw-r--r--  1 ber ber   11176 2011-04-05 17:37 Test
> new:
> 644 1 ber ber   11176 2011-04-05 17:37 Test

stat -c '%a' Test

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enig9975FAE23AC0DF3D04FD1A14
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNnxWqAAoJEKeha0olJ0NqqEoH/ioNJVqVsGDpVTNBMjVgXpKH
9nlwLhyYh/5jnTtkNf96kfUNm8m13u1vsa4+jJw6ADkoxXeXCZ4YTrzaM7r02HB6
pXSQeXSsat+8KrZ9OCG7iPocIsXSkgAQSKxoUbnH35DCKUUmcuDudpJo6EAXR145
yJzmWqh6Ue+RycKIEGK/SyMBOyyA+FQgoyT8/bhKugot5mi2NLhicBEfk3QEpE8L
MVYB5fn1zNStwBnndRppTaRyBwzLrnxlIZrXaGTaZwtFSCmROSeViXiGZOIDmqAI
7WDgLGh7EoBjma4dyz9DnOHtHH9m1CYAhfFTuXGmzK8ECWb5lzAb08NBGPPB1BY=
=Z2cV
-----END PGP SIGNATURE-----

--------------enig9975FAE23AC0DF3D04FD1A14--


From MAILER-DAEMON Fri Apr 08 13:26:30 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q8FS1-0005Gn-S0
	for mharc-coreutils@gnu.org; Fri, 08 Apr 2011 13:26:30 -0400
Received: from [140.186.70.92] (port=44725 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q8FRv-0005EF-1F
	for coreutils@gnu.org; Fri, 08 Apr 2011 13:26:27 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q8FRs-0002m5-9J
	for coreutils@gnu.org; Fri, 08 Apr 2011 13:26:21 -0400
Received: from mx.meyering.net ([82.230.74.64]:45082)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q8FRr-0002lG-WC
	for coreutils@gnu.org; Fri, 08 Apr 2011 13:26:20 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id BFF4A6019D; Fri,  8 Apr 2011 19:26:14 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Fri, 08 Apr 2011 19:26:14 +0200
Message-ID: <87ei5chcy1.fsf@rho.meyering.net>
Lines: 50
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: ready for release of coreutils-8.11?
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Apr 2011 17:26:27 -0000

We have a large array of fixes/changes so far.  NEWS below.
Is there anything pending that others would like to see in 8.11?


** Bug fixes

  cp -a --link would not create a hardlink to a symlink, instead
  copying the symlink and then not preserving its timestamp.
  [bug introduced in coreutils-8.0]

  cp now avoids FIEMAP issues with BTRFS before Linux 2.6.38,
  which could result in corrupt copies of sparse files.
  [bug introduced in coreutils-8.10]

  cut could segfault when invoked with a user-specified output
  delimiter and an unbounded range like "-f1234567890-".
  [bug introduced in coreutils-5.3.0]

  du would infloop when given --files0-from=DIR
  [bug introduced in coreutils-7.1]

  sort no longer spawns 7 worker threads to sort 16 lines
  [bug introduced in coreutils-8.6]

  touch built on Solaris 9 would segfault when run on Solaris 10
  [bug introduced in coreutils-8.8]

  wc would dereference a NULL pointer upon an early out-of-memory error
  [bug introduced in coreutils-7.1]

** New features

  dd now accepts the 'nocache' flag to the iflag and oflag options,
  which will discard any cache associated with the files, or
  processed portion thereof.

** Changes in behavior

  cp now avoids syncing files when possible, when doing a FIEMAP copy.
  The sync in only needed on Linux kernels before 2.6.39.
  [The sync was introduced in coreutils-8.10]

  cp now copies empty extents efficiently, when doing a FIEMAP copy.
  It no longer reads the zero bytes from the input, and also can efficiently
  create a hole in the output file when --sparse=always is specified.

  df now aligns columns consistently, and no longer wraps entries
  with longer device identifiers, over two lines.

  test now accepts "==" as a synonym for "="


From MAILER-DAEMON Fri Apr 08 19:38:21 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q8LFt-0007pF-29
	for mharc-coreutils@gnu.org; Fri, 08 Apr 2011 19:38:21 -0400
Received: from [140.186.70.92] (port=51870 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q8LFr-0007n5-Qy
	for coreutils@gnu.org; Fri, 08 Apr 2011 19:38:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q8LFp-0000rx-PW
	for coreutils@gnu.org; Fri, 08 Apr 2011 19:38:19 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:12923)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q8LFp-0000rd-GB
	for coreutils@gnu.org; Fri, 08 Apr 2011 19:38:17 -0400
Received: (qmail 50843 invoked from network); 8 Apr 2011 23:38:15 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 8 Apr 2011 23:38:15 -0000
Message-ID: <4D9F9C20.5020709@draigBrady.com>
Date: Sat, 09 Apr 2011 00:37:04 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: coreutils@gnu.org
References: <87ei5chcy1.fsf@rho.meyering.net>
In-Reply-To: <87ei5chcy1.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Subject: Re: ready for release of coreutils-8.11?
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Apr 2011 23:38:20 -0000

On 08/04/11 18:26, Jim Meyering wrote:
> We have a large array of fixes/changes so far.  NEWS below.
> Is there anything pending that others would like to see in 8.11?

One thing I'm worried about is fiemap with compressed file systems.
I'm unsure that the extent length refers to the length of
the unencoded data. That's what I would expect so that
user space is abstracted from encoding details and so
we know how much to read to fully process an extent.

I've not got a new BTRFS kernel readily available at present
to do a quick test. I'll try to text tomorrow night.

Otherwise, nothing else outstanding.

cheers,
Pádraig


From MAILER-DAEMON Sat Apr 09 06:35:29 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q8VVp-0006iX-97
	for mharc-coreutils@gnu.org; Sat, 09 Apr 2011 06:35:29 -0400
Received: from [140.186.70.92] (port=37283 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q8VVn-0006eo-4G
	for coreutils@gnu.org; Sat, 09 Apr 2011 06:35:28 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q8VVj-0007rf-5P
	for coreutils@gnu.org; Sat, 09 Apr 2011 06:35:27 -0400
Received: from mx.meyering.net ([82.230.74.64]:58184)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q8VVd-0007qi-PV
	for coreutils@gnu.org; Sat, 09 Apr 2011 06:35:23 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 5A05B601A8; Sat,  9 Apr 2011 12:34:58 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Date: Sat, 09 Apr 2011 12:34:58 +0200
Message-ID: <87vcyng1bh.fsf@rho.meyering.net>
Lines: 31
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Subject: odd test failure: misc/sort-spinlock-abuse
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 09 Apr 2011 10:35:28 -0000

The misc/sort-spinlock-abuse test,

  http://git.sv.gnu.org/cgit/coreutils.git/tree/tests/misc/sort-spinlock-abuse

fails regularly when it is run in parallel with others (make -j25 check,
ext4+SSD, 6/12-core, F15).  It fails because an output-restrained sort
(writing to a FIFO with a slow consumer) takes more than 1 second of CPU
time to process the regular file, "in", created by "seq 100000 > in".

In fact, it may take even more than 4 seconds of CPU time.
At first I thought it was a regression.  But no:
the problem arises even when running sort with --parallel=1.
The original bug involved a parallel-specific busy-wait
triggered by the blocked output.

What's going on?
I have traced it back to an fstat syscall that is consuming
lots of CPU time.

Here's strace -r output, where FD 3 refers to the regular input file, "in":

    3.263544 fstat(3, {st_mode=S_IFREG|0600, st_size=588895, ...}) = 0

When I run this test in isolation, it always completes successfully.
In that case, the fstat takes 30-40 microseconds.

    make check -C tests TESTS=misc/sort-spinlock-abuse VERBOSE=yes

But when I run it via "make -j25 check", it fails ~40% of the time.

Next step is probably to see if oprofile can shed some light.


From MAILER-DAEMON Mon Apr 11 07:09:19 2011
Received: from mailman by lists.gnu.org with archive (Exim 4.43)
	id 1Q9Ezf-0000cW-4h
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 07:09:19 -0400
Received: from [140.186.70.92] (port=54062 helo=eggs.gnu.org)
	by lists.gnu.org with esmtp (Exim 4.43) id 1Q9Ezc-0000ZU-8N
	for coreutils@gnu.org; Mon, 11 Apr 2011 07:09:17 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9Eza-0008Q1-4s
	for coreutils@gnu.org; Mon, 11 Apr 2011 07:09:15 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:9791)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9EzZ-0008Ow-Dn
	for coreutils@gnu.org; Mon, 11 Apr 2011 07:09:14 -0400
Received: (qmail 93638 invoked from network); 11 Apr 2011 11:09:10 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 11 Apr 2011 11:09:10 -0000
Message-ID: <4DA2E102.5080801@draigBrady.com>
Date: Mon, 11 Apr 2011 12:07:46 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: coreutils@gnu.org, linux-ext4@vger.kernel.org
References: <87ei5chcy1.fsf@rho.meyering.net> <4D9F9C20.5020709@draigBrady.com>
In-Reply-To: <4D9F9C20.5020709@draigBrady.com>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: 
Subject: Re: ready for release of coreutils-8.11?
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 11 Apr 2011 11:09:17 -0000

On 09/04/11 00:37, Pádraig Brady wrote:
> One thing I'm worried about is fiemap with compressed file systems.
> I'm unsure that the extent length refers to the length of
> the unencoded data. That's what I would expect so that
> user space is abstracted from encoding details and so
> we know how much to read to fully process an extent.
> 
> I've not got a new BTRFS kernel readily available at present
> to do a quick test. I'll try to text tomorrow night.

I've tried various BTRFS combinations here with 2.6.38.2-9.fc15.i686.
I'm using `mount -o compress -o compress-force`, but any files
copied into the file system, don't have FIEMAP_EXTENT_ENCODED flags,
and du doesn't indicate that less space is used in the file system.

So I'm assuming that the compression is below the file system,
and doesn't increase the space available. I.E. it's just
there to more efficiently write blocks to disk.

As for the FIEMAP_EXTENT_ENCODED flag, are there any uses currently?
Note coreutils currently assumes that the length of an extent, is
that required to read it fully, even in the presence of implicit conversion.

cheers,
Pádraig.


From MAILER-DAEMON Mon Apr 11 21:47:27 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9ShT-0000Kt-PV
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 21:47:27 -0400
Received: from eggs.gnu.org ([140.186.70.92]:50577)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShK-0006wb-PX
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:26 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LCC-0006WR-H9
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:41 -0400
Received: from mx.meyering.net ([82.230.74.64]:45823)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LCC-0006WN-5s
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:40 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 4F68A601B8; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: a few small changes
Date: Mon, 11 Apr 2011 19:45:56 +0200
Message-Id: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 01:47:26 -0000

Install still accepted --preserve_context.  patch-4/7 removes it.
While looking at that, I noticed install.c was one of the
very few files in coreutils with forward declarations for
static functions.  Normally we put "main" at the end
and order callees before callers.  I've removed those
forward declarations (unnecessary duplication) and reordered
the functions to make them unnecessary.

There have been a few fixes in gnulib, including
a new syntax-check rule that catches doubled words,
(yes, there were a few here), so I've updated to the latest:

[PATCH 1/7] maint: remove doubled words in comments, e.g., s/to
[PATCH 2/7] maint: rename variables for clarity...
[PATCH 3/7] maint: reorder install.c to eliminate declarations of
[PATCH 4/7] maint: install: remove support for --preserve_context
[PATCH 5/7] doc: tweak NEWS
[PATCH 6/7] cfg.mk: remove useless semicolon and backslash
[PATCH 7/7] build: update gnulib submodule to latest


From MAILER-DAEMON Mon Apr 11 21:47:27 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9ShT-0000LH-Si
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 21:47:27 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51449)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShK-0007Qq-TP
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:27 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBn-0006UC-4q
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:15 -0400
Received: from mx.meyering.net ([82.230.74.64]:34011)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBm-0006Tx-Uo
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:15 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id C7DEB602A9; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 5/7] doc: tweak NEWS
Date: Mon, 11 Apr 2011 19:46:01 +0200
Message-Id: <1302543963-9069-6-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 01:47:27 -0000

From: Jim Meyering <meyering@redhat.com>

* NEWS: Slightly obfuscate a line to avoid a false-positive
doubled-word ("is-is") match.
Fix a grammar error in news for 8.2.
* cfg.mk (old_NEWS_hash): Resync.
---
 NEWS   |    4 ++--
 cfg.mk |    2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/NEWS b/NEWS
index ceccaca..68a9a1b 100644
--- a/NEWS
+++ b/NEWS
@@ -420,7 +420,7 @@ GNU coreutils NEWS                                    -*- outline -*-
   if it uses helper processes for compression and its parent
   ignores CHLD signals. [bug introduced in coreutils-6.9]

-  tail without -f no longer access uninitialized memory
+  tail without -f no longer accesses uninitialized memory
   [bug introduced in coreutils-7.6]

   timeout is now immune to the signal handling of its parent.
@@ -549,7 +549,7 @@ GNU coreutils NEWS                                    -*- outline -*-

   ls -LR exits with status 2, not 0, when it encounters a cycle

-  ls -is is now consistent with ls -lis in ignoring values returned
+  "ls -is" is now consistent with ls -lis in ignoring values returned
   from a failed stat/lstat.  For example ls -Lis now prints "?", not "0",
   for the inode number and allocated size of a dereferenced dangling symlink.

diff --git a/cfg.mk b/cfg.mk
index 48e5a47..126968e 100644
--- a/cfg.mk
+++ b/cfg.mk
@@ -27,7 +27,7 @@ bootstrap-tools = autoconf,automake,gnulib,bison
 # Now that we have better tests, make this the default.
 export VERBOSE = yes

-old_NEWS_hash = 6830d50281733e19bb2baff7a891e1f8
+old_NEWS_hash = 99b06e7bb289add96b10127fceced4e9

 # Add an exemption for sc_makefile_at_at_check.
 _makefile_at_at_check_exceptions = ' && !/^cu_install_program =/'
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Mon Apr 11 21:47:29 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9ShV-0000NJ-0a
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 21:47:29 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51211)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShL-0007K7-6Z
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:27 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBm-0006Tk-2L
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:15 -0400
Received: from mx.meyering.net ([82.230.74.64]:34003)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBl-0006T1-Np
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:14 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 5D4586012A; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 1/7] maint: remove doubled words in comments, e.g., s/to to/to/
Date: Mon, 11 Apr 2011 19:45:57 +0200
Message-Id: <1302543963-9069-2-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 01:47:27 -0000

From: Jim Meyering <meyering@redhat.com>

* tests/ls/color-norm: s/to to/to/
* gl/lib/mbsalign.h (mbs_align_t): s/or or/or/
* src/extent-scan.c (extent_scan_read): s/the the/the/
* src/libstdbuf.c: s/the the/the/
* tests/misc/stdbuf: s/on on/on/
* ChangeLog-2005: s/for\n\tfor /for\n\t/
---
 ChangeLog-2005      |    2 +-
 gl/lib/mbsalign.h   |    2 +-
 src/extent-scan.c   |    2 +-
 src/libstdbuf.c     |    2 +-
 tests/ls/color-norm |    2 +-
 tests/misc/stdbuf   |    2 +-
 6 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/ChangeLog-2005 b/ChangeLog-2005
index d4b8cc2..8709804 100644
--- a/ChangeLog-2005
+++ b/ChangeLog-2005
@@ -9086,7 +9086,7 @@

 	split's --verbose option did nothing [broken in 4.5.10 and 5.0]
 	* src/split.c (longopts): Use `1', not `0' as the value for
-	for &verbose.  Reported by Keith Thompson.
+	&verbose.  Reported by Keith Thompson.

 	Test for the above fix.
 	* tests/misc/split-a: Also use --verbose and compare stderr
diff --git a/gl/lib/mbsalign.h b/gl/lib/mbsalign.h
index 76d5a3f..de5d767 100644
--- a/gl/lib/mbsalign.h
+++ b/gl/lib/mbsalign.h
@@ -19,7 +19,7 @@
 typedef enum { MBS_ALIGN_LEFT, MBS_ALIGN_RIGHT, MBS_ALIGN_CENTER } mbs_align_t;

 enum {
-  /* Use unibyte mode for invalid multibyte strings or
+  /* Use unibyte mode for invalid multibyte strings
      or when heap memory is exhausted.  */
   MBA_UNIBYTE_FALLBACK = 0x0001

diff --git a/src/extent-scan.c b/src/extent-scan.c
index 71a9404..e95b085 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -104,7 +104,7 @@ extent_scan_read (struct extent_scan *scan)
       fiemap->fm_extent_count = count;
       fiemap->fm_length = FIEMAP_MAX_OFFSET - scan->scan_start;

-      /* Fall back to the standard copy if call ioctl(2) failed for the
+      /* Fall back to the standard copy if call ioctl(2) failed for
          the first time.  */
       if (ioctl (scan->fd, FS_IOC_FIEMAP, fiemap) < 0)
         {
diff --git a/src/libstdbuf.c b/src/libstdbuf.c
index ce89ce0..9cb101e 100644
--- a/src/libstdbuf.c
+++ b/src/libstdbuf.c
@@ -22,7 +22,7 @@
 #include "system.h"
 #include "verify.h"

-/* Note currently for glibc (2.3.5) the following call does not change the
+/* Note currently for glibc (2.3.5) the following call does not change
    the buffer size, and more problematically does not give any indication
    that the new size request was ignored:

diff --git a/tests/ls/color-norm b/tests/ls/color-norm
index 6d8c14d..20bc121 100755
--- a/tests/ls/color-norm
+++ b/tests/ls/color-norm
@@ -27,7 +27,7 @@ umask 22
 # Output time as something constant
 export TIME_STYLE="+norm"

-# helper to to strip ls columns up to "norm" time
+# helper to strip ls columns up to "norm" time
 qls() { sed 's/-r.*norm/norm/'; }

 touch exe || framework_failure
diff --git a/tests/misc/stdbuf b/tests/misc/stdbuf
index 8bb5721..35b5710 100755
--- a/tests/misc/stdbuf
+++ b/tests/misc/stdbuf
@@ -104,7 +104,7 @@ retry_delay_ stdbuf_unbuffer .1 6 || fail=1

 # Ensure block buffering stdout takes effect
 # We don't currently test block buffering failures as
-# this doesn't work on on GLIBC-2.7 or GLIBC-2.9 at least.
+# this doesn't work on GLIBC-2.7 or GLIBC-2.9 at least.
    # stdbuf_blockbuffer()
    # {
    #   local delay="$1"
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Mon Apr 11 21:47:29 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9ShV-0000NY-4X
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 21:47:29 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51359)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShL-0007PF-AM
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:28 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBl-0006TQ-S9
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:14 -0400
Received: from mx.meyering.net ([82.230.74.64]:34002)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBl-0006T0-CV
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:13 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id B9F3A602FF; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 4/7] maint: install: remove support for --preserve_context
	("_", not "-")
Date: Mon, 11 Apr 2011 19:46:00 +0200
Message-Id: <1302543963-9069-5-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 01:47:28 -0000

From: Jim Meyering <meyering@redhat.com>

* src/install.c: Its use has elicited a warning for two years.
Use --preserve-context instead.
* NEWS (changes in behavior): Mention this.
---
 NEWS          |    3 +++
 src/install.c |    8 --------
 2 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/NEWS b/NEWS
index 9a9b97a..ceccaca 100644
--- a/NEWS
+++ b/NEWS
@@ -47,6 +47,9 @@ GNU coreutils NEWS                                    -*- outline -*-
   df now aligns columns consistently, and no longer wraps entries
   with longer device identifiers, over two lines.

+  install now rejects its long-deprecated --preserve_context option.
+  Use --preserve-context instead.
+
   test now accepts "==" as a synonym for "="


diff --git a/src/install.c b/src/install.c
index 369547b..dbff9c9 100644
--- a/src/install.c
+++ b/src/install.c
@@ -114,7 +114,6 @@ static char const *strip_program = "strip";
 enum
 {
   PRESERVE_CONTEXT_OPTION = CHAR_MAX + 1,
-  PRESERVE_CONTEXT_OPTION_DEPRECATED,
   STRIP_PROGRAM_OPTION
 };

@@ -130,9 +129,6 @@ static struct option const long_options[] =
   {"owner", required_argument, NULL, 'o'},
   {"preserve-timestamps", no_argument, NULL, 'p'},
   {"preserve-context", no_argument, NULL, PRESERVE_CONTEXT_OPTION},
-  /* --preserve_context was silently supported until Apr 2009.
-     FIXME: disable altogether in a year or so.  */
-  {"preserve_context", no_argument, NULL, PRESERVE_CONTEXT_OPTION_DEPRECATED},
   {"strip", no_argument, NULL, 's'},
   {"strip-program", required_argument, NULL, STRIP_PROGRAM_OPTION},
   {"suffix", required_argument, NULL, 'S'},
@@ -839,10 +835,6 @@ main (int argc, char **argv)
           no_target_directory = true;
           break;

-        case PRESERVE_CONTEXT_OPTION_DEPRECATED:
-          error (0, 0, _("WARNING: --preserve_context is deprecated; "
-                         "use --preserve-context instead"));
-          /* fall through */
         case PRESERVE_CONTEXT_OPTION:
           if ( ! selinux_enabled)
             {
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Mon Apr 11 21:47:31 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9ShX-0000RS-9T
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 21:47:31 -0400
Received: from eggs.gnu.org ([140.186.70.92]:40278)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShK-00062j-Pb
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:29 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBo-0006Ub-3u
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:18 -0400
Received: from mx.meyering.net ([82.230.74.64]:34004)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBl-0006T6-CX
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:16 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 90FCA601D4; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 3/7] maint: reorder install.c to eliminate declarations of
	static functions
Date: Mon, 11 Apr 2011 19:45:59 +0200
Message-Id: <1302543963-9069-4-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 01:47:29 -0000

From: Jim Meyering <meyering@redhat.com>

* src/install.c: Remove static function declarations.
---
 src/install.c |  957 ++++++++++++++++++++++++++++-----------------------------
 1 files changed, 470 insertions(+), 487 deletions(-)

diff --git a/src/install.c b/src/install.c
index f416996..369547b 100644
--- a/src/install.c
+++ b/src/install.c
@@ -68,23 +68,6 @@ static bool use_default_selinux_context = true;
 # define matchpathcon_init_prefix(a, p) /* empty */
 #endif

-static bool change_timestamps (struct stat const *from_sb, char const *to);
-static bool change_attributes (char const *name);
-static bool copy_file (const char *from, const char *to,
-                       const struct cp_options *x);
-static bool install_file_in_file_parents (char const *from, char *to,
-                                          struct cp_options *x);
-static bool install_file_in_dir (const char *from, const char *to_dir,
-                                 const struct cp_options *x);
-static bool install_file_in_file (const char *from, const char *to,
-                                  const struct cp_options *x);
-static void get_ids (void);
-static void strip (char const *name);
-static void announce_mkdir (char const *dir, void *options);
-static int make_ancestor (char const *dir, char const *component,
-                          void *options);
-void usage (int status);
-
 /* The user name that will own the files, or NULL to make the owner
    the current user ID. */
 static char *owner_name;
@@ -407,6 +390,27 @@ target_directory_operand (char const *file)
   return is_a_dir;
 }

+/* Report that directory DIR was made, if OPTIONS requests this.  */
+static void
+announce_mkdir (char const *dir, void *options)
+{
+  struct cp_options const *x = options;
+  if (x->verbose)
+    prog_fprintf (stdout, _("creating directory %s"), quote (dir));
+}
+
+/* Make ancestor directory DIR, whose last file name component is
+   COMPONENT, with options OPTIONS.  Assume the working directory is
+   COMPONENT's parent.  */
+static int
+make_ancestor (char const *dir, char const *component, void *options)
+{
+  int r = mkdir (component, DEFAULT_MODE);
+  if (r == 0)
+    announce_mkdir (dir, options);
+  return r;
+}
+
 /* Process a command-line file name, for the -d option.  */
 static int
 process_dir (char *dir, struct savewd *wd, void *options)
@@ -419,267 +423,256 @@ process_dir (char *dir, struct savewd *wd, void *options)
           : EXIT_FAILURE);
 }

-int
-main (int argc, char **argv)
-{
-  int optc;
-  int exit_status = EXIT_SUCCESS;
-  const char *specified_mode = NULL;
-  bool make_backups = false;
-  char *backup_suffix_string;
-  char *version_control_string = NULL;
-  bool mkdir_and_install = false;
-  struct cp_options x;
-  char const *target_directory = NULL;
-  bool no_target_directory = false;
-  int n_files;
-  char **file;
-  bool strip_program_specified = false;
-  security_context_t scontext = NULL;
-  /* set iff kernel has extra selinux system calls */
-  selinux_enabled = (0 < is_selinux_enabled ());
-
-  initialize_main (&argc, &argv);
-  set_program_name (argv[0]);
-  setlocale (LC_ALL, "");
-  bindtextdomain (PACKAGE, LOCALEDIR);
-  textdomain (PACKAGE);
+/* Copy file FROM onto file TO, creating TO if necessary.
+   Return true if successful.  */

-  atexit (close_stdin);
+static bool
+copy_file (const char *from, const char *to, const struct cp_options *x)
+{
+  bool copy_into_self;

-  cp_option_init (&x);
+  if (copy_only_if_needed && !need_copy (from, to, x))
+    return true;

-  owner_name = NULL;
-  group_name = NULL;
-  strip_files = false;
-  dir_arg = false;
-  umask (0);
+  /* Allow installing from non-regular files like /dev/null.
+     Charles Karney reported that some Sun version of install allows that
+     and that sendmail's installation process relies on the behavior.
+     However, since !x->recursive, the call to "copy" will fail if FROM
+     is a directory.  */

-  /* FIXME: consider not calling getenv for SIMPLE_BACKUP_SUFFIX unless
-     we'll actually use backup_suffix_string.  */
-  backup_suffix_string = getenv ("SIMPLE_BACKUP_SUFFIX");
+  return copy (from, to, false, x, &copy_into_self, NULL);
+}

-  while ((optc = getopt_long (argc, argv, "bcCsDdg:m:o:pt:TvS:Z:", long_options,
-                              NULL)) != -1)
-    {
-      switch (optc)
-        {
-        case 'b':
-          make_backups = true;
-          if (optarg)
-            version_control_string = optarg;
-          break;
-        case 'c':
-          break;
-        case 'C':
-          copy_only_if_needed = true;
-          break;
-        case 's':
-          strip_files = true;
-#ifdef SIGCHLD
-          /* System V fork+wait does not work if SIGCHLD is ignored.  */
-          signal (SIGCHLD, SIG_DFL);
-#endif
-          break;
-        case STRIP_PROGRAM_OPTION:
-          strip_program = xstrdup (optarg);
-          strip_program_specified = true;
-          break;
-        case 'd':
-          dir_arg = true;
-          break;
-        case 'D':
-          mkdir_and_install = true;
-          break;
-        case 'v':
-          x.verbose = true;
-          break;
-        case 'g':
-          group_name = optarg;
-          break;
-        case 'm':
-          specified_mode = optarg;
-          break;
-        case 'o':
-          owner_name = optarg;
-          break;
-        case 'p':
-          x.preserve_timestamps = true;
-          break;
-        case 'S':
-          make_backups = true;
-          backup_suffix_string = optarg;
-          break;
-        case 't':
-          if (target_directory)
-            error (EXIT_FAILURE, 0,
-                   _("multiple target directories specified"));
-          else
-            {
-              struct stat st;
-              if (stat (optarg, &st) != 0)
-                error (EXIT_FAILURE, errno, _("accessing %s"), quote (optarg));
-              if (! S_ISDIR (st.st_mode))
-                error (EXIT_FAILURE, 0, _("target %s is not a directory"),
-                       quote (optarg));
-            }
-          target_directory = optarg;
-          break;
-        case 'T':
-          no_target_directory = true;
-          break;
+/* Set the attributes of file or directory NAME.
+   Return true if successful.  */

-        case PRESERVE_CONTEXT_OPTION_DEPRECATED:
-          error (0, 0, _("WARNING: --preserve_context is deprecated; "
-                         "use --preserve-context instead"));
-          /* fall through */
-        case PRESERVE_CONTEXT_OPTION:
-          if ( ! selinux_enabled)
-            {
-              error (0, 0, _("WARNING: ignoring --preserve-context; "
-                             "this kernel is not SELinux-enabled"));
-              break;
-            }
-          x.preserve_security_context = true;
-          use_default_selinux_context = false;
-          break;
-        case 'Z':
-          if ( ! selinux_enabled)
-            {
-              error (0, 0, _("WARNING: ignoring --context (-Z); "
-                             "this kernel is not SELinux-enabled"));
-              break;
-            }
-          scontext = optarg;
-          use_default_selinux_context = false;
-          break;
-        case_GETOPT_HELP_CHAR;
-        case_GETOPT_VERSION_CHAR (PROGRAM_NAME, AUTHORS);
-        default:
-          usage (EXIT_FAILURE);
-        }
-    }
+static bool
+change_attributes (char const *name)
+{
+  bool ok = false;
+  /* chown must precede chmod because on some systems,
+     chown clears the set[ug]id bits for non-superusers,
+     resulting in incorrect permissions.
+     On System V, users can give away files with chown and then not
+     be able to chmod them.  So don't give files away.

-  /* Check for invalid combinations of arguments. */
-  if (dir_arg && strip_files)
-    error (EXIT_FAILURE, 0,
-           _("the strip option may not be used when installing a directory"));
-  if (dir_arg && target_directory)
-    error (EXIT_FAILURE, 0,
-           _("target directory not allowed when installing a directory"));
+     We don't normally ignore errors from chown because the idea of
+     the install command is that the file is supposed to end up with
+     precisely the attributes that the user specified (or defaulted).
+     If the file doesn't end up with the group they asked for, they'll
+     want to know.  */

-  if (x.preserve_security_context && scontext != NULL)
-    error (EXIT_FAILURE, 0,
-           _("cannot force target context to %s and preserve it"),
-           quote (scontext));
+  if (! (owner_id == (uid_t) -1 && group_id == (gid_t) -1)
+      && lchown (name, owner_id, group_id) != 0)
+    error (0, errno, _("cannot change ownership of %s"), quote (name));
+  else if (chmod (name, mode) != 0)
+    error (0, errno, _("cannot change permissions of %s"), quote (name));
+  else
+    ok = true;

-  if (backup_suffix_string)
-    simple_backup_suffix = xstrdup (backup_suffix_string);
+  if (use_default_selinux_context)
+    setdefaultfilecon (name);

-  x.backup_type = (make_backups
-                   ? xget_version (_("backup type"),
-                                   version_control_string)
-                   : no_backups);
+  return ok;
+}

-  if (scontext && setfscreatecon (scontext) < 0)
-    error (EXIT_FAILURE, errno,
-           _("failed to set default file creation context to %s"),
-           quote (scontext));
+/* Set the timestamps of file DEST to match those of SRC_SB.
+   Return true if successful.  */

-  n_files = argc - optind;
-  file = argv + optind;
+static bool
+change_timestamps (struct stat const *src_sb, char const *dest)
+{
+  struct timespec timespec[2];
+  timespec[0] = get_stat_atime (src_sb);
+  timespec[1] = get_stat_mtime (src_sb);

-  if (n_files <= ! (dir_arg || target_directory))
+  if (utimens (dest, timespec))
     {
-      if (n_files <= 0)
-        error (0, 0, _("missing file operand"));
-      else
-        error (0, 0, _("missing destination file operand after %s"),
-               quote (file[0]));
-      usage (EXIT_FAILURE);
+      error (0, errno, _("cannot set time stamps for %s"), quote (dest));
+      return false;
     }
+  return true;
+}

-  if (no_target_directory)
+/* Strip the symbol table from the file NAME.
+   We could dig the magic number out of the file first to
+   determine whether to strip it, but the header files and
+   magic numbers vary so much from system to system that making
+   it portable would be very difficult.  Not worth the effort. */
+
+static void
+strip (char const *name)
+{
+  int status;
+  pid_t pid = fork ();
+
+  switch (pid)
     {
-      if (target_directory)
-        error (EXIT_FAILURE, 0,
-               _("cannot combine --target-directory (-t) "
-                 "and --no-target-directory (-T)"));
-      if (2 < n_files)
-        {
-          error (0, 0, _("extra operand %s"), quote (file[2]));
-          usage (EXIT_FAILURE);
-        }
-    }
-  else if (! (dir_arg || target_directory))
-    {
-      if (2 <= n_files && target_directory_operand (file[n_files - 1]))
-        target_directory = file[--n_files];
-      else if (2 < n_files)
-        error (EXIT_FAILURE, 0, _("target %s is not a directory"),
-               quote (file[n_files - 1]));
+    case -1:
+      error (EXIT_FAILURE, errno, _("fork system call failed"));
+      break;
+    case 0:			/* Child. */
+      execlp (strip_program, strip_program, name, NULL);
+      error (EXIT_FAILURE, errno, _("cannot run %s"), strip_program);
+      break;
+    default:			/* Parent. */
+      if (waitpid (pid, &status, 0) < 0)
+        error (EXIT_FAILURE, errno, _("waiting for strip"));
+      else if (! WIFEXITED (status) || WEXITSTATUS (status))
+        error (EXIT_FAILURE, 0, _("strip process terminated abnormally"));
+      break;
     }
+}

-  if (specified_mode)
-    {
-      struct mode_change *change = mode_compile (specified_mode);
-      if (!change)
-        error (EXIT_FAILURE, 0, _("invalid mode %s"), quote (specified_mode));
-      mode = mode_adjust (0, false, 0, change, NULL);
-      dir_mode = mode_adjust (0, true, 0, change, &dir_mode_bits);
-      free (change);
-    }
+/* Initialize the user and group ownership of the files to install. */

-  if (strip_program_specified && !strip_files)
-    error (0, 0, _("WARNING: ignoring --strip-program option as -s option was "
-                   "not specified"));
+static void
+get_ids (void)
+{
+  struct passwd *pw;
+  struct group *gr;

-  if (copy_only_if_needed && x.preserve_timestamps)
+  if (owner_name)
     {
-      error (0, 0, _("options --compare (-C) and --preserve-timestamps are "
-                     "mutually exclusive"));
-      usage (EXIT_FAILURE);
+      pw = getpwnam (owner_name);
+      if (pw == NULL)
+        {
+          unsigned long int tmp;
+          if (xstrtoul (owner_name, NULL, 0, &tmp, NULL) != LONGINT_OK
+              || UID_T_MAX < tmp)
+            error (EXIT_FAILURE, 0, _("invalid user %s"), quote (owner_name));
+          owner_id = tmp;
+        }
+      else
+        owner_id = pw->pw_uid;
+      endpwent ();
     }
+  else
+    owner_id = (uid_t) -1;

-  if (copy_only_if_needed && strip_files)
+  if (group_name)
     {
-      error (0, 0, _("options --compare (-C) and --strip are mutually "
-                     "exclusive"));
-      usage (EXIT_FAILURE);
+      gr = getgrnam (group_name);
+      if (gr == NULL)
+        {
+          unsigned long int tmp;
+          if (xstrtoul (group_name, NULL, 0, &tmp, NULL) != LONGINT_OK
+              || GID_T_MAX < tmp)
+            error (EXIT_FAILURE, 0, _("invalid group %s"), quote (group_name));
+          group_id = tmp;
+        }
+      else
+        group_id = gr->gr_gid;
+      endgrent ();
     }
+  else
+    group_id = (gid_t) -1;
+}

-  if (copy_only_if_needed && extra_mode (mode))
-    error (0, 0, _("the --compare (-C) option is ignored when you"
-                   " specify a mode with non-permission bits"));
-
-  get_ids ();
-
-  if (dir_arg)
-    exit_status = savewd_process_files (n_files, file, process_dir, &x);
+void
+usage (int status)
+{
+  if (status != EXIT_SUCCESS)
+    fprintf (stderr, _("Try `%s --help' for more information.\n"),
+             program_name);
   else
     {
-      /* FIXME: it's a little gross that this initialization is
-         required by copy.c::copy. */
-      hash_init ();
+      printf (_("\
+Usage: %s [OPTION]... [-T] SOURCE DEST\n\
+  or:  %s [OPTION]... SOURCE... DIRECTORY\n\
+  or:  %s [OPTION]... -t DIRECTORY SOURCE...\n\
+  or:  %s [OPTION]... -d DIRECTORY...\n\
+"),
+              program_name, program_name, program_name, program_name);
+      fputs (_("\
+\n\
+This install program copies files (often just compiled) into destination\n\
+locations you choose.  If you want to download and install a ready-to-use\n\
+package on a GNU/Linux system, you should instead be using a package manager\n\
+like yum(1) or apt-get(1).\n\
+\n\
+In the first three forms, copy SOURCE to DEST or multiple SOURCE(s) to\n\
+the existing DIRECTORY, while setting permission modes and owner/group.\n\
+In the 4th form, create all components of the given DIRECTORY(ies).\n\
+\n\
+"), stdout);
+      fputs (_("\
+Mandatory arguments to long options are mandatory for short options too.\n\
+"), stdout);
+      fputs (_("\
+      --backup[=CONTROL]  make a backup of each existing destination file\n\
+  -b                  like --backup but does not accept an argument\n\
+  -c                  (ignored)\n\
+  -C, --compare       compare each pair of source and destination files, and\n\
+                        in some cases, do not modify the destination at all\n\
+  -d, --directory     treat all arguments as directory names; create all\n\
+                        components of the specified directories\n\
+"), stdout);
+      fputs (_("\
+  -D                  create all leading components of DEST except the last,\n\
+                        then copy SOURCE to DEST\n\
+  -g, --group=GROUP   set group ownership, instead of process' current group\n\
+  -m, --mode=MODE     set permission mode (as in chmod), instead of rwxr-xr-x\n\
+  -o, --owner=OWNER   set ownership (super-user only)\n\
+"), stdout);
+      fputs (_("\
+  -p, --preserve-timestamps   apply access/modification times of SOURCE files\n\
+                        to corresponding destination files\n\
+  -s, --strip         strip symbol tables\n\
+      --strip-program=PROGRAM  program used to strip binaries\n\
+  -S, --suffix=SUFFIX  override the usual backup suffix\n\
+  -t, --target-directory=DIRECTORY  copy all SOURCE arguments into DIRECTORY\n\
+  -T, --no-target-directory  treat DEST as a normal file\n\
+  -v, --verbose       print the name of each directory as it is created\n\
+"), stdout);
+      fputs (_("\
+      --preserve-context  preserve SELinux security context\n\
+  -Z, --context=CONTEXT  set SELinux security context of files and directories\
+\n\
+"), stdout);

-      if (!target_directory)
-        {
-          if (! (mkdir_and_install
-                 ? install_file_in_file_parents (file[0], file[1], &x)
-                 : install_file_in_file (file[0], file[1], &x)))
-            exit_status = EXIT_FAILURE;
-        }
-      else
-        {
-          int i;
-          dest_info_init (&x);
-          for (i = 0; i < n_files; i++)
-            if (! install_file_in_dir (file[i], target_directory, &x))
-              exit_status = EXIT_FAILURE;
-        }
+      fputs (HELP_OPTION_DESCRIPTION, stdout);
+      fputs (VERSION_OPTION_DESCRIPTION, stdout);
+      fputs (_("\
+\n\
+The backup suffix is `~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.\n\
+The version control method may be selected via the --backup option or through\n\
+the VERSION_CONTROL environment variable.  Here are the values:\n\
+\n\
+"), stdout);
+      fputs (_("\
+  none, off       never make backups (even if --backup is given)\n\
+  numbered, t     make numbered backups\n\
+  existing, nil   numbered if numbered backups exist, simple otherwise\n\
+  simple, never   always make simple backups\n\
+"), stdout);
+      emit_ancillary_info ();
     }
+  exit (status);
+}

-  exit (exit_status);
+/* Copy file FROM onto file TO and give TO the appropriate
+   attributes.
+   Return true if successful.  */
+
+static bool
+install_file_in_file (const char *from, const char *to,
+                      const struct cp_options *x)
+{
+  struct stat from_sb;
+  if (x->preserve_timestamps && stat (from, &from_sb) != 0)
+    {
+      error (0, errno, _("cannot stat %s"), quote (from));
+      return false;
+    }
+  if (! copy_file (from, to, x))
+    return false;
+  if (strip_files)
+    strip (to);
+  if (x->preserve_timestamps && (strip_files || ! S_ISREG (from_sb.st_mode))
+      && ! change_timestamps (&from_sb, to))
+    return false;
+  return change_attributes (to);
 }

 /* Copy file FROM onto file TO, creating any missing parent directories of TO.
@@ -721,30 +714,6 @@ install_file_in_file_parents (char const *from, char *to,
   return (status == EXIT_SUCCESS && install_file_in_file (from, to, x));
 }

-/* Copy file FROM onto file TO and give TO the appropriate
-   attributes.
-   Return true if successful.  */
-
-static bool
-install_file_in_file (const char *from, const char *to,
-                      const struct cp_options *x)
-{
-  struct stat from_sb;
-  if (x->preserve_timestamps && stat (from, &from_sb) != 0)
-    {
-      error (0, errno, _("cannot stat %s"), quote (from));
-      return false;
-    }
-  if (! copy_file (from, to, x))
-    return false;
-  if (strip_files)
-    strip (to);
-  if (x->preserve_timestamps && (strip_files || ! S_ISREG (from_sb.st_mode))
-      && ! change_timestamps (&from_sb, to))
-    return false;
-  return change_attributes (to);
-}
-
 /* Copy file FROM into directory TO_DIR, keeping its same name,
    and give the copy the appropriate attributes.
    Return true if successful.  */
@@ -760,251 +729,265 @@ install_file_in_dir (const char *from, const char *to_dir,
   return ret;
 }

-/* Copy file FROM onto file TO, creating TO if necessary.
-   Return true if successful.  */
-
-static bool
-copy_file (const char *from, const char *to, const struct cp_options *x)
+int
+main (int argc, char **argv)
 {
-  bool copy_into_self;
-
-  if (copy_only_if_needed && !need_copy (from, to, x))
-    return true;
-
-  /* Allow installing from non-regular files like /dev/null.
-     Charles Karney reported that some Sun version of install allows that
-     and that sendmail's installation process relies on the behavior.
-     However, since !x->recursive, the call to "copy" will fail if FROM
-     is a directory.  */
-
-  return copy (from, to, false, x, &copy_into_self, NULL);
-}
-
-/* Set the attributes of file or directory NAME.
-   Return true if successful.  */
-
-static bool
-change_attributes (char const *name)
-{
-  bool ok = false;
-  /* chown must precede chmod because on some systems,
-     chown clears the set[ug]id bits for non-superusers,
-     resulting in incorrect permissions.
-     On System V, users can give away files with chown and then not
-     be able to chmod them.  So don't give files away.
-
-     We don't normally ignore errors from chown because the idea of
-     the install command is that the file is supposed to end up with
-     precisely the attributes that the user specified (or defaulted).
-     If the file doesn't end up with the group they asked for, they'll
-     want to know.  */
+  int optc;
+  int exit_status = EXIT_SUCCESS;
+  const char *specified_mode = NULL;
+  bool make_backups = false;
+  char *backup_suffix_string;
+  char *version_control_string = NULL;
+  bool mkdir_and_install = false;
+  struct cp_options x;
+  char const *target_directory = NULL;
+  bool no_target_directory = false;
+  int n_files;
+  char **file;
+  bool strip_program_specified = false;
+  security_context_t scontext = NULL;
+  /* set iff kernel has extra selinux system calls */
+  selinux_enabled = (0 < is_selinux_enabled ());

-  if (! (owner_id == (uid_t) -1 && group_id == (gid_t) -1)
-      && lchown (name, owner_id, group_id) != 0)
-    error (0, errno, _("cannot change ownership of %s"), quote (name));
-  else if (chmod (name, mode) != 0)
-    error (0, errno, _("cannot change permissions of %s"), quote (name));
-  else
-    ok = true;
+  initialize_main (&argc, &argv);
+  set_program_name (argv[0]);
+  setlocale (LC_ALL, "");
+  bindtextdomain (PACKAGE, LOCALEDIR);
+  textdomain (PACKAGE);

-  if (use_default_selinux_context)
-    setdefaultfilecon (name);
+  atexit (close_stdin);

-  return ok;
-}
+  cp_option_init (&x);

-/* Set the timestamps of file DEST to match those of SRC_SB.
-   Return true if successful.  */
+  owner_name = NULL;
+  group_name = NULL;
+  strip_files = false;
+  dir_arg = false;
+  umask (0);

-static bool
-change_timestamps (struct stat const *src_sb, char const *dest)
-{
-  struct timespec timespec[2];
-  timespec[0] = get_stat_atime (src_sb);
-  timespec[1] = get_stat_mtime (src_sb);
+  /* FIXME: consider not calling getenv for SIMPLE_BACKUP_SUFFIX unless
+     we'll actually use backup_suffix_string.  */
+  backup_suffix_string = getenv ("SIMPLE_BACKUP_SUFFIX");

-  if (utimens (dest, timespec))
+  while ((optc = getopt_long (argc, argv, "bcCsDdg:m:o:pt:TvS:Z:", long_options,
+                              NULL)) != -1)
     {
-      error (0, errno, _("cannot set time stamps for %s"), quote (dest));
-      return false;
+      switch (optc)
+        {
+        case 'b':
+          make_backups = true;
+          if (optarg)
+            version_control_string = optarg;
+          break;
+        case 'c':
+          break;
+        case 'C':
+          copy_only_if_needed = true;
+          break;
+        case 's':
+          strip_files = true;
+#ifdef SIGCHLD
+          /* System V fork+wait does not work if SIGCHLD is ignored.  */
+          signal (SIGCHLD, SIG_DFL);
+#endif
+          break;
+        case STRIP_PROGRAM_OPTION:
+          strip_program = xstrdup (optarg);
+          strip_program_specified = true;
+          break;
+        case 'd':
+          dir_arg = true;
+          break;
+        case 'D':
+          mkdir_and_install = true;
+          break;
+        case 'v':
+          x.verbose = true;
+          break;
+        case 'g':
+          group_name = optarg;
+          break;
+        case 'm':
+          specified_mode = optarg;
+          break;
+        case 'o':
+          owner_name = optarg;
+          break;
+        case 'p':
+          x.preserve_timestamps = true;
+          break;
+        case 'S':
+          make_backups = true;
+          backup_suffix_string = optarg;
+          break;
+        case 't':
+          if (target_directory)
+            error (EXIT_FAILURE, 0,
+                   _("multiple target directories specified"));
+          else
+            {
+              struct stat st;
+              if (stat (optarg, &st) != 0)
+                error (EXIT_FAILURE, errno, _("accessing %s"), quote (optarg));
+              if (! S_ISDIR (st.st_mode))
+                error (EXIT_FAILURE, 0, _("target %s is not a directory"),
+                       quote (optarg));
+            }
+          target_directory = optarg;
+          break;
+        case 'T':
+          no_target_directory = true;
+          break;
+
+        case PRESERVE_CONTEXT_OPTION_DEPRECATED:
+          error (0, 0, _("WARNING: --preserve_context is deprecated; "
+                         "use --preserve-context instead"));
+          /* fall through */
+        case PRESERVE_CONTEXT_OPTION:
+          if ( ! selinux_enabled)
+            {
+              error (0, 0, _("WARNING: ignoring --preserve-context; "
+                             "this kernel is not SELinux-enabled"));
+              break;
+            }
+          x.preserve_security_context = true;
+          use_default_selinux_context = false;
+          break;
+        case 'Z':
+          if ( ! selinux_enabled)
+            {
+              error (0, 0, _("WARNING: ignoring --context (-Z); "
+                             "this kernel is not SELinux-enabled"));
+              break;
+            }
+          scontext = optarg;
+          use_default_selinux_context = false;
+          break;
+        case_GETOPT_HELP_CHAR;
+        case_GETOPT_VERSION_CHAR (PROGRAM_NAME, AUTHORS);
+        default:
+          usage (EXIT_FAILURE);
+        }
     }
-  return true;
-}

-/* Strip the symbol table from the file NAME.
-   We could dig the magic number out of the file first to
-   determine whether to strip it, but the header files and
-   magic numbers vary so much from system to system that making
-   it portable would be very difficult.  Not worth the effort. */
+  /* Check for invalid combinations of arguments. */
+  if (dir_arg && strip_files)
+    error (EXIT_FAILURE, 0,
+           _("the strip option may not be used when installing a directory"));
+  if (dir_arg && target_directory)
+    error (EXIT_FAILURE, 0,
+           _("target directory not allowed when installing a directory"));

-static void
-strip (char const *name)
-{
-  int status;
-  pid_t pid = fork ();
+  if (x.preserve_security_context && scontext != NULL)
+    error (EXIT_FAILURE, 0,
+           _("cannot force target context to %s and preserve it"),
+           quote (scontext));

-  switch (pid)
-    {
-    case -1:
-      error (EXIT_FAILURE, errno, _("fork system call failed"));
-      break;
-    case 0:			/* Child. */
-      execlp (strip_program, strip_program, name, NULL);
-      error (EXIT_FAILURE, errno, _("cannot run %s"), strip_program);
-      break;
-    default:			/* Parent. */
-      if (waitpid (pid, &status, 0) < 0)
-        error (EXIT_FAILURE, errno, _("waiting for strip"));
-      else if (! WIFEXITED (status) || WEXITSTATUS (status))
-        error (EXIT_FAILURE, 0, _("strip process terminated abnormally"));
-      break;
-    }
-}
+  if (backup_suffix_string)
+    simple_backup_suffix = xstrdup (backup_suffix_string);

-/* Initialize the user and group ownership of the files to install. */
+  x.backup_type = (make_backups
+                   ? xget_version (_("backup type"),
+                                   version_control_string)
+                   : no_backups);

-static void
-get_ids (void)
-{
-  struct passwd *pw;
-  struct group *gr;
+  if (scontext && setfscreatecon (scontext) < 0)
+    error (EXIT_FAILURE, errno,
+           _("failed to set default file creation context to %s"),
+           quote (scontext));

-  if (owner_name)
+  n_files = argc - optind;
+  file = argv + optind;
+
+  if (n_files <= ! (dir_arg || target_directory))
     {
-      pw = getpwnam (owner_name);
-      if (pw == NULL)
-        {
-          unsigned long int tmp;
-          if (xstrtoul (owner_name, NULL, 0, &tmp, NULL) != LONGINT_OK
-              || UID_T_MAX < tmp)
-            error (EXIT_FAILURE, 0, _("invalid user %s"), quote (owner_name));
-          owner_id = tmp;
-        }
+      if (n_files <= 0)
+        error (0, 0, _("missing file operand"));
       else
-        owner_id = pw->pw_uid;
-      endpwent ();
+        error (0, 0, _("missing destination file operand after %s"),
+               quote (file[0]));
+      usage (EXIT_FAILURE);
     }
-  else
-    owner_id = (uid_t) -1;

-  if (group_name)
+  if (no_target_directory)
     {
-      gr = getgrnam (group_name);
-      if (gr == NULL)
+      if (target_directory)
+        error (EXIT_FAILURE, 0,
+               _("cannot combine --target-directory (-t) "
+                 "and --no-target-directory (-T)"));
+      if (2 < n_files)
         {
-          unsigned long int tmp;
-          if (xstrtoul (group_name, NULL, 0, &tmp, NULL) != LONGINT_OK
-              || GID_T_MAX < tmp)
-            error (EXIT_FAILURE, 0, _("invalid group %s"), quote (group_name));
-          group_id = tmp;
+          error (0, 0, _("extra operand %s"), quote (file[2]));
+          usage (EXIT_FAILURE);
         }
-      else
-        group_id = gr->gr_gid;
-      endgrent ();
     }
-  else
-    group_id = (gid_t) -1;
-}
+  else if (! (dir_arg || target_directory))
+    {
+      if (2 <= n_files && target_directory_operand (file[n_files - 1]))
+        target_directory = file[--n_files];
+      else if (2 < n_files)
+        error (EXIT_FAILURE, 0, _("target %s is not a directory"),
+               quote (file[n_files - 1]));
+    }

-/* Report that directory DIR was made, if OPTIONS requests this.  */
-static void
-announce_mkdir (char const *dir, void *options)
-{
-  struct cp_options const *x = options;
-  if (x->verbose)
-    prog_fprintf (stdout, _("creating directory %s"), quote (dir));
-}
+  if (specified_mode)
+    {
+      struct mode_change *change = mode_compile (specified_mode);
+      if (!change)
+        error (EXIT_FAILURE, 0, _("invalid mode %s"), quote (specified_mode));
+      mode = mode_adjust (0, false, 0, change, NULL);
+      dir_mode = mode_adjust (0, true, 0, change, &dir_mode_bits);
+      free (change);
+    }

-/* Make ancestor directory DIR, whose last file name component is
-   COMPONENT, with options OPTIONS.  Assume the working directory is
-   COMPONENT's parent.  */
-static int
-make_ancestor (char const *dir, char const *component, void *options)
-{
-  int r = mkdir (component, DEFAULT_MODE);
-  if (r == 0)
-    announce_mkdir (dir, options);
-  return r;
-}
+  if (strip_program_specified && !strip_files)
+    error (0, 0, _("WARNING: ignoring --strip-program option as -s option was "
+                   "not specified"));

-void
-usage (int status)
-{
-  if (status != EXIT_SUCCESS)
-    fprintf (stderr, _("Try `%s --help' for more information.\n"),
-             program_name);
+  if (copy_only_if_needed && x.preserve_timestamps)
+    {
+      error (0, 0, _("options --compare (-C) and --preserve-timestamps are "
+                     "mutually exclusive"));
+      usage (EXIT_FAILURE);
+    }
+
+  if (copy_only_if_needed && strip_files)
+    {
+      error (0, 0, _("options --compare (-C) and --strip are mutually "
+                     "exclusive"));
+      usage (EXIT_FAILURE);
+    }
+
+  if (copy_only_if_needed && extra_mode (mode))
+    error (0, 0, _("the --compare (-C) option is ignored when you"
+                   " specify a mode with non-permission bits"));
+
+  get_ids ();
+
+  if (dir_arg)
+    exit_status = savewd_process_files (n_files, file, process_dir, &x);
   else
     {
-      printf (_("\
-Usage: %s [OPTION]... [-T] SOURCE DEST\n\
-  or:  %s [OPTION]... SOURCE... DIRECTORY\n\
-  or:  %s [OPTION]... -t DIRECTORY SOURCE...\n\
-  or:  %s [OPTION]... -d DIRECTORY...\n\
-"),
-              program_name, program_name, program_name, program_name);
-      fputs (_("\
-\n\
-This install program copies files (often just compiled) into destination\n\
-locations you choose.  If you want to download and install a ready-to-use\n\
-package on a GNU/Linux system, you should instead be using a package manager\n\
-like yum(1) or apt-get(1).\n\
-\n\
-In the first three forms, copy SOURCE to DEST or multiple SOURCE(s) to\n\
-the existing DIRECTORY, while setting permission modes and owner/group.\n\
-In the 4th form, create all components of the given DIRECTORY(ies).\n\
-\n\
-"), stdout);
-      fputs (_("\
-Mandatory arguments to long options are mandatory for short options too.\n\
-"), stdout);
-      fputs (_("\
-      --backup[=CONTROL]  make a backup of each existing destination file\n\
-  -b                  like --backup but does not accept an argument\n\
-  -c                  (ignored)\n\
-  -C, --compare       compare each pair of source and destination files, and\n\
-                        in some cases, do not modify the destination at all\n\
-  -d, --directory     treat all arguments as directory names; create all\n\
-                        components of the specified directories\n\
-"), stdout);
-      fputs (_("\
-  -D                  create all leading components of DEST except the last,\n\
-                        then copy SOURCE to DEST\n\
-  -g, --group=GROUP   set group ownership, instead of process' current group\n\
-  -m, --mode=MODE     set permission mode (as in chmod), instead of rwxr-xr-x\n\
-  -o, --owner=OWNER   set ownership (super-user only)\n\
-"), stdout);
-      fputs (_("\
-  -p, --preserve-timestamps   apply access/modification times of SOURCE files\n\
-                        to corresponding destination files\n\
-  -s, --strip         strip symbol tables\n\
-      --strip-program=PROGRAM  program used to strip binaries\n\
-  -S, --suffix=SUFFIX  override the usual backup suffix\n\
-  -t, --target-directory=DIRECTORY  copy all SOURCE arguments into DIRECTORY\n\
-  -T, --no-target-directory  treat DEST as a normal file\n\
-  -v, --verbose       print the name of each directory as it is created\n\
-"), stdout);
-      fputs (_("\
-      --preserve-context  preserve SELinux security context\n\
-  -Z, --context=CONTEXT  set SELinux security context of files and directories\
-\n\
-"), stdout);
+      /* FIXME: it's a little gross that this initialization is
+         required by copy.c::copy. */
+      hash_init ();

-      fputs (HELP_OPTION_DESCRIPTION, stdout);
-      fputs (VERSION_OPTION_DESCRIPTION, stdout);
-      fputs (_("\
-\n\
-The backup suffix is `~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX.\n\
-The version control method may be selected via the --backup option or through\n\
-the VERSION_CONTROL environment variable.  Here are the values:\n\
-\n\
-"), stdout);
-      fputs (_("\
-  none, off       never make backups (even if --backup is given)\n\
-  numbered, t     make numbered backups\n\
-  existing, nil   numbered if numbered backups exist, simple otherwise\n\
-  simple, never   always make simple backups\n\
-"), stdout);
-      emit_ancillary_info ();
+      if (!target_directory)
+        {
+          if (! (mkdir_and_install
+                 ? install_file_in_file_parents (file[0], file[1], &x)
+                 : install_file_in_file (file[0], file[1], &x)))
+            exit_status = EXIT_FAILURE;
+        }
+      else
+        {
+          int i;
+          dest_info_init (&x);
+          for (i = 0; i < n_files; i++)
+            if (! install_file_in_dir (file[i], target_directory, &x))
+              exit_status = EXIT_FAILURE;
+        }
     }
-  exit (status);
+
+  exit (exit_status);
 }
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Mon Apr 11 23:22:15 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9UBD-00048R-CI
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 23:22:15 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55186)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9SY1-00031k-Jl
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:37:45 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9NQM-000266-Gr
	for coreutils@gnu.org; Mon, 11 Apr 2011 16:09:27 -0400
Received: from mx.meyering.net ([82.230.74.64]:46340)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9NQM-00023J-7z
	for coreutils@gnu.org; Mon, 11 Apr 2011 16:09:26 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 6809060119; Mon, 11 Apr 2011 22:09:24 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
In-Reply-To: <4DA2E102.5080801@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Mon, 11 Apr 2011 12:07:46 +0100")
References: <87ei5chcy1.fsf@rho.meyering.net>
	<4D9F9C20.5020709@draigBrady.com> <4DA2E102.5080801@draigBrady.com>
Date: Mon, 11 Apr 2011 22:09:24 +0200
Message-ID: <878vvgh7nv.fsf@rho.meyering.net>
Lines: 47
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 03:22:14 -0000

P=E1draig Brady wrote:
> On 09/04/11 00:37, P=E1draig Brady wrote:
>> One thing I'm worried about is fiemap with compressed file systems.
>> I'm unsure that the extent length refers to the length of
>> the unencoded data. That's what I would expect so that
>> user space is abstracted from encoding details and so
>> we know how much to read to fully process an extent.
>>
>> I've not got a new BTRFS kernel readily available at present
>> to do a quick test. I'll try to text tomorrow night.
>
> I've tried various BTRFS combinations here with 2.6.38.2-9.fc15.i686.
> I'm using `mount -o compress -o compress-force`, but any files
> copied into the file system, don't have FIEMAP_EXTENT_ENCODED flags,
> and du doesn't indicate that less space is used in the file system.

Thanks for testing that.
I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs and x=
fs.
All tests passed on the first three FS types.
On xfs there was only one failure: cp/sparse-fiemap.
I pared it down to this stand-alone reproducer:

    rm -f j1 j2
    perl \
      -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
      -e 'for (1..21) { sysseek (*F, $n, 1)' \
      -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
    cp --sparse=3Dalways j1 j2
    diff -u <(filefrag -v j1) <(filefrag -vs j2)

Here's the output.
The difference that triggers the test failure is
the fact that the "length" numbers differ:

    --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
    +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
    @@ -1,6 +1,6 @@
     Filesystem type is: 58465342
    -File size of j1 is 301056 (74 blocks, blocksize 4096)
    +File size of j2 is 301056 (74 blocks, blocksize 4096)
      ext logical physical expected length flags
    -   0       1     7310              31
    -   1      33     7342     7340     95 eof
    -j1: 3 extents found
    +   0       1     7469              31
    +   1      33     7501     7499     63 eof
    +j2: 3 extents found


From MAILER-DAEMON Mon Apr 11 23:37:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9UQH-0008Ml-FL
	for mharc-coreutils@gnu.org; Mon, 11 Apr 2011 23:37:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:57854)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShK-0003gw-37
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:36 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LCG-0006Xi-RE
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:46 -0400
Received: from mx.meyering.net ([82.230.74.64]:45824)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LCG-0006XM-Kw
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:44 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id E5B2460362; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 7/7] build: update gnulib submodule to latest
Date: Mon, 11 Apr 2011 19:46:03 +0200
Message-Id: <1302543963-9069-8-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 03:37:47 -0000

From: Jim Meyering <meyering@redhat.com>

---
 gnulib |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/gnulib b/gnulib
index 1a22d91..a81348d 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit 1a22d910adea20751cc5649b224574d07ba72815
+Subproject commit a81348d2e136b0f2de110b49a5d0dfa5d7d76f71
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Tue Apr 12 00:45:06 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9VTO-00022t-9A
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 00:45:06 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60265)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShL-00041R-6i
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:28 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBl-0006TY-V1
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:14 -0400
Received: from mx.meyering.net ([82.230.74.64]:34001)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBl-0006Sz-CZ
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:13 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 80EE960279; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 2/7] maint: rename variables for clarity...
Date: Mon, 11 Apr 2011 19:45:58 +0200
Message-Id: <1302543963-9069-3-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 04:45:04 -0000

From: Jim Meyering <meyering@redhat.com>

and to avoid a false-positive "TO to" in new doubled word check.
* src/install.c (change_timestamps): Rename parameters for
readability.  Make the comment match the code.
---
 src/install.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/install.c b/src/install.c
index cebb642..f416996 100644
--- a/src/install.c
+++ b/src/install.c
@@ -813,19 +813,19 @@ change_attributes (char const *name)
   return ok;
 }

-/* Set the timestamps of file TO to match those of file FROM.
+/* Set the timestamps of file DEST to match those of SRC_SB.
    Return true if successful.  */

 static bool
-change_timestamps (struct stat const *from_sb, char const *to)
+change_timestamps (struct stat const *src_sb, char const *dest)
 {
   struct timespec timespec[2];
-  timespec[0] = get_stat_atime (from_sb);
-  timespec[1] = get_stat_mtime (from_sb);
+  timespec[0] = get_stat_atime (src_sb);
+  timespec[1] = get_stat_mtime (src_sb);

-  if (utimens (to, timespec))
+  if (utimens (dest, timespec))
     {
-      error (0, errno, _("cannot set time stamps for %s"), quote (to));
+      error (0, errno, _("cannot set time stamps for %s"), quote (dest));
       return false;
     }
   return true;
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Tue Apr 12 00:47:51 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9VW3-0002K1-Ul
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 00:47:51 -0400
Received: from eggs.gnu.org ([140.186.70.92]:56805)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9ShL-0003Q0-1o
	for coreutils@gnu.org; Mon, 11 Apr 2011 21:47:27 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBn-0006UH-5K
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:17 -0400
Received: from mx.meyering.net ([82.230.74.64]:34012)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1Q9LBm-0006Ty-Uq
	for coreutils@gnu.org; Mon, 11 Apr 2011 13:46:15 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id D80EF60361; Mon, 11 Apr 2011 19:46:09 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 6/7] cfg.mk: remove useless semicolon and backslash
Date: Mon, 11 Apr 2011 19:46:02 +0200
Message-Id: <1302543963-9069-7-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.1.354.ge64bd
In-Reply-To: <1302543963-9069-1-git-send-email-jim@meyering.net>
References: <1302543963-9069-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 04:47:50 -0000

From: Jim Meyering <meyering@redhat.com>

* cfg.mk (sc_NEWS_two_empty_lines): Remove semicolon and backslash.
---
 cfg.mk |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/cfg.mk b/cfg.mk
index 126968e..81aa7e0 100644
--- a/cfg.mk
+++ b/cfg.mk
@@ -173,7 +173,7 @@ sc_NEWS_two_empty_lines:
 	@sed -n 4,/Noteworthy/p $(srcdir)/NEWS				\
 	    | perl -n0e '/(^|\n)\n\n\* Noteworthy/ or exit 1'		\
 	  || { echo '$(ME): use two empty lines to separate NEWS sections' \
-		 1>&2; exit 1; } || :;					\
+		 1>&2; exit 1; } || :

 # Perl-based tests used to exec perl from a #!/bin/sh script.
 # Now they all start with #!/usr/bin/perl and the portability
-- 
1.7.5.rc1.228.g86d60b



From MAILER-DAEMON Tue Apr 12 05:14:00 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9Zfc-0005jU-E3
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 05:14:00 -0400
Received: from eggs.gnu.org ([140.186.70.92]:57857)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9Zfa-0005jE-UW
	for coreutils@gnu.org; Tue, 12 Apr 2011 05:13:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9Zfa-0007gN-2d
	for coreutils@gnu.org; Tue, 12 Apr 2011 05:13:58 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:10721)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9ZfZ-0007g8-Ps
	for coreutils@gnu.org; Tue, 12 Apr 2011 05:13:58 -0400
Received: (qmail 26102 invoked from network); 12 Apr 2011 09:13:56 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 12 Apr 2011 09:13:56 -0000
Message-ID: <4DA4177C.2090403@draigBrady.com>
Date: Tue, 12 Apr 2011 10:12:28 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com> <878vvgh7nv.fsf@rho.meyering.net>
In-Reply-To: <878vvgh7nv.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 09:13:59 -0000

On 11/04/11 21:09, Jim Meyering wrote:
> I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs and xfs.
> All tests passed on the first three FS types.
> On xfs there was only one failure: cp/sparse-fiemap.
> I pared it down to this stand-alone reproducer:
> 
>     rm -f j1 j2
>     perl \
>       -e 'BEGIN{$n=7*1024; *F=*STDOUT}' \
>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>     cp --sparse=always j1 j2
>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
> 
> Here's the output.
> The difference that triggers the test failure is
> the fact that the "length" numbers differ:
> 
>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>     @@ -1,6 +1,6 @@
>      Filesystem type is: 58465342
>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>       ext logical physical expected length flags
>     -   0       1     7310              31
>     -   1      33     7342     7340     95 eof
>     -j1: 3 extents found
>     +   0       1     7469              31
>     +   1      33     7501     7499     63 eof
>     +j2: 3 extents found

Hmm, this is with 128K block sizes?
There is a 4K hole at the start of each block which seems a bit strange.
Why are "3 extents found" but only 2 listed?
The last extent seems too long in both cases,
but that at least wouldn't cause cp to corrupt a copy.

cheers,
Pádraig.

p.s. I'll s/2.6.38/2.6.39/ in that test


From MAILER-DAEMON Tue Apr 12 06:15:54 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9adW-0007hX-46
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 06:15:54 -0400
Received: from eggs.gnu.org ([140.186.70.92]:54386)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9adT-0007hI-Ut
	for coreutils@gnu.org; Tue, 12 Apr 2011 06:15:52 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9adT-00086b-4p
	for coreutils@gnu.org; Tue, 12 Apr 2011 06:15:51 -0400
Received: from mx.meyering.net ([82.230.74.64]:51459)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9adS-000867-Rl
	for coreutils@gnu.org; Tue, 12 Apr 2011 06:15:51 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 3DCBD601F4; Tue, 12 Apr 2011 12:15:45 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] maint: remove unnecessary inclusion of <stdio.h>
Date: Tue, 12 Apr 2011 12:15:45 +0200
Message-ID: <87zknvepwu.fsf@rho.meyering.net>
Lines: 26
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 10:15:52 -0000

FYI,

>From 53ee2679452819d97d599c6fdd3c72a3eeaf7823 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Tue, 12 Apr 2011 12:14:26 +0200
Subject: [PATCH] maint: remove unnecessary inclusion of <stdio.h>

* src/extent-scan.c: Don't include <stdio.h>.  It was not used.
---
 src/extent-scan.c |    1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diff --git a/src/extent-scan.c b/src/extent-scan.c
index e95b085..da7eb9d 100644
--- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -17,7 +17,6 @@
    Written by Jie Liu (jeff.liu@oracle.com).  */

 #include <config.h>
-#include <stdio.h>
 #include <sys/types.h>
 #include <sys/ioctl.h>
 #include <sys/utsname.h>
--
1.7.5.rc1.228.g86d60b


From MAILER-DAEMON Tue Apr 12 06:59:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9bK1-0006Ye-4W
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 06:59:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:58265)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9bJx-0006XG-LD
	for coreutils@gnu.org; Tue, 12 Apr 2011 06:59:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9bJw-0003yO-QI
	for coreutils@gnu.org; Tue, 12 Apr 2011 06:59:45 -0400
Received: from mx.meyering.net ([82.230.74.64]:57606)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9bJw-0003y2-JW
	for coreutils@gnu.org; Tue, 12 Apr 2011 06:59:44 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 518356032C; Tue, 12 Apr 2011 12:59:43 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
In-Reply-To: <4DA4177C.2090403@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Tue, 12 Apr 2011 10:12:28 +0100")
References: <87ei5chcy1.fsf@rho.meyering.net>
	<4D9F9C20.5020709@draigBrady.com> <4DA2E102.5080801@draigBrady.com>
	<878vvgh7nv.fsf@rho.meyering.net> <4DA4177C.2090403@draigBrady.com>
Date: Tue, 12 Apr 2011 12:59:43 +0200
Message-ID: <87tye3envk.fsf@rho.meyering.net>
Lines: 66
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 10:59:47 -0000

P=E1draig Brady wrote:

> On 11/04/11 21:09, Jim Meyering wrote:
>> I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs an=
d xfs.
>> All tests passed on the first three FS types.
>> On xfs there was only one failure: cp/sparse-fiemap.
>> I pared it down to this stand-alone reproducer:
>>
>>     rm -f j1 j2
>>     perl \
>>       -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>     cp --sparse=3Dalways j1 j2
>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>
>> Here's the output.
>> The difference that triggers the test failure is
>> the fact that the "length" numbers differ:
>>
>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>     @@ -1,6 +1,6 @@
>>      Filesystem type is: 58465342
>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>       ext logical physical expected length flags
>>     -   0       1     7310              31
>>     -   1      33     7342     7340     95 eof
>>     -j1: 3 extents found
>>     +   0       1     7469              31
>>     +   1      33     7501     7499     63 eof
>>     +j2: 3 extents found
>
> Hmm, this is with 128K block sizes?

The perl snippet performs 21 pairs of lseek/syswrite calls,
each of length 7KiB.  Not sure about the 3 extents, or why
only the first 4KiB block and one other in the middle of the
file end up being holes on XFS.

Hey, do you feel like extending your fiemap-capable script
to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
which appears to be confused here.

> There is a 4K hole at the start of each block which seems a bit strange.
> Why are "3 extents found" but only 2 listed?
> The last extent seems too long in both cases,
> but that at least wouldn't cause cp to corrupt a copy.

mkfs shows it used 4k blocks:

    $ mkfs -t xfs .blob/xfs
    meta-data=3D.blob/xfs              isize=3D256    agcount=3D4, agsize=
=3D19200 blks
             =3D                       sectsz=3D512   attr=3D2, projid32bit=
=3D0
    data     =3D                       bsize=3D4096   blocks=3D76800, imaxp=
ct=3D25
             =3D                       sunit=3D0      swidth=3D0 blks
    naming   =3Dversion 2              bsize=3D4096   ascii-ci=3D0
    log      =3Dinternal log           bsize=3D4096   blocks=3D1200, versio=
n=3D2
             =3D                       sectsz=3D512   sunit=3D0 blks, lazy-=
count=3D1
    realtime =3Dnone                   extsz=3D4096   blocks=3D0, rtextents=
=3D0


> p.s. I'll s/2.6.38/2.6.39/ in that test

Thanks.


From MAILER-DAEMON Tue Apr 12 07:13:04 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9bWq-0000Zj-2x
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 07:13:04 -0400
Received: from eggs.gnu.org ([140.186.70.92]:58250)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9bWn-0000Yl-Cx
	for coreutils@gnu.org; Tue, 12 Apr 2011 07:13:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9bWl-0007zB-Do
	for coreutils@gnu.org; Tue, 12 Apr 2011 07:13:01 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:18387)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9bWl-0007yG-4D
	for coreutils@gnu.org; Tue, 12 Apr 2011 07:12:59 -0400
Received: (qmail 57283 invoked from network); 12 Apr 2011 11:12:55 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 12 Apr 2011 11:12:55 -0000
Message-ID: <4DA43360.4050903@draigBrady.com>
Date: Tue, 12 Apr 2011 12:11:28 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com> <87tye3envk.fsf@rho.meyering.net>
In-Reply-To: <87tye3envk.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 11:13:02 -0000

On 12/04/11 11:59, Jim Meyering wrote:
> Pádraig Brady wrote:
> 
>> On 11/04/11 21:09, Jim Meyering wrote:
>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs and xfs.
>>> All tests passed on the first three FS types.
>>> On xfs there was only one failure: cp/sparse-fiemap.
>>> I pared it down to this stand-alone reproducer:
>>>
>>>     rm -f j1 j2
>>>     perl \
>>>       -e 'BEGIN{$n=7*1024; *F=*STDOUT}' \
>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>     cp --sparse=always j1 j2
>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>
>>> Here's the output.
>>> The difference that triggers the test failure is
>>> the fact that the "length" numbers differ:
>>>
>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>     @@ -1,6 +1,6 @@
>>>      Filesystem type is: 58465342
>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>       ext logical physical expected length flags
>>>     -   0       1     7310              31
>>>     -   1      33     7342     7340     95 eof
>>>     -j1: 3 extents found
>>>     +   0       1     7469              31
>>>     +   1      33     7501     7499     63 eof
>>>     +j2: 3 extents found
>>
>> Hmm, this is with 128K block sizes?
> 
> The perl snippet performs 21 pairs of lseek/syswrite calls,
> each of length 7KiB.  Not sure about the 3 extents, or why
> only the first 4KiB block and one other in the middle of the
> file end up being holes on XFS.

> Hey, do you feel like extending your fiemap-capable script
> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
> which appears to be confused here.

I'm not sure filefrag is at fault here.
It just shifts and prints the returned lengths.
Could you post the output of this (with the -r option)
http://oss.oracle.com/~mason/fiemap-test.c

>> There is a 4K hole at the start of each block which seems a bit strange.
>> Why are "3 extents found" but only 2 listed?
>> The last extent seems too long in both cases,
>> but that at least wouldn't cause cp to corrupt a copy.
> 
> mkfs shows it used 4k blocks:

Ah OK that explains how it can handle a 4K hole.
The file system must be using a higher level size (128K)
in certain cases, for performance reasons I suppose.

cheers,
Pádraig.


From MAILER-DAEMON Tue Apr 12 08:09:57 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9cPt-0008Un-H8
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 08:09:57 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47296)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9cPq-0008Tk-T3
	for coreutils@gnu.org; Tue, 12 Apr 2011 08:09:56 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9cPp-0005zM-Fe
	for coreutils@gnu.org; Tue, 12 Apr 2011 08:09:54 -0400
Received: from mx.meyering.net ([82.230.74.64]:42788)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9cPo-0005xo-Mc
	for coreutils@gnu.org; Tue, 12 Apr 2011 08:09:53 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id BC5E56030F; Tue, 12 Apr 2011 14:09:50 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
In-Reply-To: <4DA43360.4050903@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Tue, 12 Apr 2011 12:11:28 +0100")
References: <87ei5chcy1.fsf@rho.meyering.net>
	<4D9F9C20.5020709@draigBrady.com> <4DA2E102.5080801@draigBrady.com>
	<878vvgh7nv.fsf@rho.meyering.net> <4DA4177C.2090403@draigBrady.com>
	<87tye3envk.fsf@rho.meyering.net> <4DA43360.4050903@draigBrady.com>
Date: Tue, 12 Apr 2011 14:09:50 +0200
Message-ID: <878vvfekmp.fsf@rho.meyering.net>
Lines: 106
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 12:09:56 -0000

P=E1draig Brady wrote:
> On 12/04/11 11:59, Jim Meyering wrote:
>> P=E1draig Brady wrote:
>>> On 11/04/11 21:09, Jim Meyering wrote:
>>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs =
and xfs.
>>>> All tests passed on the first three FS types.
>>>> On xfs there was only one failure: cp/sparse-fiemap.
>>>> I pared it down to this stand-alone reproducer:
>>>>
>>>>     rm -f j1 j2
>>>>     perl \
>>>>       -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>     cp --sparse=3Dalways j1 j2
>>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>>
>>>> Here's the output.
>>>> The difference that triggers the test failure is
>>>> the fact that the "length" numbers differ:
>>>>
>>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>>     @@ -1,6 +1,6 @@
>>>>      Filesystem type is: 58465342
>>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>>       ext logical physical expected length flags
>>>>     -   0       1     7310              31
>>>>     -   1      33     7342     7340     95 eof
>>>>     -j1: 3 extents found
>>>>     +   0       1     7469              31
>>>>     +   1      33     7501     7499     63 eof
>>>>     +j2: 3 extents found
>>>
>>> Hmm, this is with 128K block sizes?
>>
>> The perl snippet performs 21 pairs of lseek/syswrite calls,
>> each of length 7KiB.  Not sure about the 3 extents, or why
>> only the first 4KiB block and one other in the middle of the
>> file end up being holes on XFS.
>
>> Hey, do you feel like extending your fiemap-capable script
>> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
>> which appears to be confused here.
>
> I'm not sure filefrag is at fault here.
> It just shifts and prints the returned lengths.
> Could you post the output of this (with the -r option)
> http://oss.oracle.com/~mason/fiemap-test.c

Good idea.

  $ gcc -O -Wall fiemap-test.c
  fiemap-test.c: In function 'show_extents_table':
  fiemap-test.c:138:9: warning: variable 'last_logical' set but not used \
    [-Wunused-but-set-variable]

Note that here I'm using 1..31, not 1..21:

  $ rm -f j1 j2; perl -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
    -e 'for (1..31) { sysseek (*F, $n, 1)' \
    -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
  $ cp --sparse=3Dalways j1 j2 && cmp j1 j2

Uh oh.  du reports different sizes:

  $ du -sh j1 j2
  504K    j1
  632K    j2

Not only that, but note how the sizes are *larger*
than raw byte counts.  This means there's a hole at EOF:

  $ wc -c j1 j2
  444416 j1
  444416 j2

Here's the result of running fiemap-test:

  $ {./a.out -r j1; ./a.out -r j2}|sed 's/:   /:/g'
  Extent   0: logical:    4096 physical: 4272128 length:  126976 flags 0x000
  Extent   1: logical:  135168 physical: 4403200 length:  389120 flags 0x001
  Extent   0: logical:    4096 physical: 4792320 length:  126976 flags 0x000
  Extent   1: logical:  135168 physical: 5193728 length:  520192 flags 0x001

No wonder those final "length" numbers differed.
This looks like an XFS bug.

>>> There is a 4K hole at the start of each block which seems a bit strange.
>>> Why are "3 extents found" but only 2 listed?
>>> The last extent seems too long in both cases,
>>> but that at least wouldn't cause cp to corrupt a copy.
>>
>> mkfs shows it used 4k blocks:
>
> Ah OK that explains how it can handle a 4K hole.
> The file system must be using a higher level size (128K)
> in certain cases, for performance reasons I suppose.

I see what you mean.  If you assume XFS holes must be a multiple of 4KiB
in length and are detected only when they start on a 128KiB boundary,
that would explain why the above has only two: one at the beginning,
and the other at the 128K mark (it's 2K into a 7K sequence of NULs).
The 256KiB boundary does fall in a sequence of 0's, but there's 4KiB
before and only 3KiB after that point, so it's too short for a hole.


From MAILER-DAEMON Tue Apr 12 10:06:17 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9eET-00027I-01
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 10:06:17 -0400
Received: from eggs.gnu.org ([140.186.70.92]:32988)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9eEP-00026f-Qh
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:06:15 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9eEO-0003Yr-Dp
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:06:13 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:18981)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9eEO-0003YA-1Y
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:06:12 -0400
Received: (qmail 4561 invoked from network); 12 Apr 2011 14:06:09 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 12 Apr 2011 14:06:09 -0000
Message-ID: <4DA45BF9.3030506@draigBrady.com>
Date: Tue, 12 Apr 2011 15:04:41 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com>	<87tye3envk.fsf@rho.meyering.net>
	<4DA43360.4050903@draigBrady.com> <878vvfekmp.fsf@rho.meyering.net>
In-Reply-To: <878vvfekmp.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 14:06:15 -0000

On 12/04/11 13:09, Jim Meyering wrote:
> Pádraig Brady wrote:
>> On 12/04/11 11:59, Jim Meyering wrote:
>>> Pádraig Brady wrote:
>>>> On 11/04/11 21:09, Jim Meyering wrote:
>>>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs and xfs.
>>>>> All tests passed on the first three FS types.
>>>>> On xfs there was only one failure: cp/sparse-fiemap.
>>>>> I pared it down to this stand-alone reproducer:
>>>>>
>>>>>     rm -f j1 j2
>>>>>     perl \
>>>>>       -e 'BEGIN{$n=7*1024; *F=*STDOUT}' \
>>>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>>     cp --sparse=always j1 j2
>>>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>>>
>>>>> Here's the output.
>>>>> The difference that triggers the test failure is
>>>>> the fact that the "length" numbers differ:
>>>>>
>>>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>>>     @@ -1,6 +1,6 @@
>>>>>      Filesystem type is: 58465342
>>>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>>>       ext logical physical expected length flags
>>>>>     -   0       1     7310              31
>>>>>     -   1      33     7342     7340     95 eof
>>>>>     -j1: 3 extents found
>>>>>     +   0       1     7469              31
>>>>>     +   1      33     7501     7499     63 eof
>>>>>     +j2: 3 extents found
>>>>
>>>> Hmm, this is with 128K block sizes?
>>>
>>> The perl snippet performs 21 pairs of lseek/syswrite calls,
>>> each of length 7KiB.  Not sure about the 3 extents, or why
>>> only the first 4KiB block and one other in the middle of the
>>> file end up being holes on XFS.
>>
>>> Hey, do you feel like extending your fiemap-capable script
>>> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
>>> which appears to be confused here.
>>
>> I'm not sure filefrag is at fault here.
>> It just shifts and prints the returned lengths.
>> Could you post the output of this (with the -r option)
>> http://oss.oracle.com/~mason/fiemap-test.c
> 
> Good idea.
> 
>   $ gcc -O -Wall fiemap-test.c
>   fiemap-test.c: In function 'show_extents_table':
>   fiemap-test.c:138:9: warning: variable 'last_logical' set but not used \
>     [-Wunused-but-set-variable]
> 
> Note that here I'm using 1..31, not 1..21:
> 
>   $ rm -f j1 j2; perl -e 'BEGIN{$n=7*1024; *F=*STDOUT}' \
>     -e 'for (1..31) { sysseek (*F, $n, 1)' \
>     -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>   $ cp --sparse=always j1 j2 && cmp j1 j2
> 
> Uh oh.  du reports different sizes:
> 
>   $ du -sh j1 j2
>   504K    j1
>   632K    j2
> 
> Not only that, but note how the sizes are *larger*
> than raw byte counts.  This means there's a hole at EOF:
> 
>   $ wc -c j1 j2
>   444416 j1
>   444416 j2
> 
> Here's the result of running fiemap-test:
> 
>   $ {./a.out -r j1; ./a.out -r j2}|sed 's/:   /:/g'
>   Extent   0: logical:    4096 physical: 4272128 length:  126976 flags 0x000
>   Extent   1: logical:  135168 physical: 4403200 length:  389120 flags 0x001
>   Extent   0: logical:    4096 physical: 4792320 length:  126976 flags 0x000
>   Extent   1: logical:  135168 physical: 5193728 length:  520192 flags 0x001
> 
> No wonder those final "length" numbers differed.
> This looks like an XFS bug.

Well not necessarily.
I suppose that XFS may be allocating the extents differently
in the destination for performance reasons.
I.E. It's laying out the 128KiB chunks first and not
trimming trailing slop.

So if that was the case, we'd have to get our fiemap comparison
to discount anything after a specified length.

cheers,
Pádraig.


From MAILER-DAEMON Tue Apr 12 10:35:44 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9egy-00052m-Ek
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 10:35:44 -0400
Received: from eggs.gnu.org ([140.186.70.92]:49355)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9egw-00050r-6i
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:35:43 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9egq-0004DE-OR
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:35:42 -0400
Received: from mx1.redhat.com ([209.132.183.28]:22514)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9egq-0004Cp-F9
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:35:36 -0400
Received: from int-mx02.intmail.prod.int.phx2.redhat.com
	(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3CEZUUY030876
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Tue, 12 Apr 2011 10:35:31 -0400
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com
	[10.5.9.1])
	by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP
	id p3CEZS7f008570
	(version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO);
	Tue, 12 Apr 2011 10:35:29 -0400
Message-ID: <4DA46330.9030306@redhat.com>
Date: Tue, 12 Apr 2011 09:35:28 -0500
From: Eric Sandeen <sandeen@redhat.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com>	<87tye3envk.fsf@rho.meyering.net>
	<4DA43360.4050903@draigBrady.com> <878vvfekmp.fsf@rho.meyering.net>
	<4DA45BF9.3030506@draigBrady.com>
In-Reply-To: <4DA45BF9.3030506@draigBrady.com>
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	p3CEZUUY030876
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 14:35:43 -0000

On 4/12/11 9:04 AM, P=E1draig Brady wrote:
> On 12/04/11 13:09, Jim Meyering wrote:
>> P=E1draig Brady wrote:
>>> On 12/04/11 11:59, Jim Meyering wrote:
>>>> P=E1draig Brady wrote:
>>>>> On 11/04/11 21:09, Jim Meyering wrote:
>>>>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, bt=
rfs and xfs.
>>>>>> All tests passed on the first three FS types.
>>>>>> On xfs there was only one failure: cp/sparse-fiemap.
>>>>>> I pared it down to this stand-alone reproducer:
>>>>>>
>>>>>>     rm -f j1 j2
>>>>>>     perl \
>>>>>>       -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>>>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>>>     cp --sparse=3Dalways j1 j2
>>>>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>>>>
>>>>>> Here's the output.
>>>>>> The difference that triggers the test failure is
>>>>>> the fact that the "length" numbers differ:
>>>>>>
>>>>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>>>>     @@ -1,6 +1,6 @@
>>>>>>      Filesystem type is: 58465342
>>>>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>>>>       ext logical physical expected length flags
>>>>>>     -   0       1     7310              31
>>>>>>     -   1      33     7342     7340     95 eof
>>>>>>     -j1: 3 extents found
>>>>>>     +   0       1     7469              31
>>>>>>     +   1      33     7501     7499     63 eof
>>>>>>     +j2: 3 extents found
>>>>>
>>>>> Hmm, this is with 128K block sizes?
>>>>
>>>> The perl snippet performs 21 pairs of lseek/syswrite calls,
>>>> each of length 7KiB.  Not sure about the 3 extents, or why
>>>> only the first 4KiB block and one other in the middle of the
>>>> file end up being holes on XFS.
>>>
>>>> Hey, do you feel like extending your fiemap-capable script
>>>> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
>>>> which appears to be confused here.
>>>
>>> I'm not sure filefrag is at fault here.
>>> It just shifts and prints the returned lengths.
>>> Could you post the output of this (with the -r option)
>>> http://oss.oracle.com/~mason/fiemap-test.c
>>
>> Good idea.
>>
>>   $ gcc -O -Wall fiemap-test.c
>>   fiemap-test.c: In function 'show_extents_table':
>>   fiemap-test.c:138:9: warning: variable 'last_logical' set but not us=
ed \
>>     [-Wunused-but-set-variable]
>>
>> Note that here I'm using 1..31, not 1..21:
>>
>>   $ rm -f j1 j2; perl -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>     -e 'for (1..31) { sysseek (*F, $n, 1)' \
>>     -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>   $ cp --sparse=3Dalways j1 j2 && cmp j1 j2
>>
>> Uh oh.  du reports different sizes:
>>
>>   $ du -sh j1 j2
>>   504K    j1
>>   632K    j2
>>
>> Not only that, but note how the sizes are *larger*
>> than raw byte counts.  This means there's a hole at EOF:
>>
>>   $ wc -c j1 j2
>>   444416 j1
>>   444416 j2
>>
>> Here's the result of running fiemap-test:
>>
>>   $ {./a.out -r j1; ./a.out -r j2}|sed 's/:   /:/g'
>>   Extent   0: logical:    4096 physical: 4272128 length:  126976 flags=
 0x000
>>   Extent   1: logical:  135168 physical: 4403200 length:  389120 flags=
 0x001
>>   Extent   0: logical:    4096 physical: 4792320 length:  126976 flags=
 0x000
>>   Extent   1: logical:  135168 physical: 5193728 length:  520192 flags=
 0x001
>>
>> No wonder those final "length" numbers differed.
>> This looks like an XFS bug.
>=20
> Well not necessarily.
> I suppose that XFS may be allocating the extents differently
> in the destination for performance reasons.
> I.E. It's laying out the 128KiB chunks first and not
> trimming trailing slop.
>=20
> So if that was the case, we'd have to get our fiemap comparison
> to discount anything after a specified length.

I'd say don't get too fancy.

The only thing the fileystem needs to get right on the copy is file size =
and contents.
If those match after a copy, the test should pass.

How it allocates blocks is its own business, and can change at any time.
Trying to reverse engineer the allocator at any point in time will surely=
 lead to more pain.  :)

Giving the fs hints about holes, and doing seek/write/seek/write around t=
hose holes during the copy is great; it's less IO to do, etc, but the fil=
esystem will do as it pleases with that info.

-Eric

> cheers,
> P=E1draig.



From MAILER-DAEMON Tue Apr 12 10:49:38 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9euQ-0003Gk-Qv
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 10:49:38 -0400
Received: from eggs.gnu.org ([140.186.70.92]:50061)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9euN-0003Ff-9S
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:49:36 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9euL-0007pQ-LM
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:49:35 -0400
Received: from mx1.redhat.com ([209.132.183.28]:50393)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9euL-0007pD-8X
	for coreutils@gnu.org; Tue, 12 Apr 2011 10:49:33 -0400
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3CEnToH018203
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Tue, 12 Apr 2011 10:49:29 -0400
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com
	[10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p3CEnRY1008803
	(version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
	Tue, 12 Apr 2011 10:49:29 -0400
Message-ID: <4DA46677.7090203@redhat.com>
Date: Tue, 12 Apr 2011 09:49:27 -0500
From: Eric Sandeen <sandeen@redhat.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com>	<87tye3envk.fsf@rho.meyering.net>
	<4DA43360.4050903@draigBrady.com> <878vvfekmp.fsf@rho.meyering.net>
In-Reply-To: <878vvfekmp.fsf@rho.meyering.net>
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	p3CEnToH018203
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 12 Apr 2011 14:49:36 -0000

On 4/12/11 7:09 AM, Jim Meyering wrote:
> P=E1draig Brady wrote:
>> On 12/04/11 11:59, Jim Meyering wrote:
>>> P=E1draig Brady wrote:
>>>> On 11/04/11 21:09, Jim Meyering wrote:
>>>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, btr=
fs and xfs.
>>>>> All tests passed on the first three FS types.
>>>>> On xfs there was only one failure: cp/sparse-fiemap.
>>>>> I pared it down to this stand-alone reproducer:
>>>>>
>>>>>     rm -f j1 j2
>>>>>     perl \
>>>>>       -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>>     cp --sparse=3Dalways j1 j2
>>>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>>>
>>>>> Here's the output.
>>>>> The difference that triggers the test failure is
>>>>> the fact that the "length" numbers differ:
>>>>>
>>>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>>>     @@ -1,6 +1,6 @@
>>>>>      Filesystem type is: 58465342
>>>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>>>       ext logical physical expected length flags
>>>>>     -   0       1     7310              31
>>>>>     -   1      33     7342     7340     95 eof
>>>>>     -j1: 3 extents found
>>>>>     +   0       1     7469              31
>>>>>     +   1      33     7501     7499     63 eof
>>>>>     +j2: 3 extents found
>>>>
>>>> Hmm, this is with 128K block sizes?
>>>
>>> The perl snippet performs 21 pairs of lseek/syswrite calls,
>>> each of length 7KiB.  Not sure about the 3 extents, or why
>>> only the first 4KiB block and one other in the middle of the
>>> file end up being holes on XFS.
>>
>>> Hey, do you feel like extending your fiemap-capable script
>>> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
>>> which appears to be confused here.
>>
>> I'm not sure filefrag is at fault here.
>> It just shifts and prints the returned lengths.
>> Could you post the output of this (with the -r option)
>> http://oss.oracle.com/~mason/fiemap-test.c
>=20
> Good idea.

Heh, I actually wrote that as a quickie.  If you look at it, don't laugh!

>   $ gcc -O -Wall fiemap-test.c
>   fiemap-test.c: In function 'show_extents_table':
>   fiemap-test.c:138:9: warning: variable 'last_logical' set but not use=
d \
>     [-Wunused-but-set-variable]
>=20
> Note that here I'm using 1..31, not 1..21:
>=20
>   $ rm -f j1 j2; perl -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>     -e 'for (1..31) { sysseek (*F, $n, 1)' \
>     -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>   $ cp --sparse=3Dalways j1 j2 && cmp j1 j2
>=20
> Uh oh.  du reports different sizes:
>=20
>   $ du -sh j1 j2
>   504K    j1
>   632K    j2

so 128k more space (32 4k blocks) allocated in j2.

But that's fine.  A filesystem can allocate as it wishes, for efficiency,=
 and that's just what xfs is doing.

http://oss.sgi.com/bugzilla/show_bug.cgi?id=3D822

> Not only that, but note how the sizes are *larger*
> than raw byte counts.  This means there's a hole at EOF:
>=20
>   $ wc -c j1 j2
>   444416 j1
>   444416 j2

I'm not sure I get the "hole at EOF" conclusion...?

> Here's the result of running fiemap-test:
>=20
>   $ {./a.out -r j1; ./a.out -r j2}|sed 's/:   /:/g'
>   Extent   0: logical:    4096 physical: 4272128 length:  126976 flags =
0x000
>   Extent   1: logical:  135168 physical: 4403200 length:  389120 flags =
0x001
>   Extent   0: logical:    4096 physical: 4792320 length:  126976 flags =
0x000
>   Extent   1: logical:  135168 physical: 5193728 length:  520192 flags =
0x001
>=20
> No wonder those final "length" numbers differed.

bugs in that tester are also possible.  :(  I need to do a cleanroom mapp=
ing tool from scratch and send it to util-linux-ng, I think.

> This looks like an XFS bug.

so the 2nd/last extent has 128k more length, which matches the change in =
disk space usage.

but the last extent's logical start + length doesn't match the file size =
in either case does it?

Still not convinced of an xfs bug ;)

but I need to get an F15/rawhide system and just poke at this stuff mysel=
f a bit to be sure of what's going on.

So far I see unique allocator behavior, but not a bug.

>>>> There is a 4K hole at the start of each block which seems a bit stra=
nge.
>>>> Why are "3 extents found" but only 2 listed?
>>>> The last extent seems too long in both cases,
>>>> but that at least wouldn't cause cp to corrupt a copy.
>>>
>>> mkfs shows it used 4k blocks:
>>
>> Ah OK that explains how it can handle a 4K hole.
>> The file system must be using a higher level size (128K)
>> in certain cases, for performance reasons I suppose.

That's about right - it's not really "using 128k" but there is a speculat=
ive EOF allocation in xfs, so extending the file by hole/data/hole/data, =
I think the holes may get soaked up by that speculative allocation past E=
OF.

> I see what you mean.  If you assume XFS holes must be a multiple of 4Ki=
B

they must be a multiple of the fs block size; true of any filesystem.

> in length and are detected only when they start on a 128KiB boundary,

128k is just how it came out this time.

As I said in the other reply, don't try to reverse engineer fs allocator =
decisions in your tests...

-Eric

> that would explain why the above has only two: one at the beginning,
> and the other at the 128K mark (it's 2K into a 7K sequence of NULs).
> The 256KiB boundary does fall in a sequence of 0's, but there's 4KiB
> before and only 3KiB after that point, so it's too short for a hole.



From MAILER-DAEMON Tue Apr 12 21:34:43 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9oyh-0003H7-RL
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 21:34:43 -0400
Received: from eggs.gnu.org ([140.186.70.92]:45336)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9oye-0002OS-N2
	for coreutils@gnu.org; Tue, 12 Apr 2011 21:34:42 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9oW4-0007HI-Gw
	for coreutils@gnu.org; Tue, 12 Apr 2011 21:05:09 -0400
Received: from mx1.redhat.com ([209.132.183.28]:26061)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@redhat.com>) id 1Q9oW4-0007Eg-5r
	for coreutils@gnu.org; Tue, 12 Apr 2011 21:05:08 -0400
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3D153NM014644
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Tue, 12 Apr 2011 21:05:03 -0400
Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com
	[10.5.9.1])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p3D150Il021664
	(version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO);
	Tue, 12 Apr 2011 21:05:01 -0400
Message-ID: <4DA4F6BC.9080908@redhat.com>
Date: Tue, 12 Apr 2011 20:05:00 -0500
From: Eric Sandeen <sandeen@redhat.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com>	<87tye3envk.fsf@rho.meyering.net>
	<4DA43360.4050903@draigBrady.com> <878vvfekmp.fsf@rho.meyering.net>
	<4DA45BF9.3030506@draigBrady.com> <4DA46330.9030306@redhat.com>
	<4DA4E99A.3030102@draigBrady.com>
In-Reply-To: <4DA4E99A.3030102@draigBrady.com>
Content-Type: text/plain; charset=ISO-8859-1
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id
	p3D153NM014644
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 01:34:42 -0000

On 4/12/11 7:08 PM, P=E1draig Brady wrote:
> On 12/04/11 15:35, Eric Sandeen wrote:
>> On 4/12/11 9:04 AM, P=E1draig Brady wrote:
>>> On 12/04/11 13:09, Jim Meyering wrote:
>>>> P=E1draig Brady wrote:
>>>>> On 12/04/11 11:59, Jim Meyering wrote:
>>>>>> P=E1draig Brady wrote:
>>>>>>> On 11/04/11 21:09, Jim Meyering wrote:
>>>>>>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, =
btrfs and xfs.
>>>>>>>> All tests passed on the first three FS types.
>>>>>>>> On xfs there was only one failure: cp/sparse-fiemap.
>>>>>>>> I pared it down to this stand-alone reproducer:
>>>>>>>>
>>>>>>>>     rm -f j1 j2
>>>>>>>>     perl \
>>>>>>>>       -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>>>>>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>>>>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>>>>>     cp --sparse=3Dalways j1 j2
>>>>>>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>>>>>>
>>>>>>>> Here's the output.
>>>>>>>> The difference that triggers the test failure is
>>>>>>>> the fact that the "length" numbers differ:
>>>>>>>>
>>>>>>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>>>>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>>>>>>     @@ -1,6 +1,6 @@
>>>>>>>>      Filesystem type is: 58465342
>>>>>>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>>>>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>>>>>>       ext logical physical expected length flags
>>>>>>>>     -   0       1     7310              31
>>>>>>>>     -   1      33     7342     7340     95 eof
>>>>>>>>     -j1: 3 extents found
>>>>>>>>     +   0       1     7469              31
>>>>>>>>     +   1      33     7501     7499     63 eof
>>>>>>>>     +j2: 3 extents found
>>>>>>>
>>>>>>> Hmm, this is with 128K block sizes?
>>>>>>
>>>>>> The perl snippet performs 21 pairs of lseek/syswrite calls,
>>>>>> each of length 7KiB.  Not sure about the 3 extents, or why
>>>>>> only the first 4KiB block and one other in the middle of the
>>>>>> file end up being holes on XFS.
>>>>>
>>>>>> Hey, do you feel like extending your fiemap-capable script
>>>>>> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
>>>>>> which appears to be confused here.
>>>>>
>>>>> I'm not sure filefrag is at fault here.
>>>>> It just shifts and prints the returned lengths.
>>>>> Could you post the output of this (with the -r option)
>>>>> http://oss.oracle.com/~mason/fiemap-test.c
>>>>
>>>> Good idea.
>>>>
>>>>   $ gcc -O -Wall fiemap-test.c
>>>>   fiemap-test.c: In function 'show_extents_table':
>>>>   fiemap-test.c:138:9: warning: variable 'last_logical' set but not =
used \
>>>>     [-Wunused-but-set-variable]
>>>>
>>>> Note that here I'm using 1..31, not 1..21:
>>>>
>>>>   $ rm -f j1 j2; perl -e 'BEGIN{$n=3D7*1024; *F=3D*STDOUT}' \
>>>>     -e 'for (1..31) { sysseek (*F, $n, 1)' \
>>>>     -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>   $ cp --sparse=3Dalways j1 j2 && cmp j1 j2
>>>>
>>>> Uh oh.  du reports different sizes:
>>>>
>>>>   $ du -sh j1 j2
>>>>   504K    j1
>>>>   632K    j2
>>>>
>>>> Not only that, but note how the sizes are *larger*
>>>> than raw byte counts.  This means there's a hole at EOF:
>>>>
>>>>   $ wc -c j1 j2
>>>>   444416 j1
>>>>   444416 j2
>>>>
>>>> Here's the result of running fiemap-test:
>>>>
>>>>   $ {./a.out -r j1; ./a.out -r j2}|sed 's/:   /:/g'
>>>>   Extent   0: logical:    4096 physical: 4272128 length:  126976 fla=
gs 0x000
>>>>   Extent   1: logical:  135168 physical: 4403200 length:  389120 fla=
gs 0x001
>>>>   Extent   0: logical:    4096 physical: 4792320 length:  126976 fla=
gs 0x000
>>>>   Extent   1: logical:  135168 physical: 5193728 length:  520192 fla=
gs 0x001
>>>>
>>>> No wonder those final "length" numbers differed.
>>>> This looks like an XFS bug.
>>>
>>> Well not necessarily.
>>> I suppose that XFS may be allocating the extents differently
>>> in the destination for performance reasons.
>>> I.E. It's laying out the 128KiB chunks first and not
>>> trimming trailing slop.
>>>
>>> So if that was the case, we'd have to get our fiemap comparison
>>> to discount anything after a specified length.
>>
>> I'd say don't get too fancy.
>>
>> The only thing the fileystem needs to get right on the copy is file si=
ze and contents.
>> If those match after a copy, the test should pass.
>=20
> Right. However, we'd like to verify that holes
> are being propagated from source to dest,
> and a fiemap list is the only way I know to test that.

But unfortunately the filesystem is not actually required to put a hole a=
nywhere...

> Also it's unlikely I think that holes would be
> introduced differently in the dest, given the
> processing that cp does.

Depends on what writeback is doing during the test, etc, and probably man=
y other things.

> So I'm leaning towards relaxing the test,
> by ignoring the length of the last extent
> along the lines of the following.

I think it's fragile because the fs can change behavior at will.  But it'=
s your testcase. :)

-Eric

> cheers,
> P=E1draig.
>=20
> diff --git a/tests/filefrag-extent-compare b/tests/filefrag-extent-comp=
are
> index 2c33584..c8b840c 100644
> --- a/tests/filefrag-extent-compare
> +++ b/tests/filefrag-extent-compare
> @@ -56,11 +56,24 @@ merge_extents \@b;
>  my $i =3D 0;
>  while (defined $a[$i])
>    {
> -    $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK} && $a[$i]->{LEN} =3D=3D $b[=
$i]->{LEN}
> -      or die "$ME: differing extent:\n"
> -        . "  [$i]=3D$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
> -          . "  [$i]=3D$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
> -    $i++;
> +    if ($a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK} && $a[$i]->{LEN} =3D=3D=
 $b[$i]->{LEN})
> +      {
> +        $i++;
> +      }
> +    elsif ($i =3D=3D (@a - 1) && $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK=
})
> +      {
> +        # On XFS it was seen that the size of the last extent can vary=
,
> +        # and can extend beyond the length of the file.
> +        # So ignore the length of the last extent, because if the
> +        # file is the wrong length we'll get failures elsewhere.
> +        last;
> +      }
> +    else
> +      {
> +        die "$ME: differing extent:\n"
> +          . "  [$i]=3D$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
> +            . "  [$i]=3D$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
> +      }
>    }
>=20
>  ### Setup "GNU" style for perl-mode and cperl-mode.



From MAILER-DAEMON Tue Apr 12 21:38:01 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9p1t-0000GI-9f
	for mharc-coreutils@gnu.org; Tue, 12 Apr 2011 21:38:01 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47091)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9p1m-0002no-BF
	for coreutils@gnu.org; Tue, 12 Apr 2011 21:38:00 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9nfD-0007GX-CW
	for coreutils@gnu.org; Tue, 12 Apr 2011 20:10:33 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:15644)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9nfD-0007CZ-0d
	for coreutils@gnu.org; Tue, 12 Apr 2011 20:10:31 -0400
Received: (qmail 22870 invoked from network); 13 Apr 2011 00:10:29 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 13 Apr 2011 00:10:29 -0000
Message-ID: <4DA4E99A.3030102@draigBrady.com>
Date: Wed, 13 Apr 2011 01:08:58 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Eric Sandeen <sandeen@redhat.com>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com>	<87tye3envk.fsf@rho.meyering.net>
	<4DA43360.4050903@draigBrady.com> <878vvfekmp.fsf@rho.meyering.net>
	<4DA45BF9.3030506@draigBrady.com> <4DA46330.9030306@redhat.com>
In-Reply-To: <4DA46330.9030306@redhat.com>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 01:38:00 -0000

On 12/04/11 15:35, Eric Sandeen wrote:
> On 4/12/11 9:04 AM, Pádraig Brady wrote:
>> On 12/04/11 13:09, Jim Meyering wrote:
>>> Pádraig Brady wrote:
>>>> On 12/04/11 11:59, Jim Meyering wrote:
>>>>> Pádraig Brady wrote:
>>>>>> On 11/04/11 21:09, Jim Meyering wrote:
>>>>>>> I've run the test suite on F15 x86_64 using each of ext3, ext4, btrfs and xfs.
>>>>>>> All tests passed on the first three FS types.
>>>>>>> On xfs there was only one failure: cp/sparse-fiemap.
>>>>>>> I pared it down to this stand-alone reproducer:
>>>>>>>
>>>>>>>     rm -f j1 j2
>>>>>>>     perl \
>>>>>>>       -e 'BEGIN{$n=7*1024; *F=*STDOUT}' \
>>>>>>>       -e 'for (1..21) { sysseek (*F, $n, 1)' \
>>>>>>>       -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>>>>>     cp --sparse=always j1 j2
>>>>>>>     diff -u <(filefrag -v j1) <(filefrag -vs j2)
>>>>>>>
>>>>>>> Here's the output.
>>>>>>> The difference that triggers the test failure is
>>>>>>> the fact that the "length" numbers differ:
>>>>>>>
>>>>>>>     --- /proc/self/fd/11    2011-04-11 22:01:00.932737472 +0200
>>>>>>>     +++ /proc/self/fd/12    2011-04-11 22:01:00.932737472 +0200
>>>>>>>     @@ -1,6 +1,6 @@
>>>>>>>      Filesystem type is: 58465342
>>>>>>>     -File size of j1 is 301056 (74 blocks, blocksize 4096)
>>>>>>>     +File size of j2 is 301056 (74 blocks, blocksize 4096)
>>>>>>>       ext logical physical expected length flags
>>>>>>>     -   0       1     7310              31
>>>>>>>     -   1      33     7342     7340     95 eof
>>>>>>>     -j1: 3 extents found
>>>>>>>     +   0       1     7469              31
>>>>>>>     +   1      33     7501     7499     63 eof
>>>>>>>     +j2: 3 extents found
>>>>>>
>>>>>> Hmm, this is with 128K block sizes?
>>>>>
>>>>> The perl snippet performs 21 pairs of lseek/syswrite calls,
>>>>> each of length 7KiB.  Not sure about the 3 extents, or why
>>>>> only the first 4KiB block and one other in the middle of the
>>>>> file end up being holes on XFS.
>>>>
>>>>> Hey, do you feel like extending your fiemap-capable script
>>>>> to print FIEMAP data?  Then we wouldn't have to rely on filefrag,
>>>>> which appears to be confused here.
>>>>
>>>> I'm not sure filefrag is at fault here.
>>>> It just shifts and prints the returned lengths.
>>>> Could you post the output of this (with the -r option)
>>>> http://oss.oracle.com/~mason/fiemap-test.c
>>>
>>> Good idea.
>>>
>>>   $ gcc -O -Wall fiemap-test.c
>>>   fiemap-test.c: In function 'show_extents_table':
>>>   fiemap-test.c:138:9: warning: variable 'last_logical' set but not used \
>>>     [-Wunused-but-set-variable]
>>>
>>> Note that here I'm using 1..31, not 1..21:
>>>
>>>   $ rm -f j1 j2; perl -e 'BEGIN{$n=7*1024; *F=*STDOUT}' \
>>>     -e 'for (1..31) { sysseek (*F, $n, 1)' \
>>>     -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1
>>>   $ cp --sparse=always j1 j2 && cmp j1 j2
>>>
>>> Uh oh.  du reports different sizes:
>>>
>>>   $ du -sh j1 j2
>>>   504K    j1
>>>   632K    j2
>>>
>>> Not only that, but note how the sizes are *larger*
>>> than raw byte counts.  This means there's a hole at EOF:
>>>
>>>   $ wc -c j1 j2
>>>   444416 j1
>>>   444416 j2
>>>
>>> Here's the result of running fiemap-test:
>>>
>>>   $ {./a.out -r j1; ./a.out -r j2}|sed 's/:   /:/g'
>>>   Extent   0: logical:    4096 physical: 4272128 length:  126976 flags 0x000
>>>   Extent   1: logical:  135168 physical: 4403200 length:  389120 flags 0x001
>>>   Extent   0: logical:    4096 physical: 4792320 length:  126976 flags 0x000
>>>   Extent   1: logical:  135168 physical: 5193728 length:  520192 flags 0x001
>>>
>>> No wonder those final "length" numbers differed.
>>> This looks like an XFS bug.
>>
>> Well not necessarily.
>> I suppose that XFS may be allocating the extents differently
>> in the destination for performance reasons.
>> I.E. It's laying out the 128KiB chunks first and not
>> trimming trailing slop.
>>
>> So if that was the case, we'd have to get our fiemap comparison
>> to discount anything after a specified length.
> 
> I'd say don't get too fancy.
> 
> The only thing the fileystem needs to get right on the copy is file size and contents.
> If those match after a copy, the test should pass.

Right. However, we'd like to verify that holes
are being propagated from source to dest,
and a fiemap list is the only way I know to test that.

Also it's unlikely I think that holes would be
introduced differently in the dest, given the
processing that cp does.

So I'm leaning towards relaxing the test,
by ignoring the length of the last extent
along the lines of the following.

cheers,
Pádraig.

diff --git a/tests/filefrag-extent-compare b/tests/filefrag-extent-compare
index 2c33584..c8b840c 100644
--- a/tests/filefrag-extent-compare
+++ b/tests/filefrag-extent-compare
@@ -56,11 +56,24 @@ merge_extents \@b;
 my $i = 0;
 while (defined $a[$i])
   {
-    $a[$i]->{L_BLK} == $b[$i]->{L_BLK} && $a[$i]->{LEN} == $b[$i]->{LEN}
-      or die "$ME: differing extent:\n"
-        . "  [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
-          . "  [$i]=$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
-    $i++;
+    if ($a[$i]->{L_BLK} == $b[$i]->{L_BLK} && $a[$i]->{LEN} == $b[$i]->{LEN})
+      {
+        $i++;
+      }
+    elsif ($i == (@a - 1) && $a[$i]->{L_BLK} == $b[$i]->{L_BLK})
+      {
+        # On XFS it was seen that the size of the last extent can vary,
+        # and can extend beyond the length of the file.
+        # So ignore the length of the last extent, because if the
+        # file is the wrong length we'll get failures elsewhere.
+        last;
+      }
+    else
+      {
+        die "$ME: differing extent:\n"
+          . "  [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
+            . "  [$i]=$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
+      }
   }

 ### Setup "GNU" style for perl-mode and cperl-mode.


From MAILER-DAEMON Wed Apr 13 04:19:07 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9vI3-000244-8O
	for mharc-coreutils@gnu.org; Wed, 13 Apr 2011 04:19:07 -0400
Received: from eggs.gnu.org ([140.186.70.92]:42604)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9vHv-00021V-VN
	for coreutils@gnu.org; Wed, 13 Apr 2011 04:19:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9vHq-0000g2-Cc
	for coreutils@gnu.org; Wed, 13 Apr 2011 04:18:59 -0400
Received: from mx.meyering.net ([82.230.74.64]:36927)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9vHq-0000fr-5L
	for coreutils@gnu.org; Wed, 13 Apr 2011 04:18:54 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id A238D601A1; Wed, 13 Apr 2011 10:18:52 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
In-Reply-To: <4DA4E99A.3030102@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Wed, 13 Apr 2011 01:08:58 +0100")
References: <87ei5chcy1.fsf@rho.meyering.net>
	<4D9F9C20.5020709@draigBrady.com> <4DA2E102.5080801@draigBrady.com>
	<878vvgh7nv.fsf@rho.meyering.net> <4DA4177C.2090403@draigBrady.com>
	<87tye3envk.fsf@rho.meyering.net> <4DA43360.4050903@draigBrady.com>
	<878vvfekmp.fsf@rho.meyering.net> <4DA45BF9.3030506@draigBrady.com>
	<4DA46330.9030306@redhat.com> <4DA4E99A.3030102@draigBrady.com>
Date: Wed, 13 Apr 2011 10:18:52 +0200
Message-ID: <87bp0abm37.fsf@rho.meyering.net>
Lines: 63
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 08:19:05 -0000

P=E1draig Brady wrote:
...
> Right. However, we'd like to verify that holes
> are being propagated from source to dest,
> and a fiemap list is the only way I know to test that.
>
> Also it's unlikely I think that holes would be
> introduced differently in the dest, given the
> processing that cp does.
>
> So I'm leaning towards relaxing the test,
> by ignoring the length of the last extent
> along the lines of the following.

That works for me.
However, I find it a little too hard to visually
match the two occurrences of this expression,

    $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK}

so how about factoring it for readability?

    my $block_match =3D $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK};

also, please mention the kernel version
on which we saw this XFS behavior.

Thanks!

> diff --git a/tests/filefrag-extent-compare b/tests/filefrag-extent-compare
> index 2c33584..c8b840c 100644
> --- a/tests/filefrag-extent-compare
> +++ b/tests/filefrag-extent-compare
> @@ -56,11 +56,24 @@ merge_extents \@b;
>  my $i =3D 0;
>  while (defined $a[$i])
>    {
> -    $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK} && $a[$i]->{LEN} =3D=3D $b[$i=
]->{LEN}
> -      or die "$ME: differing extent:\n"
> -        . "  [$i]=3D$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
> -          . "  [$i]=3D$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
> -    $i++;
> +    if ($a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK} && $a[$i]->{LEN} =3D=3D $=
b[$i]->{LEN})
> +      {
> +        $i++;
> +      }
> +    elsif ($i =3D=3D (@a - 1) && $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK})
> +      {
> +        # On XFS it was seen that the size of the last extent can vary,
> +        # and can extend beyond the length of the file.
> +        # So ignore the length of the last extent, because if the
> +        # file is the wrong length we'll get failures elsewhere.
> +        last;
> +      }
> +    else
> +      {
> +        die "$ME: differing extent:\n"
> +          . "  [$i]=3D$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
> +            . "  [$i]=3D$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
> +      }
>    }
>
>  ### Setup "GNU" style for perl-mode and cperl-mode.


From MAILER-DAEMON Wed Apr 13 04:57:22 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9vt4-0006Fn-9O
	for mharc-coreutils@gnu.org; Wed, 13 Apr 2011 04:57:22 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47897)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9vsx-0006Dl-6v
	for coreutils@gnu.org; Wed, 13 Apr 2011 04:57:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9vsq-0002aA-BG
	for coreutils@gnu.org; Wed, 13 Apr 2011 04:57:15 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:14303)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1Q9vsq-0002Zg-16
	for coreutils@gnu.org; Wed, 13 Apr 2011 04:57:08 -0400
Received: (qmail 85022 invoked from network); 13 Apr 2011 08:57:05 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 13 Apr 2011 08:57:05 -0000
Message-ID: <4DA56505.9070602@draigBrady.com>
Date: Wed, 13 Apr 2011 09:55:33 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Eric Sandeen <sandeen@redhat.com>
Subject: Re: ready for release of coreutils-8.11?
References: <87ei5chcy1.fsf@rho.meyering.net>	<4D9F9C20.5020709@draigBrady.com>
	<4DA2E102.5080801@draigBrady.com>	<878vvgh7nv.fsf@rho.meyering.net>
	<4DA4177C.2090403@draigBrady.com>	<87tye3envk.fsf@rho.meyering.net>
	<4DA43360.4050903@draigBrady.com> <878vvfekmp.fsf@rho.meyering.net>
	<4DA45BF9.3030506@draigBrady.com> <4DA46330.9030306@redhat.com>
	<4DA4E99A.3030102@draigBrady.com> <4DA4F6BC.9080908@redhat.com>
In-Reply-To: <4DA4F6BC.9080908@redhat.com>
X-Enigmail-Version: 1.0.1
Content-Type: multipart/mixed; boundary="------------060809010908000709080301"
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 08:57:20 -0000

This is a multi-part message in MIME format.
--------------060809010908000709080301
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit

On 13/04/11 02:05, Eric Sandeen wrote:
> On 4/12/11 7:08 PM, Pádraig Brady wrote:
>> Right. However, we'd like to verify that holes
>> are being propagated from source to dest,
>> and a fiemap list is the only way I know to test that.
> 
> But unfortunately the filesystem is not actually required to put a hole anywhere...

Sure I understand that, but we've not seen that yet in our tests.
It would be kinda interesting to verify this possibility too :)
So I'll leave the somewhat fragile test in place for now
and we can change it to a dev only test in future if it causes issues.

I'll apply the attached soon.

thanks a lot for the feedback!
Pádraig.

--------------060809010908000709080301
Content-Type: text/x-patch;
 name="fiemap-test-relax.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="fiemap-test-relax.diff"

>From dd182dc5d1163f9bbff037560e2a0b27e416d7fb Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <P@draigBrady.com>
Date: Wed, 13 Apr 2011 07:58:02 +0100
Subject: [PATCH] tests: fix a false positive fiemap test on some file systems

* tests/filefrag-extent-compare: Don't check the length of the
last extent, as this was seen to vary on XFS, where it leaves
trailing blocks allocated for performance reasons.
* tests/cp/fiemap-empty: Though not seen as an issue in practise,
try to avoid possible issues with the allocator in file systems,
by requesting to allocate a power of 2.
---
 tests/cp/fiemap-empty         |   16 ++++++++--------
 tests/filefrag-extent-compare |   16 ++++++++++++----
 2 files changed, 20 insertions(+), 12 deletions(-)

diff --git a/tests/cp/fiemap-empty b/tests/cp/fiemap-empty
index 42d816b..64c3254 100755
--- a/tests/cp/fiemap-empty
+++ b/tests/cp/fiemap-empty
@@ -38,8 +38,8 @@ rm falloc.test
 # which would cause failure of unrelated tests run in parallel.
 require_file_system_bytes_free_ 800000000
 
-fallocate -l 600000000 space.test ||
-  skip_test_ 'this test needs at least 600MB free space'
+fallocate -l 600MiB space.test ||
+  skip_test_ 'this test needs at least 600MiB free space'
 
 # Disable this test on old BTRFS (e.g. Fedora 14)
 # which reports ordinary extents for unwritten ones.
@@ -50,7 +50,7 @@ filefrag -v space.test | grep -F 'unwritten' > /dev/null ||
 rm space.test
 
 # Ensure we read a large empty file quickly
-fallocate -l 300000000 empty.big || framework_failure
+fallocate -l 300MiB empty.big || framework_failure
 timeout 3 cp --sparse=always empty.big cp.test || fail=1
 test $(stat -c %s empty.big) = $(stat -c %s cp.test) || fail=1
 rm empty.big cp.test
@@ -58,7 +58,7 @@ rm empty.big cp.test
 # Ensure we handle extents beyond file size correctly.
 # Note until we support fallocate, we will not maintain
 # the file allocation.  FIXME: amend this test when fallocate is supported.
-fallocate -l 10000000 -n unwritten.withdata || framework_failure
+fallocate -l 10MiB -n unwritten.withdata || framework_failure
 dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=unwritten.withdata
 cp unwritten.withdata cp.test || fail=1
 test $(stat -c %s unwritten.withdata) = $(stat -c %s cp.test) || fail=1
@@ -68,16 +68,16 @@ rm unwritten.withdata cp.test
 # The following to generate unaccounted extents followed by a hole, is not
 # supported by ext4 at least. The ftruncate discards all extents not
 # accounted for in the size.
-#  fallocate -l 10000000 -n unacc.withholes
+#  fallocate -l 10MiB -n unacc.withholes
 #  dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=unacc.withholes
-#  truncate -s20000000 unacc.withholes
+#  truncate -s20M unacc.withholes
 
 # Ensure we handle a hole after empty extents correctly.
 # Since all extents are accounted for in the size,
 # we can maintain the allocation independently from
 # fallocate() support.
-fallocate -l 10000000 empty.withholes
-truncate -s 20000000 empty.withholes
+fallocate -l 10MiB empty.withholes
+truncate -s 20M empty.withholes
 sectors_per_block=$(expr $(stat -c %o .) / 512)
 cp empty.withholes cp.test || fail=1
 test $(stat -c %s empty.withholes) = $(stat -c %s cp.test) || fail=1
diff --git a/tests/filefrag-extent-compare b/tests/filefrag-extent-compare
index 2c33584..194970c 100644
--- a/tests/filefrag-extent-compare
+++ b/tests/filefrag-extent-compare
@@ -56,10 +56,18 @@ merge_extents \@b;
 my $i = 0;
 while (defined $a[$i])
   {
-    $a[$i]->{L_BLK} == $b[$i]->{L_BLK} && $a[$i]->{LEN} == $b[$i]->{LEN}
-      or die "$ME: differing extent:\n"
-        . "  [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
-          . "  [$i]=$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
+    my $start_match = $a[$i]->{L_BLK} == $b[$i]->{L_BLK};
+    my $len_match = $a[$i]->{LEN} == $b[$i]->{LEN};
+    if (!$start_match || (!$len_match && $i != (@a - 1)))
+      {
+        # On XFS on Linux kernel 2.6.38, it was seen that the size of the
+        # last extent can vary, and can extend beyond the length of the file.
+        # So we ignore the length of the last extent, because if the
+        # file is the wrong length we'll get failures elsewhere.
+        die "$ME: differing extent:\n"
+          . "  [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
+            . "  [$i]=$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
+      }
     $i++;
   }
 
-- 
1.7.4


--------------060809010908000709080301--


From MAILER-DAEMON Wed Apr 13 05:45:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1Q9wdx-0005UB-Rg
	for mharc-coreutils@gnu.org; Wed, 13 Apr 2011 05:45:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:49249)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9wdr-0005U0-5K
	for coreutils@gnu.org; Wed, 13 Apr 2011 05:45:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9wdl-0000Dk-Hd
	for coreutils@gnu.org; Wed, 13 Apr 2011 05:45:43 -0400
Received: from mx.meyering.net ([82.230.74.64]:60555)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1Q9wdl-0000DX-8C
	for coreutils@gnu.org; Wed, 13 Apr 2011 05:45:37 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 676B3601B8; Wed, 13 Apr 2011 11:45:35 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: ready for release of coreutils-8.11?
In-Reply-To: <4DA56505.9070602@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Wed, 13 Apr 2011 09:55:33 +0100")
References: <87ei5chcy1.fsf@rho.meyering.net>
	<4D9F9C20.5020709@draigBrady.com> <4DA2E102.5080801@draigBrady.com>
	<878vvgh7nv.fsf@rho.meyering.net> <4DA4177C.2090403@draigBrady.com>
	<87tye3envk.fsf@rho.meyering.net> <4DA43360.4050903@draigBrady.com>
	<878vvfekmp.fsf@rho.meyering.net> <4DA45BF9.3030506@draigBrady.com>
	<4DA46330.9030306@redhat.com> <4DA4E99A.3030102@draigBrady.com>
	<4DA4F6BC.9080908@redhat.com> <4DA56505.9070602@draigBrady.com>
Date: Wed, 13 Apr 2011 11:45:35 +0200
Message-ID: <87wriya3i8.fsf@rho.meyering.net>
Lines: 62
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Eric Sandeen <sandeen@redhat.com>, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 09:45:48 -0000

P=E1draig Brady wrote:
...
> Subject: [PATCH] tests: fix a false positive fiemap test on some file sys=
tems
>
> * tests/filefrag-extent-compare: Don't check the length of the
> last extent, as this was seen to vary on XFS, where it leaves
> trailing blocks allocated for performance reasons.
> * tests/cp/fiemap-empty: Though not seen as an issue in practise,
> try to avoid possible issues with the allocator in file systems,
> by requesting to allocate a power of 2.
> ---
>  tests/cp/fiemap-empty         |   16 ++++++++--------
>  tests/filefrag-extent-compare |   16 ++++++++++++----
>  2 files changed, 20 insertions(+), 12 deletions(-)
>
> diff --git a/tests/cp/fiemap-empty b/tests/cp/fiemap-empty
> index 42d816b..64c3254 100755
> --- a/tests/cp/fiemap-empty
> +++ b/tests/cp/fiemap-empty
> @@ -38,8 +38,8 @@ rm falloc.test
>  # which would cause failure of unrelated tests run in parallel.
>  require_file_system_bytes_free_ 800000000
>
> -fallocate -l 600000000 space.test ||
> -  skip_test_ 'this test needs at least 600MB free space'
> +fallocate -l 600MiB space.test ||
> +  skip_test_ 'this test needs at least 600MiB free space'

Nice.  More readable, too.

...
> diff --git a/tests/filefrag-extent-compare b/tests/filefrag-extent-compare
> index 2c33584..194970c 100644
> --- a/tests/filefrag-extent-compare
> +++ b/tests/filefrag-extent-compare
> @@ -56,10 +56,18 @@ merge_extents \@b;
>  my $i =3D 0;
>  while (defined $a[$i])
>    {
> -    $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK} && $a[$i]->{LEN} =3D=3D $b[$i=
]->{LEN}
> -      or die "$ME: differing extent:\n"
> -        . "  [$i]=3D$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
> -          . "  [$i]=3D$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
> +    my $start_match =3D $a[$i]->{L_BLK} =3D=3D $b[$i]->{L_BLK};
> +    my $len_match =3D $a[$i]->{LEN} =3D=3D $b[$i]->{LEN};
> +    if (!$start_match || (!$len_match && $i !=3D (@a - 1)))
> +      {
> +        # On XFS on Linux kernel 2.6.38, it was seen that the size of the
> +        # last extent can vary, and can extend beyond the length of the =
file.
> +        # So we ignore the length of the last extent, because if the
> +        # file is the wrong length we'll get failures elsewhere.
> +        die "$ME: differing extent:\n"
> +          . "  [$i]=3D$a[$i]->{L_BLK} $a[$i]->{LEN}\n"
> +            . "  [$i]=3D$b[$i]->{L_BLK} $b[$i]->{LEN}\n";
> +      }
>      $i++;
>    }

Thanks.  That looks better still.
Maybe factor out a "!" ?

  if ( ! ($start_match && ($len_match || $i =3D=3D (@a - 1))


From MAILER-DAEMON Wed Apr 13 10:53:44 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QA1Rw-0000UD-PH
	for mharc-coreutils@gnu.org; Wed, 13 Apr 2011 10:53:44 -0400
Received: from eggs.gnu.org ([140.186.70.92]:43446)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QA1Rq-0000U4-At
	for coreutils@gnu.org; Wed, 13 Apr 2011 10:53:43 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QA1Rk-0006eN-Mf
	for coreutils@gnu.org; Wed, 13 Apr 2011 10:53:38 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:14994)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QA1Rk-0006di-D5
	for coreutils@gnu.org; Wed, 13 Apr 2011 10:53:32 -0400
Received: (qmail 55301 invoked from network); 13 Apr 2011 14:53:29 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 13 Apr 2011 14:53:29 -0000
Message-ID: <4DA5B88B.40700@draigBrady.com>
Date: Wed, 13 Apr 2011 15:51:55 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Coreutils <coreutils@gnu.org>
Subject: Re: bug#8490: dd reads random number of records from pipes - named
	or	otherwise - coreutils 8.9
References: <4DA4BDD4.9010802@nikola.com>
In-Reply-To: <4DA4BDD4.9010802@nikola.com>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 14:53:43 -0000

On 12/04/11 22:02, Jesse Gordon wrote:
> I can't believe such an obvious bug would exist this long, but on the
> other hand the test is so simple I can't see where it's user error.
> 
> dd, when reading from stdin or from a named pipe sometimes (but not
> always) reads a random number of records a bit less then what it should.
> 
> I tried it like /dev/zero|dd, like yes|dd, cat somefile|dd, and even 
> mkfifo pip; yes > pipe & dd if=pipe -- and all sometimes failed.
> However, if=arealfile seems to always work perfectly.
> 
> To replicate:
> 
> yes|dd bs=1000 count=1000|wc -c

Eric's email crossed with mine and won the race to close this,
hence resending reply to main list...

With the about to be released coreutils, we now warn about this:

$ yes|dd bs=1000 count=1000|wc -c
dd: warning: partial read (960 bytes); suggest iflag=fullblock

We can't do that by default for backwards compat reasons.

cheers,
Pádraig.

p.s. I just noticed that this is not in NEWS.
I'll fix that now...


From MAILER-DAEMON Wed Apr 13 16:41:57 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QA6sv-0007KR-Lt
	for mharc-coreutils@gnu.org; Wed, 13 Apr 2011 16:41:57 -0400
Received: from eggs.gnu.org ([140.186.70.92]:54716)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QA6ss-0007KL-IZ
	for coreutils@gnu.org; Wed, 13 Apr 2011 16:41:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QA6sr-0005sK-2F
	for coreutils@gnu.org; Wed, 13 Apr 2011 16:41:54 -0400
Received: from mx.meyering.net ([82.230.74.64]:52471)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QA6sq-0005s5-SO
	for coreutils@gnu.org; Wed, 13 Apr 2011 16:41:53 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 32CF0601B8; Wed, 13 Apr 2011 22:41:20 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: Re: odd test failure: misc/sort-spinlock-abuse
In-Reply-To: <87vcyng1bh.fsf@rho.meyering.net> (Jim Meyering's message of
	"Sat, 09 Apr 2011 12:34:58 +0200")
References: <87vcyng1bh.fsf@rho.meyering.net>
Date: Wed, 13 Apr 2011 22:41:19 +0200
Message-ID: <87ipuh51g0.fsf@rho.meyering.net>
Lines: 43
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 20:41:55 -0000

Jim Meyering wrote:
> The misc/sort-spinlock-abuse test,
>
>   http://git.sv.gnu.org/cgit/coreutils.git/tree/tests/misc/sort-spinlock-abuse
>
> fails regularly when it is run in parallel with others (make -j25 check,
> ext4+SSD, 6/12-core, F15).  It fails because an output-restrained sort
> (writing to a FIFO with a slow consumer) takes more than 1 second of CPU
> time to process the regular file, "in", created by "seq 100000 > in".
>
> In fact, it may take even more than 4 seconds of CPU time.
> At first I thought it was a regression.  But no:
> the problem arises even when running sort with --parallel=1.
> The original bug involved a parallel-specific busy-wait
> triggered by the blocked output.
>
> What's going on?
> I have traced it back to an fstat syscall that is consuming
> lots of CPU time.
>
> Here's strace -r output, where FD 3 refers to the regular input file, "in":
>
>     3.263544 fstat(3, {st_mode=S_IFREG|0600, st_size=588895, ...}) = 0
>
> When I run this test in isolation, it always completes successfully.
> In that case, the fstat takes 30-40 microseconds.
>
>     make check -C tests TESTS=misc/sort-spinlock-abuse VERBOSE=yes
>
> But when I run it via "make -j25 check", it fails ~40% of the time.
>
> Next step is probably to see if oprofile can shed some light.

[composed a few days ago]

I've just updated to a newer kernel,                2.6.38.2-9.fc15.x86_64.
The one I was using when experiencing the above was 2.6.38.1-6.fc15.x86_64.
Now, the test passes every time:  38 of 38 trials, so far.

Of course, it may not be the kernel, but rather the state of
the system -- it had been running for some time.

As I said: odd.


From MAILER-DAEMON Wed Apr 13 17:50:33 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QA7xJ-00012O-CY
	for mharc-coreutils@gnu.org; Wed, 13 Apr 2011 17:50:33 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47283)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QA7xH-0000zH-Hf
	for coreutils@gnu.org; Wed, 13 Apr 2011 17:50:32 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QA7xG-0008W0-Br
	for coreutils@gnu.org; Wed, 13 Apr 2011 17:50:31 -0400
Received: from mx.meyering.net ([82.230.74.64]:60937)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>)
	id 1QA7xD-0008VR-Fw; Wed, 13 Apr 2011 17:50:27 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 42216601FF; Wed, 13 Apr 2011 23:50:25 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: info-gnu@gnu.org
Subject: coreutils-8.11 released [stable]
Date: Wed, 13 Apr 2011 22:38:38 +0200
Mail-Followup-To: bug-coreutils@gnu.org
Message-ID: <87pqopx1lq.fsf@rho.meyering.net>
Lines: 133
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coordinator@translationproject.org, coreutils@gnu.org,
	coreutils-announce@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 13 Apr 2011 21:50:32 -0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is to announce coreutils-8.11, a stable release.

This time we have an interesting variety of bug fixes,
two new features for "dd", and a few "changes in behavior".
See NEWS below for a brief summary.

Jim [on behalf of the coreutils maintainers]

- -------------------------------------------
P.S. here's the GNU Coreutils home page:
    http://www.gnu.org/software/coreutils/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v8.11
or run this command from a git-cloned coreutils directory:
  git shortlog v8.10..v8.11

To summarize the many gnulib-related changes, run these commands from
a git-cloned coreutils directory:
  git checkout v8.11
  git submodule summary v8.10

Here are the compressed sources:
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.11.tar.gz   (11MB)
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.11.tar.xz   (4.7MB)

Here are the GPG detached signatures[*]:
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.11.tar.gz.sig
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.11.tar.xz.sig

To reduce load on the main server, use a mirror listed at:
  http://www.gnu.org/order/ftp.html

[*] You can use either of the above signature files to verify that
the corresponding file (without the .sig suffix) is intact.  First,
be sure to download both the .sig file and the corresponding tarball.
Then, run a command like this:

  gpg --verify coreutils-8.11.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 000BEEEE

and rerun the `gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.68.71-af300
  Automake 1.11a
  Gnulib v0.0-5115-ga81348d
  Bison 2.4.559-6769

==========================================================
NEWS

* Noteworthy changes in release 8.11 (2011-04-13) [stable]

** Bug fixes

  cp -a --link would not create a hardlink to a symlink, instead
  copying the symlink and then not preserving its timestamp.
  [bug introduced in coreutils-8.0]

  cp now avoids FIEMAP issues with BTRFS before Linux 2.6.38,
  which could result in corrupt copies of sparse files.
  [bug introduced in coreutils-8.10]

  cut could segfault when invoked with a user-specified output
  delimiter and an unbounded range like "-f1234567890-".
  [bug introduced in coreutils-5.3.0]

  du would infloop when given --files0-from=DIR
  [bug introduced in coreutils-7.1]

  sort no longer spawns 7 worker threads to sort 16 lines
  [bug introduced in coreutils-8.6]

  touch built on Solaris 9 would segfault when run on Solaris 10
  [bug introduced in coreutils-8.8]

  wc would dereference a NULL pointer upon an early out-of-memory error
  [bug introduced in coreutils-7.1]

** New features

  dd now accepts the 'nocache' flag to the iflag and oflag options,
  which will discard any cache associated with the files, or
  processed portion thereof.

  dd now warns that 'iflag=fullblock' should be used,
  in various cases where partial reads can cause issues.

** Changes in behavior

  cp now avoids syncing files when possible, when doing a FIEMAP copy.
  The sync is only needed on Linux kernels before 2.6.39.
  [The sync was introduced in coreutils-8.10]

  cp now copies empty extents efficiently, when doing a FIEMAP copy.
  It no longer reads the zero bytes from the input, and also can efficiently
  create a hole in the output file when --sparse=always is specified.

  df now aligns columns consistently, and no longer wraps entries
  with longer device identifiers, over two lines.

  install now rejects its long-deprecated --preserve_context option.
  Use --preserve-context instead.

  test now accepts "==" as a synonym for "="

- -----
also posted as: http://savannah.gnu.org/forum/forum.php?forum_id=6785
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBCAAGBQJNphpLAAoJEH/Z/MsAC+7usF0P/0Bf9RoWCeH46ByLKaOhKedz
ZJQ9fJGWk2oGV0XCdYuejc7IKc4urTMCJIxfRTBLqgpa6D8ExwNMf9k+vFb89Wpv
lq18vmfx8gdN/mlehe8c39wPzue9+EC1w/FO8wlFgs3Koc2xUxCE/EYC/gJIklBN
JbsknZB/GACy70npeUikEFMg0+sPuf9vMBjZmwk8GpNNFpLwkkdB5KLMus0qxDw2
4xcK4PqhcuNg1EmX5UZN+64eRossmBDGScG9z1CHrEvSxqbSyhqxofB2qNTp+K2D
/EuaJHpjeoDDAfm6iHPwxVF6Mn7zEA65bBRfWyWjKT+L3B9PzmMSY1juBS+ujaja
tlLBT09D4erGskekid7vcufwNPwuj0c8RaHScj4gdbunXDpBu9/ZNfYR2Mavwr5q
G7qDBEJc3BD+peXJT25dWjJtqJ7HuwQN5DaFkVxS1gr0tpTMLSCxxMtyhl3hRByn
F3qcDK7iKg+iE85rY33dbShn5/q4SfVveczh7QwILr5jYf37XwhvUw4fkuGnY24q
a9ndmx2/FmdBkrFthO9jl47MUtCJeDQ+CC1TZGCUOK2Z9H6+GhWi2+VIpcLsqC/j
xMIW+QcT5B88NVZw0m8ur5Icup35PF2TxreUl0qeiQbaGm4KUIfJboYXV/q/zq6q
wb6Dt+lOLe94Sebr8YM+
=X0/c
-----END PGP SIGNATURE-----


From MAILER-DAEMON Thu Apr 14 09:36:03 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAMiJ-0007vE-Bb
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 09:36:03 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51373)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAMiG-0007q3-Ny
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:36:01 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAMiF-0002Ce-FY
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:36:00 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:14693)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAMiF-0002Bu-6b
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:35:59 -0400
Received: (qmail 65494 invoked from network); 14 Apr 2011 13:35:57 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 14 Apr 2011 13:35:57 -0000
Message-ID: <4DA6F7DB.4050907@draigBrady.com>
Date: Thu, 14 Apr 2011 14:34:19 +0100
From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: markus@trippelsdorf.de
Subject: fiemap issues with XFS
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 13:36:01 -0000

Hi Markus,

I noticed your fiemap issues here:
http://oss.sgi.com/pipermail/xfs/2011-April/050102.html

FAIL: cp/fiemap-empty (exit: 1)
===============================
...
+ fallocate -l 10MiB -n unwritten.withdata
+ dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock
of=unwritten.withdata
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.00219578 s, 2.3 MB/s
+ cp unwritten.withdata cp.test
++ stat -c %s unwritten.withdata
++ stat -c %s cp.test
+ test 5120 = 5120
+ cmp unwritten.withdata cp.test
unwritten.withdata cp.test differ: char 1, line 1
+ fail=1

cp was changed in 8.11 to not bother reading
an extent if it is marked as UNWRITTEN.
The comment in fiemap.h says that this means that the
space is allocated, but zero.

We tested on XFS, on F15 x86_64, which is earlier
than your 2.6.39-rc3 and didn't notice this issue.

I'm guessing so that XFS is reporting the extent
as UNWRITTEN, even though there is data in it now,
and that it might sort itself out after a while,
or after a sync I suppose (note we also stopped
using sync before fiemap for 2.6.39).

It would help a lot if you could insert this command
into the test above (just before the failing cp)
and show the test output again:

  filefrag -v unwritten.withdata

thanks,
PĂˇdraig.


From MAILER-DAEMON Thu Apr 14 09:55:41 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAN1I-0003Lp-Vo
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 09:55:40 -0400
Received: from eggs.gnu.org ([140.186.70.92]:46256)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAN1E-0003KW-GZ
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:55:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAN1D-000659-Aw
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:55:36 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:2825)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAN1D-00064z-3Q
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:55:35 -0400
Received: (qmail 69764 invoked from network); 14 Apr 2011 13:55:33 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 14 Apr 2011 13:55:33 -0000
Message-ID: <4DA6FC73.9070109@draigBrady.com>
Date: Thu, 14 Apr 2011 14:53:55 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: fiemap issues with XFS
References: <4DA6F7DB.4050907@draigBrady.com>
	<20110414134805.GA1679@x4.trippels.de>
In-Reply-To: <20110414134805.GA1679@x4.trippels.de>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 13:55:40 -0000

On 14/04/11 14:48, Markus Trippelsdorf wrote:
> On 2011.04.14 at 14:34 +0100, Pádraig Brady wrote:
>> Hi Markus,
>>
>> I noticed your fiemap issues here:
>> http://oss.sgi.com/pipermail/xfs/2011-April/050102.html
>>
>> FAIL: cp/fiemap-empty (exit: 1)
>> ===============================
>> ...
>> + fallocate -l 10MiB -n unwritten.withdata
>> + dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock
>> of=unwritten.withdata
>> 10+0 records in
>> 10+0 records out
>> 5120 bytes (5.1 kB) copied, 0.00219578 s, 2.3 MB/s
>> + cp unwritten.withdata cp.test
>> ++ stat -c %s unwritten.withdata
>> ++ stat -c %s cp.test
>> + test 5120 = 5120
>> + cmp unwritten.withdata cp.test
>> unwritten.withdata cp.test differ: char 1, line 1
>> + fail=1
>>
>> cp was changed in 8.11 to not bother reading
>> an extent if it is marked as UNWRITTEN.
>> The comment in fiemap.h says that this means that the
>> space is allocated, but zero.
>>
>> We tested on XFS, on F15 x86_64, which is earlier
>> than your 2.6.39-rc3 and didn't notice this issue.
>>
>> I'm guessing so that XFS is reporting the extent
>> as UNWRITTEN, even though there is data in it now,
>> and that it might sort itself out after a while,
>> or after a sync I suppose (note we also stopped
>> using sync before fiemap for 2.6.39).
>>
>> It would help a lot if you could insert this command
>> into the test above (just before the failing cp)
>> and show the test output again:
>>
>>   filefrag -v unwritten.withdata
> 
> Hi Pádraig,
> 
> here you go:
> + filefrag -v unwritten.withdata                                                                                                                     
> Filesystem type is: ef53                                                                                                                             
> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
>  ext logical physical expected length flags                                                                                                          
>    0       0   274432            2560 unwritten,eof                                                                                                  
> unwritten.withdata: 1 extent found
> 
> Please notice that this also happens with ext4 on the same kernel. 
> Btrfs is fine.

That looks like a bug in XFS :(
I presume if you change `filefrag -v` to `filefrag -vs` that
the output will change, and the test will pass.
I'm a bit surprised that ext4 shows the same thing
as there was supposedly a patch for this issue already
applied for 2.6.39.

It would be great if we got these fixed up before
2.6.29 was released, so that the checks in coreutils 8.11
were valid.

cheers,
Pádraig.


From MAILER-DAEMON Thu Apr 14 10:24:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QANSt-0007va-VA
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 10:24:11 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60766)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAJkb-0003Fh-Qc
	for coreutils@gnu.org; Thu, 14 Apr 2011 06:26:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAJka-0007Xe-LJ
	for coreutils@gnu.org; Thu, 14 Apr 2011 06:26:13 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:58089
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAJka-0007XU-70
	for coreutils@gnu.org; Thu, 14 Apr 2011 06:26:12 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:mime-version:content-type; q=
	dns/txt; s=beta; bh=4lcuYLRhXt6GAjm8upwOex2cpNFE5i0DD4Udb5Ruhgc=; b=
	YMZlV9V0P4WEqzmn/IsinaJLBpVmcoh1Xbxy63SS6DlBI31VNYXAbR5Bwdpr5dN8
	r+Nwbd4iKwaqex1W6Hs2nbndDx6U/srC1nevvuE0u4Bsv6Ctx3hafSYEznMb1dFY
	BfLUgpGPRax33leifIqm8vJ+qtDE8wCfnEghNDSqYYo=
Received: (qmail 21862 invoked from network); 14 Apr 2011 12:26:09 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 12:26:09 +0200
Date: Thu, 14 Apr 2011 12:26:08 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: coreutils@gnu.org
Subject: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414102608.GA1678@x4.trippels.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
X-Mailman-Approved-At: Thu, 14 Apr 2011 10:24:10 -0400
Cc: xfs@oss.sgi.com
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 10:26:14 -0000

I trashed my system this morning when I installed coreutils-8.11.

What happened is that coreutils compiles and links correctly, but then
the following command (during the installation phase):

./ginstall chroot hostid nice who users pinky stty df stdbuf [ base64 basename cat chcon chgrp chmod chown cksum comm cp csplit cut date dd dir dircolors dirname du echo env expand expr factor false fmt fold head id join link ln logname ls md5sum mkdir mkfifo mknod mktemp mv nl nproc nohup od paste pathchk pr printenv printf ptx pwd readlink rm rmdir runcon seq sha1sum sha224sum sha256sum sha384sum sha512sum shred shuf sleep sort split stat sum sync tac tail tee test timeout touch tr true truncate tsort tty uname unexpand uniq unlink vdir wc whoami yes arch '/var/tmp/portage/sys-apps/coreutils-8.11/image//usr/bin'

apparently produces files which have the length of the originals but are
full of zeros. (and these were then installed to my live system, thereby
trashing it).

Now all the above is automated, because I use gentoo.
But when I run the command above (ginstall) later again by hand,
everything is copied just fine and the resulting binaries are all
usable.

The partition in question uses xfs.

# xfs_info /var
meta-data=/dev/sda1              isize=256    agcount=4, agsize=12800000 blks
         =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=51200000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=25000, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

This is a 4kb hard drive (sectsz=4096).

I'm running the lastest vanilla git kernel (2.6.39-rc3-00087-gda768a4).

Now my question is, could this be caused by the recent FIEMAP changes in
coreutils?
-- 
Markus


From MAILER-DAEMON Thu Apr 14 10:24:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QANSu-0007wE-2q
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 10:24:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:39476)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QALJt-0003XJ-8E
	for coreutils@gnu.org; Thu, 14 Apr 2011 08:06:49 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QALJo-0002vx-BC
	for coreutils@gnu.org; Thu, 14 Apr 2011 08:06:45 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:56624
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QALJn-0002vi-RU
	for coreutils@gnu.org; Thu, 14 Apr 2011 08:06:40 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:in-reply-to; q=dns/txt; s=beta; bh=TXfw5sB2ZBiUggPO
	CX4V72oMNhtBvfzp4IVVXhQZWz4=; b=QtQFnKiGk5RTjdt4hBUWk8/eACgnIrE1
	623bAw7kEmw8Yf6cpQk6x6AySsVSO9uqgq4BK5gycx3SiJVYgQkCB8TTwxI674+6
	S8cFo9UFkyCR+YkF2rBnL/KWfCxVo5q08QnHfl6UbOfo6ra1KrP1sy/BCqCR5+E+
	PGHJl3GEMtQ=
Received: (qmail 18026 invoked from network); 14 Apr 2011 14:06:36 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 14:06:36 +0200
Date: Thu, 14 Apr 2011 14:06:35 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: coreutils@gnu.org
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414120635.GB1678@x4.trippels.de>
References: <20110414102608.GA1678@x4.trippels.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110414102608.GA1678@x4.trippels.de>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
X-Mailman-Approved-At: Thu, 14 Apr 2011 10:24:10 -0400
Cc: xfs@oss.sgi.com
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 12:06:50 -0000

On 2011.04.14 at 12:26 +0200, Markus Trippelsdorf wrote:
> I trashed my system this morning when I installed coreutils-8.11.
> 
> What happened is that coreutils compiles and links correctly, but then
> the following command (during the installation phase):
> 
> ./ginstall chroot hostid nice who users pinky stty df stdbuf [ base64 basename cat chcon chgrp chmod chown cksum comm cp csplit cut date dd dir dircolors dirname du echo env expand expr factor false fmt fold head id join link ln logname ls md5sum mkdir mkfifo mknod mktemp mv nl nproc nohup od paste pathchk pr printenv printf ptx pwd readlink rm rmdir runcon seq sha1sum sha224sum sha256sum sha384sum sha512sum shred shuf sleep sort split stat sum sync tac tail tee test timeout touch tr true truncate tsort tty uname unexpand uniq unlink vdir wc whoami yes arch '/var/tmp/portage/sys-apps/coreutils-8.11/image//usr/bin'
> 
> apparently produces files which have the length of the originals but are
> full of zeros. (and these were then installed to my live system, thereby
> trashing it).
> 
> Now all the above is automated, because I use gentoo.
> But when I run the command above (ginstall) later again by hand,
> everything is copied just fine and the resulting binaries are all
> usable.
> 
> The partition in question uses xfs.
> 
> # xfs_info /var
> meta-data=/dev/sda1              isize=256    agcount=4, agsize=12800000 blks
>          =                       sectsz=4096  attr=2
> data     =                       bsize=4096   blocks=51200000, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0
> log      =internal               bsize=4096   blocks=25000, version=2
>          =                       sectsz=4096  sunit=1 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
> 
> This is a 4kb hard drive (sectsz=4096).
> 
> I'm running the lastest vanilla git kernel (2.6.39-rc3-00087-gda768a4).
> 
> Now my question is, could this be caused by the recent FIEMAP changes in
> coreutils?

Apparently yes:

Here is a "make check" failure example:

FAIL: cp/fiemap-empty (exit: 1)
===============================
...
+ fallocate -l 10MiB -n unwritten.withdata
+ dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock
of=unwritten.withdata
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.00219578 s, 2.3 MB/s
+ cp unwritten.withdata cp.test
++ stat -c %s unwritten.withdata
++ stat -c %s cp.test
+ test 5120 = 5120
+ cmp unwritten.withdata cp.test
unwritten.withdata cp.test differ: char 1, line 1
+ fail=1

-- 
Markus


From MAILER-DAEMON Thu Apr 14 10:24:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QANSu-0007wV-7B
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 10:24:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:36040)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAMu2-00027J-50
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:48:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAMu0-0004rc-Ri
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:48:10 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:35669
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAMu0-0004rI-CO
	for coreutils@gnu.org; Thu, 14 Apr 2011 09:48:08 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:content-transfer-encoding:in-reply-to; q=dns/txt; s=
	beta; bh=VktmkKGdRmTNVBnFi87sv7JW8z0bmAmDVfNVjROMm64=; b=TirJfgo
	erId/FXzk+xyLOIhE0aDjg2eVUTboZzN4b7DWiUC9QDeBzkCfefdxKDHOaQ9K/tK
	6NIuaraYLCFqswWlmpXVZ/HlM8Ch0hyXehjYpoonQs2yud1Cs6okJvcUYCGB0TSs
	AbYfyfu6YCZZX8B1iPn998Dkv9shdWRkzksQ=
Received: (qmail 14622 invoked from network); 14 Apr 2011 15:48:05 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 15:48:05 +0200
Date: Thu, 14 Apr 2011 15:48:05 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: fiemap issues with XFS
Message-ID: <20110414134805.GA1679@x4.trippels.de>
References: <4DA6F7DB.4050907@draigBrady.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4DA6F7DB.4050907@draigBrady.com>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
X-Mailman-Approved-At: Thu, 14 Apr 2011 10:24:10 -0400
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 13:48:11 -0000

On 2011.04.14 at 14:34 +0100, Pádraig Brady wrote:
> Hi Markus,
> 
> I noticed your fiemap issues here:
> http://oss.sgi.com/pipermail/xfs/2011-April/050102.html
> 
> FAIL: cp/fiemap-empty (exit: 1)
> ===============================
> ...
> + fallocate -l 10MiB -n unwritten.withdata
> + dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock
> of=unwritten.withdata
> 10+0 records in
> 10+0 records out
> 5120 bytes (5.1 kB) copied, 0.00219578 s, 2.3 MB/s
> + cp unwritten.withdata cp.test
> ++ stat -c %s unwritten.withdata
> ++ stat -c %s cp.test
> + test 5120 = 5120
> + cmp unwritten.withdata cp.test
> unwritten.withdata cp.test differ: char 1, line 1
> + fail=1
> 
> cp was changed in 8.11 to not bother reading
> an extent if it is marked as UNWRITTEN.
> The comment in fiemap.h says that this means that the
> space is allocated, but zero.
> 
> We tested on XFS, on F15 x86_64, which is earlier
> than your 2.6.39-rc3 and didn't notice this issue.
> 
> I'm guessing so that XFS is reporting the extent
> as UNWRITTEN, even though there is data in it now,
> and that it might sort itself out after a while,
> or after a sync I suppose (note we also stopped
> using sync before fiemap for 2.6.39).
> 
> It would help a lot if you could insert this command
> into the test above (just before the failing cp)
> and show the test output again:
> 
>   filefrag -v unwritten.withdata

Hi Pádraig,

here you go:
+ filefrag -v unwritten.withdata                                                                                                                     
Filesystem type is: ef53                                                                                                                             
File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
 ext logical physical expected length flags                                                                                                          
   0       0   274432            2560 unwritten,eof                                                                                                  
unwritten.withdata: 1 extent found

Please notice that this also happens with ext4 on the same kernel. 
Btrfs is fine.
-- 
Markus


From MAILER-DAEMON Thu Apr 14 10:24:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QANSu-0007x4-Ai
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 10:24:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:42144)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAN7w-0004I0-Au
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:02:33 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAN7q-0007ci-6V
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:02:32 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:53672
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAN7p-0007cZ-Q9
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:02:26 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:content-transfer-encoding:in-reply-to; q=dns/txt; s=
	beta; bh=LIFPi9Taet2HAHKAlCt8/VUa1Y8r227Ydr5LJ6EcIpU=; b=PVNclz6
	U2Vg1z5rvF9PTZblS/xF3OnONSzl6QI932A37qg25IoGoEkiXLPpEi5Dq2w0tO95
	NpP1ieKZDVtxSehIxT8Dwh2IetSbh8C7h8OphiGs0VTdtSZ46rUcuWmXLuUC+sPQ
	e4afX4wh9PtZyoJGfD4EZiDbpAU3oLSF/PXI=
Received: (qmail 18451 invoked from network); 14 Apr 2011 16:02:23 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 16:02:23 +0200
Date: Thu, 14 Apr 2011 16:02:22 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: coreutils@gnu.org
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414140222.GB1679@x4.trippels.de>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20110414120635.GB1678@x4.trippels.de>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
X-Mailman-Approved-At: Thu, 14 Apr 2011 10:24:10 -0400
Cc: xfs@oss.sgi.com
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 14:02:33 -0000

On 2011.04.14 at 14:53 +0100, Pádraig Brady wrote:
> On 14/04/11 14:48, Markus Trippelsdorf wrote:
> > On 2011.04.14 at 14:34 +0100, Pádraig Brady wrote:
> >> Hi Markus,
> >>
> >> I noticed your fiemap issues here:
> >> http://oss.sgi.com/pipermail/xfs/2011-April/050102.html
> >>
> >> FAIL: cp/fiemap-empty (exit: 1)
> >> ===============================
> >> ...
> >> + fallocate -l 10MiB -n unwritten.withdata
> >> + dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock
> >> of=unwritten.withdata
> >> 10+0 records in
> >> 10+0 records out
> >> 5120 bytes (5.1 kB) copied, 0.00219578 s, 2.3 MB/s
> >> + cp unwritten.withdata cp.test
> >> ++ stat -c %s unwritten.withdata
> >> ++ stat -c %s cp.test
> >> + test 5120 = 5120
> >> + cmp unwritten.withdata cp.test
> >> unwritten.withdata cp.test differ: char 1, line 1
> >> + fail=1
> >>
> >> cp was changed in 8.11 to not bother reading
> >> an extent if it is marked as UNWRITTEN.
> >> The comment in fiemap.h says that this means that the
> >> space is allocated, but zero.
> >>
> >> We tested on XFS, on F15 x86_64, which is earlier
> >> than your 2.6.39-rc3 and didn't notice this issue.
> >>
> >> I'm guessing so that XFS is reporting the extent
> >> as UNWRITTEN, even though there is data in it now,
> >> and that it might sort itself out after a while,
> >> or after a sync I suppose (note we also stopped
> >> using sync before fiemap for 2.6.39).
> >>
> >> It would help a lot if you could insert this command
> >> into the test above (just before the failing cp)
> >> and show the test output again:
> >>
> >>   filefrag -v unwritten.withdata
> > 
> > Hi Pádraig,
> > 
> > here you go:
> > + filefrag -v unwritten.withdata                                                                                                                     
> > Filesystem type is: ef53                                                                                                                             
> > File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
> >  ext logical physical expected length flags                                                                                                          
> >    0       0   274432            2560 unwritten,eof                                                                                                  
> > unwritten.withdata: 1 extent found
> > 
> > Please notice that this also happens with ext4 on the same kernel. 
> > Btrfs is fine.
> 
> That looks like a bug in XFS :(
> I presume if you change `filefrag -v` to `filefrag -vs` that
> the output will change, and the test will pass.
> I'm a bit surprised that ext4 shows the same thing
> as there was supposedly a patch for this issue already
> applied for 2.6.39.
> 
> It would be great if we got these fixed up before
> 2.6.29 was released, so that the checks in coreutils 8.11
> were valid.

You're right `filefrag -vs` fixes the issue on both xfs and ext4.

-- 
Markus


From MAILER-DAEMON Thu Apr 14 10:24:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QANSu-0007xF-Gk
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 10:24:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:42475)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QANAA-0004mx-0n
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:05:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QANA7-0007yp-6o
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:04:49 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:35578
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QANA6-0007yL-SO
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:04:47 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:content-transfer-encoding:in-reply-to; q=dns/txt; s=
	beta; bh=LIFPi9Taet2HAHKAlCt8/VUa1Y8r227Ydr5LJ6EcIpU=; b=l+0EdZU
	nuje/Kkk3qElta3yb1FycUrDlUOJedNpaiW9RtcR6xTSUClu4Mi9Mu9KGSYt+m8l
	yCdv4VPsf+QTU3YANbEYKtwmMsaywrQBRWDXb0tTROT4SfDvJyrGzqiVS+CvKbUZ
	fXGsbQITNqNtWL0Af7Ud39LFg/ABEL5DqzuA=
Received: (qmail 19034 invoked from network); 14 Apr 2011 16:04:45 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 16:04:45 +0200
Date: Thu, 14 Apr 2011 16:04:45 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: fiemap issues with XFS
Message-ID: <20110414140445.GC1679@x4.trippels.de>
References: <4DA6F7DB.4050907@draigBrady.com>
	<20110414134805.GA1679@x4.trippels.de>
	<4DA6FC73.9070109@draigBrady.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4DA6FC73.9070109@draigBrady.com>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
X-Mailman-Approved-At: Thu, 14 Apr 2011 10:24:10 -0400
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 14:05:02 -0000

On 2011.04.14 at 14:53 +0100, Pádraig Brady wrote:
> On 14/04/11 14:48, Markus Trippelsdorf wrote:
> > On 2011.04.14 at 14:34 +0100, Pádraig Brady wrote:
> >> Hi Markus,
> >>
> >> I noticed your fiemap issues here:
> >> http://oss.sgi.com/pipermail/xfs/2011-April/050102.html
> >>
> >> FAIL: cp/fiemap-empty (exit: 1)
> >> ===============================
> >> ...
> >> + fallocate -l 10MiB -n unwritten.withdata
> >> + dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock
> >> of=unwritten.withdata
> >> 10+0 records in
> >> 10+0 records out
> >> 5120 bytes (5.1 kB) copied, 0.00219578 s, 2.3 MB/s
> >> + cp unwritten.withdata cp.test
> >> ++ stat -c %s unwritten.withdata
> >> ++ stat -c %s cp.test
> >> + test 5120 = 5120
> >> + cmp unwritten.withdata cp.test
> >> unwritten.withdata cp.test differ: char 1, line 1
> >> + fail=1
> >>
> >> cp was changed in 8.11 to not bother reading
> >> an extent if it is marked as UNWRITTEN.
> >> The comment in fiemap.h says that this means that the
> >> space is allocated, but zero.
> >>
> >> We tested on XFS, on F15 x86_64, which is earlier
> >> than your 2.6.39-rc3 and didn't notice this issue.
> >>
> >> I'm guessing so that XFS is reporting the extent
> >> as UNWRITTEN, even though there is data in it now,
> >> and that it might sort itself out after a while,
> >> or after a sync I suppose (note we also stopped
> >> using sync before fiemap for 2.6.39).
> >>
> >> It would help a lot if you could insert this command
> >> into the test above (just before the failing cp)
> >> and show the test output again:
> >>
> >>   filefrag -v unwritten.withdata
> > 
> > Hi Pádraig,
> > 
> > here you go:
> > + filefrag -v unwritten.withdata                                                                                                                     
> > Filesystem type is: ef53                                                                                                                             
> > File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
> >  ext logical physical expected length flags                                                                                                          
> >    0       0   274432            2560 unwritten,eof                                                                                                  
> > unwritten.withdata: 1 extent found
> > 
> > Please notice that this also happens with ext4 on the same kernel. 
> > Btrfs is fine.
> 
> That looks like a bug in XFS :(
> I presume if you change `filefrag -v` to `filefrag -vs` that
> the output will change, and the test will pass.
> I'm a bit surprised that ext4 shows the same thing
> as there was supposedly a patch for this issue already
> applied for 2.6.39.
> 
> It would be great if we got these fixed up before
> 2.6.29 was released, so that the checks in coreutils 8.11
> were valid.

You're right `filefrag -vs` fixes the issue on both xfs and ext4.

-- 
Markus


From MAILER-DAEMON Thu Apr 14 10:39:33 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QANhl-00015k-QA
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 10:39:33 -0400
Received: from eggs.gnu.org ([140.186.70.92]:42365)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QANhK-00015B-KT
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:39:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QANhJ-0006JN-Js
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:39:06 -0400
Received: from sandeen.net ([63.231.237.45]:48798 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QANhJ-0006J1-CW
	for coreutils@gnu.org; Thu, 14 Apr 2011 10:39:05 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id CAC6948FDA82;
	Thu, 14 Apr 2011 09:39:02 -0500 (CDT)
Message-ID: <4DA70706.4050305@sandeen.net>
Date: Thu, 14 Apr 2011 09:39:02 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>
In-Reply-To: <20110414102608.GA1678@x4.trippels.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
X-Mailman-Approved-At: Thu, 14 Apr 2011 10:39:32 -0400
Cc: coreutils@gnu.org, xfs@oss.sgi.com
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 14:39:07 -0000

On 4/14/11 5:26 AM, Markus Trippelsdorf wrote:
> I trashed my system this morning when I installed coreutils-8.11.
> 
> What happened is that coreutils compiles and links correctly, but
> then the following command (during the installation phase):
> 
> ./ginstall chroot hostid nice who users pinky stty df stdbuf [ base64
> basename cat chcon chgrp chmod chown cksum comm cp csplit cut date dd
> dir dircolors dirname du echo env expand expr factor false fmt fold
> head id join link ln logname ls md5sum mkdir mkfifo mknod mktemp mv
> nl nproc nohup od paste pathchk pr printenv printf ptx pwd readlink
> rm rmdir runcon seq sha1sum sha224sum sha256sum sha384sum sha512sum
> shred shuf sleep sort split stat sum sync tac tail tee test timeout
> touch tr true truncate tsort tty uname unexpand uniq unlink vdir wc
> whoami yes arch
> '/var/tmp/portage/sys-apps/coreutils-8.11/image//usr/bin'
> 
> apparently produces files which have the length of the originals but
> are full of zeros. (and these were then installed to my live system,
> thereby trashing it).
> 
> Now all the above is automated, because I use gentoo. But when I run
> the command above (ginstall) later again by hand, everything is
> copied just fine and the resulting binaries are all usable.
> 
> The partition in question uses xfs.
> 
> # xfs_info /var meta-data=/dev/sda1              isize=256
> agcount=4, agsize=12800000 blks =                       sectsz=4096
> attr=2 data     =                       bsize=4096   blocks=51200000,
> imaxpct=25 =                       sunit=0      swidth=0 blks naming
> =version 2              bsize=4096   ascii-ci=0 log      =internal
> bsize=4096   blocks=25000, version=2 =
> sectsz=4096  sunit=1 blks, lazy-count=1 realtime =none
> extsz=4096   blocks=0, rtextents=0
> 
> This is a 4kb hard drive (sectsz=4096).
> 
> I'm running the lastest vanilla git kernel
> (2.6.39-rc3-00087-gda768a4).
> 
> Now my question is, could this be caused by the recent FIEMAP changes
> in coreutils?


Well damn.  Looking into it ...

-Eric


From MAILER-DAEMON Thu Apr 14 11:01:19 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAO2p-0003xx-PB
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 11:01:19 -0400
Received: from eggs.gnu.org ([140.186.70.92]:36793)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAO2j-0003xg-A6
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:01:18 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAO2i-0002W8-8R
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:01:13 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:7765)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAO2i-0002Vd-0S
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:01:12 -0400
Received: (qmail 83857 invoked from network); 14 Apr 2011 15:01:09 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 14 Apr 2011 15:01:09 -0000
Message-ID: <4DA70BD3.1070409@draigBrady.com>
Date: Thu, 14 Apr 2011 15:59:31 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: xfs-oss <xfs@oss.sgi.com>, linux-ext4@vger.kernel.org
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
In-Reply-To: <20110414140222.GB1679@x4.trippels.de>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org, Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 15:01:18 -0000

On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>> Hi Pádraig,
>>>
>>> here you go:
>>> + filefrag -v unwritten.withdata                                                                                                                     
>>> Filesystem type is: ef53                                                                                                                             
>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
>>>  ext logical physical expected length flags                                                                                                          
>>>    0       0   274432            2560 unwritten,eof                                                                                                  
>>> unwritten.withdata: 1 extent found
>>>
>>> Please notice that this also happens with ext4 on the same kernel. 
>>> Btrfs is fine.
>>
> `filefrag -vs` fixes the issue on both xfs and ext4.

So in summary, currently on (2.6.39-rc3), the following
will (usually?) report a single unwritten extent,
on both ext4 and xfs

  fallocate -l 10MiB -n k
  dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
  filefrag -v k # grep for an extent without unwritten || fail

This particular issue has been discussed so far at:
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
Note there it was stated there that ext4 had this
fixed as of 2.6.39-rc1, so maybe there is something lurking?

cheers,
Pádraig.


From MAILER-DAEMON Thu Apr 14 11:50:35 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAOoV-0006zr-2T
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 11:50:35 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35903)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAOoJ-0006wb-W5
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:50:33 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAOo8-0006qp-J8
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:50:23 -0400
Received: from sandeen.net ([63.231.237.45]:34003 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAOo8-0006q4-6J
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:50:12 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 703AA48FDAA3;
	Thu, 14 Apr 2011 10:50:10 -0500 (CDT)
Message-ID: <4DA717B2.3020305@sandeen.net>
Date: Thu, 14 Apr 2011 10:50:10 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>	<20110414120635.GB1678@x4.trippels.de>	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
In-Reply-To: <4DA70BD3.1070409@draigBrady.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 15:50:33 -0000

On 4/14/11 9:59 AM, P=E1draig Brady wrote:
> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>> Hi P=E1draig,
>>>>
>>>> here you go:
>>>> + filefrag -v unwritten.withdata                                    =
                                                                         =
       =20
>>>> Filesystem type is: ef53                                            =
                                                                         =
       =20
>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)  =
                                                                         =
       =20
>>>>  ext logical physical expected length flags                         =
                                                                         =
       =20
>>>>    0       0   274432            2560 unwritten,eof                 =
                                                                         =
       =20
>>>> unwritten.withdata: 1 extent found
>>>>
>>>> Please notice that this also happens with ext4 on the same kernel.=20
>>>> Btrfs is fine.
>>>
>> `filefrag -vs` fixes the issue on both xfs and ext4.
>=20
> So in summary, currently on (2.6.39-rc3), the following
> will (usually?) report a single unwritten extent,
> on both ext4 and xfs
>=20
>   fallocate -l 10MiB -n k
>   dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock of=3D=
k
>   filefrag -v k # grep for an extent without unwritten || fail

right, that's what I see too in testing.

But would the coreutils install have done a preallocation of the destinat=
ion file?

Otherwise this looks like a different bug...

> This particular issue has been discussed so far at:
> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
> Note there it was stated there that ext4 had this
> fixed as of 2.6.39-rc1, so maybe there is something lurking?

ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I th=
ink I started looking into it, but it's clearly still broken.

Still, I don't know for sure what happened to Markus - did something prea=
llocate, in his case?

-Eric
=20
> cheers,
> P=E1draig.
>=20
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>=20



From MAILER-DAEMON Thu Apr 14 11:53:55 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAOrj-0007DQ-56
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 11:53:55 -0400
Received: from eggs.gnu.org ([140.186.70.92]:50228)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAOrh-0007Bv-4X
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:53:54 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAOrg-0007sm-4h
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:53:53 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:5923)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAOrf-0007sh-TS
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:53:52 -0400
Received: (qmail 95042 invoked from network); 14 Apr 2011 15:53:49 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 14 Apr 2011 15:53:49 -0000
Message-ID: <4DA7182B.8050409@draigBrady.com>
Date: Thu, 14 Apr 2011 16:52:11 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Eric Sandeen <sandeen@sandeen.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>	<20110414120635.GB1678@x4.trippels.de>	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
In-Reply-To: <4DA717B2.3020305@sandeen.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 15:53:54 -0000

On 14/04/11 16:50, Eric Sandeen wrote:
> On 4/14/11 9:59 AM, Pádraig Brady wrote:
>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>> Hi Pádraig,
>>>>>
>>>>> here you go:
>>>>> + filefrag -v unwritten.withdata                                                                                                                     
>>>>> Filesystem type is: ef53                                                                                                                             
>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
>>>>>  ext logical physical expected length flags                                                                                                          
>>>>>    0       0   274432            2560 unwritten,eof                                                                                                  
>>>>> unwritten.withdata: 1 extent found
>>>>>
>>>>> Please notice that this also happens with ext4 on the same kernel. 
>>>>> Btrfs is fine.
>>>>
>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>
>> So in summary, currently on (2.6.39-rc3), the following
>> will (usually?) report a single unwritten extent,
>> on both ext4 and xfs
>>
>>   fallocate -l 10MiB -n k
>>   dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
>>   filefrag -v k # grep for an extent without unwritten || fail
> 
> right, that's what I see too in testing.
> 
> But would the coreutils install have done a preallocation of the destination file?
> 
> Otherwise this looks like a different bug...
> 
>> This particular issue has been discussed so far at:
>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
>> Note there it was stated there that ext4 had this
>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
> 
> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
> 
> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?

Well that preallocate test is failing for him
when the source file is either on ext4 or xfs.
He noticed the issue initially on XFS when copying
none preallocated files, so XFS probably just has
the general issue of needing a sync before fiemap,
where as EXT4 just has this preallocate one
(though I've not seen it myself).

cheers,
Pádraig.


From MAILER-DAEMON Thu Apr 14 11:56:23 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAOu6-0007Uz-WF
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 11:56:23 -0400
Received: from eggs.gnu.org ([140.186.70.92]:59858)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAOu4-0007UG-9y
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:56:21 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAOu3-0008Gp-1V
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:56:20 -0400
Received: from sandeen.net ([63.231.237.45]:44173 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAOu2-0008GQ-DG
	for coreutils@gnu.org; Thu, 14 Apr 2011 11:56:19 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 1F94A48FDAA4;
	Thu, 14 Apr 2011 10:56:17 -0500 (CDT)
Message-ID: <4DA71920.9@sandeen.net>
Date: Thu, 14 Apr 2011 10:56:16 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>	<20110414120635.GB1678@x4.trippels.de>	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com>
In-Reply-To: <4DA7182B.8050409@draigBrady.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 15:56:21 -0000

On 4/14/11 10:52 AM, P=E1draig Brady wrote:
> On 14/04/11 16:50, Eric Sandeen wrote:
>> On 4/14/11 9:59 AM, P=E1draig Brady wrote:
>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>> Hi P=E1draig,
>>>>>>
>>>>>> here you go:
>>>>>> + filefrag -v unwritten.withdata                                  =
                                                                         =
         =20
>>>>>> Filesystem type is: ef53                                          =
                                                                         =
         =20
>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)=
                                                                         =
         =20
>>>>>>  ext logical physical expected length flags                       =
                                                                         =
         =20
>>>>>>    0       0   274432            2560 unwritten,eof               =
                                                                         =
         =20
>>>>>> unwritten.withdata: 1 extent found
>>>>>>
>>>>>> Please notice that this also happens with ext4 on the same kernel.=
=20
>>>>>> Btrfs is fine.
>>>>>
>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>
>>> So in summary, currently on (2.6.39-rc3), the following
>>> will (usually?) report a single unwritten extent,
>>> on both ext4 and xfs
>>>
>>>   fallocate -l 10MiB -n k
>>>   dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock of=
=3Dk
>>>   filefrag -v k # grep for an extent without unwritten || fail
>>
>> right, that's what I see too in testing.
>>
>> But would the coreutils install have done a preallocation of the desti=
nation file?
>>
>> Otherwise this looks like a different bug...
>>
>>> This particular issue has been discussed so far at:
>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
>>> Note there it was stated there that ext4 had this
>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>
>> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I=
 think I started looking into it, but it's clearly still broken.
>>
>> Still, I don't know for sure what happened to Markus - did something p=
reallocate, in his case?
>=20
> Well that preallocate test is failing for him
> when the source file is either on ext4 or xfs.
> He noticed the issue initially on XFS when copying
> none preallocated files, so XFS probably just has
> the general issue of needing a sync before fiemap,
> where as EXT4 just has this preallocate one
> (though I've not seen it myself).
>=20
> cheers,
> P=E1draig.
>=20

well, if I simply take the preallocation step out of the testcase, it wor=
ks fine on xfs without a sync.

So I still don't know what Markus hit...

-Eric


From MAILER-DAEMON Thu Apr 14 12:03:50 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAP1K-0002lj-CY
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:03:50 -0400
Received: from eggs.gnu.org ([140.186.70.92]:40639)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAP1I-0002lM-5H
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:03:49 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAP1G-000262-Tx
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:03:48 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:41674
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAP1G-00025i-HW
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:03:46 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:content-transfer-encoding:in-reply-to; q=dns/txt; s=
	beta; bh=utnN7A/7i22aY22YYByVUxdnAYfP8jgQoZ5RbsCadqA=; b=qiLCsgl
	jZRPjmRX1LZmkHdoPMZbRJwx+FNtAUvqCUZVX6EH47zyLPea75k0Y0V4yX2uToOp
	RQ+Oq0siFhUN0ewycM8fHQ5V02RFGr+vASNHPfs26tP3vMcGu+9R1hrwnpSRB3YA
	/XezBzjtoAJ3LGQVVIycj2u3Ywy6CpH634Ic=
Received: (qmail 21136 invoked from network); 14 Apr 2011 18:03:44 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 18:03:44 +0200
Date: Thu, 14 Apr 2011 18:03:43 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: Eric Sandeen <sandeen@sandeen.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414160343.GA12787@x4.trippels.de>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4DA71920.9@sandeen.net>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:03:49 -0000

On 2011.04.14 at 10:56 -0500, Eric Sandeen wrote:
> On 4/14/11 10:52 AM, Pádraig Brady wrote:
> > On 14/04/11 16:50, Eric Sandeen wrote:
> >> On 4/14/11 9:59 AM, Pádraig Brady wrote:
> >>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
> >>>>>> Hi Pádraig,
> >>>>>>
> >>>>>> here you go:
> >>>>>> + filefrag -v unwritten.withdata                                                                                                                     
> >>>>>> Filesystem type is: ef53                                                                                                                             
> >>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
> >>>>>>  ext logical physical expected length flags                                                                                                          
> >>>>>>    0       0   274432            2560 unwritten,eof                                                                                                  
> >>>>>> unwritten.withdata: 1 extent found
> >>>>>>
> >>>>>> Please notice that this also happens with ext4 on the same kernel. 
> >>>>>> Btrfs is fine.
> >>>>>
> >>>> `filefrag -vs` fixes the issue on both xfs and ext4.
> >>>
> >>> So in summary, currently on (2.6.39-rc3), the following
> >>> will (usually?) report a single unwritten extent,
> >>> on both ext4 and xfs
> >>>
> >>>   fallocate -l 10MiB -n k
> >>>   dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> >>>   filefrag -v k # grep for an extent without unwritten || fail
> >>
> >> right, that's what I see too in testing.
> >>
> >> But would the coreutils install have done a preallocation of the destination file?
> >>
> >> Otherwise this looks like a different bug...
> >>
> >>> This particular issue has been discussed so far at:
> >>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
> >>> Note there it was stated there that ext4 had this
> >>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
> >>
> >> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
> >>
> >> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
> > 
> > Well that preallocate test is failing for him
> > when the source file is either on ext4 or xfs.
> > He noticed the issue initially on XFS when copying
> > none preallocated files, so XFS probably just has
> > the general issue of needing a sync before fiemap,
> > where as EXT4 just has this preallocate one
> > (though I've not seen it myself).
> > 
> > cheers,
> > Pádraig.
> > 
> 
> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
> 
> So I still don't know what Markus hit...

Maybe it's delalloc:

x4 /tmp # dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k                                                                               
10+0 records in
10+0 records out
5120 bytes (5.1 kB) copied, 0.0021822 s, 2.3 MB/s
x4 /tmp # filefrag -v k 
Filesystem type is: 58465342
File size of k is 5120 (2 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0        0              16 unknown,delalloc,eof
k: 1 extent found
x4 /tmp # sync
x4 /tmp # filefrag -v k 
Filesystem type is: 58465342
File size of k is 5120 (2 blocks, blocksize 4096)
 ext logical physical expected length flags
   0       0 26960045              16 eof
k: 1 extent found

-- 
Markus


From MAILER-DAEMON Thu Apr 14 12:15:04 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPCC-0004xw-LN
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:15:04 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55704)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPCA-0004xl-PI
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:15:03 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPC8-0004VL-Ih
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:15:02 -0400
Received: from sandeen.net ([63.231.237.45]:39720 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPC8-0004V1-7u
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:15:00 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 4595948FDA82;
	Thu, 14 Apr 2011 11:14:59 -0500 (CDT)
Message-ID: <4DA71D83.5040601@sandeen.net>
Date: Thu, 14 Apr 2011 11:14:59 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<20110414160343.GA12787@x4.trippels.de>
In-Reply-To: <20110414160343.GA12787@x4.trippels.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:15:03 -0000

On 4/14/11 11:03 AM, Markus Trippelsdorf wrote:
> On 2011.04.14 at 10:56 -0500, Eric Sandeen wrote:

...

>> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
>>
>> So I still don't know what Markus hit...
> 
> Maybe it's delalloc:
> 
> x4 /tmp # dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k                                                                               
> 10+0 records in
> 10+0 records out
> 5120 bytes (5.1 kB) copied, 0.0021822 s, 2.3 MB/s
> x4 /tmp # filefrag -v k 
> Filesystem type is: 58465342
> File size of k is 5120 (2 blocks, blocksize 4096)
>  ext logical physical expected length flags
>    0       0        0              16 unknown,delalloc,eof
> k: 1 extent found
> x4 /tmp # sync
> x4 /tmp # filefrag -v k 
> Filesystem type is: 58465342
> File size of k is 5120 (2 blocks, blocksize 4096)
>  ext logical physical expected length flags
>    0       0 26960045              16 eof
> k: 1 extent found
> 

well filefrag still returns that it knows of a valid range from logical blocks 0 to 16, so cp should have something to copy...

-Eric


From MAILER-DAEMON Thu Apr 14 12:17:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPEr-00051A-By
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:17:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:40762)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAP1g-0002vi-T4
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:04:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAP1f-00029Y-Bq
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:04:12 -0400
Received: from mail-px0-f179.google.com ([209.85.212.179]:51376)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAP1f-000298-50
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:04:11 -0400
Received: by pxi2 with SMTP id 2so934204pxi.38
	for <coreutils@gnu.org>; Thu, 14 Apr 2011 09:04:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=KqHYLsAePM4jw1FjqbIQ1N4R6jC9Vy7nKhgsWVTUQes=;
	b=ov/KkYS3fc6mCC7XZ84wr96p2ibD59lLSyquxPLPhQ+or0FFC9+Hs5YffDNct8Y152
	bPdF/sIUgEaI4xOBZMUQKD8aZt8fpMaPiBB4/e46PksiwGQ8/1r0LBFTamwWa20ICCxa
	8DLfviaO1KQiBjRJ3n+4ovOpqlqIAxiXk5Md0=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=UT3znAME/29LDakV9fHucopoV4jL+XkF/sN2wA9kzO482KRXZ3h1aly0sDr/ZQtPNX
	kl4RRtxOJ2ZjpwrqhiMV/dtGalX9F5RBpdbSspTcW7E/wgX8TGZpWPKbcGbO+cvf+OLC
	+AReYeoWFl4WQBSG7s/crOtQsv6swGodZm4O0=
MIME-Version: 1.0
Received: by 10.68.15.130 with SMTP id x2mr543516pbc.35.1302797049087; Thu, 14
	Apr 2011 09:04:09 -0700 (PDT)
Received: by 10.68.60.7 with HTTP; Thu, 14 Apr 2011 09:04:09 -0700 (PDT)
In-Reply-To: <4DA71920.9@sandeen.net>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
Date: Fri, 15 Apr 2011 00:04:09 +0800
Message-ID: <BANLkTikW72q_2pEoVTmag8+gMsCcZ-fR8w@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Eric Sandeen <sandeen@sandeen.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.212.179
X-Mailman-Approved-At: Thu, 14 Apr 2011 12:17:49 -0400
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:04:14 -0000

2011/4/14 Eric Sandeen <sandeen@sandeen.net>:
> On 4/14/11 10:52 AM, P=E1draig Brady wrote:
>> On 14/04/11 16:50, Eric Sandeen wrote:
>>> On 4/14/11 9:59 AM, P=E1draig Brady wrote:
>>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>>> Hi P=E1draig,
>>>>>>>
>>>>>>> here you go:
>>>>>>> + filefrag -v unwritten.withdata
>>>>>>> Filesystem type is: ef53
>>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>>>> =A0ext logical physical expected length flags
>>>>>>> =A0 =A00 =A0 =A0 =A0 0 =A0 274432 =A0 =A0 =A0 =A0 =A0 =A02560 unwri=
tten,eof
>>>>>>> unwritten.withdata: 1 extent found
>>>>>>>
>>>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>>>> Btrfs is fine.
>>>>>>
>>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>>
>>>> So in summary, currently on (2.6.39-rc3), the following
>>>> will (usually?) report a single unwritten extent,
>>>> on both ext4 and xfs
>>>>
>>>> =A0 fallocate -l 10MiB -n k
>>>> =A0 dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock o=
f=3Dk
>>>> =A0 filefrag -v k # grep for an extent without unwritten || fail
>>>
>>> right, that's what I see too in testing.
>>>
>>> But would the coreutils install have done a preallocation of the destin=
ation file?
>>>
>>> Otherwise this looks like a different bug...
>>>
>>>> This particular issue has been discussed so far at:
>>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
>>>> Note there it was stated there that ext4 had this
>>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>>
>>> ext4 got a fix, but not xfs, I guess. =A0My poor brain can't remember, =
I think I started looking into it, but it's clearly still broken.
>>>
>>> Still, I don't know for sure what happened to Markus - did something pr=
eallocate, in his case?
>>
>> Well that preallocate test is failing for him
>> when the source file is either on ext4 or xfs.
>> He noticed the issue initially on XFS when copying
>> none preallocated files, so XFS probably just has
>> the general issue of needing a sync before fiemap,
>> where as EXT4 just has this preallocate one
>> (though I've not seen it myself).
>>
>> cheers,
>> P=E1draig.
>>
>
> well, if I simply take the preallocation step out of the testcase, it wor=
ks fine on xfs without a sync.
>
> So I still don't know what Markus hit...
Sorry for that my patch ignored fallocate.  The situation is like this:
An user allocated space for a file with fallocate, and write a small
part of it. and the written is not flushed.
The extent stays one unwritten extent in disk and memory with delayed
allocation.

In ext4 ext4_ext_walk_space() thinks an extent does not exist only if
there is no any extents on disk.  So ext4_ext_walk_space()
thinks there is a extent and   ext4_ext_fiemap_cb() thus ignores pagecache.

I think ext4_ext_walk_space() should take unwritten extent not exist.

Yongqiang.
>
> -Eric
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Thu Apr 14 12:17:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPEr-00051M-DO
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:17:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:54809)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAP83-00040m-Rq
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:10:49 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAP82-0003Kq-P9
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:10:47 -0400
Received: from mail-pw0-f41.google.com ([209.85.160.41]:47702)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAP82-0003Kk-DZ
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:10:46 -0400
Received: by pwi10 with SMTP id 10so929222pwi.0
	for <coreutils@gnu.org>; Thu, 14 Apr 2011 09:10:45 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=4gGrC37RqWqA/KN42xxySE2i5UgJ6doZ8pe9l56sMQ4=;
	b=OSTUehiXTyemzpZBGoDuhxrbZxSTCSQjt3cvCeNOGT09pYBJe/S0eIYCzwq4+t72nn
	qY/1VPM0PV3/rwUFGXNndlI7xmJctXEHH6sox09wfJXxjvjRj0J4HkeZo+YUhEPP3uhU
	KVp7qlHoZhaMewq8uubkpYz+rGjfEVxD2nNRk=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=jBGxnNfLrwDlSxDfEethwT5SWmzv2nqUrQAeAGOuDEM1j1BrOdfvl8G0PPXYyQ1Xu6
	ibgyJQk/W++RnFsyGfkLKWZ9KxWpfDVNkG9Oc47vakuSvh44OqPn7hlkqT0zd4Vn/AkU
	VF8tdpuxAvASGZaCdpe8u0uvWyyzkq4Fo2994=
MIME-Version: 1.0
Received: by 10.68.4.199 with SMTP id m7mr606072pbm.370.1302797445527; Thu, 14
	Apr 2011 09:10:45 -0700 (PDT)
Received: by 10.68.60.7 with HTTP; Thu, 14 Apr 2011 09:10:45 -0700 (PDT)
In-Reply-To: <BANLkTikW72q_2pEoVTmag8+gMsCcZ-fR8w@mail.gmail.com>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<BANLkTikW72q_2pEoVTmag8+gMsCcZ-fR8w@mail.gmail.com>
Date: Fri, 15 Apr 2011 00:10:45 +0800
Message-ID: <BANLkTimoLeWMJgNFGW+zdeUeJyZ-_+8fMQ@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Eric Sandeen <sandeen@sandeen.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.160.41
X-Mailman-Approved-At: Thu, 14 Apr 2011 12:17:49 -0400
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:10:49 -0000

Hi,

I am off my working computer.  Maybe below fix could fix the problem.

fs/ext4/extent.c
static int ext4_ext_walk_space(struct inode *inode, ext4_lblk_t block,
1877                 } else if (block >=3D le32_to_cpu(ex->ee_block)) {
1878                         /*
1879                          * some part of requested space is covered
1880                          * by found extent
1881                          */
1882                         start =3D block;
1883                         end =3D le32_to_cpu(ex->ee_block)
1884                                 + ext4_ext_get_actual_len(ex);
1885                         if (block + num < end)
1886                                 end =3D block + num;
       +                        if (!ext4_ext_is_uninitialized(ex))
1887                         exists =3D 1;
1888                 } else {
1889                         BUG();
1890                 }


On Fri, Apr 15, 2011 at 12:04 AM, Yongqiang Yang <xiaoqiangnk@gmail.com> wr=
ote:
> 2011/4/14 Eric Sandeen <sandeen@sandeen.net>:
>> On 4/14/11 10:52 AM, P=E1draig Brady wrote:
>>> On 14/04/11 16:50, Eric Sandeen wrote:
>>>> On 4/14/11 9:59 AM, P=E1draig Brady wrote:
>>>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>>>> Hi P=E1draig,
>>>>>>>>
>>>>>>>> here you go:
>>>>>>>> + filefrag -v unwritten.withdata
>>>>>>>> Filesystem type is: ef53
>>>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>>>>> =A0ext logical physical expected length flags
>>>>>>>> =A0 =A00 =A0 =A0 =A0 0 =A0 274432 =A0 =A0 =A0 =A0 =A0 =A02560 unwr=
itten,eof
>>>>>>>> unwritten.withdata: 1 extent found
>>>>>>>>
>>>>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>>>>> Btrfs is fine.
>>>>>>>
>>>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>>>
>>>>> So in summary, currently on (2.6.39-rc3), the following
>>>>> will (usually?) report a single unwritten extent,
>>>>> on both ext4 and xfs
>>>>>
>>>>> =A0 fallocate -l 10MiB -n k
>>>>> =A0 dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock =
of=3Dk
>>>>> =A0 filefrag -v k # grep for an extent without unwritten || fail
>>>>
>>>> right, that's what I see too in testing.
>>>>
>>>> But would the coreutils install have done a preallocation of the desti=
nation file?
>>>>
>>>> Otherwise this looks like a different bug...
>>>>
>>>>> This particular issue has been discussed so far at:
>>>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
>>>>> Note there it was stated there that ext4 had this
>>>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>>>
>>>> ext4 got a fix, but not xfs, I guess. =A0My poor brain can't remember,=
 I think I started looking into it, but it's clearly still broken.
>>>>
>>>> Still, I don't know for sure what happened to Markus - did something p=
reallocate, in his case?
>>>
>>> Well that preallocate test is failing for him
>>> when the source file is either on ext4 or xfs.
>>> He noticed the issue initially on XFS when copying
>>> none preallocated files, so XFS probably just has
>>> the general issue of needing a sync before fiemap,
>>> where as EXT4 just has this preallocate one
>>> (though I've not seen it myself).
>>>
>>> cheers,
>>> P=E1draig.
>>>
>>
>> well, if I simply take the preallocation step out of the testcase, it wo=
rks fine on xfs without a sync.
>>
>> So I still don't know what Markus hit...
> Sorry for that my patch ignored fallocate. =A0The situation is like this:
> An user allocated space for a file with fallocate, and write a small
> part of it. and the written is not flushed.
> The extent stays one unwritten extent in disk and memory with delayed
> allocation.
>
> In ext4 ext4_ext_walk_space() thinks an extent does not exist only if
> there is no any extents on disk. =A0So ext4_ext_walk_space()
> thinks there is a extent and =A0 ext4_ext_fiemap_cb() thus ignores pageca=
che.
>
> I think ext4_ext_walk_space() should take unwritten extent not exist.
>
> Yongqiang.
>>
>> -Eric
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>>
>
>
>
> --
> Best Wishes
> Yongqiang Yang
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Thu Apr 14 12:21:36 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPIW-0005QN-1c
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:21:36 -0400
Received: from eggs.gnu.org ([140.186.70.92]:48865)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAPIS-0005PU-8F
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:21:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAPIQ-0005EC-Vp
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:21:32 -0400
Received: from mail-pw0-f41.google.com ([209.85.160.41]:40698)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAPIQ-0005Dy-R4
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:21:30 -0400
Received: by pwi10 with SMTP id 10so934325pwi.0
	for <coreutils@gnu.org>; Thu, 14 Apr 2011 09:21:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=+/vQgSlqPd8P1FlLUL1MT8kfVhXp+Id1Efnew86VoRI=;
	b=eWAQ6ujm3KXFd/nb5hx11kuRnJCHUtfJS8D/Yd9FBU57pLblrN4WJW2JvBg4+JT+xM
	hgw3W63yR5mgT40q+0KloSyjqiQK81pZH7bPutx2gCsuHe72+47Gdz7K0WcSFmcyOgM8
	3XcOsfGEY1j1JVITkNlV6r4W3wRVth+b+Ws8M=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=uFq7EMsbN7XOuczL00FJXNZgJPt+a3DwDZpzoxGJ4c1o+RVWarV0JcV4FYljphsQAF
	u0a3j/TuSlGWk/uGDiN8v82zvShJ6wkImBz0VIPXKfNQKZGZ2QIVzF729G/umgTL6fTD
	nDJ02gYgdUidBz/h2hSpYmIkeRtk2RrLI5/G0=
MIME-Version: 1.0
Received: by 10.68.15.130 with SMTP id x2mr555705pbc.35.1302798089741; Thu, 14
	Apr 2011 09:21:29 -0700 (PDT)
Received: by 10.68.60.7 with HTTP; Thu, 14 Apr 2011 09:21:29 -0700 (PDT)
In-Reply-To: <20110414160343.GA12787@x4.trippels.de>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<20110414160343.GA12787@x4.trippels.de>
Date: Fri, 15 Apr 2011 00:21:29 +0800
Message-ID: <BANLkTimRxvBMp9M7zwiUY_UmmFOY5N58+A@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.160.41
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:21:34 -0000

On Fri, Apr 15, 2011 at 12:03 AM, Markus Trippelsdorf
<markus@trippelsdorf.de> wrote:
> On 2011.04.14 at 10:56 -0500, Eric Sandeen wrote:
>> On 4/14/11 10:52 AM, P=E1draig Brady wrote:
>> > On 14/04/11 16:50, Eric Sandeen wrote:
>> >> On 4/14/11 9:59 AM, P=E1draig Brady wrote:
>> >>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>> >>>>>> Hi P=E1draig,
>> >>>>>>
>> >>>>>> here you go:
>> >>>>>> + filefrag -v unwritten.withdata
>> >>>>>> Filesystem type is: ef53
>> >>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096=
)
>> >>>>>> =A0ext logical physical expected length flags
>> >>>>>> =A0 =A00 =A0 =A0 =A0 0 =A0 274432 =A0 =A0 =A0 =A0 =A0 =A02560 unw=
ritten,eof
>> >>>>>> unwritten.withdata: 1 extent found
>> >>>>>>
>> >>>>>> Please notice that this also happens with ext4 on the same kernel=
.
>> >>>>>> Btrfs is fine.
>> >>>>>
>> >>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>> >>>
>> >>> So in summary, currently on (2.6.39-rc3), the following
>> >>> will (usually?) report a single unwritten extent,
>> >>> on both ext4 and xfs
>> >>>
>> >>> =A0 fallocate -l 10MiB -n k
>> >>> =A0 dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock=
 of=3Dk
>> >>> =A0 filefrag -v k # grep for an extent without unwritten || fail
>> >>
>> >> right, that's what I see too in testing.
>> >>
>> >> But would the coreutils install have done a preallocation of the dest=
ination file?
>> >>
>> >> Otherwise this looks like a different bug...
>> >>
>> >>> This particular issue has been discussed so far at:
>> >>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
>> >>> Note there it was stated there that ext4 had this
>> >>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>> >>
>> >> ext4 got a fix, but not xfs, I guess. =A0My poor brain can't remember=
, I think I started looking into it, but it's clearly still broken.
>> >>
>> >> Still, I don't know for sure what happened to Markus - did something =
preallocate, in his case?
>> >
>> > Well that preallocate test is failing for him
>> > when the source file is either on ext4 or xfs.
>> > He noticed the issue initially on XFS when copying
>> > none preallocated files, so XFS probably just has
>> > the general issue of needing a sync before fiemap,
>> > where as EXT4 just has this preallocate one
>> > (though I've not seen it myself).
>> >
>> > cheers,
>> > P=E1draig.
>> >
>>
>> well, if I simply take the preallocation step out of the testcase, it wo=
rks fine on xfs without a sync.
>>
>> So I still don't know what Markus hit...
>
> Maybe it's delalloc:
>
> x4 /tmp # dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullbloc=
k of=3Dk
> 10+0 records in
> 10+0 records out
> 5120 bytes (5.1 kB) copied, 0.0021822 s, 2.3 MB/s
> x4 /tmp # filefrag -v k
> Filesystem type is: 58465342
> File size of k is 5120 (2 blocks, blocksize 4096)
> =A0ext logical physical expected length flags
> =A0 0 =A0 =A0 =A0 0 =A0 =A0 =A0 =A00 =A0 =A0 =A0 =A0 =A0 =A0 =A016 unknow=
n,delalloc,eof
> k: 1 extent found
> x4 /tmp # sync
> x4 /tmp # filefrag -v k
> Filesystem type is: 58465342
> File size of k is 5120 (2 blocks, blocksize 4096)
> =A0ext logical physical expected length flags
> =A0 0 0 26960045 16 eof
> k: 1 extent found
There is no preallocation in this case. Problem comes when fallocate
and dealloc work together.

>
> --
> Markus
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Thu Apr 14 12:29:04 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPPj-0007Fz-VE
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:29:04 -0400
Received: from eggs.gnu.org ([140.186.70.92]:40613)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAPPh-0007Ft-4s
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:29:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAPPf-0007C5-KJ
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:29:00 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:52247
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAPPf-0007Bv-97
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:28:59 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:content-transfer-encoding:in-reply-to; q=dns/txt; s=
	beta; bh=YIOQVYCvxUtbxWX9OKnab3VFkXIz+XpiyvmDb87MSB4=; b=T9Fo12d
	OWSzlaFrkgpkEK6WqVBZr51oTuhE9rw/H4n2Xx4o0c4hwxPJBCjLwzESLMkYLDSB
	NCXeBTCJinGink2iY/4I8OKIyhbX5Fchz9ufT9P9Y2f8V38mGohiwsxa/iSEb0WS
	7h2N/rznVagC8L1rABMDc+ToAv21OsAz1GDU=
Received: (qmail 27797 invoked from network); 14 Apr 2011 18:28:56 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 18:28:56 +0200
Date: Thu, 14 Apr 2011 18:28:56 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: Yongqiang Yang <xiaoqiangnk@gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414162856.GA21658@x4.trippels.de>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<20110414160343.GA12787@x4.trippels.de>
	<BANLkTimRxvBMp9M7zwiUY_UmmFOY5N58+A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <BANLkTimRxvBMp9M7zwiUY_UmmFOY5N58+A@mail.gmail.com>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:29:02 -0000

On 2011.04.15 at 00:21 +0800, Yongqiang Yang wrote:
> On Fri, Apr 15, 2011 at 12:03 AM, Markus Trippelsdorf
> <markus@trippelsdorf.de> wrote:
> > On 2011.04.14 at 10:56 -0500, Eric Sandeen wrote:
> >> On 4/14/11 10:52 AM, Pádraig Brady wrote:
> >> > On 14/04/11 16:50, Eric Sandeen wrote:
> >> >> On 4/14/11 9:59 AM, Pádraig Brady wrote:
> >> >>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
> >> >>>>>> Hi Pádraig,
> >> >>>>>>
> >> >>>>>> here you go:
> >> >>>>>> + filefrag -v unwritten.withdata
> >> >>>>>> Filesystem type is: ef53
> >> >>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
> >> >>>>>>  ext logical physical expected length flags
> >> >>>>>>    0       0   274432            2560 unwritten,eof
> >> >>>>>> unwritten.withdata: 1 extent found
> >> >>>>>>
> >> >>>>>> Please notice that this also happens with ext4 on the same kernel.
> >> >>>>>> Btrfs is fine.
> >> >>>>>
> >> >>>> `filefrag -vs` fixes the issue on both xfs and ext4.
> >> >>>
> >> >>> So in summary, currently on (2.6.39-rc3), the following
> >> >>> will (usually?) report a single unwritten extent,
> >> >>> on both ext4 and xfs
> >> >>>
> >> >>>   fallocate -l 10MiB -n k
> >> >>>   dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> >> >>>   filefrag -v k # grep for an extent without unwritten || fail
> >> >>
> >> >> right, that's what I see too in testing.
> >> >>
> >> >> But would the coreutils install have done a preallocation of the destination file?
> >> >>
> >> >> Otherwise this looks like a different bug...
> >> >>
> >> >>> This particular issue has been discussed so far at:
> >> >>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
> >> >>> Note there it was stated there that ext4 had this
> >> >>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
> >> >>
> >> >> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
> >> >>
> >> >> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
> >> >
> >> > Well that preallocate test is failing for him
> >> > when the source file is either on ext4 or xfs.
> >> > He noticed the issue initially on XFS when copying
> >> > none preallocated files, so XFS probably just has
> >> > the general issue of needing a sync before fiemap,
> >> > where as EXT4 just has this preallocate one
> >> > (though I've not seen it myself).
> >> >
> >> > cheers,
> >> > Pádraig.
> >> >
> >>
> >> well, if I simply take the preallocation step out of the testcase, it works fine on xfs without a sync.
> >>
> >> So I still don't know what Markus hit...
> >
> > Maybe it's delalloc:
> >
> > x4 /tmp # dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> > 10+0 records in
> > 10+0 records out
> > 5120 bytes (5.1 kB) copied, 0.0021822 s, 2.3 MB/s
> > x4 /tmp # filefrag -v k
> > Filesystem type is: 58465342
> > File size of k is 5120 (2 blocks, blocksize 4096)
> >  ext logical physical expected length flags
> >   0       0        0              16 unknown,delalloc,eof
> > k: 1 extent found
> > x4 /tmp # sync
> > x4 /tmp # filefrag -v k
> > Filesystem type is: 58465342
> > File size of k is 5120 (2 blocks, blocksize 4096)
> >  ext logical physical expected length flags
> >   0 0 26960045 16 eof
> > k: 1 extent found
> There is no preallocation in this case. Problem comes when fallocate
> and dealloc work together.

Yes, but we're still trying to find out what caused the zeros in the
binaries that coreutils installed on my system.

Now the failure only happens when I use "gold" as my linker. With GNU ld
everything is OK. But I thought this must be a timing issue, because
gold is faster and the binaries in coreutils-8.11/src are all fine.

-- 
Markus


From MAILER-DAEMON Thu Apr 14 12:31:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPRo-0007S2-JL
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:31:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60163)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPRm-0007Rv-Fb
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:31:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPRl-0007Vs-L0
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:31:10 -0400
Received: from sandeen.net ([63.231.237.45]:46116 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPRl-0007Vn-GH
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:31:09 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 4277048FDAA4;
	Thu, 14 Apr 2011 11:31:08 -0500 (CDT)
Message-ID: <4DA7214C.4070602@sandeen.net>
Date: Thu, 14 Apr 2011 11:31:08 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<20110414160343.GA12787@x4.trippels.de>
	<BANLkTimRxvBMp9M7zwiUY_UmmFOY5N58+A@mail.gmail.com>
	<20110414162856.GA21658@x4.trippels.de>
In-Reply-To: <20110414162856.GA21658@x4.trippels.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:31:11 -0000

On 4/14/11 11:28 AM, Markus Trippelsdorf wrote:

<snip>

> Yes, but we're still trying to find out what caused the zeros in the
> binaries that coreutils installed on my system.
> 
> Now the failure only happens when I use "gold" as my linker. With GNU ld
> everything is OK. But I thought this must be a timing issue, because
> gold is faster and the binaries in coreutils-8.11/src are all fine.

maybe xfs_bmap (or filefrag) of the binaries with both linkers would be instructive; are they laid out significantly differently?

does gold preallocate?

-Eric


From MAILER-DAEMON Thu Apr 14 12:48:19 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPiN-0000lV-Ec
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:48:19 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55221)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAPiK-0000lO-Iv
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:48:17 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAPiJ-0002YF-G9
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:48:16 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:48258
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QAPiJ-0002Y2-03
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:48:15 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:in-reply-to; q=dns/txt; s=beta; bh=pKwFCLPCHq3dymGp
	05qTSuS0MY/kvJujHgqrxCQec40=; b=KtoIHk3FRvFeBAhaJGWCgvn1llUWTDpz
	V9JnaiuZ0F2mbiwAjECS/dRxBnx/Q+Xvb7gkIasI8gfa8bn1dF3WG4Wn1kJ1R4aV
	Vasw5/ehyx6tLT9jZt+f6ZinC8u4s1rCuBKu8U1pRg7/U5Mo80uPeFRAVmjEGJlY
	V26YW2HowXQ=
Received: (qmail 32470 invoked from network); 14 Apr 2011 18:48:12 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 14 Apr 2011 18:48:12 +0200
Date: Thu, 14 Apr 2011 18:48:11 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: Eric Sandeen <sandeen@sandeen.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414164811.GB21658@x4.trippels.de>
References: <20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<20110414160343.GA12787@x4.trippels.de>
	<BANLkTimRxvBMp9M7zwiUY_UmmFOY5N58+A@mail.gmail.com>
	<20110414162856.GA21658@x4.trippels.de>
	<4DA7214C.4070602@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4DA7214C.4070602@sandeen.net>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:48:17 -0000

On 2011.04.14 at 11:31 -0500, Eric Sandeen wrote:
> On 4/14/11 11:28 AM, Markus Trippelsdorf wrote:
> 
> <snip>
> 
> > Yes, but we're still trying to find out what caused the zeros in the
> > binaries that coreutils installed on my system.
> > 
> > Now the failure only happens when I use "gold" as my linker. With GNU ld
> > everything is OK. But I thought this must be a timing issue, because
> > gold is faster and the binaries in coreutils-8.11/src are all fine.
> 
> maybe xfs_bmap (or filefrag) of the binaries with both linkers would be instructive; are they laid out significantly differently?
> 
> does gold preallocate?

Just checked and yes it does. That should explain the issue I was
seeing.

bool
Output_file::map_no_anonymous()
{
  const int o = this->o_;

  // If the output file is not a regular file, don't try to mmap it;
  // instead, we'll mmap a block of memory (an anonymous buffer), and
  // then later write the buffer to the file.
  void* base;
  struct stat statbuf;
  if (o == STDOUT_FILENO || o == STDERR_FILENO
      || ::fstat(o, &statbuf) != 0
      || !S_ISREG(statbuf.st_mode)
      || this->is_temporary_)
    return false;

  // Ensure that we have disk space available for the file.  If we
  // don't do this, it is possible that we will call munmap, close,
  // and exit with dirty buffers still in the cache with no assigned
  // disk blocks.  If the disk is out of space at that point, the
  // output file will wind up incomplete, but we will have already
  // exited.  The alternative to fallocate would be to use fdatasync,
  // but that would be a more significant performance hit.
  if (::posix_fallocate(o, 0, this->file_size_) < 0)
    gold_fatal(_("%s: %s"), this->name_, strerror(errno));

  // Map the file into memory.
  base = ::mmap(NULL, this->file_size_, PROT_READ | PROT_WRITE,
		MAP_SHARED, o, 0);

  // The mmap call might fail because of file system issues: the file
  // system might not support mmap at all, or it might not support
  // mmap with PROT_WRITE.
  if (base == MAP_FAILED)
    return false;

  this->map_is_anonymous_ = false;
  this->base_ = static_cast<unsigned char*>(base);
  return true;
}


-- 
Markus


From MAILER-DAEMON Thu Apr 14 12:49:25 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAPjR-0001tz-BQ
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 12:49:25 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55442)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPjO-0001sO-IN
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:49:23 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPjN-0002fD-KY
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:49:22 -0400
Received: from sandeen.net ([63.231.237.45]:54881 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QAPjN-0002f5-Bm
	for coreutils@gnu.org; Thu, 14 Apr 2011 12:49:21 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id D3AFD48FDA82;
	Thu, 14 Apr 2011 11:49:19 -0500 (CDT)
Message-ID: <4DA7258F.1080302@sandeen.net>
Date: Thu, 14 Apr 2011 11:49:19 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <4DA71920.9@sandeen.net>
	<20110414160343.GA12787@x4.trippels.de>
	<BANLkTimRxvBMp9M7zwiUY_UmmFOY5N58+A@mail.gmail.com>
	<20110414162856.GA21658@x4.trippels.de>
	<4DA7214C.4070602@sandeen.net>
	<20110414164811.GB21658@x4.trippels.de>
In-Reply-To: <20110414164811.GB21658@x4.trippels.de>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 16:49:23 -0000

On 4/14/11 11:48 AM, Markus Trippelsdorf wrote:
> On 2011.04.14 at 11:31 -0500, Eric Sandeen wrote:
>> On 4/14/11 11:28 AM, Markus Trippelsdorf wrote:
>>
>> <snip>
>>
>>> Yes, but we're still trying to find out what caused the zeros in the
>>> binaries that coreutils installed on my system.
>>>
>>> Now the failure only happens when I use "gold" as my linker. With GNU ld
>>> everything is OK. But I thought this must be a timing issue, because
>>> gold is faster and the binaries in coreutils-8.11/src are all fine.
>>
>> maybe xfs_bmap (or filefrag) of the binaries with both linkers would be instructive; are they laid out significantly differently?
>>
>> does gold preallocate?
> 
> Just checked and yes it does. That should explain the issue I was
> seeing.

Well, mystery solved there, at least!

Now for the fixing part :)

Thanks for checking, at least my view of the world is still intact ;)

-Eric

> bool
> Output_file::map_no_anonymous()
> {
>   const int o = this->o_;
> 
>   // If the output file is not a regular file, don't try to mmap it;
>   // instead, we'll mmap a block of memory (an anonymous buffer), and
>   // then later write the buffer to the file.
>   void* base;
>   struct stat statbuf;
>   if (o == STDOUT_FILENO || o == STDERR_FILENO
>       || ::fstat(o, &statbuf) != 0
>       || !S_ISREG(statbuf.st_mode)
>       || this->is_temporary_)
>     return false;
> 
>   // Ensure that we have disk space available for the file.  If we
>   // don't do this, it is possible that we will call munmap, close,
>   // and exit with dirty buffers still in the cache with no assigned
>   // disk blocks.  If the disk is out of space at that point, the
>   // output file will wind up incomplete, but we will have already
>   // exited.  The alternative to fallocate would be to use fdatasync,
>   // but that would be a more significant performance hit.
>   if (::posix_fallocate(o, 0, this->file_size_) < 0)
>     gold_fatal(_("%s: %s"), this->name_, strerror(errno));
> 
>   // Map the file into memory.
>   base = ::mmap(NULL, this->file_size_, PROT_READ | PROT_WRITE,
> 		MAP_SHARED, o, 0);
> 
>   // The mmap call might fail because of file system issues: the file
>   // system might not support mmap at all, or it might not support
>   // mmap with PROT_WRITE.
>   if (base == MAP_FAILED)
>     return false;
> 
>   this->map_is_anonymous_ = false;
>   this->base_ = static_cast<unsigned char*>(base);
>   return true;
> }
> 
> 



From MAILER-DAEMON Thu Apr 14 13:27:37 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAQKP-0006qT-C8
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 13:27:37 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35503)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QAQKN-0006qI-RT
	for coreutils@gnu.org; Thu, 14 Apr 2011 13:27:36 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QAQKM-0001na-Ci
	for coreutils@gnu.org; Thu, 14 Apr 2011 13:27:35 -0400
Received: from mx.meyering.net ([82.230.74.64]:43727)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QAQKM-0001kq-0J
	for coreutils@gnu.org; Thu, 14 Apr 2011 13:27:34 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id F275560110; Thu, 14 Apr 2011 19:27:31 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
In-Reply-To: <4DA7182B.8050409@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Thu, 14 Apr 2011 16:52:11 +0100")
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com>
Date: Thu, 14 Apr 2011 19:27:31 +0200
Message-ID: <878vvcspz0.fsf@rho.meyering.net>
Lines: 125
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, Markus Trippelsdorf <markus@trippelsdorf.de>,
	xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 17:27:37 -0000

P=E1draig Brady wrote:

> On 14/04/11 16:50, Eric Sandeen wrote:
>> On 4/14/11 9:59 AM, P=E1draig Brady wrote:
>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>> Hi P=E1draig,
>>>>>>
>>>>>> here you go:
>>>>>> + filefrag -v unwritten.withdata
>>>>>> Filesystem type is: ef53
>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)
>>>>>>  ext logical physical expected length flags
>>>>>>    0       0   274432            2560 unwritten,eof
>>>>>> unwritten.withdata: 1 extent found
>>>>>>
>>>>>> Please notice that this also happens with ext4 on the same kernel.
>>>>>> Btrfs is fine.
>>>>>
>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>
>>> So in summary, currently on (2.6.39-rc3), the following
>>> will (usually?) report a single unwritten extent,
>>> on both ext4 and xfs
>>>
>>>   fallocate -l 10MiB -n k
>>>   dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock of=
=3Dk
>>>   filefrag -v k # grep for an extent without unwritten || fail
>>
>> right, that's what I see too in testing.
>>
>> But would the coreutils install have done a preallocation of the
>> destination file?
>>
>> Otherwise this looks like a different bug...
>>
>>> This particular issue has been discussed so far at:
>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
>>> Note there it was stated there that ext4 had this
>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>
>> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember,
>> I think I started looking into it, but it's clearly still broken.
>>
>> Still, I don't know for sure what happened to Markus - did something
>> preallocate, in his case?
>
> Well that preallocate test is failing for him
> when the source file is either on ext4 or xfs.
> He noticed the issue initially on XFS when copying
> none preallocated files, so XFS probably just has
> the general issue of needing a sync before fiemap,
> where as EXT4 just has this preallocate one
> (though I've not seen it myself).

FYI, I see the same failure now using ext3 (and but not w/ext4)
with rawhide's 2.6.39-0.rc2.git0.0.fc16.x86_64:

  + df -t ext3 .
  + require_root_
  + uid_is_privileged_
  ++ id -u
  + my_uid=3D0
  + case $my_uid in
  + NON_ROOT_USERNAME=3Dnobody
  ++ id -g nobody
  + NON_ROOT_GROUP=3D99
  + cwd=3D/t/m/ext3/tmp/coreutils-8.11.1-5995ed-dirty/tests/gt-sparse-fiema=
p.Qhjo
  + skip=3D0
  + dd if=3D/dev/zero of=3Dblob bs=3D32k count=3D1000
  1000+0 records in
  1000+0 records out
  32768000 bytes (33 MB) copied, 1.02932 s, 31.8 MB/s
  + mkdir mnt
  + mkfs -t ext4 -F blob
  mke2fs 1.41.14 (22-Dec-2010)
  Filesystem label=3D
  OS type: Linux
  Block size=3D1024 (log=3D0)
  Fragment size=3D1024 (log=3D0)
  Stride=3D0 blocks, Stripe width=3D0 blocks
  8000 inodes, 32000 blocks
  1600 blocks (5.00%) reserved for the super user
  First data block=3D1
  Maximum filesystem blocks=3D32768000
  4 block groups
  8192 blocks per group, 8192 fragments per group
  2000 inodes per group
  Superblock backups stored on blocks:
          8193, 24577

  Writing inode tables: done
  Creating journal (1024 blocks): done
  Writing superblocks and filesystem accounting information: done

  This filesystem will be automatically checked every 21 mounts or
  180 days, whichever comes first.  Use tune2fs -c or -i to override.
  + mount -oloop blob mnt
  + cd mnt
  + echo test
  + test -s f
  + test 0 =3D 1
  ++ seq 1 2 21
  + for i in '$(seq 1 2 21)'
  + for j in 1 2 31 100
  + perl -e 'BEGIN { $n =3D 1 * 1024; *F =3D *STDOUT }' -e 'for (1..1) { sy=
sseek (*F, $n, 1)' -e '&& syswrite (*F, chr($_)x$n) or die "$!"}'
  + cp --sparse=3Dalways j1 j2
  + cmp j1 j2
  + filefrag -vs j1
  + grep -F extent
  + filefrag -v j1
  + filefrag -vs j2
  + f ff1
  + perl /t/m/ext3/tmp/coreutils-8.11.1-5995ed-dirty/tests/filefrag-extent-=
compare
  + sed 's/ [a-z,][a-z,]*$//' ff1
  + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""=
}'
  + f ff2
  + sed 's/ [a-z,][a-z,]*$//' ff2
  + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""=
}'
  + test 0 =3D 1
  + for j in 1 2 31 100
  + perl -e 'BEGIN { $n =3D 1 * 1024; *F =3D *STDOUT }' -e 'for (1..2) { sy=
sseek (*F, $n, 1)' -e '&& syswrite (*F, chr($_)x$n) or die "$!"}'
  + cp --sparse=3Dalways j1 j2
  + cmp j1 j2
  j1 j2 differ: char 1, line 1     <<<<=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
  + fail=3D1


From MAILER-DAEMON Thu Apr 14 15:15:37 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAS0v-0008At-Tm
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 15:15:37 -0400
Received: from eggs.gnu.org ([140.186.70.92]:46460)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAS0u-00088U-DM
	for coreutils@gnu.org; Thu, 14 Apr 2011 15:15:37 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAS0t-0007Lw-FA
	for coreutils@gnu.org; Thu, 14 Apr 2011 15:15:36 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:19037)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAS0t-0007Ls-5y
	for coreutils@gnu.org; Thu, 14 Apr 2011 15:15:35 -0400
Received: (qmail 31448 invoked from network); 14 Apr 2011 19:15:33 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 14 Apr 2011 19:15:33 -0000
Message-ID: <4DA74771.5010103@draigBrady.com>
Date: Thu, 14 Apr 2011 20:13:53 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>	<20110414120635.GB1678@x4.trippels.de>	<20110414140222.GB1679@x4.trippels.de>	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net>	<4DA7182B.8050409@draigBrady.com>
	<878vvcspz0.fsf@rho.meyering.net>
In-Reply-To: <878vvcspz0.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, Markus Trippelsdorf <markus@trippelsdorf.de>,
	xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 19:15:37 -0000

On 14/04/11 18:27, Jim Meyering wrote:
> 
> FYI, I see the same failure now using ext3 (and but not w/ext4)
> with rawhide's 2.6.39-0.rc2.git0.0.fc16.x86_64:
> 
>   + df -t ext3 .

Not with ext3 ...

>   + mkfs -t ext4 -F blob

... but with ext4 loop back

>   + perl -e 'BEGIN { $n = 1 * 1024; *F = *STDOUT }' -e 'for (1..1) { sysseek (*F, $n, 1)' -e '&& syswrite (*F, chr($_)x$n) or die "$!"}'

>   + cp --sparse=always j1 j2
>   + cmp j1 j2
>   j1 j2 differ: char 1, line 1     <<<<================

But there was no preallocation done above.
So this was the original sync issue, which doesn't seem to be working :(
Is there a chance the rawhide kernel hasn't included that change?
Unlikely as it's 2.6.39-rc2.

cheers,
Pádraig.

p.s. I will do some checking with ext3 to ensure everything is OK


From MAILER-DAEMON Thu Apr 14 15:39:47 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QASOJ-0002xF-1d
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 15:39:47 -0400
Received: from eggs.gnu.org ([140.186.70.92]:45074)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QASOG-0002vB-JM
	for coreutils@gnu.org; Thu, 14 Apr 2011 15:39:45 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QASOF-0004AI-Qj
	for coreutils@gnu.org; Thu, 14 Apr 2011 15:39:44 -0400
Received: from mx.meyering.net ([82.230.74.64]:55490)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QASOF-000481-LF
	for coreutils@gnu.org; Thu, 14 Apr 2011 15:39:43 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id C2708600B2; Thu, 14 Apr 2011 21:39:39 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
In-Reply-To: <878vvcspz0.fsf@rho.meyering.net> (Jim Meyering's message of
	"Thu, 14 Apr 2011 19:27:31 +0200")
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<4DA7182B.8050409@draigBrady.com> <878vvcspz0.fsf@rho.meyering.net>
Date: Thu, 14 Apr 2011 21:39:39 +0200
Message-ID: <8739lksjus.fsf@rho.meyering.net>
Lines: 21
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, Markus Trippelsdorf <markus@trippelsdorf.de>,
	xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 19:39:45 -0000

Jim Meyering wrote:
...
> FYI, I see the same failure now using ext3 (and but not w/ext4)
> with rawhide's 2.6.39-0.rc2.git0.0.fc16.x86_64:

Correction.
The failure below is not on ext3, but on a loopback-mounted
ext4 file system.

The test detected that the current FS was ext3 -- and it's known
that this test is not useful on ext3 -- and since I was running it
as root, it creates a loopback ext4 file system on which to run the test.

>   + df -t ext3 .
>   + require_root_
>   + uid_is_privileged_
>   ++ id -u
>   + my_uid=0

However, other strange things are happening here.
I'm investigating.


From MAILER-DAEMON Thu Apr 14 19:14:08 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAVjj-0007N8-Vw
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 19:14:07 -0400
Received: from eggs.gnu.org ([140.186.70.92]:38225)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAVVP-0003F9-Ej
	for coreutils@gnu.org; Thu, 14 Apr 2011 18:59:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAVVO-0006SU-4Y
	for coreutils@gnu.org; Thu, 14 Apr 2011 18:59:19 -0400
Received: from ipmail06.adl2.internode.on.net ([150.101.137.129]:18329)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAVVN-0006S0-OT
	for coreutils@gnu.org; Thu, 14 Apr 2011 18:59:18 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Ao8EAHN5p015LHHJgWdsb2JhbAClexUBARYmJcQeDoVgBA
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl2.internode.on.net with ESMTP; 15 Apr 2011 08:29:06 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QAVVB-0006Ot-1G; Fri, 15 Apr 2011 08:59:05 +1000
Date: Fri, 15 Apr 2011 08:59:04 +1000
From: Dave Chinner <david@fromorbit.com>
To: Eric Sandeen <sandeen@sandeen.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110414225904.GK21395@dastard>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4DA717B2.3020305@sandeen.net>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.129
X-Mailman-Approved-At: Thu, 14 Apr 2011 19:14:06 -0400
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>, xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 22:59:20 -0000

On Thu, Apr 14, 2011 at 10:50:10AM -0500, Eric Sandeen wrote:
> On 4/14/11 9:59 AM, Pádraig Brady wrote:
> > On 14/04/11 15:02, Markus Trippelsdorf wrote:
> >>>> Hi Pádraig,
> >>>>
> >>>> here you go:
> >>>> + filefrag -v unwritten.withdata                                                                                                                     
> >>>> Filesystem type is: ef53                                                                                                                             
> >>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
> >>>>  ext logical physical expected length flags                                                                                                          
> >>>>    0       0   274432            2560 unwritten,eof                                                                                                  
> >>>> unwritten.withdata: 1 extent found
> >>>>
> >>>> Please notice that this also happens with ext4 on the same kernel. 
> >>>> Btrfs is fine.
> >>>
> >> `filefrag -vs` fixes the issue on both xfs and ext4.
> > 
> > So in summary, currently on (2.6.39-rc3), the following
> > will (usually?) report a single unwritten extent,
> > on both ext4 and xfs
> > 
> >   fallocate -l 10MiB -n k
> >   dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> >   filefrag -v k # grep for an extent without unwritten || fail
> 
> right, that's what I see too in testing.
> 
> But would the coreutils install have done a preallocation of the destination file?
> 
> Otherwise this looks like a different bug...
> 
> > This particular issue has been discussed so far at:
> > http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
> > Note there it was stated there that ext4 had this
> > fixed as of 2.6.39-rc1, so maybe there is something lurking?
> 
> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
> 
> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?

Unwritten extent mapping behaves in an unexpected way due to
buffered writeback not occurring immediately. Extent conversion
doesn't occur until the data is on disk, and for buffered IO you
need an fdatasync to ensure that has occurred.

That is: 

$ xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c "bmap -vp" /mnt/test/foo
wrote 5120/5120 bytes at offset 0
5 KiB, 2 ops; 0.0000 sec (62.600 MiB/sec and 25641.0256 ops/sec)
/mnt/test/foo:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
   0: [0..20479]:      268984..289463    0 (268984..289463) 20480 10000

Data has not been written yet, so it is still unwritten. The same
test with a fsync shows:

$ sudo xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c fsync -c "bmap -vp" /mnt/test/foo
wrote 5120/5120 bytes at offset 0
5 KiB, 2 ops; 0.0000 sec (87.193 MiB/sec and 35714.2857 ops/sec)
/mnt/test/foo:
 EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
   0: [0..15]:         268984..268999    0 (268984..268999)    16 00000
   1: [16..20479]:     269000..289463    0 (269000..289463) 20464 10000

Everything is fine.

So this seems like an application error to me. If you are going to
use fiemap to determine what ranges to copy, then you have to
fdatasync the source file first to guarantee that preallocated
extents have been converted to written state before mapping the
file....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Thu Apr 14 19:31:33 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAW0b-0000xQ-GW
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 19:31:33 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35318)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAW0Y-0000wL-8F
	for coreutils@gnu.org; Thu, 14 Apr 2011 19:31:31 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAW0X-0003th-3f
	for coreutils@gnu.org; Thu, 14 Apr 2011 19:31:30 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:16096)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QAW0W-0003td-Mc
	for coreutils@gnu.org; Thu, 14 Apr 2011 19:31:29 -0400
Received: (qmail 66471 invoked from network); 14 Apr 2011 23:31:26 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 14 Apr 2011 23:31:26 -0000
Message-ID: <4DA7836A.5040604@draigBrady.com>
Date: Fri, 15 Apr 2011 00:29:46 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Dave Chinner <david@fromorbit.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net> <20110414225904.GK21395@dastard>
In-Reply-To: <20110414225904.GK21395@dastard>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, Markus Trippelsdorf <markus@trippelsdorf.de>,
	xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 14 Apr 2011 23:31:31 -0000

On 14/04/11 23:59, Dave Chinner wrote:
> On Thu, Apr 14, 2011 at 10:50:10AM -0500, Eric Sandeen wrote:
>> On 4/14/11 9:59 AM, Pádraig Brady wrote:
>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>> Hi Pádraig,
>>>>>>
>>>>>> here you go:
>>>>>> + filefrag -v unwritten.withdata                                                                                                                     
>>>>>> Filesystem type is: ef53                                                                                                                             
>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
>>>>>>  ext logical physical expected length flags                                                                                                          
>>>>>>    0       0   274432            2560 unwritten,eof                                                                                                  
>>>>>> unwritten.withdata: 1 extent found
>>>>>>
>>>>>> Please notice that this also happens with ext4 on the same kernel. 
>>>>>> Btrfs is fine.
>>>>>
>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>
>>> So in summary, currently on (2.6.39-rc3), the following
>>> will (usually?) report a single unwritten extent,
>>> on both ext4 and xfs
>>>
>>>   fallocate -l 10MiB -n k
>>>   dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
>>>   filefrag -v k # grep for an extent without unwritten || fail
>>
>> right, that's what I see too in testing.
>>
>> But would the coreutils install have done a preallocation of the destination file?
>>
>> Otherwise this looks like a different bug...
>>
>>> This particular issue has been discussed so far at:
>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
>>> Note there it was stated there that ext4 had this
>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>
>> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
>>
>> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
> 
> Unwritten extent mapping behaves in an unexpected way due to
> buffered writeback not occurring immediately. Extent conversion
> doesn't occur until the data is on disk, and for buffered IO you
> need an fdatasync to ensure that has occurred.
> 
> That is: 
> 
> $ xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c "bmap -vp" /mnt/test/foo
> wrote 5120/5120 bytes at offset 0
> 5 KiB, 2 ops; 0.0000 sec (62.600 MiB/sec and 25641.0256 ops/sec)
> /mnt/test/foo:
>  EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
>    0: [0..20479]:      268984..289463    0 (268984..289463) 20480 10000
> 
> Data has not been written yet, so it is still unwritten. The same
> test with a fsync shows:
> 
> $ sudo xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c fsync -c "bmap -vp" /mnt/test/foo
> wrote 5120/5120 bytes at offset 0
> 5 KiB, 2 ops; 0.0000 sec (87.193 MiB/sec and 35714.2857 ops/sec)
> /mnt/test/foo:
>  EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
>    0: [0..15]:         268984..268999    0 (268984..268999)    16 00000
>    1: [16..20479]:     269000..289463    0 (269000..289463) 20464 10000
> 
> Everything is fine.
> 
> So this seems like an application error to me. If you are going to
> use fiemap to determine what ranges to copy, then you have to
> fdatasync the source file first to guarantee that preallocated
> extents have been converted to written state before mapping the
> file....

Well IMHO there should be a difference between
knowing where you are going to write, and actually writing to disk.
I.E. one shouldn't need to write the whole way to the device
before returning a valid fiemap.  If a particular file system
implementation needs to sync to return a valid fiemap,
then it should be implicit.

cheers,
Pádraig.


From MAILER-DAEMON Thu Apr 14 20:09:51 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAWbf-0003uv-1e
	for mharc-coreutils@gnu.org; Thu, 14 Apr 2011 20:09:51 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41705)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAWbd-0003u4-60
	for coreutils@gnu.org; Thu, 14 Apr 2011 20:09:50 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAWbb-00025w-U0
	for coreutils@gnu.org; Thu, 14 Apr 2011 20:09:49 -0400
Received: from ipmail06.adl2.internode.on.net ([150.101.137.129]:54886)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAWbb-00023d-7W
	for coreutils@gnu.org; Thu, 14 Apr 2011 20:09:47 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Ao8EABmLp015LHHJgWdsb2JhbAClfBUBARYmJcREDoVgBA
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl2.internode.on.net with ESMTP; 15 Apr 2011 09:39:42 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QAWbU-0006Vb-2j; Fri, 15 Apr 2011 10:09:40 +1000
Date: Fri, 15 Apr 2011 10:09:40 +1000
From: Dave Chinner <david@fromorbit.com>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110415000940.GL21395@dastard>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4DA7836A.5040604@draigBrady.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.129
Cc: linux-ext4@vger.kernel.org, Eric Sandeen <sandeen@sandeen.net>,
	coreutils@gnu.org, Markus Trippelsdorf <markus@trippelsdorf.de>,
	xfs-oss <xfs@oss.sgi.com>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 00:09:50 -0000

On Fri, Apr 15, 2011 at 12:29:46AM +0100, Pádraig Brady wrote:
> On 14/04/11 23:59, Dave Chinner wrote:
> > On Thu, Apr 14, 2011 at 10:50:10AM -0500, Eric Sandeen wrote:
> >> On 4/14/11 9:59 AM, Pádraig Brady wrote:
> >>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
> >>>>>> Hi Pádraig,
> >>>>>>
> >>>>>> here you go:
> >>>>>> + filefrag -v unwritten.withdata                                                                                                                     
> >>>>>> Filesystem type is: ef53                                                                                                                             
> >>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096)                                                                                   
> >>>>>>  ext logical physical expected length flags                                                                                                          
> >>>>>>    0       0   274432            2560 unwritten,eof                                                                                                  
> >>>>>> unwritten.withdata: 1 extent found
> >>>>>>
> >>>>>> Please notice that this also happens with ext4 on the same kernel. 
> >>>>>> Btrfs is fine.
> >>>>>
> >>>> `filefrag -vs` fixes the issue on both xfs and ext4.
> >>>
> >>> So in summary, currently on (2.6.39-rc3), the following
> >>> will (usually?) report a single unwritten extent,
> >>> on both ext4 and xfs
> >>>
> >>>   fallocate -l 10MiB -n k
> >>>   dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=k
> >>>   filefrag -v k # grep for an extent without unwritten || fail
> >>
> >> right, that's what I see too in testing.
> >>
> >> But would the coreutils install have done a preallocation of the destination file?
> >>
> >> Otherwise this looks like a different bug...
> >>
> >>> This particular issue has been discussed so far at:
> >>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=8411
> >>> Note there it was stated there that ext4 had this
> >>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
> >>
> >> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I think I started looking into it, but it's clearly still broken.
> >>
> >> Still, I don't know for sure what happened to Markus - did something preallocate, in his case?
> > 
> > Unwritten extent mapping behaves in an unexpected way due to
> > buffered writeback not occurring immediately. Extent conversion
> > doesn't occur until the data is on disk, and for buffered IO you
> > need an fdatasync to ensure that has occurred.
> > 
> > That is: 
> > 
> > $ xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c "bmap -vp" /mnt/test/foo
> > wrote 5120/5120 bytes at offset 0
> > 5 KiB, 2 ops; 0.0000 sec (62.600 MiB/sec and 25641.0256 ops/sec)
> > /mnt/test/foo:
> >  EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
> >    0: [0..20479]:      268984..289463    0 (268984..289463) 20480 10000
> > 
> > Data has not been written yet, so it is still unwritten. The same
> > test with a fsync shows:
> > 
> > $ sudo xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c fsync -c "bmap -vp" /mnt/test/foo
> > wrote 5120/5120 bytes at offset 0
> > 5 KiB, 2 ops; 0.0000 sec (87.193 MiB/sec and 35714.2857 ops/sec)
> > /mnt/test/foo:
> >  EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
> >    0: [0..15]:         268984..268999    0 (268984..268999)    16 00000
> >    1: [16..20479]:     269000..289463    0 (269000..289463) 20464 10000
> > 
> > Everything is fine.
> > 
> > So this seems like an application error to me. If you are going to
> > use fiemap to determine what ranges to copy, then you have to
> > fdatasync the source file first to guarantee that preallocated
> > extents have been converted to written state before mapping the
> > file....
> 
> Well IMHO there should be a difference between
> knowing where you are going to write, and actually writing to disk.
> I.E. one shouldn't need to write the whole way to the device
> before returning a valid fiemap.  If a particular file system
> implementation needs to sync to return a valid fiemap,
> then it should be implicit.

No, this was explicitly laid out in the fiemap interface discussions
- it's up to the applicaiton to decide if it needs to do a sync
first. That's what the FIEMAP_FLAG_SYNC control flag is for.
This forces the fiemap call to do a fsync _before_ getting the
mapping. If you want to know the exact layout of the file is, then
you must use this flag.

Even so, it is recognised that this is racy - any use of the block
map has a time-of-read-to-time-of-use race condition that means you
have to _verify_ the copy after it completes. FYI, that's what
xfs_fsr does when copying based on extent maps - if the inode has
changed in _any way_ during the copy, it aborts the copy of that
file.

i.e. using fiemap for copying is at best a *hint* about the regions
that need copying, and it is in no way a guarantee that you'll get
all the information you need to make accurate copy even if you do
use the synchronous variant.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Fri Apr 15 04:53:52 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAemm-0003OK-J6
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 04:53:52 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47260)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QAeml-0003OB-B1
	for coreutils@gnu.org; Fri, 15 Apr 2011 04:53:52 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QAemk-0001xD-Aj
	for coreutils@gnu.org; Fri, 15 Apr 2011 04:53:51 -0400
Received: from mx.meyering.net ([82.230.74.64]:49253)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QAemj-0001x7-Va
	for coreutils@gnu.org; Fri, 15 Apr 2011 04:53:50 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 795C3600D4; Fri, 15 Apr 2011 10:53:48 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Dave Chinner <david@fromorbit.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
In-Reply-To: <20110415000940.GL21395@dastard> (Dave Chinner's message of "Fri, 
	15 Apr 2011 10:09:40 +1000")
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
Date: Fri, 15 Apr 2011 10:53:48 +0200
Message-ID: <87sjtjq4ir.fsf@rho.meyering.net>
Lines: 39
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	coreutils@gnu.org, linux-ext4@vger.kernel.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 08:53:52 -0000

Dave Chinner wrote:
> On Fri, Apr 15, 2011 at 12:29:46AM +0100, P=E1draig Brady wrote:
...
>> Well IMHO there should be a difference between
>> knowing where you are going to write, and actually writing to disk.
>> I.E. one shouldn't need to write the whole way to the device
>> before returning a valid fiemap.  If a particular file system
>> implementation needs to sync to return a valid fiemap,
>> then it should be implicit.
>
> No, this was explicitly laid out in the fiemap interface discussions
> - it's up to the applicaiton to decide if it needs to do a sync
> first. That's what the FIEMAP_FLAG_SYNC control flag is for.
> This forces the fiemap call to do a fsync _before_ getting the
> mapping. If you want to know the exact layout of the file is, then
> you must use this flag.
>
> Even so, it is recognised that this is racy - any use of the block
> map has a time-of-read-to-time-of-use race condition that means you
> have to _verify_ the copy after it completes. FYI, that's what
> xfs_fsr does when copying based on extent maps - if the inode has
> changed in _any way_ during the copy, it aborts the copy of that
> file.
>
> i.e. using fiemap for copying is at best a *hint* about the regions
> that need copying, and it is in no way a guarantee that you'll get
> all the information you need to make accurate copy even if you do
> use the synchronous variant.

Hi Dave,

Can you or anyone else point to authoritative documentation
(or even a summary of those "discussions") of FIEMAP semantics?
I'm hoping the semantics are the same for all file system types.

I had understood that cp's use of FIEMAP_FLAG_SYNC was not only
unnecessary, but even undesirable, given a new-enough kernel.
That's why coreutils-8.11 resorts to using the workaround of
FIEMAP_FLAG_SYNC only when uname says the kernel is 2.6.[0..38].


From MAILER-DAEMON Fri Apr 15 10:04:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAjd6-0003MT-3T
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 10:04:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60641)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QAb9e-000370-2Z
	for coreutils@gnu.org; Fri, 15 Apr 2011 01:01:15 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QAb9c-0005NX-Jb
	for coreutils@gnu.org; Fri, 15 Apr 2011 01:01:13 -0400
Received: from shawmail.shawcable.com ([64.59.128.220]:12462
	helo=mail.shawcable.com) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QAb9c-0005MT-64
	for coreutils@gnu.org; Fri, 15 Apr 2011 01:01:12 -0400
Received: from bpd2mi4no-svcs.prod.shawcable.com ([10.0.184.123])
	by bpd2mo3no-ssvc.prod.shawcable.com with ESMTP;
	14 Apr 2011 23:01:08 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.1 cv=W1EbViUdRLrB3xZls61b9cpNy9JNgdRUyrcMJe3sx/g=
	c=1 sm=1
	a=nduc_ontD6wA:10 a=BLceEmwcHowA:10 a=IkcTkHD0fZMA:10
	a=c23vf5CSMVc0QQz9B4a6RA==:17 a=7-415B0cAAAA:8 a=mDV3o1hIAAAA:8
	a=2QAW-TQMWCG-mY2DWDQA:9 a=BZZr46TI8cr7sYEVFNkA:7 a=QEXdDO2ut3YA:10
	a=FtYAf7_czLgA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117
X-IronPort-AV: E=Sophos;i="4.64,216,1301896800"; d="scan'208";a="331383893"
Received: from unknown (HELO bpd2mi4no-cmts.prod.shawcable.com)
	([192.168.183.123]) by bpd2mi4no-cmts.prod.shawcable.com with ESMTP;
	14 Apr 2011 23:01:07 -0600
X-reinject: true
Received: from unknown (HELO [192.168.10.170]) ([68.147.195.121])
	by bpd2mi4no-dmz.prod.shawcable.com with ESMTP;
	14 Apr 2011 23:01:07 -0600
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net> <20110414225904.GK21395@dastard>
	<4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
In-Reply-To: <20110415000940.GL21395@dastard>
Mime-Version: 1.0 (iPhone Mail 8G4)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8
Message-Id: <76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
X-Mailer: iPhone Mail (8G4)
From: Andreas Dilger <adilger@dilger.ca>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Date: Thu, 14 Apr 2011 23:01:04 -0600
To: Dave Chinner <david@fromorbit.com>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 64.59.128.220
X-Mailman-Approved-At: Fri, 15 Apr 2011 10:04:11 -0400
Cc: Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 05:01:15 -0000

On 2011-04-14, at 6:09 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Fri, Apr 15, 2011 at 12:29:46AM +0100, P=C3=A1draig Brady wrote:
>> On 14/04/11 23:59, Dave Chinner wrote:
>>> On Thu, Apr 14, 2011 at 10:50:10AM -0500, Eric Sandeen wrote:
>>>> On 4/14/11 9:59 AM, P=C3=A1draig Brady wrote:
>>>>> On 14/04/11 15:02, Markus Trippelsdorf wrote:
>>>>>>>> Hi P=C3=A1draig,
>>>>>>>>=20
>>>>>>>> here you go:
>>>>>>>> + filefrag -v unwritten.withdata                                   =
                                                                            =
     =20
>>>>>>>> Filesystem type is: ef53                                           =
                                                                            =
     =20
>>>>>>>> File size of unwritten.withdata is 5120 (2 blocks, blocksize 4096) =
                                                                            =
     =20
>>>>>>>> ext logical physical expected length flags                         =
                                                                            =
    =20
>>>>>>>>   0       0   274432            2560 unwritten,eof                 =
                                                                            =
    =20
>>>>>>>> unwritten.withdata: 1 extent found
>>>>>>>>=20
>>>>>>>> Please notice that this also happens with ext4 on the same kernel.=20=

>>>>>>>> Btrfs is fine.
>>>>>>>=20
>>>>>> `filefrag -vs` fixes the issue on both xfs and ext4.
>>>>>=20
>>>>> So in summary, currently on (2.6.39-rc3), the following
>>>>> will (usually?) report a single unwritten extent,
>>>>> on both ext4 and xfs
>>>>>=20
>>>>>  fallocate -l 10MiB -n k
>>>>>  dd count=3D10 if=3D/dev/urandom conv=3Dnotrunc iflag=3Dfullblock of=3D=
k
>>>>>  filefrag -v k # grep for an extent without unwritten || fail
>>>>=20
>>>> right, that's what I see too in testing.
>>>>=20
>>>> But would the coreutils install have done a preallocation of the destin=
ation file?
>>>>=20
>>>> Otherwise this looks like a different bug...
>>>>=20
>>>>> This particular issue has been discussed so far at:
>>>>> http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D8411
>>>>> Note there it was stated there that ext4 had this
>>>>> fixed as of 2.6.39-rc1, so maybe there is something lurking?
>>>>=20
>>>> ext4 got a fix, but not xfs, I guess.  My poor brain can't remember, I t=
hink I started looking into it, but it's clearly still broken.
>>>>=20
>>>> Still, I don't know for sure what happened to Markus - did something pr=
eallocate, in his case?
>>>=20
>>> Unwritten extent mapping behaves in an unexpected way due to
>>> buffered writeback not occurring immediately. Extent conversion
>>> doesn't occur until the data is on disk, and for buffered IO you
>>> need an fdatasync to ensure that has occurred.
>>>=20
>>> That is:=20
>>>=20
>>> $ xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c "bmap -vp" /mnt/test=
/foo
>>> wrote 5120/5120 bytes at offset 0
>>> 5 KiB, 2 ops; 0.0000 sec (62.600 MiB/sec and 25641.0256 ops/sec)
>>> /mnt/test/foo:
>>> EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
>>>   0: [0..20479]:      268984..289463    0 (268984..289463) 20480 10000
>>>=20
>>> Data has not been written yet, so it is still unwritten. The same
>>> test with a fsync shows:
>>>=20
>>> $ sudo xfs_io -f -c "resvsp 0 10m" -c "pwrite 0 5120" -c fsync -c "bmap -=
vp" /mnt/test/foo
>>> wrote 5120/5120 bytes at offset 0
>>> 5 KiB, 2 ops; 0.0000 sec (87.193 MiB/sec and 35714.2857 ops/sec)
>>> /mnt/test/foo:
>>> EXT: FILE-OFFSET      BLOCK-RANGE      AG AG-OFFSET        TOTAL FLAGS
>>>   0: [0..15]:         268984..268999    0 (268984..268999)    16 00000
>>>   1: [16..20479]:     269000..289463    0 (269000..289463) 20464 10000
>>>=20
>>> Everything is fine.
>>>=20
>>> So this seems like an application error to me. If you are going to
>>> use fiemap to determine what ranges to copy, then you have to
>>> fdatasync the source file first to guarantee that preallocated
>>> extents have been converted to written state before mapping the
>>> file....
>>=20
>> Well IMHO there should be a difference between
>> knowing where you are going to write, and actually writing to disk.
>> I.E. one shouldn't need to write the whole way to the device
>> before returning a valid fiemap.  If a particular file system
>> implementation needs to sync to return a valid fiemap,
>> then it should be implicit.
>=20
> No, this was explicitly laid out in the fiemap interface discussions
> - it's up to the applicaiton to decide if it needs to do a sync
> first. That's what the FIEMAP_FLAG_SYNC control flag is for.
> This forces the fiemap call to do a fsync _before_ getting the
> mapping. If you want to know the exact layout of the file is, then
> you must use this flag.
>=20
> Even so, it is recognised that this is racy - any use of the block
> map has a time-of-read-to-time-of-use race condition that means you
> have to _verify_ the copy after it completes. FYI, that's what
> xfs_fsr does when copying based on extent maps - if the inode has
> changed in _any way_ during the copy, it aborts the copy of that
> file.
>=20
> i.e. using fiemap for copying is at best a *hint* about the regions
> that need copying, and it is in no way a guarantee that you'll get
> all the information you need to make accurate copy even if you do
> use the synchronous variant.

I would tend to agree with P=C3=A1draig. If there is data in the mapping (re=
gardless of whether it is on disk or not), the FIEMAP should return this to t=
he caller. The SYNC flag is only intended to flush the data to disk for tool=
s that are doing direct-to-disk operations on the data.=20

Otherwise the UNMAPPED flag is useless, since even with "check, copy, check"=
 there is no guarantee that the inode is changed _during_ the copy operation=
. It could have been written into the cache _before_ the FIEMAP and remain u=
nchanged and in your case there would be no way to know any data was ever wr=
itten to the file without SYNC on ever single file before FIEMAP.

Cheers, Andreas=


From MAILER-DAEMON Fri Apr 15 13:18:43 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAmfL-0000o4-V7
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 13:18:43 -0400
Received: from eggs.gnu.org ([140.186.70.92]:43111)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+67ae08b727c9c2676b85+2791+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QAmdP-0008Oa-Qd
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:16:44 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from
	<BATV+67ae08b727c9c2676b85+2791+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QAmdP-00018I-1a
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:16:43 -0400
Received: from bombadil.infradead.org ([18.85.46.34]:59813)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+67ae08b727c9c2676b85+2791+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QAmdO-00017z-Vx
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:16:42 -0400
Received: from hch by bombadil.infradead.org with local (Exim 4.72 #1 (Red Hat
	Linux)) id 1QAmdB-0003DX-7v; Fri, 15 Apr 2011 17:16:29 +0000
Date: Fri, 15 Apr 2011 13:16:29 -0400
From: Christoph Hellwig <hch@infradead.org>
To: Jim Meyering <jim@meyering.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110415171629.GA9088@infradead.org>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard> <87sjtjq4ir.fsf@rho.meyering.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87sjtjq4ir.fsf@rho.meyering.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SRS-Rewrite: SMTP reverse-path rewritten from <hch@infradead.org> by
	bombadil.infradead.org See http://www.infradead.org/rpr.html
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 18.85.46.34
X-Mailman-Approved-At: Fri, 15 Apr 2011 13:18:42 -0400
Cc: Eric Sandeen <sandeen@sandeen.net>, Dave Chinner <david@fromorbit.com>,
	xfs-oss <xfs@oss.sgi.com>, coreutils@gnu.org, linux-ext4@vger.kernel.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 17:16:44 -0000

On Fri, Apr 15, 2011 at 10:53:48AM +0200, Jim Meyering wrote:
> Can you or anyone else point to authoritative documentation
> (or even a summary of those "discussions") of FIEMAP semantics?

A large part of the problem is that there's none and the semantics
is a hode-podge of random flags that mostly make little sense
outside of fs developers debugging the layout that we created.  The
closest to a description is Documentation/filesystems/fiemap.txt
in the kernel tree, but that doesn't document most of the delicate
points.



From MAILER-DAEMON Fri Apr 15 13:24:32 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAmky-0002I7-MS
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 13:24:32 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55599)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QAmkw-0002HR-LK
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:24:31 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QAmks-0002YF-Jc
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:24:30 -0400
Received: from mx1.redhat.com ([209.132.183.28]:62241)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QAmks-0002YA-9k
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:24:26 -0400
Received: from int-mx01.intmail.prod.int.phx2.redhat.com
	(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3FHOKQ6029028
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 15 Apr 2011 13:24:20 -0400
Received: from [10.3.113.38] (ovpn-113-38.phx2.redhat.com [10.3.113.38])
	by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP
	id p3FHOJs6029615; Fri, 15 Apr 2011 13:24:19 -0400
Message-ID: <4DA87F43.5040807@redhat.com>
Date: Fri, 15 Apr 2011 11:24:19 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: Christoph Hellwig <hch@infradead.org>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414102608.GA1678@x4.trippels.de>	<20110414120635.GB1678@x4.trippels.de>	<20110414140222.GB1679@x4.trippels.de>	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net>	<20110414225904.GK21395@dastard>
	<4DA7836A.5040604@draigBrady.com>	<20110415000940.GL21395@dastard>
	<87sjtjq4ir.fsf@rho.meyering.net>
	<20110415171629.GA9088@infradead.org>
In-Reply-To: <20110415171629.GA9088@infradead.org>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enigD99D3507C9612B7769610428"
X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: Eric Sandeen <sandeen@sandeen.net>, Dave Chinner <david@fromorbit.com>,
	xfs-oss <xfs@oss.sgi.com>, coreutils@gnu.org, linux-ext4@vger.kernel.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 17:24:31 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigD99D3507C9612B7769610428
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 04/15/2011 11:16 AM, Christoph Hellwig wrote:
> On Fri, Apr 15, 2011 at 10:53:48AM +0200, Jim Meyering wrote:
>> Can you or anyone else point to authoritative documentation
>> (or even a summary of those "discussions") of FIEMAP semantics?
>=20
> A large part of the problem is that there's none and the semantics
> is a hode-podge of random flags that mostly make little sense
> outside of fs developers debugging the layout that we created.  The
> closest to a description is Documentation/filesystems/fiemap.txt
> in the kernel tree, but that doesn't document most of the delicate
> points.

Would it be worth borrowing from Solaris' semantics and adding SEEK_HOLE
and SEEK_DATA to lseek(2), as a higher level (less-detailed, but easier
to define and easier to use) interface for discovering the regions of a
file that only contain NUL bytes?  FIEMAP is a great interface at
exposing everything possible, but coreutils doesn't necessarily need all
that complexity just for determining when it is safe to skip over a
portion of a file that is either unallocated or unchanged from the
initial 0 contents.

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enigD99D3507C9612B7769610428
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNqH9DAAoJEKeha0olJ0Nql8sH/0k7s6dpcKOMaJPWHvPcV4PI
eAHboTT6J6t4Z2zovlFLyO4THFPa6Ib9KA8hENqHPQ4VOAeGeeqp+2ACrJXjwSaG
nxe1u3gDGR/DTC2psJys/cjA+6oaIJubTUHIPMrdTVjwjYysdyHBaromTCHxXN+c
VLn+fjo+O/pPFGDYD37aXzhgTJBvpacUSltfN/sdae4ap+u6LQB53eV8u9Arm7lH
BS8RcV16srDVjR9W+Vw2eEkgmZAG20NtM+SjXz6did8iX5e8KDS8nEdAIbVbOCy4
7Vy0D2B9ak5b3mwFGV+7n/nF35A3LLNlmdP4NGGQLvdwlfMBMxdMgEB8LreVpf8=
=JVjL
-----END PGP SIGNATURE-----

--------------enigD99D3507C9612B7769610428--


From MAILER-DAEMON Fri Apr 15 13:26:12 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAmma-0002TS-O0
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 13:26:12 -0400
Received: from eggs.gnu.org ([140.186.70.92]:38578)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+67ae08b727c9c2676b85+2791+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QAmmY-0002SH-U4
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:26:11 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from
	<BATV+67ae08b727c9c2676b85+2791+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QAmmV-0002i2-81
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:26:10 -0400
Received: from bombadil.infradead.org ([18.85.46.34]:42271)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+67ae08b727c9c2676b85+2791+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QAmmV-0002hx-6A
	for coreutils@gnu.org; Fri, 15 Apr 2011 13:26:07 -0400
Received: from hch by bombadil.infradead.org with local (Exim 4.72 #1 (Red Hat
	Linux)) id 1QAmmR-0005Wl-Pz; Fri, 15 Apr 2011 17:26:03 +0000
Date: Fri, 15 Apr 2011 13:26:03 -0400
From: Christoph Hellwig <hch@infradead.org>
To: Eric Blake <eblake@redhat.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110415172603.GA20086@infradead.org>
References: <20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard> <87sjtjq4ir.fsf@rho.meyering.net>
	<20110415171629.GA9088@infradead.org> <4DA87F43.5040807@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4DA87F43.5040807@redhat.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SRS-Rewrite: SMTP reverse-path rewritten from <hch@infradead.org> by
	bombadil.infradead.org See http://www.infradead.org/rpr.html
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 18.85.46.34
Cc: Eric Sandeen <sandeen@sandeen.net>, Dave Chinner <david@fromorbit.com>,
	xfs-oss <xfs@oss.sgi.com>, Christoph Hellwig <hch@infradead.org>,
	coreutils@gnu.org, linux-ext4@vger.kernel.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 17:26:11 -0000

On Fri, Apr 15, 2011 at 11:24:19AM -0600, Eric Blake wrote:
> Would it be worth borrowing from Solaris' semantics and adding SEEK_HOLE
> and SEEK_DATA to lseek(2), as a higher level (less-detailed, but easier
> to define and easier to use) interface for discovering the regions of a
> file that only contain NUL bytes?

Yes, I've already suggested that both in this thread and on IRC.

For efficient copies it's the only usable interface.


From MAILER-DAEMON Fri Apr 15 18:28:43 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QArVL-0001Cg-TH
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 18:28:43 -0400
Received: from eggs.gnu.org ([140.186.70.92]:44992)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QArVJ-0001CZ-Og
	for coreutils@gnu.org; Fri, 15 Apr 2011 18:28:42 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QArVI-0001WO-OZ
	for coreutils@gnu.org; Fri, 15 Apr 2011 18:28:41 -0400
Received: from idcmail-mo1so.shaw.ca ([24.71.223.10]:47272)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QArVI-0001W2-K8
	for coreutils@gnu.org; Fri, 15 Apr 2011 18:28:40 -0400
Received: from pd4ml3so-ssvc.prod.shaw.ca ([10.0.141.150])
	by pd3mo1so-svcs.prod.shaw.ca with ESMTP; 15 Apr 2011 16:28:38 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.1 cv=fp/YrIG6YJuYYWILSlVONgBms0XWtFqTjqHWlNKfnDg=
	c=1 sm=1
	a=nduc_ontD6wA:10 a=BLceEmwcHowA:10 a=kj9zAlcOel0A:10
	a=c23vf5CSMVc0QQz9B4a6RA==:17 a=nsgWAoxlMCuh_diYnmYA:9
	a=CjuIK1q_8ugA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117
Received: from unknown (HELO [192.168.10.101]) ([68.147.195.121])
	by pd4ml3so-dmz.prod.shaw.ca with ESMTP; 15 Apr 2011 16:28:37 -0600
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: text/plain; charset=us-ascii
From: Andreas Dilger <adilger@dilger.ca>
In-Reply-To: <20110415172603.GA20086@infradead.org>
Date: Fri, 15 Apr 2011 16:28:37 -0600
Content-Transfer-Encoding: quoted-printable
Message-Id: <DD59E27A-FD96-4983-A274-B76CCE99AE7A@dilger.ca>
References: <20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net> <20110414225904.GK21395@dastard>
	<4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
	<87sjtjq4ir.fsf@rho.meyering.net>
	<20110415171629.GA9088@infradead.org> <4DA87F43.5040807@redhat.com>
	<20110415172603.GA20086@infradead.org>
To: Christoph Hellwig <hch@infradead.org>
X-Mailer: Apple Mail (2.1082)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 24.71.223.10
Cc: Dave Chinner <david@fromorbit.com>, Eric Sandeen <sandeen@sandeen.net>,
	xfs-oss <xfs@oss.sgi.com>, coreutils@gnu.org, linux-ext4@vger.kernel.org,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 15 Apr 2011 22:28:42 -0000

On 2011-04-15, at 11:26 AM, Christoph Hellwig wrote:
> On Fri, Apr 15, 2011 at 11:24:19AM -0600, Eric Blake wrote:
>> Would it be worth borrowing from Solaris' semantics and adding =
SEEK_HOLE
>> and SEEK_DATA to lseek(2), as a higher level (less-detailed, but =
easier
>> to define and easier to use) interface for discovering the regions of =
a
>> file that only contain NUL bytes?
>=20
> Yes, I've already suggested that both in this thread and on IRC.
>=20
> For efficient copies it's the only usable interface.

I suspect that these bugs would have still existed whether the interface =
is SEEK_HOLE/SEEK_DATA, or FIEMAP.  The main problem is that the =
delalloc pages were not accounted for correctly during layout traversal.

For ext4 I think it is sufficient to add another case to =
ext4_ext_fiemap_cb() to check the pagecache for unmapped pages when it =
finds an uninitialized extent on disk.  This will be very similar to the =
fix for finding holes in the on-disk mapping.

Cheers, Andreas







From MAILER-DAEMON Fri Apr 15 20:25:14 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAtK5-00014A-Vx
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 20:25:13 -0400
Received: from eggs.gnu.org ([140.186.70.92]:54841)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAtK3-00013t-KX
	for coreutils@gnu.org; Fri, 15 Apr 2011 20:25:12 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAtK2-0000Mz-Cv
	for coreutils@gnu.org; Fri, 15 Apr 2011 20:25:11 -0400
Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:51091)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAtK2-0000Lx-1p
	for coreutils@gnu.org; Fri, 15 Apr 2011 20:25:10 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AosDAAvgqE15LHHJgWdsb2JhbACmDBUBARYmJcRpDoVgBA
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl6.internode.on.net with ESMTP; 16 Apr 2011 09:55:03 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QAtJs-0000HP-LD; Sat, 16 Apr 2011 10:25:00 +1000
Date: Sat, 16 Apr 2011 10:25:00 +1000
From: Dave Chinner <david@fromorbit.com>
To: Andreas Dilger <adilger@dilger.ca>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110416002500.GO21395@dastard>
References: <4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard> <87sjtjq4ir.fsf@rho.meyering.net>
	<20110415171629.GA9088@infradead.org> <4DA87F43.5040807@redhat.com>
	<20110415172603.GA20086@infradead.org>
	<DD59E27A-FD96-4983-A274-B76CCE99AE7A@dilger.ca>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <DD59E27A-FD96-4983-A274-B76CCE99AE7A@dilger.ca>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.145
Cc: Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	Christoph Hellwig <hch@infradead.org>, coreutils@gnu.org,
	linux-ext4@vger.kernel.org, Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Apr 2011 00:25:12 -0000

On Fri, Apr 15, 2011 at 04:28:37PM -0600, Andreas Dilger wrote:
> On 2011-04-15, at 11:26 AM, Christoph Hellwig wrote:
> > On Fri, Apr 15, 2011 at 11:24:19AM -0600, Eric Blake wrote:
> >> Would it be worth borrowing from Solaris' semantics and adding SEEK_HOLE
> >> and SEEK_DATA to lseek(2), as a higher level (less-detailed, but easier
> >> to define and easier to use) interface for discovering the regions of a
> >> file that only contain NUL bytes?
> > 
> > Yes, I've already suggested that both in this thread and on IRC.
> > 
> > For efficient copies it's the only usable interface.
> 
> I suspect that these bugs would have still existed whether the
> interface is SEEK_HOLE/SEEK_DATA, or FIEMAP.  The main problem is
> that the delalloc pages were not accounted for correctly during
> layout traversal..

It's not delalloc that is the problem - XFS accounts for them just
fine in the extent map when asked. However, XFS does speculative
delayed allocation over regions that contain no data, so if the
core-utils folk are assuming that delalloc extents contain data and
need to be copied, they're in for a nasty surprise.

However, every example I've seen in this thread has had to do with
unwritten extents not changing state when data is written into the
page cache. i.e. people are struggling with the expected behaviour
of unwritten extents.

That is, unwritten extent remain unwritten extents until data has
been _physically_ written to them. If there is data in the page
cache over the unwritten extent, it is still an unwritten extent.
If the system crashes while in this state, then the extent _must_
remain an unwritten extent after recovery, otherwise it exposes
stale data.

Further, using FIEMAP to determine where the data is that needs
copying is extremely fragile. What happens when FIEMAP grows a
different type of extent that contains data? cp will break, because
it doesn't think it needs to copy data in extents of an unknown
type. Or it will break because it thinks it needs to copy it and
there's something in it that should not be copied.

Also, cp shoul dnot be trying to replicate the physical layout of
the file when copying it - that's for the filesystem to decide and
having userspace try to do this is a sure recipe for causing severe
filesystem fragmentation. The filesystems already do an excellent
job of optimising allocation - userspace should not be trying to
second guess what is optimal layout for the filesystem.

Fundamentally, what the core-utils guys want is FIEMAP to tell them
where data is in the file, regardless of whether it is in memory or
on disk. That is not what FIEMAP is intended for and matches
SEEK_HOLE/SEEK_DATA precisely.

SEEK_HOLE/SEEK_DATA have very well understood semantics and is
designed specifically for optimising acceess to sparse files. This
interface abstracts all the details of how different filesystems
store their data so the application doesn't need to care about it.
The API is so, so much simpler to use and understand, to. And if the
filesystem has data in cache over an unwritten extent, then by
definition it's still data to be returned by SEEK_DATA. If it fails
to return the range as such then the implementation is broken.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Fri Apr 15 20:50:50 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAtis-0005Tu-5R
	for mharc-coreutils@gnu.org; Fri, 15 Apr 2011 20:50:50 -0400
Received: from eggs.gnu.org ([140.186.70.92]:44336)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAtip-0005TQ-Fz
	for coreutils@gnu.org; Fri, 15 Apr 2011 20:50:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAtio-0003NC-Dc
	for coreutils@gnu.org; Fri, 15 Apr 2011 20:50:47 -0400
Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:61795)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QAtio-0003Mg-0T
	for coreutils@gnu.org; Fri, 15 Apr 2011 20:50:46 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AosDABznqE15LHHJgWdsb2JhbACmDBUBARYmJYhvu10OhWAE
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl6.internode.on.net with ESMTP; 16 Apr 2011 10:20:41 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QAtii-0000K9-GC; Sat, 16 Apr 2011 10:50:40 +1000
Date: Sat, 16 Apr 2011 10:50:40 +1000
From: Dave Chinner <david@fromorbit.com>
To: Andreas Dilger <adilger@dilger.ca>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110416005040.GP21395@dastard>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.145
Cc: Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Apr 2011 00:50:48 -0000

On Thu, Apr 14, 2011 at 11:01:04PM -0600, Andreas Dilger wrote:
> On 2011-04-14, at 6:09 PM, Dave Chinner <david@fromorbit.com>
> wrote:
> > No, this was explicitly laid out in the fiemap interface
> > discussions - it's up to the applicaiton to decide if it needs
> > to do a sync first. That's what the FIEMAP_FLAG_SYNC control
> > flag is for.  This forces the fiemap call to do a fsync _before_
> > getting the mapping. If you want to know the exact layout of the
> > file is, then you must use this flag.
> > 
> > Even so, it is recognised that this is racy - any use of the
> > block map has a time-of-read-to-time-of-use race condition that
> > means you have to _verify_ the copy after it completes. FYI,
> > that's what xfs_fsr does when copying based on extent maps - if
> > the inode has changed in _any way_ during the copy, it aborts
> > the copy of that file.
> > 
> > i.e. using fiemap for copying is at best a *hint* about the
> > regions that need copying, and it is in no way a guarantee that
> > you'll get all the information you need to make accurate copy
> > even if you do use the synchronous variant.
> 
> I would tend to agree with Pádraig. If there is data in the
> mapping (regardless of whether it is on disk or not), the FIEMAP
> should return this to the caller.  The SYNC flag is only intended
> to flush the data to disk for tools that are doing
> direct-to-disk operations on the data.

What you are suggesting is that FIEMAP needs to be page cache
coherent, and that is far, far away from the intended use of the
interface. Even consiering that you need to looking for active pages
in the page cache when mapping extents say to me that you are
doing something very wrong.

Unwritten extents remain unwritten until the data is physically
written to them. Therefore, to change their state, you need to sync
the data covering the range.  _Lying_ about whether an extent is in
the unwritten state is a really bad precedence to set, especially as
it is then guaranteed to change state when a crash occurs (Why did
recovery zero out my file? FIEMAP said it contained data before my
system crashed!).

Don't try to mangle the API semantics every time someone doesn't
understand how to use FIEMAP reliably. If you need the extent list
returned by FIEMAP to match what is in the page cache *regardless of

> Otherwise the UNMAPPED flag is useless, since even with "check,
> copy, check" there is no guarantee that the inode is changed
> _during_ the copy operation. It could have been written into the
> cache _before_ the FIEMAP and remain unchanged and in your case
> there would be no way to know any data was ever written to the
> file without SYNC on ever single file before FIEMAP.

I can't find any UNMAPPED flag in the FIEMAP interface, so I have no
idea what you are refering to here.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Sat Apr 16 01:11:33 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAxnB-00008y-Lx
	for mharc-coreutils@gnu.org; Sat, 16 Apr 2011 01:11:33 -0400
Received: from eggs.gnu.org ([140.186.70.92]:53242)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QAxn9-00006Z-Ib
	for coreutils@gnu.org; Sat, 16 Apr 2011 01:11:32 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QAxn8-0005BE-CS
	for coreutils@gnu.org; Sat, 16 Apr 2011 01:11:31 -0400
Received: from shawmail.shawcable.com ([64.59.128.220]:48603
	helo=mail.shawcable.com) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QAxn8-0005At-8h
	for coreutils@gnu.org; Sat, 16 Apr 2011 01:11:30 -0400
Received: from bpd2mi2no-svcs.prod.shawcable.com ([10.0.184.121])
	by bpd2mo1no-ssvc.prod.shawcable.com with ESMTP;
	15 Apr 2011 23:11:26 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.1 cv=l0hTUOwXBBZC3xuFbE1ochJnP84ph5EBJB8ZSEz2E7A=
	c=1 sm=1
	a=nduc_ontD6wA:10 a=BLceEmwcHowA:10 a=kj9zAlcOel0A:10
	a=c23vf5CSMVc0QQz9B4a6RA==:17 a=7-415B0cAAAA:8 a=DKKZMC6X0ut7eAvZZIcA:9
	a=CjuIK1q_8ugA:10 a=FtYAf7_czLgA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117
X-IronPort-AV: E=Sophos;i="4.64,223,1301896800"; d="scan'208";a="423781857"
Received: from unknown (HELO bpd2mi2no-cmts.prod.shawcable.com)
	([192.168.183.121]) by bpd2mi2no-cmts.prod.shawcable.com with ESMTP;
	15 Apr 2011 23:11:25 -0600
X-reinject: true
Received: from unknown (HELO [192.168.10.170]) ([68.147.195.121])
	by bpd2mi2no-dmz.prod.shawcable.com with ESMTP;
	15 Apr 2011 23:11:25 -0600
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net> <20110414225904.GK21395@dastard>
	<4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
In-Reply-To: <20110416005040.GP21395@dastard>
Mime-Version: 1.0 (iPhone Mail 8G4)
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii
Message-Id: <E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
X-Mailer: iPhone Mail (8G4)
From: Andreas Dilger <adilger@dilger.ca>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Date: Fri, 15 Apr 2011 23:11:23 -0600
To: Dave Chinner <david@fromorbit.com>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 64.59.128.220
Cc: Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Apr 2011 05:11:32 -0000

On 2011-04-15, at 6:50 PM, Dave Chinner <david@fromorbit.com> wrote:
> What you are suggesting is that FIEMAP needs to be page cache
> coherent, and that is far, far away from the intended use of the
> interface. Even consiering that you need to looking for active pages
> in the page cache when mapping extents say to me that you are
> doing something very wrong.
>=20
> Unwritten extents remain unwritten until the data is physically
> written to them. Therefore, to change their state, you need to sync
> the data covering the range.

In that case, it means cp should just always use FIEMAP_FLAG_SYNC, which is f=
ine.=20

Cheers, Andreas



From MAILER-DAEMON Sat Apr 16 02:05:57 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QAydp-00071k-9s
	for mharc-coreutils@gnu.org; Sat, 16 Apr 2011 02:05:57 -0400
Received: from eggs.gnu.org ([140.186.70.92]:59857)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAydm-00070x-PM
	for coreutils@gnu.org; Sat, 16 Apr 2011 02:05:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAydl-0005A2-KX
	for coreutils@gnu.org; Sat, 16 Apr 2011 02:05:54 -0400
Received: from mail-px0-f179.google.com ([209.85.212.179]:51881)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QAydl-00059q-CK
	for coreutils@gnu.org; Sat, 16 Apr 2011 02:05:53 -0400
Received: by pxi2 with SMTP id 2so1993020pxi.38
	for <coreutils@gnu.org>; Fri, 15 Apr 2011 23:05:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=TUqK3ZntR6+MQN9wLiP1e3YRlMt72D9+pBpjoOBk31M=;
	b=s5FGQJSfuc8Nvh4gPM2TOPomAYhSCIxAjrNiqQJBEbf/AIiD3n1agH+VqagJpyGaxu
	Z1PWpjaTZ82uvLyDRsOoiKZV6iLQmrN6CGx7yCl/Y0V/Hj39AIvaacQOxNpkNO5OrLLQ
	FJM/9F7LU7tQ9cGRQRlkDXQ9vjzQEkgOgQJ90=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=q+VGLmMHsmBo/7hXQrBzPrwkMLHTdbWt+kNOdYlbtZPAN6aZ8EKfLKdyjkMbpzlcKh
	uOANsLF2CXjAFgJjKDBQbctPkLSO29hbcAlTSAW1vw0m9AQXvD+LsyQb9wBetImQwnJ/
	hVXGgu7Sk9ZszAvj4zTqAG6DipcCFeTo+ks10=
MIME-Version: 1.0
Received: by 10.68.15.130 with SMTP id x2mr3019130pbc.35.1302933951928; Fri,
	15 Apr 2011 23:05:51 -0700 (PDT)
Received: by 10.68.60.7 with HTTP; Fri, 15 Apr 2011 23:05:51 -0700 (PDT)
In-Reply-To: <20110416005040.GP21395@dastard>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
Date: Sat, 16 Apr 2011 14:05:51 +0800
Message-ID: <BANLkTikEeXcvjgREoRCgriWAhZfnxJVtKQ@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Dave Chinner <david@fromorbit.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.212.179
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Apr 2011 06:05:55 -0000

On Sat, Apr 16, 2011 at 8:50 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Thu, Apr 14, 2011 at 11:01:04PM -0600, Andreas Dilger wrote:
>> On 2011-04-14, at 6:09 PM, Dave Chinner <david@fromorbit.com>
>> wrote:
>> > No, this was explicitly laid out in the fiemap interface
>> > discussions - it's up to the applicaiton to decide if it needs
>> > to do a sync first. That's what the FIEMAP_FLAG_SYNC control
>> > flag is for. =A0This forces the fiemap call to do a fsync _before_
>> > getting the mapping. If you want to know the exact layout of the
>> > file is, then you must use this flag.
>> >
>> > Even so, it is recognised that this is racy - any use of the
>> > block map has a time-of-read-to-time-of-use race condition that
>> > means you have to _verify_ the copy after it completes. FYI,
>> > that's what xfs_fsr does when copying based on extent maps - if
>> > the inode has changed in _any way_ during the copy, it aborts
>> > the copy of that file.
>> >
>> > i.e. using fiemap for copying is at best a *hint* about the
>> > regions that need copying, and it is in no way a guarantee that
>> > you'll get all the information you need to make accurate copy
>> > even if you do use the synchronous variant.
>>
>> I would tend to agree with P=E1draig. If there is data in the
>> mapping (regardless of whether it is on disk or not), the FIEMAP
>> should return this to the caller. =A0The SYNC flag is only intended
>> to flush the data to disk for tools that are doing
>> direct-to-disk operations on the data.
>
> What you are suggesting is that FIEMAP needs to be page cache
> coherent, and that is far, far away from the intended use of the
> interface. Even consiering that you need to looking for active pages
> in the page cache when mapping extents say to me that you are
> doing something very wrong.
>
> Unwritten extents remain unwritten until the data is physically
> written to them. Therefore, to change their state, you need to sync
No, buffered writes change their state without sync.

> the data covering the range. =A0_Lying_ about whether an extent is in
> the unwritten state is a really bad precedence to set, especially as
> it is then guaranteed to change state when a crash occurs (Why did
> recovery zero out my file? FIEMAP said it contained data before my
> system crashed!).

All filesystems have metadata in memory which is not flushed to
permanent storage. e.g. if a extent exists in memory, but itself and
corresponding data are not flushed to permanent storage. So you said
above can only be achieved by sync before FIEMAP.  Otherwise if a
crash occurs, FIEMAP can not find data before system crashed.

Without delayed allocation, there is no difference between
preallocation case(fallocate) and normal cases.

> Don't try to mangle the API semantics every time someone doesn't
> understand how to use FIEMAP reliably. If you need the extent list
> returned by FIEMAP to match what is in the page cache *regardless of
>
>> Otherwise the UNMAPPED flag is useless, since even with "check,
>> copy, check" there is no guarantee that the inode is changed
>> _during_ the copy operation. It could have been written into the
>> cache _before_ the FIEMAP and remain unchanged and in your case
>> there would be no way to know any data was ever written to the
>> file without SYNC on ever single file before FIEMAP.
>
> I can't find any UNMAPPED flag in the FIEMAP interface, so I have no
> idea what you are refering to here.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Sat Apr 16 10:54:59 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QB6tn-0000Yt-Rz
	for mharc-coreutils@gnu.org; Sat, 16 Apr 2011 10:54:59 -0400
Received: from eggs.gnu.org ([140.186.70.92]:48918)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@mit.edu>) id 1QB4VO-0004Mb-PI
	for coreutils@gnu.org; Sat, 16 Apr 2011 08:21:39 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <tytso@mit.edu>) id 1QB4VO-0006Oh-4B
	for coreutils@gnu.org; Sat, 16 Apr 2011 08:21:38 -0400
Received: from dmz-mailsec-scanner-2.mit.edu ([18.9.25.13]:48729)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@mit.edu>) id 1QB4VO-0006Ob-2R
	for coreutils@gnu.org; Sat, 16 Apr 2011 08:21:38 -0400
X-AuditID: 1209190d-b7c48ae000004826-e3-4da989a5ee45
Received: from mailhub-auth-4.mit.edu ( [18.7.62.39])
	by dmz-mailsec-scanner-2.mit.edu (Symantec Messaging Gateway) with SMTP
	id 86.E2.18470.5A989AD4; Sat, 16 Apr 2011 08:20:53 -0400 (EDT)
Received: from outgoing.mit.edu (OUTGOING-AUTH.MIT.EDU [18.7.22.103])
	by mailhub-auth-4.mit.edu (8.13.8/8.9.2) with ESMTP id p3GCLYfF009092; 
	Sat, 16 Apr 2011 08:21:34 -0400
Received: from [10.0.42.109] (c-98-216-98-217.hsd1.ma.comcast.net
	[98.216.98.217]) (authenticated bits=0)
	(User authenticated as tytso@ATHENA.MIT.EDU)
	by outgoing.mit.edu (8.13.6/8.12.4) with ESMTP id p3GCLUqv004836
	(version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT);
	Sat, 16 Apr 2011 08:21:31 -0400 (EDT)
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: text/plain; charset=us-ascii
From: Theodore Tso <tytso@MIT.EDU>
In-Reply-To: <E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
Date: Sat, 16 Apr 2011 08:21:28 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
References: <20110414102608.GA1678@x4.trippels.de>
	<20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net> <20110414225904.GK21395@dastard>
	<4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
To: Andreas Dilger <adilger@dilger.ca>
X-Mailer: Apple Mail (2.1084)
X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAKsWRmVeSWpSXmKPExsUixG6nrru0c6WvwcGVshbLTj5isth79BGz
	xZZj9xgtZs67w2bx//9tJot1c00tWq9oWSzqu8XowOHRsrnc49bUdnaPU4skPNqmmXmsvrCV
	0WPL4odMHv27LjB5fN4kF8ARxWWTkpqTWZZapG+XwJXx+etz1oIPzBU9F/6zNzC2M3cxcnJI
	CJhIPPl8D8oWk7hwbz1bFyMXh5DAPkaJvzsfsEM4Gxgltq9axQrhnGaSOPlrBhtIi7CAn8S/
	c5fA2nkFDCWWbmpnB7GZBbQkbvx7yQRiswkoSdz5tJ8FxOYUsJW4tvUuK4jNIqAqcev+RrB1
	zAKPmCSmnXoG1awtsWzha6ihVhIdvWeZIDavYJa4c/4u2CQRATWJ/1d+ABVxAB0uK9G0LGMC
	o+AsJHfMQnLHLCRjFzAyr2KUTcmt0s1NzMwpTk3WLU5OzMtLLdI10svNLNFLTSndxAiKGE5J
	3h2M7w4qHWIU4GBU4uG1LF/hK8SaWFZcmXuIUZKDSUmU1xgYb0J8SfkplRmJxRnxRaU5qcWH
	GCU4mJVEeJPdgXK8KYmVValF+TApaQ4WJXHemZLqvkIC6YklqdmpqQWpRTBZGQ4OJQleU5Ch
	gkWp6akVaZk5JQhpJg5OkOE8QMPvdIAMLy5IzC3OTIfIn2JUlBLnlQJpFgBJZJTmwfXCEtor
	RnGgV4R5dUGqeIDJEK77FdBgJqDBNxuWgwwuSURISTUwCkTJiN9b2evoLPX9yhFTPrtQf+d/
	gp/WsDbv9p55aqoR22SdrLvyzWye7MUXfJU50l6IC875furB0ThFDpFfTbsPMal7/7qrpCXx
	c+39zSpVa1aHdW6fUmJwOrh77hLFd/VSlxsd9F27dO5smNGS+3PmMTklY7eylwwsCzaLzdl1
	L9GFI/iWEktxRqKhFnNRcSIAnPCLB0MDAAA=
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 18.9.25.13
X-Mailman-Approved-At: Sat, 16 Apr 2011 10:54:58 -0400
Cc: Dave Chinner <david@fromorbit.com>, Eric Sandeen <sandeen@sandeen.net>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Apr 2011 12:21:39 -0000


On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:

> In that case, it means cp should just always use FIEMAP_FLAG_SYNC, =
which is fine.=20

Except that if someone is copying a large delay allocated file, it will =
cause=20
the file to immediately snapped to disk, which might not be the greatest
thing in the world.  Christoph is write, SEEK_HOLE and SEEK_DATA are
a much better API for what cp woulld lke to do.  Unfortunately it hasn't
been implemented yet in the VFS...

- Ted=


From MAILER-DAEMON Sat Apr 16 21:36:02 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBGuA-0003mQ-Nk
	for mharc-coreutils@gnu.org; Sat, 16 Apr 2011 21:36:02 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41004)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adekunleadekoya@gmail.com>) id 1QBCi6-00064n-Qd
	for coreutils@gnu.org; Sat, 16 Apr 2011 17:07:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <adekunleadekoya@gmail.com>) id 1QBCi5-00026L-La
	for coreutils@gnu.org; Sat, 16 Apr 2011 17:07:18 -0400
Received: from mail-ew0-f41.google.com ([209.85.215.41]:48459)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adekunleadekoya@gmail.com>)
	id 1QBChy-000256-Sn; Sat, 16 Apr 2011 17:07:11 -0400
Received: by ewy9 with SMTP id 9so1160441ewy.0
	for <multiple recipients>; Sat, 16 Apr 2011 14:07:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:x-rim-org-msg-ref-id:message-id
	:content-transfer-encoding:reply-to:x-priority:references
	:in-reply-to:sensitivity:importance:subject:to:cc:from:date
	:content-type:mime-version;
	bh=tjmbONXTYrRULKZDRJGi+xtVDjxeknS0GC/DtBpXF8g=;
	b=i6cx8vyfgJEtzNqpOv2oA5pxVjTiW5p0eJvht9rQaUHd2lhbvl3iRavGQnCxAP20Dg
	y6EAm7x/10gt8ml8VUiDArk9C9VkJwSR2RcbDLk23uGsMsbWqVnRBlQEvd1gKKtL/jSS
	Y22SU3puRVwNjOBi+HdsA7Na0/AhEoaSjQwNg=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=x-rim-org-msg-ref-id:message-id:content-transfer-encoding:reply-to
	:x-priority:references:in-reply-to:sensitivity:importance:subject:to
	:cc:from:date:content-type:mime-version;
	b=BzAgnB8AYsMEA0YPHoFJRyjtNaplalocTZrcBfVijup73kS3qOudz9MxyThV4UWGci
	t5wcoFi+pEizbXq20zr4mnykuYmweyfLCQTbQptl8cm13YW9QwD0dFjcIFEyMfYS0xy3
	EaGbh2mIE/fx0uY8b0HfTHAIL5Bh39bazm0vk=
Received: by 10.213.34.211 with SMTP id m19mr643768ebd.146.1302988029180;
	Sat, 16 Apr 2011 14:07:09 -0700 (PDT)
Received: from 172.18.203.192 (bda-178-239-85-124.bis7.eu.blackberry.com
	[178.239.85.124])
	by mx.google.com with ESMTPS id k51sm2954608eei.3.2011.04.16.14.07.07
	(version=SSLv3 cipher=OTHER); Sat, 16 Apr 2011 14:07:08 -0700 (PDT)
X-rim-org-msg-ref-id: 56752866
Message-ID: <56752866-1302988026-cardhu_decombobulator_blackberry.rim.net-1192226888-@b11.c10.bise7.blackberry>
Content-Transfer-Encoding: base64
X-Priority: Normal
References: <87pqopx1lq.fsf@rho.meyering.net>
In-Reply-To: <87pqopx1lq.fsf@rho.meyering.net>
Sensitivity: Normal
Importance: Normal
Subject: Re: coreutils-8.11 released [stable]
To: "Jim Meyering" <jim@meyering.net>,
	info-gnu-bounces+adekunleadekoya=gmail.com@gnu.org, info-gnu@gnu.org
From: adekunleadekoya@gmail.com
Date: Sat, 16 Apr 2011 21:07:40 +0000
Content-Type: text/plain
MIME-Version: 1.0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.215.41
X-Mailman-Approved-At: Sat, 16 Apr 2011 21:36:01 -0400
Cc: coordinator@translationproject.org, coreutils@gnu.org,
	coreutils-announce@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
Reply-To: adekunleadekoya@gmail.com
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 16 Apr 2011 21:07:20 -0000

Q2FuIGFueW9uZSBoZWxwIHdpdGggdGhlIHJlcXVlc3QgYmVsb3cgPw0KDQpTaXIsDQoNClBhcmRv
biBtZSB0aGF0IEkgaGF2ZSB0byBhc2sgdGhpcyxzaXIhDQoNCk15IHByb2plY3QgaXMgYWJvdXQg
Y29tcGlsZXIgZGV2ZWxvcG1lbnQuIEkgc3VyZWx5IGhhdmUgYSBudW1iZXIgb2YgaW1wb3J0YW50
IGJ1aWxkaW5nIGJsb2NrcyB0aGF0IEkgbmVlZCB0byBnZXQgdGhlIGpvYiBkb25lIGV4Y2VwdCBm
b3IgdGhlIGludGVyZmFjZSBiZXR3ZWVuIHRoZSBjb21waWxlZCBjb2RlIGFuZCB0aGUgaW50ZXJw
cmV0ZXIvVk0vYXNzZW1ibGVyIHRoYXQgd291bGQgZXhlY3V0ZSB0aGUgdGFyZ2V0IGNvZGUuDQoN
CkkgYWN0dWFsbHkgbmVlZCBteSBqYXZhIGFwcGxpY2F0aW9uKGFuIElERSB0aGF0IGludm9rZXMg
dGhlIGNvbXBpbGVyKSB0byBpbnZva2UgdGhlIGludGVycHJldGVyLiBJIHdhbnQgYW4gZW1iZWRk
YWJsZSBpbnRlcnByZXRlciB0aGF0IGNvdWxkIGdldCBpbnB1dChjb21waWxlZCBjb2RlKSBmcm9t
IG15IGphdmEgYXBwbGljYXRpb24gYW5kIGFsc28gc2VuZCBpdHMgb3V0cHV0IGJhY2sgdG8gbXkg
YXBwbGljYXRpb24uDQoNCkkgYW0gd3JpdGluZyBpZiB5b3UgaGF2ZSBnb29kIHJlZmVyZW5jZXMg
b24gaG93IHRvIGFjaGlldmUgdGhpcyB3aXRoIGphdmEgYW5kIGEgd2VsbCBkb2N1bWVudGVkIGlu
dGVycHJldGVyID8gSSB3b3VsZCBwcmVmZXIgYW4gYXNzZW1ibGVyIGJ1dCBvdGhlciBpbnRlcnBy
ZXRlcnMgb3IgdmlydHVhbCBtYWNoaW5lcyBtYXkgc3VmZmljZSBpbiBhcyBtdWNoIGFzIHRoZSBp
bnN0cnVjdGlvbiBzZXRzIGFuZCBkYXRhIHN0cnVjdHVyZXMoZWcgcmVnaXN0ZXJzKSAgYXJlIHBy
b3Blcmx5IGRvY3VtZW50ZWQgYW5kIHRoZSBqYXZhIGludGVyZmFjaW5nIGlzIGFsc28gd2VsbCBk
b2N1bWVudGVkLg0KDQpJIGhhdmUgc2VudCBvdXQgYSBzaW1pbGFyIGVtYWlsIHRvIGEgbnVtYmVy
IG9mIG9wZW4gc291cmNlIG1haWxpbmcgbGlzdHMgYnV0IEkgcmVjYWxsIHRoYXQgeW91IGhhdmUg
ZG9uZSBhIG51bWJlciBvZiByZXNlYXJjaCBpbiB0aGlzIGFyZWEgYW5kIHRodXMgZmVlbCB5b3Ug
bWF5IGhhdmUgZ29vZCByZWZlcmVuY2VzID8NCg0KSSB3b3VsZCBhcHByZWNpYXRlIHlvdXIga2lu
ZCBvcGluaW9uIG9uIHRoZSByZXF1ZXN0Lg0KDQpUaGFua3Mgc2lyIGFuZCBHb2QgYmxlc3MgeW91
IQ0KDQpSZWdhcmRzDQoNCkFkZWtveWEgQWRla3VubGUNClNlbnQgZnJvbSBteSBCbGFja0JlcnJ5
IHdpcmVsZXNzIGRldmljZSBmcm9tIE1UTg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0K
RnJvbTogSmltIE1leWVyaW5nIDxqaW1AbWV5ZXJpbmcubmV0Pg0KU2VuZGVyOiBpbmZvLWdudS1i
b3VuY2VzK2FkZWt1bmxlYWRla295YT1nbWFpbC5jb21AZ251Lm9yZ0RhdGU6IFdlZCwgMTMgQXBy
IDIwMTEgMjI6Mzg6MzggDQpUbzogPGluZm8tZ251QGdudS5vcmc+DQpDYzogPGNvb3JkaW5hdG9y
QHRyYW5zbGF0aW9ucHJvamVjdC5vcmc+OyA8Y29yZXV0aWxzQGdudS5vcmc+OyA8Y29yZXV0aWxz
LWFubm91bmNlQGdudS5vcmc+DQpTdWJqZWN0OiBjb3JldXRpbHMtOC4xMSByZWxlYXNlZCBbc3Rh
YmxlXQ0KDQotLS0tLUJFR0lOIFBHUCBTSUdORUQgTUVTU0FHRS0tLS0tDQpIYXNoOiBTSEEyNTYN
Cg0KVGhpcyBpcyB0byBhbm5vdW5jZSBjb3JldXRpbHMtOC4xMSwgYSBzdGFibGUgcmVsZWFzZS4N
Cg0KVGhpcyB0aW1lIHdlIGhhdmUgYW4gaW50ZXJlc3RpbmcgdmFyaWV0eSBvZiBidWcgZml4ZXMs
DQp0d28gbmV3IGZlYXR1cmVzIGZvciAiZGQiLCBhbmQgYSBmZXcgImNoYW5nZXMgaW4gYmVoYXZp
b3IiLg0KU2VlIE5FV1MgYmVsb3cgZm9yIGEgYnJpZWYgc3VtbWFyeS4NCg0KSmltIFtvbiBiZWhh
bGYgb2YgdGhlIGNvcmV1dGlscyBtYWludGFpbmVyc10NCg0KLSAtLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpQLlMuIGhlcmUncyB0aGUgR05VIENvcmV1dGlscyBo
b21lIHBhZ2U6DQogICAgaHR0cDovL3d3dy5nbnUub3JnL3NvZnR3YXJlL2NvcmV1dGlscy8NCg0K
Rm9yIGEgc3VtbWFyeSBvZiBjaGFuZ2VzIGFuZCBjb250cmlidXRvcnMsIHNlZToNCiAgaHR0cDov
L2dpdC5zdi5nbnUub3JnL2dpdHdlYi8/cD1jb3JldXRpbHMuZ2l0O2E9c2hvcnRsb2c7aD12OC4x
MQ0Kb3IgcnVuIHRoaXMgY29tbWFuZCBmcm9tIGEgZ2l0LWNsb25lZCBjb3JldXRpbHMgZGlyZWN0
b3J5Og0KICBnaXQgc2hvcnRsb2cgdjguMTAuLnY4LjExDQoNClRvIHN1bW1hcml6ZSB0aGUgbWFu
eSBnbnVsaWItcmVsYXRlZCBjaGFuZ2VzLCBydW4gdGhlc2UgY29tbWFuZHMgZnJvbQ0KYSBnaXQt
Y2xvbmVkIGNvcmV1dGlscyBkaXJlY3Rvcnk6DQogIGdpdCBjaGVja291dCB2OC4xMQ0KICBnaXQg
c3VibW9kdWxlIHN1bW1hcnkgdjguMTANCg0KSGVyZSBhcmUgdGhlIGNvbXByZXNzZWQgc291cmNl
czoNCiAgaHR0cDovL2Z0cC5nbnUub3JnL2dudS9jb3JldXRpbHMvY29yZXV0aWxzLTguMTEudGFy
Lmd6ICAgKDExTUIpDQogIGh0dHA6Ly9mdHAuZ251Lm9yZy9nbnUvY29yZXV0aWxzL2NvcmV1dGls
cy04LjExLnRhci54eiAgICg0LjdNQikNCg0KSGVyZSBhcmUgdGhlIEdQRyBkZXRhY2hlZCBzaWdu
YXR1cmVzWypdOg0KICBodHRwOi8vZnRwLmdudS5vcmcvZ251L2NvcmV1dGlscy9jb3JldXRpbHMt
OC4xMS50YXIuZ3ouc2lnDQogIGh0dHA6Ly9mdHAuZ251Lm9yZy9nbnUvY29yZXV0aWxzL2NvcmV1
dGlscy04LjExLnRhci54ei5zaWcNCg0KVG8gcmVkdWNlIGxvYWQgb24gdGhlIG1haW4gc2VydmVy
LCB1c2UgYSBtaXJyb3IgbGlzdGVkIGF0Og0KICBodHRwOi8vd3d3LmdudS5vcmcvb3JkZXIvZnRw
Lmh0bWwNCg0KWypdIFlvdSBjYW4gdXNlIGVpdGhlciBvZiB0aGUgYWJvdmUgc2lnbmF0dXJlIGZp
bGVzIHRvIHZlcmlmeSB0aGF0DQp0aGUgY29ycmVzcG9uZGluZyBmaWxlICh3aXRob3V0IHRoZSAu
c2lnIHN1ZmZpeCkgaXMgaW50YWN0LiAgRmlyc3QsDQpiZSBzdXJlIHRvIGRvd25sb2FkIGJvdGgg
dGhlIC5zaWcgZmlsZSBhbmQgdGhlIGNvcnJlc3BvbmRpbmcgdGFyYmFsbC4NClRoZW4sIHJ1biBh
IGNvbW1hbmQgbGlrZSB0aGlzOg0KDQogIGdwZyAtLXZlcmlmeSBjb3JldXRpbHMtOC4xMS50YXIu
Z3ouc2lnDQoNCklmIHRoYXQgY29tbWFuZCBmYWlscyBiZWNhdXNlIHlvdSBkb24ndCBoYXZlIHRo
ZSByZXF1aXJlZCBwdWJsaWMga2V5LA0KdGhlbiBydW4gdGhpcyBjb21tYW5kIHRvIGltcG9ydCBp
dDoNCg0KICBncGcgLS1rZXlzZXJ2ZXIga2V5cy5nbnVwZy5uZXQgLS1yZWN2LWtleXMgMDAwQkVF
RUUNCg0KYW5kIHJlcnVuIHRoZSBgZ3BnIC0tdmVyaWZ5JyBjb21tYW5kLg0KDQpUaGlzIHJlbGVh
c2Ugd2FzIGJvb3RzdHJhcHBlZCB3aXRoIHRoZSBmb2xsb3dpbmcgdG9vbHM6DQogIEF1dG9jb25m
IDIuNjguNzEtYWYzMDANCiAgQXV0b21ha2UgMS4xMWENCiAgR251bGliIHYwLjAtNTExNS1nYTgx
MzQ4ZA0KICBCaXNvbiAyLjQuNTU5LTY3NjkNCg0KPT09PT09PT09PT09PT09PT09PT09PT09PT09
PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KTkVXUw0KDQoqIE5vdGV3b3J0aHkgY2hh
bmdlcyBpbiByZWxlYXNlIDguMTEgKDIwMTEtMDQtMTMpIFtzdGFibGVdDQoNCioqIEJ1ZyBmaXhl
cw0KDQogIGNwIC1hIC0tbGluayB3b3VsZCBub3QgY3JlYXRlIGEgaGFyZGxpbmsgdG8gYSBzeW1s
aW5rLCBpbnN0ZWFkDQogIGNvcHlpbmcgdGhlIHN5bWxpbmsgYW5kIHRoZW4gbm90IHByZXNlcnZp
bmcgaXRzIHRpbWVzdGFtcC4NCiAgW2J1ZyBpbnRyb2R1Y2VkIGluIGNvcmV1dGlscy04LjBdDQoN
CiAgY3Agbm93IGF2b2lkcyBGSUVNQVAgaXNzdWVzIHdpdGggQlRSRlMgYmVmb3JlIExpbnV4IDIu
Ni4zOCwNCiAgd2hpY2ggY291bGQgcmVzdWx0IGluIGNvcnJ1cHQgY29waWVzIG9mIHNwYXJzZSBm
aWxlcy4NCiAgW2J1ZyBpbnRyb2R1Y2VkIGluIGNvcmV1dGlscy04LjEwXQ0KDQogIGN1dCBjb3Vs
ZCBzZWdmYXVsdCB3aGVuIGludm9rZWQgd2l0aCBhIHVzZXItc3BlY2lmaWVkIG91dHB1dA0KICBk
ZWxpbWl0ZXIgYW5kIGFuIHVuYm91bmRlZCByYW5nZSBsaWtlICItZjEyMzQ1Njc4OTAtIi4NCiAg
W2J1ZyBpbnRyb2R1Y2VkIGluIGNvcmV1dGlscy01LjMuMF0NCg0KICBkdSB3b3VsZCBpbmZsb29w
IHdoZW4gZ2l2ZW4gLS1maWxlczAtZnJvbT1ESVINCiAgW2J1ZyBpbnRyb2R1Y2VkIGluIGNvcmV1
dGlscy03LjFdDQoNCiAgc29ydCBubyBsb25nZXIgc3Bhd25zIDcgd29ya2VyIHRocmVhZHMgdG8g
c29ydCAxNiBsaW5lcw0KICBbYnVnIGludHJvZHVjZWQgaW4gY29yZXV0aWxzLTguNl0NCg0KICB0
b3VjaCBidWlsdCBvbiBTb2xhcmlzIDkgd291bGQgc2VnZmF1bHQgd2hlbiBydW4gb24gU29sYXJp
cyAxMA0KICBbYnVnIGludHJvZHVjZWQgaW4gY29yZXV0aWxzLTguOF0NCg0KICB3YyB3b3VsZCBk
ZXJlZmVyZW5jZSBhIE5VTEwgcG9pbnRlciB1cG9uIGFuIGVhcmx5IG91dC1vZi1tZW1vcnkgZXJy
b3INCiAgW2J1ZyBpbnRyb2R1Y2VkIGluIGNvcmV1dGlscy03LjFdDQoNCioqIE5ldyBmZWF0dXJl
cw0KDQogIGRkIG5vdyBhY2NlcHRzIHRoZSAnbm9jYWNoZScgZmxhZyB0byB0aGUgaWZsYWcgYW5k
IG9mbGFnIG9wdGlvbnMsDQogIHdoaWNoIHdpbGwgZGlzY2FyZCBhbnkgY2FjaGUgYXNzb2NpYXRl
ZCB3aXRoIHRoZSBmaWxlcywgb3INCiAgcHJvY2Vzc2VkIHBvcnRpb24gdGhlcmVvZi4NCg0KICBk
ZCBub3cgd2FybnMgdGhhdCAnaWZsYWc9ZnVsbGJsb2NrJyBzaG91bGQgYmUgdXNlZCwNCiAgaW4g
dmFyaW91cyBjYXNlcyB3aGVyZSBwYXJ0aWFsIHJlYWRzIGNhbiBjYXVzZSBpc3N1ZXMuDQoNCioq
IENoYW5nZXMgaW4gYmVoYXZpb3INCg0KICBjcCBub3cgYXZvaWRzIHN5bmNpbmcgZmlsZXMgd2hl
biBwb3NzaWJsZSwgd2hlbiBkb2luZyBhIEZJRU1BUCBjb3B5Lg0KICBUaGUgc3luYyBpcyBvbmx5
IG5lZWRlZCBvbiBMaW51eCBrZXJuZWxzIGJlZm9yZSAyLjYuMzkuDQogIFtUaGUgc3luYyB3YXMg
aW50cm9kdWNlZCBpbiBjb3JldXRpbHMtOC4xMF0NCg0KICBjcCBub3cgY29waWVzIGVtcHR5IGV4
dGVudHMgZWZmaWNpZW50bHksIHdoZW4gZG9pbmcgYSBGSUVNQVAgY29weS4NCiAgSXQgbm8gbG9u
Z2VyIHJlYWRzIHRoZSB6ZXJvIGJ5dGVzIGZyb20gdGhlIGlucHV0LCBhbmQgYWxzbyBjYW4gZWZm
aWNpZW50bHkNCiAgY3JlYXRlIGEgaG9sZSBpbiB0aGUgb3V0cHV0IGZpbGUgd2hlbiAtLXNwYXJz
ZT1hbHdheXMgaXMgc3BlY2lmaWVkLg0KDQogIGRmIG5vdyBhbGlnbnMgY29sdW1ucyBjb25zaXN0
ZW50bHksIGFuZCBubyBsb25nZXIgd3JhcHMgZW50cmllcw0KICB3aXRoIGxvbmdlciBkZXZpY2Ug
aWRlbnRpZmllcnMsIG92ZXIgdHdvIGxpbmVzLg0KDQogIGluc3RhbGwgbm93IHJlamVjdHMgaXRz
IGxvbmctZGVwcmVjYXRlZCAtLXByZXNlcnZlX2NvbnRleHQgb3B0aW9uLg0KICBVc2UgLS1wcmVz
ZXJ2ZS1jb250ZXh0IGluc3RlYWQuDQoNCiAgdGVzdCBub3cgYWNjZXB0cyAiPT0iIGFzIGEgc3lu
b255bSBmb3IgIj0iDQoNCi0gLS0tLS0NCmFsc28gcG9zdGVkIGFzOiBodHRwOi8vc2F2YW5uYWgu
Z251Lm9yZy9mb3J1bS9mb3J1bS5waHA/Zm9ydW1faWQ9Njc4NQ0KLS0tLS1CRUdJTiBQR1AgU0lH
TkFUVVJFLS0tLS0NClZlcnNpb246IEdudVBHIHYxLjQuMTEgKEdOVS9MaW51eCkNCg0KaVFJY0JB
RUJDQUFHQlFKTnBocExBQW9KRUgvWi9Nc0FDKzd1c0YwUC8wQmY5Um9XQ2VINDZCeUxLYU9oS2Vk
eg0KWkpROWZKR1drMm9HVjBYQ2RZdWVqYzdJS2M0dXJUTUNKSXhmUlRCTHFncGE2RDhFeHdOTWY5
ayt2RmI4OVdwdg0KbHExOHZtZng4Z2ROL21sZWhlOGMzOXdQenVlOStFQzF3L0ZPOHdsRmdzM0tv
YzJ4VXhDRS9FWUMvZ0pJa2xCTg0KSmJza25aQi9HQUN5NzBucGVVaWtFRk1nMCtzUHVmOXZNQmpa
bXdrOEdwTk5GcEx3a2tkQjVLTE11czBxeER3Mg0KNHhjSzRQcWhjdU5nMUVtWDVVWk4rNjRlUm9z
c21CREdTY0c5ejFDSHJFdlN4cWJTeWhxeG9mQjJxTlRwK0syRA0KL0V1YUpIcGplb0REQWZtNmlI
UHd4VkY2TW43ekVBNjViQlJmV3lXaktUK0wzQjlQem1NU1kxanVCUyt1amFqYQ0KdGxMQlQwOUQ0
ZXJHc2tla2lkN3ZjdWZ3TlB3dWowYzhSYUhTY2o0Z2RidW5YRHBCdTkvWk5mWVIyTWF2d3I1cQ0K
RzdxREJFSmMzQkQrcGVYSlQyNWRXakp0cUo3SHV3UU41RGFGa1Z4UzFncjB0cFRNTFNDeHhNdHlo
bDNoUkJ5bg0KRjNxY0RLN2lLZytpRTg1clkzM2RiU2huNS9xNFNmVnZlY3poN1F3SUxyNWpZZjM3
WHdodlV3NGZrdUduWTI0cQ0KYTluZG14Mi9GbWRCa3JGdGhPOWpsNDdNVXRDSmVEUStDQzFUWkdD
VU9LMlo5SDYrR2hXaTIrVklwY0xzcUMvag0KeE1JVytRY1Q1Qjg4TlZadzBtOHVyNUljdXAzNVBG
MlR4cmVVbDBxZWlRYmFHbTRLVUlmSmJvWVhWL3EvenE2cQ0Kd2I2RHQrbE9MZTk0U2VicjhZTSsN
Cj1YMC9jDQotLS0tLUVORCBQR1AgU0lHTkFUVVJFLS0tLS0NCg0KX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX18NCkdOVSBBbm5vdW5jZW1lbnQgbWFpbGluZyBs
aXN0IDxpbmZvLWdudUBnbnUub3JnPg0KaHR0cDovL2xpc3RzLmdudS5vcmcvbWFpbG1hbi9saXN0
aW5mby9pbmZvLWdudQ0K



From MAILER-DAEMON Sun Apr 17 20:36:09 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBcRl-0007gh-1Y
	for mharc-coreutils@gnu.org; Sun, 17 Apr 2011 20:36:09 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41293)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QBcRi-0007fV-IA
	for coreutils@gnu.org; Sun, 17 Apr 2011 20:36:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QBcRg-0007oR-1w
	for coreutils@gnu.org; Sun, 17 Apr 2011 20:36:06 -0400
Received: from ipmail06.adl2.internode.on.net ([150.101.137.129]:4474)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QBcRf-0007mt-LG
	for coreutils@gnu.org; Sun, 17 Apr 2011 20:36:04 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AqgEAJOGq015LHHJgWdsb2JhbAClWRUBARYmJYhvuhIOhWME
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl2.internode.on.net with ESMTP; 18 Apr 2011 10:05:55 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QBcRW-00042A-0C; Mon, 18 Apr 2011 10:35:54 +1000
Date: Mon, 18 Apr 2011 10:35:53 +1000
From: Dave Chinner <david@fromorbit.com>
To: Yongqiang Yang <xiaoqiangnk@gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110418003553.GR21395@dastard>
References: <20110414120635.GB1678@x4.trippels.de>
	<20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<BANLkTikEeXcvjgREoRCgriWAhZfnxJVtKQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <BANLkTikEeXcvjgREoRCgriWAhZfnxJVtKQ@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.129
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 00:36:07 -0000

On Sat, Apr 16, 2011 at 02:05:51PM +0800, Yongqiang Yang wrote:
> On Sat, Apr 16, 2011 at 8:50 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Thu, Apr 14, 2011 at 11:01:04PM -0600, Andreas Dilger wrote:
> >> On 2011-04-14, at 6:09 PM, Dave Chinner <david@fromorbit.com>
> >> wrote:
> >> > No, this was explicitly laid out in the fiemap interface
> >> > discussions - it's up to the applicaiton to decide if it needs
> >> > to do a sync first. That's what the FIEMAP_FLAG_SYNC control
> >> > flag is for.  This forces the fiemap call to do a fsync _before_
> >> > getting the mapping. If you want to know the exact layout of the
> >> > file is, then you must use this flag.
> >> >
> >> > Even so, it is recognised that this is racy - any use of the
> >> > block map has a time-of-read-to-time-of-use race condition that
> >> > means you have to _verify_ the copy after it completes. FYI,
> >> > that's what xfs_fsr does when copying based on extent maps - if
> >> > the inode has changed in _any way_ during the copy, it aborts
> >> > the copy of that file.
> >> >
> >> > i.e. using fiemap for copying is at best a *hint* about the
> >> > regions that need copying, and it is in no way a guarantee that
> >> > you'll get all the information you need to make accurate copy
> >> > even if you do use the synchronous variant.
> >>
> >> I would tend to agree with Pádraig. If there is data in the
> >> mapping (regardless of whether it is on disk or not), the FIEMAP
> >> should return this to the caller.  The SYNC flag is only intended
> >> to flush the data to disk for tools that are doing
> >> direct-to-disk operations on the data.
> >
> > What you are suggesting is that FIEMAP needs to be page cache
> > coherent, and that is far, far away from the intended use of the
> > interface. Even consiering that you need to looking for active pages
> > in the page cache when mapping extents say to me that you are
> > doing something very wrong.
> >
> > Unwritten extents remain unwritten until the data is physically
> > written to them. Therefore, to change their state, you need to sync
> No, buffered writes change their state without sync.

They shouldn't.

> > the data covering the range.  _Lying_ about whether an extent is in
> > the unwritten state is a really bad precedence to set, especially as
> > it is then guaranteed to change state when a crash occurs (Why did
> > recovery zero out my file? FIEMAP said it contained data before my
> > system crashed!).
> 
> All filesystems have metadata in memory which is not flushed to
> permanent storage. e.g. if a extent exists in memory, but itself and
> corresponding data are not flushed to permanent storage.

Sure, but in the case of unwritten extents, XFS does not change the
metadata state in memory until *after the physical IO is completed*.
I'm pretty sure that btrfs is the same.

IOWs, despite the fact that a buffered write has occurred, no
metadata has changed state in memory, and the extents are still
unwritten in both memory and on disk....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Sun Apr 17 20:40:52 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBcWK-0007sX-UK
	for mharc-coreutils@gnu.org; Sun, 17 Apr 2011 20:40:52 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35326)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QBcWI-0007qT-P8
	for coreutils@gnu.org; Sun, 17 Apr 2011 20:40:51 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QBcWE-0008UZ-3b
	for coreutils@gnu.org; Sun, 17 Apr 2011 20:40:50 -0400
Received: from ipmail05.adl6.internode.on.net ([150.101.137.143]:62062)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QBcWD-0008UD-Jb
	for coreutils@gnu.org; Sun, 17 Apr 2011 20:40:46 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AqgEAJOGq015LHHJgWdsb2JhbACETqELFQEBFiYliG+qS49HDoEbg056BJs2
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail05.adl6.internode.on.net with ESMTP; 18 Apr 2011 10:10:41 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QBcW8-00042j-Db; Mon, 18 Apr 2011 10:40:40 +1000
Date: Mon, 18 Apr 2011 10:40:40 +1000
From: Dave Chinner <david@fromorbit.com>
To: Theodore Tso <tytso@MIT.EDU>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110418004040.GS21395@dastard>
References: <20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.143
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 00:40:51 -0000

On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
> 
> On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
> 
> > In that case, it means cp should just always use FIEMAP_FLAG_SYNC, which is fine. 
> 
> Except that if someone is copying a large delay allocated file, it will cause 
> the file to immediately snapped to disk, which might not be the greatest
> thing in the world. 

Obvious workaround - if the initial fiemap call shows unwritten
extents, redo it with the sync flag set. Though that assumeŃ• that
you can trust things like delalloc extents to only cover the range
that valid data exists in. Which, of course, you can't assume,
either. :/

> Christoph is write, SEEK_HOLE and SEEK_DATA are
> a much better API for what cp woulld lke to do.  Unfortunately it hasn't
> been implemented yet in the VFS...

Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Sun Apr 17 22:46:08 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBeTY-0005Cp-6A
	for mharc-coreutils@gnu.org; Sun, 17 Apr 2011 22:46:08 -0400
Received: from eggs.gnu.org ([140.186.70.92]:48541)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QBeTV-0005Ch-D1
	for coreutils@gnu.org; Sun, 17 Apr 2011 22:46:06 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QBeTU-0007Ue-A0
	for coreutils@gnu.org; Sun, 17 Apr 2011 22:46:05 -0400
Received: from shawmail.shawcable.com ([64.59.128.220]:7100
	helo=mail.shawcable.com) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <adilger@dilger.ca>) id 1QBeTU-0007UI-0v
	for coreutils@gnu.org; Sun, 17 Apr 2011 22:46:04 -0400
Received: from bpd2mi1no-svcs.prod.shawcable.com ([10.0.184.120])
	by bpd2mo1no-ssvc.prod.shawcable.com with ESMTP;
	17 Apr 2011 20:46:01 -0600
X-Cloudmark-SP-Filtered: true
X-Cloudmark-SP-Result: v=1.1 cv=rhzBfl3TH6jSClqh7E8qe//hARbvbzCI0ETuu7/2kJU=
	c=1 sm=1
	a=nduc_ontD6wA:10 a=BLceEmwcHowA:10 a=c23vf5CSMVc0QQz9B4a6RA==:17
	a=7-415B0cAAAA:8 a=pkP_T5-Yn166WMDsEoYA:9 a=QEXdDO2ut3YA:10
	a=FtYAf7_czLgA:10 a=KxZ_BtGwYEi7FNd2bawA:9
	a=HpAAvcLHHh0Zw7uRqdWCyQ==:117
X-IronPort-AV: E=Sophos;i="4.64,229,1301896800"; 
	d="scan'208,217";a="533873946"
Received: from unknown (HELO bpd2mi1no-cmts.prod.shawcable.com)
	([192.168.183.120]) by bpd2mi1no-cmts.prod.shawcable.com with ESMTP;
	17 Apr 2011 20:46:01 -0600
X-reinject: true
Received: from unknown (HELO [192.168.10.170]) ([68.147.195.121])
	by bpd2mi1no-dmz.prod.shawcable.com with ESMTP;
	17 Apr 2011 20:46:00 -0600
References: <20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com>
	<4DA717B2.3020305@sandeen.net> <20110414225904.GK21395@dastard>
	<4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
In-Reply-To: <20110418004040.GS21395@dastard>
Mime-Version: 1.0 (iPhone Mail 8G4)
Content-Transfer-Encoding: 7bit
Content-Type: multipart/alternative;
	boundary=Apple-Mail-5-517528027
Message-Id: <6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
X-Mailer: iPhone Mail (8G4)
From: Andreas Dilger <adilger@dilger.ca>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Date: Sun, 17 Apr 2011 20:45:56 -0600
To: Dave Chinner <david@fromorbit.com>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 64.59.128.220
Cc: Theodore Tso <tytso@MIT.EDU>, Eric Sandeen <sandeen@sandeen.net>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 02:46:06 -0000


--Apple-Mail-5-517528027
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

On 2011-04-17, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
>>=20
>> On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
>>> In that case, it means cp should just always use FIEMAP_FLAG_SYNC, which=
 is fine.=20
>>=20
>> Except that if someone is copying a large delay allocated file, it will c=
ause=20
>> the file to immediately snapped to disk, which might not be the greatest
>> thing in the world.=20
>=20
> Obvious workaround - if the initial fiemap call shows unwritten
> extents, redo it with the sync flag set. Though that assume=D1=95 that
> you can trust things like delalloc extents to only cover the range
> that valid data exists in. Which, of course, you can't assume,
> either. :/

Always passing FIEMAP_FLAG_SYNC is fine in this case. It should only do anyt=
hing if there is unwritten data, which is the only case we are concerned wit=
h at this point.  In any case, this is a simple solution for coreutils until=
 such a time that a more complex solution is added in the kernel (if ever).

>> Christoph is write, SEEK_HOLE and SEEK_DATA are
>> a much better API for what cp woulld lke to do.  Unfortunately it hasn't
>> been implemented yet in the VFS...
>=20
> Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.

I don't see how this will change the problem in any meaningful way. There wi=
ll still need to be code that is traversing the on-disk mapping, and also ke=
eping it coherent with unwritten data in the page cache.

Since FIEMAP already exists for most Linux filesystems, it probably makes se=
nse to implement SEEK_{HOLE,DATA} by calling FIEMAP to get the disk mapping i=
n the first place.

I agree that SEEK_{HOLE,DATA} is an easier programming interface, and probab=
ly what cp, tar, etc should use, once it is implemented.=20

Cheers, Andreas=

--Apple-Mail-5-517528027
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html><body bgcolor=3D"#FFFFFF"><div><span class=3D"Apple-style-span" style=3D=
"-webkit-tap-highlight-color: rgba(26, 26, 26, 0.296875); -webkit-compositio=
n-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color=
: rgba(77, 128, 180, 0.230469); ">On 2011-04-17, at 6:40 PM, Dave Chinner &l=
t;<a href=3D"mailto:david@fromorbit.com">david@fromorbit.com</a>&gt; wrote:<=
/span></div><div></div><blockquote type=3D"cite"><div><span>On Sat, Apr 16, 2=
011 at 08:21:28AM -0400, Theodore Tso wrote:</span><blockquote type=3D"cite"=
><span>On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:</span></blockquote=
><blockquote type=3D"cite"><blockquote type=3D"cite"><span>In that case, it m=
eans cp should just always use FIEMAP_FLAG_SYNC, which is fine. </span><br><=
/blockquote></blockquote><blockquote type=3D"cite"><span></span><br></blockq=
uote><blockquote type=3D"cite"><span>Except that if someone is copying a lar=
ge delay allocated file, it will cause </span><br></blockquote><blockquote t=
ype=3D"cite"><span>the file to immediately snapped to disk, which might not b=
e the greatest</span><br></blockquote><blockquote type=3D"cite"><span>thing i=
n the world. </span><br></blockquote><span></span><br><span>Obvious workarou=
nd - if the initial fiemap call shows unwritten</span><br><span>extents, red=
o it with the sync flag set. Though that assume=D1=95 that</span><br><span>y=
ou can trust things like delalloc extents to only cover the range</span><br>=
<span>that valid data exists in. Which, of course, you can't assume,</span><=
br><span>either. :/</span><br></div></blockquote><div><br></div>Always passi=
ng&nbsp;<span class=3D"Apple-style-span" style=3D"-webkit-tap-highlight-colo=
r: rgba(26, 26, 26, 0.292969); -webkit-composition-fill-color: rgba(175, 192=
, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.2304=
69); ">FIEMAP_FLAG_SYNC is fine in this case. It should only do anything if t=
here is unwritten data, which is the only case we are concerned with at this=
 point. &nbsp;In any case, this is a simple solution for coreutils until suc=
h a time that a more complex solution is added in the kernel (if ever).</spa=
n><div><span class=3D"Apple-style-span" style=3D"-webkit-tap-highlight-color=
: rgba(26, 26, 26, 0.292969); -webkit-composition-fill-color: rgba(175, 192,=
 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.23046=
9);"><br></span><blockquote type=3D"cite"><div><blockquote type=3D"cite"><sp=
an>Christoph is write, SEEK_HOLE and SEEK_DATA are</span><br></blockquote><b=
lockquote type=3D"cite"><span>a much better API for what cp woulld lke to do=
. &nbsp;Unfortunately it hasn't</span><br></blockquote><blockquote type=3D"c=
ite"><span>been implemented yet in the VFS...</span><br></blockquote><span><=
/span><br><span>Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this p=
roblem.</span><font class=3D"Apple-style-span" color=3D"#000000"><font class=
=3D"Apple-style-span" color=3D"#0023A3"><br></font></font></div></blockquote=
><br></div><div>I don't see how this will change the problem in any meaningf=
ul way. There will still need to be code that is traversing the on-disk mapp=
ing, and also keeping it coherent with unwritten data in the page cache.</di=
v><div><br></div><div>Since FIEMAP already exists for most Linux filesystems=
, it probably makes s<span class=3D"Apple-style-span" style=3D"-webkit-tap-h=
ighlight-color: rgba(26, 26, 26, 0.296875); -webkit-composition-fill-color: r=
gba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128,=
 180, 0.230469); ">ense to implement SEEK_{HOLE,DATA} by calling FIEMAP to g=
et the disk mapping in the first place.</span></div><div><span class=3D"Appl=
e-style-span" style=3D"-webkit-tap-highlight-color: rgba(26, 26, 26, 0.29687=
5); -webkit-composition-fill-color: rgba(175, 192, 227, 0.230469); -webkit-c=
omposition-frame-color: rgba(77, 128, 180, 0.230469); "><br></span></div><di=
v><span class=3D"Apple-style-span" style=3D"-webkit-tap-highlight-color: rgb=
a(26, 26, 26, 0.296875); -webkit-composition-fill-color: rgba(175, 192, 227,=
 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180, 0.230469); "=
>I agree that&nbsp;</span><span class=3D"Apple-style-span" style=3D"-webkit-=
tap-highlight-color: rgba(26, 26, 26, 0.292969); -webkit-composition-fill-co=
lor: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77=
, 128, 180, 0.230469); ">SEEK_{HOLE,DATA} is an easier programming interface=
, and probably what cp, tar, etc should use, once it is implemented.&nbsp;</=
span></div><div><span class=3D"Apple-style-span" style=3D"-webkit-tap-highli=
ght-color: rgba(26, 26, 26, 0.292969); -webkit-composition-fill-color: rgba(=
175, 192, 227, 0.230469); -webkit-composition-frame-color: rgba(77, 128, 180=
, 0.230469); "><br></span></div><div><span class=3D"Apple-style-span" style=3D=
"-webkit-tap-highlight-color: rgba(26, 26, 26, 0.292969); -webkit-compositio=
n-fill-color: rgba(175, 192, 227, 0.230469); -webkit-composition-frame-color=
: rgba(77, 128, 180, 0.230469); ">Cheers, Andreas</span></div></body></html>=

--Apple-Mail-5-517528027--


From MAILER-DAEMON Mon Apr 18 09:21:54 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBoOo-0008TL-IF
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 09:21:54 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41135)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QBoOf-0008Ro-8u
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:21:53 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QBoOZ-0006pl-6t
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:21:45 -0400
Received: from mx1.redhat.com ([209.132.183.28]:18980)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QBoOY-0006pb-Sa
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:21:39 -0400
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3IDLahl027638
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Mon, 18 Apr 2011 09:21:37 -0400
Received: from [10.3.113.106] (ovpn-113-106.phx2.redhat.com [10.3.113.106])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p3IDLaEP004569; Mon, 18 Apr 2011 09:21:36 -0400
Message-ID: <4DAC3AE0.5060507@redhat.com>
Date: Mon, 18 Apr 2011 07:21:36 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: adekunleadekoya@gmail.com
Subject: Re: coreutils-8.11 released [stable]
References: <87pqopx1lq.fsf@rho.meyering.net>
	<56752866-1302988026-cardhu_decombobulator_blackberry.rim.net-1192226888-@b11.c10.bise7.blackberry>
In-Reply-To: <56752866-1302988026-cardhu_decombobulator_blackberry.rim.net-1192226888-@b11.c10.bise7.blackberry>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enigFC3C2EC8F4688D26056F407E"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 13:21:53 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigFC3C2EC8F4688D26056F407E
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

[dropping spurious CC's - this is _not_ worthy of an announce post, and
using reply-all was the wrong thing to do]

On 04/16/2011 03:07 PM, adekunleadekoya@gmail.com wrote:
> Can anyone help with the request below ?

In general, it is better to ask a new question by starting a new thread,
rather than replying to an unrelated post.  Additionally, top-posting
and quoting the full body of an unrelated post are frowned upon on
technical mailing lists.

>=20
> Sir,
>=20
> Pardon me that I have to ask this,sir!
>=20
> My project is about compiler development. I surely have a number of imp=
ortant building blocks that I need to get the job done except for the int=
erface between the compiled code and the interpreter/VM/assembler that wo=
uld execute the target code.
>=20
> I actually need my java application(an IDE that invokes the compiler) t=
o invoke the interpreter. I want an embeddable interpreter that could get=
 input(compiled code) from my java application and also send its output b=
ack to my application.

Unfortunately, the GNU coreutils list is not involved with anything
java-related.  You will not find any good answers on this list, and all
I can do is recommend that you google for a better forum that is more
suited to your particular line of questions.

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enigFC3C2EC8F4688D26056F407E
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNrDrgAAoJEKeha0olJ0NqorkH/jR/4wu2Y7SD0RFqtqNjo854
RONPzfJpoM39HSSdS4ZNUzQKyzaE5W2vfbMnCj/wsEWekRpwZxOMtVvFZA3GyXLz
sebk2uwN7Xo0ca+gcYFEZZW7enOKdWrWrTNHfX5dGjZRpYMc3kNCmYPABAlN2c/v
JbiJkwWVzUEz4XOPW6hVCLgZC0s/LSjB+/rZ2FMK/So14Nsv/pr63qwI5U3QJxs5
xqe+gytlq7J4LZotYN0Wq0QVDmHZlFZB/kIK7oqNFDJ2Zap63jtLg2TYX5SAaRiB
UzKBb3KLPG8pBSeHTjn+dPltUTTImnL85nqbq2aHF6kBiYFyIYoA1xps2CF/WPI=
=06LC
-----END PGP SIGNATURE-----

--------------enigFC3C2EC8F4688D26056F407E--


From MAILER-DAEMON Mon Apr 18 09:34:58 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBobS-0003hx-DS
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 09:34:58 -0400
Received: from eggs.gnu.org ([140.186.70.92]:36721)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1QBobM-0003ec-51
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:34:56 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <pth@suse.de>) id 1QBobH-00014p-U5
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:34:52 -0400
Received: from cantor.suse.de ([195.135.220.2]:54077 helo=mx1.suse.de)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <pth@suse.de>)
	id 1QBobH-00014j-Oy
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:34:47 -0400
Received: from relay1.suse.de (charybdis-ext.suse.de [195.135.221.2])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mx1.suse.de (Postfix) with ESMTP id 0D1CA93987;
	Mon, 18 Apr 2011 15:34:45 +0200 (CEST)
Date: Mon, 18 Apr 2011 15:34:44 +0200
From: Philipp Thomas <pth@suse.de>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: Supporting reflinked/cow files in coreutils
Message-ID: <20110418133444.GE28014@paradies.suse.de>
References: <20110414143352.GC5231@paradies.suse.de>
	<4DA71296.8040000@draigBrady.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <4DA71296.8040000@draigBrady.com>
X-Operating-System: openSUSE - Kernel Linux 2.6.38-31-desktop x86_64
Organization: SUSE LINUX Products GmbH, =?iso-8859-1?Q?G?=
	=?iso-8859-1?Q?F=3A_Markus_Rex=2C_HRB_16746_=28AG_N=FCrnberg=29?=
User-Agent: Mutt/1.5.21 (2010-09-15)
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4-2.6
X-Received-From: 195.135.220.2
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 13:34:56 -0000

* P=E1draig Brady (P@draigBrady.com) [20110414 17:30]:

> I don't think there is anything else you could do,
> like minimizing reads from a set of files with
> shared extents, as there is no way to ID the extent.
> Oh I suppose you could by using device_id, physical_offset.
> You could extend the di_set stuff we have for matching
> hardlinks to do that I suppose. But then one can currently
> only clone a file, not an extent (on BTRFS at least).
> What is your use case exactly?

As far as I understood the request, it's for reflink support in du so tha=
t
it's possible to see the space savings reflinks give you. Given your hint
I'll have a look at the di_set stuff to see how this can possibly be
extended to use the FIEMAP info.

> Also it would be better to send details to coreutils@gnu.org
> as others might have suggestions.

Which I'm doing hereby. BTW, could somebody point me to the documents for
copyright assignment? I have one on file for GCC but now I'd need one for
coreutils.

Philpp


From MAILER-DAEMON Mon Apr 18 09:54:41 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBouX-000805-Ag
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 09:54:41 -0400
Received: from eggs.gnu.org ([140.186.70.92]:53426)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QBouR-0007wZ-60
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:54:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QBouN-00058w-1v
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:54:35 -0400
Received: from rcsinet10.oracle.com ([148.87.113.121]:54985)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QBouM-00058b-SZ
	for coreutils@gnu.org; Mon, 18 Apr 2011 09:54:31 -0400
Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117])
	by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id
	p3IDsNns010419
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Mon, 18 Apr 2011 13:54:25 GMT
Received: from acsmt357.oracle.com (acsmt357.oracle.com [141.146.40.157])
	by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id
	p3IDsN86004655
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 18 Apr 2011 13:54:23 GMT
Received: from abhmt021.oracle.com (abhmt021.oracle.com [141.146.116.30])
	by acsmt357.oracle.com (8.12.11.20060308/8.12.11) with ESMTP id
	p3IDsMjM011816; Mon, 18 Apr 2011 08:54:22 -0500
Received: from [192.168.1.100] (/123.119.97.134)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Mon, 18 Apr 2011 06:54:22 -0700
Subject: Re: Supporting reflinked/cow files in coreutils
Mime-Version: 1.0 (Apple Message framework v1082)
Content-Type: multipart/alternative; boundary=Apple-Mail-2-557620598
From: Jeff liu <jeff.liu@oracle.com>
In-Reply-To: <20110418133444.GE28014@paradies.suse.de>
Date: Mon, 18 Apr 2011 21:54:11 +0800
Message-Id: <6AA787AA-590D-4C24-A7C0-18A8C17E74BC@oracle.com>
References: <20110414143352.GC5231@paradies.suse.de>
	<4DA71296.8040000@draigBrady.com>
	<20110418133444.GE28014@paradies.suse.de>
To: Philipp Thomas <pth@suse.de>
X-Mailer: Apple Mail (2.1082)
X-Source-IP: acsmt357.oracle.com [141.146.40.157]
X-Auth-Type: Internal IP
X-CT-RefId: str=0001.0A090206.4DAC428F.0183:SCFSTAT5015188, ss=1, pt=DBB_65838,
	fgs=0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 148.87.113.121
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 13:54:40 -0000


--Apple-Mail-2-557620598
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=GB2312

FYI, I once implemented a similar patch to teach du(1) to show the disk =
footprint with reflinked stuff for OCFS2 only.

It is make use of rb-tree to figure out the shared extents and calculate =
the exactly used space eventually.
http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007293.html
http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007288.html
http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007287.html

I have not submit the patch set to Coreutils since I guess it is hard to =
accept by upstream as reflink and FIEMAP_EXTENT_SHARED flag are not =
spread over the general filesystems.

Thanks,
-Jeff

> * P=A8=A2draig Brady (P@draigBrady.com) [20110414 17:30]:
>=20
>> I don't think there is anything else you could do,
>> like minimizing reads from a set of files with
>> shared extents, as there is no way to ID the extent.
>> Oh I suppose you could by using device_id, physical_offset.
>> You could extend the di_set stuff we have for matching
>> hardlinks to do that I suppose. But then one can currently
>> only clone a file, not an extent (on BTRFS at least).
>> What is your use case exactly?
>=20
> As far as I understood the request, it's for reflink support in du so =
that
> it's possible to see the space savings reflinks give you. Given your =
hint
> I'll have a look at the di_set stuff to see how this can possibly be
> extended to use the FIEMAP info.
>=20
>> Also it would be better to send details to coreutils@gnu.org
>> as others might have suggestions.
>=20
> Which I'm doing hereby. BTW, could somebody point me to the documents =
for
> copyright assignment? I have one on file for GCC but now I'd need one =
for
> coreutils.
>=20
> Philpp
>=20


--Apple-Mail-2-557620598
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=GB2312

<html><head></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space; =
"><div><div>FYI,&nbsp;I once implemented a similar patch to teach du(1) =
to show the disk footprint with reflinked stuff for OCFS2 =
only.</div><div><br></div><div>It is make use of rb-tree to figure out =
the shared extents and calculate the exactly used space =
eventually.</div><div><a =
href=3D"http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007293.=
html">http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007293.ht=
ml</a></div><div><a =
href=3D"http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007288.=
html">http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007288.ht=
ml</a></div><div><a =
href=3D"http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007287.=
html">http://oss.oracle.com/pipermail/ocfs2-devel/2010-September/007287.ht=
ml</a></div><div><br></div><div><div>I have not submit the patch =
set&nbsp;to Coreutils since I guess it is hard to accept by upstream as =
reflink and FIEMAP_EXTENT_SHARED flag are<font class=3D"Apple-style-span" =
face=3D"monospace"><span class=3D"Apple-style-span" style=3D"white-space: =
pre; ">&nbsp;</span></font>not spread over the general =
filesystems.</div></div><div><br></div><div>Thanks,</div><div>-Jeff</div><=
br class=3D"Apple-interchange-newline"><blockquote type=3D"cite"><div>* =
P=A8=A2draig Brady (<a =
href=3D"mailto:P@draigBrady.com">P@draigBrady.com</a>) [20110414 =
17:30]:<br><br><blockquote type=3D"cite">I don't think there is anything =
else you could do,<br></blockquote><blockquote type=3D"cite">like =
minimizing reads from a set of files with<br></blockquote><blockquote =
type=3D"cite">shared extents, as there is no way to ID the =
extent.<br></blockquote><blockquote type=3D"cite">Oh I suppose you could =
by using device_id, physical_offset.<br></blockquote><blockquote =
type=3D"cite">You could extend the di_set stuff we have for =
matching<br></blockquote><blockquote type=3D"cite">hardlinks to do that =
I suppose. But then one can currently<br></blockquote><blockquote =
type=3D"cite">only clone a file, not an extent (on BTRFS at =
least).<br></blockquote><blockquote type=3D"cite">What is your use case =
exactly?<br></blockquote><br>As far as I understood the request, it's =
for reflink support in du so that<br>it's possible to see the space =
savings reflinks give you. Given your hint<br>I'll have a look at the =
di_set stuff to see how this can possibly be<br>extended to use the =
FIEMAP info.<br><br><blockquote type=3D"cite">Also it would be better to =
send details to <a =
href=3D"mailto:coreutils@gnu.org">coreutils@gnu.org</a><br></blockquote><b=
lockquote type=3D"cite">as others might have =
suggestions.<br></blockquote><br>Which I'm doing hereby. BTW, could =
somebody point me to the documents for<br>copyright assignment? I have =
one on file for GCC but now I'd need one =
for<br>coreutils.<br><br>Philpp<br><br></div></blockquote></div><br></body=
></html>=

--Apple-Mail-2-557620598--


From MAILER-DAEMON Mon Apr 18 10:06:48 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QBp6G-0005YL-NG
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 10:06:48 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41471)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QBp67-0005Xh-Vc
	for coreutils@gnu.org; Mon, 18 Apr 2011 10:06:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QBp63-0007Vq-G0
	for coreutils@gnu.org; Mon, 18 Apr 2011 10:06:39 -0400
Received: from mx.meyering.net ([82.230.74.64]:51919)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QBp63-0007Vh-2i
	for coreutils@gnu.org; Mon, 18 Apr 2011 10:06:35 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 671C76021D; Mon, 18 Apr 2011 16:06:33 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Philipp Thomas <pth@suse.de>
Subject: Re: Supporting reflinked/cow files in coreutils
In-Reply-To: <20110418133444.GE28014@paradies.suse.de> (Philipp Thomas's
	message of "Mon, 18 Apr 2011 15:34:44 +0200")
References: <20110414143352.GC5231@paradies.suse.de>
	<4DA71296.8040000@draigBrady.com>
	<20110418133444.GE28014@paradies.suse.de>
Date: Mon, 18 Apr 2011 16:06:33 +0200
Message-ID: <87vcyb7ixi.fsf@rho.meyering.net>
Lines: 27
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 18 Apr 2011 14:06:48 -0000

Philipp Thomas wrote:
> * P=E1draig Brady (P@draigBrady.com) [20110414 17:30]:
>
>> I don't think there is anything else you could do,
>> like minimizing reads from a set of files with
>> shared extents, as there is no way to ID the extent.
>> Oh I suppose you could by using device_id, physical_offset.
>> You could extend the di_set stuff we have for matching
>> hardlinks to do that I suppose. But then one can currently
>> only clone a file, not an extent (on BTRFS at least).
>> What is your use case exactly?
>
> As far as I understood the request, it's for reflink support in du so that
> it's possible to see the space savings reflinks give you. Given your hint
> I'll have a look at the di_set stuff to see how this can possibly be
> extended to use the FIEMAP info.
>
>> Also it would be better to send details to coreutils@gnu.org
>> as others might have suggestions.
>
> Which I'm doing hereby. BTW, could somebody point me to the documents for
> copyright assignment? I have one on file for GCC but now I'd need one for
> coreutils.

Sure:

    http://git.sv.gnu.org/cgit/coreutils.git/tree/HACKING#n429


From MAILER-DAEMON Mon Apr 18 21:58:28 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC0Cx-0007GM-TR
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 21:58:28 -0400
Received: from eggs.gnu.org ([140.186.70.92]:45984)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC0Cv-0007Fe-3G
	for coreutils@gnu.org; Mon, 18 Apr 2011 21:58:26 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC0Ct-0002cY-T5
	for coreutils@gnu.org; Mon, 18 Apr 2011 21:58:25 -0400
Received: from mail-px0-f179.google.com ([209.85.212.179]:58302)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC0Ct-0002cP-OK
	for coreutils@gnu.org; Mon, 18 Apr 2011 21:58:23 -0400
Received: by pxi2 with SMTP id 2so3991190pxi.38
	for <coreutils@gnu.org>; Mon, 18 Apr 2011 18:58:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=Df9d21Fxj3hIfTxIxNaFHoYtPg7I7OlViEoyr7Nbmt8=;
	b=lNW6UBJPjtTHKT/oLV5F+sAphpoJ9DpdVIrM29TDO8KnEX2AHIY+ewJRr0LaeZSWKH
	wS+yBgpHliyeSC5DCqA7O55xLQSjqJCkQtCkR1g/JuWV6K6kH8gCPRvEogb489EG3V1e
	VCKhvZKUekiyXcPOUp1FhdzEqnZZvO/zbH5c4=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=GYGD/RvWewwwReZ53+5FcXxIsCcq1U8qhVopaTQ9cdq5tgohEi24cVo3vRnXNsUpXy
	TiVLrEqEa+eCSJqMBT/PxCoQsHEnFdzShigHWOQL6MeTZ4eQddoGVxROv+p+N3pMJ995
	z4xwAllYO3tmn2TFwCpeDa5BQcNcvC5wYgUN0=
MIME-Version: 1.0
Received: by 10.68.36.170 with SMTP id r10mr7614961pbj.504.1303178295471; Mon,
	18 Apr 2011 18:58:15 -0700 (PDT)
Received: by 10.68.66.164 with HTTP; Mon, 18 Apr 2011 18:58:15 -0700 (PDT)
In-Reply-To: <6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
References: <20110414140222.GB1679@x4.trippels.de>
	<4DA70BD3.1070409@draigBrady.com> <4DA717B2.3020305@sandeen.net>
	<20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
Date: Tue, 19 Apr 2011 09:58:15 +0800
Message-ID: <BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Andreas Dilger <adilger@dilger.ca>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.212.179
Cc: Theodore Tso <tytso@mit.edu>, Dave Chinner <david@fromorbit.com>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 01:58:26 -0000

On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wrote:
> On 2011-04-17, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote:
>
> On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
>
> On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
>
> In that case, it means cp should just always use FIEMAP_FLAG_SYNC, which =
is
> fine.
>
> Except that if someone is copying a large delay allocated file, it will
> cause
>
> the file to immediately snapped to disk, which might not be the greatest
>
> thing in the world.
>
> Obvious workaround - if the initial fiemap call shows unwritten
> extents, redo it with the sync flag set. Though that assume=D1=95 that
> you can trust things like delalloc extents to only cover the range
> that valid data exists in. Which, of course, you can't assume,
> either. :/
>
> Always passing=C2=A0FIEMAP_FLAG_SYNC is fine in this case. It should only=
 do
> anything if there is unwritten data, which is the only case we are concer=
ned
> with at this point. =C2=A0In any case, this is a simple solution for core=
utils
> until such a time that a more complex solution is added in the kernel (if
> ever).
>
> Christoph is write, SEEK_HOLE and SEEK_DATA are
>
> a much better API for what cp woulld lke to do. =C2=A0Unfortunately it ha=
sn't
>
> been implemented yet in the VFS...
>
> Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
>
> I don't see how this will change the problem in any meaningful way. There
> will still need to be code that is traversing the on-disk mapping, and al=
so
> keeping it coherent with unwritten data in the page cache.

It seems that we are being messed up by page cache and disk.
Unwritten flag returned from FIEMAP indicates blocks on disk are not
written, but it does not say if there is data in page cache.  So
FIEMAP itself just tells user the map on disk.  However there is an
exception for delayed allocation,  FIEMAP tells users the data is in
page cache.

Maybe FIEMAP should return all known messages for unwritten extent, if
unwritten data exists in page cache, FIEMAP should let users know that
data is in page cache and space on disk has been preallocated, but
data has not been flushed into disk.  Actually, delayed allocation has
done like this. Then user-space applications can determine how to do.
Taking cp as an example, it will copy from page cache rather ignore
it.


We need a definite definition for FIEMAP, in other words, it tells
users map on disk or both disk and page cache.

If the former one is taken, then FIEMAP should not consider delayed allocat=
ion.
otherwise, FIEMAP should return all known messages for unwritten case
like delayed allocation.

> Since FIEMAP already exists for most Linux filesystems, it probably makes
> sense to implement SEEK_{HOLE,DATA} by calling FIEMAP to get the disk
> mapping in the first place.
> I agree that=C2=A0SEEK_{HOLE,DATA} is an easier programming interface, an=
d
> probably what cp, tar, etc should use, once it is implemented.
> Cheers, Andreas
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
>
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Mon Apr 18 23:00:10 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC1Ag-0003Rm-FG
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 23:00:10 -0400
Received: from eggs.gnu.org ([140.186.70.92]:48580)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QC1Ae-0003Re-M8
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:00:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QC1Ad-0003Sr-GV
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:00:08 -0400
Received: from li9-11.members.linode.com ([67.18.176.11]:37313
	helo=test.thunk.org) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QC1Ad-0003RJ-Bz
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:00:07 -0400
Received: from root (helo=tytso-glaptop)
	by test.thunk.org with local-esmtp (Exim 4.69)
	(envelope-from <tytso@thunk.org>)
	id 1QC1AO-0000QP-2C; Tue, 19 Apr 2011 02:59:52 +0000
Received: from tytso by tytso-glaptop with local (Exim 4.71)
	(envelope-from <tytso@thunk.org>)
	id 1QC1AL-0004kp-Vi; Mon, 18 Apr 2011 22:59:49 -0400
Date: Mon, 18 Apr 2011 22:59:49 -0400
From: Ted Ts'o <tytso@mit.edu>
To: Yongqiang Yang <xiaoqiangnk@gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419025949.GA3030@thunk.org>
References: <20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on test.thunk.org); SAEximRunCond expanded to false
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 1)
X-Received-From: 67.18.176.11
Cc: Andreas Dilger <adilger@dilger.ca>, Dave Chinner <david@fromorbit.com>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 03:00:09 -0000

On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
> On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wrote:
> > Always passing FIEMAP_FLAG_SYNC is fine in this case. It should
> > only do anything if there is unwritten data, which is the only
> > case we are concerned with at this point.  In any case, this is a
> > simple solution for coreutils until such a time that a more
> > complex solution is added in the kernel (if ever).

I would recommend that coreutils check i_blocks and i_size and only
try using fiemap (with FIEMAP_FLAG_SYNC) if the file appears to be
sparse.  That's because FIEMAP_FLAG_SYNC will do the effectively
equivalent of an fsync() system call.  Otherwise, in the case of a
freshly untar'ed directory hierarchy which is then copied using "cp
-r", cp would end up calling fsync() for each file in the directory,
with the disastrous performance result that one might expect.

If cp only tries the fiemap optimization on files that appear to be
sparse, it should avoid this problem.

> > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
> >
> > I don't see how this will change the problem in any meaningful way. There
> > will still need to be code that is traversing the on-disk mapping, and also
> > keeping it coherent with unwritten data in the page cache.

The advantage of SEEK_HOLE/SEEK_DATA is that we don't need to force an
fsync() of the data.

> It seems that we are being messed up by page cache and disk.
> Unwritten flag returned from FIEMAP indicates blocks on disk are not
> written, but it does not say if there is data in page cache.  So
> FIEMAP itself just tells user the map on disk.  However there is an
> exception for delayed allocation,  FIEMAP tells users the data is in
> page cache.
> 
> Maybe FIEMAP should return all known messages for unwritten extent, if
> unwritten data exists in page cache, FIEMAP should let users know that
> data is in page cache and space on disk has been preallocated, but
> data has not been flushed into disk.  Actually, delayed allocation has
> done like this. Then user-space applications can determine how to do.
> Taking cp as an example, it will copy from page cache rather ignore
> it.
>
> We need a definite definition for FIEMAP, in other words, it tells
> users map on disk or both disk and page cache.
> 
> If the former one is taken, then FIEMAP should not consider delayed
> allocation.  otherwise, FIEMAP should return all known messages for
> unwritten case like delayed allocation.

The fact that the FIEMAP interface deifnition includes an delayed
allocation bit could be a strong indication that unlike the XFS's bmap
interface, that this interface is supposed to return information
taking into account both on-disk and page cache information.  If this
is the case, then even though there might be a single on-disk
(uninitialized) extent, if there are pages in the page cache that have
not yet been written out yet, but which are described by that on-disk
extent, then instead of returning a single struct fiemap_extent for
that on-disk extent, the fiemap ioctl would need to return multiple
struct fiemap_extents, where some would have the FIEMAP_UNWRITTEN bit,
and others would not (since data has been written to the page cache,
even if it hasn't been flushed to disk yet).

But yes, if we're going to make the case that the FIEMAP interface is
only intended to reflect the on-disk information, then the DELALLOC
bit shouldn't be returned at all, and we should deprecate it.
Anything else leads us to a inconsistent interface.

> > Since FIEMAP already exists for most Linux filesystems, it probably makes
> > sense to implement SEEK_{HOLE,DATA} by calling FIEMAP to get the disk
> > mapping in the first place.

Not if it means forcing an FIEMAP_FLAG_SYNC, which implies an fsync().
If the only way to get consistent data across ext4, btrfs, xfs,
etc. is to force userspace to issue a FIEMAP_FLAG_SYNC, then we need
to have a separate interface of SEEK_HOLE/SEEK_DATA that doesn't
require flushing data to the disk first.

Maybe coreutils will need to use FIEMAP_FLAG_SYNC initially, since
it's the only way to guarantee correct behaviour for XFS.  But I would
really rather that be the long-term way we leave things!

						- Ted


From MAILER-DAEMON Mon Apr 18 23:05:20 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC1Fg-0004IA-MC
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 23:05:20 -0400
Received: from eggs.gnu.org ([140.186.70.92]:36277)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QC1Fe-0004HQ-Ve
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:05:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QC1Fd-0004Wc-W7
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:05:18 -0400
Received: from sandeen.net ([63.231.237.45]:48498 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QC1Fd-0004WA-Nj
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:05:17 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 940B648FDA82;
	Mon, 18 Apr 2011 22:05:15 -0500 (CDT)
Message-ID: <4DACFBEB.9040909@sandeen.net>
Date: Mon, 18 Apr 2011 22:05:15 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Ted Ts'o <tytso@mit.edu>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419025949.GA3030@thunk.org>
In-Reply-To: <20110419025949.GA3030@thunk.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: Andreas Dilger <adilger@dilger.ca>, Dave Chinner <david@fromorbit.com>,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 03:05:19 -0000

On 4/18/11 9:59 PM, Ted Ts'o wrote:

...

> Maybe coreutils will need to use FIEMAP_FLAG_SYNC initially, since
> it's the only way to guarantee correct behaviour for XFS.  But I would
> really rather that be the long-term way we leave things!

XFS ... or ext4:

# xfs_io -Ff -c "falloc 0 1m" -c "pwrite 0 512k" testfile; /root/fiemap-test testfile
wrote 524288/524288 bytes at offset 0
512 KiB, 128 ops; 0.0000 sec (161.342 MiB/sec and 41303.6463 ops/sec)
start 0 length -1 flags 0x0 count 32
ext:   0 logical: [       0..     255] phys:    34048..   34303 flags: 0x801 tot: 256

# uname -r
2.6.39-0.rc3.git2.0.fc16.x86_64

Above is on ext4.  It behaves exactly like XFS in my testing; data in the page cache does not cause fiemap to return anything other than "unwritten" for preallocated extents.

-Eric


From MAILER-DAEMON Mon Apr 18 23:30:25 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC1dw-0005F6-TG
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 23:30:25 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60489)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC1du-0005F0-MH
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:30:23 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC1dt-0000TP-Ec
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:30:22 -0400
Received: from mail-px0-f179.google.com ([209.85.212.179]:45372)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC1dt-0000TD-32
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:30:21 -0400
Received: by pxi2 with SMTP id 2so4063789pxi.38
	for <coreutils@gnu.org>; Mon, 18 Apr 2011 20:30:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=cHcZUkbL1U9uGqA9FmWsNDBvYXp58u6SZ4rbjhtohuo=;
	b=wHSs8ytGL1EbiXmC74NtSIcR83yR4eZaMCxXxfamKJLJkJtquFs9GSssSXB/a3044L
	ZUfG41ZZmXZjltV9TfX/dg/OxD8ptO9udQ4qbV1KtzrqlfuDbvPMQMLBojmt4Sgah46D
	EUXL/eZ0T1214fDH33QU2i2CEbHpIZoII/pFk=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=SM/xEcixgB+tIsVfuMW74KmEGYcZN1KffekCxSLXKwpfZtmAPR+OMR1p1nOKVe+OJM
	CbnR6DAa5f0/fc4rxfx1M0tnqJlbvEVYtnCk5oIXoL2RAFWE3A8iDMiMSU9/Jn7pcou1
	aSHwZs6v866OrjFKuzPH2I4MS162Bc6N+cFtE=
MIME-Version: 1.0
Received: by 10.68.25.1 with SMTP id y1mr1668103pbf.169.1303183819737; Mon, 18
	Apr 2011 20:30:19 -0700 (PDT)
Received: by 10.68.66.164 with HTTP; Mon, 18 Apr 2011 20:30:19 -0700 (PDT)
In-Reply-To: <20110419025949.GA3030@thunk.org>
References: <20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419025949.GA3030@thunk.org>
Date: Tue, 19 Apr 2011 11:30:19 +0800
Message-ID: <BANLkTinM5yWeT7X7pQ3AYYYvK9gTSLO8XA@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: "Ted Ts'o" <tytso@mit.edu>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.212.179
Cc: Andreas Dilger <adilger@dilger.ca>, Dave Chinner <david@fromorbit.com>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 03:30:23 -0000

On Tue, Apr 19, 2011 at 10:59 AM, Ted Ts'o <tytso@mit.edu> wrote:
> On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
>> On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wro=
te:
>> > Always passing FIEMAP_FLAG_SYNC is fine in this case. It should
>> > only do anything if there is unwritten data, which is the only
>> > case we are concerned with at this point. =A0In any case, this is a
>> > simple solution for coreutils until such a time that a more
>> > complex solution is added in the kernel (if ever).
>
> I would recommend that coreutils check i_blocks and i_size and only
> try using fiemap (with FIEMAP_FLAG_SYNC) if the file appears to be
> sparse. =A0That's because FIEMAP_FLAG_SYNC will do the effectively
> equivalent of an fsync() system call. =A0Otherwise, in the case of a
> freshly untar'ed directory hierarchy which is then copied using "cp
> -r", cp would end up calling fsync() for each file in the directory,
> with the disastrous performance result that one might expect.
>
> If cp only tries the fiemap optimization on files that appear to be
> sparse, it should avoid this problem.
>
>> > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
>> >
>> > I don't see how this will change the problem in any meaningful way. Th=
ere
>> > will still need to be code that is traversing the on-disk mapping, and=
 also
>> > keeping it coherent with unwritten data in the page cache.
>
> The advantage of SEEK_HOLE/SEEK_DATA is that we don't need to force an
> fsync() of the data.
>
>> It seems that we are being messed up by page cache and disk.
>> Unwritten flag returned from FIEMAP indicates blocks on disk are not
>> written, but it does not say if there is data in page cache. =A0So
>> FIEMAP itself just tells user the map on disk. =A0However there is an
>> exception for delayed allocation, =A0FIEMAP tells users the data is in
>> page cache.
>>
>> Maybe FIEMAP should return all known messages for unwritten extent, if
>> unwritten data exists in page cache, FIEMAP should let users know that
>> data is in page cache and space on disk has been preallocated, but
>> data has not been flushed into disk. =A0Actually, delayed allocation has
>> done like this. Then user-space applications can determine how to do.
>> Taking cp as an example, it will copy from page cache rather ignore
>> it.
>>
>> We need a definite definition for FIEMAP, in other words, it tells
>> users map on disk or both disk and page cache.
>>
>> If the former one is taken, then FIEMAP should not consider delayed
>> allocation. =A0otherwise, FIEMAP should return all known messages for
>> unwritten case like delayed allocation.
>
> The fact that the FIEMAP interface deifnition includes an delayed
> allocation bit could be a strong indication that unlike the XFS's bmap
> interface, that this interface is supposed to return information
> taking into account both on-disk and page cache information. =A0If this
> is the case, then even though there might be a single on-disk
> (uninitialized) extent, if there are pages in the page cache that have
> not yet been written out yet, but which are described by that on-disk
> extent, then instead of returning a single struct fiemap_extent for
> that on-disk extent, the fiemap ioctl would need to return multiple
> struct fiemap_extents, where some would have the FIEMAP_UNWRITTEN bit,
> and others would not (since data has been written to the page cache,
> even if it hasn't been flushed to disk yet).
Maybe we can add a SPLIT flag like MERGE for ext3, which is set if
there are pages in page cache that have not been written out, but
which are described by unwritten extent on disk, and which does not
cover the whole extent.

Thus, an extent returned by FIEMAP may have UNWRITTEN, NOBYPASS and SPLIT f=
lags.

I noticed that there is a NOBYPASS flag in initial FIEMAP, which
indicates data has not been written out to disk.  But it does not
exist in current implementation any more.

>
> But yes, if we're going to make the case that the FIEMAP interface is
> only intended to reflect the on-disk information, then the DELALLOC
> bit shouldn't be returned at all, and we should deprecate it.
> Anything else leads us to a inconsistent interface.
>
>> > Since FIEMAP already exists for most Linux filesystems, it probably ma=
kes
>> > sense to implement SEEK_{HOLE,DATA} by calling FIEMAP to get the disk
>> > mapping in the first place.
>
> Not if it means forcing an FIEMAP_FLAG_SYNC, which implies an fsync().
> If the only way to get consistent data across ext4, btrfs, xfs,
> etc. is to force userspace to issue a FIEMAP_FLAG_SYNC, then we need
> to have a separate interface of SEEK_HOLE/SEEK_DATA that doesn't
> require flushing data to the disk first.
>
> Maybe coreutils will need to use FIEMAP_FLAG_SYNC initially, since
> it's the only way to guarantee correct behaviour for XFS. =A0But I would
> really rather that be the long-term way we leave things!
>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0- Ted
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Mon Apr 18 23:45:11 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC1sF-0005oo-2I
	for mharc-coreutils@gnu.org; Mon, 18 Apr 2011 23:45:11 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35111)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC1sC-0005o1-60
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:45:09 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC1sA-0002X0-SG
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:45:08 -0400
Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:51747)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC1s9-0002WA-Vx
	for coreutils@gnu.org; Mon, 18 Apr 2011 23:45:06 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AhAEAEkCrU15LHHJgWdsb2JhbACET6BxFQEBFiYliG+sGZEKDoEbg056BA
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl6.internode.on.net with ESMTP; 19 Apr 2011 13:14:57 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QC1rz-0006ZH-8T; Tue, 19 Apr 2011 13:44:55 +1000
Date: Tue, 19 Apr 2011 13:44:55 +1000
From: Dave Chinner <david@fromorbit.com>
To: Yongqiang Yang <xiaoqiangnk@gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419034455.GB23985@dastard>
References: <20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.145
Cc: Andreas Dilger <adilger@dilger.ca>, Theodore Tso <tytso@mit.edu>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 03:45:09 -0000

On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
> On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wrote:
> > On 2011-04-17, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote:
> >
> > On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
> >
> > On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
> >
> > In that case, it means cp should just always use FIEMAP_FLAG_SYNC, which is
> > fine.
> >
> > Except that if someone is copying a large delay allocated file, it will
> > cause
> >
> > the file to immediately snapped to disk, which might not be the greatest
> >
> > thing in the world.
> >
> > Obvious workaround - if the initial fiemap call shows unwritten
> > extents, redo it with the sync flag set. Though that assumeŃ• that
> > you can trust things like delalloc extents to only cover the range
> > that valid data exists in. Which, of course, you can't assume,
> > either. :/
> >
> > Always passingÂ FIEMAP_FLAG_SYNC is fine in this case. It should only do
> > anything if there is unwritten data, which is the only case we are concerned
> > with at this point. Â In any case, this is a simple solution for coreutils
> > until such a time that a more complex solution is added in the kernel (if
> > ever).
> >
> > Christoph is write, SEEK_HOLE and SEEK_DATA are
> >
> > a much better API for what cp woulld lke to do. Â Unfortunately it hasn't
> >
> > been implemented yet in the VFS...
> >
> > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
> >
> > I don't see how this will change the problem in any meaningful way. There
> > will still need to be code that is traversing the on-disk mapping, and also
> > keeping it coherent with unwritten data in the page cache.
> 
> It seems that we are being messed up by page cache and disk.
> Unwritten flag returned from FIEMAP indicates blocks on disk are not
> written, but it does not say if there is data in page cache.  So
> FIEMAP itself just tells user the map on disk.  However there is an
> exception for delayed allocation,  FIEMAP tells users the data is in
> page cache.

No, FIEMAP does not tell the user there is data in the page cache.
It tells there user there is a delayed allocation extent. For XFS, a
delayed allocation extent can cover a range _greater_ than there is
data in the page cache - we do allocation allignment, speculative
allocation and other tricks to avoid fragmentation via
delayed allocation. When XFSs says there is a delalloc extent, it is
simply showing the in-memory representation of the extent. if you
want to know where the data in the page cache actually is, you need
to sync the file to disk to get those ranges converted to real
extents. This is how xfs_bmap has worked for 15 years....

> Maybe FIEMAP should return all known messages for unwritten extent, if
> unwritten data exists in page cache, FIEMAP should let users know that
> data is in page cache and space on disk has been preallocated, but
> data has not been flushed into disk.  Actually, delayed allocation has
> done like this. Then user-space applications can determine how to do.
> Taking cp as an example, it will copy from page cache rather ignore
> it.

Once again, FIEMAP is for showing the filesystem's current extent
state, not the page cache state. Ext4 may implement FIEMAP by doing
page cache walks, but that is a filesystem specific implementation
detail.

> We need a definite definition for FIEMAP, in other words, it tells
> users map on disk or both disk and page cache.

We already have a definition - and it has nothing to do with the
page cache state.

> If the former one is taken, then FIEMAP should not consider
> delayed allocation.

Not at all. the delayed allocation extent is a first class extent
type in XFS and it is reported directly from the extent list. Your
viewpoint is very ext4-specific and ignores the fact that other
filesystems were doing this sort of mapping long before even ext3
(let alone ext4) was a glint in the designer's eye....

> otherwise, FIEMAP should return all known messages for unwritten case
> like delayed allocation.

See my previous comments about extents being unwritten until data is
physically written to them.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Tue Apr 19 00:14:20 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC2KS-0007jH-5c
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 00:14:20 -0400
Received: from eggs.gnu.org ([140.186.70.92]:57423)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC2KP-0007ip-Rl
	for coreutils@gnu.org; Tue, 19 Apr 2011 00:14:18 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC2KO-0006e5-C5
	for coreutils@gnu.org; Tue, 19 Apr 2011 00:14:17 -0400
Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:38059)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC2KN-0006da-MS
	for coreutils@gnu.org; Tue, 19 Apr 2011 00:14:16 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Ag8EAE0JrU15LHHJgWdsb2JhbAClQBUBARYmJYhvvRkOgwSCXwSbQw
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl6.internode.on.net with ESMTP; 19 Apr 2011 13:44:10 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QC2KG-0006bk-HT; Tue, 19 Apr 2011 14:14:08 +1000
Date: Tue, 19 Apr 2011 14:14:08 +1000
From: Dave Chinner <david@fromorbit.com>
To: Ted Ts'o <tytso@mit.edu>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419041408.GC23985@dastard>
References: <4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419025949.GA3030@thunk.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110419025949.GA3030@thunk.org>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.145
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 04:14:18 -0000

On Mon, Apr 18, 2011 at 10:59:49PM -0400, Ted Ts'o wrote:
> On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
> > On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wrote:
> > > Always passing FIEMAP_FLAG_SYNC is fine in this case. It should
> > > only do anything if there is unwritten data, which is the only
> > > case we are concerned with at this point.  In any case, this is a
> > > simple solution for coreutils until such a time that a more
> > > complex solution is added in the kernel (if ever).
> 
> I would recommend that coreutils check i_blocks and i_size and only
> try using fiemap (with FIEMAP_FLAG_SYNC) if the file appears to be
> sparse.  That's because FIEMAP_FLAG_SYNC will do the effectively
> equivalent of an fsync() system call.  Otherwise, in the case of a
> freshly untar'ed directory hierarchy which is then copied using "cp
> -r", cp would end up calling fsync() for each file in the directory,
> with the disastrous performance result that one might expect.
> 
> If cp only tries the fiemap optimization on files that appear to be
> sparse, it should avoid this problem.
> 
> > > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
> > >
> > > I don't see how this will change the problem in any meaningful way. There
> > > will still need to be code that is traversing the on-disk mapping, and also
> > > keeping it coherent with unwritten data in the page cache.
> 
> The advantage of SEEK_HOLE/SEEK_DATA is that we don't need to force an
> fsync() of the data.
> 
> > It seems that we are being messed up by page cache and disk.
> > Unwritten flag returned from FIEMAP indicates blocks on disk are not
> > written, but it does not say if there is data in page cache.  So
> > FIEMAP itself just tells user the map on disk.  However there is an
> > exception for delayed allocation,  FIEMAP tells users the data is in
> > page cache.
> > 
> > Maybe FIEMAP should return all known messages for unwritten extent, if
> > unwritten data exists in page cache, FIEMAP should let users know that
> > data is in page cache and space on disk has been preallocated, but
> > data has not been flushed into disk.  Actually, delayed allocation has
> > done like this. Then user-space applications can determine how to do.
> > Taking cp as an example, it will copy from page cache rather ignore
> > it.
> >
> > We need a definite definition for FIEMAP, in other words, it tells
> > users map on disk or both disk and page cache.
> > 
> > If the former one is taken, then FIEMAP should not consider delayed
> > allocation.  otherwise, FIEMAP should return all known messages for
> > unwritten case like delayed allocation.
> 
> The fact that the FIEMAP interface deifnition includes an delayed
> allocation bit could be a strong indication that unlike the XFS's bmap
> interface, that this interface is supposed to return information
> taking into account both on-disk and page cache information.

As I said in a previous email, XFS uses delalloc as a first class
extent and reporting them does not require looking at the page
cache. Therefore whatever historical behaviour xfs_bmap used is
irrelevant - supporting delalloc extents was a 2 or 3 line change
and in no way was intended to report anything other than the current
extents. Even at that time, "dirty page cache ranges" != delalloc
extents, and this appears to be the way ext4 has _implemented_
reporting of delalloc extents. 

Indeed, I was the one that suggested it be supported because it is
useful to know the delalloc state _for debugging purposes_.  Now you
are trying to redefine what a delalloc extent is to match the ext4
implementation, and then extend that same reasoning to change what
an unwritten extent means to match how _you think_ the ext4
implementation works(*).

And besides, if I use your same logical progression you've
applied to FIEMAP via the ext4 delalloc extent implementation, using
the XFS delalloc extent implementation in no way implies page cache
coherency for FIEMAP. :)

FIEMAP is for reporting extent state. What that means is filesystem
specific, and requires knowledge of the filesystem to use
effectively. If you want to report coherent state for working out
what ranges to copy, implement SEEK_HOLE/SEEK_DATA (which would use
much of the FIEMAP infrastructure). Redefining the FIEMAP API will
not solve the problem of different filesystems behaving in a manner
that is not useful for coreutils....

> Maybe coreutils will need to use FIEMAP_FLAG_SYNC initially, since
> it's the only way to guarantee correct behaviour for XFS.  But I would
> really rather that be the long-term way we leave things!

(*) It's not XFS specific - ext4 behaves exactly the same way (as
eric kindly pointed out). IOWs, it's likely that all filesystems
need the SYNC flag for one reason or another and that indicates to
me that FIEMAP is simply not the right interface for coreutils to be
using for their intended purpose.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Tue Apr 19 02:53:25 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC4oP-0002Zh-0H
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 02:53:25 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51479)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC4oN-0002Yb-9w
	for coreutils@gnu.org; Tue, 19 Apr 2011 02:53:24 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC4oL-0007RO-Rh
	for coreutils@gnu.org; Tue, 19 Apr 2011 02:53:23 -0400
Received: from mail-pv0-f169.google.com ([74.125.83.169]:38641)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC4oL-0007RH-Ds
	for coreutils@gnu.org; Tue, 19 Apr 2011 02:53:21 -0400
Received: by pvg4 with SMTP id 4so3647649pvg.0
	for <coreutils@gnu.org>; Mon, 18 Apr 2011 23:53:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=VGXUh19pa3ZwgDbtAgVxbFYEuKqJ2cTU5prRbsD4Qz8=;
	b=JLDwnWlPcTuau0uM2ns6MgQbMGNjcYdwU8XKn2dxPFoP/raPjM8v7Khv4+jMMj3BgR
	ztmRXFyukdGAWYDPrBSsciW/uB6kibOqxR+d6j7r3OxwX9/m8141m9F/EdiLYoCNauQR
	D3v11w7btWcKYY1nfMsE7PVNdicKz3s2vEVR8=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=xEbL6G/k/diBvECFYO/B6XANYhLSaVgOOZWAN3SzbmtXIWGm81Sfbtud4Vo0MYfNay
	B6mLCKNgZ5ZdJsUpnWSJPvg+7DyBzwWJlqOlrtL6AnKS7r1hj4SgiGG/xyN3SIZfEJE6
	0QrJo4t677wO2Jrp5TnDCnXPqFkcn2Abu8e8M=
MIME-Version: 1.0
Received: by 10.68.48.163 with SMTP id m3mr8024242pbn.236.1303196000540; Mon,
	18 Apr 2011 23:53:20 -0700 (PDT)
Received: by 10.68.66.164 with HTTP; Mon, 18 Apr 2011 23:53:20 -0700 (PDT)
In-Reply-To: <20110419034455.GB23985@dastard>
References: <20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
Date: Tue, 19 Apr 2011 14:53:20 +0800
Message-ID: <BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Dave Chinner <david@fromorbit.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 74.125.83.169
Cc: Andreas Dilger <adilger@dilger.ca>, Theodore Tso <tytso@mit.edu>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 06:53:24 -0000

On Tue, Apr 19, 2011 at 11:44 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
>> On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wro=
te:
>> > On 2011-04-17, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote:
>> >
>> > On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
>> >
>> > On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
>> >
>> > In that case, it means cp should just always use FIEMAP_FLAG_SYNC, whi=
ch is
>> > fine.
>> >
>> > Except that if someone is copying a large delay allocated file, it wil=
l
>> > cause
>> >
>> > the file to immediately snapped to disk, which might not be the greate=
st
>> >
>> > thing in the world.
>> >
>> > Obvious workaround - if the initial fiemap call shows unwritten
>> > extents, redo it with the sync flag set. Though that assume=D1=95 that
>> > you can trust things like delalloc extents to only cover the range
>> > that valid data exists in. Which, of course, you can't assume,
>> > either. :/
>> >
>> > Always passing=C2=A0FIEMAP_FLAG_SYNC is fine in this case. It should o=
nly do
>> > anything if there is unwritten data, which is the only case we are con=
cerned
>> > with at this point. =C2=A0In any case, this is a simple solution for c=
oreutils
>> > until such a time that a more complex solution is added in the kernel =
(if
>> > ever).
>> >
>> > Christoph is write, SEEK_HOLE and SEEK_DATA are
>> >
>> > a much better API for what cp woulld lke to do. =C2=A0Unfortunately it=
 hasn't
>> >
>> > been implemented yet in the VFS...
>> >
>> > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
>> >
>> > I don't see how this will change the problem in any meaningful way. Th=
ere
>> > will still need to be code that is traversing the on-disk mapping, and=
 also
>> > keeping it coherent with unwritten data in the page cache.
>>
>> It seems that we are being messed up by page cache and disk.
>> Unwritten flag returned from FIEMAP indicates blocks on disk are not
>> written, but it does not say if there is data in page cache. =C2=A0So
>> FIEMAP itself just tells user the map on disk. =C2=A0However there is an
>> exception for delayed allocation, =C2=A0FIEMAP tells users the data is i=
n
>> page cache.
>
> No, FIEMAP does not tell the user there is data in the page cache.
> It tells there user there is a delayed allocation extent. For XFS, a
> delayed allocation extent can cover a range _greater_ than there is
> data in the page cache - we do allocation allignment, speculative
> allocation and other tricks to avoid fragmentation via
> delayed allocation. When XFSs says there is a delalloc extent, it is
> simply showing the in-memory representation of the extent. if you
> want to know where the data in the page cache actually is, you need
> to sync the file to disk to get those ranges converted to real
> extents. This is how xfs_bmap has worked for 15 years....
>
>> Maybe FIEMAP should return all known messages for unwritten extent, if
>> unwritten data exists in page cache, FIEMAP should let users know that
>> data is in page cache and space on disk has been preallocated, but
>> data has not been flushed into disk. =C2=A0Actually, delayed allocation =
has
>> done like this. Then user-space applications can determine how to do.
>> Taking cp as an example, it will copy from page cache rather ignore
>> it.
>
> Once again, FIEMAP is for showing the filesystem's current extent
> state, not the page cache state. Ext4 may implement FIEMAP by doing
> page cache walks, but that is a filesystem specific implementation
> detail.
>
>> We need a definite definition for FIEMAP, in other words, it tells
>> users map on disk or both disk and page cache.
>
> We already have a definition - and it has nothing to do with the
> page cache state.
>
>> If the former one is taken, then FIEMAP should not consider
>> delayed allocation.
>
> Not at all. the delayed allocation extent is a first class extent
> type in XFS and it is reported directly from the extent list. Your
> viewpoint is very ext4-specific and ignores the fact that other
> filesystems were doing this sort of mapping long before even ext3
> (let alone ext4) was a glint in the designer's eye....
>
>> otherwise, FIEMAP should return all known messages for unwritten case
>> like delayed allocation.
>
> See my previous comments about extents being unwritten until data is
> physically written to them.
Understood, thank you for your explanation.

Ok.  Let's look at it from a higher view.  What you described about
extent state is specific to xfs.

I think there are 2 ways to provide a definite definition for FIEMAP
for all filesystems:

1. FIEMAP returns extent state on disk.
2. FIEMAP returns extent both in memory and on disk.

Now, the question comes in case 2.   How to define extent's state in
memory? Every filesystem has its own implementation regarding extent
in memory, especially for delayed and unwritten extents, and I think
they are all reasonable. For example, ext4 without delayed allocation
change unwritten extents to written ones immediately in memory, while
the changing is delayed until flush time in delayed allocation case.

It seems that there are only 1 way to provide a definite definition of
in-memory-extent - FIEMAP should return what an user has written.
this works for all filesystems.

Yongqiang.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Tue Apr 19 03:45:58 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC5dG-0007q8-1U
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 03:45:58 -0400
Received: from eggs.gnu.org ([140.186.70.92]:44817)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC5dD-0007q2-LB
	for coreutils@gnu.org; Tue, 19 Apr 2011 03:45:56 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC5dC-00009O-Ai
	for coreutils@gnu.org; Tue, 19 Apr 2011 03:45:55 -0400
Received: from ipmail06.adl6.internode.on.net ([150.101.137.145]:65140)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QC5dB-00008z-LK
	for coreutils@gnu.org; Tue, 19 Apr 2011 03:45:54 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AhUEAJQ6rU15LHHJgWdsb2JhbACET6ByFQEBFiYliG+raZEVDoEbg056BA
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl6.internode.on.net with ESMTP; 19 Apr 2011 17:15:46 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QC5d2-0006vX-51; Tue, 19 Apr 2011 17:45:44 +1000
Date: Tue, 19 Apr 2011 17:45:38 +1000
From: Dave Chinner <david@fromorbit.com>
To: Yongqiang Yang <xiaoqiangnk@gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419074538.GG23985@dastard>
References: <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.145
Cc: Andreas Dilger <adilger@dilger.ca>, Theodore Tso <tytso@mit.edu>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 07:45:56 -0000

On Tue, Apr 19, 2011 at 02:53:20PM +0800, Yongqiang Yang wrote:
> On Tue, Apr 19, 2011 at 11:44 AM, Dave Chinner <david@fromorbit.com> wrote:
> > On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
> >> On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> wrote:
> >> > On 2011-04-17, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote:
> >> >
> >> > On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
> >> >
> >> > On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
> >> >
> >> > In that case, it means cp should just always use FIEMAP_FLAG_SYNC, which is
> >> > fine.
> >> >
> >> > Except that if someone is copying a large delay allocated file, it will
> >> > cause
> >> >
> >> > the file to immediately snapped to disk, which might not be the greatest
> >> >
> >> > thing in the world.
> >> >
> >> > Obvious workaround - if the initial fiemap call shows unwritten
> >> > extents, redo it with the sync flag set. Though that assumeŃ• that
> >> > you can trust things like delalloc extents to only cover the range
> >> > that valid data exists in. Which, of course, you can't assume,
> >> > either. :/
> >> >
> >> > Always passingÂ FIEMAP_FLAG_SYNC is fine in this case. It should only do
> >> > anything if there is unwritten data, which is the only case we are concerned
> >> > with at this point. Â In any case, this is a simple solution for coreutils
> >> > until such a time that a more complex solution is added in the kernel (if
> >> > ever).
> >> >
> >> > Christoph is write, SEEK_HOLE and SEEK_DATA are
> >> >
> >> > a much better API for what cp woulld lke to do. Â Unfortunately it hasn't
> >> >
> >> > been implemented yet in the VFS...
> >> >
> >> > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
> >> >
> >> > I don't see how this will change the problem in any meaningful way. There
> >> > will still need to be code that is traversing the on-disk mapping, and also
> >> > keeping it coherent with unwritten data in the page cache.
> >>
> >> It seems that we are being messed up by page cache and disk.
> >> Unwritten flag returned from FIEMAP indicates blocks on disk are not
> >> written, but it does not say if there is data in page cache. Â So
> >> FIEMAP itself just tells user the map on disk. Â However there is an
> >> exception for delayed allocation, Â FIEMAP tells users the data is in
> >> page cache.
> >
> > No, FIEMAP does not tell the user there is data in the page cache.
> > It tells there user there is a delayed allocation extent. For XFS, a
> > delayed allocation extent can cover a range _greater_ than there is
> > data in the page cache - we do allocation allignment, speculative
> > allocation and other tricks to avoid fragmentation via
> > delayed allocation. When XFSs says there is a delalloc extent, it is
> > simply showing the in-memory representation of the extent. if you
> > want to know where the data in the page cache actually is, you need
> > to sync the file to disk to get those ranges converted to real
> > extents. This is how xfs_bmap has worked for 15 years....
> >
> >> Maybe FIEMAP should return all known messages for unwritten extent, if
> >> unwritten data exists in page cache, FIEMAP should let users know that
> >> data is in page cache and space on disk has been preallocated, but
> >> data has not been flushed into disk. Â Actually, delayed allocation has
> >> done like this. Then user-space applications can determine how to do.
> >> Taking cp as an example, it will copy from page cache rather ignore
> >> it.
> >
> > Once again, FIEMAP is for showing the filesystem's current extent
> > state, not the page cache state. Ext4 may implement FIEMAP by doing
> > page cache walks, but that is a filesystem specific implementation
> > detail.
> >
> >> We need a definite definition for FIEMAP, in other words, it tells
> >> users map on disk or both disk and page cache.
> >
> > We already have a definition - and it has nothing to do with the
> > page cache state.
> >
> >> If the former one is taken, then FIEMAP should not consider
> >> delayed allocation.
> >
> > Not at all. the delayed allocation extent is a first class extent
> > type in XFS and it is reported directly from the extent list. Your
> > viewpoint is very ext4-specific and ignores the fact that other
> > filesystems were doing this sort of mapping long before even ext3
> > (let alone ext4) was a glint in the designer's eye....
> >
> >> otherwise, FIEMAP should return all known messages for unwritten case
> >> like delayed allocation.
> >
> > See my previous comments about extents being unwritten until data is
> > physically written to them.
> Understood, thank you for your explanation.
> 
> Ok.  Let's look at it from a higher view.  What you described about
> extent state is specific to xfs.
> 
> I think there are 2 ways to provide a definite definition for FIEMAP
> for all filesystems:
> 
> 1. FIEMAP returns extent state on disk.
> 2. FIEMAP returns extent both in memory and on disk.

You are *not listening*. There is no #2. FIEMAP returns the extent
state _on disk_ at the time of the call. If you want it to reflect
the in-memory state at the time of the call (for data or metadata),
you *must* use the the SYNC flag to convert that in-memory state to
on-disk state, which FIEMAP then reports just fine.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Tue Apr 19 04:11:42 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QC62A-0001kY-5c
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 04:11:42 -0400
Received: from eggs.gnu.org ([140.186.70.92]:54387)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC628-0001kO-A9
	for coreutils@gnu.org; Tue, 19 Apr 2011 04:11:41 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC626-0005HS-Kd
	for coreutils@gnu.org; Tue, 19 Apr 2011 04:11:40 -0400
Received: from mail-px0-f179.google.com ([209.85.212.179]:52605)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QC626-0005HD-Cb
	for coreutils@gnu.org; Tue, 19 Apr 2011 04:11:38 -0400
Received: by pxi2 with SMTP id 2so4269105pxi.38
	for <coreutils@gnu.org>; Tue, 19 Apr 2011 01:11:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=2tb7XCIuWQxQ1JLF5QQ0SI5l2fISgszMCdGeecW2FEQ=;
	b=JBjpWqfPFvmt1nczWZFAmMr+vE9tlRMaxf0CH//fIX/x4FZ+mu4gUtmtUwJ8eYS2yS
	KK7sDfGJjJhNs8esLwxNldGQ0vKag+CndQ6KqeYsFY7EsMjqrzlzuqeu8gKzLtBHNkU1
	QSp+8i5WI7dguc4u4xsY6evU0s7rzbTHH7TIY=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=vIBB+KqteRSe6AHOffQdC79SgBK920TuG/nJcMNub4cW5WmCo7MMPzrhvAsrE6/aDq
	u6uLfiLYLqrDq4hhUaFJ3sLD+nWXW1SkFSKZijwQYeZk+p8oYfoU8mGLKAfcgITfEH6s
	zIDOXSNMy7cPx/8Z1VjMLVnw/FA5R9dIz9qTc=
MIME-Version: 1.0
Received: by 10.68.15.130 with SMTP id x2mr8105513pbc.35.1303200697683; Tue,
	19 Apr 2011 01:11:37 -0700 (PDT)
Received: by 10.68.66.164 with HTTP; Tue, 19 Apr 2011 01:11:37 -0700 (PDT)
In-Reply-To: <20110419074538.GG23985@dastard>
References: <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard>
Date: Tue, 19 Apr 2011 16:11:37 +0800
Message-ID: <BANLkTinozFL1Q5BmWr=7=Pua+0g_BciVvA@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: Dave Chinner <david@fromorbit.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 209.85.212.179
Cc: Andreas Dilger <adilger@dilger.ca>, Theodore Tso <tytso@mit.edu>,
	Eric Sandeen <sandeen@sandeen.net>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 08:11:41 -0000

On Tue, Apr 19, 2011 at 3:45 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Tue, Apr 19, 2011 at 02:53:20PM +0800, Yongqiang Yang wrote:
>> On Tue, Apr 19, 2011 at 11:44 AM, Dave Chinner <david@fromorbit.com> wro=
te:
>> > On Tue, Apr 19, 2011 at 09:58:15AM +0800, Yongqiang Yang wrote:
>> >> On Mon, Apr 18, 2011 at 10:45 AM, Andreas Dilger <adilger@dilger.ca> =
wrote:
>> >> > On 2011-04-17, at 6:40 PM, Dave Chinner <david@fromorbit.com> wrote=
:
>> >> >
>> >> > On Sat, Apr 16, 2011 at 08:21:28AM -0400, Theodore Tso wrote:
>> >> >
>> >> > On Apr 16, 2011, at 1:11 AM, Andreas Dilger wrote:
>> >> >
>> >> > In that case, it means cp should just always use FIEMAP_FLAG_SYNC, =
which is
>> >> > fine.
>> >> >
>> >> > Except that if someone is copying a large delay allocated file, it =
will
>> >> > cause
>> >> >
>> >> > the file to immediately snapped to disk, which might not be the gre=
atest
>> >> >
>> >> > thing in the world.
>> >> >
>> >> > Obvious workaround - if the initial fiemap call shows unwritten
>> >> > extents, redo it with the sync flag set. Though that assume=D1=95 t=
hat
>> >> > you can trust things like delalloc extents to only cover the range
>> >> > that valid data exists in. Which, of course, you can't assume,
>> >> > either. :/
>> >> >
>> >> > Always passing=C2=A0FIEMAP_FLAG_SYNC is fine in this case. It shoul=
d only do
>> >> > anything if there is unwritten data, which is the only case we are =
concerned
>> >> > with at this point. =C2=A0In any case, this is a simple solution fo=
r coreutils
>> >> > until such a time that a more complex solution is added in the kern=
el (if
>> >> > ever).
>> >> >
>> >> > Christoph is write, SEEK_HOLE and SEEK_DATA are
>> >> >
>> >> > a much better API for what cp woulld lke to do. =C2=A0Unfortunately=
 it hasn't
>> >> >
>> >> > been implemented yet in the VFS...
>> >> >
>> >> > Agreed, SEEK_HOLE/SEEK_DATA is the right way to solve this problem.
>> >> >
>> >> > I don't see how this will change the problem in any meaningful way.=
 There
>> >> > will still need to be code that is traversing the on-disk mapping, =
and also
>> >> > keeping it coherent with unwritten data in the page cache.
>> >>
>> >> It seems that we are being messed up by page cache and disk.
>> >> Unwritten flag returned from FIEMAP indicates blocks on disk are not
>> >> written, but it does not say if there is data in page cache. =C2=A0So
>> >> FIEMAP itself just tells user the map on disk. =C2=A0However there is=
 an
>> >> exception for delayed allocation, =C2=A0FIEMAP tells users the data i=
s in
>> >> page cache.
>> >
>> > No, FIEMAP does not tell the user there is data in the page cache.
>> > It tells there user there is a delayed allocation extent. For XFS, a
>> > delayed allocation extent can cover a range _greater_ than there is
>> > data in the page cache - we do allocation allignment, speculative
>> > allocation and other tricks to avoid fragmentation via
>> > delayed allocation. When XFSs says there is a delalloc extent, it is
>> > simply showing the in-memory representation of the extent. if you
>> > want to know where the data in the page cache actually is, you need
>> > to sync the file to disk to get those ranges converted to real
>> > extents. This is how xfs_bmap has worked for 15 years....
>> >
>> >> Maybe FIEMAP should return all known messages for unwritten extent, i=
f
>> >> unwritten data exists in page cache, FIEMAP should let users know tha=
t
>> >> data is in page cache and space on disk has been preallocated, but
>> >> data has not been flushed into disk. =C2=A0Actually, delayed allocati=
on has
>> >> done like this. Then user-space applications can determine how to do.
>> >> Taking cp as an example, it will copy from page cache rather ignore
>> >> it.
>> >
>> > Once again, FIEMAP is for showing the filesystem's current extent
>> > state, not the page cache state. Ext4 may implement FIEMAP by doing
>> > page cache walks, but that is a filesystem specific implementation
>> > detail.
>> >
>> >> We need a definite definition for FIEMAP, in other words, it tells
>> >> users map on disk or both disk and page cache.
>> >
>> > We already have a definition - and it has nothing to do with the
>> > page cache state.
>> >
>> >> If the former one is taken, then FIEMAP should not consider
>> >> delayed allocation.
>> >
>> > Not at all. the delayed allocation extent is a first class extent
>> > type in XFS and it is reported directly from the extent list. Your
>> > viewpoint is very ext4-specific and ignores the fact that other
>> > filesystems were doing this sort of mapping long before even ext3
>> > (let alone ext4) was a glint in the designer's eye....
>> >
>> >> otherwise, FIEMAP should return all known messages for unwritten case
>> >> like delayed allocation.
>> >
>> > See my previous comments about extents being unwritten until data is
>> > physically written to them.
>> Understood, thank you for your explanation.
>>
>> Ok. =C2=A0Let's look at it from a higher view. =C2=A0What you described =
about
>> extent state is specific to xfs.
>>
>> I think there are 2 ways to provide a definite definition for FIEMAP
>> for all filesystems:
>>
>> 1. FIEMAP returns extent state on disk.
>> 2. FIEMAP returns extent both in memory and on disk.
>
> You are *not listening*. There is no #2. FIEMAP returns the extent
> state _on disk_ at the time of the call. If you want it to reflect
> the in-memory state at the time of the call (for data or metadata),
> you *must* use the the SYNC flag to convert that in-memory state to
> on-disk state, which FIEMAP then reports just fine.

Sorry for being dense.

I think delayed extent is an exception. because it is not on the disk.

Yongqiang.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Tue Apr 19 09:55:19 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCBOh-0000QB-49
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 09:55:19 -0400
Received: from eggs.gnu.org ([140.186.70.92]:42518)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+cd2d0cc2584a24e57a4c+2795+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QC3TZ-0003W3-1N
	for coreutils@gnu.org; Tue, 19 Apr 2011 01:27:49 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from
	<BATV+cd2d0cc2584a24e57a4c+2795+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QC3TY-0002Hy-2A
	for coreutils@gnu.org; Tue, 19 Apr 2011 01:27:49 -0400
Received: from bombadil.infradead.org ([18.85.46.34]:37445)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+cd2d0cc2584a24e57a4c+2795+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QC3TX-0002Hq-Vw
	for coreutils@gnu.org; Tue, 19 Apr 2011 01:27:48 -0400
Received: from hch by bombadil.infradead.org with local (Exim 4.72 #1 (Red Hat
	Linux)) id 1QC3TI-0007aa-RX; Tue, 19 Apr 2011 05:27:32 +0000
Date: Tue, 19 Apr 2011 01:27:32 -0400
From: Christoph Hellwig <hch@infradead.org>
To: Ted Ts'o <tytso@mit.edu>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419052732.GA19952@infradead.org>
References: <4DA7836A.5040604@draigBrady.com> <20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419025949.GA3030@thunk.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=unknown-8bit
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20110419025949.GA3030@thunk.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SRS-Rewrite: SMTP reverse-path rewritten from <hch@infradead.org> by
	bombadil.infradead.org See http://www.infradead.org/rpr.html
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 18.85.46.34
X-Mailman-Approved-At: Tue, 19 Apr 2011 09:55:18 -0400
Cc: Andreas Dilger <adilger@dilger.ca>, Dave Chinner <david@fromorbit.com>,
	Eric Sandeen <sandeen@sandeen.net>, Yongqiang Yang <xiaoqiangnk@gmail.com>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 05:27:50 -0000

On Mon, Apr 18, 2011 at 10:59:49PM -0400, Ted Ts'o wrote:
> Maybe coreutils will need to use FIEMAP_FLAG_SYNC initially, since
> it's the only way to guarantee correct behaviour for XFS.  But I would
> really rather that be the long-term way we leave things!

As Eric pointed out both ext4 and XFS have the same behaviour when
writing into unwritten extent.  I think you are a bit confused because
ext4 also got basic handling of delalloc extents wrong before commit
6d9c85eb700bd3ac59e63bb9de463dea1aca084c, which never was a problem with
XFS.  It would be nice if ext4 developers had sent the included
regression test for xfs so that everyone could verify this behaviour,
btw.

To report written to but not synced unwritten extents properly we'd
need to move fiemap away from the onđdisk state reporting done so far
and do something that is purely in-memory.  It would be doable by
walking the pagecache and checking for the buffer unwritten flag
in a loop over the pages, but I'm honestly not sure it's going to
help much.  In fact given that unwritten extent were specifically
allocated before it doesn't seem like an overly smart idea to skip
them in a copy - yes it will save space but it also undoes the
previous explicit preallocation.  If people want it they should rather
add a new option to cp to turn zeroes into holes.



From MAILER-DAEMON Tue Apr 19 10:05:16 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCBYK-0003pn-L4
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 10:05:16 -0400
Received: from eggs.gnu.org ([140.186.70.92]:52263)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QCBYH-0003pZ-VM
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:05:15 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QCBYG-0002I4-AI
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:05:13 -0400
Received: from sandeen.net ([63.231.237.45]:58720 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QCBYF-0002Hf-W1
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:05:12 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 9391548FDA82;
	Tue, 19 Apr 2011 09:05:10 -0500 (CDT)
Message-ID: <4DAD96A4.40008@sandeen.net>
Date: Tue, 19 Apr 2011 09:05:24 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Yongqiang Yang <xiaoqiangnk@gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <20110415000940.GL21395@dastard>	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>	<20110416005040.GP21395@dastard>	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>	<20110418004040.GS21395@dastard>	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>	<20110419034455.GB23985@dastard>	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>	<20110419074538.GG23985@dastard>
	<BANLkTinozFL1Q5BmWr=7=Pua+0g_BciVvA@mail.gmail.com>
In-Reply-To: <BANLkTinozFL1Q5BmWr=7=Pua+0g_BciVvA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: Andreas Dilger <adilger@dilger.ca>, Theodore Tso <tytso@mit.edu>,
	Dave Chinner <david@fromorbit.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 14:05:15 -0000

On 4/19/11 3:11 AM, Yongqiang Yang wrote:
> On Tue, Apr 19, 2011 at 3:45 PM, Dave Chinner <david@fromorbit.com> wrote:
>> On Tue, Apr 19, 2011 at 02:53:20PM +0800, Yongqiang Yang wrote:


...

>>> I think there are 2 ways to provide a definite definition for FIEMAP
>>> for all filesystems:
>>>
>>> 1. FIEMAP returns extent state on disk.
>>> 2. FIEMAP returns extent both in memory and on disk.
>>
>> You are *not listening*. There is no #2. FIEMAP returns the extent
>> state _on disk_ at the time of the call. If you want it to reflect
>> the in-memory state at the time of the call (for data or metadata),
>> you *must* use the the SYNC flag to convert that in-memory state to
>> on-disk state, which FIEMAP then reports just fine.
> 
> Sorry for being dense.
> 
> I think delayed extent is an exception. because it is not on the disk.
> 
> Yongqiang.

I don't think you're being dense, I think that the interface specification is just messed up.

By including flags for both unwritten and delalloc in the interface, we have hopelessly intertwined on-disk and in-memory state.

If you preallocate 1M and then do a buffered IO to that same range without a sync, and then a fiemap, what on earth should the interface return?

Writing the first part of that testcase is simple but I have no idea what the correct behavior is.

(FIEMAP_EXTENT_UNWRITTEN|FIEMAP_EXTENT_DELALLOC) ? that makes no sense.  But no other combination of flags makes sense to me either, unless we get into tortured redefinitions of what "delalloc" means.

And if we can't say what it -should- return.... well, too bad for coreutils.  :(

-Eric


From MAILER-DAEMON Tue Apr 19 10:09:30 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCBcQ-0004ff-Gn
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 10:09:30 -0400
Received: from eggs.gnu.org ([140.186.70.92]:53077)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCBcK-0004es-MQ
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:09:28 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCBcF-000393-6D
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:09:24 -0400
Received: from li9-11.members.linode.com ([67.18.176.11]:46360
	helo=test.thunk.org) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCBcF-00038p-0C
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:09:19 -0400
Received: from root (helo=tytso-glaptop)
	by test.thunk.org with local-esmtp (Exim 4.69)
	(envelope-from <tytso@thunk.org>)
	id 1QCBc7-0000qA-U7; Tue, 19 Apr 2011 14:09:11 +0000
Received: from tytso by tytso-glaptop with local (Exim 4.71)
	(envelope-from <tytso@thunk.org>)
	id 1QCBc6-0006vT-6z; Tue, 19 Apr 2011 10:09:10 -0400
Date: Tue, 19 Apr 2011 10:09:09 -0400
From: Ted Ts'o <tytso@mit.edu>
To: Dave Chinner <david@fromorbit.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419140909.GD3030@thunk.org>
References: <76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110419074538.GG23985@dastard>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on test.thunk.org); SAEximRunCond expanded to false
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 1)
X-Received-From: 67.18.176.11
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 14:09:28 -0000

On Tue, Apr 19, 2011 at 05:45:38PM +1000, Dave Chinner wrote:
> You are *not listening*. There is no #2. FIEMAP returns the extent
> state _on disk_ at the time of the call.

Dave, you're being rather strident about your insistence about what
FIEMAP's semantics are.  Part of the problem here is that it's *not*
clear or settled.

If it really is the state _on_ _disk_, does XFS really have a DELALLOC
bit _on_ _disk_?

					- Ted


From MAILER-DAEMON Tue Apr 19 10:13:16 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCBg4-0005KX-0N
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 10:13:16 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60832)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QCBfx-000578-Im
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:13:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QCBfw-0003sZ-EU
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:13:09 -0400
Received: from sandeen.net ([63.231.237.45]:57751 helo=mail.sandeen.net)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <sandeen@sandeen.net>) id 1QCBfw-0003sO-7p
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:13:08 -0400
Received: from liberator.sandeen.net (liberator.sandeen.net [10.0.0.4])
	(using TLSv1 with cipher AES256-SHA (256/256 bits))
	(No client certificate requested)
	by mail.sandeen.net (Postfix) with ESMTP id 27C0848FDA82;
	Tue, 19 Apr 2011 09:13:05 -0500 (CDT)
Message-ID: <4DAD987F.5000506@sandeen.net>
Date: Tue, 19 Apr 2011 09:13:19 -0500
From: Eric Sandeen <sandeen@sandeen.net>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: Ted Ts'o <tytso@mit.edu>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
References: <76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
In-Reply-To: <20110419140909.GD3030@thunk.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 63.231.237.45
Cc: Andreas Dilger <adilger@dilger.ca>, Dave Chinner <david@fromorbit.com>,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 14:13:14 -0000

On 4/19/11 9:09 AM, Ted Ts'o wrote:
> On Tue, Apr 19, 2011 at 05:45:38PM +1000, Dave Chinner wrote:
>> You are *not listening*. There is no #2. FIEMAP returns the extent
>> state _on disk_ at the time of the call.
> 
> Dave, you're being rather strident about your insistence about what
> FIEMAP's semantics are.  Part of the problem here is that it's *not*
> clear or settled.
> 
> If it really is the state _on_ _disk_, does XFS really have a DELALLOC
> bit _on_ _disk_?
> 
> 					- Ted
> 

no of course it doesn't....

But I too am confused about Dave's assertion that it only reflects ondisk state when we have that pesky delalloc flag.

Whose idea was that, anyway? ;)

I'd certainly buy the argument that it -should- only reflect ondisk state, and we should nuke the delalloc flag from orbit, if we could, though.

-Eric


From MAILER-DAEMON Tue Apr 19 10:23:09 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCBpd-0006DX-Dw
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 10:23:09 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41660)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <crockabiscuit@yahoo.com>) id 1QCBpa-0006DR-TP
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:23:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <crockabiscuit@yahoo.com>) id 1QCBpZ-00061E-TL
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:23:06 -0400
Received: from nm25-vm4.bullet.mail.ne1.yahoo.com ([98.138.91.185]:29243)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <crockabiscuit@yahoo.com>) id 1QCBpZ-00060d-Ow
	for coreutils@gnu.org; Tue, 19 Apr 2011 10:23:05 -0400
Received: from [98.138.90.52] by nm25.bullet.mail.ne1.yahoo.com with NNFMP;
	19 Apr 2011 14:23:03 -0000
Received: from [98.138.89.193] by tm5.bullet.mail.ne1.yahoo.com with NNFMP;
	19 Apr 2011 14:23:03 -0000
Received: from [127.0.0.1] by omp1051.mail.ne1.yahoo.com with NNFMP;
	19 Apr 2011 14:23:03 -0000
X-Yahoo-Newman-Property: ymail-3
X-Yahoo-Newman-Id: 694430.59803.bm@omp1051.mail.ne1.yahoo.com
Received: (qmail 7803 invoked by uid 60001); 19 Apr 2011 14:23:03 -0000
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024;
	t=1303222983; bh=qhMPuZ7qpsqpKCzTmof99iGDBPApvLrIY9s1pLhadXw=;
	h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
	b=DW5pTyy5UkYd7QxjpIsLlhrjLg79Q1W6NYbzd7uqoyBGlGycujcfe4l1ny8SfPS2lZEnG+w26X5pMxLR5wrClOZPGipja+4XW9tXA5wZW30bC1AUvCZk0cdaSN78p5Y2N3AEInZn6TlJYzkWDXZmyZohFU/94cpSaP/WrwdMQNM=
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com;
	h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
	b=AddsABWnGLDvz0co4Ujr7t9XI/EgRHJnIvfRZ4KU7UXYcMptkx73RXu+sk0WSqVW2vYiVyeA8+/ELv4mYGapwJH0xBaXUWPi/WXtch15Rgwr4iZBo6fmOOt0s7srrnjFODXdUFlmiyyicQTr3DHnAhg9+gljEYTPP3fDjm1/NxI=;
Message-ID: <483711.6463.qm@web120319.mail.ne1.yahoo.com>
X-YMail-OSG: 5qq60KkVM1kN6XvGjTsaONRw9wJjgM1RnHb7TBJ0NrLSNUg
	v.5_ejiBI.mzP22TQ9ANecIf4D8epO3_foByZXaQc6by7ewCeATsx_KOovGk
	mzNdD8sScWSy43ey6pYczJPC7Xqhj.xVjutq5lPPDk0RlmtFEPl08AjUsTU8
	ollwnqbd9_9Yhn4DB1RFxpt6P1cza1ZlZkRxBvLkSVDWDp4ZX3OwgZ7tmbjh
	UR2F8LCYDjXRbvQyGZFsOELGtiJqlruuPqv60DQWuanDQi8EudXCgdX1WL4W
	xU9oH41RZu9K0o3gwzzvhipgrTcdEHJqw6rPx9UP1ND8yn.MEwR4-
Received: from [183.107.21.20] by web120319.mail.ne1.yahoo.com via HTTP;
	Tue, 19 Apr 2011 07:23:03 PDT
X-Mailer: YahooMailRC/559 YahooMailWebService/0.8.109.295617
Date: Tue, 19 Apr 2011 07:23:03 -0700 (PDT)
From: crocket <crockabiscuit@yahoo.com>
Subject: cp/mv in coreutils don't respect the default ACL of parent
	directories.
To: coreutils@gnu.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 98.138.91.185
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 14:23:08 -0000

I copied a file into a directory whose default ACL entries were d:u::rwx, 
d:g::rwx, d:o::rx.
The original file's permission was 775.
The copied file's permission should have been 664 if default ACL was respected.
But the copied file's permission was 775, which is the same as the original 
file.

The same happens with mv.

It seems like a bug.
I want every copied/moved file to respect default ACL by default.



From MAILER-DAEMON Tue Apr 19 12:01:28 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCDMm-0007ql-RM
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 12:01:28 -0400
Received: from eggs.gnu.org ([140.186.70.92]:44271)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCDMk-0007mX-Ia
	for coreutils@gnu.org; Tue, 19 Apr 2011 12:01:27 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCDMj-0000ql-GG
	for coreutils@gnu.org; Tue, 19 Apr 2011 12:01:26 -0400
Received: from li9-11.members.linode.com ([67.18.176.11]:45489
	helo=test.thunk.org) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCDMj-0000qX-8O
	for coreutils@gnu.org; Tue, 19 Apr 2011 12:01:25 -0400
Received: from root (helo=tytso-glaptop)
	by test.thunk.org with local-esmtp (Exim 4.69)
	(envelope-from <tytso@thunk.org>)
	id 1QCDMa-0000u1-LV; Tue, 19 Apr 2011 16:01:16 +0000
Received: from tytso by tytso-glaptop with local (Exim 4.71)
	(envelope-from <tytso@thunk.org>)
	id 1QCDMZ-0007UE-28; Tue, 19 Apr 2011 12:01:15 -0400
Date: Tue, 19 Apr 2011 12:01:14 -0400
From: Ted Ts'o <tytso@mit.edu>
To: Eric Sandeen <sandeen@sandeen.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419160114.GE3030@thunk.org>
References: <E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
	<4DAD987F.5000506@sandeen.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4DAD987F.5000506@sandeen.net>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on test.thunk.org); SAEximRunCond expanded to false
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 1)
X-Received-From: 67.18.176.11
Cc: Andreas Dilger <adilger@dilger.ca>, Dave Chinner <david@fromorbit.com>,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 16:01:27 -0000

On Tue, Apr 19, 2011 at 09:13:19AM -0500, Eric Sandeen wrote:
> 
> But I too am confused about Dave's assertion that it only reflects ondisk state when we have that pesky delalloc flag.
> 
> Whose idea was that, anyway? ;)
> 
> I'd certainly buy the argument that it -should- only reflect ondisk state, and we should nuke the delalloc flag from orbit, if we could, though.

I see three options of how we can clarify FIEMAP's semantics:

1) We define it as only reflecting ondisk state, and nuke the delalloc
flag from orbit.

2) We state that if the file is currently has unflushed pages in the
page cache, and FIEMAP_FLAG_SYNC is not passed, whether or not extents
return the DELALLOC flag or how they handle the UNWRITTEN flag is
undefined.

3) We state that FIEMAP is supposed to return information which
reflects the union of the on-disk and page cache state, with all that
this implies.

All of these are internally consistent definitions --- we need to
chose one, document, and then tell the shellutils folks what they
should do.

In the case of #1 and #2, we really need to implement support for
SEEK_HOLE/SEEK_DATA for userspace programs like cp who want to know
this information.

Do we all agree on the problem statement, at least?  If so, then we
can try to come consensus on what is the appropriate solution. 

    	   		     	     - Ted


From MAILER-DAEMON Tue Apr 19 17:08:35 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCI9z-0001zG-TK
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 17:08:35 -0400
Received: from eggs.gnu.org ([140.186.70.92]:49732)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QCI9x-0001xJ-W0
	for coreutils@gnu.org; Tue, 19 Apr 2011 17:08:34 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QCI9w-000870-Jx
	for coreutils@gnu.org; Tue, 19 Apr 2011 17:08:33 -0400
Received: from ipmail06.adl2.internode.on.net ([150.101.137.129]:3808)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <david@fromorbit.com>) id 1QCI9w-00086r-4v
	for coreutils@gnu.org; Tue, 19 Apr 2011 17:08:32 -0400
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AkMEAJb4rU15LHHJgWdsb2JhbAClMxUBARYmJYhvvEQOhWMEm1I
Received: from ppp121-44-113-201.lns20.syd6.internode.on.net (HELO dastard)
	([121.44.113.201])
	by ipmail06.adl2.internode.on.net with ESMTP; 20 Apr 2011 06:38:27 +0930
Received: from dave by dastard with local (Exim 4.72)
	(envelope-from <david@fromorbit.com>)
	id 1QCI9p-0008MJ-N8; Wed, 20 Apr 2011 07:08:25 +1000
Date: Wed, 20 Apr 2011 07:08:25 +1000
From: Dave Chinner <david@fromorbit.com>
To: Ted Ts'o <tytso@mit.edu>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110419210825.GJ23985@dastard>
References: <20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110419140909.GD3030@thunk.org>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 150.101.137.129
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	Yongqiang Yang <xiaoqiangnk@gmail.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 19 Apr 2011 21:08:34 -0000

On Tue, Apr 19, 2011 at 10:09:09AM -0400, Ted Ts'o wrote:
> On Tue, Apr 19, 2011 at 05:45:38PM +1000, Dave Chinner wrote:
> > You are *not listening*. There is no #2. FIEMAP returns the extent
> > state _on disk_ at the time of the call.
> 
> Dave, you're being rather strident about your insistence about what
> FIEMAP's semantics are. 

The bit about the page cache state being relevant? That's what I was
refering to here.

> Part of the problem here is that it's *not*
> clear or settled.
> 
> If it really is the state _on_ _disk_, does XFS really have a DELALLOC
> bit _on_ _disk_?

Sigh. No.

This whole thing blew up because of unwritten extent behaviour when
there is dirty page cache covering and unwritten extent. Delalloc
was not the issue - what I said is absolutely true for unwritten
extents.  Somewhere in the middle someone started talking about
delalloc extents and conflating their behaviour with unwritten
extents, but I continued to talk about unwritten extents and
cached pages.

Even so, for delalloc extents the dirty page state in the page cache
is irrelevant. I've said earlier that XFS delalloc extents can span
regions that have no page cache state - they don't get reported as
holes by FIEMAP because they are tracked as delalloc. IOWs, like
unwritten extents, you can't rely on delalloc extents to tell you
where the data is in the file.

So, it logically follws that you need to use the SYNC flag for both
unwritten extents and delalloc extents to find out where there data
realy lies by converting them to real, written extents. i.e. the
only extents you can trust contain data from FIEMAP are the real
extents on disk....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


From MAILER-DAEMON Tue Apr 19 21:53:09 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCMbN-0003oA-96
	for mharc-coreutils@gnu.org; Tue, 19 Apr 2011 21:53:09 -0400
Received: from eggs.gnu.org ([140.186.70.92]:59510)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QCMbK-0003mY-T9
	for coreutils@gnu.org; Tue, 19 Apr 2011 21:53:07 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QCMbJ-0007Xb-Rs
	for coreutils@gnu.org; Tue, 19 Apr 2011 21:53:06 -0400
Received: from mail-pv0-f169.google.com ([74.125.83.169]:35627)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <xiaoqiangnk@gmail.com>) id 1QCMbJ-0007XR-K6
	for coreutils@gnu.org; Tue, 19 Apr 2011 21:53:05 -0400
Received: by pvg4 with SMTP id 4so216487pvg.0
	for <coreutils@gnu.org>; Tue, 19 Apr 2011 18:53:04 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma;
	h=domainkey-signature:mime-version:in-reply-to:references:date
	:message-id:subject:from:to:cc:content-type
	:content-transfer-encoding;
	bh=Z+kLWRVsPhMioW6WrNwWjMKJd0oAM8duXIx5tP1Haws=;
	b=EkIYF6UwpY7mL3s3AgZs38psTH8zqSlI53xOGyixO2r50JzMy2bERRn8fQii13lwU6
	BbHzp8/etIBrX4bu2DnKJp88sZ8JMYuY51ao2gZoE2cQSpU1A+aVrkFbOKLBloWS5Nt6
	cG/nCtTO/YILYtTg3i0DJqSHyV4UTss5GSfHY=
DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=AhGqM1UJOJ39cBvNgQ6yu0nLsR5HWFD7EK4YmJuiyKPcwXg3PlCtK/NWK61A5tGcfQ
	ZSzFx8JCg07q2ir0nJ3DeUWUbhVTq8ywxIhk1K9o9hbYr4xGXj3FiuceAeUfLDMcY7yE
	mFY9Sf3JouSK3CwNlzvyotARgJ//j75smZlJQ=
MIME-Version: 1.0
Received: by 10.68.15.130 with SMTP id x2mr9416494pbc.35.1303264384742; Tue,
	19 Apr 2011 18:53:04 -0700 (PDT)
Received: by 10.68.66.164 with HTTP; Tue, 19 Apr 2011 18:53:04 -0700 (PDT)
In-Reply-To: <20110419160114.GE3030@thunk.org>
References: <E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
	<4DAD987F.5000506@sandeen.net> <20110419160114.GE3030@thunk.org>
Date: Wed, 20 Apr 2011 09:53:04 +0800
Message-ID: <BANLkTinHsYgLbGBTnjRD1cJ0HAJXdebW-w@mail.gmail.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
From: Yongqiang Yang <xiaoqiangnk@gmail.com>
To: "Ted Ts'o" <tytso@mit.edu>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2)
X-Received-From: 74.125.83.169
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	Dave Chinner <david@fromorbit.com>, xfs-oss <xfs@oss.sgi.com>,
	"coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Apr 2011 01:53:07 -0000

On Wed, Apr 20, 2011 at 12:01 AM, Ted Ts'o <tytso@mit.edu> wrote:
> On Tue, Apr 19, 2011 at 09:13:19AM -0500, Eric Sandeen wrote:
>>
>> But I too am confused about Dave's assertion that it only reflects ondis=
k state when we have that pesky delalloc flag.
>>
>> Whose idea was that, anyway? ;)
>>
>> I'd certainly buy the argument that it -should- only reflect ondisk stat=
e, and we should nuke the delalloc flag from orbit, if we could, though.
>
> I see three options of how we can clarify FIEMAP's semantics:
>
> 1) We define it as only reflecting ondisk state, and nuke the delalloc
> flag from orbit.
>
> 2) We state that if the file is currently has unflushed pages in the
> page cache, and FIEMAP_FLAG_SYNC is not passed, whether or not extents
> return the DELALLOC flag or how they handle the UNWRITTEN flag is
> undefined.
>
> 3) We state that FIEMAP is supposed to return information which
> reflects the union of the on-disk and page cache state, with all that
> this implies.
>
> All of these are internally consistent definitions --- we need to
> chose one, document, and then tell the shellutils folks what they
> should do.
>
> In the case of #1 and #2, we really need to implement support for
> SEEK_HOLE/SEEK_DATA for userspace programs like cp who want to know
> this information.
>
> Do we all agree on the problem statement, at least? =A0If so, then we
> can try to come consensus on what is the appropriate solution.

I agree on the problem statement.  Users need to know what FIEMAP
returns definitely.

It seems that Dave is looking at the problem from a different view.

Dave thinks that FIEMAP returns where data exists on disk finally.
Then there are 2 possibilities: unknown and known.  delayed extent is
unknown and others are known.  Although we know where data in
unwritten extent exist finally on disk, we cannot know whether or not
it is being in memory.  However we know data in delayed extent is in
memory. it sounds strange.

>
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 -=
 Ted
>



--=20
Best Wishes
Yongqiang Yang


From MAILER-DAEMON Wed Apr 20 10:39:39 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCYZ9-0006rH-89
	for mharc-coreutils@gnu.org; Wed, 20 Apr 2011 10:39:39 -0400
Received: from eggs.gnu.org ([140.186.70.92]:34659)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QCYZ7-0006rA-3j
	for coreutils@gnu.org; Wed, 20 Apr 2011 10:39:38 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QCYZ5-0001VU-Am
	for coreutils@gnu.org; Wed, 20 Apr 2011 10:39:37 -0400
Received: from mx.meyering.net ([82.230.74.64]:60896)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QCYZ4-0001V3-Qy
	for coreutils@gnu.org; Wed, 20 Apr 2011 10:39:35 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 2BD9660230; Wed, 20 Apr 2011 16:39:30 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
In-Reply-To: <20110414102608.GA1678@x4.trippels.de> (Markus Trippelsdorf's
	message of "Thu, 14 Apr 2011 12:26:08 +0200")
References: <20110414102608.GA1678@x4.trippels.de>
Date: Wed, 20 Apr 2011 16:39:30 +0200
Message-ID: <87d3khugv1.fsf@rho.meyering.net>
Lines: 303
MIME-Version: 1.0
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org, xfs@oss.sgi.com
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Apr 2011 14:39:38 -0000

Markus Trippelsdorf wrote:
> I trashed my system this morning when I installed coreutils-8.11.
>
> What happened is that coreutils compiles and links correctly, but then
> the following command (during the installation phase):
>
> ./ginstall chroot hostid nice who users pinky stty df stdbuf [ base64
...
>
> apparently produces files which have the length of the originals but are
> full of zeros. (and these were then installed to my live system, thereby
> trashing it).
...

Thanks again for the report.
I believe that the following series addresses this problem
and have confirmed that tests pass with 2.6.39-rc3 on all
of ext3, ext4, btrfs and xfs -- though there was what appears
to be a spurious failure in tests/cp/sparse-fiemap when run on xfs.
On one iteration of this loop, with j=3D31, in these loops

  for i in $(seq 1 2 21); do
    for j in 1 2 31 100; do

[in http://git.savannah.gnu.org/cgit/coreutils.git/tree/tests/cp/sparse-fie=
map]
the two files compared equal, yet their extents did not match,
even after merging.  I'm inclined to skip the extent-comparing check
at least for XFS, now.

Here's the unusually-technical-for-NEWS summary:

** Changes in behavior

  cp's extent-based (FIEMAP) copying code is more reliable in the face
  of varying and undocumented file system semantics:
  - it no longer treats unwritten extents specially
  - a FIEMAP-based extent copy always uses the FIEMAP_FLAG_SYNC flag.
      Before, it would incur the performance penalty of that sync only
      for 2.6.38 and older kernels.  We thought all problems would be
      resolved for 2.6.39.
  - it now attempts a FIEMAP copy only on a file that appears sparse.
      Sparse files are relatively unusual, and the copying code incurs
      the performance penalty of the now-mandatory sync only for them.




From=202783b52b273dd8fca824d8e1a64f8c4f41a54c00 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 09:49:15 +0200
Subject: [PATCH 1/4] copy: always use FIEMAP_FLAG_SYNC, for now

* src/extent-scan.c (extent_need_sync): Always return true,
to make the sole caller always use FIEMAP_FLAG_SYNC.
This will doubtless have an undesirable performance impact,
but we'll mitigate that shortly, by using extent_copy only on
files with holes.
=2D--
 src/extent-scan.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/src/extent-scan.c b/src/extent-scan.c
index da7eb9d..596e7f7 100644
=2D-- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -36,6 +36,13 @@
 static bool
 extent_need_sync (void)
 {
+  /* For now always return true, to be on the safe side.
+     If/when FIEMAP semantics are well defined (before SEEK_HOLE support
+     is usable) and kernels implementing them are in use, we may relax
+     this once again.  */
+  return true;
+
+#if FIEMAP_BEHAVIOR_IS_DEFINED_AND_USABLE
   static int need_sync =3D -1;

   if (need_sync =3D=3D -1)
@@ -57,6 +64,7 @@ extent_need_sync (void)
     }

   return need_sync;
+#endif
 }

 /* Allocate space for struct extent_scan, initialize the entries if
=2D-
1.7.5.rc2.295.g19c42


From=20f35019b45e2b1ff6e1940db7b452dcb8f674f190 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 10:15:15 +0200
Subject: [PATCH 2/4] copy: do not treat unwritten extents specially (avoid
 XFS corruption)

* src/copy.c (extent_copy): Do not treat "unwritten extents" specially.
Otherwise, with XFS and a release-candidate 2.6.39-rc3 kernel, and
when using gold as your linker[*], and if you don't run "make check",
you could end up installing files full of zeros instead of the
expected binaries.  For a lot of discussion, see
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/37895

[*] Gold preallocates space for the files it writes, which is good.
However, on XFS, that conspired with the other conditions to result
the malfunctioning of the just-built install binary.
=2D--
 src/copy.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index 9b53127..f6f9ea6 100644
=2D-- a/src/copy.c
+++ b/src/copy.c
@@ -398,7 +398,10 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_=
t buf_size,
           /* Treat an unwritten but allocated extent much like a hole.
              I.E. don't read, but don't convert to a hole in the destinati=
on,
              unless SPARSE_ALWAYS.  */
=2D          if (scan.ext_info[i].ext_flags & FIEMAP_EXTENT_UNWRITTEN)
+          /* For now, do not treat FIEMAP_EXTENT_UNWRITTEN specially,
+             because that (in combination with no sync) would lead to data
+             loss at least on XFS and ext4 when using 2.6.39-rc3 kernels. =
 */
+          if (0 && (scan.ext_info[i].ext_flags & FIEMAP_EXTENT_UNWRITTEN))
             {
               empty_extent =3D true;
               last_ext_len =3D 0;
=2D-
1.7.5.rc2.295.g19c42


From=20489261905dfee95cc1ebb708e8302bb246519b8b Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 10:23:32 +0200
Subject: [PATCH 3/4] copy: factor out a tiny sparse-testing function

* src/copy.c (HAVE_STRUCT_STAT_ST_BLOCKS): Define to 0 if undefined,
so we can use it in the return expression, here:
(is_probably_sparse): New function, factored out of...
(copy_reg): ...here.  Use the new function.
=2D--
 src/copy.c |   23 +++++++++++++++++++----
 1 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index f6f9ea6..3db07b5 100644
=2D-- a/src/copy.c
+++ b/src/copy.c
@@ -764,6 +764,23 @@ fchmod_or_lchmod (int desc, char const *name, mode_t m=
ode)
   return lchmod (name, mode);
 }

+#ifndef HAVE_STRUCT_STAT_ST_BLOCKS
+# define HAVE_STRUCT_STAT_ST_BLOCKS 0
+#endif
+
+/* Use a heuristic to determine whether stat buffer SB comes from a file
+   with sparse blocks.  If the file has fewer blocks than would normally
+   be needed for a file of its size, then at least one of the blocks in
+   the file is a hole.  In that case, return true.  */
+static bool
+is_probably_sparse (struct stat const *sb)
+{
+  return (HAVE_STRUCT_STAT_ST_BLOCKS
+          && S_ISREG (sb->st_mode)
+          && ST_NBLOCKS (*sb) < sb->st_size / ST_NBLOCKSIZE);
+}
+
+
 /* Copy a regular file from SRC_NAME to DST_NAME.
    If the source file contains holes, copies holes and blocks of zeros
    in the source file as holes in the destination file.
@@ -984,15 +1001,13 @@ copy_reg (char const *src_name, char const *dst_name,
           if (x->sparse_mode =3D=3D SPARSE_ALWAYS)
             make_holes =3D true;

=2D#if HAVE_STRUCT_STAT_ST_BLOCKS
           /* Use a heuristic to determine whether SRC_NAME contains any sp=
arse
              blocks.  If the file has fewer blocks than would normally be
              needed for a file of its size, then at least one of the block=
s in
              the file is a hole.  */
=2D          if (x->sparse_mode =3D=3D SPARSE_AUTO && S_ISREG (src_open_sb.=
st_mode)
=2D              && ST_NBLOCKS (src_open_sb) < src_open_sb.st_size / ST_NBL=
OCKSIZE)
+          if (x->sparse_mode =3D=3D SPARSE_AUTO
+              && is_probably_sparse (&src_open_sb))
             make_holes =3D true;
=2D#endif
         }

       /* If not making a sparse file, try to use a more-efficient
=2D-
1.7.5.rc2.295.g19c42


From=2039fdf629729319ab4011cf15c0a16cba4e4aba1b Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 11:21:09 +0200
Subject: [PATCH 4/4] copy: use FIEMAP (extent_copy) only for
 apparently-sparse files,

to avoid the expense of extent_copy's unconditional use of
FIEMAP_FLAG_SYNC.
* src/copy.c (copy_reg): Do not attempt extent_copy on a file
that appears to have no holes.
* NEWS (Changes in behavior): Document this.  At first I labeled this
as a bug fix, but that would be inaccurate, considering there is no
documentation of FIEMAP semantics, nor even consensus among kernel
FS developers.  Here's hoping SEEK_HOLE/SEEK_DATA support will soon
make it into the linux kernel.
=2D--
 NEWS       |   13 +++++++++++++
 src/copy.c |   37 +++++++++++++++++++++----------------
 2 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/NEWS b/NEWS
index 4873b5a..7bc2ef3 100644
=2D-- a/NEWS
+++ b/NEWS
@@ -2,6 +2,19 @@ GNU coreutils NEWS                                    -*- =
outline -*-

 * Noteworthy changes in release ?.? (????-??-??) [?]

+** Changes in behavior
+
+  cp's extent-based (FIEMAP) copying code is more reliable in the face
+  of varying and undocumented file system semantics:
+  - it no longer treats unwritten extents specially
+  - a FIEMAP-based extent copy always uses the FIEMAP_FLAG_SYNC flag.
+      Before, it would incur the performance penalty of that sync only
+      for 2.6.38 and older kernels.  We thought all problems would be
+      resolved for 2.6.39.
+  - it now attempts a FIEMAP copy only on a file that appears sparse.
+      Sparse files are relatively unusual, and the copying code incurs
+      the performance penalty of the now-mandatory sync only for them.
+

 * Noteworthy changes in release 8.11 (2011-04-13) [stable]

diff --git a/src/copy.c b/src/copy.c
index 3db07b5..6edf52e 100644
=2D-- a/src/copy.c
+++ b/src/copy.c
@@ -993,6 +993,7 @@ copy_reg (char const *src_name, char const *dst_name,

       /* Deal with sparse files.  */
       bool make_holes =3D false;
+      bool sparse_src =3D false;

       if (S_ISREG (sb.st_mode))
         {
@@ -1005,8 +1006,8 @@ copy_reg (char const *src_name, char const *dst_name,
              blocks.  If the file has fewer blocks than would normally be
              needed for a file of its size, then at least one of the block=
s in
              the file is a hole.  */
=2D          if (x->sparse_mode =3D=3D SPARSE_AUTO
=2D              && is_probably_sparse (&src_open_sb))
+          sparse_src =3D is_probably_sparse (&src_open_sb);
+          if (x->sparse_mode =3D=3D SPARSE_AUTO && sparse_src)
             make_holes =3D true;
         }

@@ -1038,21 +1039,25 @@ copy_reg (char const *src_name, char const *dst_nam=
e,
       buf_alloc =3D xmalloc (buf_size + buf_alignment_slop);
       buf =3D ptr_align (buf_alloc, buf_alignment);

=2D      bool normal_copy_required;
=2D      /* Perform an efficient extent-based copy, falling back to the
=2D         standard copy only if the initial extent scan fails.  If the
=2D         '--sparse=3Dnever' option is specified, write all data but use
=2D         any extents to read more efficiently.  */
=2D      if (extent_copy (source_desc, dest_desc, buf, buf_size,
=2D                       src_open_sb.st_size,
=2D                       S_ISREG (sb.st_mode) ? x->sparse_mode : SPARSE_NE=
VER,
=2D                       src_name, dst_name, &normal_copy_required))
=2D        goto preserve_metadata;
=2D
=2D      if (! normal_copy_required)
+      if (sparse_src)
         {
=2D          return_val =3D false;
=2D          goto close_src_and_dst_desc;
+          bool normal_copy_required;
+
+          /* Perform an efficient extent-based copy, falling back to the
+             standard copy only if the initial extent scan fails.  If the
+             '--sparse=3Dnever' option is specified, write all data but use
+             any extents to read more efficiently.  */
+          if (extent_copy (source_desc, dest_desc, buf, buf_size,
+                           src_open_sb.st_size,
+                           S_ISREG (sb.st_mode) ? x->sparse_mode : SPARSE_=
NEVER,
+                           src_name, dst_name, &normal_copy_required))
+            goto preserve_metadata;
+
+          if (! normal_copy_required)
+            {
+              return_val =3D false;
+              goto close_src_and_dst_desc;
+            }
         }

       off_t n_read;
=2D-
1.7.5.rc2.295.g19c42


From MAILER-DAEMON Wed Apr 20 11:22:18 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCZEQ-0008En-2U
	for mharc-coreutils@gnu.org; Wed, 20 Apr 2011 11:22:18 -0400
Received: from eggs.gnu.org ([140.186.70.92]:59099)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+cabedcb8a302d715f734+2796+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QCZDv-0008Dj-8f
	for coreutils@gnu.org; Wed, 20 Apr 2011 11:21:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from
	<BATV+cabedcb8a302d715f734+2796+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QCZDu-0002BN-5D
	for coreutils@gnu.org; Wed, 20 Apr 2011 11:21:47 -0400
Received: from bombadil.infradead.org ([18.85.46.34]:46588)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+cabedcb8a302d715f734+2796+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QCZDu-00025l-3a
	for coreutils@gnu.org; Wed, 20 Apr 2011 11:21:46 -0400
Received: from hch by bombadil.infradead.org with local (Exim 4.72 #1 (Red Hat
	Linux)) id 1QCZDf-0005mH-6C; Wed, 20 Apr 2011 15:21:31 +0000
Date: Wed, 20 Apr 2011 11:21:31 -0400
From: Christoph Hellwig <hch@infradead.org>
To: Ted Ts'o <tytso@mit.edu>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110420152131.GA7123@infradead.org>
References: <4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
	<4DAD987F.5000506@sandeen.net> <20110419160114.GE3030@thunk.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110419160114.GE3030@thunk.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SRS-Rewrite: SMTP reverse-path rewritten from <hch@infradead.org> by
	bombadil.infradead.org See http://www.infradead.org/rpr.html
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 18.85.46.34
X-Mailman-Approved-At: Wed, 20 Apr 2011 11:22:16 -0400
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	Dave Chinner <david@fromorbit.com>, Yongqiang Yang <xiaoqiangnk@gmail.com>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Apr 2011 15:21:48 -0000

On Tue, Apr 19, 2011 at 12:01:14PM -0400, Ted Ts'o wrote:
> 1) We define it as only reflecting ondisk state, and nuke the delalloc
> flag from orbit.
> 
> 2) We state that if the file is currently has unflushed pages in the
> page cache, and FIEMAP_FLAG_SYNC is not passed, whether or not extents
> return the DELALLOC flag or how they handle the UNWRITTEN flag is
> undefined.

That seems like a weird option, as the pagecache state really has
nothing to do at all with the extent layout, and the existence of dirty
pages really has nothing to do with the unwritten flag.

> 3) We state that FIEMAP is supposed to return information which
> reflects the union of the on-disk and page cache state, with all that
> this implies.

How do you want to union the existance of an extent with a state
on disk, with a pending modification to it that is still in-memory
and not flushed out to disk yet?  This is looking into an uncertain
future, as the extent map might change in various other ways before
the transaction to conver the unwritten extents goes to disk.

And if we do this it would need to be a new option to FIEMAP, as
it changes the semantics from the existing one that returns the
actual state on disk (plus the magic delalloc bit).

And even if you find semantics that take pending unwrittent extent
conversions into account and still make sense how do you plan to
implement them?  For buffered writes into unwritten extents it could
be done by walking the pagecache and buffers after adding a new
flag for an already converted unwritten extent to the buffer head
state.  But there's no easy way to do that for direct I/O.

> In the case of #1 and #2, we really need to implement support for
> SEEK_HOLE/SEEK_DATA for userspace programs like cp who want to know
> this information.

We need to do that anyway, as fiemap is a horrible interface for
tools that just want to skip holes.  



From MAILER-DAEMON Wed Apr 20 11:30:46 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCZMc-0000PG-AS
	for mharc-coreutils@gnu.org; Wed, 20 Apr 2011 11:30:46 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55508)
	by lists.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+cabedcb8a302d715f734+2796+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QCZLs-0000J2-8M
	for coreutils@gnu.org; Wed, 20 Apr 2011 11:30:01 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from
	<BATV+cabedcb8a302d715f734+2796+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QCZLr-0003Jg-Dv
	for coreutils@gnu.org; Wed, 20 Apr 2011 11:30:00 -0400
Received: from bombadil.infradead.org ([18.85.46.34]:35468)
	by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from
	<BATV+cabedcb8a302d715f734+2796+infradead.org+hch@bombadil.srs.infradead.org>)
	id 1QCZLr-0003Jc-CE
	for coreutils@gnu.org; Wed, 20 Apr 2011 11:29:59 -0400
Received: from hch by bombadil.infradead.org with local (Exim 4.72 #1 (Red Hat
	Linux)) id 1QCZLo-0008Aj-8t; Wed, 20 Apr 2011 15:29:56 +0000
Date: Wed, 20 Apr 2011 11:29:56 -0400
From: Christoph Hellwig <hch@infradead.org>
To: Dave Chinner <david@fromorbit.com>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110420152956.GA24641@infradead.org>
References: <E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
	<20110419210825.GJ23985@dastard>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110419210825.GJ23985@dastard>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SRS-Rewrite: SMTP reverse-path rewritten from <hch@infradead.org> by
	bombadil.infradead.org See http://www.infradead.org/rpr.html
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 18.85.46.34
X-Mailman-Approved-At: Wed, 20 Apr 2011 11:30:45 -0400
Cc: Andreas Dilger <adilger@dilger.ca>, Ted Ts'o <tytso@mit.edu>,
	Eric Sandeen <sandeen@sandeen.net>, Yongqiang Yang <xiaoqiangnk@gmail.com>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Apr 2011 15:30:01 -0000

On Wed, Apr 20, 2011 at 07:08:25AM +1000, Dave Chinner wrote:
> So, it logically follws that you need to use the SYNC flag for both
> unwritten extents and delalloc extents to find out where there data
> realy lies by converting them to real, written extents. i.e. the
> only extents you can trust contain data from FIEMAP are the real
> extents on disk....

Even more funny is that the bug report that started this thread involved
software that didn't actually care about the location on disk, at all.

cp from coreutils really just wanted an efficient way to skip holes
in sparse files, and we got into a chain reaction of various flaws
and oversights :

 (1) Linux lacks the SEEK_HOLE/SEEK_DATA interface that would make
     skipping holes trivial and thus coreutils has to use FIEMAP.
 (2) ext4 and btrfs in some cases mishandled reporting delalloc
     extents, which means coreutils had to add the sync flag,
     despite not caring where data is on disk
 (3) coreutils tried to treat unwrittent extents as holes.  Which
     makes some sense given their high-level description, although
     probably not too much in practice given that we explicitly
     allocated blocks to these "holes" to optimize performance.
     But the main issue here is that there is no documentation
     that clearly states that unwrittent extents reported by
     FIEMAP may actually contain useful data.  In fact there's
     no useful documentation for FIEMAP outside the kernel tree.
     And interface that complex really needs a manpage.



From MAILER-DAEMON Wed Apr 20 13:22:15 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCb6V-00086V-52
	for mharc-coreutils@gnu.org; Wed, 20 Apr 2011 13:22:15 -0400
Received: from eggs.gnu.org ([140.186.70.92]:43341)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCb64-00081s-VZ
	for coreutils@gnu.org; Wed, 20 Apr 2011 13:21:49 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCb64-0007c4-6x
	for coreutils@gnu.org; Wed, 20 Apr 2011 13:21:48 -0400
Received: from li9-11.members.linode.com ([67.18.176.11]:57291
	helo=test.thunk.org) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <tytso@thunk.org>) id 1QCb64-0007bJ-0W
	for coreutils@gnu.org; Wed, 20 Apr 2011 13:21:48 -0400
Received: from root (helo=tytso-glaptop)
	by test.thunk.org with local-esmtp (Exim 4.69)
	(envelope-from <tytso@thunk.org>)
	id 1QCb5l-0001eV-SX; Wed, 20 Apr 2011 17:21:29 +0000
Received: from tytso by tytso-glaptop with local (Exim 4.71)
	(envelope-from <tytso@thunk.org>)
	id 1QCb5j-0003qM-Nn; Wed, 20 Apr 2011 13:21:27 -0400
Date: Wed, 20 Apr 2011 13:21:27 -0400
From: Ted Ts'o <tytso@mit.edu>
To: Christoph Hellwig <hch@infradead.org>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
Message-ID: <20110420172127.GF3030@thunk.org>
References: <20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419034455.GB23985@dastard>
	<BANLkTinjh968ECqAobQ677hnV5yzke1ncw@mail.gmail.com>
	<20110419074538.GG23985@dastard> <20110419140909.GD3030@thunk.org>
	<4DAD987F.5000506@sandeen.net> <20110419160114.GE3030@thunk.org>
	<20110420152131.GA7123@infradead.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20110420152131.GA7123@infradead.org>
User-Agent: Mutt/1.5.20 (2009-06-14)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: tytso@thunk.org
X-SA-Exim-Scanned: No (on test.thunk.org); SAEximRunCond expanded to false
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 1)
X-Received-From: 67.18.176.11
X-Mailman-Approved-At: Wed, 20 Apr 2011 13:22:13 -0400
Cc: Andreas Dilger <adilger@dilger.ca>, Eric Sandeen <sandeen@sandeen.net>,
	Dave Chinner <david@fromorbit.com>, Yongqiang Yang <xiaoqiangnk@gmail.com>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <http://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <http://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Apr 2011 17:21:50 -0000

On Wed, Apr 20, 2011 at 11:21:31AM -0400, Christoph Hellwig wrote:
> 
> How do you want to union the existance of an extent with a state
> on disk, with a pending modification to it that is still in-memory
> and not flushed out to disk yet?  This is looking into an uncertain
> future, as the extent map might change in various other ways before
> the transaction to conver the unwritten extents goes to disk.

So for example, suppose you have a single unwritten extent on disk,
but there are 3 regions within that extent range's that have unwritten
pages, you return 3 or 4 fiemap_extent structures, reflecting the
state if the unwritten pages were pushed out to disk at the time of
the fiemap ioctl --- but without actually doing the expensive sync
operation.  The one case where you can't do that is in the case of
delayed allocation blocks, since you won't know where on disk they
would be going, necessarily --- but hey, conveniently we have a
DELALLOC bit already defined....

> And if we do this it would need to be a new option to FIEMAP, as
> it changes the semantics from the existing one that returns the
> actual state on disk (plus the magic delalloc bit).

Well, we seem to have inconsistent semantics right now, because we
never defined the semantics clearly enough from the beginning.  So no
matter which choice we choose, including "the on-disk extent state
only, and nuke the delalloc bit", we will be changing semantics.  I'm
not sure we can get around that.

> And even if you find semantics that take pending unwrittent extent
> conversions into account and still make sense how do you plan to
> implement them?  For buffered writes into unwritten extents it could
> be done by walking the pagecache and buffers after adding a new
> flag for an already converted unwritten extent to the buffer head
> state.  But there's no easy way to do that for direct I/O.

If the file is being actively modified (for example with direct I/O),
there will be inevitably race conditions.  If only some of the pending
conversions have been taken into account, that seems like it's
reasonable result.  If a file is actively being modified by many DIO
writes, even using FIEMAP_FLAG_SYNC isn't going to help you get a
coherent view of the file, so this seems to be a previously unsolved
problem....

> > In the case of #1 and #2, we really need to implement support for
> > SEEK_HOLE/SEEK_DATA for userspace programs like cp who want to know
> > this information.
> 
> We need to do that anyway, as fiemap is a horrible interface for
> tools that just want to skip holes.

I agree that implementing SEEK_HOLE/SEEK_DATA is a good thing
regardless of which choice we end up choosing.

	      	    	      	     - Ted


From MAILER-DAEMON Thu Apr 21 15:40:38 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QCzjx-0006VL-W1
	for mharc-coreutils@gnu.org; Thu, 21 Apr 2011 15:40:38 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51119)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <matthew@linuxfromscratch.org>) id 1QCzas-0004aV-P0
	for coreutils@gnu.org; Thu, 21 Apr 2011 15:31:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <matthew@linuxfromscratch.org>) id 1QCzar-0004zB-F2
	for coreutils@gnu.org; Thu, 21 Apr 2011 15:31:14 -0400
Received: from quantum.linuxfromscratch.org ([216.171.237.234]:44420)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <matthew@linuxfromscratch.org>) id 1QCzar-0004yY-AH
	for coreutils@gnu.org; Thu, 21 Apr 2011 15:31:13 -0400
Received: by quantum.linuxfromscratch.org (Postfix, from userid 108)
	id CBB55140A1; Thu, 21 Apr 2011 13:31:10 -0600 (MDT)
Received: from localhost (quantum.linuxfromscratch.org [216.171.237.234])
	by quantum.linuxfromscratch.org (Postfix) with ESMTP id 0842D14089;
	Thu, 21 Apr 2011 13:31:03 -0600 (MDT)
MIME-Version: 1.0
Date: Thu, 21 Apr 2011 13:31:03 -0600
From: Matthew Burgess <matthew@linuxfromscratch.org>
To: coreutils@gnu.org
Subject: Problem with Coreutils & i18n patch
Message-ID: <ad1ffe36a1b5e6969cc5035281b005b9@quantum.linuxfromscratch.org>
X-Sender: matthew@linuxfromscratch.org
User-Agent: RoundCube Webmail/0.1b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 216.171.237.234
X-Mailman-Approved-At: Thu, 21 Apr 2011 15:40:37 -0400
Cc: matthew@linuxfromscratch.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Apr 2011 19:31:19 -0000

Hi all,

I realise that the i18n patch [0] has already been rejected by
upstream coreutils developers, but was wondering whether
you might be able to tell me why it's having the effect we're
seeing on builds of Coreutils-8.11.

When running 'make check-root', the following is output to
tests/.built-programs:

------
 cd .. && /bin/sh ./config.status src/Makefile depfiles
config.status: creating src/Makefile
config.status: executing depfiles commands
[ base64 basename cat chcon chgrp chmod chown chroot cksum comm cp csplit=
 cut date dd df dir dircolors dirname du echo env expand expr factor fals=
e fmt fold ginstall groups head hostid id join kill link ln logname ls md=
5sum mkdir mkfifo mknod mktemp mv nice nl nohup nproc od paste pathchk pi=
nky pr printenv printf ptx pwd readlink rm rmdir runcon seq sha1sum sha22=
4sum sha256sum sha384sum sha512sum shred shuf sleep sort split stat stdbu=
f stty sum sync tac tail tee test timeout touch tr true truncate tsort tt=
y uname unexpand uniq unlink uptime users vdir wc who whoami yes
------

Now, obviously, when the misc/help-version test is run, which
read the .built-programs file, they fail because it'll try to
run 'cd --version'.

So, the questions I have are why src/Makefile need to be rebuilt?rebuildi=
ng,
and why/how do those rebuild commands end up in tests/.built-programs

I presume it's being triggered by the patching of tests/Makefile.am
but I've no idea why/how the two are linked, and how we may be able
to work around this (aside from obviously dropping the i18n patch
completely).

Thanks,

Matt.

[0] http://www.linuxfromscratch.org/patches/lfs/development/coreutils-8.1=
1-i18n-1.patch



From MAILER-DAEMON Thu Apr 21 16:01:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QD04T-0008Vn-G0
	for mharc-coreutils@gnu.org; Thu, 21 Apr 2011 16:01:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47324)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD04Q-0008UH-LQ
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:01:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD04O-0001ZJ-Ou
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:01:46 -0400
Received: from mx.meyering.net ([82.230.74.64]:41729)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD04O-0001ZD-8E
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:01:44 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 951EE601A4; Thu, 21 Apr 2011 22:01:43 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
In-Reply-To: <87d3khugv1.fsf@rho.meyering.net> (Jim Meyering's message of
	"Wed, 20 Apr 2011 16:39:30 +0200")
References: <20110414102608.GA1678@x4.trippels.de>
	<87d3khugv1.fsf@rho.meyering.net>
Date: Thu, 21 Apr 2011 22:01:43 +0200
Message-ID: <87r58vqspk.fsf@rho.meyering.net>
Lines: 398
MIME-Version: 1.0
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: linux-ext4@vger.kernel.org, coreutils@gnu.org, xfs@oss.sgi.com
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Apr 2011 20:01:48 -0000

Jim Meyering wrote:
> Markus Trippelsdorf wrote:
>> I trashed my system this morning when I installed coreutils-8.11.
>>
>> What happened is that coreutils compiles and links correctly, but then
>> the following command (during the installation phase):
>>
>> ./ginstall chroot hostid nice who users pinky stty df stdbuf [ base64
> ...
>>
>> apparently produces files which have the length of the originals but are
>> full of zeros. (and these were then installed to my live system, thereby
>> trashing it).
> ...
>
> Thanks again for the report.
> I believe that the following series addresses this problem
> and have confirmed that tests pass with 2.6.39-rc3 on all
> of ext3, ext4, btrfs and xfs -- though there was what appears
> to be a spurious failure in tests/cp/sparse-fiemap when run on xfs.
> On one iteration of this loop, with j=3D31, in these loops
>
>   for i in $(seq 1 2 21); do
>     for j in 1 2 31 100; do
>
> [in http://git.savannah.gnu.org/cgit/coreutils.git/tree/tests/cp/sparse-f=
iemap]
> the two files compared equal, yet their extents did not match,
> even after merging.  I'm inclined to skip the extent-comparing check
> at least for XFS, now.
>
> Here's the unusually-technical-for-NEWS summary:

[slightly updated and pushed, along with test-adjusting changes]

** Changes in behavior

  cp's extent-based (FIEMAP) copying code is more reliable in the face
  of varying and undocumented file system semantics:
  - it no longer treats unwritten extents specially
  - a FIEMAP-based extent copy always uses the FIEMAP_FLAG_SYNC flag.
      Before, it would incur the performance penalty of that sync only
      for 2.6.38 and older kernels.  We thought all problems would be
      resolved for 2.6.39.
  - it now attempts a FIEMAP copy only on a file that appears sparse.
      Sparse files are relatively unusual, and the copying code incurs
      the performance penalty of the now-mandatory sync only for them.


From=209bcd045f812a75cf96ba392bc45529422f87c088 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 09:49:15 +0200
Subject: [PATCH 1/6] copy: always use FIEMAP_FLAG_SYNC, for now

* src/extent-scan.c (extent_need_sync): Always return true,
to make the sole caller always use FIEMAP_FLAG_SYNC.
This will doubtless have an undesirable performance impact,
but we'll mitigate that shortly, by using extent_copy only on
files with holes.
=2D--
 src/extent-scan.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/src/extent-scan.c b/src/extent-scan.c
index da7eb9d..596e7f7 100644
=2D-- a/src/extent-scan.c
+++ b/src/extent-scan.c
@@ -36,6 +36,13 @@
 static bool
 extent_need_sync (void)
 {
+  /* For now always return true, to be on the safe side.
+     If/when FIEMAP semantics are well defined (before SEEK_HOLE support
+     is usable) and kernels implementing them are in use, we may relax
+     this once again.  */
+  return true;
+
+#if FIEMAP_BEHAVIOR_IS_DEFINED_AND_USABLE
   static int need_sync =3D -1;

   if (need_sync =3D=3D -1)
@@ -57,6 +64,7 @@ extent_need_sync (void)
     }

   return need_sync;
+#endif
 }

 /* Allocate space for struct extent_scan, initialize the entries if
=2D-
1.7.5.rc3.291.g63e4e


From=20bef4fa1e1a20c636979db159647a93e5954bc542 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 10:15:15 +0200
Subject: [PATCH 2/6] copy: do not treat unwritten extents specially: avoid
 XFS/ext4 data loss

* src/copy.c (extent_copy): Do not treat "unwritten extents" specially.
Otherwise, with a release-candidate 2.6.39-rc3 kernel, XFS or ext4,
when using gold as your linker, and if you forget to run "make check",
you could end up installing files full of zeros instead of the expected
binaries.  For a lot of discussion, see
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/37895
* tests/cp/fiemap-empty: Disable this test.
=2D--
 src/copy.c            |    5 ++++-
 tests/cp/fiemap-empty |    5 +++++
 2 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index 9b53127..f6f9ea6 100644
=2D-- a/src/copy.c
+++ b/src/copy.c
@@ -398,7 +398,10 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_=
t buf_size,
           /* Treat an unwritten but allocated extent much like a hole.
              I.E. don't read, but don't convert to a hole in the destinati=
on,
              unless SPARSE_ALWAYS.  */
=2D          if (scan.ext_info[i].ext_flags & FIEMAP_EXTENT_UNWRITTEN)
+          /* For now, do not treat FIEMAP_EXTENT_UNWRITTEN specially,
+             because that (in combination with no sync) would lead to data
+             loss at least on XFS and ext4 when using 2.6.39-rc3 kernels. =
 */
+          if (0 && (scan.ext_info[i].ext_flags & FIEMAP_EXTENT_UNWRITTEN))
             {
               empty_extent =3D true;
               last_ext_len =3D 0;
diff --git a/tests/cp/fiemap-empty b/tests/cp/fiemap-empty
index 64c3254..836668e 100755
=2D-- a/tests/cp/fiemap-empty
+++ b/tests/cp/fiemap-empty
@@ -19,6 +19,11 @@
 . "${srcdir=3D.}/init.sh"; path_prepend_ ../src
 print_ver_ cp

+# FIXME: enable any part of this test that is still relevant,
+# or, if none are relevant (now that cp does not handle unwritten
+# extents), just remove the test altogether.
+skip_test_ 'disabled for now'
+
 touch fiemap_chk
 fiemap_capable_ fiemap_chk ||
   skip_test_ 'this file system lacks FIEMAP support'
=2D-
1.7.5.rc3.291.g63e4e


From=20846d826096fc8fb621d751f8b3db1da68a8bbd06 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 10:23:32 +0200
Subject: [PATCH 3/6] copy: factor out a tiny sparse-testing function

* src/copy.c (HAVE_STRUCT_STAT_ST_BLOCKS): Define to 0 if undefined,
so we can use it in the return expression, here:
(is_probably_sparse): New function, factored out of...
(copy_reg): ...here.  Use the new function.
=2D--
 src/copy.c |   23 +++++++++++++++++++----
 1 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index f6f9ea6..3db07b5 100644
=2D-- a/src/copy.c
+++ b/src/copy.c
@@ -764,6 +764,23 @@ fchmod_or_lchmod (int desc, char const *name, mode_t m=
ode)
   return lchmod (name, mode);
 }

+#ifndef HAVE_STRUCT_STAT_ST_BLOCKS
+# define HAVE_STRUCT_STAT_ST_BLOCKS 0
+#endif
+
+/* Use a heuristic to determine whether stat buffer SB comes from a file
+   with sparse blocks.  If the file has fewer blocks than would normally
+   be needed for a file of its size, then at least one of the blocks in
+   the file is a hole.  In that case, return true.  */
+static bool
+is_probably_sparse (struct stat const *sb)
+{
+  return (HAVE_STRUCT_STAT_ST_BLOCKS
+          && S_ISREG (sb->st_mode)
+          && ST_NBLOCKS (*sb) < sb->st_size / ST_NBLOCKSIZE);
+}
+
+
 /* Copy a regular file from SRC_NAME to DST_NAME.
    If the source file contains holes, copies holes and blocks of zeros
    in the source file as holes in the destination file.
@@ -984,15 +1001,13 @@ copy_reg (char const *src_name, char const *dst_name,
           if (x->sparse_mode =3D=3D SPARSE_ALWAYS)
             make_holes =3D true;

=2D#if HAVE_STRUCT_STAT_ST_BLOCKS
           /* Use a heuristic to determine whether SRC_NAME contains any sp=
arse
              blocks.  If the file has fewer blocks than would normally be
              needed for a file of its size, then at least one of the block=
s in
              the file is a hole.  */
=2D          if (x->sparse_mode =3D=3D SPARSE_AUTO && S_ISREG (src_open_sb.=
st_mode)
=2D              && ST_NBLOCKS (src_open_sb) < src_open_sb.st_size / ST_NBL=
OCKSIZE)
+          if (x->sparse_mode =3D=3D SPARSE_AUTO
+              && is_probably_sparse (&src_open_sb))
             make_holes =3D true;
=2D#endif
         }

       /* If not making a sparse file, try to use a more-efficient
=2D-
1.7.5.rc3.291.g63e4e


From=2018a474d755aa10d881243d9457d2c420c5e4ea77 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Wed, 20 Apr 2011 11:21:09 +0200
Subject: [PATCH 4/6] copy: use FIEMAP (extent_copy) only for
 apparently-sparse files,

to avoid the expense of extent_copy's unconditional use of
FIEMAP_FLAG_SYNC.
* src/copy.c (copy_reg): Do not attempt extent_copy on a file
that appears to have no holes.
* NEWS (Changes in behavior): Document this.  At first I labeled this
as a bug fix, but that would be inaccurate, considering there is no
documentation of FIEMAP semantics, nor even consensus among kernel
FS developers.  Here's hoping SEEK_HOLE/SEEK_DATA support will soon
make it into the linux kernel.
=2D--
 NEWS       |   13 +++++++++++++
 src/copy.c |   37 +++++++++++++++++++++----------------
 2 files changed, 34 insertions(+), 16 deletions(-)

diff --git a/NEWS b/NEWS
index 4873b5a..7bc2ef3 100644
=2D-- a/NEWS
+++ b/NEWS
@@ -2,6 +2,19 @@ GNU coreutils NEWS                                    -*- =
outline -*-

 * Noteworthy changes in release ?.? (????-??-??) [?]

+** Changes in behavior
+
+  cp's extent-based (FIEMAP) copying code is more reliable in the face
+  of varying and undocumented file system semantics:
+  - it no longer treats unwritten extents specially
+  - a FIEMAP-based extent copy always uses the FIEMAP_FLAG_SYNC flag.
+      Before, it would incur the performance penalty of that sync only
+      for 2.6.38 and older kernels.  We thought all problems would be
+      resolved for 2.6.39.
+  - it now attempts a FIEMAP copy only on a file that appears sparse.
+      Sparse files are relatively unusual, and the copying code incurs
+      the performance penalty of the now-mandatory sync only for them.
+

 * Noteworthy changes in release 8.11 (2011-04-13) [stable]

diff --git a/src/copy.c b/src/copy.c
index 3db07b5..6edf52e 100644
=2D-- a/src/copy.c
+++ b/src/copy.c
@@ -993,6 +993,7 @@ copy_reg (char const *src_name, char const *dst_name,

       /* Deal with sparse files.  */
       bool make_holes =3D false;
+      bool sparse_src =3D false;

       if (S_ISREG (sb.st_mode))
         {
@@ -1005,8 +1006,8 @@ copy_reg (char const *src_name, char const *dst_name,
              blocks.  If the file has fewer blocks than would normally be
              needed for a file of its size, then at least one of the block=
s in
              the file is a hole.  */
=2D          if (x->sparse_mode =3D=3D SPARSE_AUTO
=2D              && is_probably_sparse (&src_open_sb))
+          sparse_src =3D is_probably_sparse (&src_open_sb);
+          if (x->sparse_mode =3D=3D SPARSE_AUTO && sparse_src)
             make_holes =3D true;
         }

@@ -1038,21 +1039,25 @@ copy_reg (char const *src_name, char const *dst_nam=
e,
       buf_alloc =3D xmalloc (buf_size + buf_alignment_slop);
       buf =3D ptr_align (buf_alloc, buf_alignment);

=2D      bool normal_copy_required;
=2D      /* Perform an efficient extent-based copy, falling back to the
=2D         standard copy only if the initial extent scan fails.  If the
=2D         '--sparse=3Dnever' option is specified, write all data but use
=2D         any extents to read more efficiently.  */
=2D      if (extent_copy (source_desc, dest_desc, buf, buf_size,
=2D                       src_open_sb.st_size,
=2D                       S_ISREG (sb.st_mode) ? x->sparse_mode : SPARSE_NE=
VER,
=2D                       src_name, dst_name, &normal_copy_required))
=2D        goto preserve_metadata;
=2D
=2D      if (! normal_copy_required)
+      if (sparse_src)
         {
=2D          return_val =3D false;
=2D          goto close_src_and_dst_desc;
+          bool normal_copy_required;
+
+          /* Perform an efficient extent-based copy, falling back to the
+             standard copy only if the initial extent scan fails.  If the
+             '--sparse=3Dnever' option is specified, write all data but use
+             any extents to read more efficiently.  */
+          if (extent_copy (source_desc, dest_desc, buf, buf_size,
+                           src_open_sb.st_size,
+                           S_ISREG (sb.st_mode) ? x->sparse_mode : SPARSE_=
NEVER,
+                           src_name, dst_name, &normal_copy_required))
+            goto preserve_metadata;
+
+          if (! normal_copy_required)
+            {
+              return_val =3D false;
+              goto close_src_and_dst_desc;
+            }
         }

       off_t n_read;
=2D-
1.7.5.rc3.291.g63e4e


From=20223e3832eb5a9b1aadf0a69d076f40116389565c Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Thu, 21 Apr 2011 18:08:20 +0200
Subject: [PATCH 5/6] tests: sparse-fiemap: report more detail upon failure;
 ignore an FP

* tests/cp/sparse-fiemap: Fail right away with details, when cmp fails.
When extent maps are found to differ, display them and merely warn.
=2D--
 tests/cp/sparse-fiemap |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
index 2c6a250..2e8c95b 100755
=2D-- a/tests/cp/sparse-fiemap
+++ b/tests/cp/sparse-fiemap
@@ -75,7 +75,7 @@ for i in $(seq 1 2 21); do
     # for the same reasons.
     cp --sparse=3Dalways j1 j2 || fail=3D1

=2D    cmp j1 j2 || fail=3D1
+    cmp j1 j2 || fail_ "data loss i=3D$i j=3D$j"
     if ! filefrag -vs j1 | grep -F extent >/dev/null; then
       test $skip !=3D 1 && warn_ 'skipping part; you lack filefrag'
       skip=3D1
@@ -98,8 +98,12 @@ for i in $(seq 1 2 21); do
       # exclude the physical block numbers; they always differ
       filefrag -v j1 > ff1 || framework_failure
       filefrag -vs j2 > ff2 || framework_failure
=2D      { f ff1; f ff2; } | $PERL $abs_top_srcdir/tests/filefrag-extent-co=
mpare ||
=2D        fail=3D1
+      { f ff1; f ff2; } | $PERL $abs_top_srcdir/tests/filefrag-extent-comp=
are \
+        || {
+             warn_ ignoring filefrag-reported extent map differences
+             # Show the differing extent maps.
+             head -99 ff1 ff2
+           }
     fi
     test $fail =3D 1 && break 2
   done
=2D-
1.7.5.rc3.291.g63e4e


From=208c0b1de42c615a82ce7e32901ad1e4dca95b3657 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Thu, 21 Apr 2011 21:01:13 +0200
Subject: [PATCH 6/6] tests: sparse-fiemap: with root/ext3, do not create an
 ext4 FS

* tests/cp/sparse-fiemap: When this test was run as root on an ext3
file system, (ext3 had known problems), it would trickily create and
mount a loopback ext4 file system and use that instead.  However, due
to a bug in 2.6.39-rc1..rc3, this loopback test (when run in another
loopback FS) exposed a bug with 1k-blocksize ext4 whereby non-NUL
data would be read from a hole.  For details, see this:
http://thread.gmane.org/gmane.comp.file-systems.ext4/24495
=2D--
 tests/cp/sparse-fiemap |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
index 2e8c95b..1394060 100755
=2D-- a/tests/cp/sparse-fiemap
+++ b/tests/cp/sparse-fiemap
@@ -26,6 +26,10 @@ touch fiemap_chk
 if fiemap_capable_ fiemap_chk && ! df -t ext3 . >/dev/null; then
   : # Current partition has working extents.  Good!
 else
+  # FIXME: temporarily(?) skip this variant, at least until after this bug
+  # is fixed: http://thread.gmane.org/gmane.comp.file-systems.ext4/24495
+  skip_test_ "current file system has insufficient FIEMAP support"
+
   # It's not;  we need to create one, hence we need root access.
   require_root_

=2D-
1.7.5.rc3.291.g63e4e


From MAILER-DAEMON Thu Apr 21 16:09:38 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QD0C2-0001oj-9e
	for mharc-coreutils@gnu.org; Thu, 21 Apr 2011 16:09:38 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51718)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD0C0-0001oc-3W
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:09:36 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD0By-0003Pm-V9
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:09:36 -0400
Received: from mx.meyering.net ([82.230.74.64]:51101)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD0By-0003Pd-PN
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:09:34 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 670E2601A4; Thu, 21 Apr 2011 22:09:32 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: new snapshot available: coreutils-8.11.9-8c0b1d
Date: Thu, 21 Apr 2011 22:09:32 +0200
Message-ID: <87liz3qscj.fsf@rho.meyering.net>
Lines: 50
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coordinator@translationproject.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Apr 2011 20:09:37 -0000

Considering the potential for data loss when using coreutils-8.11 on
ext4 and xfs when using e.g., 2.6.39-rc3[*], here's a snapshot that will
soon become coreutils-8.12.  Please give it a try.

Here is the NEWS addition:

** Changes in behavior

  cp's extent-based (FIEMAP) copying code is more reliable in the face
  of varying and undocumented file system semantics:
  - it no longer treats unwritten extents specially
  - a FIEMAP-based extent copy always uses the FIEMAP_FLAG_SYNC flag.
      Before, it would incur the performance penalty of that sync only
      for 2.6.38 and older kernels.  We thought all problems would be
      resolved for 2.6.39.
  - it now attempts a FIEMAP copy only on a file that appears sparse.
      Sparse files are relatively unusual, and the copying code incurs
      the performance penalty of the now-mandatory sync only for them.

=2D--
[*] http://thread.gmane.org/gmane.comp.gnu.coreutils.general/1041

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D
coreutils snapshot: (.gz files are here, too)
  http://meyering.net/cu/coreutils-ss.tar.xz      4.7 MB
  http://meyering.net/cu/coreutils-ss.tar.xz.sig
  http://meyering.net/cu/coreutils-8.11.9-8c0b1d.tar.xz

There are .gz and .sig files here, too:
  http://people.redhat.com/meyering/cu/coreutils-ss.tar.xz
  http://people.redhat.com/meyering/cu/coreutils-8.11.9-8c0b1d.tar.xz

Changes in coreutils since v8.11:

Jim Meyering (8):
      post-release administrivia
      tests: remove spurious syntax from a perl snippet
      copy: always use FIEMAP_FLAG_SYNC, for now
      copy: do not treat unwritten extents specially: avoid XFS/ext4 data l=
oss
      copy: factor out a tiny sparse-testing function
      copy: use FIEMAP (extent_copy) only for apparently-sparse files,
      tests: sparse-fiemap: report more detail upon failure; ignore an FP
      tests: sparse-fiemap: with root/ext3, do not create an ext4 FS

Ond=C5=99ej Va=C5=A1=C3=ADk (1):
      dircolors: add .ear, .war, .sar, for Java jar-like archives


Changes in gnulib since v8.11:
  none taken


From MAILER-DAEMON Thu Apr 21 16:12:07 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QD0ER-0002gz-EE
	for mharc-coreutils@gnu.org; Thu, 21 Apr 2011 16:12:07 -0400
Received: from eggs.gnu.org ([140.186.70.92]:48160)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD0EP-0002gi-2u
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:12:06 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD0EO-00041p-3Q
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:12:05 -0400
Received: from mx.meyering.net ([82.230.74.64]:55418)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QD0EN-00041f-Ju
	for coreutils@gnu.org; Thu, 21 Apr 2011 16:12:03 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 94DA2601A4; Thu, 21 Apr 2011 22:12:02 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Eric Sandeen <sandeen@sandeen.net>
Subject: Re: Files full of zeros with coreutils-8.11 and xfs (FIEMAP related?)
In-Reply-To: <4DACFBEB.9040909@sandeen.net> (Eric Sandeen's message of "Mon,
	18 Apr 2011 22:05:15 -0500")
References: <20110414225904.GK21395@dastard> <4DA7836A.5040604@draigBrady.com>
	<20110415000940.GL21395@dastard>
	<76FFF648-CA02-494B-A862-566C66A8CB82@dilger.ca>
	<20110416005040.GP21395@dastard>
	<E7438588-5C5A-42B9-9B6C-FADF7CAC6D3A@dilger.ca>
	<4EEEA16E-1FDB-4430-A372-8F8701196E4C@mit.edu>
	<20110418004040.GS21395@dastard>
	<6C89E159-A5F6-4A06-A3D2-273BE4CFB9B5@dilger.ca>
	<BANLkTin=WEpSf6ddiOMNMOpCPP-wiEttSw@mail.gmail.com>
	<20110419025949.GA3030@thunk.org> <4DACFBEB.9040909@sandeen.net>
Date: Thu, 21 Apr 2011 22:12:02 +0200
Message-ID: <87fwpbqs8d.fsf@rho.meyering.net>
Lines: 28
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Andreas Dilger <adilger@dilger.ca>, Ted Ts'o <tytso@mit.edu>,
	Dave Chinner <david@fromorbit.com>, Yongqiang Yang <xiaoqiangnk@gmail.com>,
	xfs-oss <xfs@oss.sgi.com>, "coreutils@gnu.org" <coreutils@gnu.org>,
	"linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Apr 2011 20:12:06 -0000

Eric Sandeen wrote:
> On 4/18/11 9:59 PM, Ted Ts'o wrote:
> ...
>> Maybe coreutils will need to use FIEMAP_FLAG_SYNC initially, since
>> it's the only way to guarantee correct behaviour for XFS.  But I would
>> really rather that be the long-term way we leave things!
>
> XFS ... or ext4:
>
> # xfs_io -Ff -c "falloc 0 1m" -c "pwrite 0 512k" testfile;
> /root/fiemap-test testfile
> wrote 524288/524288 bytes at offset 0
> 512 KiB, 128 ops; 0.0000 sec (161.342 MiB/sec and 41303.6463 ops/sec)
> start 0 length -1 flags 0x0 count 32
> ext: 0 logical: [ 0..  255] phys: 34048..  34303 flags: 0x801 tot: 256
>
> # uname -r
> 2.6.39-0.rc3.git2.0.fc16.x86_64
>
> Above is on ext4.  It behaves exactly like XFS in my testing; data in
> the page cache does not cause fiemap to return anything other than
> "unwritten" for preallocated extents.

Thanks for the feedback.
In case anyone wants to test or review,
I've just made a coreutils snapshot:

  http://thread.gmane.org/gmane.comp.gnu.coreutils.general/1108


From MAILER-DAEMON Thu Apr 21 17:13:07 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QD1BT-000736-0n
	for mharc-coreutils@gnu.org; Thu, 21 Apr 2011 17:13:07 -0400
Received: from eggs.gnu.org ([140.186.70.92]:39907)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QD1BQ-00070x-Rh
	for coreutils@gnu.org; Thu, 21 Apr 2011 17:13:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QD1BO-00053V-Ud
	for coreutils@gnu.org; Thu, 21 Apr 2011 17:13:04 -0400
Received: from ud10.udmedia.de ([194.117.254.50]:59964
	helo=mail.ud10.udmedia.de) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <markus@trippelsdorf.de>) id 1QD1BO-00052v-I2
	for coreutils@gnu.org; Thu, 21 Apr 2011 17:13:02 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=simple; d=mail.ud10.udmedia.de; h=
	date:from:to:cc:subject:message-id:references:mime-version:
	content-type:in-reply-to; q=dns/txt; s=beta; bh=xu7cvn1L+9p0gjYv
	WcFLtX7MdpkQoAywXNy0JJu2coE=; b=vNRPpV97umCD9kMhvGxFTY4eQqkLqZ7c
	wmg9VtoWQs3hPij5D9j1CKP2nxjuc8PSNU87xmF6he4oGZPhE0XV/QEQP1crnCGm
	433udqabcKJidX3YSe0dVQvI2JZoQjzCLTc4rGSvXf4TACPwSUQzkbx/Hhz/hoK0
	2ipSSm8sIKI=
Received: (qmail 18974 invoked from network); 21 Apr 2011 23:12:59 +0200
Received: from unknown (HELO x4.trippels.de) (ud10?360p3@91.66.182.48)
	by mail.ud10.udmedia.de with ESMTPSA (DHE-RSA-AES256-SHA encrypted,
	authenticated); 21 Apr 2011 23:12:59 +0200
Date: Thu, 21 Apr 2011 23:12:59 +0200
From: Markus Trippelsdorf <markus@trippelsdorf.de>
To: Jim Meyering <jim@meyering.net>
Subject: Re: new snapshot available: coreutils-8.11.9-8c0b1d
Message-ID: <20110421211259.GA1627@x4.trippels.de>
References: <87liz3qscj.fsf@rho.meyering.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87liz3qscj.fsf@rho.meyering.net>
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 194.117.254.50
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Apr 2011 21:13:05 -0000

On 2011.04.21 at 22:09 +0200, Jim Meyering wrote:
> Considering the potential for data loss when using coreutils-8.11 on
> ext4 and xfs when using e.g., 2.6.39-rc3[*], here's a snapshot that will
> soon become coreutils-8.12.  Please give it a try.

Looks good. The "files full of zeros" problem is gone and all 367 tests
pass.

The first (tiny) steps for SEEK_HOLE and SEEK_DATA support were just
posted: http://thread.gmane.org/gmane.linux.kernel/1129696

-- 
Markus


From MAILER-DAEMON Thu Apr 21 18:33:22 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QD2R8-0005T1-Ne
	for mharc-coreutils@gnu.org; Thu, 21 Apr 2011 18:33:22 -0400
Received: from eggs.gnu.org ([140.186.70.92]:33766)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QD2R6-0005Sl-VH
	for coreutils@gnu.org; Thu, 21 Apr 2011 18:33:21 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QD2R5-0002uD-PZ
	for coreutils@gnu.org; Thu, 21 Apr 2011 18:33:20 -0400
Received: from mx1.redhat.com ([209.132.183.28]:42466)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QD2R5-0002u4-H8
	for coreutils@gnu.org; Thu, 21 Apr 2011 18:33:19 -0400
Received: from int-mx10.intmail.prod.int.phx2.redhat.com
	(int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3LMXHS1019064
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Thu, 21 Apr 2011 18:33:17 -0400
Received: from [10.3.113.38] (ovpn-113-38.phx2.redhat.com [10.3.113.38])
	by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p3LMXG1j010365; Thu, 21 Apr 2011 18:33:16 -0400
Message-ID: <4DB0B0AB.7070406@redhat.com>
Date: Thu, 21 Apr 2011 16:33:15 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: new snapshot available: coreutils-8.11.9-8c0b1d
References: <87liz3qscj.fsf@rho.meyering.net>
	<20110421211259.GA1627@x4.trippels.de>
In-Reply-To: <20110421211259.GA1627@x4.trippels.de>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enig3FFF7D7B6D2CD884E4BA1907"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 21 Apr 2011 22:33:21 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig3FFF7D7B6D2CD884E4BA1907
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 04/21/2011 03:12 PM, Markus Trippelsdorf wrote:
> On 2011.04.21 at 22:09 +0200, Jim Meyering wrote:
>> Considering the potential for data loss when using coreutils-8.11 on
>> ext4 and xfs when using e.g., 2.6.39-rc3[*], here's a snapshot that wi=
ll
>> soon become coreutils-8.12.  Please give it a try.
>=20
> Looks good. The "files full of zeros" problem is gone and all 367 tests=

> pass.
>=20
> The first (tiny) steps for SEEK_HOLE and SEEK_DATA support were just
> posted: http://thread.gmane.org/gmane.linux.kernel/1129696

Thanks for that pointer.

Here's hoping that the kernel implements the same documented semantics
as Solaris:

SEEK_HOLE - returns the offset of the next hole without repositioning
the file (all files have a virtual hole after the last data byte, so the
only time this fails is with ENXIO if you seek relative to an offset
beyond the end of the file).

SEEK_DATA - move the pointer to the start of the next non-hole, or fail
with ENXIO if there are no data sections after the requested position.

Typical usage would then be a loop of:

lseek(SEEK_HOLE) to learn where the next hole starts, and process all
bytes from current position to that point
lseek(SEEK_DATA) to skip past the hole that you just arrived at, where
ENXIO tells you you reached the end of the file.

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enig3FFF7D7B6D2CD884E4BA1907
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNsLCrAAoJEKeha0olJ0Nqie4H/108U3pAjfQbZcZgO5/SFoi6
FPlQjZkTbMmXawLrmNRR/0H883TcAV7d2iL2ZVHOtdM4DfFR1Yo3em41yQBRZj+g
VahXoD+ZAdqgJuM0lJPXby6gkjcJm70o6bkD74+l9gdmYr7p1NCSbQchSzqcZJ2s
XkAI0ebAsLbU34p0KdZ7OutRovkuPgOqyCGCE43NM0SADQyl6k/Rln2NuHxzLLaX
VcA+xlSPc/HTAO/p57XZY5m1a0nxJckl65UoFTggT0UaFhlCh3wADh1ChKg/FA+G
W6jyWo6gpbQnLl09CN20QHHv3umvXfamO/m3VNMDOeZBoLxBXep2PAYpyi5LUtY=
=70YO
-----END PGP SIGNATURE-----

--------------enig3FFF7D7B6D2CD884E4BA1907--


From MAILER-DAEMON Fri Apr 22 09:06:41 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QDG4H-0007ke-3k
	for mharc-coreutils@gnu.org; Fri, 22 Apr 2011 09:06:41 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47754)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QDG4F-0007kV-He
	for coreutils@gnu.org; Fri, 22 Apr 2011 09:06:40 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QDG4E-0001rL-F3
	for coreutils@gnu.org; Fri, 22 Apr 2011 09:06:39 -0400
Received: from mx1.redhat.com ([209.132.183.28]:50244)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QDG4E-0001rG-38
	for coreutils@gnu.org; Fri, 22 Apr 2011 09:06:38 -0400
Received: from int-mx02.intmail.prod.int.phx2.redhat.com
	(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3MD6alF015208
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Fri, 22 Apr 2011 09:06:36 -0400
Received: from [10.3.113.38] (ovpn-113-38.phx2.redhat.com [10.3.113.38])
	by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP
	id p3MD6Z1P014122; Fri, 22 Apr 2011 09:06:35 -0400
Message-ID: <4DB17D5B.2090303@redhat.com>
Date: Fri, 22 Apr 2011 07:06:35 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: Markus Trippelsdorf <markus@trippelsdorf.de>
Subject: Re: [PATCH 1/2] fs: add SEEK_HOLE and SEEK_DATA flags
References: <1303414954-3315-1-git-send-email-josef@redhat.com>
	<loom.20110422T001650-760@post.gmane.org>
	<BANLkTiknb+hzFAjpwESwMcqMVtkFc0HFQQ@mail.gmail.com>
	<20110422045054.GB17795@infradead.org>
	<20110422112852.GB1627@x4.trippels.de>
In-Reply-To: <20110422112852.GB1627@x4.trippels.de>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enig3440113740B5193E68F3D547"
X-Scanned-By: MIMEDefang 2.67 on 10.5.11.12
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: Josef Bacik <josef@toxicpanda.com>, linux-kernel@vger.kernel.org,
	Christoph Hellwig <hch@infradead.org>,
	Coreutils <coreutils@gnu.org>, linux-fsdevel@vger.kernel.org,
	linux-btrfs@vger.kernel.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 22 Apr 2011 13:06:40 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig3440113740B5193E68F3D547
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 04/22/2011 05:28 AM, Markus Trippelsdorf wrote:
>> I would be surprised if the bug is around for such a long time, but
>> otherwise I concur.
>=20
> It's a bug. Let me quote what Jeff Bonwick wrote on his blog:
>=20
> =BBI'm not sure where you got the impression that either SEEK_HOLE or
> SEEK_DATA doesn't set the file pointer. They do. (If they didn't, that
> would be weird, and we'd call it out explicitly.)=AB
>=20
> http://blogs.sun.com/bonwick/entry/seek_hole_and_seek_data

I've created a request to standardize SEEK_HOLE and SEEK_DATA in the
next revision of POSIX; comments are welcome to make sure that everyone
is happy with wording:
http://austingroupbugs.net/view.php?id=3D415

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enig3440113740B5193E68F3D547
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNsX1bAAoJEKeha0olJ0NqwNMH/RFHCct8i8aE8TKB3gvFAfMB
Mf+TiHebzAitBX6/FfdoWb1+MPPG4kNVXd8sjOyjHG/AfeqfCtUHr6gJ/kRbS17p
b8uuEaH4OO/see4Q4ghuSo/dELCthm1IVIwHg7TBF5LBsuFWI561GIgM+bRL0iPU
bt0SFrddLWEjZQaWxG/myZeQwzV1BHDr8p4S1K5TkKfw6D3K5XHW9hISWo/drBuW
haOLA7y26bupunEZrTuK9igS9vgcMrpnYui/Ly+/87c0Dn+eE04FtxHArsV4v66g
16o7NsBl8qIdXhGJiGOBg8CJouRDfOcPlLakbQXnAFPrjsoJeKRrd2iO7HlANk0=
=iKrP
-----END PGP SIGNATURE-----

--------------enig3440113740B5193E68F3D547--


From MAILER-DAEMON Sun Apr 24 05:31:00 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QDvee-0000HZ-3T
	for mharc-coreutils@gnu.org; Sun, 24 Apr 2011 05:31:00 -0400
Received: from eggs.gnu.org ([140.186.70.92]:41271)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QDveb-0000GH-RO
	for coreutils@gnu.org; Sun, 24 Apr 2011 05:30:58 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QDvea-0007Zs-Qp
	for coreutils@gnu.org; Sun, 24 Apr 2011 05:30:57 -0400
Received: from rcsinet10.oracle.com ([148.87.113.121]:45093)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QDvea-0007ZN-La
	for coreutils@gnu.org; Sun, 24 Apr 2011 05:30:56 -0400
Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227])
	by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id
	p3O9Uo1e031273
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Sun, 24 Apr 2011 09:30:51 GMT
Received: from acsmt357.oracle.com (acsmt357.oracle.com [141.146.40.157])
	by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id
	p3O9UmKE022487
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Sun, 24 Apr 2011 09:30:48 GMT
Received: from abhmt018.oracle.com (abhmt018.oracle.com [141.146.116.27])
	by acsmt357.oracle.com (8.12.11.20060308/8.12.11) with ESMTP id
	p3O9UlEk029944; Sun, 24 Apr 2011 04:30:47 -0500
Received: from [123.119.100.37] (/123.119.100.37)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Sun, 24 Apr 2011 02:30:47 -0700
Message-ID: <4DB3EDBF.8090703@oracle.com>
Date: Sun, 24 Apr 2011 17:30:39 +0800
From: "jeff.liu" <jeff.liu@oracle.com>
User-Agent: Thunderbird 2.0.0.14 (X11/20080505)
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: [coreutils] cp/reflink-perm fails on btrfs (and probably on ocfs2,
	too)
References: <87sjztcmxd.fsf@meyering.net>	<20101026103539.GA19482@sushi.unix-ag.uni-kl.de>	<87mxq1cjcr.fsf@meyering.net>
	<4CC6C02A.6020001@draigBrady.com>	<4CC6CD8D.2050708@draigBrady.com>
	<878w1lcb58.fsf@meyering.net>	<4CC6E89B.8040709@draigBrady.com>
	<4CC71813.2050307@draigBrady.com> <87zku0akh4.fsf@meyering.net>
In-Reply-To: <87zku0akh4.fsf@meyering.net>
Content-Type: text/plain; charset=utf-8
X-Source-IP: acsinet15.oracle.com [141.146.126.227]
X-Auth-Type: Internal IP
X-CT-RefId: str=0001.0A090207.4DB3EDCB.00AA:SCFMA922111,ss=1,fgs=0
Content-Transfer-Encoding: quoted-printable
X-MIME-Autoconverted: from 8bit to quoted-printable by rcsinet10.oracle.com id
	p3O9Uo1e031273
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 148.87.113.121
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 24 Apr 2011 09:30:58 -0000

Hi Jim and P=C3=A1draig,

Is it better to adjust the error string from "failed to clone dst_name" t=
o "failed to clone
src_name" when clone operation fails? or maybe it would confuse the user =
IMHO.

If it make sense, one line patch fix it as below:

>From ffb747674f1318a1fec2c2db335d5f29c310fada Mon Sep 17 00:00:00 2001
From: Jie Liu <jeff.liu@oracle.com>
Date: Sun, 24 Apr 2011 17:17:40 +0800
Subject: [PATCH 1/1] copy: adjust error string from dst_name to src_name =
if clone failed

* src/copy.c: Print "faild to clone src_name" as the error string if
  Btrfs clone operation fails.

Signed-off-by: Jie Liu <jeff.liu@oracle.com>
---
 src/copy.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index 6edf52e..1ac1f95 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -974,7 +974,7 @@ copy_reg (char const *src_name, char const *dst_name,
         {
           if (!clone_ok)
             {
-              error (0, errno, _("failed to clone %s"), quote (dst_name)=
);
+              error (0, errno, _("failed to clone %s"), quote (src_name)=
);
               return_val =3D false;
               goto close_src_and_dst_desc;
             }
--=20
1.5.4.3


Thanks,
-Jeff


Jim Meyering wrote:
> P=C3=A1draig Brady wrote:
>> Actually I remember now why we didn't want --reflink
>> mutually exclusive with other options.
>> It's so it can be used as configuration to change
>> the data copy method employed. I.E. it operates
>> on much the same level as --sparse in that the
>> results are mostly transparent to the user.
>> I.E. cp could be aliased like this to minimize
>> the amount of space that cp uses:
>>
>> alias cp=3D'cp --reflink=3Dauto --sparse=3Dalways'
>>
>> So therefore I've changed the patch as per the attached.
>> I'll apply in a few hours.
>>
>> >From 3dc499ba27b1b0b6365179cb6d7bc9e5f6456600 Mon Sep 17 00:00:00 200=
1
>> Subject: [PATCH] cp: make --attributes-only override --reflink complet=
ely
>>
>> * doc/coreutils.texi (cp invocation): Change the description slightly
>> so as users might not immediately discount using this option.
>> Mention that --reflink is overridden by the other linking options and
>> --attributes-only, and give an example where this might be useful.
>> * src/copy.c (copy_internal): Bypass the reflink if
>> --attributes-only is specifed.
>> * tests/cp/reflink-perm: Ensure both --reflink modes are
>> overridden by --attributes-only.
>> * NEWS: Mention the change in behavior.
>=20
> Glad you caught that.  Thanks!
>=20



From MAILER-DAEMON Sun Apr 24 10:44:35 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QE0Y7-0008EV-Nz
	for mharc-coreutils@gnu.org; Sun, 24 Apr 2011 10:44:35 -0400
Received: from eggs.gnu.org ([140.186.70.92]:59642)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QE0Y4-0008EP-Rs
	for coreutils@gnu.org; Sun, 24 Apr 2011 10:44:33 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QE0Y3-0007pK-Rf
	for coreutils@gnu.org; Sun, 24 Apr 2011 10:44:32 -0400
Received: from mx.meyering.net ([82.230.74.64]:41411)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QE0Y3-0007pA-L7
	for coreutils@gnu.org; Sun, 24 Apr 2011 10:44:31 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id A5665600B4; Sun, 24 Apr 2011 16:44:28 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: "jeff.liu" <jeff.liu@oracle.com>
Subject: Re: [coreutils] cp/reflink-perm fails on btrfs (and probably on ocfs2,
	too)
In-Reply-To: <4DB3EDBF.8090703@oracle.com> (jeff liu's message of "Sun, 24 Apr
	2011 17:30:39 +0800")
References: <87sjztcmxd.fsf@meyering.net>
	<20101026103539.GA19482@sushi.unix-ag.uni-kl.de>
	<87mxq1cjcr.fsf@meyering.net> <4CC6C02A.6020001@draigBrady.com>
	<4CC6CD8D.2050708@draigBrady.com> <878w1lcb58.fsf@meyering.net>
	<4CC6E89B.8040709@draigBrady.com> <4CC71813.2050307@draigBrady.com>
	<87zku0akh4.fsf@meyering.net> <4DB3EDBF.8090703@oracle.com>
Date: Sun, 24 Apr 2011 16:44:28 +0200
Message-ID: <87y62ziu9f.fsf@rho.meyering.net>
Lines: 44
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sun, 24 Apr 2011 14:44:33 -0000

jeff.liu wrote:
> Is it better to adjust the error string from "failed to clone dst_name" to "failed to clone
> src_name" when clone operation fails? or maybe it would confuse the user IMHO.

Hi Jeff,

Thanks for the patch.  Mentioning only one of source and destination,
I prefer to list the destination name, since it is more often relevant
than the source.  Sometimes it's useful to know both (e.g., EXDEV) so
I propose this instead:

I changed the log message, too, but left your name as author,
so I'll wait for an ACK from you before pushing.


>From d5bd22605a9ff8289d6c02cd2d3e81d460bae873 Mon Sep 17 00:00:00 2001
From: Jeff Liu <jeff.liu@oracle.com>
Date: Sun, 24 Apr 2011 16:25:31 +0200
Subject: [PATCH] copy: include both src and dest names in clone failure
 diagnostic

* src/copy.c (copy_reg): Upon btrfs clone failure, print not just
the destination file name, but also the source file name.
That may be useful upon failure of a cross-device clone attempt.
---
 src/copy.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/src/copy.c b/src/copy.c
index 6edf52e..65566a0 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -974,7 +974,8 @@ copy_reg (char const *src_name, char const *dst_name,
         {
           if (!clone_ok)
             {
-              error (0, errno, _("failed to clone %s"), quote (dst_name));
+              error (0, errno, _("failed to clone %s from %s"),
+                     quote (dst_name), quote (src_name));
               return_val = false;
               goto close_src_and_dst_desc;
             }
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Sun Apr 24 23:25:00 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QECQ0-0004F1-TG
	for mharc-coreutils@gnu.org; Sun, 24 Apr 2011 23:25:00 -0400
Received: from eggs.gnu.org ([140.186.70.92]:34758)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QECPy-0004Ea-Rl
	for coreutils@gnu.org; Sun, 24 Apr 2011 23:24:59 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QECPx-0001Mm-PK
	for coreutils@gnu.org; Sun, 24 Apr 2011 23:24:58 -0400
Received: from rcsinet10.oracle.com ([148.87.113.121]:22072)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jeff.liu@oracle.com>) id 1QECPx-0001Mf-Fw
	for coreutils@gnu.org; Sun, 24 Apr 2011 23:24:57 -0400
Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117])
	by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id
	p3P3OrAZ020410
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Mon, 25 Apr 2011 03:24:54 GMT
Received: from acsmt358.oracle.com (acsmt358.oracle.com [141.146.40.158])
	by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id
	p3P3OqHG024985
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO);
	Mon, 25 Apr 2011 03:24:52 GMT
Received: from abhmt009.oracle.com (abhmt009.oracle.com [141.146.116.18])
	by acsmt358.oracle.com (8.12.11.20060308/8.12.11) with ESMTP id
	p3P3Op8E019406; Sun, 24 Apr 2011 22:24:52 -0500
Received: from [10.182.121.39] (/10.182.121.39)
	by default (Oracle Beehive Gateway v4.0)
	with ESMTP ; Sun, 24 Apr 2011 20:24:51 -0700
Message-ID: <4DB4E97F.50108@oracle.com>
Date: Mon, 25 Apr 2011 11:24:47 +0800
From: "jeff.liu" <jeff.liu@oracle.com>
User-Agent: Thunderbird 2.0.0.14 (X11/20080505)
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: [coreutils] cp/reflink-perm fails on btrfs (and probably on ocfs2,
	too)
References: <87sjztcmxd.fsf@meyering.net>	<20101026103539.GA19482@sushi.unix-ag.uni-kl.de>	<87mxq1cjcr.fsf@meyering.net>
	<4CC6C02A.6020001@draigBrady.com>	<4CC6CD8D.2050708@draigBrady.com>
	<878w1lcb58.fsf@meyering.net>	<4CC6E89B.8040709@draigBrady.com>
	<4CC71813.2050307@draigBrady.com>	<87zku0akh4.fsf@meyering.net>
	<4DB3EDBF.8090703@oracle.com> <87y62ziu9f.fsf@rho.meyering.net>
In-Reply-To: <87y62ziu9f.fsf@rho.meyering.net>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-Source-IP: rcsinet15.oracle.com [148.87.113.117]
X-Auth-Type: Internal IP
X-CT-RefId: str=0001.0A090208.4DB4E986.0058:SCFMA4539811,ss=1,fgs=0
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 148.87.113.121
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 03:24:59 -0000

Hi Jim,

Jim Meyering wrote:
> jeff.liu wrote:
>> Is it better to adjust the error string from "failed to clone dst_name" to "failed to clone
>> src_name" when clone operation fails? or maybe it would confuse the user IMHO.
> 
> Hi Jeff,
> 
> Thanks for the patch.  Mentioning only one of source and destination,
> I prefer to list the destination name, since it is more often relevant
> than the source.  Sometimes it's useful to know both (e.g., EXDEV) so
> I propose this instead:
Exactly, it's more meaningful to list the destination file too.

> 
> I changed the log message, too, but left your name as author,
thank you.

> so I'll wait for an ACK from you before pushing.
Acked.


Regards,
-Jeff
> 
> 
> From d5bd22605a9ff8289d6c02cd2d3e81d460bae873 Mon Sep 17 00:00:00 2001
> From: Jeff Liu <jeff.liu@oracle.com>
> Date: Sun, 24 Apr 2011 16:25:31 +0200
> Subject: [PATCH] copy: include both src and dest names in clone failure
>  diagnostic
> 
> * src/copy.c (copy_reg): Upon btrfs clone failure, print not just
> the destination file name, but also the source file name.
> That may be useful upon failure of a cross-device clone attempt.
> ---
>  src/copy.c |    3 ++-
>  1 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/src/copy.c b/src/copy.c
> index 6edf52e..65566a0 100644
> --- a/src/copy.c
> +++ b/src/copy.c
> @@ -974,7 +974,8 @@ copy_reg (char const *src_name, char const *dst_name,
>          {
>            if (!clone_ok)
>              {
> -              error (0, errno, _("failed to clone %s"), quote (dst_name));
> +              error (0, errno, _("failed to clone %s from %s"),
> +                     quote (dst_name), quote (src_name));
>                return_val = false;
>                goto close_src_and_dst_desc;
>              }
> --
> 1.7.5.rc3.316.gd1ff9



From MAILER-DAEMON Mon Apr 25 04:58:01 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEHcH-0001LZ-6Z
	for mharc-coreutils@gnu.org; Mon, 25 Apr 2011 04:58:01 -0400
Received: from eggs.gnu.org ([140.186.70.92]:33230)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHcF-0001LK-S7
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:58:00 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHcF-0007zf-4S
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:57:59 -0400
Received: from mx.meyering.net ([82.230.74.64]:34176)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHcE-0007zZ-QW
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:57:59 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id EE6F760051; Mon, 25 Apr 2011 10:57:57 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] maint: explicitly list full-read and full-write module names
Date: Mon, 25 Apr 2011 10:57:57 +0200
Message-ID: <87r58qhfmy.fsf@rho.meyering.net>
Lines: 30
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 08:58:00 -0000

FYI, preparing to update to newer gnulib (though not for 8.12),

>From 925182174cd6c429f01350635af45c87bdfcfa68 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Sat, 23 Apr 2011 23:47:37 +0200
Subject: [PATCH] maint: explicitly list full-read and full-write module names

* bootstrap.conf (gnulib_modules): Include full-read and full-write
explicitly.  Before, we'd get them via safe-read, but with newer
gnulib, that is no longer enough: link failure due to undefined
references to full_write.
---
 bootstrap.conf |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/bootstrap.conf b/bootstrap.conf
index 618fa53..5633e13 100644
--- a/bootstrap.conf
+++ b/bootstrap.conf
@@ -91,6 +91,8 @@ gnulib_modules="
   ftello
   ftoastr
   fts
+  full-read
+  full-write
   getgroups
   gethrxtime
   getline
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Mon Apr 25 04:59:03 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEHdH-0001g9-Ux
	for mharc-coreutils@gnu.org; Mon, 25 Apr 2011 04:59:03 -0400
Received: from eggs.gnu.org ([140.186.70.92]:33571)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHdF-0001g2-Io
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:59:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHdE-0008D4-LV
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:59:01 -0400
Received: from mx.meyering.net ([82.230.74.64]:34233)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHdE-0008Cs-8N
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:59:00 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 8E6F3601A4; Mon, 25 Apr 2011 10:58:59 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] doc: tail/inotify does use --sleep-interval=S, with --pid=P
Date: Mon, 25 Apr 2011 10:58:59 +0200
Message-ID: <87liyyhfl8.fsf@rho.meyering.net>
Lines: 63
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 08:59:02 -0000

In fixing the recent tail --follow=name bug, I noticed that
--sleep-interval=S *is* used in inotify mode, so adjusted the
documentation:

>From 249d9a3ca429e508958e87c636c1b6a4a8d75f12 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Sat, 23 Apr 2011 16:08:08 +0200
Subject: [PATCH] doc: tail/inotify does use --sleep-interval=S, with --pid=P

* doc/coreutils.texi (tail invocation): Mention it.
* src/tail.c (usage): Likewise.
(tail_forever_inotify): Clarify comment.
---
 doc/coreutils.texi |    5 ++++-
 src/tail.c         |   10 ++++++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index b7ccf38..d2377f4 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -2852,7 +2852,10 @@ tail invocation
 Historical implementations of @command{tail} have required that
 @var{number} be an integer.  However, GNU @command{tail} accepts
 an arbitrary floating point number.  @xref{Floating point}.
-When @command{tail} uses inotify, this polling-related option is ignored.
+When @command{tail} uses inotify, this polling-related option
+is usually ignored.  However, if you also specify @option{--pid=@var{p}},
+@command{tail} checks whether process @var{p} is alive at least
+every @var{number} seconds.

 @itemx --pid=@var{pid}
 @opindex --pid
diff --git a/src/tail.c b/src/tail.c
index bcd2d99..88fbd10 100644
--- a/src/tail.c
+++ b/src/tail.c
@@ -302,7 +302,8 @@ Mandatory arguments to long options are mandatory for short options too.\n\
      fputs (_("\
   -s, --sleep-interval=N   with -f, sleep for approximately N seconds\n\
                              (default 1.0) between iterations.\n\
-                             With inotify, this option is rarely useful.\n\
+                             With inotify and --pid=P, check process P at\n\
+                             least once every N seconds.\n\
   -v, --verbose            always output headers giving file names\n\
 "), stdout);
      fputs (HELP_OPTION_DESCRIPTION, stdout);
@@ -1422,9 +1423,10 @@ tail_forever_inotify (int wd, struct File_spec *f, size_t n_files,
   evlen += sizeof (struct inotify_event) + 1;
   evbuf = xmalloc (evlen);

-  /* Wait for inotify events and handle them.  Events on directories make sure
-     that watched files can be re-added when -F is used.
-     This loop sleeps on the `safe_read' call until a new event is notified.  */
+  /* Wait for inotify events and handle them.  Events on directories
+     ensure that watched files can be re-added when following by name.
+     This loop blocks on the `safe_read' call until a new event is notified.
+     But when --pid=P is specified, tail usually waits via the select.  */
   while (1)
     {
       struct File_spec *fspec;
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Mon Apr 25 04:59:49 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEHe1-0001ll-EH
	for mharc-coreutils@gnu.org; Mon, 25 Apr 2011 04:59:49 -0400
Received: from eggs.gnu.org ([140.186.70.92]:56373)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHe0-0001lf-20
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:59:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHdz-0008K2-3p
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:59:48 -0400
Received: from mx.meyering.net ([82.230.74.64]:34373)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEHdy-0008Ju-SU
	for coreutils@gnu.org; Mon, 25 Apr 2011 04:59:47 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id DFEF8601A4; Mon, 25 Apr 2011 10:59:45 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] tests: tail-2/pipe-f2: avoid false-positive failure
Date: Mon, 25 Apr 2011 10:59:45 +0200
Message-ID: <87fwp6hfjy.fsf@rho.meyering.net>
Lines: 30
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 08:59:48 -0000

FYI, noticed a rare false-positive failure.
Fixed it like this:

>From 0e6c96e68cc38954c6d77e2c9361708a383b2cea Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Sat, 23 Apr 2011 23:54:11 +0200
Subject: [PATCH] tests: tail-2/pipe-f2: avoid false-positive failure

Otherwise, this would fail (albeit rarely) on a "make -j24 check" run.
* tests/tail-2/pipe-f2: Increase timeout from 1 second to 10,
to avoid false positive failure.
---
 tests/tail-2/pipe-f2 |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tests/tail-2/pipe-f2 b/tests/tail-2/pipe-f2
index c258fee..3422c23 100755
--- a/tests/tail-2/pipe-f2
+++ b/tests/tail-2/pipe-f2
@@ -24,7 +24,7 @@ mkfifo_or_skip_ fifo
 echo 1 > fifo &
 echo 1 > exp || framework_failure

-timeout 1 tail -f fifo > out
+timeout 10 tail -f fifo > out
 test $? = 124 || fail=1

 compare out exp || fail=1
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Mon Apr 25 08:44:03 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEL91-00008T-DH
	for mharc-coreutils@gnu.org; Mon, 25 Apr 2011 08:44:03 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55291)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEL8z-00007t-6G
	for coreutils@gnu.org; Mon, 25 Apr 2011 08:44:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEL8y-0003rl-8y
	for coreutils@gnu.org; Mon, 25 Apr 2011 08:44:01 -0400
Received: from mx.meyering.net ([82.230.74.64]:44852)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEL8y-0003rd-2i
	for coreutils@gnu.org; Mon, 25 Apr 2011 08:44:00 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 14BB360051; Mon, 25 Apr 2011 14:43:57 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] stat: recognize V9FS and ECRYPTFS file systems
Date: Mon, 25 Apr 2011 14:43:56 +0200
Message-ID: <87k4eifqlv.fsf@rho.meyering.net>
Lines: 44
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 12:44:02 -0000

I ran "make -C src fs-magic-compare" and it showed that there
are two new file system types:

    # define S_MAGIC_ECRYPTFS 0xF15F
    # define S_MAGIC_V9FS 0x01021997

So I've added them with names "v9fs" and "ecryptfs".
If some other name(s) would be appropriate, please let me know.


>From bd6cea345ceb25e10b4ecd7245d8c5725e126e97 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Mon, 25 Apr 2011 14:40:45 +0200
Subject: [PATCH] stat: recognize V9FS and ECRYPTFS file systems

* src/stat.c (human_fstype): Add magic numbers for V9FS and ECRYPTFS.
---
 src/stat.c |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/src/stat.c b/src/stat.c
index f26dced..4e6e1c3 100644
--- a/src/stat.c
+++ b/src/stat.c
@@ -268,6 +268,8 @@ human_fstype (STRUCT_STATVFS const *statfsbuf)
       return "devfs";
     case S_MAGIC_DEVPTS: /* 0x1CD1 */
       return "devpts";
+    case S_MAGIC_ECRYPTFS: /* 0xF15F */
+      return "ecryptfs";
     case S_MAGIC_EFS: /* 0x00414A53 */
       return "efs";
     case S_MAGIC_EXT: /* 0x137D */
@@ -374,6 +376,8 @@ human_fstype (STRUCT_STATVFS const *statfsbuf)
       return "ufs";
     case S_MAGIC_USBDEVFS: /* 0x9FA2 */
       return "usbdevfs";
+    case S_MAGIC_V9FS: /* 0x01021997 */
+      return "v9fs";
     case S_MAGIC_VXFS: /* 0xA501FCF5 */
       return "vxfs";
     case S_MAGIC_XENFS: /* 0xABBA1974 */
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Mon Apr 25 11:49:04 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEO24-0001af-Kp
	for mharc-coreutils@gnu.org; Mon, 25 Apr 2011 11:49:04 -0400
Received: from eggs.gnu.org ([140.186.70.92]:53710)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QEO22-0001ZP-Kx
	for coreutils@gnu.org; Mon, 25 Apr 2011 11:49:03 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QEO21-0002Tt-CH
	for coreutils@gnu.org; Mon, 25 Apr 2011 11:49:02 -0400
Received: from mx1.redhat.com ([209.132.183.28]:62457)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QEO20-0002Tp-QL
	for coreutils@gnu.org; Mon, 25 Apr 2011 11:49:01 -0400
Received: from int-mx09.intmail.prod.int.phx2.redhat.com
	(int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22])
	by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p3PFms2k019256
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK);
	Mon, 25 Apr 2011 11:48:54 -0400
Received: from [10.3.113.102] (ovpn-113-102.phx2.redhat.com [10.3.113.102])
	by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP
	id p3PFmqwQ024460; Mon, 25 Apr 2011 11:48:52 -0400
Message-ID: <4DB597E3.7090600@redhat.com>
Date: Mon, 25 Apr 2011 09:48:51 -0600
From: Eric Blake <eblake@redhat.com>
Organization: Red Hat
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110307 Fedora/3.1.9-0.39.b3pre.fc14
	Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.9
MIME-Version: 1.0
To: Nick Bowler <nbowler@elliptictech.com>
Subject: Re: [PATCH 1/2] fs: add SEEK_HOLE and SEEK_DATA flags
References: <1303414954-3315-1-git-send-email-josef@redhat.com>
	<loom.20110422T001650-760@post.gmane.org>
	<BANLkTiknb+hzFAjpwESwMcqMVtkFc0HFQQ@mail.gmail.com>
	<20110422045054.GB17795@infradead.org>
	<20110422112852.GB1627@x4.trippels.de>
	<4DB17D5B.2090303@redhat.com>
	<20110425150227.GA10653@elliptictech.com>
In-Reply-To: <20110425150227.GA10653@elliptictech.com>
X-Enigmail-Version: 1.1.2
OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature";
	boundary="------------enig8B7BBB404659570B8A8ECBF7"
X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.132.183.28
Cc: Josef Bacik <josef@toxicpanda.com>, linux-kernel@vger.kernel.org,
	Christoph Hellwig <hch@infradead.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>,
	linux-fsdevel@vger.kernel.org, linux-btrfs@vger.kernel.org,
	Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 15:49:03 -0000

This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig8B7BBB404659570B8A8ECBF7
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On 04/25/2011 09:02 AM, Nick Bowler wrote:

Hi Nick,
> * File A (sparse file created by lseek/write beyond end-of-file):
>=20
>     data | hole 0 | data || hole 1 (virtual)
>=20
> * File B (sparse file created by truncate beyond end-of-file):
>=20
>     data | hole 0 || hole 1 (virtual)
>=20
> Excluding the error description, the term "the last hole" is used in
> two places in your proposal:
>=20
>   * (for SEEK_HOLE): if offset falls within "the last hole", then the
>     file offset may be set to the file size instead.
>=20
>   * (for SEEK_DATA): it shall be an error ... if offset falls within th=
e
>     last hole.
>=20
> I imagine that both of these conditions are intended to address the
> case where the offset falls within hole 0 in File B, that is, when
> there is no non-hole data beyond the specified offset but the offset
> is nevertheless less than the file size.

Correct.

>  However, this looks (to me)
> like the penultimate hole in the file, not the last hole.  Furthermore,=

> these conditions are presumably *not* intended to apply to the
> penultimate hole in File A, which has data after it.

Good catch.

>=20
> I think my confusion can be avoided by talking about the last non-hole
> data byte in the file (which is unambigious), instead of by talking
> about the last hole.  For instance, the SEEK_HOLE/SEEK_DATA description=
s
> could be written as follows:
>=20
>   If whence is SEEK_HOLE, the file offset shall be set to the smallest
>   location of a byte within a hole and not less than offset, except tha=
t
>   if offset falls beyond the last byte not within a hole, then the file=

>   offset may be set to the file size instead.  It shall be an error if
>   offset is greater or equal to the size of the file.
>=20
>   If whence is SEEK_DATA, the file offset shall be set to the smallest
>   location of a byte not within a hole and not less than offset.  It sh=
all
>   be an error if no such byte exists.
>=20
> plus a corresponding update to the ENXIO description:
>=20
>   ... or the whence argument is SEEK_DATA and the offset falls beyond
>   the last byte not within a hole.

I've added your improved wording as a comment at
http://austingroupbugs.net/view.php?id=3D415

--=20
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


--------------enig8B7BBB404659570B8A8ECBF7
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Public key at http://people.redhat.com/eblake/eblake.gpg
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iQEcBAEBCAAGBQJNtZfjAAoJEKeha0olJ0NqQWgIAIRdXY+1Os6dnyI7OfkncGmz
/4+2wNGUBmXtEdOn52oCOfe3gp0zP6s6SbIWAz80P72pcTMzxDIjFjc3kmHRY+64
LHFxRS3pIDRDfLmqyI7w1H89qfbRi8Gem9jAoF6BPeuhcfu0MQ27+9K4byXxrhTU
lzFEGTHHUH6QC4mS+qS9YIgPUZKnipOIFRJPDDzeJKFuElDpGvUFF7TyyR9eGKXb
bh0sZSXMfhyd5oVp08pk6OBJBfMRF1Z3ysArWOng5QoJ7cttYcjc7u+D98Bi5GXS
ijofbM6/qDFt/W2NJ+W22U/7TIvKz9EkiAir3MGFYcXqWN/eGmOECf2p2biQhOs=
=qasl
-----END PGP SIGNATURE-----

--------------enig8B7BBB404659570B8A8ECBF7--


From MAILER-DAEMON Mon Apr 25 11:59:44 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEOCN-0002Te-WC
	for mharc-coreutils@gnu.org; Mon, 25 Apr 2011 11:59:44 -0400
Received: from eggs.gnu.org ([140.186.70.92]:44473)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <nbowler@elliptictech.com>) id 1QENJO-0005ao-HV
	for coreutils@gnu.org; Mon, 25 Apr 2011 11:02:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <nbowler@elliptictech.com>) id 1QENJN-00034i-Df
	for coreutils@gnu.org; Mon, 25 Apr 2011 11:02:54 -0400
Received: from mail.elliptictech.com ([209.217.122.41]:47304
	helo=mail.ellipticsemi.com) by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <nbowler@elliptictech.com>) id 1QENJN-00034E-9Z
	for coreutils@gnu.org; Mon, 25 Apr 2011 11:02:53 -0400
Received: from nbowler by mail.ellipticsemi.com with local (Exim 4.74)
	(envelope-from <nbowler@elliptictech.com>)
	id 1QENIx-0002sC-M1; Mon, 25 Apr 2011 11:02:27 -0400
Date: Mon, 25 Apr 2011 11:02:27 -0400
From: Nick Bowler <nbowler@elliptictech.com>
To: Eric Blake <eblake@redhat.com>
Subject: Re: [PATCH 1/2] fs: add SEEK_HOLE and SEEK_DATA flags
Message-ID: <20110425150227.GA10653@elliptictech.com>
References: <1303414954-3315-1-git-send-email-josef@redhat.com>
	<loom.20110422T001650-760@post.gmane.org>
	<BANLkTiknb+hzFAjpwESwMcqMVtkFc0HFQQ@mail.gmail.com>
	<20110422045054.GB17795@infradead.org>
	<20110422112852.GB1627@x4.trippels.de>
	<4DB17D5B.2090303@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4DB17D5B.2090303@redhat.com>
Organization: Elliptic Technologies Inc.
User-Agent: Mutt/1.5.21 (2010-09-15)
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 209.217.122.41
X-Mailman-Approved-At: Mon, 25 Apr 2011 11:59:43 -0400
Cc: Josef Bacik <josef@toxicpanda.com>, linux-kernel@vger.kernel.org,
	Christoph Hellwig <hch@infradead.org>,
	Markus Trippelsdorf <markus@trippelsdorf.de>,
	linux-fsdevel@vger.kernel.org, linux-btrfs@vger.kernel.org,
	Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Mon, 25 Apr 2011 15:02:55 -0000

Hi Eric,

On 2011-04-22 07:06 -0600, Eric Blake wrote:
> I've created a request to standardize SEEK_HOLE and SEEK_DATA in the
> next revision of POSIX; comments are welcome to make sure that everyone
> is happy with wording:
> http://austingroupbugs.net/view.php?id=415

Reading through your proposal, I think there is one thing that could be
clarified: the meaning of "the last hole" in the file.  Consider the
following two file layouts -- in the "diagrams", a bar (|) indicates a
boundary between holes and non-hole data, and a double bar (||)
indicates end-of-file.

* File A (sparse file created by lseek/write beyond end-of-file):

    data | hole 0 | data || hole 1 (virtual)

* File B (sparse file created by truncate beyond end-of-file):

    data | hole 0 || hole 1 (virtual)

Excluding the error description, the term "the last hole" is used in
two places in your proposal:

  * (for SEEK_HOLE): if offset falls within "the last hole", then the
    file offset may be set to the file size instead.

  * (for SEEK_DATA): it shall be an error ... if offset falls within the
    last hole.

I imagine that both of these conditions are intended to address the
case where the offset falls within hole 0 in File B, that is, when
there is no non-hole data beyond the specified offset but the offset
is nevertheless less than the file size.  However, this looks (to me)
like the penultimate hole in the file, not the last hole.  Furthermore,
these conditions are presumably *not* intended to apply to the
penultimate hole in File A, which has data after it.

I think my confusion can be avoided by talking about the last non-hole
data byte in the file (which is unambigious), instead of by talking
about the last hole.  For instance, the SEEK_HOLE/SEEK_DATA descriptions
could be written as follows:

  If whence is SEEK_HOLE, the file offset shall be set to the smallest
  location of a byte within a hole and not less than offset, except that
  if offset falls beyond the last byte not within a hole, then the file
  offset may be set to the file size instead.  It shall be an error if
  offset is greater or equal to the size of the file.

  If whence is SEEK_DATA, the file offset shall be set to the smallest
  location of a byte not within a hole and not less than offset.  It shall
  be an error if no such byte exists.

plus a corresponding update to the ENXIO description:

  ... or the whence argument is SEEK_DATA and the offset falls beyond
  the last byte not within a hole.

-- 
Nick Bowler, Elliptic Technologies (http://www.elliptictech.com/)


From MAILER-DAEMON Tue Apr 26 03:40:55 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEctD-0000JC-Ep
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 03:40:55 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47906)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEct6-0000J5-Bq
	for coreutils@gnu.org; Tue, 26 Apr 2011 03:40:54 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEct1-0002Is-HV
	for coreutils@gnu.org; Tue, 26 Apr 2011 03:40:48 -0400
Received: from mx.meyering.net ([82.230.74.64]:60460)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEct1-0002Ib-6i
	for coreutils@gnu.org; Tue, 26 Apr 2011 03:40:43 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id E329560113; Tue, 26 Apr 2011 09:40:40 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 2/2] maint: tweak sc_tight_scope rule
Date: Tue, 26 Apr 2011 09:40:40 +0200
Message-ID: <87ei4pcvev.fsf@rho.meyering.net>
Lines: 98
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 07:40:54 -0000

FYI, I've added a NEWS entry for the AIX fix, and in a separate commit
tweaked src/Makefile.am sc_tight_scope rule:

>From 740516cd1205b58ad07f2318335d8177586a4ca4 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Tue, 26 Apr 2011 09:09:43 +0200
Subject: [PATCH 1/2] dd: work around compilation failure on AIX 5.1 and 5.2

* src/dd.c (O_NOCACHE): Undefine.  This symbol is defined
via AIX's <fcntl.h>, yet used as an enum name in dd.c.
Reported by Gary V. Vaughan in http://debbugs.gnu.org/8555
* NEWS (Portability): Mention this.
---
 NEWS     |    4 ++++
 src/dd.c |    4 ++++
 2 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index 3307e71..025b564 100644
--- a/NEWS
+++ b/NEWS
@@ -20,6 +20,10 @@ GNU coreutils NEWS                                    -*- outline -*-
       Sparse files are relatively unusual, and the copying code incurs
       the performance penalty of the now-mandatory sync only for them.

+** Portability
+
+  dd once again compiles on AIX 5.1 and 5.2
+

 * Noteworthy changes in release 8.11 (2011-04-13) [stable]

diff --git a/src/dd.c b/src/dd.c
index cb626fc..072410e 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -66,6 +66,10 @@
 # define O_CIO 0
 #endif

+/* On AIX 5.1 and AIX 5.2, O_NOCACHE is defined via <fcntl.h>
+   and would interfere with our use of that name, below.  */
+#undef O_NOCACHE
+
 #if ! HAVE_FDATASYNC
 # define fdatasync(fd) (errno = ENOSYS, -1)
 #endif
--
1.7.5.rc3.316.gd1ff9


>From 74e5ad9cf7c515b69ae9d5a83168c70eb2604fec Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Tue, 26 Apr 2011 09:27:32 +0200
Subject: [PATCH 2/2] maint: tweak sc_tight_scope rule

* src/Makefile.am (sc_tight_scope): Remove useless quotes,
change 1>&2 to >&2, and combine a few short lines.
---
 src/Makefile.am |   20 ++++++++------------
 1 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/src/Makefile.am b/src/Makefile.am
index de0000d..548f5ed 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -730,21 +730,17 @@ sc_tight_scope: $(bin_PROGRAMS)
 	  perl -lne '/^extern (?:enum )?\S+ \**(\S*) \(/ and print $$1' \
 		$$hdr;							\
 	) | $(ASSORT) -u | sed 's/^/^/;s/$$/$$/' > $$t;			\
-	nm -e *.$(OBJEXT)						\
-	    | sed -n 's/.* T //p'					\
-	    | sed 's/^_//'						\
-	    | grep -Ev -f $$t &&					\
-	  { echo 'the above functions should have static scope' 1>&2;	\
-	    exit 1; } || : ;						\
+	nm -e *.$(OBJEXT) | sed -n 's/.* T //p'				\
+	    | sed 's/^_//' | grep -Ev -f $$t				\
+	  && { echo the above functions should have static scope >&2;	\
+	       exit 1; } || : ;						\
 	( printf '^program_name$$\n';					\
 	  perl -lne '/^extern .*?\**(\w+);/ and print "^$$1\$$"'	\
 	    $$hdr *.h ) | $(ASSORT) -u > $$t;				\
-	nm -e *.$(OBJEXT)						\
-	    | sed -n 's/.* [BD] //p'					\
-	    | sed 's/^_//'						\
-	    | grep -Ev -f $$t &&					\
-	  { echo 'the above variables should have static scope' 1>&2;	\
-	    exit 1; } || :
+	nm -e *.$(OBJEXT) | sed -n 's/.* [BD] //p'			\
+	    | sed 's/^_//' | grep -Ev -f $$t				\
+	  && { echo the above variables should have static scope >&2;	\
+	       exit 1; } || :

 # Use the just-built ./ginstall, when not cross-compiling.
 if CROSS_COMPILING
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Tue Apr 26 03:42:11 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEcuR-00013O-Gs
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 03:42:11 -0400
Received: from eggs.gnu.org ([140.186.70.92]:48276)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEcuK-000128-Tz
	for coreutils@gnu.org; Tue, 26 Apr 2011 03:42:10 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEcuF-0002Wj-7a
	for coreutils@gnu.org; Tue, 26 Apr 2011 03:42:04 -0400
Received: from mx.meyering.net ([82.230.74.64]:50544)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEcuF-0002WX-0w
	for coreutils@gnu.org; Tue, 26 Apr 2011 03:41:59 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 5F9996029B; Tue, 26 Apr 2011 09:41:58 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] maint: move two small functions, so we can remove a fwd decl
Date: Tue, 26 Apr 2011 09:41:58 +0200
Message-ID: <878vuxcvcp.fsf@rho.meyering.net>
Lines: 100
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 07:42:10 -0000

Looking at dd.c I noticed an unnecessary forward declaration
and saw it was easy to remove, so...

>From d891fe81f7f774aa19857c888e5924aecf421ebe Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Tue, 26 Apr 2011 09:02:44 +0200
Subject: [PATCH] maint: move two small functions, so we can remove a fwd
 decl

* src/dd.c (cleanup, quit): Move the definition of quit to follow the
definition of process_signals, so we can remove the declaration of
the latter.
---
 src/dd.c |   50 ++++++++++++++++++++++++--------------------------
 1 files changed, 24 insertions(+), 26 deletions(-)

diff --git a/src/dd.c b/src/dd.c
index 3472442..cb626fc 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -37,8 +37,6 @@
 #include "xstrtol.h"
 #include "xtime.h"

-static void process_signals (void);
-
 /* The official name of this program (e.g., no `g' prefix).  */
 #define PROGRAM_NAME "dd"

@@ -674,30 +672,6 @@ print_stats (void)
   fprintf (stderr, _(", %g s, %s/s\n"), delta_s, bytes_per_second);
 }

-static void
-cleanup (void)
-{
-  if (close (STDIN_FILENO) < 0)
-    error (EXIT_FAILURE, errno,
-           _("closing input file %s"), quote (input_file));
-
-  /* Don't remove this call to close, even though close_stdout
-     closes standard output.  This close is necessary when cleanup
-     is called as part of a signal handler.  */
-  if (close (STDOUT_FILENO) < 0)
-    error (EXIT_FAILURE, errno,
-           _("closing output file %s"), quote (output_file));
-}
-
-static void ATTRIBUTE_NORETURN
-quit (int code)
-{
-  cleanup ();
-  print_stats ();
-  process_signals ();
-  exit (code);
-}
-
 /* An ordinary signal was received; arrange for the program to exit.  */

 static void
@@ -772,6 +746,21 @@ install_signal_handlers (void)
 #endif
 }

+static void
+cleanup (void)
+{
+  if (close (STDIN_FILENO) < 0)
+    error (EXIT_FAILURE, errno,
+           _("closing input file %s"), quote (input_file));
+
+  /* Don't remove this call to close, even though close_stdout
+     closes standard output.  This close is necessary when cleanup
+     is called as part of a signal handler.  */
+  if (close (STDOUT_FILENO) < 0)
+    error (EXIT_FAILURE, errno,
+           _("closing output file %s"), quote (output_file));
+}
+
 /* Process any pending signals.  If signals are caught, this function
    should be called periodically.  Ideally there should never be an
    unbounded amount of time when signals are not being processed.  */
@@ -805,6 +794,15 @@ process_signals (void)
     }
 }

+static void ATTRIBUTE_NORETURN
+quit (int code)
+{
+  cleanup ();
+  print_stats ();
+  process_signals ();
+  exit (code);
+}
+
 /* Return LEN rounded down to a multiple of PAGE_SIZE
    while storing the remainder internally per FD.
    Pass LEN == 0 to get the current remainder.  */
--
1.7.5.rc3.316.gd1ff9


From MAILER-DAEMON Tue Apr 26 06:24:02 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEfR4-0001Es-8G
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 06:24:02 -0400
Received: from eggs.gnu.org ([140.186.70.92]:40489)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QEfQy-0001E9-0J
	for coreutils@gnu.org; Tue, 26 Apr 2011 06:24:00 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QEfQw-0000eG-Qy
	for coreutils@gnu.org; Tue, 26 Apr 2011 06:23:55 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:17550)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QEfQw-0000ds-9C
	for coreutils@gnu.org; Tue, 26 Apr 2011 06:23:54 -0400
Received: (qmail 56677 invoked from network); 26 Apr 2011 10:23:51 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 26 Apr 2011 10:23:51 -0000
Message-ID: <4DB69C9B.60202@draigBrady.com>
Date: Tue, 26 Apr 2011 11:21:15 +0100
From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Coreutils <coreutils@gnu.org>
Subject: [PATCH] maint: enable -Wmissing-field-initializers
X-Enigmail-Version: 1.0.1
Content-Type: multipart/mixed; boundary="------------030006010104050303000105"
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 10:24:00 -0000

This is a multi-part message in MIME format.
--------------030006010104050303000105
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

I'll apply the attached after the pending release.

cheers,
PĂˇdraig.

--------------030006010104050303000105
Content-Type: text/x-patch;
 name="enable-missing-field-init-warning.diff"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename="enable-missing-field-init-warning.diff"

>From b1e2512f1ad88d88e1654cb0d78860f32bc76549 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <P@draigBrady.com>
Date: Tue, 26 Apr 2011 10:30:05 +0100
Subject: [PATCH] maint: enable -Wmissing-field-initializers

* configure.ac: Enable -Wmissing-field-initializers for src/
but leave disabled for lib/.
* src/system.h: Enable the static variable hack only for
non gcc compilers and versions of gcc older than 4.7.
Starting with that version, the warning is no longer issued for
variables initialized with a single zero value.
Rename DECLARE_ZEROED_AGGREGATE to the more concise
and accurate DECLARE_ZEROED_AUTO.
* src/ls.c: Use the renamed macro.
* src/pathchk.c: Likewise.
* src/shred.c: Likewise.
* src/stty.c: Likewise.
* src/wc.c: Likewise.
---
 configure.ac  |    6 +++++-
 src/ls.c      |    2 +-
 src/pathchk.c |    2 +-
 src/shred.c   |    2 +-
 src/stty.c    |    4 ++--
 src/system.h  |    9 +++++----
 src/wc.c      |    2 +-
 7 files changed, 16 insertions(+), 11 deletions(-)

diff --git a/configure.ac b/configure.ac
index 6407bea..41d0442 100644
--- a/configure.ac
+++ b/configure.ac
@@ -96,7 +96,6 @@ if test "$gl_gcc_warnings" = yes; then
   for w in $ws; do
     gl_WARN_ADD([$w])
   done
-  gl_WARN_ADD([-Wno-missing-field-initializers]) # We need this one
   gl_WARN_ADD([-Wno-sign-compare])     # Too many warnings for now
   gl_WARN_ADD([-Wno-unused-parameter]) # Too many warnings for now
 
@@ -122,6 +121,11 @@ if test "$gl_gcc_warnings" = yes; then
   nw="$nw -Wmissing-prototypes"
   nw="$nw -Wold-style-definition"
   gl_MANYWARN_COMPLEMENT([GNULIB_WARN_CFLAGS], [$WARN_CFLAGS], [$nw])
+
+  # Disable this warning too, as gnulib
+  # doesn't use DECLARE_ZEROED_AUTO from system.h
+  gl_WARN_ADD([-Wno-missing-field-initializers], [GNULIB_WARN_CFLAGS])
+
   AC_SUBST([GNULIB_WARN_CFLAGS])
 
   # For gnulib-tests, the set is slightly smaller still.
diff --git a/src/ls.c b/src/ls.c
index a7ea8c2..49192d1 100644
--- a/src/ls.c
+++ b/src/ls.c
@@ -3891,7 +3891,7 @@ quote_name (FILE *out, const char *name, struct quoting_options const *options,
                      reach its end, replacing each non-printable multibyte
                      character with a single question mark.  */
                   {
-                    DECLARE_ZEROED_AGGREGATE (mbstate_t, mbstate);
+                    DECLARE_ZEROED_AUTO (mbstate_t, mbstate);
                     do
                       {
                         wchar_t wc;
diff --git a/src/pathchk.c b/src/pathchk.c
index f0f99dd..c721058 100644
--- a/src/pathchk.c
+++ b/src/pathchk.c
@@ -196,7 +196,7 @@ portable_chars_only (char const *file, size_t filelen)
 
   if (*invalid)
     {
-      DECLARE_ZEROED_AGGREGATE (mbstate_t, mbstate);
+      DECLARE_ZEROED_AUTO (mbstate_t, mbstate);
       size_t charlen = mbrlen (invalid, filelen - validlen, &mbstate);
       error (0, 0,
              _("nonportable character %s in file name %s"),
diff --git a/src/shred.c b/src/shred.c
index 10425a3..5906cdb 100644
--- a/src/shred.c
+++ b/src/shred.c
@@ -1098,7 +1098,7 @@ int
 main (int argc, char **argv)
 {
   bool ok = true;
-  DECLARE_ZEROED_AGGREGATE (struct Options, flags);
+  DECLARE_ZEROED_AUTO (struct Options, flags);
   char **file;
   int n_files;
   int c;
diff --git a/src/stty.c b/src/stty.c
index 674a19a..9080114 100644
--- a/src/stty.c
+++ b/src/stty.c
@@ -730,7 +730,7 @@ main (int argc, char **argv)
 {
   /* Initialize to all zeroes so there is no risk memcmp will report a
      spurious difference in an uninitialized portion of the structure.  */
-  DECLARE_ZEROED_AGGREGATE (struct termios, mode);
+  DECLARE_ZEROED_AUTO (struct termios, mode);
 
   enum output_type output_type;
   int optc;
@@ -1003,7 +1003,7 @@ main (int argc, char **argv)
     {
       /* Initialize to all zeroes so there is no risk memcmp will report a
          spurious difference in an uninitialized portion of the structure.  */
-      DECLARE_ZEROED_AGGREGATE (struct termios, new_mode);
+      DECLARE_ZEROED_AUTO (struct termios, new_mode);
 
       if (tcsetattr (STDIN_FILENO, TCSADRAIN, &mode))
         error (EXIT_FAILURE, errno, "%s", device_name);
diff --git a/src/system.h b/src/system.h
index 1c351bd..65890ac 100644
--- a/src/system.h
+++ b/src/system.h
@@ -469,14 +469,15 @@ enum
 
 /* With -Dlint, avoid warnings from gcc about code like mbstate_t m = {0,};
    by wasting space on a static variable of the same type, that is thus
-   guaranteed to be initialized to 0, and use that on the RHS.  */
+   guaranteed to be initialized to 0, and use that on the RHS.
+   This issue was fixed in gcc 4.7.  */
 #define DZA_CONCAT0(x,y) x ## y
 #define DZA_CONCAT(x,y) DZA_CONCAT0 (x, y)
-#ifdef lint
-# define DECLARE_ZEROED_AGGREGATE(Type, Var) \
+#if defined lint && (__GNUC__ < 4 || (__GNUC__ == 4 && __GNUC_MINOR__ < 7))
+# define DECLARE_ZEROED_AUTO(Type, Var) \
    static Type DZA_CONCAT (s0_, __LINE__); Type Var = DZA_CONCAT (s0_, __LINE__)
 #else
-# define DECLARE_ZEROED_AGGREGATE(Type, Var) \
+# define DECLARE_ZEROED_AUTO(Type, Var) \
   Type Var = { 0, }
 #endif
 
diff --git a/src/wc.c b/src/wc.c
index 0399214..5f9c7e5 100644
--- a/src/wc.c
+++ b/src/wc.c
@@ -285,7 +285,7 @@ wc (int fd, char const *file_x, struct fstatus *fstatus)
     {
       bool in_word = false;
       uintmax_t linepos = 0;
-      DECLARE_ZEROED_AGGREGATE (mbstate_t, state);
+      DECLARE_ZEROED_AUTO (mbstate_t, state);
       bool in_shift = false;
 # if SUPPORT_OLD_MBRTOWC
       /* Back-up the state before each multibyte character conversion and
-- 
1.7.4


--------------030006010104050303000105--


From MAILER-DAEMON Tue Apr 26 10:30:21 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEjHR-00050i-Ci
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 10:30:21 -0400
Received: from eggs.gnu.org ([140.186.70.92]:55598)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEjHO-0004zL-UG
	for coreutils@gnu.org; Tue, 26 Apr 2011 10:30:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEjHO-0006tN-7i
	for coreutils@gnu.org; Tue, 26 Apr 2011 10:30:18 -0400
Received: from mx.meyering.net ([82.230.74.64]:55804)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEjHN-0006sq-UR
	for coreutils@gnu.org; Tue, 26 Apr 2011 10:30:18 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id CF85E603A0; Tue, 26 Apr 2011 16:30:15 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: coreutils-8.12 in an hour or two
Date: Tue, 26 Apr 2011 16:30:15 +0200
Message-ID: <87sjt59jbc.fsf@rho.meyering.net>
Lines: 6
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 14:30:19 -0000

FYI, I've tagged the latest and, assuming last-minute testing
exposes nothing serious, I'll release coreutils-8.12 within
an hour or two.

So, if you're sitting on a bug report, this is your last chance
to get it fixed in 8.12.


From MAILER-DAEMON Tue Apr 26 14:41:55 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEnCt-0001lF-Uc
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 14:41:55 -0400
Received: from eggs.gnu.org ([140.186.70.92]:47860)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEnCs-0001iE-9M
	for coreutils@gnu.org; Tue, 26 Apr 2011 14:41:55 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QEnCr-0003Yo-6x
	for coreutils@gnu.org; Tue, 26 Apr 2011 14:41:54 -0400
Received: from mx.meyering.net ([82.230.74.64]:36570)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>)
	id 1QEnCo-0003YO-Ax; Tue, 26 Apr 2011 14:41:50 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 379166029B; Tue, 26 Apr 2011 20:41:48 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: info-gnu@gnu.org
Subject: coreutils-8.12 released [stable]
Mail-Followup-To: coreutils@gnu.org
Date: Tue, 26 Apr 2011 20:41:48 +0200
Message-ID: <87y62w97o3.fsf@rho.meyering.net>
Lines: 118
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coordinator@translationproject.org, coreutils@gnu.org,
	coreutils-announce@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 18:41:55 -0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is to announce coreutils-8.12, a stable release.

We released coreutils-8.11 less than two weeks ago.
Why a new release so soon?  Because under unusual conditions,
coreutils-8.11's copying code could cause trouble.  Data loss trouble.
The trouble could arise only when these conditions are all met:
  - when using linux-2.6.39-related kernels (including at least -rc3) and
  - using an xfs file system and
  - copying (via cp, install, mv) a file with a so-called "unwritten
      extent" shortly after it has been created, yet before some
      data in that unwritten extent has made it to disk.
      This would happen if you're using the "gold" linker, which
      preallocates using fallocate and then writes its output
      (the binaries) into those unwritten extents, and you then
      immediately copy those binaries into place via "make install".
Under those conditions, just building coreutils and running "make
install" quickly enough after compile and link would result in
installing files containing all 0 bytes.

See the commit logs for links to plenty of discussion.
See the NEWS below for a brief summary.

Jim [on behalf of the coreutils maintainers]

- - -------------------------------------------
P.S. here's the GNU Coreutils home page:
    http://www.gnu.org/software/coreutils/

For a summary of changes and contributors, see:
  http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=shortlog;h=v8.12
or run this command from a git-cloned coreutils directory:
  git shortlog v8.11..v8.12

To summarize the many gnulib-related changes, run these commands from
a git-cloned coreutils directory:
  git checkout v8.12
  git submodule summary v8.11

Here are the compressed sources:
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.12.tar.gz   (11MB)
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.12.tar.xz   (4.7MB)

Here are the GPG detached signatures[*]:
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.12.tar.gz.sig
  http://ftp.gnu.org/gnu/coreutils/coreutils-8.12.tar.xz.sig

To reduce load on the main server, use a mirror listed at:
  http://www.gnu.org/order/ftp.html

[*] You can use either of the above signature files to verify that
the corresponding file (without the .sig suffix) is intact.  First,
be sure to download both the .sig file and the corresponding tarball.
Then, run a command like this:

  gpg --verify coreutils-8.12.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

  gpg --keyserver keys.gnupg.net --recv-keys 000BEEEE

and rerun the `gpg --verify' command.

This release was bootstrapped with the following tools:
  Autoconf 2.68.71-af300
  Automake 1.11a
  Gnulib v0.0-5115-ga81348d
  Bison 2.4.588-2f658

===================================================================
NEWS

* Noteworthy changes in release 8.12 (2011-04-26) [stable]

** Bug fixes

  tail's --follow=name option no longer implies --retry on systems
  with inotify support.  [bug introduced in coreutils-7.5]

** Changes in behavior

  cp's extent-based (FIEMAP) copying code is more reliable in the face
  of varying and undocumented file system semantics:
  - it no longer treats unwritten extents specially
  - a FIEMAP-based extent copy always uses the FIEMAP_FLAG_SYNC flag.
      Before, it would incur the performance penalty of that sync only
      for 2.6.38 and older kernels.  We thought all problems would be
      resolved for 2.6.39.
  - it now attempts a FIEMAP copy only on a file that appears sparse.
      Sparse files are relatively unusual, and the copying code incurs
      the performance penalty of the now-mandatory sync only for them.

** Portability

  dd once again compiles on AIX 5.1 and 5.2

- -----
also posted as: https://savannah.gnu.org/forum/forum.php?forum_id=6798
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBCAAGBQJNtxCCAAoJEH/Z/MsAC+7uLWYQAJSe7I4Gp92cA4m/1SxF1t/S
S8dpD3ay7DWivX6EHFjhrdXu6y/urZjVQAlALNQLBgyzZ+khkwV3KllKWzo6K/Qw
6bU9rF7tbfuzb0/GkOEaFOM3Jp87RzP7Ju31YR/zLx0/0aPGdV/7vGmKmLjdrdoD
0BsW3118cp6i4TT7tFoknNqfDmtBW/B/N8tnpYVBG3UmMK8+UprCdbD63JJ1aiIV
0MP601699T3+iK7N3xSk4c8G6KDV+clCYCxmBiq72IJUDN8qBeuYJ2InVtQZ7ilA
3CG5aLL5sDCBL+lSoy7JtmY9pJ5BbMM+wn/k4jCoqiWjv+cTR1U39RLIG7QQMbg2
7c0sY2LkN2eMCc/mxuCYCMmOtDBkA8eRg1l5bcvRm4SVaMr4OlsmMaLFTfgzxQwh
19OpMlkQya0lmhj/K1k0ONdrox5xvkuIoabYsqOBoNiESaoQkwQNMfJFUltY0pKv
GXQt6o++SQcba+BpFTA+sLbOqahlcpZz3jJG0xoC53X3WKWVPaONou1VE3o8NV4c
7ER727eYq/2/P3wd1IIFP3HpoWPUxxoEEhZb4GqXE7yXCro5vBYn8fV7CKfnivQV
k11mAZRHg9d4Zc+NudxdysSalq84c173Ix9AMqT1iE6sCpiDhzv082k3maLiDZfS
TdzjjS+6yN5h0Kx8H2RT
=iMOO
-----END PGP SIGNATURE-----


From MAILER-DAEMON Tue Apr 26 16:04:19 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEoUd-0002Wp-4m
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 16:04:19 -0400
Received: from eggs.gnu.org ([140.186.70.92]:44023)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <btwood@ufl.edu>) id 1QEoB0-00083T-Ig
	for coreutils@gnu.org; Tue, 26 Apr 2011 15:44:03 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <btwood@ufl.edu>) id 1QEoAz-0005OD-KY
	for coreutils@gnu.org; Tue, 26 Apr 2011 15:44:02 -0400
Received: from smtp03.osg.ufl.edu ([128.227.74.70]:57127 helo=smtp.ufl.edu)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <btwood@ufl.edu>) id 1QEoAz-0005O8-Fh
	for coreutils@gnu.org; Tue, 26 Apr 2011 15:44:01 -0400
Received: from [10.227.18.43] (43-18-227-10.unnamed.ees.ufl.edu [10.227.18.43]
	(may be forged)) (authenticated bits=0)
	by smtp.ufl.edu (8.14.0/8.14.0/3.0.0) with ESMTP id p3QJi0OI022867
	for <coreutils@gnu.org>; Tue, 26 Apr 2011 15:44:00 -0400
Message-ID: <4DB72080.5070506@ufl.edu>
Date: Tue, 26 Apr 2011 15:44:00 -0400
From: Brandon Wood <btwood@ufl.edu>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
	rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9
MIME-Version: 1.0
To: coreutils@gnu.org
Subject: cut feature request: treat multiple consecutive delimiters as one
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.2.15, 1.0.148,
	0.0.0000
	definitions=2011-04-26_06:2011-04-26, 2011-04-26,
	1970-01-01 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0
	ipscore=0 suspectscore=1
	phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0
	reason=mlx
	engine=6.0.2-1012030000 definitions=main-1104260134
X-UFL-Spam-Level: *
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3)
X-Received-From: 128.227.74.70
X-Mailman-Approved-At: Tue, 26 Apr 2011 16:04:18 -0400
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 19:44:03 -0000

When dealing with stdout or delimited files that have multiple 
consecutive delimiters, I think that allowing cut to treat these as a 
single delimiter may be advantageous.

When considering a trivial example,
$ wc -l *.dat | grep data
1107 data_01.dat
  180 data_02.dat
  317 data_03.dat
  295 data_04.dat
   10 data_05.dat
$ !! | cut --mult-delim -d " " -f 1,2
1107 data_01.dat
180 data_02.dat
317 data_03.dat
295 data_04.dat
10 data_05.dat

Another example with a data file,
$ tail -n 3 data.dat
AAAA         BBBB         CCCC
1111         2222         3333
XXXX         YYYY         ZZZZ
$ !! | cut --mult-delim -d " " -f 2
BBBB
2222
YYYY

Obviously the flag can be something more suiting.

Though this may be more useful for those users dealing with large 
delimited files that are not properly formatted to be cut, though more 
creative folks may be able to use it in some more interesting ways.

While this may disturb files that have legitimately blank fields, it 
would be off by default to avoid any problems. Treating multiple 
delimiters as one is common in most spreadsheet applications when 
importing delimited data.

Any thoughts? My apologies if this is a previously requested feature, 
though I did not find it in the archives.


From MAILER-DAEMON Tue Apr 26 16:33:06 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEowU-0007p2-BA
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 16:33:06 -0400
Received: from eggs.gnu.org ([140.186.70.92]:40921)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QEowR-0007ot-L8
	for coreutils@gnu.org; Tue, 26 Apr 2011 16:33:04 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QEowQ-0004TN-AR
	for coreutils@gnu.org; Tue, 26 Apr 2011 16:33:03 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:12191)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QEowQ-0004T2-19
	for coreutils@gnu.org; Tue, 26 Apr 2011 16:33:02 -0400
Received: (qmail 73525 invoked from network); 26 Apr 2011 20:32:59 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 26 Apr 2011 20:32:59 -0000
Message-ID: <4DB72B5D.4090406@draigBrady.com>
Date: Tue, 26 Apr 2011 21:30:21 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Brandon Wood <btwood@ufl.edu>
Subject: Re: cut feature request: treat multiple consecutive delimiters as one
References: <4DB72080.5070506@ufl.edu>
In-Reply-To: <4DB72080.5070506@ufl.edu>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 20:33:04 -0000

On 26/04/11 20:44, Brandon Wood wrote:
> When dealing with stdout or delimited files that have multiple
> consecutive delimiters, I think that allowing cut to treat these as a
> single delimiter may be advantageous.
> 
> When considering a trivial example,
> $ wc -l *.dat | grep data
> 1107 data_01.dat
>  180 data_02.dat
>  317 data_03.dat
>  295 data_04.dat
>   10 data_05.dat
> $ !! | cut --mult-delim -d " " -f 1,2
> 1107 data_01.dat
> 180 data_02.dat
> 317 data_03.dat
> 295 data_04.dat
> 10 data_05.dat

> Any thoughts? My apologies if this is a previously requested feature,
> though I did not find it in the archives.

Yes, we're maybe leaning towards adding cut --blank-separated
http://lists.gnu.org/archive/html/bug-coreutils/2010-11/msg00203.html

For the moment, we've documented `awk` and `join` alternatives:
http://www.gnu.org/software/coreutils/manual/coreutils.html#cut-invocation

cheers,
Pádraig.


From MAILER-DAEMON Tue Apr 26 16:34:53 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QEoyD-00085e-Kx
	for mharc-coreutils@gnu.org; Tue, 26 Apr 2011 16:34:53 -0400
Received: from eggs.gnu.org ([140.186.70.92]:57156)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <public@plane.gmane.org>) id 1QEoqe-00071w-Ka
	for coreutils@gnu.org; Tue, 26 Apr 2011 16:27:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <public@plane.gmane.org>) id 1QEoqd-0003Q6-HX
	for coreutils@gnu.org; Tue, 26 Apr 2011 16:27:04 -0400
Received: from plane.gmane.org ([80.91.229.3]:41418)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <public@plane.gmane.org>) id 1QEoqd-0003Pg-9U
	for coreutils@gnu.org; Tue, 26 Apr 2011 16:27:03 -0400
Received: from public by plane.gmane.org with local (Exim 4.69)
	(envelope-from <public@plane.gmane.org>) id 1QEoqS-0006V8-Ng
	for coreutils@gnu.org; Tue, 26 Apr 2011 22:26:52 +0200
Received: from mail-out.m-online.net ([212.18.0.10])
	by plane.gmane.org with esmtp (Exim 4.69)
	(envelope-from <whitebox@nefkom.net>)
	id 1QEoqL-0006KN-4z; Tue, 26 Apr 2011 22:26:45 +0200
Received: from frontend1.mail.m-online.net (frontend1.mail.intern.m-online.net
	[192.168.8.180])
	by mail-out.m-online.net (Postfix) with ESMTP id 33930180092B;
	Tue, 26 Apr 2011 22:26:44 +0200 (CEST)
Received: from localhost (dynscan1.mnet-online.de [192.168.8.164])
	by mail.m-online.net (Postfix) with ESMTP id 390441C000B3;
	Tue, 26 Apr 2011 22:26:44 +0200 (CEST)
X-Virus-Scanned: amavisd-new at mnet-online.de
Received: from mail.mnet-online.de ([192.168.8.180])
	by localhost (dynscan1.mail.m-online.net [192.168.8.164]) (amavisd-new,
	port 10024)
	with ESMTP id ET63X2h10uAH; Tue, 26 Apr 2011 22:26:41 +0200 (CEST)
Received: from igel.home (ppp-88-217-123-4.dynamic.mnet-online.de
	[88.217.123.4]) by mail.mnet-online.de (Postfix) with ESMTP;
	Tue, 26 Apr 2011 22:26:40 +0200 (CEST)
Received: by igel.home (Postfix, from userid 501)
	id 555ECCA29C; Tue, 26 Apr 2011 22:26:40 +0200 (CEST)
From: Andreas Schwab <schwab@linux-m68k.org>
To: Brandon Wood <public-btwood-hnhdhkBXzx8@plane.gmane.org>
Subject: Re: cut feature request: treat multiple consecutive delimiters as one
References: <4DB72080.5070506@ufl.edu>
X-Yow: On the other hand, life can be an endless parade of TRANSSEXUAL
	QUILTING BEES aboard a cruise ship to DISNEYWORLD if only we let it!!
Date: Tue, 26 Apr 2011 22:26:40 +0200
In-Reply-To: <4DB72080.5070506@ufl.edu> (Brandon Wood's message of "Tue, 26
	Apr 2011 15:44:00 -0400")
Message-ID: <m2mxjc7o8v.fsf@igel.home>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 80.91.229.3
X-Mailman-Approved-At: Tue, 26 Apr 2011 16:34:52 -0400
Cc: public-coreutils-mXXj517/zsQ@plane.gmane.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Tue, 26 Apr 2011 20:27:05 -0000



Brandon Wood <btwood-hnhdhkBXzx8@public.gmane.org> writes:

> When considering a trivial example,
> $ wc -l *.dat | grep data
> 1107 data_01.dat
>  180 data_02.dat
>  317 data_03.dat
>  295 data_04.dat
>   10 data_05.dat
> $ !! | cut --mult-delim -d " " -f 1,2
> 1107 data_01.dat
> 180 data_02.dat
> 317 data_03.dat
> 295 data_04.dat
> 10 data_05.dat
>
> Another example with a data file,
> $ tail -n 3 data.dat
> AAAA         BBBB         CCCC
> 1111         2222         3333
> XXXX         YYYY         ZZZZ
> $ !! | cut --mult-delim -d " " -f 2
> BBBB
> 2222
> YYYY

You can do that with awk:

$ awk '{print $1,$2}'
$ awk '{print $2}'

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."



From MAILER-DAEMON Wed Apr 27 16:14:38 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFB8A-00017O-HF
	for mharc-coreutils@gnu.org; Wed, 27 Apr 2011 16:14:38 -0400
Received: from eggs.gnu.org ([140.186.70.92]:56253)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFB88-00016Q-DH
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:14:37 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFB87-0005wb-JI
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:14:36 -0400
Received: from mx.meyering.net ([82.230.74.64]:34052)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFB87-0005wX-BD
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:14:35 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id C866A600E1; Wed, 27 Apr 2011 22:14:33 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: [PATCH] maint: enable -Wmissing-field-initializers
In-Reply-To: <4DB69C9B.60202@draigBrady.com> (=?iso-8859-1?Q?=22P=E1draig?=
	Brady"'s message of "Tue, 26 Apr 2011 11:21:15 +0100")
References: <4DB69C9B.60202@draigBrady.com>
Date: Wed, 27 Apr 2011 22:14:33 +0200
Message-ID: <8762pz4fkm.fsf@rho.meyering.net>
Lines: 21
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Apr 2011 20:14:37 -0000

P=E1draig Brady wrote:
> I'll apply the attached after the pending release.
>
> Subject: [PATCH] maint: enable -Wmissing-field-initializers
>
> * configure.ac: Enable -Wmissing-field-initializers for src/
> but leave disabled for lib/.
> * src/system.h: Enable the static variable hack only for
> non gcc compilers and versions of gcc older than 4.7.
> Starting with that version, the warning is no longer issued for
> variables initialized with a single zero value.
> Rename DECLARE_ZEROED_AGGREGATE to the more concise
> and accurate DECLARE_ZEROED_AUTO.
> * src/ls.c: Use the renamed macro.
> * src/pathchk.c: Likewise.
> * src/shred.c: Likewise.
> * src/stty.c: Likewise.
> * src/wc.c: Likewise.

Nice.  You beat me to it.
Thanks.


From MAILER-DAEMON Wed Apr 27 16:34:25 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFBRJ-0005RB-6D
	for mharc-coreutils@gnu.org; Wed, 27 Apr 2011 16:34:25 -0400
Received: from eggs.gnu.org ([140.186.70.92]:50213)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QFBRH-0005R5-Mj
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:34:24 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QFBRG-0000ZN-Nm
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:34:23 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:9718)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QFBRG-0000ZF-Ck
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:34:22 -0400
Received: (qmail 12847 invoked from network); 27 Apr 2011 20:34:19 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 27 Apr 2011 20:34:19 -0000
Message-ID: <4DB87D28.5010007@draigBrady.com>
Date: Wed, 27 Apr 2011 21:31:36 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: [PATCH] maint: enable -Wmissing-field-initializers
References: <4DB69C9B.60202@draigBrady.com> <8762pz4fkm.fsf@rho.meyering.net>
In-Reply-To: <8762pz4fkm.fsf@rho.meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Apr 2011 20:34:24 -0000

On 27/04/11 21:14, Jim Meyering wrote:
> Pádraig Brady wrote:
>> I'll apply the attached after the pending release.
>>
>> Subject: [PATCH] maint: enable -Wmissing-field-initializers
>>
>> * configure.ac: Enable -Wmissing-field-initializers for src/
>> but leave disabled for lib/.
>> * src/system.h: Enable the static variable hack only for
>> non gcc compilers and versions of gcc older than 4.7.
>> Starting with that version, the warning is no longer issued for
>> variables initialized with a single zero value.
>> Rename DECLARE_ZEROED_AGGREGATE to the more concise
>> and accurate DECLARE_ZEROED_AUTO.
>> * src/ls.c: Use the renamed macro.
>> * src/pathchk.c: Likewise.
>> * src/shred.c: Likewise.
>> * src/stty.c: Likewise.
>> * src/wc.c: Likewise.
> 
> Nice.  You beat me to it.
> Thanks.

I was thinking actually of changing this, and doing this solely within
manywarnings.m4 in gnulib. I.E. apply -Wno-missing-field-initializers
conditionally there (depending on result of compiling {0,}).

Then we could remove all this stuff from coreutils,
including the DECLARE_ZEROED_AGGREGATE macro.


From MAILER-DAEMON Wed Apr 27 16:42:48 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFBZQ-0006Fh-U9
	for mharc-coreutils@gnu.org; Wed, 27 Apr 2011 16:42:48 -0400
Received: from eggs.gnu.org ([140.186.70.92]:38422)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFBZO-0006FJ-6E
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:42:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFBZN-0001zi-8j
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:42:46 -0400
Received: from mx.meyering.net ([82.230.74.64]:39591)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFBZN-0001ze-2b
	for coreutils@gnu.org; Wed, 27 Apr 2011 16:42:45 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 45A3B600FB; Wed, 27 Apr 2011 22:42:44 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: [PATCH] maint: enable -Wmissing-field-initializers
In-Reply-To: <4DB87D28.5010007@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Wed, 27 Apr 2011 21:31:36 +0100")
References: <4DB69C9B.60202@draigBrady.com> <8762pz4fkm.fsf@rho.meyering.net>
	<4DB87D28.5010007@draigBrady.com>
Date: Wed, 27 Apr 2011 22:42:44 +0200
Message-ID: <87pqo72zp7.fsf@rho.meyering.net>
Lines: 34
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: Coreutils <coreutils@gnu.org>
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Wed, 27 Apr 2011 20:42:47 -0000

P=E1draig Brady wrote:
> On 27/04/11 21:14, Jim Meyering wrote:
>> P=E1draig Brady wrote:
>>> I'll apply the attached after the pending release.
>>>
>>> Subject: [PATCH] maint: enable -Wmissing-field-initializers
>>>
>>> * configure.ac: Enable -Wmissing-field-initializers for src/
>>> but leave disabled for lib/.
>>> * src/system.h: Enable the static variable hack only for
>>> non gcc compilers and versions of gcc older than 4.7.
>>> Starting with that version, the warning is no longer issued for
>>> variables initialized with a single zero value.
>>> Rename DECLARE_ZEROED_AGGREGATE to the more concise
>>> and accurate DECLARE_ZEROED_AUTO.
>>> * src/ls.c: Use the renamed macro.
>>> * src/pathchk.c: Likewise.
>>> * src/shred.c: Likewise.
>>> * src/stty.c: Likewise.
>>> * src/wc.c: Likewise.
>>
>> Nice.  You beat me to it.
>> Thanks.
>
> I was thinking actually of changing this, and doing this solely within
> manywarnings.m4 in gnulib. I.E. apply -Wno-missing-field-initializers
> conditionally there (depending on result of compiling {0,}).
>
> Then we could remove all this stuff from coreutils,
> including the DECLARE_ZEROED_AGGREGATE macro.

That sounds even better.
I like the idea of not having to put this logic in configure.ac
in every package where I'd like to use it.


From MAILER-DAEMON Thu Apr 28 17:17:23 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFYaR-0006N7-8M
	for mharc-coreutils@gnu.org; Thu, 28 Apr 2011 17:17:23 -0400
Received: from eggs.gnu.org ([140.186.70.92]:50625)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QFYaP-0006N0-8l
	for coreutils@gnu.org; Thu, 28 Apr 2011 17:17:22 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QFYaO-0001yU-AK
	for coreutils@gnu.org; Thu, 28 Apr 2011 17:17:21 -0400
Received: from qmta08.westchester.pa.mail.comcast.net ([76.96.62.80]:44599)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <eblake@redhat.com>) id 1QFYaO-0001yQ-7c
	for coreutils@gnu.org; Thu, 28 Apr 2011 17:17:20 -0400
Received: from omta18.westchester.pa.mail.comcast.net ([76.96.62.90])
	by qmta08.westchester.pa.mail.comcast.net with comcast
	id d99T1g0021wpRvQ589HLti; Thu, 28 Apr 2011 21:17:20 +0000
Received: from office.redhat.com ([24.10.251.25])
	by omta18.westchester.pa.mail.comcast.net with comcast
	id d9HG1g01u0ZdyUg3e9HK5e; Thu, 28 Apr 2011 21:17:20 +0000
From: Eric Blake <eblake@redhat.com>
To: coreutils@gnu.org
Subject: [PATCH] build: update to latest gnulib
Date: Thu, 28 Apr 2011 15:17:15 -0600
Message-Id: <1304025435-22407-1-git-send-email-eblake@redhat.com>
X-Mailer: git-send-email 1.7.4.4
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 76.96.62.80
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Thu, 28 Apr 2011 21:17:22 -0000

* gnulib: Update to latest.
* bootstrap.conf (gnulib_modules): Add xgetgroups.
---

Okay to push?

* gnulib a81348d...d6d01fd (70):
  > tests: drop unused link dependency
  > hash, mgetgroups: drop xalloc dependency
  > xalloc-oversized: new module
  > utimecmp: drop dependency on xmalloc
  > getcwd: fix mingw bugs
  > Typo in comment.
  > mkstemps: Ensure declaration on MacOS X 10.5.
  > mkstemp: More documentation.
  > mkstemp: Tweak configure message when cross-compiling.
  > clean-temp: Clarify what it does.
  > fchdir: avoid extra chdir and fix test
  > filenamecat-lgpl: fix licence
  > linkat, renameat: add missing dependency
  > tests: reduce dependencies
  > save-cwd: reduce default dependency
  > getcwd: enhance tests
  > getcwd-lgpl: new module
  > getcwd: tweak comments
  > mkstemp: replace if system version uses wrong permissions
  > passfd: avoid compiler warning
  > Change gnulib-tool to support NetBSD's join
  > mkstemp: mention clean-temp module
  > inttypes: also provide default values for 32-bit tests
  > strtoumax: remove dependency on strtoimax
  > inttypes-incomplete: new module
  > inttypes: omit now-redundant strtoimax and strtoumax work
  > strtoimax, strtoumax: simplify, port to HP-UX 11.00 64-bit
  > gnumakefile: Replace TAB with SPCs.
  > strtol, strtoul: Mark modules as obsolete.
  > strtod: Remove check for strtod, unless supporting old platforms.
  > strcase: Make module obsolete.
  > dup2: Remove check for dup2, unless supporting old obsolete platforms.
  > strnlen: Avoid memchr related link error on old obsolete platforms.
  > Revert "use _GL_ATTRIBUTE_CONST and _GL_ATTRIBUTE_PURE"
  > maint.mk: makefile_at_at_check extend and clean up
  > use _GL_ATTRIBUTE_CONST and _GL_ATTRIBUTE_PURE
  > strtoul: remove dependency on strtol
  > strtoull: remove dependency on strtoul
  > strtoll: remove dependency on strtol
  > inttypes: Move some configure check to module 'imaxdiv'.
  > inttypes: Move some configure check to module 'imaxabs'.
  > inttypes: Remove configure tests that are not needed since 2009-12-31.
  > * modules/strnlen (Depends-on): Remove memchr.
  > strtol: remove dependency on wchar
  > passfd: fix test regression on Linux
  > passfd: speed up configure and drop unused code
  > passfd: allow compilation on mingw
  > sys_uio: new module
  > useless-if-before-free: avoid false-positive
  > doc: update users.txt.
  > ioctl: Remove link dependency on native Windows.
  > Move the support of O_NONBLOCK in open() to the 'open' module.
  > nonblocking tests: Tweak comment.
  > pipe2: Simplify code.
  > nonblocking: Add comment.
  > nonblocking: Add tests for sockets.
  > nonblocking: Add tests for pipes.
  > autoupdate
  > gettext: Clarify the needed programmer actions.
  > strchrnul: Tweak last commit.
  > strchrnul: work around cygwin bug
  > Don't require lib/stdio-write.c when only module 'stdio' is used.
  > Support non-blocking pipe I/O in read() on native Windows.
  > Support non-blocking pipe I/O in write() on native Windows.
  > wchar: Ensure that wchar_t gets defined on uClibc.
  > safe-write, full-read: Avoid unnecessary compilation units.
  > Support non-blocking pipe I/O and SIGPIPE in pwrite().
  > Support non-blocking pipe I/O in write() on native Windows.
  > autoupdate
  > autoupdate

 bootstrap.conf |    1 +
 gnulib         |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/bootstrap.conf b/bootstrap.conf
index 5633e13..6ff11a4 100644
--- a/bootstrap.conf
+++ b/bootstrap.conf
@@ -241,6 +241,7 @@ gnulib_modules="
   xalloc
   xfreopen
   xgetcwd
+  xgetgroups
   xgethostname
   xmemcoll
   xnanosleep
diff --git a/gnulib b/gnulib
index a81348d..d6d01fd 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit a81348d2e136b0f2de110b49a5d0dfa5d7d76f71
+Subproject commit d6d01fdde58da00d0d839711097e3acb83f61754
-- 
1.7.4.4



From MAILER-DAEMON Fri Apr 29 04:20:45 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFiwO-0003Yo-Te
	for mharc-coreutils@gnu.org; Fri, 29 Apr 2011 04:20:44 -0400
Received: from eggs.gnu.org ([140.186.70.92]:33777)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFiwM-0003VT-MI
	for coreutils@gnu.org; Fri, 29 Apr 2011 04:20:43 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFiwL-0005QY-K9
	for coreutils@gnu.org; Fri, 29 Apr 2011 04:20:42 -0400
Received: from mx.meyering.net ([82.230.74.64]:56382)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>)
	id 1QFiwI-0005Py-Tf; Fri, 29 Apr 2011 04:20:39 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 9B9B860098; Fri, 29 Apr 2011 10:20:36 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Eric Blake <eblake@redhat.com>
Subject: [PATCH] test-getcwd.c: avoid new set-but-not-used warning
In-Reply-To: <1304025435-22407-1-git-send-email-eblake@redhat.com> (Eric
	Blake's message of "Thu, 28 Apr 2011 15:17:15 -0600")
References: <1304025435-22407-1-git-send-email-eblake@redhat.com>
Date: Fri, 29 Apr 2011 10:20:36 +0200
Message-ID: <87y62tv57v.fsf@rho.meyering.net>
Lines: 135
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: bug-gnulib@gnu.org, coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Apr 2011 08:20:43 -0000

Eric Blake wrote:
> * gnulib: Update to latest.
> * bootstrap.conf (gnulib_modules): Add xgetgroups.
> ---
>
> Okay to push?
>
> * gnulib a81348d...d6d01fd (70):

Thanks, but not yet.
That would pull in changes that induce two build failures
when configuring coreutils with --enable-gcc-warnings:

    test-hash.c: In function 'main':
    test-hash.c:213:23: error: declaration of 'dup' shadows a global declaration\
      [-Werror=shadow]
    test-getcwd.c: In function 'test_abort_bug':
    test-getcwd.c:49:7: error: variable 'fail' set but not used\
      [-Werror=unused-but-set-variable]

I've already pushed the trivial fix for test-hash.c:

  http://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=20abe9a92c71d7cc

I'm about to push this, to fix the second one:

>From c1bb7ba6ba9e685da05f48da5368d9342ba272c1 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Fri, 29 Apr 2011 09:53:25 +0200
Subject: [PATCH] test-getcwd.c: avoid new set-but-not-used warning

* tests/test-getcwd.c (test_abort_bug): Exit nonzero for any problem,
not just the glibc/abort one that getcwd-abort-bug.m4 detects.
This also adds an additional test for an unlikely chdir("..") failure.
* m4/getcwd-abort-bug.m4: Update this now-duplicated code to match,
and adjust the code that sets gl_cv_func_getcwd_abort_bug accordingly.
---
 ChangeLog              |    6 ++++++
 m4/getcwd-abort-bug.m4 |   17 ++++++++++++-----
 tests/test-getcwd.c    |    9 ++++++---
 3 files changed, 24 insertions(+), 8 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 3cc245e..dc01efd 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,11 @@
 2011-04-29  Jim Meyering  <meyering@redhat.com>

+	test-getcwd.c: avoid new set-but-not-used warning
+	* tests/test-getcwd.c (test_abort_bug): Exit nonzero for any problem,
+	not just the glibc/abort one that getcwd-abort-bug.m4 detects.
+	* m4/getcwd-abort-bug.m4: Update this now-duplicated code to match,
+	and adjust the code that sets gl_cv_func_getcwd_abort_bug accordingly.
+
 	test-hash.c: avoid a new shadowing warning
 	* tests/test-hash.c (main): Don't shadow "dup".

diff --git a/m4/getcwd-abort-bug.m4 b/m4/getcwd-abort-bug.m4
index 0b59a6b..fd6820d 100644
--- a/m4/getcwd-abort-bug.m4
+++ b/m4/getcwd-abort-bug.m4
@@ -1,4 +1,4 @@
-# serial 3
+# serial 4
 # Determine whether getcwd aborts when the length of the working directory
 # name is unusually large.  Any length between 4k and 16k trigger the bug
 # when using glibc-2.4.90-9 or older.
@@ -65,7 +65,7 @@ main ()

   cwd = getcwd (NULL, 0);
   if (cwd == NULL)
-    return 0;
+    return 2;

   initial_cwd_len = strlen (cwd);
   free (cwd);
@@ -92,15 +92,22 @@ main ()
   while (0 < d--)
     {
       if (chdir ("..") < 0)
-        break;
+        {
+          fail = 5;
+          break;
+        }
       rmdir (dir_name);
     }

-  return 0;
+  return fail;
 }
           ]])],
     [gl_cv_func_getcwd_abort_bug=no],
-    [gl_cv_func_getcwd_abort_bug=yes],
+    dnl A "regular" nonzero return does not indicate this bug.
+    dnl An abort will provoke an exit code of something like 134 (128 + 6).
+    [test $? -gt 128 \
+      && gl_cv_func_getcwd_abort_bug=yes \
+      || gl_cv_func_getcwd_abort_bug=no],
     [gl_cv_func_getcwd_abort_bug=yes])
   ])
   AS_IF([test $gl_cv_func_getcwd_abort_bug = yes], [$1], [$2])
diff --git a/tests/test-getcwd.c b/tests/test-getcwd.c
index 0e0f90f..14a526f 100644
--- a/tests/test-getcwd.c
+++ b/tests/test-getcwd.c
@@ -55,7 +55,7 @@ test_abort_bug (void)

   cwd = getcwd (NULL, 0);
   if (cwd == NULL)
-    return 0;
+    return 2;

   initial_cwd_len = strlen (cwd);
   free (cwd);
@@ -82,11 +82,14 @@ test_abort_bug (void)
   while (0 < d--)
     {
       if (chdir ("..") < 0)
-        break;
+        {
+          fail = 5;
+          break;
+        }
       rmdir (dir_name);
     }

-  return 0;
+  return fail;
 }

 /* The length of this name must be 8.  */
--
1.7.5.421.g9d34e


From MAILER-DAEMON Fri Apr 29 05:03:22 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFjbe-0002VC-Al
	for mharc-coreutils@gnu.org; Fri, 29 Apr 2011 05:03:22 -0400
Received: from eggs.gnu.org ([140.186.70.92]:34790)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFjbc-0002SA-30
	for coreutils@gnu.org; Fri, 29 Apr 2011 05:03:21 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFjbb-0003Xs-0w
	for coreutils@gnu.org; Fri, 29 Apr 2011 05:03:20 -0400
Received: from mx.meyering.net ([82.230.74.64]:58843)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFjba-0003XD-Qh
	for coreutils@gnu.org; Fri, 29 Apr 2011 05:03:18 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 79B54601D4; Fri, 29 Apr 2011 11:03:17 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] tests: distribute new file, CuSkip.pm
Date: Fri, 29 Apr 2011 11:03:17 +0200
Message-ID: <87mxj9v38q.fsf@rho.meyering.net>
Lines: 28
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Apr 2011 09:03:21 -0000

Oops.  Without this change, "make distcheck" would fail
due to lack of the new file:

>From 1346856368e675e2c085349674d3348c99dfcd55 Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Fri, 29 Apr 2011 10:31:09 +0200
Subject: [PATCH] tests: distribute new file, CuSkip.pm

Without this, most perl-based tests would fail in "make distcheck".
* tests/Makefile.am (EXTRA_DIST): Add CuSkip.pm
---
 tests/Makefile.am |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 1a80f84..52417bc 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -7,6 +7,7 @@ ASSORT = LC_ALL=C sort

 EXTRA_DIST =		\
   Coreutils.pm		\
+  CuSkip.pm		\
   CuTmpdir.pm		\
   check.mk		\
   envvar-check		\
--
1.7.5.421.g9d34e


From MAILER-DAEMON Fri Apr 29 05:06:39 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QFjep-0003c3-6B
	for mharc-coreutils@gnu.org; Fri, 29 Apr 2011 05:06:39 -0400
Received: from eggs.gnu.org ([140.186.70.92]:60033)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFjel-0003VM-Qx
	for coreutils@gnu.org; Fri, 29 Apr 2011 05:06:37 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFjel-00045T-3N
	for coreutils@gnu.org; Fri, 29 Apr 2011 05:06:35 -0400
Received: from mx.meyering.net ([82.230.74.64]:51017)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QFjek-00045J-Sl
	for coreutils@gnu.org; Fri, 29 Apr 2011 05:06:35 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id EE7EA600E1; Fri, 29 Apr 2011 11:06:33 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: Eric Blake <eblake@redhat.com>
Subject: Re: [PATCH] build: update to latest gnulib
In-Reply-To: <1304025435-22407-1-git-send-email-eblake@redhat.com> (Eric
	Blake's message of "Thu, 28 Apr 2011 15:17:15 -0600")
References: <1304025435-22407-1-git-send-email-eblake@redhat.com>
Date: Fri, 29 Apr 2011 11:06:33 +0200
Message-ID: <87hb9hv33a.fsf@rho.meyering.net>
Lines: 18
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Fri, 29 Apr 2011 09:06:37 -0000

Eric Blake wrote:
> * gnulib: Update to latest.
> * bootstrap.conf (gnulib_modules): Add xgetgroups.
> ---
>
> Okay to push?

> diff --git a/bootstrap.conf b/bootstrap.conf
...
>    xgetcwd
> +  xgetgroups
> diff --git a/gnulib b/gnulib
...
> -Subproject commit a81348d2e136b0f2de110b49a5d0dfa5d7d76f71
> +Subproject commit d6d01fdde58da00d0d839711097e3acb83f61754

With my two additional fixes in gnulib and the CuSkip.pm addition,
coreutils' "make distcheck" passes, so I've adjusted and pushed this change.


From MAILER-DAEMON Sat Apr 30 09:21:39 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGA79-0005Ip-F9
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:21:39 -0400
Received: from eggs.gnu.org ([140.186.70.92]:51811)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGA78-0005Ij-2B
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:21:38 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGA77-0007CS-0m
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:21:38 -0400
Received: from mx.meyering.net ([82.230.74.64]:49202)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGA76-0007CO-Ld
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:21:36 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 5787160355; Sat, 30 Apr 2011 15:21:35 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH] maint: adjust split.c formatting to conform
Date: Sat, 30 Apr 2011 15:21:35 +0200
Message-ID: <87d3k3nacg.fsf@rho.meyering.net>
Lines: 92
MIME-Version: 1.0
Content-Type: text/plain
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:21:39 -0000

FYI, I've just pushed this.  No semantic change:

>From b83d2896854de3823665d32630fb68e2c887c17b Mon Sep 17 00:00:00 2001
From: Jim Meyering <meyering@redhat.com>
Date: Sat, 30 Apr 2011 07:48:08 +0200
Subject: [PATCH] maint: adjust split.c formatting to conform

* src/split.c (usage): Correct indentation.
(ofile_open): Likewise.
(create): "char *name", not "char* name".
(struct of_info) [ofile]: Similar.
(parse_chunk): Add spaces around "+".
---
 src/split.c |   20 ++++++++++----------
 1 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/src/split.c b/src/split.c
index 3a630a0..65e44dd 100644
--- a/src/split.c
+++ b/src/split.c
@@ -105,7 +105,7 @@ static struct option const longopts[] =
   {"numeric-suffixes", no_argument, NULL, 'd'},
   {"verbose", no_argument, NULL, VERBOSE_OPTION},
   {"-io-blksize", required_argument, NULL,
-    IO_BLKSIZE_OPTION}, /* do not document */
+   IO_BLKSIZE_OPTION}, /* do not document */
   {GETOPT_HELP_OPTION_DECL},
   {GETOPT_VERSION_OPTION_DECL},
   {NULL, 0, NULL, 0}
@@ -155,7 +155,7 @@ usage (int status)
 Usage: %s [OPTION]... [INPUT [PREFIX]]\n\
 "),
               program_name);
-    fputs (_("\
+      fputs (_("\
 Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default\n\
 size is 1000 lines, and default PREFIX is `x'.  With no INPUT, or when INPUT\n\
 is -, read standard input.\n\
@@ -181,7 +181,7 @@ Mandatory arguments to long options are mandatory for short options too.\n\
       fputs (HELP_OPTION_DESCRIPTION, stdout);
       fputs (VERSION_OPTION_DESCRIPTION, stdout);
       emit_size_note ();
-fputs (_("\n\
+      fputs (_("\n\
 CHUNKS may be:\n\
 N       split into N files based on size of input\n\
 K/N     output Kth of N to stdout\n\
@@ -254,7 +254,7 @@ next_file_name (void)
 /* Create or truncate a file.  */

 static int
-create (const char* name)
+create (const char *name)
 {
   if (verbose)
     fprintf (stdout, _("creating file %s\n"), quote (name));
@@ -574,7 +574,7 @@ typedef struct of_info
 {
   char *of_name;
   int ofd;
-  FILE* ofile;
+  FILE *ofile;
 } of_t;

 enum
@@ -616,9 +616,9 @@ ofile_open (of_t *files, size_t i_check, size_t nfiles)
                  from the fifo, terminating on conditions in the data
                  itself, or perhaps never in the case of `tail -f`.
                  I.E. for fifos it is valid to attempt this reopen.  */
-               fd = open (files[i_check].of_name,
-                          O_WRONLY | O_BINARY | O_APPEND | O_NONBLOCK);
-             }
+              fd = open (files[i_check].of_name,
+                         O_WRONLY | O_BINARY | O_APPEND | O_NONBLOCK);
+            }

           if (-1 < fd)
             break;
@@ -779,9 +779,9 @@ static void
 parse_chunk (uintmax_t *k_units, uintmax_t *n_units, char *slash)
 {
   *slash = '\0';
-  if (xstrtoumax (slash+1, NULL, 10, n_units, "") != LONGINT_OK
+  if (xstrtoumax (slash + 1, NULL, 10, n_units, "") != LONGINT_OK
       || *n_units == 0)
-    error (EXIT_FAILURE, 0, _("%s: invalid number of chunks"), slash+1);
+    error (EXIT_FAILURE, 0, _("%s: invalid number of chunks"), slash + 1);
   if (slash != optarg           /* a leading number is specified.  */
       && (xstrtoumax (optarg, NULL, 10, k_units, "") != LONGINT_OK
           || *k_units == 0 || *n_units < *k_units))
--
1.7.5.134.g1c08b


From MAILER-DAEMON Sat Apr 30 09:31:21 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGX-00062A-BG
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:21 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35269)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0005zp-9d
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000CD-CL
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:18 -0400
Received: from mx.meyering.net ([82.230.74.64]:35094)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000Bz-3q
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:17 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id EE7186038B; Sat, 30 Apr 2011 15:31:15 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: split --filter, nearly ready
Date: Sat, 30 Apr 2011 15:31:01 +0200
Message-Id: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:19 -0000

Here's what I have so far.

One remaining task is to write the ChangeLog entry for
Karl's initial change.

[1/8] split: accept new output --filter option
[2/8] split: remove short-named -f option; improve diagnostics
[3/8] split: add tests
[4/8] split: update NEWS
[5/8] split: mark a string for translation
[6/8] strip: style: use braces around 1-stmt-but-multi-line
[7/8] split: don't rely on knowledge that INT_BUFSIZE_BOUND <=
[8/8] doc: document split's new --filter=CMD option

 b/NEWS               |   10 ++
 b/doc/coreutils.texi |   24 +++++
 b/src/split.c        |  211 +++++++++++++++++++++++++++++++++++++++++++++------
 b/tests/Makefile.am  |    1
 b/tests/split/filter |   41 +++++++++
 src/split.c          |   42 ++++++----
 6 files changed, 287 insertions(+), 42 deletions(-)

I've already merged a few small changes into Karl's initial 1/8,
and will merge others (probably 2, 5, 6, 7), but have left them
separate for now to facilitate review.

One oddity that I have not documented is how --elide-empty-files
works with --filter, but I did add a test to demonstrate
the existing behavior:

  # Show how --elide-empty-files works with --filter:
  # split does not run the command (and effectively elides the file)
  # only when the output to that command would have been empty.
  split -e -n 10 --filter='xz > $FILE.xz' /dev/null || fail=1
  stat x?? 2>/dev/null && fail=1

Feedback welcome.


From MAILER-DAEMON Sat Apr 30 09:31:22 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGX-00063F-UW
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:22 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35290)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0005zt-W1
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000CJ-DC
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:18 -0400
Received: from mx.meyering.net ([82.230.74.64]:35097)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000C5-3z
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:17 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 3523660355; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 3/8] split: add tests
Date: Sat, 30 Apr 2011 15:31:04 +0200
Message-Id: <1304170269-17085-4-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:20 -0000

From: Jim Meyering <meyering@redhat.com>

* tests/Makefile.am (TESTS): Add split/filter.
* tests/split/filter: New file.
---
 tests/Makefile.am  |    1 +
 tests/split/filter |   41 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+), 0 deletions(-)
 create mode 100755 tests/split/filter

diff --git a/tests/Makefile.am b/tests/Makefile.am
index 52417bc..6a8b79d 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -247,6 +247,7 @@ TESTS =						\
   misc/sort-unique				\
   misc/sort-unique-segv				\
   misc/sort-version				\
+  split/filter					\
   misc/split-a					\
   misc/split-bchunk				\
   misc/split-fail				\
diff --git a/tests/split/filter b/tests/split/filter
new file mode 100755
index 0000000..52075f9
--- /dev/null
+++ b/tests/split/filter
@@ -0,0 +1,41 @@
+#!/bin/sh
+# Exercise split's new --filter option.
+
+# Copyright (C) 2011 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. "${srcdir=.}/init.sh"; path_prepend_ ../src
+print_ver_ split
+
+for total_n_lines in 5 3000 20000; do
+  for i in 2 51 598; do
+    # Don't create too many files/processes.
+    case $i:$total_n_lines in 2:5);; *) continue;; esac
+
+    seq $total_n_lines > in || framework_failure_
+    split -l$i --filter='xz > $FILE.xz' in out- || fail=1
+    xz -dc out-* > out || fail=1
+    compare in out || fail=1
+    rm -f in out*
+  done
+done
+
+# Show how --elide-empty-files works with --filter:
+# split does not run the command (and effectively elides the file)
+# only when the output to that command would have been empty.
+split -e -n 10 --filter='xz > $FILE.xz' /dev/null || fail=1
+stat x?? 2>/dev/null && fail=1
+
+Exit $fail
-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:22 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGY-000640-AY
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:22 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35314)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGV-0005zw-H8
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000DO-Es
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:19 -0400
Received: from mx.meyering.net ([82.230.74.64]:35102)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000Co-8J
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:18 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 790DA604CB; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 6/8] strip: style: use braces around 1-stmt-but-multi-line
	blocks
Date: Sat, 30 Apr 2011 15:31:07 +0200
Message-Id: <1304170269-17085-7-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:20 -0000

From: Jim Meyering <meyering@redhat.com>

* src/strip.c (closeout): As above, per guidelines in HACKING.
---
 src/split.c |   17 +++++++++++------
 1 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/src/split.c b/src/split.c
index 4aa7d79..bd63fcf 100644
--- a/src/split.c
+++ b/src/split.c
@@ -354,9 +354,12 @@ closeout (FILE *fp, int fd, pid_t pid, char const *name)
         error (EXIT_FAILURE, errno, "%s", name);
       int j;
       for (j = 0; j < n_open_pipes; ++j)
-        if (open_pipes[j] == fd) {
-          open_pipes[j] = open_pipes[--n_open_pipes];
-          break;
+        {
+          if (open_pipes[j] == fd)
+            {
+              open_pipes[j] = open_pipes[--n_open_pipes];
+              break;
+            }
         }
     }
   if (pid > 0)
@@ -385,9 +388,11 @@ closeout (FILE *fp, int fd, pid_t pid, char const *name)
                    name, ex, filter_command);
         }
       else
-        /* shouldn't happen.  */
-        error (EXIT_FAILURE, 0,
-               _("unknown status from command (0x%X)"), wstatus);
+        {
+          /* shouldn't happen.  */
+          error (EXIT_FAILURE, 0,
+                 _("unknown status from command (0x%X)"), wstatus);
+        }
     }
 }

-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:23 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGZ-00066j-E2
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:23 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35331)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGV-0005zx-Sd
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000CX-KU
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:19 -0400
Received: from mx.meyering.net ([82.230.74.64]:35096)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000C3-3s
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:17 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 2793060391; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 2/8] split: remove short-named -f option; improve diagnostics
Date: Sat, 30 Apr 2011 15:31:03 +0200
Message-Id: <1304170269-17085-3-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:20 -0000

From: Jim Meyering <meyering@redhat.com>

* src/split.c (FILTER_OPTION): New anonymous enum member.
(longopts): Use it in place of 'f'.
(main): Likewise.
(create): Use "failed to ...", rather than "cannot ..." in diagnostics.
Include the full command in diagnostic when it fails to run.
---
 src/split.c |   20 +++++++++++---------
 1 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/src/split.c b/src/split.c
index 7aab357..59473c5 100644
--- a/src/split.c
+++ b/src/split.c
@@ -108,6 +108,7 @@ enum Split_type
 enum
 {
   VERBOSE_OPTION = CHAR_MAX + 1,
+  FILTER_OPTION,
   IO_BLKSIZE_OPTION
 };

@@ -121,7 +122,7 @@ static struct option const longopts[] =
   {"unbuffered", no_argument, NULL, 'u'},
   {"suffix-length", required_argument, NULL, 'a'},
   {"numeric-suffixes", no_argument, NULL, 'd'},
-  {"filter", required_argument, NULL, 'f'},
+  {"filter", required_argument, NULL, FILTER_OPTION},
   {"verbose", no_argument, NULL, VERBOSE_OPTION},
   {"-io-blksize", required_argument, NULL,
    IO_BLKSIZE_OPTION}, /* do not document */
@@ -189,7 +190,7 @@ Mandatory arguments to long options are mandatory for short options too.\n\
   -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file\n\
   -d, --numeric-suffixes  use numeric suffixes instead of alphabetic\n\
   -e, --elide-empty-files  do not generate empty output files with `-n'\n\
-  -f, --filter=COMMAND    write to shell COMMAND; file name is $FILE\n\
+      --filter=COMMAND    write to shell COMMAND; file name is $FILE\n\
   -l, --lines=NUMBER      put NUMBER lines per output file\n\
   -n, --number=CHUNKS     generate CHUNKS output files.  See below\n\
   -u, --unbuffered        immediately copy input to output with `-n r/...'\n\
@@ -291,11 +292,11 @@ create (const char *name)
       if (shell_prog == NULL)
         shell_prog = "/bin/sh";
       if (setenv ("FILE", name, 1) != 0)
-        error (EXIT_FAILURE, errno, "cannot set environment variable");
+        error (EXIT_FAILURE, errno, "failed to set FILE environment variable");
       if (verbose)
         fprintf (stdout, _("executing with FILE=%s\n"), quote (name));
       if (pipe (fd_pair) != 0)
-        error (EXIT_FAILURE, errno, _("cannot create pipe"));
+        error (EXIT_FAILURE, errno, _("failed to create pipe"));
       child_pid = fork ();
       if (child_pid == 0)
         {
@@ -322,12 +323,13 @@ create (const char *name)
           sigprocmask (SIG_SETMASK, &oldblocked, NULL);
           execl (shell_prog, last_component (shell_prog), "-c",
                  filter_command, (char *) NULL);
-          error (EXIT_FAILURE, errno, "%s", shell_prog);
+          error (EXIT_FAILURE, errno, _("failed to run command: \"%s -c %s\""),
+                 shell_prog, filter_command);
         }
       if (child_pid == -1)
-        error (EXIT_FAILURE, errno, _("cannot spawn new process"));
+        error (EXIT_FAILURE, errno, _("fork system call failed"));
       if (close (fd_pair[0]) != 0)
-        error (EXIT_FAILURE, errno, _("closing input pipe"));
+        error (EXIT_FAILURE, errno, _("failed to close input pipe"));
       filter_pid = child_pid;
       if (n_open_pipes == open_pipes_alloc)
         open_pipes = x2nrealloc (open_pipes, &open_pipes_alloc,
@@ -971,7 +973,7 @@ main (int argc, char **argv)
       int this_optind = optind ? optind : 1;
       char *slash;

-      c = getopt_long (argc, argv, "0123456789C:a:b:def:l:n:u",
+      c = getopt_long (argc, argv, "0123456789C:a:b:del:n:u",
                        longopts, NULL);
       if (c == -1)
         break;
@@ -1103,7 +1105,7 @@ main (int argc, char **argv)
           elide_empty_files = true;
           break;

-        case 'f':
+        case FILTER_OPTION:
           filter_command = optarg;
           break;

-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:26 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGb-0006Cx-UN
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:25 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35337)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGW-00060E-Ai
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:22 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000Cd-Po
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from mx.meyering.net ([82.230.74.64]:35095)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000By-3o
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:17 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 09AF660029; Sat, 30 Apr 2011 15:31:15 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 1/8] split: accept new output --filter option
Date: Sat, 30 Apr 2011 15:31:02 +0200
Message-Id: <1304170269-17085-2-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:22 -0000

From: Karl Heuer <kwzh@gnu.org>

* src/split.c (FIXME): FIXME
---
 src/split.c |  211 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 188 insertions(+), 23 deletions(-)

diff --git a/src/split.c b/src/split.c
index 65e44dd..7aab357 100644
--- a/src/split.c
+++ b/src/split.c
@@ -25,7 +25,9 @@
 #include <assert.h>
 #include <stdio.h>
 #include <getopt.h>
+#include <signal.h>
 #include <sys/types.h>
+#include <sys/wait.h>

 #include "system.h"
 #include "error.h"
@@ -35,6 +37,7 @@
 #include "full-write.h"
 #include "quote.h"
 #include "safe-read.h"
+#include "sig2str.h"
 #include "xfreopen.h"
 #include "xstrtol.h"

@@ -45,6 +48,21 @@
   proper_name_utf8 ("Torbjorn Granlund", "Torbj\303\266rn Granlund"), \
   proper_name ("Richard M. Stallman")

+/* Shell command to filter through, instead of creating files.  */
+static char const *filter_command;
+
+/* Process ID of the filter.  */
+static int filter_pid;
+
+/* Array of open pipes.  */
+static int *open_pipes;
+static size_t open_pipes_alloc;
+static size_t n_open_pipes;
+
+/* Blocked signals.  */
+static sigset_t oldblocked;
+static sigset_t newblocked;
+
 /* Base name of output files.  */
 static char const *outbase;

@@ -103,6 +121,7 @@ static struct option const longopts[] =
   {"unbuffered", no_argument, NULL, 'u'},
   {"suffix-length", required_argument, NULL, 'a'},
   {"numeric-suffixes", no_argument, NULL, 'd'},
+  {"filter", required_argument, NULL, 'f'},
   {"verbose", no_argument, NULL, VERBOSE_OPTION},
   {"-io-blksize", required_argument, NULL,
    IO_BLKSIZE_OPTION}, /* do not document */
@@ -170,6 +189,7 @@ Mandatory arguments to long options are mandatory for short options too.\n\
   -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file\n\
   -d, --numeric-suffixes  use numeric suffixes instead of alphabetic\n\
   -e, --elide-empty-files  do not generate empty output files with `-n'\n\
+  -f, --filter=COMMAND    write to shell COMMAND; file name is $FILE\n\
   -l, --lines=NUMBER      put NUMBER lines per output file\n\
   -n, --number=CHUNKS     generate CHUNKS output files.  See below\n\
   -u, --unbuffered        immediately copy input to output with `-n r/...'\n\
@@ -256,10 +276,116 @@ next_file_name (void)
 static int
 create (const char *name)
 {
-  if (verbose)
-    fprintf (stdout, _("creating file %s\n"), quote (name));
-  return open (name, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
-               (S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH));
+  if (!filter_command)
+    {
+      if (verbose)
+        fprintf (stdout, _("creating file %s\n"), quote (name));
+      return open (name, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY,
+                   (S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH | S_IWOTH));
+    }
+  else
+    {
+      int fd_pair[2];
+      pid_t child_pid;
+      char const *shell_prog = getenv ("SHELL");
+      if (shell_prog == NULL)
+        shell_prog = "/bin/sh";
+      if (setenv ("FILE", name, 1) != 0)
+        error (EXIT_FAILURE, errno, "cannot set environment variable");
+      if (verbose)
+        fprintf (stdout, _("executing with FILE=%s\n"), quote (name));
+      if (pipe (fd_pair) != 0)
+        error (EXIT_FAILURE, errno, _("cannot create pipe"));
+      child_pid = fork ();
+      if (child_pid == 0)
+        {
+          /* This is the child process.  If an error occurs here, the
+             parent will eventually learn about it after doing a wait,
+             at which time it will emit its own error message.  */
+          int j;
+          /* We have to close any pipes that were opened during an
+             earlier call, otherwise this process will be holding a
+             write-pipe that will prevent the earlier process from
+             reading an EOF on the corresponding read-pipe.  */
+          for (j = 0; j < n_open_pipes; ++j)
+            if (close (open_pipes[j]) != 0)
+              error (EXIT_FAILURE, errno, _("closing prior pipe"));
+          if (close (fd_pair[1]))
+            error (EXIT_FAILURE, errno, _("closing output pipe"));
+          if (fd_pair[0] != STDIN_FILENO)
+            {
+              if (dup2 (fd_pair[0], STDIN_FILENO) != STDIN_FILENO)
+                error (EXIT_FAILURE, errno, _("moving input pipe"));
+              if (close (fd_pair[0]) != 0)
+                error (EXIT_FAILURE, errno, _("closing input pipe"));
+            }
+          sigprocmask (SIG_SETMASK, &oldblocked, NULL);
+          execl (shell_prog, last_component (shell_prog), "-c",
+                 filter_command, (char *) NULL);
+          error (EXIT_FAILURE, errno, "%s", shell_prog);
+        }
+      if (child_pid == -1)
+        error (EXIT_FAILURE, errno, _("cannot spawn new process"));
+      if (close (fd_pair[0]) != 0)
+        error (EXIT_FAILURE, errno, _("closing input pipe"));
+      filter_pid = child_pid;
+      if (n_open_pipes == open_pipes_alloc)
+        open_pipes = x2nrealloc (open_pipes, &open_pipes_alloc,
+                                 sizeof *open_pipes);
+      open_pipes[n_open_pipes++] = fd_pair[1];
+      return fd_pair[1];
+    }
+}
+
+/* Close the output file, and do any associated cleanup.
+   If FP and FD are both specified, they refer to the same open file;
+   in this case FP is closed, but FD is still used in cleanup.  */
+static void
+closeout (FILE *fp, int fd, pid_t pid, char const *name)
+{
+  if (fp != NULL && fclose (fp) != 0 && errno != EPIPE)
+    error (EXIT_FAILURE, errno, "%s", name);
+  if (fd >= 0)
+    {
+      if (fp == NULL && close (fd) < 0)
+        error (EXIT_FAILURE, errno, "%s", name);
+      int j;
+      for (j = 0; j < n_open_pipes; ++j)
+        if (open_pipes[j] == fd) {
+          open_pipes[j] = open_pipes[--n_open_pipes];
+          break;
+        }
+    }
+  if (pid > 0)
+    {
+      int wstatus = 0;
+      if (waitpid (pid, &wstatus, 0) == -1 && errno != ECHILD)
+        error (EXIT_FAILURE, errno, _("waiting for child process"));
+      if (WIFSIGNALED (wstatus))
+        {
+          int sig = WTERMSIG (wstatus);
+          if (sig != SIGPIPE)
+            {
+              char signame[SIG2STR_MAX+1];
+              if (sig2str (sig, signame) != 0)
+                sprintf (signame, "%d", sig);
+              error (sig + 128, 0,
+                     _("with FILE=%s, signal %s (%s) from command: %s"),
+                     name, signame, strsignal (sig), filter_command);
+            }
+        }
+      else if (WIFEXITED (wstatus))
+        {
+          int ex = WEXITSTATUS (wstatus);
+          if (ex != 0)
+            error (ex, 0, _("with FILE=%s, exit %d from command: %s"),
+                   name, ex, filter_command);
+        }
+      else
+        /* shouldn't happen.  */
+        error (EXIT_FAILURE, 0,
+               _("unknown status from command (0x%X)"), wstatus);
+    }
 }

 /* Write BYTES bytes at BP to an output file.
@@ -273,13 +399,12 @@ cwrite (bool new_file_flag, const char *bp, size_t bytes)
     {
       if (!bp && bytes == 0 && elide_empty_files)
         return;
-      if (output_desc >= 0 && close (output_desc) < 0)
-        error (EXIT_FAILURE, errno, "%s", outfile);
+      closeout (NULL, output_desc, filter_pid, outfile);
       next_file_name ();
       if ((output_desc = create (outfile)) < 0)
         error (EXIT_FAILURE, errno, "%s", outfile);
     }
-  if (full_write (output_desc, bp, bytes) != bytes)
+  if (full_write (output_desc, bp, bytes) != bytes && errno != EPIPE)
     error (EXIT_FAILURE, errno, "%s", outfile);
 }

@@ -501,7 +626,8 @@ lines_chunk_split (uintmax_t k, uintmax_t n, char *buf, size_t bufsize,
               /* We don't use the stdout buffer here since we're writing
                  large chunks from an existing file, so it's more efficient
                  to write out directly.  */
-              if (full_write (STDOUT_FILENO, bp, to_write) != to_write)
+              if (full_write (STDOUT_FILENO, bp, to_write) != to_write
+                  && errno != EPIPE)
                 error (EXIT_FAILURE, errno, "%s", _("write error"));
             }
           else
@@ -564,7 +690,7 @@ bytes_chunk_extract (uintmax_t k, uintmax_t n, char *buf, size_t bufsize,
         error (EXIT_FAILURE, errno, "%s", infile);
       else if (n_read == 0)
         break; /* eof.  */
-      if (full_write (STDOUT_FILENO, buf, n_read) != n_read)
+      if (full_write (STDOUT_FILENO, buf, n_read) != n_read && errno != EPIPE)
         error (EXIT_FAILURE, errno, "%s", quote ("-"));
       start += n_read;
     }
@@ -575,6 +701,7 @@ typedef struct of_info
   char *of_name;
   int ofd;
   FILE *ofile;
+  int opid;
 } of_t;

 enum
@@ -637,14 +764,17 @@ ofile_open (of_t *files, size_t i_check, size_t nfiles)
                 error (EXIT_FAILURE, errno, "%s", files[i_check].of_name);
             }

-          if (fclose (files[i_reopen].ofile) != 0)
+          if (fclose (files[i_reopen].ofile) != 0 && errno != EPIPE)
             error (EXIT_FAILURE, errno, "%s", files[i_reopen].of_name);
+          files[i_reopen].ofile = NULL;
           files[i_reopen].ofd = OFD_APPEND;
         }

       files[i_check].ofd = fd;
       if (!(files[i_check].ofile = fdopen (fd, "a")))
         error (EXIT_FAILURE, errno, "%s", files[i_check].of_name);
+      files[i_check].opid = filter_pid;
+      filter_pid = 0;
     }

   return file_limit;
@@ -658,6 +788,7 @@ ofile_open (of_t *files, size_t i_check, size_t nfiles)
 static void
 lines_rr (uintmax_t k, uintmax_t n, char *buf, size_t bufsize)
 {
+  bool wrapped = false;
   bool file_limit;
   size_t i_file;
   of_t *files IF_LINT (= NULL);
@@ -678,6 +809,7 @@ lines_rr (uintmax_t k, uintmax_t n, char *buf, size_t bufsize)
           files[i_file].of_name = xstrdup (outfile);
           files[i_file].ofd = OFD_NEW;
           files[i_file].ofile = NULL;
+          files[i_file].opid = 0;
         }
       i_file = 0;
       file_limit = false;
@@ -715,10 +847,12 @@ lines_rr (uintmax_t k, uintmax_t n, char *buf, size_t bufsize)
             {
               if (line_no == k && unbuffered)
                 {
-                  if (full_write (STDOUT_FILENO, bp, to_write) != to_write)
+                  if (full_write (STDOUT_FILENO, bp, to_write) != to_write
+                      && errno != EPIPE)
                     error (EXIT_FAILURE, errno, "%s", _("write error"));
                 }
-              else if (line_no == k && fwrite (bp, to_write, 1, stdout) != 1)
+              else if (line_no == k && fwrite (bp, to_write, 1, stdout) != 1
+                       && errno != EPIPE)
                 {
                   clearerr (stdout); /* To silence close_stdout().  */
                   error (EXIT_FAILURE, errno, "%s", _("write error"));
@@ -734,19 +868,25 @@ lines_rr (uintmax_t k, uintmax_t n, char *buf, size_t bufsize)
                 {
                   /* Note writing to fd, rather than flushing the FILE gives
                      an 8% performance benefit, due to reduced data copying.  */
-                  if (full_write (files[i_file].ofd, bp, to_write) != to_write)
+                  if (full_write (files[i_file].ofd, bp, to_write) != to_write
+                      && errno != EPIPE)
                     error (EXIT_FAILURE, errno, "%s", files[i_file].of_name);
                 }
-              else if (fwrite (bp, to_write, 1, files[i_file].ofile) != 1)
+              else if (fwrite (bp, to_write, 1, files[i_file].ofile) != 1
+                       && errno != EPIPE)
                 error (EXIT_FAILURE, errno, "%s", files[i_file].of_name);
               if (file_limit)
                 {
-                  if (fclose (files[i_file].ofile) != 0)
+                  if (fclose (files[i_file].ofile) != 0 && errno != EPIPE)
                     error (EXIT_FAILURE, errno, "%s", files[i_file].of_name);
+                  files[i_file].ofile = NULL;
                   files[i_file].ofd = OFD_APPEND;
                 }
               if (next && ++i_file == n)
-                i_file = 0;
+                {
+                  wrapped = true;
+                  i_file = 0;
+                }
             }

           bp = bp_out;
@@ -757,11 +897,18 @@ lines_rr (uintmax_t k, uintmax_t n, char *buf, size_t bufsize)
      and to signal any waiting fifo consumers.
      Also, close any open file descriptors.
      FIXME: Should we do this before EXIT_FAILURE?  */
-  for (i_file = 0; !k && !elide_empty_files && i_file < n; i_file++)
+  if (!k)
     {
-      file_limit |= ofile_open (files, i_file, n);
-      if (fclose (files[i_file].ofile) != 0)
-        error (EXIT_FAILURE, errno, "%s", files[i_file].of_name);
+      int ceiling = (wrapped ? n : i_file);
+      for (i_file = 0; i_file < n; i_file++)
+        {
+          if (i_file >= ceiling && !elide_empty_files)
+            file_limit |= ofile_open (files, i_file, n);
+          if (files[i_file].ofd >= 0)
+            closeout (files[i_file].ofile, files[i_file].ofd,
+                      files[i_file].opid, files[i_file].of_name);
+          files[i_file].ofd = OFD_APPEND;
+        }
     }
 }

@@ -824,7 +971,8 @@ main (int argc, char **argv)
       int this_optind = optind ? optind : 1;
       char *slash;

-      c = getopt_long (argc, argv, "0123456789C:a:b:del:n:u", longopts, NULL);
+      c = getopt_long (argc, argv, "0123456789C:a:b:def:l:n:u",
+                       longopts, NULL);
       if (c == -1)
         break;

@@ -955,6 +1103,10 @@ main (int argc, char **argv)
           elide_empty_files = true;
           break;

+        case 'f':
+          filter_command = optarg;
+          break;
+
         case IO_BLKSIZE_OPTION:
           {
             uintmax_t tmp_blk_size;
@@ -1048,6 +1200,18 @@ main (int argc, char **argv)

   buf = ptr_align (xmalloc (in_blk_size + 1 + page_size - 1), page_size);

+  /* When filtering, closure of one pipe must not terminate the process,
+     as there may still be other streams expecting input from us.  */
+  sigemptyset (&newblocked);
+  if (filter_command)
+    {
+      struct sigaction act;
+      sigaction (SIGPIPE, NULL, &act);
+      if (act.sa_handler != SIG_IGN)
+        sigaddset (&newblocked, SIGPIPE);
+    }
+  sigprocmask (SIG_BLOCK, &newblocked, &oldblocked);
+
   switch (split_type)
     {
     case type_digits:
@@ -1084,10 +1248,11 @@ main (int argc, char **argv)
       abort ();
     }

+  sigprocmask (SIG_SETMASK, &oldblocked, NULL);
+
   if (close (STDIN_FILENO) != 0)
     error (EXIT_FAILURE, errno, "%s", infile);
-  if (output_desc >= 0 && close (output_desc) < 0)
-    error (EXIT_FAILURE, errno, "%s", outfile);
+  closeout (NULL, output_desc, filter_pid, outfile);

   exit (EXIT_SUCCESS);
 }
-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:21 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGX-00062S-N5
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:21 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35301)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGV-0005zu-6e
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000CO-Db
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:19 -0400
Received: from mx.meyering.net ([82.230.74.64]:35098)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGT-0000C7-41
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:17 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 531406047F; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 4/8] split: update NEWS
Date: Sat, 30 Apr 2011 15:31:05 +0200
Message-Id: <1304170269-17085-5-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:20 -0000

From: Jim Meyering <meyering@redhat.com>

* NEWS (New feature): Mention it.
---
 NEWS |   10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/NEWS b/NEWS
index c90e02f..82ce53c 100644
--- a/NEWS
+++ b/NEWS
@@ -2,6 +2,16 @@ GNU coreutils NEWS                                    -*- outline -*-

 * Noteworthy changes in release ?.? (????-??-??) [?]

+** New features
+
+  split accepts a new --filter=CMD option.  With it, split filters output
+  through CMD.  CMD may use the $FILE environment variable, which is set to
+  the nominal output file name for each invocation of CMD.  For example, to
+  split a file into 3 approximately equal parts, which are then compressed:
+    split -n3 --filter='xz > $FILE.xz' big
+  Note the use of single quotes, not double quotes.
+  That creates files named xaa.xz, xab.xz and xac.xz.
+

 * Noteworthy changes in release 8.12 (2011-04-26) [stable]

-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:22 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGX-00062t-PL
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:21 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35309)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGV-0005zv-DX
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000D0-BD
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:19 -0400
Received: from mx.meyering.net ([82.230.74.64]:35100)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000Ci-5M
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:18 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 87C6160483; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 7/8] split: don't rely on knowledge that INT_BUFSIZE_BOUND <=
	SIG2STR_MAX
Date: Sat, 30 Apr 2011 15:31:08 +0200
Message-Id: <1304170269-17085-8-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:20 -0000

From: Jim Meyering <meyering@redhat.com>

* src/split.c (closeout): SIG2STR_MAX is a buffer size, so there's
no need to add 1.  However, since we may also write the decimal
representation of an "int" into this same buffer, include
INT_BUFSIZE_BOUND in the equation.
---
 src/split.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/src/split.c b/src/split.c
index bd63fcf..66f44fc 100644
--- a/src/split.c
+++ b/src/split.c
@@ -372,7 +372,7 @@ closeout (FILE *fp, int fd, pid_t pid, char const *name)
           int sig = WTERMSIG (wstatus);
           if (sig != SIGPIPE)
             {
-              char signame[SIG2STR_MAX+1];
+              char signame[MAX (SIG2STR_MAX, INT_BUFSIZE_BOUND (int))];
               if (sig2str (sig, signame) != 0)
                 sprintf (signame, "%d", sig);
               error (sig + 128, 0,
-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:23 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGZ-00066x-JK
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:23 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35343)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGW-00060J-Fm
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:21 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000DG-E2
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from mx.meyering.net ([82.230.74.64]:35099)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000Ch-5P
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:18 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 5DFA160453; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 5/8] split: mark a string for translation
Date: Sat, 30 Apr 2011 15:31:06 +0200
Message-Id: <1304170269-17085-6-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:21 -0000

From: Jim Meyering <meyering@redhat.com>

---
 src/split.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/src/split.c b/src/split.c
index 59473c5..4aa7d79 100644
--- a/src/split.c
+++ b/src/split.c
@@ -292,7 +292,8 @@ create (const char *name)
       if (shell_prog == NULL)
         shell_prog = "/bin/sh";
       if (setenv ("FILE", name, 1) != 0)
-        error (EXIT_FAILURE, errno, "failed to set FILE environment variable");
+        error (EXIT_FAILURE, errno,
+               _("failed to set FILE environment variable"));
       if (verbose)
         fprintf (stdout, _("executing with FILE=%s\n"), quote (name));
       if (pipe (fd_pair) != 0)
-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 09:31:23 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGAGZ-00067S-Oh
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 09:31:23 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35348)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGW-00060K-KS
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:21 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000DB-Dw
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:20 -0400
Received: from mx.meyering.net ([82.230.74.64]:35101)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <meyering@mx.meyering.net>) id 1QGAGU-0000Cj-5O
	for coreutils@gnu.org; Sat, 30 Apr 2011 09:31:18 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 981B1604FE; Sat, 30 Apr 2011 15:31:16 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: coreutils@gnu.org
Subject: [PATCH 8/8] doc: document split's new --filter=CMD option
Date: Sat, 30 Apr 2011 15:31:09 +0200
Message-Id: <1304170269-17085-9-git-send-email-jim@meyering.net>
X-Mailer: git-send-email 1.7.4.4
In-Reply-To: <1304170269-17085-1-git-send-email-jim@meyering.net>
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 13:31:21 -0000

From: Jim Meyering <meyering@redhat.com>

* doc/coreutils.texi (split invocation): Describe --filter=CMD.
---
 doc/coreutils.texi |   24 ++++++++++++++++++++++--
 1 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index d2377f4..88e363b 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -2992,8 +2992,8 @@ split invocation
 Put @var{lines} lines of @var{input} into each output file.

 For compatibility @command{split} also supports an obsolete
-option syntax @option{-@var{lines}}.  New scripts should use @option{-l
-@var{lines}} instead.
+option syntax @option{-@var{lines}}.  New scripts should use
+@option{-l @var{lines}} instead.

 @item -b @var{size}
 @itemx --bytes=@var{size}
@@ -3011,6 +3011,26 @@ split invocation
 @var{size} bytes are broken into multiple files.
 @var{size} has the same format as for the @option{--bytes} option.

+@itemx --filter=@var{command}
+@opindex --filter
+With this option, rather than simply writing to each output file,
+@command{split} writes through a pipe to the specified shell @var{command}
+for each output file.
+@var{command} may use the $FILE environment variable, which is set
+to a different output file name for each invocation of the command.
+For example, imagine that you have a 1TiB compressed file
+that, if uncompressed, would be too large to reside on disk,
+yet you must split it into individually-compressed pieces
+of a more manageable size.
+To do that, you might run this command:
+
+@example
+xz -dc BIG.xz big- | split -b200G --filter='xz > $FILE.xz'
+@end example
+
+Assuming a 10:1 compression ratio, that would create about fifty 20GiB files
+with names @file{big-xaa.xz}, @file{big-xab.xz}, @file{big-xac.xz}, etc.
+
 @item -n @var{chunks}
 @itemx --number=@var{chunks}
 @opindex -n
-- 
1.7.5.134.g1c08b



From MAILER-DAEMON Sat Apr 30 12:07:57 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGCi5-0002lG-Sk
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 12:07:57 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35896)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGCi4-0002l8-11
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:07:56 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGCi3-0005aF-8w
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:07:55 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:14531)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGCi3-0005a2-0J
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:07:55 -0400
Received: (qmail 15556 invoked from network); 30 Apr 2011 16:07:53 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 30 Apr 2011 16:07:53 -0000
Message-ID: <4DBC33D2.8070403@draigBrady.com>
Date: Sat, 30 Apr 2011 17:07:46 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: [PATCH 6/8] strip: style: use braces around 1-stmt-but-multi-line
	blocks
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
	<1304170269-17085-7-git-send-email-jim@meyering.net>
In-Reply-To: <1304170269-17085-7-git-send-email-jim@meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 16:07:57 -0000

On 30/04/11 14:31, Jim Meyering wrote:
> From: Jim Meyering <meyering@redhat.com>
> 
> * src/strip.c (closeout): As above, per guidelines in HACKING.

s/strip/split/ in the subject and file name


From MAILER-DAEMON Sat Apr 30 12:21:20 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGCv2-00043D-AI
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 12:21:20 -0400
Received: from eggs.gnu.org ([140.186.70.92]:59791)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGCv0-00042v-96
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:21:19 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGCuz-00079r-6j
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:21:18 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:10082)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGCuy-00079m-QZ
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:21:17 -0400
Received: (qmail 17290 invoked from network); 30 Apr 2011 16:21:15 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 30 Apr 2011 16:21:15 -0000
Message-ID: <4DBC36F5.8000308@draigBrady.com>
Date: Sat, 30 Apr 2011 17:21:09 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: [PATCH 8/8] doc: document split's new --filter=CMD option
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
	<1304170269-17085-9-git-send-email-jim@meyering.net>
In-Reply-To: <1304170269-17085-9-git-send-email-jim@meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 16:21:19 -0000

On 30/04/11 14:31, Jim Meyering wrote:
> From: Jim Meyering <meyering@redhat.com>
> 
> * doc/coreutils.texi (split invocation): Describe --filter=CMD.
> ---
>  doc/coreutils.texi |   24 ++++++++++++++++++++++--
>  1 files changed, 22 insertions(+), 2 deletions(-)
> 
> diff --git a/doc/coreutils.texi b/doc/coreutils.texi
> index d2377f4..88e363b 100644
> --- a/doc/coreutils.texi
> +++ b/doc/coreutils.texi
> @@ -2992,8 +2992,8 @@ split invocation
>  Put @var{lines} lines of @var{input} into each output file.
> 
>  For compatibility @command{split} also supports an obsolete
> -option syntax @option{-@var{lines}}.  New scripts should use @option{-l
> -@var{lines}} instead.
> +option syntax @option{-@var{lines}}.  New scripts should use
> +@option{-l @var{lines}} instead.
> 
>  @item -b @var{size}
>  @itemx --bytes=@var{size}
> @@ -3011,6 +3011,26 @@ split invocation
>  @var{size} bytes are broken into multiple files.
>  @var{size} has the same format as for the @option{--bytes} option.
> 
> +@itemx --filter=@var{command}
> +@opindex --filter
> +With this option, rather than simply writing to each output file,
> +@command{split} writes through a pipe to the specified shell @var{command}

s/@command{split} writes/write/

> +for each output file.
> +@var{command} may use the $FILE environment variable, which is set
> +to a different output file name for each invocation of the command.
> +For example, imagine that you have a 1TiB compressed file
> +that, if uncompressed, would be too large to reside on disk,
> +yet you must split it into individually-compressed pieces
> +of a more manageable size.
> +To do that, you might run this command:
> +
> +@example
> +xz -dc BIG.xz big- | split -b200G --filter='xz > $FILE.xz'

xz -dc BIG.xz | split -b20G --filter='xz > $FILE.xz' - big-

> +@end example
> +
> +Assuming a 10:1 compression ratio, that would create about fifty 20GiB files
> +with names @file{big-xaa.xz}, @file{big-xab.xz}, @file{big-xac.xz}, etc.
> +
>  @item -n @var{chunks}
>  @itemx --number=@var{chunks}
>  @opindex -n



From MAILER-DAEMON Sat Apr 30 12:52:03 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGDOl-0006RV-PL
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 12:52:03 -0400
Received: from eggs.gnu.org ([140.186.70.92]:50560)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGDOk-0006RO-06
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:52:02 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGDOj-0002vH-4U
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:52:01 -0400
Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:2299)
	by eggs.gnu.org with smtp (Exim 4.71)
	(envelope-from <P@draigBrady.com>) id 1QGDOi-0002vB-QT
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:52:01 -0400
Received: (qmail 21214 invoked from network); 30 Apr 2011 16:51:59 -0000
Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218)
	by mail1.slb.deg.dub.stisp.net with SMTP; 30 Apr 2011 16:51:59 -0000
Message-ID: <4DBC3E28.4070009@draigBrady.com>
Date: Sat, 30 Apr 2011 17:51:52 +0100
From: =?ISO-8859-1?Q?P=E1draig_Brady?= <P@draigBrady.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
	rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3
MIME-Version: 1.0
To: Jim Meyering <jim@meyering.net>
Subject: Re: [PATCH 3/8] split: add tests
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
	<1304170269-17085-4-git-send-email-jim@meyering.net>
In-Reply-To: <1304170269-17085-4-git-send-email-jim@meyering.net>
X-Enigmail-Version: 1.0.1
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 8bit
X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9
X-Received-From: 84.203.253.98
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 16:52:03 -0000

On 30/04/11 14:31, Jim Meyering wrote:
> +for total_n_lines in 5 3000 20000; do
> +  for i in 2 51 598; do
> +    # Don't create too many files/processes.
> +    case $i:$total_n_lines in 2:5);; *) continue;; esac

So the other combos are only for developer debug?
A comment to clariy would be good.
Perhaps they could be enabled if RUN_VERY_EXPENSIVE_TESTS ?

> +
> +    seq $total_n_lines > in || framework_failure_

The `seq` line should be moved to the outer loop

> +    split -l$i --filter='xz > $FILE.xz' in out- || fail=1
> +    xz -dc out-* > out || fail=1
> +    compare in out || fail=1
> +    rm -f in out*

Ditto for `rm -f in`

> +  done
> +done

cheers,
Pádraig.


From MAILER-DAEMON Sat Apr 30 12:55:15 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGDRr-0006lF-2j
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 12:55:15 -0400
Received: from eggs.gnu.org ([140.186.70.92]:34941)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGDRq-0006l9-3V
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:55:14 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGDRp-0003aY-9x
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:55:14 -0400
Received: from mx.meyering.net ([82.230.74.64]:41200)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGDRp-0003aP-2L
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:55:13 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id 18EE960355; Sat, 30 Apr 2011 18:55:11 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: [PATCH 8/8] doc: document split's new --filter=CMD option
In-Reply-To: <4DBC36F5.8000308@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Sat, 30 Apr 2011 17:21:09 +0100")
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
	<1304170269-17085-9-git-send-email-jim@meyering.net>
	<4DBC36F5.8000308@draigBrady.com>
Date: Sat, 30 Apr 2011 18:55:10 +0200
Message-ID: <87tydfllw1.fsf@rho.meyering.net>
Lines: 28
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 16:55:14 -0000

P=E1draig Brady wrote:
> On 30/04/11 14:31, Jim Meyering wrote:
...
>> +@itemx --filter=3D@var{command}
>> +@opindex --filter
>> +With this option, rather than simply writing to each output file,
>> +@command{split} writes through a pipe to the specified shell @var{comma=
nd}
>
> s/@command{split} writes/write/

Thanks.  That sounds better.

>> +for each output file.
>> +@var{command} may use the $FILE environment variable, which is set
>> +to a different output file name for each invocation of the command.
>> +For example, imagine that you have a 1TiB compressed file
>> +that, if uncompressed, would be too large to reside on disk,
>> +yet you must split it into individually-compressed pieces
>> +of a more manageable size.
>> +To do that, you might run this command:
>> +
>> +@example
>> +xz -dc BIG.xz big- | split -b200G --filter=3D'xz > $FILE.xz'
>
> xz -dc BIG.xz | split -b20G --filter=3D'xz > $FILE.xz' - big-

That's definitely what I meant.
I've merged both.  Thanks again.


From MAILER-DAEMON Sat Apr 30 12:57:48 2011
Received: from list by lists.gnu.org with archive (Exim 4.71)
	id 1QGDUK-0007A7-RY
	for mharc-coreutils@gnu.org; Sat, 30 Apr 2011 12:57:48 -0400
Received: from eggs.gnu.org ([140.186.70.92]:35258)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGDUJ-0007A1-0P
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:57:47 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGDUI-0003wD-6N
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:57:46 -0400
Received: from mx.meyering.net ([82.230.74.64]:41415)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <jim@meyering.net>) id 1QGDUH-0003w7-VL
	for coreutils@gnu.org; Sat, 30 Apr 2011 12:57:46 -0400
Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000)
	id E88F36038B; Sat, 30 Apr 2011 18:57:44 +0200 (CEST)
From: Jim Meyering <jim@meyering.net>
To: =?iso-8859-1?Q?P=E1draig?= Brady <P@draigBrady.com>
Subject: Re: [PATCH 6/8] strip: style: use braces around 1-stmt-but-multi-line
	blocks
In-Reply-To: <4DBC33D2.8070403@draigBrady.com> (=?iso-8859-1?Q?=22P=E1drai?=
	=?iso-8859-1?Q?g?= Brady"'s message of
	"Sat, 30 Apr 2011 17:07:46 +0100")
References: <1304170269-17085-1-git-send-email-jim@meyering.net>
	<1304170269-17085-7-git-send-email-jim@meyering.net>
	<4DBC33D2.8070403@draigBrady.com>
Date: Sat, 30 Apr 2011 18:57:44 +0200
Message-ID: <87oc3nllrr.fsf@rho.meyering.net>
Lines: 14
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
	recognized.
X-Received-From: 82.230.74.64
Cc: coreutils@gnu.org
X-BeenThere: coreutils@gnu.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: GNU coreutils General Discussion <coreutils.gnu.org>
List-Unsubscribe: <https://lists.gnu.org/mailman/options/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=unsubscribe>
List-Archive: <http://lists.gnu.org/archive/html/coreutils>
List-Post: <mailto:coreutils@gnu.org>
List-Help: <mailto:coreutils-request@gnu.org?subject=help>
List-Subscribe: <https://lists.gnu.org/mailman/listinfo/coreutils>,
	<mailto:coreutils-request@gnu.org?subject=subscribe>
X-List-Received-Date: Sat, 30 Apr 2011 16:57:47 -0000

P=E1draig Brady wrote:
> On 30/04/11 14:31, Jim Meyering wrote:
>> From: Jim Meyering <meyering@redhat.com>
>>
>> * src/strip.c (closeout): As above, per guidelines in HACKING.
>
> s/strip/split/ in the subject and file name

Good catch.
Odd how that works.  Since then, I was trying to edit that file
and it took a couple seconds to realize why src/str<TAB> wasn't
showing any possible completions.

Thanks for the prompt reviews!


