Re: Bash scripting and large files: input with the read builtin from a r
From:
Chet Ramey
Subject:
Re: Bash scripting and large files: input with the read builtin from a redirection gives unexpected result with files larger than 2GB.
Date:
Sun, 04 Mar 2012 21:04:01 -0500
User-agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:8.0) Gecko/20111105 Thunderbird/8.0
On 3/2/12 6:47 AM, Jean-François Gagné wrote:
> Description:
> When reading data with the 'read' buildin from a redirection, read has
> unexpected behavior after reading 2G of data.
>
> Repeat-By:
>
>
> yes "0123456789abcdefghijklmnopqrs" | head -n 100000000 > file
> while read line; do file=${line:0:10}; echo $file; done < file | uniq -c
>
>
> results in
>
>
> 71582790 0123456789
> 1 mnopqrs
> 3 0123456789
> 1 mnopqrs
> 3 0123456789
> 1 mnopqrs
> 3 0123456789
> 1 mnopqrs
> 3 0123456789
> ...
>
> So the problem happens after reading 71.582.790 x30 = 2.147.483.700 bytes of
> data, just a little over 2^31.
>
> but the following:
>
> cat file | while read line; do file=${line:0:10}; echo $file; done | uniq -c
>
> works fine:
>
> 100000000 0123456789
This works fine with the patch I posted.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, ITS, CWRU chet@case.edu http://cnswww.cns.cwru.edu/~chet/