bug#9321: repeated segfaults sorting large files in 8.12
From:
Andras Salamon
Subject:
bug#9321: repeated segfaults sorting large files in 8.12
Date:
Sat, 20 Aug 2011 21:58:57 +0100
User-agent:
Mutt/1.5.21 (2010-09-15)
On Fri, Aug 19, 2011 at 11:54:46PM +0100, PƔdraig Brady wrote:
On 08/18/2011 03:30 PM, Andras Salamon wrote:
I am seeing repeated (but not reliably repeatable) segmentation faults
sorting datasets in the 100MB-100GB range on a 64-bit Debian system
using GNU sort 8.12 (and also 8.9). Stack traces seem to indicate
problems during the merge phase, usually when the temporary files
are being combined.
Andras, could you give the exact command line your having issue with,
and perhaps make sort inputs available too?
The sort inputs are several-gigabyte-range files containing strings,
each typically 60 to 140 bytes long, one per line. There are
many duplicates, and the first reason to sort is to establish the
distribution of duplicates. I would be happy to make available data
if I could find a reasonably sized file that causes a reproducible
segfault. The problem seems easier to reproduce with larger files,
unfortunately.
Do the --batch-size=NMERGE or --compress-program=PROG options change anything?
Thanks for the suggestion, I will try forcing smaller batches.
Compressing batches was something I tried early on with no apparent
change in likelihood of failure, but it led to much slower runtimes.
Here are some of the relevant-seeming parts of a gdb session for
coreutils-7.1. Here ?.xz is a compressed file which has already been
sorted, around 35MB in size.
Built with: configure CFLAGS=-g --disable-nls
Commandline:
% nohup xzcat 1.xz 2.xz 3.xz 4.xz | sort -S 100M -T /home/a/tmp | xz > o.xz &
Segmentation fault ../bin/sort -T /home/a/tmp -S 100M | (core dumped)
During the run there were 435 temp files active at one point.
There may have been more at a later stage, but these were reduced
to a final 32 which remained after the crash. There is around 600GB
free disk space on this volume.
% du -smc sort* | tail -1
29556 total
% ls -sktr sort*
62776 sortR07gPu
62056 sortS3H1Mu
10848 sortECN8Nx
951020 sortlk9Xd1
1001668 sortrDhnFQ
1001420 sortItDvPu
1001216 sortIBlIVY
1001500 sortDWg5Vj
1012504 sortOulxqu
916424 sortOTNgnn
907976 sortRlRPsA
997840 sortuQbWXj
1001328 sortoWTS4K
1001436 sort3GpGf2
1001544 sortVudEk7
1009412 sortJou3Y3
926628 sortL2SeVF
950584 sortSTuAkJ
1001376 sortX9rCaf
1000928 sortAjXZkz
1001120 sortQzXcgK
1001412 sortLwoe9K
1012704 sortM4WHnD
955044 sort1c8ja8
981680 sortJhX3rd
1001040 sortqGq4yV
1000596 sort7obBHs
1000540 sortW4fLHR
1000800 sortSzB3s6
999624 sortMD7K0b
305892 sortqSxpe4
3183480 sortcOqzkh
(gdb) bt
#0 0x000000000040e6bc in memcoll (
(gdb) p address@hidden
$8 = {0, 8, 4, 9, 1, 5, 10, 2, 6, 3, 7}
(gdb) p ord0
$9 = 0
(gdb) p probe
$10 = 1
(gdb) p *(const struct line *)0x2a9ff30
$15 = {
text = 0x245ff30
"_:httpx3Ax2Fx2Fapix2Ehi5x2Ecomx2Frestx2Fprofilex2Ffoafx2F350598182xxbnode337",
length = 77, keybeg = 0x0, keylim = 0x0}
(gdb) p *(const struct line *)0x228b5e0
$16 = {text = 0x600000000226d720 <Address 0x600000000226d720 out of bounds>,
length = 14843864371813154892,
keybeg = 0x756566736f4e2f72 <Address 0x756566736f4e2f72 out of bounds>,
keylim = 0x66626f5f6f746c61 <Address 0x66626f5f6f746c61 out of bounds>}
(gdb) p *(const struct line *)0x75a0170
$18 = {
text = 0x6f60170
"_:httpx3Ax2Fx2Fapix2Ehi5x2Ecomx2Frestx2Fprofilex2Ffoafx2F492419832xxbnode215",
length = 77, keybeg = 0x0, keylim = 0x0}
(gdb) p *buffer
$33 = {