[PATCH] split: add content-defined chunking to --bytes
From:
Leonid Evdokimov
Subject:
[PATCH] split: add content-defined chunking to --bytes
Date:
Mon, 2 Mar 2026 15:32:10 +0300
Hello,
Here is the patch to implement CDC in `split --bytes`. I'm submitting
it for review before proceeding with adding CDC to --line-bytes.
I've tested the patch on x86_64, ppc64be and Apple M1, with gcc and clang.
Texinfo documentation is currently missing.
I'm mostly unsure about the following:
1) autoconf and l10n logic being right, as I'm not familiar with AC/AM
and gettext.
2) embedding PCG RNG into make-buz-table. Is there a better way to
accomplish the goal and is it a better way needed?
3) licensing/authorship headers. There might be guidelines I'm missing.
4) right place for getcachelinesize(). Should it be a separate file
and/or part of gnulib?
5) busy-loop of randperm_new() on random-source being stream of 0xFF.
On one hand, that's a "bug" in randint_choose() and randpem_bound(),
on the other hand - one may say that it's just a foot-shooting case.
6) moving `+1` byte allocation to be specific for lines_split(). I've
not run asan build to test correctness. +1 is there for 35 years and,
seems, lines_split() is the only user of that extra byte, but my eye
might miss something.
7) 40 MiB limit for 32-bit CDC hashes, it's tempting to say "42 MB".
Should we? :-)
I've tried to add enough comments to make the code easy to understand,
but I can add more if that's helpful as the memory is still fresh.
The patch patch is also available at github:
https://github.com/coreutils/coreutils/compare/master...darkk:coreutils:cdc
--
WBRBW, Leonid Evdokimov, https://darkk.net.ru tel:+79816800702
PGP: 6691 DE6B 4CCD C1C1 76A0 0D4A E1F2 A980 7F50 FAB2