On Sat, Nov 19, 2005 at 10:49:07AM -0500, Phillip Susi wrote:
> I don't see why the filesystem's cluster size should have a thing to do
> with the buffer size used to copy files. For optimal performance, the
> larger the buffer, the better. Diminishing returns applies of course,
> so at some point the increase in buffer size results in little to no
> further increase in performance, so that's the size you should use. I
> believe that the optimal size is about 64 KB.
In local file systems, i'm sure you are correct. If you are working
with a remote file system, however, the optimal size is on the order
of megabytes, not kilobytes. For a specific example, consider the
PVFS2 file system, where the plateau in "blocksize vs. bandwitdh" is
two orders of magnitude larger than 64 KB. PVFS2 is a parallel file
system for linux clusters. I am not nearly as familiar with Lustre,
GPFS, or GFS, but I suspect those filesystems too would benefit from
block sizes larger than 64 KB.
Are you taking umbrage at the idea of using st_blksize to direct how
large the transfer size should be for I/O? I don't know what other
purpose st_blksize should have, nor are there any other fields which
are remotely valid for that purpose.
Thanks for your feedback.
==rob
--
Rob Latham
Mathematics and Computer Science Division A215 0178 EA2D B059 8CDF
Argonne National Labs, IL USA B29D F333 664A 4280 315B