Re: avoid mbox file fragmentation

Peter Grandi
2010-10-20T11:19:58Z

[ ... ]

> I added allocsize=1m as my read of the man page suggests this
> will preallocate an additional 1MB of extent space at the end
> of each mbox file each time it is written, which I would think
> should eliminate fragmentation of these files. [ ... ] I've
> probably totally misread what allocsize= actually does. [
> ... ]

That is not really the point. The point is that it is extremely
difficult if not pointless to ensure contiguous allocation for N
slowly growing streams (see below) on a 1-dimensional storage
device in the general case. One does not need to read any file
system introductory tutorial to figure that out, just a bit of
thinking it through will do.

> If allocsize= doesn't help prevent fragmentation of mbox
> files, what can I do to mitigate this, other than regularly
> running xfs_fsr?

'xfs_fsr' is not really a defragmenter, it just fixes some of
the worst cases, if there is a lot of free space.

> I've recently run xfs_fsr thus the low 7% figure. Every
> couple of weeks the fragmentation reaches ~30%. I save a lot
> of list mail, dozens to hundreds per day for each of about 7
> foss mailing lists.

That is the case where one has a number of slowly growing
"stream" files, such as multiple parallel downloads, or log
files.

> As I say, in just a couple of weeks, these mbox files become
> fragmented, on the order of a dozen to a few dozen extents per
> mbox file.

That's not a lot and not necessarily something to worry about.
What matters is fragment size and their distance in terms of arm
positioning times.

Anyhow in general file system allocators are designed to give
decent average performance, not to give perfect allocations in
edge cases (especially not those requiring predictive instead of
adaptive algorithms), unlike so many newbies posting to this
list seem to assume.

If you want perfect allocation strategies for edge cases, please
write your own abstract storage system. DBMS authors do that for
example.