Re: XFS: Abysmal write performance because of excessive seeking (allocation groups to blame?)

Peter Grandi
2012-04-06T14:36:41Z

[ ... ]

> [ ... ] general purpose filesystems cannot handle well large
> groups of small files,
[ ... ]

>> As can be seen from the time scale in the bottom part, the
>> ext4 version performed about 5 times as fast because of a much
>> more disk-friendly write pattern.

As to 'ext4' and doing (euphemism) insipid tests involving
peculiar setups, there is an interesting story in this post:

http://oss.sgi.com/archives/xfs/2012-03/msg00465.html

on the perils of using 'tar x' as a "test" of something
meaningful (illustrated using a much smaller "test" than
yours).

The telling details was that there was a ratio of 227 times (6
seconds versus 23 minutes) between running 'tar x' without any
safety and with most safeties.

A ratio of 227 times indicates that there is something big going
on, which is that contemporary disk drives have 2 orders of
magnitude between bulk sequential and small random "speed" (which
is the major reason why «general purpose filesystems cannot
handle well large groups of small files»), and that in between
one can choose a vast number of different safety/speed tradeoffs
(or introduce performance problems :->).

Does that means that 'ext4' has "Abysmal write performance" in
the 23 minutes case? No, just a different tradeoff.

Similarly XFS has had for a long time a mostly undeserved
reputation for being "slow" on small-IO/metadata intensive
workloads, in large part because traditionally it has been
designed to deliver a higher level of (implicit, metadata) safety
than other filesystems; for good reasons.

Therefore as I argued in other comments the «excessive seeking»
you report seems due to me more to storage layer issues and
perhaps stricter interpretation of safety by XFS, than to
something really wrong with XFS, which is a tool that has to be
deployed with consideration.

As to that a comparison that does point a finger at the
underlying storage system:

* In your graphs 'ext4' writes out 2.5GB of small files at
around 100MB/s (and with relatively few long seeks on that
workload) on an "enterprise" storage system that has 4+2
disks each capable of 130MB/s.

* In the 6s "test" I did reported above in a similar situation
'ext4' wrote out 370MB also at not much less than 100MB/s,
but on a single "consumer" disk on a much slower destktop.