Re: Premature "No Space left on device" on XFS

[Bryan J Smith <b.j.smith@ieee.org>]

[-- Attachment #1.1: Type: text/plain, Size: 3140 bytes --]

I figured someone would fill in the gaps in my experience-assumptions. Excellent info, especially on those fs comparisons.
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Dave Chinner <david@fromorbit.com> wrote:

On Fri, Oct 07, 2011 at 06:58:53AM -0700, Bryan J Smith wrote:
> [ Not really adding any technical meat, but just some past
> experience with XFS, plus Ext3 experience ]
> 
> I remember running into this a long time ago when I was first
> playing with XFS for /tmp and /var (I was still a Linux/XFS noob
> at the time, not that I'm an expert today).  I ran into the same
> case where both free block and inodes were still available
> (although similarly well utilized), and the median file size was
> around 1KiB.  It was also in the case of many small files being
> written out in a short period.
> 
> In my case, I didn't use the XFS debugger to get into the
> allocation of the extents (would have if I wasn't such a noob,
> good, discrete command to know, thanx!).
> 
> Extents are outstanding for data and similar directories, ordering
> and placing large and small files to mitigate fragmentation.  But
> in this case, and correct me if I'm wrong, it's really just a
> wasteful use for the extents approach, as the files typically fit
> in a single data block or two. 

And single blocks still an extent, so there's nothing "wasted" by
having a single block extent.

....

> I've used Ext3 with around 8 million files with a median size well
> under 4KiB (under 32GiB total).  It works "well enough."  I'm
> curious how Ext4 would do though.  I think Ric Wheeler's team (at
> Red Hat) has done some benchmarks on 7+ figure file counts on Ext3
> and Ext4.

And against XFS, too. In case you didn't realise, you're talking to
the person who ran a large number of those tests. ;)

The results were ext4 is good for create/delete workloads up to 2-4
threads and about 100k files per directory on a decent disk
subsystem (4000 iops). It's much better than ext3, and for those
workloads about 2x as fast as XFS at 1-2 threads. This pattern held
true as long as the disk subsystem could handle the number of iops
that ext4 threw at it. XFS performance came at a much, much lower
iops cost (think order of magnitude), so shoul dbe more consistent
on a wider range of storage hardware than ext4.

However, XFS was about 3x faster on cold cache lookups than ext4, so
if you're workload is dominated by lookups, XFS is definitely the
faster filesystem to use even if creates/unlinks on ext4 are
faster..

As soon as you have more parallelism than 2-4 threads or large
directories, XFS create/unlink speed surpasses ext4 by a large
amount - the best I got out of ext4 was ~80k creates a second, while
XFS topped 130k creates/s at 8 threads. And the lookup speed
differential increase in XFS's favour at larger thread counts as
well.

So it really depends on your workload as to which filesystem will
handle your small files best. Mail spools tend to have lots of
parallelism, which is why XFS works pretty well, even though it is
a small file workload.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com


[-- Attachment #1.2: Type: text/html, Size: 3886 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs