XFS filesystem performance drop in kernels 2.6.16+

XFS filesystem performance drop in kernels 2.6.16+

[Igor A. Valcov]

Hello,

For one of our projects we have a test program that measures file
system performance by writing up to 1000 files simultaneously. After
installing kernel v2.6.16 we noticed that XFS performance dropped by a
factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
take around 20 minutes to complete). We then checked all kernels
starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
The funny thing about all this is that we chose XFS for that
particular project specifically because it was about 5 times faster
with the tests than the other file systems. Now they all take about
the same time.

I also noticed that I/O barriers were introduced in v2.6.16 and
thought they may be the cause, but mounting the file system with
'nobarrier' doesn't seem to affect the performance in any way.

Any thoughts on the matter are appreciated.

Thanks in advance,

-- 
Igor A. Valcov

Re: XFS filesystem performance drop in kernels 2.6.16+

[Eric Sandeen]

Igor A. Valcov wrote:
> Hello,
> 
> For one of our projects we have a test program that measures file
> system performance by writing up to 1000 files simultaneously. After
> installing kernel v2.6.16 we noticed that XFS performance dropped by a
> factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> take around 20 minutes to complete). 

Ouch.

> We then checked all kernels
> starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> The funny thing about all this is that we chose XFS for that
> particular project specifically because it was about 5 times faster
> with the tests than the other file systems. Now they all take about
> the same time.
> 
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.
> 
> Any thoughts on the matter are appreciated.

Can you provide the test?

-Eric

Re: XFS filesystem performance drop in kernels 2.6.16+

[Russell Cattelan]

[-- Attachment #1: Type: text/plain, Size: 1255 bytes --]

On Thu, 2006-11-09 at 20:30 +0300, Igor A. Valcov wrote:
> Hello,
> 
> For one of our projects we have a test program that measures file
> system performance by writing up to 1000 files simultaneously. After
> installing kernel v2.6.16 we noticed that XFS performance dropped by a
> factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> take around 20 minutes to complete). We then checked all kernels
> starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> The funny thing about all this is that we chose XFS for that
> particular project specifically because it was about 5 times faster
> with the tests than the other file systems. Now they all take about
> the same time.
> 
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.
> 
> Any thoughts on the matter are appreciated.
I would try verifying the problem on a non ide disk just
to confirm the write barrier theory.

Also file a bug.
http://oss/sgi.com/bugzilla
include test case and hard description if possible.


> 
> Thanks in advance,

-- 
Russell Cattelan <cattelan@thebarn.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: XFS filesystem performance drop in kernels 2.6.16+

[David Chinner]

On Thu, Nov 09, 2006 at 12:08:35PM -0600, Russell Cattelan wrote:
> On Thu, 2006-11-09 at 20:30 +0300, Igor A. Valcov wrote:
> > Hello,
> > 
> > For one of our projects we have a test program that measures file
> > system performance by writing up to 1000 files simultaneously. After
> > installing kernel v2.6.16 we noticed that XFS performance dropped by a
> > factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> > take around 20 minutes to complete). We then checked all kernels
> > starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> > The funny thing about all this is that we chose XFS for that
> > particular project specifically because it was about 5 times faster
> > with the tests than the other file systems. Now they all take about
> > the same time.
> > 
> > I also noticed that I/O barriers were introduced in v2.6.16 and
> > thought they may be the cause, but mounting the file system with
> > 'nobarrier' doesn't seem to affect the performance in any way.
> > 
> > Any thoughts on the matter are appreciated.
> I would try verifying the problem on a non ide disk just
> to confirm the write barrier theory.
> 
> Also file a bug.
> http://oss/sgi.com/bugzilla
> include test case and hard description if possible.

and cc xfs@oss.sgi.com on XFS bug reports ;)

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS filesystem performance drop in kernels 2.6.16+

[Eric Sandeen]

Igor A. Valcov wrote:

> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.


did this happen to be a remount with nobarrier, or a fresh mount?

-Eric

Re: XFS filesystem performance drop in kernels 2.6.16+

[Jan Engelhardt]

>> I also noticed that I/O barriers were introduced in v2.6.16 and
>> thought they may be the cause, but mounting the file system with
>> 'nobarrier' doesn't seem to affect the performance in any way.
>
>
> did this happen to be a remount with nobarrier, or a fresh mount?

For the barrier stuff, see
http://lkml.org/lkml/2006/5/19/33


	-`J'
-- 

Re: XFS filesystem performance drop in kernels 2.6.16+

[Igor A. Valcov]

Below is a simplified version of the test program, and results of
testing different kernels/filesystems/mount options. The results are a
little different from the ones described in the initial post (this
time performance decreased "only" 2 times), but the general tendency
is clearly the same.


============ 2.6.19-rc5-git2 ============

mount -t xfs -o noatime,barrier /dev/sdc1 /mnt/disc

real    16m40.516s
user    0m17.989s
sys    9m36.320s

mount -t xfs -o noatime,nobarrier /dev/sdc1 /mnt/disc

real    15m40.212s
user    0m17.549s
sys    9m29.692s

mount -t ext3 -o noatime /dev/sdc1 /mnt/disc

real    49m44.728s
user    0m27.678s
sys    14m15.689s

============ 2.6.14.6 ============

mount -t xfs -o noatime /dev/sdc1 /mnt/disc

real    9m58.974s
user    0m17.373s
sys    8m4.850s

mount -t ext3 -o noatime /dev/sdc1 /mnt/disc

real    49m7.526s
user    0m26.278s
sys    12m37.627s

========================================

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

#define __BYTES        8192
#define __FILES        1000

char    buf [__BYTES];

int main ()
{
    char    fname [1024];
    int        nFiles [__FILES];
    int     f, i;

    /* Fill buf */
    for (i = 0; i < __BYTES; i++)
        buf [i] = i % 128;

    /* Create and open files */
    for (f = 0; f < __FILES; f++) {
        sprintf (fname, "/mnt/disc/storage/file-%d", f);
        nFiles [f] = open (fname, O_WRONLY | O_CREAT | O_TRUNC, 0644);
    }

    for (i = 0; i < 262144; i++) {
        /* Write data to a big file */
        write (nFiles [0], buf, __BYTES);

        /* Write data to small files */
        for (f = 1; f < __FILES; f++)
            write (nFiles [f], &f, sizeof (f));
    }

    for (f = 0; f < __FILES; f++) {
        fsync (nFiles [f]);
        close (nFiles [f]);
    }

    return 0;
}

========================================

-- 
Igor A. Valcov

Re: XFS filesystem performance drop in kernels 2.6.16+

[Igor A. Valcov]

Is a same test on ext4dev

============ 2.6.19-rc5-git2 ============

mount -t ext4dev -o extents /dev/sdc1 /mnt/disc

real    41m55.445s
user    0m24.242s
sys     14m20.302s

-- 
Igor A. Valcov

Re: XFS filesystem performance drop in kernels 2.6.16+

[Eric Sandeen]

Igor A. Valcov wrote:
> Below is a simplified version of the test program, and results of
> testing different kernels/filesystems/mount options. The results are a
> little different from the ones described in the initial post (this
> time performance decreased "only" 2 times), but the general tendency
> is clearly the same.

I imagine that I know the answer, but to be sure you might put some time
checks into your test app to see -which- portion of the test is taking
the bulk of the time.

-Eric

Re: XFS filesystem performance drop in kernels 2.6.16+

[Andrew Morton]

On Fri, 10 Nov 2006 16:16:27 +0300
"Igor A. Valcov" <viaprog@gmail.com> wrote:

> Below is a simplified version of the test program,

Boy, I hope not.  The results of this test program are of very little interest.

>     for (i = 0; i < 262144; i++) {
>         /* Write data to a big file */
>         write (nFiles [0], buf, __BYTES);
> 
>         /* Write data to small files */
>         for (f = 1; f < __FILES; f++)
>             write (nFiles [f], &f, sizeof (f));
>     }

This sits in a loop doing write(fd, buf, 4).  This is wildly inefficient -
you'd get a 10x throughput benefit and maybe 100x reduction in CPU cost
simply by switching to fwrite().

I suspect something went wrong here.

Re: XFS filesystem performance drop in kernels 2.6.16+

[Jan Engelhardt]

>>     for (i = 0; i < 262144; i++) {
>>         /* Write data to a big file */
>>         write (nFiles [0], buf, __BYTES);
>> 
>>         /* Write data to small files */
>>         for (f = 1; f < __FILES; f++)
>>             write (nFiles [f], &f, sizeof (f));
>>     }
>
>This sits in a loop doing write(fd, buf, 4).  This is wildly inefficient -
>you'd get a 10x throughput benefit and maybe 100x reduction in CPU cost
>simply by switching to fwrite().

Well yes and no. The problem here is the syscall overhead. fwrite 
buffers things, so needless syscalls are avoided.
The same could be done by changing the program logic and increasing the 
size argument to read/write.

>I suspect something went wrong here.

Design error. :)


	-`J'
-- 

Re: XFS filesystem performance drop in kernels 2.6.16+

[Tom Vier]

On Thu, Nov 09, 2006 at 08:30:23PM +0300, Igor A. Valcov wrote:
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.

I don't know if it's related, but i played with a few fs'es a few months ago
and both xfs and reiser4 were very slow. iirc, they both took about 45min
longer than reiserfs and jfs to copy 70 or 80gigs of files from another
drive.

I suspected delayed allocation was the culprit (both r4 and xfs use DA).
Never really looked into it much, however.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE