Re: XFS filesystem performance drop in kernels 2.6.16+

Re: XFS filesystem performance drop in kernels 2.6.16+

[David Chinner]

On Thu, Nov 09, 2006 at 12:08:35PM -0600, Russell Cattelan wrote:
> On Thu, 2006-11-09 at 20:30 +0300, Igor A. Valcov wrote:
> > Hello,
> > 
> > For one of our projects we have a test program that measures file
> > system performance by writing up to 1000 files simultaneously. After
> > installing kernel v2.6.16 we noticed that XFS performance dropped by a
> > factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> > take around 20 minutes to complete). We then checked all kernels
> > starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> > The funny thing about all this is that we chose XFS for that
> > particular project specifically because it was about 5 times faster
> > with the tests than the other file systems. Now they all take about
> > the same time.
> > 
> > I also noticed that I/O barriers were introduced in v2.6.16 and
> > thought they may be the cause, but mounting the file system with
> > 'nobarrier' doesn't seem to affect the performance in any way.
> > 
> > Any thoughts on the matter are appreciated.
> I would try verifying the problem on a non ide disk just
> to confirm the write barrier theory.
> 
> Also file a bug.
> http://oss/sgi.com/bugzilla
> include test case and hard description if possible.

and cc xfs@oss.sgi.com on XFS bug reports ;)

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: XFS filesystem performance drop in kernels 2.6.16+

[Eric Sandeen]

Igor A. Valcov wrote:

> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.


did this happen to be a remount with nobarrier, or a fresh mount?

-Eric

Re: XFS filesystem performance drop in kernels 2.6.16+

[Jan Engelhardt]

>> I also noticed that I/O barriers were introduced in v2.6.16 and
>> thought they may be the cause, but mounting the file system with
>> 'nobarrier' doesn't seem to affect the performance in any way.
>
>
> did this happen to be a remount with nobarrier, or a fresh mount?

For the barrier stuff, see
http://lkml.org/lkml/2006/5/19/33


	-`J'
-- 

Re: XFS filesystem performance drop in kernels 2.6.16+

[Igor A. Valcov]

Below is a simplified version of the test program, and results of
testing different kernels/filesystems/mount options. The results are a
little different from the ones described in the initial post (this
time performance decreased "only" 2 times), but the general tendency
is clearly the same.


============ 2.6.19-rc5-git2 ============

mount -t xfs -o noatime,barrier /dev/sdc1 /mnt/disc

real    16m40.516s
user    0m17.989s
sys    9m36.320s

mount -t xfs -o noatime,nobarrier /dev/sdc1 /mnt/disc

real    15m40.212s
user    0m17.549s
sys    9m29.692s

mount -t ext3 -o noatime /dev/sdc1 /mnt/disc

real    49m44.728s
user    0m27.678s
sys    14m15.689s

============ 2.6.14.6 ============

mount -t xfs -o noatime /dev/sdc1 /mnt/disc

real    9m58.974s
user    0m17.373s
sys    8m4.850s

mount -t ext3 -o noatime /dev/sdc1 /mnt/disc

real    49m7.526s
user    0m26.278s
sys    12m37.627s

========================================

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

#define __BYTES        8192
#define __FILES        1000

char    buf [__BYTES];

int main ()
{
    char    fname [1024];
    int        nFiles [__FILES];
    int     f, i;

    /* Fill buf */
    for (i = 0; i < __BYTES; i++)
        buf [i] = i % 128;

    /* Create and open files */
    for (f = 0; f < __FILES; f++) {
        sprintf (fname, "/mnt/disc/storage/file-%d", f);
        nFiles [f] = open (fname, O_WRONLY | O_CREAT | O_TRUNC, 0644);
    }

    for (i = 0; i < 262144; i++) {
        /* Write data to a big file */
        write (nFiles [0], buf, __BYTES);

        /* Write data to small files */
        for (f = 1; f < __FILES; f++)
            write (nFiles [f], &f, sizeof (f));
    }

    for (f = 0; f < __FILES; f++) {
        fsync (nFiles [f]);
        close (nFiles [f]);
    }

    return 0;
}

========================================

-- 
Igor A. Valcov

Re: XFS filesystem performance drop in kernels 2.6.16+

[Eric Sandeen]

Igor A. Valcov wrote:
> Below is a simplified version of the test program, and results of
> testing different kernels/filesystems/mount options. The results are a
> little different from the ones described in the initial post (this
> time performance decreased "only" 2 times), but the general tendency
> is clearly the same.

I imagine that I know the answer, but to be sure you might put some time
checks into your test app to see -which- portion of the test is taking
the bulk of the time.

-Eric

Re: XFS filesystem performance drop in kernels 2.6.16+

[Andrew Morton]

On Fri, 10 Nov 2006 16:16:27 +0300
"Igor A. Valcov" <viaprog@gmail.com> wrote:

> Below is a simplified version of the test program,

Boy, I hope not.  The results of this test program are of very little interest.

>     for (i = 0; i < 262144; i++) {
>         /* Write data to a big file */
>         write (nFiles [0], buf, __BYTES);
> 
>         /* Write data to small files */
>         for (f = 1; f < __FILES; f++)
>             write (nFiles [f], &f, sizeof (f));
>     }

This sits in a loop doing write(fd, buf, 4).  This is wildly inefficient -
you'd get a 10x throughput benefit and maybe 100x reduction in CPU cost
simply by switching to fwrite().

I suspect something went wrong here.

Re: XFS filesystem performance drop in kernels 2.6.16+

[Jan Engelhardt]

>>     for (i = 0; i < 262144; i++) {
>>         /* Write data to a big file */
>>         write (nFiles [0], buf, __BYTES);
>> 
>>         /* Write data to small files */
>>         for (f = 1; f < __FILES; f++)
>>             write (nFiles [f], &f, sizeof (f));
>>     }
>
>This sits in a loop doing write(fd, buf, 4).  This is wildly inefficient -
>you'd get a 10x throughput benefit and maybe 100x reduction in CPU cost
>simply by switching to fwrite().

Well yes and no. The problem here is the syscall overhead. fwrite 
buffers things, so needless syscalls are avoided.
The same could be done by changing the program logic and increasing the 
size argument to read/write.

>I suspect something went wrong here.

Design error. :)


	-`J'
-- 

No Mails

[Shailendra Tripathi]

Not getting any mail on the mailing list. I am not sure if there is no 
activity or I am not getting any mail.
I suspect the latter, how do I re-enable myself.
-shailendra

Re: No Mails

[Eric Sandeen]

Shailendra Tripathi wrote:
> Not getting any mail on the mailing list. I am not sure if there is no 
> activity or I am not getting any mail.
> I suspect the latter, how do I re-enable myself.
> -shailendra
> 
> 
odd, looks like you are not subscribed now...

try following the instructions at
http://oss.sgi.com/projects/xfs/mail.html to resubscribe...

-Eric

Re: No Mails

[Russell Cattelan]

Shailendra Tripathi wrote:

> Not getting any mail on the mailing list. I am not sure if there is no 
> activity or I am not getting any mail.
> I suspect the latter, how do I re-enable myself.
> -shailendra
>
the list software will unsubscribe any address that has to many failures.
Normally this is a good thing as the list maintainers don't have to keep
pruning out dead email addresses. Unfortunately the list software has
no way of knowing if the failure was due a bad address or a rejected spam.
In your case it turns out to be to many rejected spams.

Unsub      0     10    Oct 22 - Nov 09      stripathi@agami.com
           550 5.7.1 mail containing es.geocities.com rejected - sbl; 
see http://www.spamhaus.org/query/bl?ip=66.218.77.68
Unsub      0     10    Oct 22 - Nov 09      miken@agami.com
           550 5.7.1 mail containing es.geocities.com rejected - sbl; 
see http://www.spamhaus.org/query/bl?ip=66.218.77.68


This may have something to do with the fact that oss's mail is running 
through a barracuda box.

Guess you will have to resubscribe to the list.
sorry out the inconvenience.

-Russell Cattelan
cattelan@xfs.org