OSDL aio-stress results on latest kernels show buffered random read issue

OSDL aio-stress results on latest kernels show buffered random read issue

[Judith Lebzelter]

Hello;

I am running aio-stress on the most recent kernels and have
found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
performance of buffered random reads is poor compared to the
buffered random writes:

               2.6.8      2.6.9-rc2     2.6.9-rc2-mm4
             --------------------------------------------
random write 35.66 MB/s   34.80 MB/s    29.89 MB/s
random read   7.69 MB/s    7.50 MB/s     7.68 MB/s

** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.


This shows up on our 4CPU host as well. (striped AACRAID.4G
RAM. 8G File):
             2.6.9-rc2     2.6.9-rc2-mm4   2.6.9-rc2-mm1
             -------------------------------------------
random write 31.36 MB/s     18.92 MB/s      18.97 MB/s
random read  11.13 MB/s      9.74 MB/s      11.05 MB/s


There seems to be an issue with the reads.  Usually, reads
should be at least as fast as writes of the same type.

Also, there seems to be a substantial drop-off in the performance
of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
40% on 4CPU)

Regards;
Judith Lebzelter
OSDL

Re: OSDL aio-stress results on latest kernels show buffered random read issue

[William Lee Irwin III]

On Wed, Sep 29, 2004 at 04:29:08PM -0700, Judith Lebzelter wrote:
> There seems to be an issue with the reads.  Usually, reads
> should be at least as fast as writes of the same type.
> Also, there seems to be a substantial drop-off in the performance
> of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> 40% on 4CPU)

Okay, is it cpu time or idle/iowait? If it's cpu time, where do
profiles show it appears?


-- wli

Re: OSDL aio-stress results on latest kernels show buffered random read issue

[Judith Lebzelter]

On Wed, 29 Sep 2004, William Lee Irwin III wrote:

> On Wed, Sep 29, 2004 at 04:29:08PM -0700, Judith Lebzelter wrote:
> > There seems to be an issue with the reads.  Usually, reads
> > should be at least as fast as writes of the same type.
> > Also, there seems to be a substantial drop-off in the performance
> > of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> > 40% on 4CPU)
>
> Okay, is it cpu time or idle/iowait? If it's cpu time, where do
> profiles show it appears?

Th CPU is not that busy:

2.6.9-rc2-mm4 Results and iostat outputs:
http://khack.osdl.org/stp/297714/
http://khack.osdl.org/stp/297714/results/bufferrand/iostat.txt

2.6.9-rc2 Results and iostat outputs:
http://khack.osdl.org/stp/297545/
http://khack.osdl.org/stp/297545/results/bufferrand/iostat.txt


The iostat has the write stats followed by the reads, taken every 15
seconds.

>
>
> -- wli
>

Re: OSDL aio-stress results on latest kernels show buffered random read issue

[Andrew Morton]

Judith Lebzelter <judith@osdl.org> wrote:
>
> Hello;
> 
> I am running aio-stress on the most recent kernels and have
> found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
> performance of buffered random reads is poor compared to the
> buffered random writes:
> 
>                2.6.8      2.6.9-rc2     2.6.9-rc2-mm4
>              --------------------------------------------
> random write 35.66 MB/s   34.80 MB/s    29.89 MB/s
> random read   7.69 MB/s    7.50 MB/s     7.68 MB/s
> 
> ** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
> 
> 
> This shows up on our 4CPU host as well. (striped AACRAID.4G
> RAM. 8G File):
>              2.6.9-rc2     2.6.9-rc2-mm4   2.6.9-rc2-mm1
>              -------------------------------------------
> random write 31.36 MB/s     18.92 MB/s      18.97 MB/s
> random read  11.13 MB/s      9.74 MB/s      11.05 MB/s
> 
> 
> There seems to be an issue with the reads.  Usually, reads
> should be at least as fast as writes of the same type.
> 
> Also, there seems to be a substantial drop-off in the performance
> of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> 40% on 4CPU)
> 

Well one would expect writes to be much faster than reads because writes
usually do not involve performing physical I/O, and when pagecache
writeback finally happens it has vastly more data to work with and hence
can schedule I/O more efficiently.

Unless you are using O_SYNC or fsync(), in which case ignore the above.

The regression within random write performance is unexpected.  Can you
please provide a URL to the current version of the test tool, and a
description of how you are invoking it?  What sort of I/O system, what
filesystem, etc.

Thanks.

Re: OSDL aio-stress results on latest kernels show buffered random read issue

[Judith Lebzelter]

On Wed, 29 Sep 2004, Andrew Morton wrote:

> Judith Lebzelter <judith@osdl.org> wrote:
> >
> > Hello;
> >
> > I am running aio-stress on the most recent kernels and have
> > found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
> > performance of buffered random reads is poor compared to the
> > buffered random writes:
> >
> >                2.6.8      2.6.9-rc2     2.6.9-rc2-mm4
> >              --------------------------------------------
> > random write 35.66 MB/s   34.80 MB/s    29.89 MB/s
> > random read   7.69 MB/s    7.50 MB/s     7.68 MB/s
> >
> > ** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
> >
> >
> > This shows up on our 4CPU host as well. (striped AACRAID.4G
> > RAM. 8G File):
> >              2.6.9-rc2     2.6.9-rc2-mm4   2.6.9-rc2-mm1
> >              -------------------------------------------
> > random write 31.36 MB/s     18.92 MB/s      18.97 MB/s
> > random read  11.13 MB/s      9.74 MB/s      11.05 MB/s
> >
> >
> > There seems to be an issue with the reads.  Usually, reads
> > should be at least as fast as writes of the same type.
> >
> > Also, there seems to be a substantial drop-off in the performance
> > of AIO buffered-random writes in the mm kernels. (14% on 2CPU,
> > 40% on 4CPU)
> >
>
> Well one would expect writes to be much faster than reads because writes
> usually do not involve performing physical I/O, and when pagecache
> writeback finally happens it has vastly more data to work with and hence
> can schedule I/O more efficiently.
>
> Unless you are using O_SYNC or fsync(), in which case ignore the above.

It should be doing an fsync() between the write and read stages.  For all
other types of reads the performance is substantially better than the
writes.
                       2.6.9-rc2-mm4
Direct Random write:     11.17 MB/s
Direct Random read:      44.71 MB/s

>
> The regression within random write performance is unexpected.  Can you
> please provide a URL to the current version of the test tool, and a
> description of how you are invoking it?  What sort of I/O system, what
> filesystem, etc.

I am running aio-stress from Chris Mason;  you can get it from OSDL's
aio-stress test kit:
  bk clone bk://developer.osdl.org/stp-tests/aio-stress

This is how I call it (with only one file):
    aio-stress -o 2 -o 3 -s 4g -r 64k -l -L <files_name>

The 2 CPU host has MegaRAID, with 5 18G disks, striped.  The 4CPU has
Adaptec 2200S x2, with 5 18G disks, striped.

The filesystem is ext2.

>
> Thanks.
>

Re: OSDL aio-stress results on latest kernels show buffered random read issue

[Bill Davidsen]

Judith Lebzelter wrote:
> Hello;
> 
> I am running aio-stress on the most recent kernels and have
> found that on linux-2.6.8, 2.6.9-rc2 and 2.6.9-rc2-mm4 the
> performance of buffered random reads is poor compared to the
> buffered random writes:
> 
>                2.6.8      2.6.9-rc2     2.6.9-rc2-mm4
>              --------------------------------------------
> random write 35.66 MB/s   34.80 MB/s    29.89 MB/s
> random read   7.69 MB/s    7.50 MB/s     7.68 MB/s
> 
> ** 2CPU hosts with striped Megaraid. 1G RAM. 4G File.
> 
> 
> This shows up on our 4CPU host as well. (striped AACRAID.4G
> RAM. 8G File):
>              2.6.9-rc2     2.6.9-rc2-mm4   2.6.9-rc2-mm1
>              -------------------------------------------
> random write 31.36 MB/s     18.92 MB/s      18.97 MB/s
> random read  11.13 MB/s      9.74 MB/s      11.05 MB/s
> 
> 
> There seems to be an issue with the reads.  Usually, reads
> should be at least as fast as writes of the same type.

Usually not, since the write goes to cache in most cases while the read 
must really happen. The 40% slowdown is troubling, though, that's 
unexpected.


-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me