PROBLEM: Consistently slower 3ware RAID performance under 2.6.4

PROBLEM: Consistently slower 3ware RAID performance under 2.6.4

[Timothy Miller]

## Background:

I'm doing a lot of sequential read and write performance tests on my 
3ware 7000-2 controller (RAID1 with 2xWD1200JB), because I'm getting 
very poor write performance.  This is a problem that I'm currently 
working with 3ware to resolve, and they are working enthusiastically 
with me to fix it.

Read test:    time dd if=/dev/sda2 of=/dev/null bs=1024k count=1024
Write test:   time dd if=/dev/zero of=/dev/sda2 bs=1024k count=1024

NOTE:  /dev/sda2 is the swap partition which is NEAR the outer-most tracks.


## The new problem I discovered specific to Linux:

Regardless of the above-mentioned problem, I am noticing a very 
significant performance drop between a 2.4 kernel and a 2.6 kernel.


## Performance with "2.4.25-gentoo":

The read test here takes 21.6 seconds which is about 47MB/sec.  This is 
a correct number, because I have measured the maximum read throughput 
from each drive to be 47MB/sec.

The write test here takes 2 minutes, 2.5 seconds.  This translates to 
8.35MB/sec.  This is what I'm working with 3ware to correct, but let's 
call this the baseline write performance.


## Performance with "2.6.4-gentoo-r1":

The read test here takes 33.9 seconds.  That's down to about 30MB/sec.

The write test here takes 2 minutes, 44.2 seconds.  That is down to 
6.2MB/sec.


## HELP?

How can I help kernel developers to diagnose this problem?  What 
information do I need to provide that is missing?

What is responsible for such a significant performance drop on LONG 
sequential disk accesses?


## My computer:

Athlon 2800+
1GB RAM  (Corsair 2700LL)
ABIT KD7 (KT400 chipset)


Thanks in advance for the help!

Re: PROBLEM: Consistently slower 3ware RAID performance under 2.6.4

[Andrew Morton]

Timothy Miller <miller@techsource.com> wrote:
>
> ## Performance with "2.4.25-gentoo":
> 
> The read test here takes 21.6 seconds which is about 47MB/sec.  This is 
> a correct number, because I have measured the maximum read throughput 
> from each drive to be 47MB/sec.
> 
> The write test here takes 2 minutes, 2.5 seconds.  This translates to 
> 8.35MB/sec.  This is what I'm working with 3ware to correct, but let's 
> call this the baseline write performance.
> 
> 
> ## Performance with "2.6.4-gentoo-r1":
> 
> The read test here takes 33.9 seconds.  That's down to about 30MB/sec.
> 
> The write test here takes 2 minutes, 44.2 seconds.  That is down to 
> 6.2MB/sec.
> 

You cannot compare 2.4 and 2.6 kernel write performance with `dd', because
the kernels are tuned differently.  2.6 kernels are tuned to leave less
dirty pages in memory than a 2.4 kernel.  Hence when your dd has finished,
40% of memory will be dirty (needing writeout) under 2.6, but this figure
is 60% on 2.4.

So the 2.6 kernel does more writeout during the dd run and less writeout
after dd has finished.  The 2.4 kernel does less writeout during the dd run
and more writeout after dd has finished.

To compare IO performance you'll need to set 2.6's /proc/sys/vm/dirty_ratio
to 60 and /proc/sys/vm/dirty_async_ratio to 30.  Or use write-and-fsync
from http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz with
the `-f' option.

I don't know why the read bandwidth appears to be lower.  Try increasing
the readahead with `blockdev --setra'?

Re: PROBLEM: Consistently slower 3ware RAID performance under 2.6.4

[Timothy Miller]

Andrew Morton wrote:

> 
> You cannot compare 2.4 and 2.6 kernel write performance with `dd', because
> the kernels are tuned differently.  2.6 kernels are tuned to leave less
> dirty pages in memory than a 2.4 kernel.  Hence when your dd has finished,
> 40% of memory will be dirty (needing writeout) under 2.6, but this figure
> is 60% on 2.4.
> 
> So the 2.6 kernel does more writeout during the dd run and less writeout
> after dd has finished.  The 2.4 kernel does less writeout during the dd run
> and more writeout after dd has finished.
> 
> To compare IO performance you'll need to set 2.6's /proc/sys/vm/dirty_ratio
> to 60 and /proc/sys/vm/dirty_async_ratio to 30.  Or use write-and-fsync
> from http://www.zip.com.au/~akpm/linux/patches/stuff/ext3-tools.tar.gz with
> the `-f' option.
> 
> I don't know why the read bandwidth appears to be lower.  Try increasing
> the readahead with `blockdev --setra'?


That's odd.  I've tried different mem= kernel parameters with no change 
in throughput under 2.4.  That is, 1G vs. 128M vs. 32M all perform the 
same under 2.4 when doing dd to a block device.  On the other hand, 
available RAM makes a HUGE difference when doing dd to a _file_system_.

I haven't tried that under 2.6 yet.

Anyhow, I chose a 1GB test run so as diminish cache effects and file 
system overhead.  Is my logic flawed here?

Thanks.