poor raid0 performance in 2.6.16-rc1-mm2?

poor raid0 performance in 2.6.16-rc1-mm2?

[Ed Sweetman]

I'll have to reboot to double check that this is specific to the above 
kernel version, but It seems something is either wrong with my 
particular kernel config for raid0, or my raid0 is setup wrong.

my raid0 uses 64k chunk sizes on an ext3 fs that's 367GB large, (across 
two identical sata disks on nforce4 chipset)

I have partitions on both drives of equal size (2 altogether) that are 
outside of the raid0.  I dbenched those partitions, the raid0 device, 
and libata pata devices i also have (same rpm, less cache, same company).  

pata disk : 403MB/sec
sata disk 1: 446MB/sec
raid0 : between 336MB/sec and 386MB/sec

now the sata disks alone, get 446MB/sec, but the raid device that's 
comprised of them is getting >60MB/sec less throughput. 

This difference is much more drastic when say only 1 process is used 
with dbench,

sata disk 1: 230MB/sec
raid0 : 96MB/sec


Something definitely feels wrong with these numbers. 

All filesystems are ext3, on an athlon 64 x2 system, during each test no 
other io was performed, there is no swap and no other cpu intensive 
operations were going on. 

the filesystems were all created the same way, and all have the same 
blocksizes and such.  All are mounted with default options too. 

Re: poor raid0 performance in 2.6.16-rc1-mm2?

[J.A. Magallon]

[-- Attachment #1: Type: text/plain, Size: 3227 bytes --]

On Tue, 24 Jan 2006 18:27:53 -0500, Ed Sweetman <safemode@comcast.net> wrote:

> I'll have to reboot to double check that this is specific to the above 
> kernel version, but It seems something is either wrong with my 
> particular kernel config for raid0, or my raid0 is setup wrong.
> 
> my raid0 uses 64k chunk sizes on an ext3 fs that's 367GB large, (across 
> two identical sata disks on nforce4 chipset)
> 
> I have partitions on both drives of equal size (2 altogether) that are 
> outside of the raid0.  I dbenched those partitions, the raid0 device, 
> and libata pata devices i also have (same rpm, less cache, same company).  
> 
> pata disk : 403MB/sec
> sata disk 1: 446MB/sec
> raid0 : between 336MB/sec and 386MB/sec
> 

Uh ? I want some of those disks....
How are you measuring that ?

Some more real numbers.

A SATA raid5:
nada:~# lsscsi
[1:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
[2:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
[3:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
[4:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
[5:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
[6:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
[7:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -
nada:~# hdparm -t /dev/sdb /dev/sdc

/dev/sdb:
 Timing buffered disk reads:  156 MB in  3.02 seconds =  51.70 MB/sec

/dev/sdc:
 Timing buffered disk reads:  154 MB in  3.02 seconds =  50.96 MB/sec

nada:~# hdparm -t /dev/md0

/dev/md0:
 Timing buffered disk reads:  148 MB in  3.01 seconds =  49.20 MB/sec

An SCSI raid0 one (160 and 320 mixed...)

annwn:~# hdparm -t /dev/sdc /dev/sdb

/dev/sdc:
 Timing buffered disk reads:  226 MB in  3.02 seconds =  74.72 MB/sec

/dev/sdb:
 Timing buffered disk reads:  122 MB in  3.00 seconds =  40.64 MB/sec

annwn:~# hdparm -t /dev/md0

/dev/md0:
 Timing buffered disk reads:  242 MB in  3.00 seconds =  80.62 MB/sec


> now the sata disks alone, get 446MB/sec, but the raid device that's 
> comprised of them is getting >60MB/sec less throughput. 
> 
> This difference is much more drastic when say only 1 process is used 
> with dbench,
> 
> sata disk 1: 230MB/sec
> raid0 : 96MB/sec
> 
> 
> Something definitely feels wrong with these numbers. 
> 
> All filesystems are ext3, on an athlon 64 x2 system, during each test no 
> other io was performed, there is no swap and no other cpu intensive 
> operations were going on. 
> 
> the filesystems were all created the same way, and all have the same 
> blocksizes and such.  All are mounted with default options too. 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



--
J.A. Magallon <jamagallon()able!es>     \               Software is like sex:
werewolf!able!es                         \         It's better when it's free
Mandriva Linux release 2006.1 (Cooker) for i586
Linux 2.6.15-jam5 (gcc 4.0.2 (4.0.2-1mdk for Mandriva Linux release 2006.1))

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: poor raid0 performance in 2.6.16-rc1-mm2?

[Ed Sweetman]

J.A. Magallon wrote:

>On Tue, 24 Jan 2006 18:27:53 -0500, Ed Sweetman <safemode@comcast.net> wrote:
>
>  
>
>>I'll have to reboot to double check that this is specific to the above 
>>kernel version, but It seems something is either wrong with my 
>>particular kernel config for raid0, or my raid0 is setup wrong.
>>
>>my raid0 uses 64k chunk sizes on an ext3 fs that's 367GB large, (across 
>>two identical sata disks on nforce4 chipset)
>>
>>I have partitions on both drives of equal size (2 altogether) that are 
>>outside of the raid0.  I dbenched those partitions, the raid0 device, 
>>and libata pata devices i also have (same rpm, less cache, same company).  
>>
>>pata disk : 403MB/sec
>>sata disk 1: 446MB/sec
>>raid0 : between 336MB/sec and 386MB/sec
>>
>>    
>>
>
>Uh ? I want some of those disks....
>How are you measuring that ?
>
>Some more real numbers.
>
>A SATA raid5:
>nada:~# lsscsi
>[1:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
>[2:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
>[3:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
>[4:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
>[5:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
>[6:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -       
>[7:0:0:0]    disk    ATA      Maxtor 7Y250M0   YAR5  -
>nada:~# hdparm -t /dev/sdb /dev/sdc
>
>/dev/sdb:
> Timing buffered disk reads:  156 MB in  3.02 seconds =  51.70 MB/sec
>
>/dev/sdc:
> Timing buffered disk reads:  154 MB in  3.02 seconds =  50.96 MB/sec
>
>nada:~# hdparm -t /dev/md0
>
>/dev/md0:
> Timing buffered disk reads:  148 MB in  3.01 seconds =  49.20 MB/sec
>
>An SCSI raid0 one (160 and 320 mixed...)
>
>annwn:~# hdparm -t /dev/sdc /dev/sdb
>
>/dev/sdc:
> Timing buffered disk reads:  226 MB in  3.02 seconds =  74.72 MB/sec
>
>/dev/sdb:
> Timing buffered disk reads:  122 MB in  3.00 seconds =  40.64 MB/sec
>
>annwn:~# hdparm -t /dev/md0
>
>/dev/md0:
> Timing buffered disk reads:  242 MB in  3.00 seconds =  80.62 MB/sec
>  
>

I was using dbench  with three threads.  In retrospect, I am pretty sure 
my 2GB of ram was creating that performance.  Though, the poor 
performance when run on the raid0 device still stands.

This performance hit doesn't exist when using hdparm to test the 
bandwidth of the devices

/dev/sdc:
 Timing buffered disk reads:  166 MB in  3.01 seconds =  55.19 MB/sec
psuedomode:/mnt# hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:  126 MB in  3.02 seconds =  41.72 MB/sec
psuedomode:/mnt# hdparm -t /dev/md0

/dev/md0:
 Timing buffered disk reads:  338 MB in  3.02 seconds = 112.07 MB/sec


sda is pata libata
sdc is sata libata
md0 is two identical sata libata devices (sdc and sdd) in RAID0

So apparently, this may be filesystem related, as it pertains to being 
in a raid setup (since the filesystems are all the same otherwise except 
size).



>  
>
>>now the sata disks alone, get 446MB/sec, but the raid device that's 
>>comprised of them is getting >60MB/sec less throughput. 
>>
>>This difference is much more drastic when say only 1 process is used 
>>with dbench,
>>
>>sata disk 1: 230MB/sec
>>raid0 : 96MB/sec
>>
>>
>>Something definitely feels wrong with these numbers. 
>>
>>All filesystems are ext3, on an athlon 64 x2 system, during each test no 
>>other io was performed, there is no swap and no other cpu intensive 
>>operations were going on. 
>>
>>the filesystems were all created the same way, and all have the same 
>>blocksizes and such.  All are mounted with default options too. 
>>-
>>    
>>

Re: poor raid0 performance in 2.6.16-rc1-mm2?

[Grant Coady]

On 1/25/06, Ed Sweetman <safemode@comcast.net> wrote:
> I'll have to reboot to double check that this is specific to the above
> kernel version, but It seems something is either wrong with my
> particular kernel config for raid0, or my raid0 is setup wrong.
>
> my raid0 uses 64k chunk sizes on an ext3 fs that's 367GB large, (across
> two identical sata disks on nforce4 chipset)
>
> I have partitions on both drives of equal size (2 altogether) that are
> outside of the raid0.  I dbenched those partitions, the raid0 device,
> and libata pata devices i also have (same rpm, less cache, same company).
>
> pata disk : 403MB/sec
> sata disk 1: 446MB/sec
> raid0 : between 336MB/sec and 386MB/sec

Hmm, looks optimistic, I performed some raid0, raid1 testing on SATA recently
here: http://bugsplatter.mine.nu/test/raid_times

Grant.