Re: RAID-5 streaming read performance

[Ming Zhang <mingz@ele.uri.edu>]

On Mon, 2005-07-11 at 11:11 -0400, Dan Christensen wrote:
> I was wondering what I should expect in terms of streaming read
> performance when using (software) RAID-5 with four SATA drives.  I
> thought I would get a noticeable improvement compared to reads from a
> single device, but that's not the case.  I tested this by using dd to
> read 300MB directly from disk partitions /dev/sda7, etc, and also using
> dd to read 300MB directly from the raid device (/dev/md2 in this case).
> I get around 57MB/s from each of the disk partitions that make up the
> raid device, and about 58MB/s from the raid device.  On the other
> hand, if I run parallel reads from the component partitions, I get
> 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.
> 
> Before each read, I try to clear the kernel's cache by reading
> 900MB from an unrelated partition on the disk.  (Is this guaranteed
> to work?  Is there a better and/or faster way to clear cache?)
> 
> I have AAM quiet mode/low performance enabled on the drives, but (a)
> this shouldn't matter too much for streaming reads, and (b) it's the
> relative performance of the reads from the partitions and the RAID
> device that I'm curious about.
> 
> I also get poor write performance, but that's harder to isolate
> because I have to go through the lvm and filesystem layers too.
> 
> I also get poor performance from my RAID-1 array and my other
> RAID-5 arrays.
> 
> Details of my tests and set-up below.
> 
> Thanks for any suggestions,
> 
> Dan
> 
> 
> System:
> - Athlon 2500+
> - kernel 2.6.12.2 (also tried 2.6.11.11)
> - four SATA drives (3 160G, 1 200G); Samsung Spinpoint
> - SiI3114 controller (latency_timer=32 by default; tried 128 too)
only 1 card? 4 port? try some other brand card and try to use several
cards at the same time. i met some poor cards before.



> - 1G ram
> - blockdev --getra /dev/sda  -->  256   (didn't play with these)
> - blockdev --getra /dev/md2  -->  768   (didn't play with this)
> - tried anticipatory, deadline and cfq schedules, with no significant
>   difference.
> - machine essentially idle during tests
> 
> Here is part of /proc/mdstat (the full output is below):
> 
> md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
>       218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>       
> Here's the test script and output:
> 
> # Clear cache:
> dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1
> for f in sda7 sdb5 sdc5 sdd5 ; do 
>   echo $f
>   dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec
>   echo
> done
> 
> # Clear cache:
> dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1
> for f in md2 ; do 
>   echo $f
>   dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec
>   echo
> done
> 
> Output:
> 
> sda7
> 314572800 bytes transferred in 5.401071 seconds (58242671 bytes/sec)
> 
> sdb5
> 314572800 bytes transferred in 5.621170 seconds (55962158 bytes/sec)
> 
> sdc5
> 314572800 bytes transferred in 5.635491 seconds (55819947 bytes/sec)
> 
> sdd5
> 314572800 bytes transferred in 5.333374 seconds (58981951 bytes/sec)
> 
> md2
> 314572800 bytes transferred in 5.386627 seconds (58398846 bytes/sec)
> 
> # cat /proc/mdstat
> md1 : active raid5 sdd1[2] sdc1[1] sda2[0]
>       578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
>       
> md4 : active raid5 sdd2[3] sdc2[2] sdb2[1] sda6[0]
>       30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>       
> md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
>       218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>       
> md3 : active raid5 sdd6[3] sdc6[2] sdb6[1] sda8[0]
>       218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
>       
> md0 : active raid1 sdb1[0] sda5[1]
>       289024 blocks [2/2] [UU]
> 
> # mdadm --detail /dev/md2
> /dev/md2:
>         Version : 00.90.01
>   Creation Time : Mon Jul  4 23:54:34 2005
>      Raid Level : raid5
>      Array Size : 218612160 (208.48 GiB 223.86 GB)
>     Device Size : 72870720 (69.49 GiB 74.62 GB)
>    Raid Devices : 4
>   Total Devices : 4
> Preferred Minor : 2
>     Persistence : Superblock is persistent
> 
>     Update Time : Thu Jul  7 21:52:50 2005
>           State : clean
>  Active Devices : 4
> Working Devices : 4
>  Failed Devices : 0
>   Spare Devices : 0
> 
>          Layout : left-symmetric
>      Chunk Size : 64K
> 
>            UUID : c4056d19:7b4bb550:44925b88:91d5bc8a
>          Events : 0.10873823
> 
>     Number   Major   Minor   RaidDevice State
>        0       8        7        0      active sync   /dev/sda7
>        1       8       21        1      active sync   /dev/sdb5
>        2       8       37        2      active sync   /dev/sdc5
>        3       8       53        3      active sync   /dev/sdd5
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html