RAID-5 streaming read performance

[Dan Christensen <jdc@uwo.ca>]

I was wondering what I should expect in terms of streaming read
performance when using (software) RAID-5 with four SATA drives.  I
thought I would get a noticeable improvement compared to reads from a
single device, but that's not the case.  I tested this by using dd to
read 300MB directly from disk partitions /dev/sda7, etc, and also using
dd to read 300MB directly from the raid device (/dev/md2 in this case).
I get around 57MB/s from each of the disk partitions that make up the
raid device, and about 58MB/s from the raid device.  On the other
hand, if I run parallel reads from the component partitions, I get
25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.

Before each read, I try to clear the kernel's cache by reading
900MB from an unrelated partition on the disk.  (Is this guaranteed
to work?  Is there a better and/or faster way to clear cache?)

I have AAM quiet mode/low performance enabled on the drives, but (a)
this shouldn't matter too much for streaming reads, and (b) it's the
relative performance of the reads from the partitions and the RAID
device that I'm curious about.

I also get poor write performance, but that's harder to isolate
because I have to go through the lvm and filesystem layers too.

I also get poor performance from my RAID-1 array and my other
RAID-5 arrays.

Details of my tests and set-up below.

Thanks for any suggestions,

Dan


System:
- Athlon 2500+
- kernel 2.6.12.2 (also tried 2.6.11.11)
- four SATA drives (3 160G, 1 200G); Samsung Spinpoint
- SiI3114 controller (latency_timer=32 by default; tried 128 too)
- 1G ram
- blockdev --getra /dev/sda  -->  256   (didn't play with these)
- blockdev --getra /dev/md2  -->  768   (didn't play with this)
- tried anticipatory, deadline and cfq schedules, with no significant
  difference.
- machine essentially idle during tests

Here is part of /proc/mdstat (the full output is below):

md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
      218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
Here's the test script and output:

# Clear cache:
dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1
for f in sda7 sdb5 sdc5 sdd5 ; do 
  echo $f
  dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec
  echo
done

# Clear cache:
dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1
for f in md2 ; do 
  echo $f
  dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec
  echo
done

Output:

sda7
314572800 bytes transferred in 5.401071 seconds (58242671 bytes/sec)

sdb5
314572800 bytes transferred in 5.621170 seconds (55962158 bytes/sec)

sdc5
314572800 bytes transferred in 5.635491 seconds (55819947 bytes/sec)

sdd5
314572800 bytes transferred in 5.333374 seconds (58981951 bytes/sec)

md2
314572800 bytes transferred in 5.386627 seconds (58398846 bytes/sec)

# cat /proc/mdstat
md1 : active raid5 sdd1[2] sdc1[1] sda2[0]
      578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]
      
md4 : active raid5 sdd2[3] sdc2[2] sdb2[1] sda6[0]
      30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
      218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md3 : active raid5 sdd6[3] sdc6[2] sdb6[1] sda8[0]
      218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
      
md0 : active raid1 sdb1[0] sda5[1]
      289024 blocks [2/2] [UU]

# mdadm --detail /dev/md2
/dev/md2:
        Version : 00.90.01
  Creation Time : Mon Jul  4 23:54:34 2005
     Raid Level : raid5
     Array Size : 218612160 (208.48 GiB 223.86 GB)
    Device Size : 72870720 (69.49 GiB 74.62 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Thu Jul  7 21:52:50 2005
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : c4056d19:7b4bb550:44925b88:91d5bc8a
         Events : 0.10873823

    Number   Major   Minor   RaidDevice State
       0       8        7        0      active sync   /dev/sda7
       1       8       21        1      active sync   /dev/sdb5
       2       8       37        2      active sync   /dev/sdc5
       3       8       53        3      active sync   /dev/sdd5