Re: Slow Soft-RAID 5 performance

["J.A. Magallón" <jamagallon@ono.com>]

On Wed, 18 Jul 2007 10:56:11 +0100, Rui Santos <rsantos@grupopie.com> wrote:

> Hi,
> 
> I'm getting a strange slow performance behavior on a recently installed
> Server. Here are the details:
> 
...
> 
> I can get a write throughput of 60 MB/sec on each HD by issuing the
> command 'time `dd if=/dev/zero of=test.raw bs=4k count=$(( 1024 * 1024 /
> 4 )); sync`'
> 
...
> 
> The RAID device I'm testing on is /dev/md2. Now, by issuing the same
> command 'dd if=/dev/zero of=test.raw bs=4k count=$(( 1024 * 1024 / 4 ));
> sync`' on the raid device mount point, I get the following speeds:
> With stripe_cache_size at default '265': 51 MB/sec
> With stripe_cache_size at '8192': 73 MB/sec
> 

I know many people consider this stupid, but can you post some hdparm -tT
data ?
The culprit can be the filesystem+pagecache, the md driver or the disk
driver, so I think trying just hdparm will show if the disk o md are
going nuts...

In my case, I have a box with 2 raids, one with SCSI disks and one with
IDE ones.

Some results:

lsscsi:
[0:0:0:0]    disk    IBM      DDYS-T18350N     S96H  /dev/sda
[2:0:0:0]    disk    SEAGATE  ST336807LW       0C01  /dev/sdb
[2:0:1:0]    disk    SEAGATE  ST336807LW       0C01  /dev/sdc
[2:0:2:0]    disk    SEAGATE  ST336807LW       0C01  /dev/sdd
[2:0:3:0]    disk    SEAGATE  ST336807LW       0C01  /dev/sde
[3:0:0:0]    disk    ATA      ST3120022A       3.06  /dev/sdf
[3:0:1:0]    cd/dvd  HL-DT-ST DVDRAM GSA-4040B A300  /dev/sr0
[4:0:0:0]    disk    ATA      ST3120022A       3.76  /dev/sdg


/dev/md0:
        Version : 00.90.03
  Creation Time : Mon Jun 18 13:40:57 2007
     Raid Level : raid5
     Array Size : 107522304 (102.54 GiB 110.10 GB)
  Used Dev Size : 35840768 (34.18 GiB 36.70 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Wed Jul 18 13:31:22 2007
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

           UUID : 51ad72a7:a4d20d15:0f3ea3a1:5ccb49a0
         Events : 0.2

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1

This is, four scsi disks on a Adaptec U320, doing raid5:

/dev/sdb:
 Timing cached reads:   904 MB in  2.00 seconds = 451.84 MB/sec
 Timing buffered disk reads:  228 MB in  3.00 seconds =  75.90 MB/sec
/dev/sdc:
 Timing buffered disk reads:  226 MB in  3.01 seconds =  75.01 MB/sec
/dev/sdd:
 Timing buffered disk reads:  228 MB in  3.00 seconds =  75.88 MB/sec
/dev/sde:
 Timing buffered disk reads:  226 MB in  3.00 seconds =  75.31 MB/sec

/dev/md0:
 Timing buffered disk reads:  562 MB in  3.01 seconds = 186.88 MB/sec

Nearly 75x3 = 215 Mb/s. And this looks like a small regression, I remember
to have seen 200Mb on this setup on previous kernels.
Performance is like 186/215 = 86%.

And /dev/md1, raid0 on 2 IDE disks:

/dev/sdf:
 Timing buffered disk reads:  148 MB in  3.02 seconds =  48.93 MB/sec
/dev/sdg:
 Timing buffered disk reads:  124 MB in  3.00 seconds =  41.33 MB/sec

/dev/md1:
 Timing buffered disk reads:  204 MB in  3.01 seconds =  67.68 MB/sec

Performance: 67 / 90 = 75%, more or less...not too good.

Now that I read the hdparm man page, perhaps would be better to repeat
the tests with hdparm --direct.

--
J.A. Magallon <jamagallon()ono!com>     \               Software is like sex:
                                         \         It's better when it's free
Mandriva Linux release 2008.0 (Cooker) for i586
Linux 2.6.21-jam12 (gcc 4.2.1 20070704 (4.2.1-3mdv2008.0)) SMP PREEMPT
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0