New RAID array syncs very slow

New RAID array syncs very slow

[Jay Roberts]

I just set up a new Raid5 array using 4 200GB disks on 2 controllers. I 
created the array, and it begins to sync up, but it will not sync at 
over 600Kb. It has been over 12 hours now, and the sync is only 7% done. 
From reading, I know that sync should only take a few hours, but at the 
current rate mine will take 91 hours which seems like an insane amount 
of time. I am looking for any advice on this.  Details follow:

Hardware:
Disks: 4 Seagate 200GB 7200 RPM ST3200822A
Controllers: 2 Promise Ultra133 TX2 IDE controllers, one disk on each 
channel, Ultra133 cables.
CPU: PIII - 450 Mhz.
MB: Abit BE-6II (intel 440 bx)
512MB Ram.
OS boots and runs off a Atlas 10k SCSI drive attached to an Adaptec 
U19160 scsi card.

Software:
Debian Testing, kernel 2.6.3, mdadm -- whatever came with apt-get

I created the array using
#mdadm --create /dev/md0 --verbose --chunk=128 --level=5 
--parity=left-symmetric --raid-devices=4 --spare-devices=0

I also tried bumping up /proc/sys/dev/raid/speed_limit_min to 100000

Output of #cat /proc/mdstat

Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6]
md0 : active raid5 hdk[3] hdi[2] hdg[1] hde[0]
      586082688 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      [=>...................]  resync =  8.0% (15647744/195360896) 
finish=5491.2min speed=543K/sec
unused devices: <none>

Output of #mdadm --detail /dev/md0

        Version : 00.90.01
  Creation Time : Mon Mar 22 00:18:13 2004
     Raid Level : raid5
     Array Size : 586082688 (558.93 GiB 600.15 GB)
    Device Size : 195360896 (186.31 GiB 200.05 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Mar 22 00:18:13 2004
          State : clean, no-errors
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 128K

    Number   Major   Minor   RaidDevice State
       0      33        0        0      active sync   /dev/hde
       1      34        0        1      active sync   /dev/hdg
       2      56        0        2      active sync   /dev/hdi
       3      57        0        3      active sync   /dev/hdk
           UUID : 5152287c:34d31d31:19a993d7:946f87c9
         Events : 0.2

After this, I moved all the hardware to an Asus A7V MB, with a 1.3 
Duron, and ran a stock Sarge install. I got Faster results, but still it 
is going to take about 18 hours to sync the set of disks. Now after 
about 5 hours in the new rig i Get:

debian:~# cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 ide/host4/bus1/target0/lun0/disc[3] 
ide/host4/bus0/target0/lun0/disc[2] ide/host2/bus1/target0/lun0/disc[1] 
ide/host2/bus0/target0/lun0/disc[0]
      586082688 blocks level 5, 128k chunk, algorithm 2 [4/4] [UUUU]
      [======>..............]  resync = 30.2% (59040244/195360896) 
finish=781.7min speed=2904K/sec
unused devices: <none>


Based on things I have read this seems way too long. I must be screwing 
up some where I can't see.

Any help at all is appreciated. Quite a few hours of googling has 
yielded little.

Re: New RAID array syncs very slow

[Gordon Henderson]

On Tue, 23 Mar 2004, Jay Roberts wrote:

> I just set up a new Raid5 array using 4 200GB disks on 2 controllers. I
> created the array, and it begins to sync up, but it will not sync at
> over 600Kb. It has been over 12 hours now, and the sync is only 7% done.
> >From reading, I know that sync should only take a few hours, but at the
> current rate mine will take 91 hours which seems like an insane amount
> of time. I am looking for any advice on this.  Details follow:

You probably don't have DMA enabled on the controller. See what

  hdparm /dev/hde

and for all the other drives you have gives.

Do a crude speed test with

  hdparm -Tt /dev/hde

too. You ought to get at least 45MB/sec on a modern IDE drive.

You might have to build a custom kernel to get the right driver for the
chipset.

Gordon

Re: New RAID array syncs very slow

[Sandro Dentella]

>   hdparm -Tt /dev/hde
> 
> too. You ought to get at least 45MB/sec on a modern IDE drive.

On my pc I get:
/dev/hda:
 Timing buffer-cache reads:   128 MB in  1.73 seconds = 73.99 MB/sec
 Timing buffered disk reads:  64 MB in  2.98 seconds = 21.48 MB/sec

On my laptop I get
/dev/hda:
 Timing buffer-cache reads:   596 MB in  2.01 seconds = 296.52 MB/sec
 Timing buffered disk reads:   54 MB in  3.05 seconds =  17.70 MB/sec

On other raid server I have on production I get
/dev/hda:
 Timing buffer-cache reads:   128 MB in  0.24 seconds =533.33 MB/sec
 Timing buffered disk reads:  64 MB in  1.29 seconds = 49.61 MB/sec

Which "line" should be "at least 45MB/sec"? I read the man page for -tT opt
but I know too little of how it works to really understand it.

sandro
*:-)

-- 
Sandro Dentella  *:-)
e-mail: sandro.dentella@tin.it 
http://www.tksql.org                    TkSQL Home page - My GPL work

Re: New RAID array syncs very slow

[Jay Roberts]

This turned out to be really helpful. While investigating why one of  
the drives had so much lower numbers than the others using the test you 
suggested, I discovered a bent down pin in the cable socket. After 
fixing , the stats popped back up to the other drives' levels, and the 
estimated sync time on the 1.3 Duron box dropped to around 4 hours. I 
moved all the hardware back to the original P3-450 box, made sure DMA 
was enabled, and the sync time was now only 8 hours vs 91. A huge 
improvement and a livable time period. I was surprised at the difference 
in sync time between the two boxes.  Almost double on the P3-450 over 
the 1.3 Duron. 

Gordon Henderson wrote:

>On Tue, 23 Mar 2004, Jay Roberts wrote:
>
>  
>
>>I just set up a new Raid5 array using 4 200GB disks on 2 controllers. I
>>created the array, and it begins to sync up, but it will not sync at
>>over 600Kb. It has been over 12 hours now, and the sync is only 7% done.
>>>From reading, I know that sync should only take a few hours, but at the
>>current rate mine will take 91 hours which seems like an insane amount
>>of time. I am looking for any advice on this.  Details follow:
>>    
>>
>
>You probably don't have DMA enabled on the controller. See what
>
>  hdparm /dev/hde
>
>and for all the other drives you have gives.
>
>Do a crude speed test with
>
>  hdparm -Tt /dev/hde
>
>too. You ought to get at least 45MB/sec on a modern IDE drive.
>
>You might have to build a custom kernel to get the right driver for the
>chipset.
>
>Gordon
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>