Raid5 resync slow with one linear array

Raid5 resync slow with one linear array

[Mike Fedyk]

Hi,

I just setup a 4 (5 really) drive raid5 array.

It is syncing up right now and nothing else is running on the system.

I have three 18GB SCA scsi drives and 2x9GB linear array in a four "drive"
raid5 array.

Unfortunately, it is syncing up quite slowly.  Only about 2MB/sec on a
40MB/sec array.  The system is idle.

2.4.19-pre4-ac3

Is there something about this config that says "Don't do that!"?  I've
heard about RAID10, but not Linear+RAID5...

This way, I get more space, and one filesystem (except a 50mb raid1 for
booting).

Thanks

Mike

Personalities : [linear] [raid0] [raid1] [raid5] 
read_ahead 1024 sectors
md2 : active raid1 scsi/host0/bus0/target1/lun0/part1[1] scsi/host0/bus0/target0/lun0/part1[0]
      48064 blocks [2/2] [UU]
      
md1 : active linear scsi/host0/bus0/target0/lun0/part2[1] scsi/host0/bus0/target1/lun0/part2[0]
      17671296 blocks 32k rounding
      
md0 : active raid5 md/1[4] scsi/host0/bus0/target4/lun0/part2[2] scsi/host0/bus0/target3/lun0/part2[1] scsi/host0/bus0/target2/lun0/part2[0]
      52990080 blocks level 5, 32k chunk, algorithm 2 [4/3] [UUU_]
      [>....................]  recovery =  2.1% (373984/17663360) finish=143.2min speed=2010K/sec
unused devices: <none>

Re: Raid5 resync slow with one linear array

[Mike Fedyk]

On Tue, Apr 02, 2002 at 06:35:29AM +0100, Mark Cooke wrote:
> Hi Mike,
> 
> Have you checked individual transfer rates to the drives / md devices ?

Each drive transfers 18-20MB/sec, and all have been tested with badblocks -w.

> 
> I saw (non-reproducible unfortunately) a bad-transfer rate between one 
> of my drives in an ide raid setup, where the drive was only pushing ~ 
> 2MB/sec, even after being explictly zapped with hdparm.  A power cycle 
> fixed it, and it hasn't happenned since.
> 

Hmm, I don't think I'm dealing with bad hardware.  All are SCA SCSI and tested.

> Did you try building it with different chunk sizes ?
>

No, my root filesystem is on this array.  I think 32k chunks are ok.

Mike

Re: Raid5 resync slow with one linear array

[Neil Brown]

On Monday April 1, mfedyk@matchmail.com wrote:
> Hi,
> 
> I just setup a 4 (5 really) drive raid5 array.
> 
> It is syncing up right now and nothing else is running on the system.
> 
> I have three 18GB SCA scsi drives and 2x9GB linear array in a four "drive"
> raid5 array.
> 
> Unfortunately, it is syncing up quite slowly.  Only about 2MB/sec on a
> 40MB/sec array.  The system is idle.
> 
> 2.4.19-pre4-ac3
> 
> Is there something about this config that says "Don't do that!"?  I've
> heard about RAID10, but not Linear+RAID5...

echo 10000  > /proc/sys/dev/raid/speed_limit_min

md tries to monitor the activity on the component devices and limits
rebuild activity when there appears to be other activity.
It measures "other activity" as "blocks added to kstat.dk_drive_?blk",
minus "block due to resync activity".

When a component device is an md array, nothing gets recorded
in kstat, but lots is recorded as resync activity, so this "other 
activity" appears as a negative number which, due to storage in an
unsigned long, appears rather large.

NeilBrown

Re: Raid5 resync slow with one linear array

[Mike Fedyk]

On Thu, Apr 04, 2002 at 10:56:02PM +0100, Mark Cooke wrote:
> 
> Hmm,
> 
> Well you have me stumped then, Mike.  I've only been playing with the 
> raid stuff for a little while myself, and haven't had this trouble.
> 
> Hope someone else on the list can help!

I think it is the linear array that is part of the raid5 array.  RAID5
writing to the linear array (because one of the members of the RAID5 is
another linear pair of drives) probably is accounted as normal userspace IO,
and that causes the code to choose a slower transfer rate to sync.

Can someone confirm?

Re: Raid5 resync slow with one linear array

[Mike Fedyk]

On Thu, Apr 04, 2002 at 02:03:16PM +1000, Neil Brown wrote:
> On Monday April 1, mfedyk@matchmail.com wrote:
> > Hi,
> > 
> > I just setup a 4 (5 really) drive raid5 array.
> > 
> > It is syncing up right now and nothing else is running on the system.
> > 
> > I have three 18GB SCA scsi drives and 2x9GB linear array in a four "drive"
> > raid5 array.
> > 
> > Unfortunately, it is syncing up quite slowly.  Only about 2MB/sec on a
> > 40MB/sec array.  The system is idle.
> > 
> > 2.4.19-pre4-ac3
> > 
> > Is there something about this config that says "Don't do that!"?  I've
> > heard about RAID10, but not Linear+RAID5...
> 
> echo 10000  > /proc/sys/dev/raid/speed_limit_min
> 
> md tries to monitor the activity on the component devices and limits
> rebuild activity when there appears to be other activity.
> It measures "other activity" as "blocks added to kstat.dk_drive_?blk",
> minus "block due to resync activity".
> 
> When a component device is an md array, nothing gets recorded
> in kstat, but lots is recorded as resync activity, so this "other 
> activity" appears as a negative number which, due to storage in an
> unsigned long, appears rather large.
> 
> NeilBrown

Yeah, thought it was something like that.  Thanks for the explanation.  I
didn't see your message until I sent the other one.

Mike