very slow rebuilt for RAID15 and RAID51

very slow rebuilt for RAID15 and RAID51

[Patrick Schaaf]

Dear software-RAID hackers and users,

I'm in the process of setting up RAID15 and RAID51. Right now, this is
for performance testing, but if operational characteristics are good,
it will go into production. The hardware platform is a dual-processor
P-III box (1Ghz processors), an ASUS CUR-DLS board, 2GB RAM, and two
3ware 8-channel IDE controllers (sitting on seperate PCI busses).
There are 16 disk drives, Maxtor 80GB beasts.  The kernel is 2.4.17-rc2,
with the MD code compiled in (no modules). I use the raidtools2 package
of current Debian woody.  As a testing platform, the machine does not
do anything besides the things described here:

In order to compare RAID15 to RAID51, I split the drives into two equal-sized
partitions, sd[a-p]2 for the RAID15, and sd[a-p]3 for the RAID51 tests.

The first layer was created yesterday: 8 mirror pairs, each pair with one
drive on each 3ware controller, rebuilt in parallel at a rate of about
7.5MB/s per mirror pair, 60MB/s aggregate. Next, two RAID5 sets were
created in parallel, each spanning the "3" partition of the 8 drives
of one, and the other, controller. This initial rebuild progressed
at about 12MB/s/set, 24MB/s aggregate. I waited until all this layer 1
rebuild was done.

So far, everything is well. I/O characteristics for sequential read and
write are OK on both variants. Sequential read from the RAID5 sets gives
me a whopping 150MB/s transfer speed. Great.

Then I started the second layer, making a RAID5 set out of the 8 mirror pairs
out of the "2" partitions, and making a mirror pair out of the two RAID5 sets
on the "3" partition. Technically, this seems to work, but the initial rebuild
runs extremely slow, compared to the above rates: the RAID15 rebuild goes
at about 380kB/s, and the RAID51 build at slightly over 100kB/s. I will report
on operational characteristics, once these rebuilds are through (in about
30 days, for the RAID51...)

Anybody got any explanation, hints, or even convincing discouragement?

best regards
  Patrick

Please Cc: me on replies, I'm not subscribed to linux-kernel.

Here is the /etc/raidtab I am using. md[0-7] are the layer 1 mirror pairs,
md1[0-1] are the layer 1 RAID5 arrays, md8 is the RAID15 thing, and md9
is the RAID51 thing.

raiddev /dev/md0
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sda2
 raid-disk 0
 device /dev/sdi2
 raid-disk 1
raiddev /dev/md1
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sdb2
 raid-disk 0
 device /dev/sdj2
 raid-disk 1
raiddev /dev/md2
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sdc2
 raid-disk 0
 device /dev/sdk2
 raid-disk 1
raiddev /dev/md3
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sdd2
 raid-disk 0
 device /dev/sdl2
 raid-disk 1
raiddev /dev/md4
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sde2
 raid-disk 0
 device /dev/sdm2
 raid-disk 1
raiddev /dev/md5
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sdf2
 raid-disk 0
 device /dev/sdn2
 raid-disk 1
raiddev /dev/md6
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sdg2
 raid-disk 0
 device /dev/sdo2
 raid-disk 1
raiddev /dev/md7
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/sdh2
 raid-disk 0
 device /dev/sdp2
 raid-disk 1

raiddev /dev/md8
 raid-level 5
 nr-raid-disks 8
 persistent-superblock 1
 chunk-size 128
 device /dev/md0
 raid-disk 0
 device /dev/md1
 raid-disk 1
 device /dev/md2
 raid-disk 2
 device /dev/md3
 raid-disk 3
 device /dev/md4
 raid-disk 4
 device /dev/md5
 raid-disk 5
 device /dev/md6
 raid-disk 6
 device /dev/md7
 raid-disk 7

raiddev /dev/md9
 raid-level 1
 nr-raid-disks 2
 persistent-superblock 1
 chunk-size 8
 device /dev/md10
 raid-disk 0
 device /dev/md11
 raid-disk 1

raiddev /dev/md10
 raid-level 5
 nr-raid-disks 8
 persistent-superblock 1
 chunk-size 32
 device /dev/sda3
 raid-disk 0
 device /dev/sdb3
 raid-disk 1
 device /dev/sdc3
 raid-disk 2
 device /dev/sdd3
 raid-disk 3
 device /dev/sde3
 raid-disk 4
 device /dev/sdf3
 raid-disk 5
 device /dev/sdg3
 raid-disk 6
 device /dev/sdh3
 raid-disk 7

raiddev /dev/md11
 raid-level 5
 nr-raid-disks 8
 persistent-superblock 1
 chunk-size 32
 device /dev/sdi3
 raid-disk 0
 device /dev/sdj3
 raid-disk 1
 device /dev/sdk3
 raid-disk 2
 device /dev/sdl3
 raid-disk 3
 device /dev/sdm3
 raid-disk 4
 device /dev/sdn3
 raid-disk 5
 device /dev/sdo3
 raid-disk 6
 device /dev/sdp3
 raid-disk 7

Finally, here's a current snapshot of /proc/mdstat:

Personalities : [raid0] [raid1] [raid5] 
read_ahead 1024 sectors
md8 : active raid5 md7[7] md6[6] md5[5] md4[4] md3[3] md2[2] md1[1] md0[0]
      272702080 blocks level 5, 128k chunk, algorithm 0 [8/8] [UUUUUUUU]
      [========>............]  resync = 43.9% (17120184/38957440) finish=909.8min speed=396K/sec
md9 : active raid1 md11[1] md10[0]
      272814912 blocks [2/2] [UU]
      [>....................]  resync =  1.7% (4651736/272814912) finish=39435.7min speed=111K/sec
      
md11 : active raid5 sdp3[7] sdo3[6] sdn3[5] sdm3[4] sdl3[3] sdk3[2] sdj3[1] sdi3[0]
      272814976 blocks level 5, 32k chunk, algorithm 0 [8/8] [UUUUUUUU]
      
md10 : active raid5 sdh3[7] sdg3[6] sdf3[5] sde3[4] sdd3[3] sdc3[2] sdb3[1] sda3[0]
      272814976 blocks level 5, 32k chunk, algorithm 0 [8/8] [UUUUUUUU]
      
md0 : active raid1 sdi2[1] sda2[0]
      38957504 blocks [2/2] [UU]
      
md1 : active raid1 sdj2[1] sdb2[0]
      38957504 blocks [2/2] [UU]
      
md2 : active raid1 sdk2[1] sdc2[0]
      38957504 blocks [2/2] [UU]
      
md3 : active raid1 sdl2[1] sdd2[0]
      38957504 blocks [2/2] [UU]
      
md4 : active raid1 sdm2[1] sde2[0]
      38957504 blocks [2/2] [UU]
      
md5 : active raid1 sdn2[1] sdf2[0]
      38957504 blocks [2/2] [UU]
      
md6 : active raid1 sdo2[1] sdg2[0]
      38957504 blocks [2/2] [UU]
      
md7 : active raid1 sdp2[1] sdh2[0]
      38957504 blocks [2/2] [UU]
      
unused devices: <none>

Re: very slow rebuilt for RAID15 and RAID51

[José Luis Domingo López]

On Saturday, 22 December 2001, at 10:38:32 +0100,
Patrick Schaaf wrote:

> Then I started the second layer, making a RAID5 set out of the 8 mirror pairs
> out of the "2" partitions, and making a mirror pair out of the two RAID5 sets
> on the "3" partition. Technically, this seems to work, but the initial rebuild
> runs extremely slow, compared to the above rates: the RAID15 rebuild goes
> at about 380kB/s, and the RAID51 build at slightly over 100kB/s. I will report
> on operational characteristics, once these rebuilds are through (in about
> 30 days, for the RAID51...)
> 
There are two kernel tunnables that can modify RAID reconstruction speed:
/proc/sys/dev/raid/speed_limit_min
/proc/sys/dev/raid/speed_limit_max

They both are integer values in KB/s, the latter limiting the absolute
maximun RAID reconstruction speed (200 MB/s if I recall correctly, at
least for prior kernel versions). The former gives you the minimun
guaranteed reconstruction speed, and has a default (again, at least for
kernel kersions up to 2.4.7) of exactly 100 KB/s.

RAID reconstruction code is expected to only "access" disks when there
are no disk activity due to applications, with a guaranteed minimun of
"speed_limit_min". If this RAID code feels disks are busy due to
applications using the disks, reconstruction will take place at just
this minimun guaranteed speed (exactly 100 KB/s).

As far as I remember this happens with "layered" RAIDs, such as the
setup your are trying. To get a minimun reconstruction speed greater
than the default, just (for example):
echo "50000" > /proc/sys/dev/raid/speed_limit_min

Be careful not to give a too high value, because you will get a very
unresponsive system in case you set this parameter too high.

A couple of days ago Ingo Molnar submitted some patches for RAID in the
2.5.x series, with some improvements that maybe will find their way into
2.4.x. Check this threads:
http://marc.theaimsgroup.com/?t=100863779600002&r=1&w=2&n=5
http://marc.theaimsgroup.com/?l=linux-kernel&m=100876771508736&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=100876826309961&w=2

-- 
José Luis Domingo López
Linux Registered User #189436     Debian Linux Woody (P166 64 MB RAM)
 
jdomingo AT internautas DOT org  => Spam at your own risk