raid5 grow reshaping speed is unchangeable

raid5 grow reshaping speed is unchangeable

[Cody Yellan]

I had a 4x500GB SATA2 array, md0.  I added one 500GB drive and
reshaping began at ~2500K/sec.  Changing
/proc/sys/dev/raid/speed_limit_m{in,ax} or
/sys/block/md0/md/sync_speed_m{in,ax} had no effect.  I shut down all
unnecessary services and the array is offline (not mounted).  I have
read that the throttling code is "fragile" (esp. with regard to
raid5) but does this make sense?  I will wait (in)patiently for it to
finish, but I do wonder why the configuration parameters have no
effect.  This is a dual quad 2GHz Xeon machine with 8GB of memory
running RHEL5.  Is this the maximum speed?

Re: raid5 grow reshaping speed is unchangeable

[Nagilum]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

----- Message from cody@mochamail.com ---------
     Date: Wed, 26 Dec 2007 21:22:42 -0800
     From: Cody Yellan <cody@mochamail.com>

> I had a 4x500GB SATA2 array, md0.  I added one 500GB drive and
> reshaping began at ~2500K/sec.  Changing
> /proc/sys/dev/raid/speed_limit_m{in,ax} or
> /sys/block/md0/md/sync_speed_m{in,ax} had no effect.  I shut down all
> unnecessary services and the array is offline (not mounted).  I have
> read that the throttling code is "fragile" (esp. with regard to
> raid5) but does this make sense?  I will wait (in)patiently for it to
> finish, but I do wonder why the configuration parameters have no
> effect.  This is a dual quad 2GHz Xeon machine with 8GB of memory
> running RHEL5.  Is this the maximum speed?
----- End message from cody@mochamail.com -----

Certainly not. I've personally used a similar setup with >80MB/s  
reshape speeds.
There must be some other limiting factor.
Try simultanous dd's from the drives, if that doesn't work as fast as  
it should there might be some bus contention?


========================================================================
#    _  __          _ __     http://www.nagilum.org/ \n icq://69646724 #
#   / |/ /__ ____ _(_) /_ ____ _  nagilum@nagilum.org \n +491776461165 #
#  /    / _ `/ _ `/ / / // /  ' \  Amiga (68k/PPC): AOS/NetBSD/Linux   #
# /_/|_/\_,_/\_, /_/_/\_,_/_/_/_/   Mac (PPC): MacOS-X / NetBSD /Linux #
#           /___/     x86: FreeBSD/Linux/Solaris/Win2k  ARM9: EPOC EV6 #
========================================================================


----------------------------------------------------------------
cakebox.homeunix.net - all the machine one needs..


[-- Attachment #2: PGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 187 bytes --]

Re: raid5 grow reshaping speed is unchangeable

[Bill Davidsen]

Cody Yellan wrote:
> I had a 4x500GB SATA2 array, md0.  I added one 500GB drive and
> reshaping began at ~2500K/sec.  Changing
> /proc/sys/dev/raid/speed_limit_m{in,ax} or
> /sys/block/md0/md/sync_speed_m{in,ax} had no effect.  I shut down all
> unnecessary services and the array is offline (not mounted).  I have
> read that the throttling code is "fragile" (esp. with regard to
> raid5) but does this make sense?  I will wait (in)patiently for it to
> finish, but I do wonder why the configuration parameters have no
> effect.  This is a dual quad 2GHz Xeon machine with 8GB of memory
> running RHEL5.  Is this the maximum speed?
>   

Something else going on, I do better than that adding drives on USB!

I don't have a clue what the issue is, and I don't see anything in your 
information which looks unusual.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: raid5 grow reshaping speed is unchangeable

[Richard Scobie]

> Cody Yellan wrote:
>> I had a 4x500GB SATA2 array, md0.  I added one 500GB drive and
>> reshaping began at ~2500K/sec.  Changing
>> /proc/sys/dev/raid/speed_limit_m{in,ax} or
>> /sys/block/md0/md/sync_speed_m{in,ax} had no effect.  I shut down all
>> unnecessary services and the array is offline (not mounted).  I have
>> read that the throttling code is "fragile" (esp. with regard to
>> raid5) but does this make sense?  I will wait (in)patiently for it to
>> finish, but I do wonder why the configuration parameters have no
>> effect.  This is a dual quad 2GHz Xeon machine with 8GB of memory
>> running RHEL5.  Is this the maximum speed?

Are you running a non-standard /sys/block/md5/md/stripe_cache_size?

I found when running a kernel of about that vintage on FC6 (which I 
guess became RHEL5), that with a stripe_cache_size of 16384, resyncs 
dropped down to about the speed you are seeing.

This changed back to more normal speeds with a later kernel, but I do 
not recall which.

Regards,

Richard

RE: raid5 grow reshaping speed is unchangeable

[Cody Yellan]

I forgot the version information:

mdadm - v2.5.4 - 13 October 2006
kernel 2.6.18-53.el5 #1 SMP

Would anyone consider it unsafe to upgrade to the latest version of
mdadm on a production machine using Neil Brown's srpm?

Re: raid5 grow reshaping speed is unchangeable

[Bill Davidsen]

Cody Yellan wrote:
> I forgot the version information:
>
> mdadm - v2.5.4 - 13 October 2006
> kernel 2.6.18-53.el5 #1 SMP
>
> Would anyone consider it unsafe to upgrade to the latest version of
> mdadm on a production machine using Neil Brown's srpm?
>   

I wouldn't expect any problems, although I don't think there will be a 
benefit, either. The problem with RHEL is that while it's stable in 
terms of bug fixes, it also doesn't get any performance benefits.

Why not wait until Neil gets back from the holiday and see if he has any 
words of advice?

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: raid5 grow reshaping speed is unchangeable

[Cody Yellan]

>>> Cody Yellan wrote:
>>>> I had a 4x500GB SATA2 array, md0.  I added one 500GB drive and
>>>> reshaping began at ~2500K/sec.  Changing
>>>> /proc/sys/dev/raid/speed_limit_m{in,ax} or
>>>> /sys/block/md0/md/sync_speed_m{in,ax} had no effect.  I shut down
>all
>>>> unnecessary services and the array is offline (not mounted).  I
>have
>>>> read that the throttling code is "fragile" (esp. with regard to
>>>> raid5) but does this make sense?  I will wait (in)patiently for
>it to
>>>> finish, but I do wonder why the configuration parameters have no
>>>> effect.  This is a dual quad 2GHz Xeon machine with 8GB of memory
>>>> running RHEL5.  Is this the maximum speed?
>>
>>Are you running a non-standard /sys/block/md5/md/stripe_cache_size?
>>
>>I found when running a kernel of about that vintage on FC6 (which I 
>>guess became RHEL5), that with a stripe_cache_size of 16384, resyncs
>
>>dropped down to about the speed you are seeing.
>>
>>This changed back to more normal speeds with a later kernel, but I
>do 
>>not recall which.
>>
>>Regards,
>>
>>Richard

You are right, Richard.  RHEL5 had a stripe_cache_size of 256 when
the reshape began.  I increased it to 1024 and the reshape speed
doubled to 4500K/s.  I did not see any increase in memory usage.  I
tried 2048 and then 4096 but saw no difference in speed.

The reshape finished and the array is back online.  mdadm works very
well, but I would like to better understand how it works.  For
instance, the mdadm man page says a backup-file is necessary to grow
a RAID5, but this does not appear necessary now.  Anyway, it's a
wonderful tool.

Re: raid5 grow reshaping speed is unchangeable

[Richard Scobie]

Cody Yellan wrote:

> You are right, Richard.  RHEL5 had a stripe_cache_size of 256 when
> the reshape began.  I increased it to 1024 and the reshape speed
> doubled to 4500K/s.  I did not see any increase in memory usage.  I
> tried 2048 and then 4096 but saw no difference in speed.

Sorry, I did not make myself clear.

Back when I was seeing the slow resync speeds, it was due to larger than 
default stripe_cache_size - I had increased to 16384.

When this was reduced back to the default, which I suspect is 256, 
speeds were back up to more normal levels - much faster than the 4.5MB/s 
you are seeing.

At some point, this issue was fixed and currently on 2.6.22, with a 
stripe_cache_size of 16384 and a 4 x 500GB SATA II md RAID 5, resync 
speed is 162MB/s.

Perhaps somthing different occurs on a reshape, (which I have only now 
realised you were doing).

Regards,

Richard

Re: raid5 grow reshaping speed is unchangeable

[Cody Yellan]

>>2007/12/27, Cody Yellan <cody@mochamail.com>:
>>> I had a 4x500GB SATA2 array, md0.  I added one 500GB drive and
>>> reshaping began at ~2500K/sec.  Changing
>>> /proc/sys/dev/raid/speed_limit_m{in,ax} or
>>> /sys/block/md0/md/sync_speed_m{in,ax} had no effect.  I shut down
>all
>>> unnecessary services and the array is offline (not mounted).  I
>have
>>> read that the throttling code is "fragile" (esp. with regard to
>>> raid5) but does this make sense?  I will wait (in)patiently for it
>to
>>> finish, but I do wonder why the configuration parameters have no
>>> effect.  This is a dual quad 2GHz Xeon machine with 8GB of memory
>>> running RHEL5.  Is this the maximum speed?
>>>
>>I think you may check each SATA2 disk to make sure they are working
>fine.
>>I had a similar situation and it turned out to be caused by one of
>>disks in array. It is  a failed disk indicated by its S.M.A.R.T
>>information.

No problems indicated by SMART but I did have some concerns about the
drive I added.  It contained some data so I tried to delete the
partition and create a new filesystem-free partition.  Both parted
and fdisk simply restored the already existing partition, complete
with the data.  So I just added that drive, figuring mdadm would
overwrite the partition.    It worked but from now on I will try to
use only blank disks.