Raid 5 to 6 reshape q

Raid 5 to 6 reshape q

[Jon Hardcastle]

Hi guys, thanks for your help getting me going with this over the last week. I kicked it off last night and all is well.

The only Q i have is it seems to be abit slow? When i kicked it off it insisted that i give a backup file --backup-file that i set to /root/backupfile and now the reshape seems to be constantly accessing my md3 as well as the md4 (the one that is being reshaped) i have set the min and max that is used when i am rebuilding the array/testing

cat /sys/block/md4/md/sync_speed_min
200000 (local)
cat /sys/block/md4/md/sync_speed_max
7168000 (local)

but still the speed is abit slow?

md4 : active raid6 sdg1[4] sdf1[0] sde1[1] sdd1[2] sdc1[5] sdb1[3] sda1[6]
      2441919680 blocks super 0.91 level 6, 64k chunk, algorithm 18 [7/6] [UUUUUU_]
      [===>.................]  reshape = 17.9% (87600896/488383936) finish=2881.3min speed=2317K/sec

vg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.03    0.00   13.42   41.82    0.00   44.73

Device:            tps    MB_read/s    MB_wrtn/s    MB_read    MB_wrtn
hda              31.65         0.00        11.50          0        689
hdc              31.67         0.00        11.51          0        690
sda              77.35         0.00         2.33          0        139
sdb              80.05         4.59         2.33        275        139
sdc              82.28         4.59         2.33        275        139
sdd             135.65         4.59         2.33        275        139
sde              81.72         4.59         2.33        275        139
sdf              83.80         4.59         2.33        275        139
sdg              78.52         4.59         2.33        275        139
md3            2904.82         0.00        11.35          0        680
md4               0.00         0.00         0.00          0          0

As you can see lots of write access to md3 and low level read/write to the constituent drives of md4 

Does this seem ok? or is it slow (like i suspect!) 

-----------------------
N: Jon Hardcastle
E: Jon@eHardcastle.com
'Do not worry about tomorrow, for tomorrow will bring worries of its own.'

***********
Please note, I am phasing out jd_hardcastle AT yahoo.com and replacing it with jon AT eHardcastle.com
***********

-----------------------


      

Re: Raid 5 to 6 reshape q

[John Robinson]

On 02/01/2010 08:52, Jon Hardcastle wrote:
> Hi guys, thanks for your help getting me going with this over the last week. I kicked it off last night and all is well.
> 
> The only Q i have is it seems to be abit slow? When i kicked it off it insisted that i give a backup file --backup-file that i set to /root/backupfile and now the reshape seems to be constantly accessing my md3 as well as the md4 (the one that is being reshaped) i have set the min and max that is used when i am rebuilding the array/testing
[...]
> As you can see lots of write access to md3 and low level read/write to the constituent drives of md4 
> 
> Does this seem ok? or is it slow (like i suspect!) 

Because the entire reshape has to happen "in place", every chunk/stripe 
must be backed up. This makes things slow, as Neil explains in his blog 
here at http://neil.brown.name/blog/20090817000931#2 :
> This means that all the data is copied twice, once to the backup and once to the new layout on the array. This clearly means that such a reshape will go very slowly. But that is the price we have to pay for safety. It is like insurance. You might hate having to pay it, but you would hate it much more if you didn't and found that you needed it.

So I think the answer is that you'll have to be very patient.

Cheers,

John.

Re: Raid 5 to 6 reshape q

[Kristleifur Daðason]

On Sat, Jan 2, 2010 at 8:52 AM, Jon Hardcastle <jd_hardcastle@yahoo.com> wrote:
> Hi guys, thanks for your help getting me going with this over the last week. I kicked it off last night and all is well.
> ...
> The only Q i have is it seems to be abit slow?
> ...
> md4 : active raid6 sdg1[4] sdf1[0] sde1[1] sdd1[2] sdc1[5] sdb1[3] sda1[6]
>      2441919680 blocks super 0.91 level 6, 64k chunk, algorithm 18 [7/6] [UUUUUU_]
>      [===>.................]  reshape = 17.9% (87600896/488383936) finish=2881.3min speed=2317K/sec

That's very slow. An order of magnitude too slow in my view. I was
initially seeing speeds like this during my recent reshape, but got it
up to around 30MBps. I can't recall exactly what I did to get it
running better - it was the equivalent of giving the television a good
whack - but I seem to recall that increasing the md stripe cache size
helped the most.

I don't currently have access to my work computer where the command
history lives, so this is all I have for now. If I find anything more
to try, I'll chime in.

-- Kristleifur
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 5 to 6 reshape q

[Jon Hardcastle]

--- On Sat, 2/1/10, Kristleifur Daðason <kristleifur@gmail.com> wrote:

> From: Kristleifur Daðason <kristleifur@gmail.com>
> Subject: Re: Raid 5 to 6 reshape q
> To: Jon@ehardcastle.com
> Cc: linux-raid@vger.kernel.org
> Date: Saturday, 2 January, 2010, 12:19
> On Sat, Jan 2, 2010 at 8:52 AM, Jon
> Hardcastle <jd_hardcastle@yahoo.com>
> wrote:
> > Hi guys, thanks for your help getting me going with
> this over the last week. I kicked it off last night and all
> is well.
> > ...
> > The only Q i have is it seems to be abit slow?
> > ...
> > md4 : active raid6 sdg1[4] sdf1[0] sde1[1] sdd1[2]
> sdc1[5] sdb1[3] sda1[6]
> >      2441919680 blocks super 0.91 level 6, 64k
> chunk, algorithm 18 [7/6] [UUUUUU_]
> >      [===>.................]  reshape = 17.9%
> (87600896/488383936) finish=2881.3min speed=2317K/sec
> 
> That's very slow. An order of magnitude too slow in my
> view. I was
> initially seeing speeds like this during my recent reshape,
> but got it
> up to around 30MBps. I can't recall exactly what I did to
> get it
> running better - it was the equivalent of giving the
> television a good
> whack - but I seem to recall that increasing the md stripe
> cache size
> helped the most.
> 
> I don't currently have access to my work computer where the
> command
> history lives, so this is all I have for now. If I find
> anything more
> to try, I'll chime in.
> 
> -- Kristleifur
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Well it is trundling along at about that speed still if only i had know that if i was adding 2 drives it would be much quciker as i plan to add another drive at the end of the week! i figured it was more to potentially go wrong!

Anyways if you can suggest any commands that might speed it up i'd be very grateful!


-----------------------
N: Jon Hardcastle
E: Jon@eHardcastle.com
'Do not worry about tomorrow, for tomorrow will bring worries of its own.'

***********
Please note, I am phasing out jd_hardcastle AT yahoo.com and replacing it with jon AT eHardcastle.com
***********

-----------------------


      
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 5 to 6 reshape q

[Billy Crook]

Here's a couple trivial scripts I've used to pause execution until a
sync is completed.  They are handy for when you want to do multiple
grow operations, each no sooner than after the previous completed.
Modify for your own purposes.

[root@Eight bin]# cat raid-wait-till-sync-finish
#! /bin/bash

export interval=300

if [[ $1 -gt 0 ]]
then
	interval=$1
fi

echo -n Checking every ${interval} seconds for raid sync to finish.

while [[ $( cat /sys/block/md*/md/sync_action | \
            grep -v idle | \
            wc -l ) -gt 0 ]]
do
	echo -n .
	sleep ${interval}
done

echo Raid System Healthy!

exit 0
[root@Eight bin]# cat raid-wait-till-all-healthy
#! /bin/bash

export clean=no
while [[ $clean==no ]]
do
	#set clean to yes
	clean=yes

	#Give each array the chance to set clean to no
	for array in `ls -1 /sys/block/md*/md -d`
	do
		
		if [[ $( cat $array/raid_disks ) -gt $( ls -1 $array/ | grep
^rd[0-9]*$ | wc -l ) ]]
		then
			echo $array is still unclean
			clean=no
		fi
	done
	sleep 5
done



On Sun, Jan 3, 2010 at 12:07, Jon Hardcastle <jd_hardcastle@yahoo.com> wrote:
>
>
>
> --- On Sat, 2/1/10, Kristleifur Daðason <kristleifur@gmail.com> wrote:
>
>> From: Kristleifur Daðason <kristleifur@gmail.com>
>> Subject: Re: Raid 5 to 6 reshape q
>> To: Jon@ehardcastle.com
>> Cc: linux-raid@vger.kernel.org
>> Date: Saturday, 2 January, 2010, 12:19
>> On Sat, Jan 2, 2010 at 8:52 AM, Jon
>> Hardcastle <jd_hardcastle@yahoo.com>
>> wrote:
>> > Hi guys, thanks for your help getting me going with
>> this over the last week. I kicked it off last night and all
>> is well.
>> > ...
>> > The only Q i have is it seems to be abit slow?
>> > ...
>> > md4 : active raid6 sdg1[4] sdf1[0] sde1[1] sdd1[2]
>> sdc1[5] sdb1[3] sda1[6]
>> >      2441919680 blocks super 0.91 level 6, 64k
>> chunk, algorithm 18 [7/6] [UUUUUU_]
>> >      [===>.................]  reshape = 17.9%
>> (87600896/488383936) finish=2881.3min speed=2317K/sec
>>
>> That's very slow. An order of magnitude too slow in my
>> view. I was
>> initially seeing speeds like this during my recent reshape,
>> but got it
>> up to around 30MBps. I can't recall exactly what I did to
>> get it
>> running better - it was the equivalent of giving the
>> television a good
>> whack - but I seem to recall that increasing the md stripe
>> cache size
>> helped the most.
>>
>> I don't currently have access to my work computer where the
>> command
>> history lives, so this is all I have for now. If I find
>> anything more
>> to try, I'll chime in.
>>
>> -- Kristleifur
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
> Well it is trundling along at about that speed still if only i had know that if i was adding 2 drives it would be much quciker as i plan to add another drive at the end of the week! i figured it was more to potentially go wrong!
>
> Anyways if you can suggest any commands that might speed it up i'd be very grateful!
>
>
> -----------------------
> N: Jon Hardcastle
> E: Jon@eHardcastle.com
> 'Do not worry about tomorrow, for tomorrow will bring worries of its own.'
>
> ***********
> Please note, I am phasing out jd_hardcastle AT yahoo.com and replacing it with jon AT eHardcastle.com
> ***********
>
> -----------------------
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Raid 5 to 6 reshape q

[Peter Rabbitson]

Billy Crook wrote:
> Here's a couple trivial scripts I've used to pause execution until a
> sync is completed.  They are handy for when you want to do multiple
> grow operations, each no sooner than after the previous completed.
> Modify for your own purposes.
> 
> [root@Eight bin]# cat raid-wait-till-sync-finish
> #! /bin/bash
> 
> export interval=300
> 
> if [[ $1 -gt 0 ]]
> then
> 	interval=$1
> fi
> 
> echo -n Checking every ${interval} seconds for raid sync to finish.
> 
> while [[ $( cat /sys/block/md*/md/sync_action | \
>             grep -v idle | \
>             wc -l ) -gt 0 ]]
> do
> 	echo -n .
> 	sleep ${interval}
> done
> 
> echo Raid System Healthy!
> 
> exit 0
> [root@Eight bin]# cat raid-wait-till-all-healthy
> #! /bin/bash
> 
> export clean=no
> while [[ $clean==no ]]
> do
> 	#set clean to yes
> 	clean=yes
> 
> 	#Give each array the chance to set clean to no
> 	for array in `ls -1 /sys/block/md*/md -d`
> 	do
> 		
> 		if [[ $( cat $array/raid_disks ) -gt $( ls -1 $array/ | grep
> ^rd[0-9]*$ | wc -l ) ]]
> 		then
> 			echo $array is still unclean
> 			clean=no
> 		fi
> 	done
> 	sleep 5
> done
> 

Both of these can be replaced by mdadm --wait $device