Raid stalls during --replace and other disk activity...

Raid stalls during --replace and other disk activity...

[Scott D'Vileskis]

I have a 4 disk RAID5.  I added a spare disk and am in the process of
--replace[ing] a disk in the array.

~# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md5 : active raid5 sde1[4](R) sda1[1] sdb1[7] sdd1[8] sdf1[0]
      4395302400 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      [==============>......]  recovery = 71.7%
(1051281152/1465100800) finish=334.3min speed=20628K/sec

I am replacing /dev/sdf with /dev/sde. This WAS running at about 120MB/second.

However, when I started a format of an unrelated partition on
/dev/sda, the RAID 'recovery' disk-replacement completely stopped. I
suspect the RAID rebuild mechanism is pausing because of the activity
on /dev/sda.

I monkeyed with
echo 90000 > speed_limit_min
and now I get 10 second bursts of 20-100MB/sec every 30 seconds or so,
so at least the jobs will finish in parallel..

I feel the RAID --replace option should only concern itself with the
two disks involved not get caught up because of activity on another
disk. Is this a bug?

Thoughts?

Re: Raid stalls during --replace and other disk activity...

[Phil Turmel]

On 03/15/2014 12:28 AM, Scott D'Vileskis wrote:

> However, when I started a format of an unrelated partition on
> /dev/sda, the RAID 'recovery' disk-replacement completely stopped. I
> suspect the RAID rebuild mechanism is pausing because of the activity
> on /dev/sda.
> 
> I monkeyed with
> echo 90000 > speed_limit_min
> and now I get 10 second bursts of 20-100MB/sec every 30 seconds or so,
> so at least the jobs will finish in parallel..
> 
> I feel the RAID --replace option should only concern itself with the
> two disks involved not get caught up because of activity on another
> disk. Is this a bug?

This is deliberate, and in my opinion, desirable.  Same thing happens
when doing a check or any kind of rebuild on multiple arrays that use
partitions on shared underlying devices.  If you don't limit the raid
background activity, the extra seek load on the devices can severely cut
performance, especially on spinning rust.

Phil

Re: Raid stalls during --replace and other disk activity...

[Scott D'Vileskis]

>
> This is deliberate, and in my opinion, desirable.  Same thing happens
> when doing a check or any kind of rebuild on multiple arrays that use
> partitions on shared underlying devices.  If you don't limit the raid
> background activity, the extra seek load on the devices can severely cut
> performance, especially on spinning rust.
>

If I was doing a 'normal' resync, I would also agree.

But my point was this...
My raid was doing a --replace of /dev/sdf1 with /dev/sde1
It was only reading/writing from/to those two devices, not the other
disks in the raid. (i.e. /dev/sd[a-d] were not being used)
The other job I started (dd if=/dev/zero of=/dev/sda2 bs=1M) was only
writing to /dev/sda

These jobs should be are perfectly capable of running in parallel
without slowing eachother down.

I think the logic that pauses a resync on 'related' disk activity
should treat a --replace slightly different

Re: Raid stalls during --replace and other disk activity...

[Phil Turmel]

On 03/15/2014 03:53 PM, Scott D'Vileskis wrote:
>>
>> This is deliberate, and in my opinion, desirable.  Same thing happens
>> when doing a check or any kind of rebuild on multiple arrays that use
>> partitions on shared underlying devices.  If you don't limit the raid
>> background activity, the extra seek load on the devices can severely cut
>> performance, especially on spinning rust.
>>
> 
> If I was doing a 'normal' resync, I would also agree.
> 
> But my point was this...
> My raid was doing a --replace of /dev/sdf1 with /dev/sde1
> It was only reading/writing from/to those two devices, not the other
> disks in the raid. (i.e. /dev/sd[a-d] were not being used)
> The other job I started (dd if=/dev/zero of=/dev/sda2 bs=1M) was only
> writing to /dev/sda
> 
> These jobs should be are perfectly capable of running in parallel
> without slowing eachother down.

Good point.  If I were Neil, I'd say "Patch Welcome!".  :-)

> I think the logic that pauses a resync on 'related' disk activity
> should treat a --replace slightly different

But I would insist that the solution be generic -- make the existing
logic notice that the activity is on unrelated members, so all
background activity would benefit from the new algorithm.  IMHO.

Phil