Curious problem with asynchronous RAID1 rebuild operations

Curious problem with asynchronous RAID1 rebuild operations

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

I tried to rebuild/sync two RAID1 arrays at the same time, with strange results.

When a new drive was added into md0, it started to sync fine at
123,000kb/sec. Whilst that was running, I also added a new drive into
md1, and this causes both arrays sync speed to slow down to
20,000kb/sec. I checked the /proc/sys/dev/raid/speed_limit_{min|max}
and these were both set fine.

I removed the drive from md1, and the md0 sync went back to the normal speed.

RAID Setup:
md0 - 2 drives (2TB Western Digital)
md1 - 2 drives (500GB M3 SDD)

So my question is, does anyone know why the sync speed (on all arrays)
slowed down to almost 6-7 times the norm, when two (or more?) sync
operations are taking place?

Thanks
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Curious problem with asynchronous RAID1 rebuild operations

[NeilBrown]

On Sat, 2 Jul 2011 12:07:54 +0100 "Cal Leeming [Simplicity Media Ltd]"
<cal.leeming@simplicitymedialtd.co.uk> wrote:

> Hi all,
> 
> I tried to rebuild/sync two RAID1 arrays at the same time, with strange results.
> 
> When a new drive was added into md0, it started to sync fine at
> 123,000kb/sec. Whilst that was running, I also added a new drive into
> md1, and this causes both arrays sync speed to slow down to
> 20,000kb/sec. I checked the /proc/sys/dev/raid/speed_limit_{min|max}
> and these were both set fine.
> 
> I removed the drive from md1, and the md0 sync went back to the normal speed.
> 
> RAID Setup:
> md0 - 2 drives (2TB Western Digital)
> md1 - 2 drives (500GB M3 SDD)
> 
> So my question is, does anyone know why the sync speed (on all arrays)
> slowed down to almost 6-7 times the norm, when two (or more?) sync
> operations are taking place?
> 

Dodgy controller?  I assume both arrays are on the same controller.

Can you plug in a separate controller even if only temporarily and see what
happens?

I can see how any interaction with md would have this effect.

NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Curious problem with asynchronous RAID1 rebuild operations

[Cal Leeming [Simplicity Media Ltd]]

Thanks for the reply.

Sadly, this specific server is 150 miles away in colo, and only has a
single on-board controller (it's a Dell R310).

I'd be surprised if it was controller/contention related, but I
certainly won't rule it out.

Cal

On Tue, Jul 5, 2011 at 1:34 AM, NeilBrown <neilb@suse.de> wrote:
> On Sat, 2 Jul 2011 12:07:54 +0100 "Cal Leeming [Simplicity Media Ltd]"
> <cal.leeming@simplicitymedialtd.co.uk> wrote:
>
>> Hi all,
>>
>> I tried to rebuild/sync two RAID1 arrays at the same time, with strange results.
>>
>> When a new drive was added into md0, it started to sync fine at
>> 123,000kb/sec. Whilst that was running, I also added a new drive into
>> md1, and this causes both arrays sync speed to slow down to
>> 20,000kb/sec. I checked the /proc/sys/dev/raid/speed_limit_{min|max}
>> and these were both set fine.
>>
>> I removed the drive from md1, and the md0 sync went back to the normal speed.
>>
>> RAID Setup:
>> md0 - 2 drives (2TB Western Digital)
>> md1 - 2 drives (500GB M3 SDD)
>>
>> So my question is, does anyone know why the sync speed (on all arrays)
>> slowed down to almost 6-7 times the norm, when two (or more?) sync
>> operations are taking place?
>>
>
> Dodgy controller?  I assume both arrays are on the same controller.
>
> Can you plug in a separate controller even if only temporarily and see what
> happens?
>
> I can see how any interaction with md would have this effect.
>
> NeilBrown
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html