Software RAID (non-preempt) server blocking question. (2.6.20.4)

Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Justin Piszcz]

I ran a check on my SW RAID devices this morning.  However, when I did so, 
I had a few lftp sessions open pulling files.  After I executed the check, 
the lftp processes entered 'D' state and I could do 'nothing' in the 
process until the check finished.  Is this normal?  Should a check block 
all I/O to the device and put the processes writing to a particular device 
in 'D' state until it is finished?

Justin.

Re: Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Neil Brown]

On Tuesday March 27, jpiszcz@lucidpixels.com wrote:
> I ran a check on my SW RAID devices this morning.  However, when I did so, 
> I had a few lftp sessions open pulling files.  After I executed the check, 
> the lftp processes entered 'D' state and I could do 'nothing' in the 
> process until the check finished.  Is this normal?  Should a check block 
> all I/O to the device and put the processes writing to a particular device 
> in 'D' state until it is finished?

No, that shouldn't happen.  The 'check' should notice any other disk
activity and slow down if anything else is happening on the device.

Did the check run to completion?  And if so, did the 'lftp' start
working normally again?

Did you look at "cat /proc/mdstat" ?? What sort of speed was the check
running at?

NeilBrown

Re: Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Justin Piszcz]

On Thu, 29 Mar 2007, Neil Brown wrote:

> On Tuesday March 27, jpiszcz@lucidpixels.com wrote:
>> I ran a check on my SW RAID devices this morning.  However, when I did so,
>> I had a few lftp sessions open pulling files.  After I executed the check,
>> the lftp processes entered 'D' state and I could do 'nothing' in the
>> process until the check finished.  Is this normal?  Should a check block
>> all I/O to the device and put the processes writing to a particular device
>> in 'D' state until it is finished?
>
> No, that shouldn't happen.  The 'check' should notice any other disk
> activity and slow down if anything else is happening on the device.
>
> Did the check run to completion?  And if so, did the 'lftp' start
> working normally again?
Yes it did and the lftp did start working normally again.

>
> Did you look at "cat /proc/mdstat" ?? What sort of speed was the check
> running at?
Around 44MB/s.

I do use the following optimization, perhaps a bad idea if I want other 
processes to 'stay alive'?

echo "Setting minimum resync speed to 200MB/s..."
echo "This improves the resync speed from 2.1MB/s to 44MB/s"
echo 200000 > /sys/block/md0/md/sync_speed_min
echo 200000 > /sys/block/md1/md/sync_speed_min
echo 200000 > /sys/block/md2/md/sync_speed_min
echo 200000 > /sys/block/md3/md/sync_speed_min
echo 200000 > /sys/block/md4/md/sync_speed_min



>
> NeilBrown
>

Re: Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Henrique de Moraes Holschuh]

On Thu, 29 Mar 2007, Justin Piszcz wrote:
> >Did you look at "cat /proc/mdstat" ?? What sort of speed was the check
> >running at?
> Around 44MB/s.
> 
> I do use the following optimization, perhaps a bad idea if I want other 
> processes to 'stay alive'?
> 
> echo "Setting minimum resync speed to 200MB/s..."
> echo "This improves the resync speed from 2.1MB/s to 44MB/s"
> echo 200000 > /sys/block/md0/md/sync_speed_min
> echo 200000 > /sys/block/md1/md/sync_speed_min
> echo 200000 > /sys/block/md2/md/sync_speed_min
> echo 200000 > /sys/block/md3/md/sync_speed_min
> echo 200000 > /sys/block/md4/md/sync_speed_min

md RAID1 resync reacts *extremely* badly to CFQ.  Just a data point, you may
want to check on it.  Might mean other RAID types also get screwed, and also
that md "check" is also disturbed by CFQ (or disturbs CFQ, whatever).

I reverted everything here to non-CFQ while the RAID did its resync (which
fixed all issues immediately), and we went back to 2.6.16.x later for other
reasons.

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Justin Piszcz]

On Thu, 29 Mar 2007, Henrique de Moraes Holschuh wrote:

> On Thu, 29 Mar 2007, Justin Piszcz wrote:
>>> Did you look at "cat /proc/mdstat" ?? What sort of speed was the check
>>> running at?
>> Around 44MB/s.
>>
>> I do use the following optimization, perhaps a bad idea if I want other
>> processes to 'stay alive'?
>>
>> echo "Setting minimum resync speed to 200MB/s..."
>> echo "This improves the resync speed from 2.1MB/s to 44MB/s"
>> echo 200000 > /sys/block/md0/md/sync_speed_min
>> echo 200000 > /sys/block/md1/md/sync_speed_min
>> echo 200000 > /sys/block/md2/md/sync_speed_min
>> echo 200000 > /sys/block/md3/md/sync_speed_min
>> echo 200000 > /sys/block/md4/md/sync_speed_min
>
> md RAID1 resync reacts *extremely* badly to CFQ.  Just a data point, you may
> want to check on it.  Might mean other RAID types also get screwed, and also
> that md "check" is also disturbed by CFQ (or disturbs CFQ, whatever).
>
> I reverted everything here to non-CFQ while the RAID did its resync (which
> fixed all issues immediately), and we went back to 2.6.16.x later for other
> reasons.
>
> --
>  "One disk to rule them all, One disk to find them. One disk to bring
>  them all and in the darkness grind them. In the Land of Redmond
>  where the shadows lie." -- The Silicon Valley Tarot
>  Henrique Holschuh
>

I am using the AS scheduler; not CFQ.

$ find /sys 2>/dev/null|grep -i scheduler|xargs -n1 cat
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]
noop [anticipatory]

Justin.

Re: Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Neil Brown]

On Thursday March 29, jpiszcz@lucidpixels.com wrote:
> 
> >
> > Did you look at "cat /proc/mdstat" ?? What sort of speed was the check
> > running at?
> Around 44MB/s.
> 
> I do use the following optimization, perhaps a bad idea if I want other 
> processes to 'stay alive'?
> 
> echo "Setting minimum resync speed to 200MB/s..."
> echo "This improves the resync speed from 2.1MB/s to 44MB/s"
> echo 200000 > /sys/block/md0/md/sync_speed_min
> echo 200000 > /sys/block/md1/md/sync_speed_min
> echo 200000 > /sys/block/md2/md/sync_speed_min
> echo 200000 > /sys/block/md3/md/sync_speed_min
> echo 200000 > /sys/block/md4/md/sync_speed_min
> 

Yes, well....

You told it to use up to 200MB/s and the drives are only delivering
44MB/s, so they will be taking nearly all of the available bandwidth.
You shouldn't be too surprised if other things suffer.

NeilBrown

Re: Software RAID (non-preempt) server blocking question. (2.6.20.4)

[Justin Piszcz]

On Fri, 30 Mar 2007, Neil Brown wrote:

> On Thursday March 29, jpiszcz@lucidpixels.com wrote:
>>
>>>
>>> Did you look at "cat /proc/mdstat" ?? What sort of speed was the check
>>> running at?
>> Around 44MB/s.
>>
>> I do use the following optimization, perhaps a bad idea if I want other
>> processes to 'stay alive'?
>>
>> echo "Setting minimum resync speed to 200MB/s..."
>> echo "This improves the resync speed from 2.1MB/s to 44MB/s"
>> echo 200000 > /sys/block/md0/md/sync_speed_min
>> echo 200000 > /sys/block/md1/md/sync_speed_min
>> echo 200000 > /sys/block/md2/md/sync_speed_min
>> echo 200000 > /sys/block/md3/md/sync_speed_min
>> echo 200000 > /sys/block/md4/md/sync_speed_min
>>
>
> Yes, well....
>
> You told it to use up to 200MB/s and the drives are only delivering
> 44MB/s, so they will be taking nearly all of the available bandwidth.
> You shouldn't be too surprised if other things suffer.
>
> NeilBrown
>

Understood, will reduce this, thanks.

Justin.