New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[Niccolo Rigacci]

Thanks to the several guys in this list, I have solved my problem 
and elaborated this, can be a new FAQ entry?



Q: Sometimes when a RAID volume is resyncing, the system seems to 
locks-up: every disk activity is blocked until resync is done.

A: This is not strictly related to Linux RAID, this is a problem 
related to the Linux kernel and the disk subsytem: in no 
circumstances a process should get all the disk resources 
preventing others to access them.

You can control the max speed at which RAID reconstruction is 
done by setting it, say at 5 Mb/s:

  echo 5000 > /proc/sys/dev/raid/speed_limit_max

This is just a workaround, you have to determine the max speed 
that does not lock your system by trial and error and you cannot 
predict what will be the disk load in the future when the RAID 
will be resyncing for some reason.

Starting from version 2.6, Linux kernel has several choices about 
the I/O scheduler to be used. The default is the anticipatory 
scheduler, which seems to be sub-optimal on resync high load. If 
your kernel has the CFQ scheduler compiled in, use it during 
resync.

From the command line you can see which schedulers are supported 
and change it on the fly (remember to do it for each RAID disk):

  # cat /sys/block/hda/queue/scheduler
  noop [anticipatory] deadline cfq
  # echo cfq > /sys/block/hda/queue/scheduler

Otherwise you can recompile your kernel and set CFQ as the 
default I/O scheduler (CONFIG_DEFAULT_CFQ=y in Block layer, IO 
Schedulers, Default I/O scheduler).


-- 
Niccolo Rigacci
Firenze - Italy

Iraq, missione di pace: 38475 morti - www.iraqbodycount.net

Re: New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[David Greaves]

OK :)

David

Niccolo Rigacci wrote:
> Thanks to the several guys in this list, I have solved my problem 
> and elaborated this, can be a new FAQ entry?
>
>
>
> Q: Sometimes when a RAID volume is resyncing, the system seems to 
> locks-up: every disk activity is blocked until resync is done.
>
> A: This is not strictly related to Linux RAID, this is a problem 
> related to the Linux kernel and the disk subsytem: in no 
> circumstances a process should get all the disk resources 
> preventing others to access them.
>
> You can control the max speed at which RAID reconstruction is 
> done by setting it, say at 5 Mb/s:
>
>   echo 5000 > /proc/sys/dev/raid/speed_limit_max
>
> This is just a workaround, you have to determine the max speed 
> that does not lock your system by trial and error and you cannot 
> predict what will be the disk load in the future when the RAID 
> will be resyncing for some reason.
>
> Starting from version 2.6, Linux kernel has several choices about 
> the I/O scheduler to be used. The default is the anticipatory 
> scheduler, which seems to be sub-optimal on resync high load. If 
> your kernel has the CFQ scheduler compiled in, use it during 
> resync.
>
> >From the command line you can see which schedulers are supported 
> and change it on the fly (remember to do it for each RAID disk):
>
>   # cat /sys/block/hda/queue/scheduler
>   noop [anticipatory] deadline cfq
>   # echo cfq > /sys/block/hda/queue/scheduler
>
> Otherwise you can recompile your kernel and set CFQ as the 
> default I/O scheduler (CONFIG_DEFAULT_CFQ=y in Block layer, IO 
> Schedulers, Default I/O scheduler).
>
>
>   


-- 

Re: New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[Michael Tokarev]

Niccolo Rigacci wrote:
[]
> From the command line you can see which schedulers are supported 
> and change it on the fly (remember to do it for each RAID disk):
> 
>   # cat /sys/block/hda/queue/scheduler
>   noop [anticipatory] deadline cfq
>   # echo cfq > /sys/block/hda/queue/scheduler
> 
> Otherwise you can recompile your kernel and set CFQ as the 
> default I/O scheduler (CONFIG_DEFAULT_CFQ=y in Block layer, IO 
> Schedulers, Default I/O scheduler).

There's much easier/simpler way to set default scheduler.  As
someone suggested, RTFM Documentation/kernel-parameters.txt.
Passing elevator=cfq (or whatever) will do the trick much simpler
than kernel recompile.

/mjt

Re: New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[Mark Hahn]

> There's much easier/simpler way to set default scheduler.  As

personally, I don't see any point to worrying about the default,
compile-time or boot time:

for f in `find /sys/block/* -name scheduler`; do echo cfq > $f; done

Re: New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[Gil]

Mark Hahn wrote:
>> There's much easier/simpler way to set default scheduler.  As
> 
> personally, I don't see any point to worrying about the default,
> compile-time or boot time:
> 
> for f in `find /sys/block/* -name scheduler`; do echo cfq > $f; done

I agree -- if you're talking about changing the io scheduler for the
duration of a resync you should take this approach rather than
changing kernels or rebooting.

--Gil

Re: New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[Niccolo Rigacci]

> > personally, I don't see any point to worrying about the default,
> > compile-time or boot time:
> > 
> > for f in `find /sys/block/* -name scheduler`; do echo cfq > $f; done

I tested this case:

- reboot as per power failure (RAID goes dirty)
- RAID start resyncing as soon as the kernel assemble it
- every disk activity is blocked, even DHCP failed!
- host services are unavailable

This is why I changed the kernel default.

-- 
Niccolo Rigacci
Firenze - Italy

Iraq, missione di pace: 38475 morti - www.iraqbodycount.net

Re: New FAQ entry? (was IBM xSeries stop responding during RAID1 reconstruction)

[Bill Davidsen]

Niccolo Rigacci wrote:

>>>personally, I don't see any point to worrying about the default,
>>>compile-time or boot time:
>>>
>>>for f in `find /sys/block/* -name scheduler`; do echo cfq > $f; done
>>>      
>>>
>
>I tested this case:
>
>- reboot as per power failure (RAID goes dirty)
>- RAID start resyncing as soon as the kernel assemble it
>- every disk activity is blocked, even DHCP failed!
>- host services are unavailable
>
>This is why I changed the kernel default.
>
>  
>
Changing on the command line assumes that you built all of the 
schedulers in... but making that assumption, perhaps the correct 
fail-safe is to have cfq as the default, and at the end of rc.local 
check for rebuild, and if everything is clean change to whatever work 
best at the end of the boot. If the raid is not clean stay with cfq.

Has anyone tried deadline for this? I think I had this as deafult and 
didn't hand on a raid5 fail/rebuild.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979