sw raid5 hungs on resync and high IO load, 2.6.32.23

[Martin Hamrle <martin.hamrle@nangu.tv>]

Hi,

I'm having this issue on several boxes with several configuration.
One of them is a box with 8 drives attached to ARC-1160 in pass through
mode and build sw raid5 from these drives. There is also one drive to OS.

During resync or check and heavy IO load, process tscpd (tscpd is IO
load maker) hungs, the machine is still alive but there are many blocked
processes.
After tscpd hungs, IO load is generated only by resync. In traceback you
can see blocked processes (ps, htop cat) accessing tscpd cmdline in
proc. Some tscpd threads is blocked during writing files into fs on
raid5. Reading these files is also blocking, reading other files in
filesystem is fast as usual.  This state takes 110 minutes. After that
all blocked processes continue their work.

I am not sure what is the reason of the end of the weird state. I think
the end was caused by starting copying kernel source into array.

Note that this is first time when hung processes wake up I never wait so
long.

I think that it is related to sw raid because I do not see this issue on
hw raid or on sw raid without resync.

kern.log contains initial "INFO: task collectd:2577 blocked for more
than 120 seconds"
   and two dumps
echo w > /proc/sysrq-trigger

log is located http://files.nangu.tv/kernel/kern.log
Let me know if you need more info.

Martin