Re: Raid5 hang in 3.14.19

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

On Sun, 28 Sep 2014 22:56:19 -0500 BillStuff <billstuff2001@sbcglobal.net>
wrote:

> On 09/28/2014 09:25 PM, NeilBrown wrote:
> > On Fri, 26 Sep 2014 17:33:58 -0500 BillStuff <billstuff2001@sbcglobal.net>
> > wrote:
> >
> >> Hi Neil,
> >>
> >> I found something that looks similar to the problem described in
> >> "Re: seems like a deadlock in workqueue when md do a flush" from Sept 14th.
> >>
> >> It's on 3.14.19 with 7 recent patches for fixing raid1 recovery hangs.
> >>
> >> on this array:
> >> md3 : active raid5 sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1] sda1[0]
> >>         104171200 blocks level 5, 64k chunk, algorithm 2 [6/6] [UUUUUU]
> >>         bitmap: 1/5 pages [4KB], 2048KB chunk
> >>
> >> I was running a test doing parallel kernel builds, read/write loops, and
> >> disk add / remove / check loops,
> >> on both this array and a raid1 array.
> >>
> >> I was trying to stress test your recent raid1 fixes, which went well,
> >> but then after 5 days,
> >> the raid5 array hung up with this in dmesg:
> > I think this is different to the workqueue problem you mentioned, though as I
> > don't know exactly what caused either I cannot be certain.
> >
> >   From the data you provided it looks like everything is waiting on
> > get_active_stripe(), or on a process that is waiting on that.
> > That seems pretty common whenever anything goes wrong in raid5 :-(
> >
> > The md3_raid5 task is listed as blocked, but not stack trace is given.
> > If the machine is still in the state, then
> >
> >   cat /proc/1698/stack
> >
> > might be useful.
> > (echo t > /proc/sysrq-trigger is always a good idea)
> 
> Might this help? I believe the array was doing a "check" when things 
> hung up.

It looks like it was trying to start doing a 'check'.
The 'resync' thread hadn't been started yet.
What is 'kthreadd' doing?
My guess is that it is in try_to_free_pages() waiting for writeout
for some xfs file page onto the md array ... which won't progress until
the thread gets started.

That would suggest that we need an async way to start threads...

Thanks,
NeilBrown


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]