Re: rcu_prempt stalls / lockup

[Dave Jones <davej@redhat.com>]

On Wed, Apr 02, 2014 at 06:48:40PM -0400, Dave Jones wrote:
 
 >  > > >  > Waiting uninterruptibly.  Presumably blocked on mutex_lock().  But
 >  > > >  > you have CONFIG_PROVE_LOCKING(), so any deadlocks should have been
 >  > > >  > reported.
 >  > > > 
 >  > > > Lockdep had reported something a little earlier (timestamped at 1108.xxxxxx)
 >  > > > but that's a known false-positive in xfs.
 >  > > 
 >  > > Yep, I would be very surprised if that was related to the grace-period hang.
 >  > 
 >  > Ah, but it could be suppressing later lockdep splats.  So if this can be
 >  > reproduced without xfs, we might get additional information from lockdep.
 > 
 > Hrmph.
 > 
 > $ git bisect bad
 > The merge base 5cb480f6b488128140c940abff3c36f524a334a8 is bad.
 > This means the bug has been fixed between 5cb480f6b488128140c940abff3c36f524a334a8 and [455c6fdbd219161bd09b1165f11699d6d73de11c 62c206bd514600d4d73751ade00dca8e488390a3 e086481baf9d0436bdd6e9b739bfa4a83fb89ef5].
 > 
 > Not sure where to go from here..
 > 
 > The 'good' news is I can reproduce it pretty reliably now.
 > I start my fuzz tester, and immediately do a git diff in my working tree,
 > and then boom..

Even better, now I realise I don't even need my fuzzer in the mix. Just doing
a fair amount of disk io (like a git diff on a dirty tree) will trigger it.

I've tried adding a show_state() call when the stall happens, but another stall
seems to occur before it gets a chance to even dump everything over the usb-serial console.
And of course nothing ever makes it to disk, even though I can sysrq-sync, on the
next reboot systemd has stuffed a bunch of ^@ in the log where the interesting
stuff should be.

Any other ideas ?

	Dave