Re: [regression, 3.1, rcu] rcu_sched_state detected stall on CPU 8 (t=15000 jiffies)

[Dave Chinner <david@fromorbit.com>]

On Tue, Aug 02, 2011 at 11:30:50PM -0700, Paul E. McKenney wrote:
> On Wed, Aug 03, 2011 at 12:52:22PM +1000, Dave Chinner wrote:
> > On Wed, Aug 03, 2011 at 12:28:57PM +1000, Dave Chinner wrote:
> > > Hi Paul,
> > > 
> > > I've had this hang a couple of times now, so I figured it isn't an
> > > isolated event. I am getting kernels occassionally hanging with the
> > > following output occurring:
> > > 
> > > [   62.812011] INFO: rcu_sched_state detected stall on CPU 8 (t=15000 jiffies)
> > > [  242.936009] INFO: rcu_sched_state detected stall on CPU 8 (t=60031 jiffies)

....

> > This might be a false alarm - I've just diagnosed(*) that a kernel
> > thread was stuck in a hard loop therefore not giving up the CPU.
> 
> Ah, that is indeed one of the conditions that RCU CPU stall warnings
> can catch.
> 
> > Perhaps this is error message could be more informative?
> > The detector is acting like the hung task detector, except it's
> > working on kernel code stuck in a loop burning CPU, so maybe dumping
> > a stack trace of the spinning CPU (i.e. similar to sysrq-l output)
> > might be a useful addition to tracking down such stalls?
> 
> Strange.  There is a trigger_all_cpu_backtrace() call that is supposed
> to dump all CPUs' stacks.  It has been working in the past, but you are
> the second person in a couple of weeks to report that it isn't doing
> its job.  (Though the other one was running the -rt tree.)

Ok, so it is supposed to be dumping the stack. Good.

> 
> Wait a minute...  Here is the definition:
> 
> 	#ifdef arch_trigger_all_cpu_backtrace
> 	static inline bool trigger_all_cpu_backtrace(void)
> 	{
> 		arch_trigger_all_cpu_backtrace();
> 
> 		return true;
> 	}
> 	#else
> 	static inline bool trigger_all_cpu_backtrace(void)
> 	{
> 		return false;
> 	}
> 	#endif
> 
> Passing a lower-case symbol to #ifdef is a bit of a red flag.  Where
> is it defined?
> 
> o	arch/sparc/include/asm/irq_64.h:
> 
> 	#define arch_trigger_all_cpu_backtrace arch_trigger_all_cpu_backtrace
> 
> o	arch/sparc/kernel/process_64.c:
> 
> 	void arch_trigger_all_cpu_backtrace(void)
> 	{
> 		...
> 	}
> 
> o	arch/x86/include/asm/nmi.h:
> 
> 	#define arch_trigger_all_cpu_backtrace arch_trigger_all_cpu_backtrace
> 
> o	arch/x86/kernel/apic/hw_nmi.c:
> 
> 	void arch_trigger_all_cpu_backtrace(void)
> 	{
> 		...
> 	}
> 
> So I am guessing that you are running some architecture other than
> x86 or SPARC.  And the implementation is a bit hostile on other
> architectures.  So I suggest adding a dump_stack() before the
> "return false" in trigger_all_cpu_backtrace(), as in the patch
> shown below.

I'm running on x86_64 (inside a KVM VM) so it should be present.

Hmmm - I note that sysrq-l has a fallback implementation that uses
smp_call_function() should trigger_all_cpu_backtrace() return false.
I'd bet that's why sysrq-l is working and the rcu stall detection
isn't. i.e arch_trigger_all_cpu_backtrace() is either broken or for
some reason not compiled in. I can't tell why - I get lost in all
the different ways that arch specific code is inlined by
preprocessor magic...

> But this is still strange.  I -know- I have seen stack dumps for
> all CPUs when running on Power...  But the code has not changed
> for quite some time.
> 
> Nevertheless, could you please try out the patch below?  It should
> get you at least the stack dump for the current CPU, which in your
> case was the offending CPU.

I'll give it a go, though perhaps using the same fallback as sysrq-l
might be a better idea?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com