Re: [regression, 3.1, rcu] rcu_sched_state detected stall on CPU 8 (t=15000 jiffies)

Paul E. McKenney
2011-08-05T06:41:31Z

On Fri, Aug 05, 2011 at 10:33:16AM +1000, Dave Chinner wrote:
> On Tue, Aug 02, 2011 at 11:30:50PM -0700, Paul E. McKenney wrote:
> > On Wed, Aug 03, 2011 at 12:52:22PM +1000, Dave Chinner wrote:
> > > On Wed, Aug 03, 2011 at 12:28:57PM +1000, Dave Chinner wrote:
> > > > Hi Paul,
> > > >
> > > > I've had this hang a couple of times now, so I figured it isn't an
> > > > isolated event. I am getting kernels occassionally hanging with the
> > > > following output occurring:
> > > >
> > > > [ 62.812011] INFO: rcu_sched_state detected stall on CPU 8 (t=15000 jiffies)
> > > > [ 242.936009] INFO: rcu_sched_state detected stall on CPU 8 (t=60031 jiffies)
>
> ....
>
> > > This might be a false alarm - I've just diagnosed(*) that a kernel
> > > thread was stuck in a hard loop therefore not giving up the CPU.
> >
> > Ah, that is indeed one of the conditions that RCU CPU stall warnings
> > can catch.
> >
> > > Perhaps this is error message could be more informative?
> > > The detector is acting like the hung task detector, except it's
> > > working on kernel code stuck in a loop burning CPU, so maybe dumping
> > > a stack trace of the spinning CPU (i.e. similar to sysrq-l output)
> > > might be a useful addition to tracking down such stalls?
> >
> > Strange. There is a trigger_all_cpu_backtrace() call that is supposed
> > to dump all CPUs' stacks. It has been working in the past, but you are
> > the second person in a couple of weeks to report that it isn't doing
> > its job. (Though the other one was running the -rt tree.)
>
> Ok, so it is supposed to be dumping the stack. Good.

Yep!

> > Wait a minute... Here is the definition:
> >
> > #ifdef arch_trigger_all_cpu_backtrace
> > static inline bool trigger_all_cpu_backtrace(void)
> > {
> > arch_trigger_all_cpu_backtrace();
> >
> > return true;
> > }
> > #else
> > static inline bool trigger_all_cpu_backtrace(void)
> > {
> > return false;
> > }
> > #endif
> >
> > Passing a lower-case symbol to #ifdef is a bit of a red flag. Where
> > is it defined?
> >
> > o arch/sparc/include/asm/irq_64.h:
> >
> > #define arch_trigger_all_cpu_backtrace arch_trigger_all_cpu_backtrace
> >
> > o arch/sparc/kernel/process_64.c:
> >
> > void arch_trigger_all_cpu_backtrace(void)
> > {
> > ...
> > }
> >
> > o arch/x86/include/asm/nmi.h:
> >
> > #define arch_trigger_all_cpu_backtrace arch_trigger_all_cpu_backtrace
> >
> > o arch/x86/kernel/apic/hw_nmi.c:
> >
> > void arch_trigger_all_cpu_backtrace(void)
> > {
> > ...
> > }
> >
> > So I am guessing that you are running some architecture other than
> > x86 or SPARC. And the implementation is a bit hostile on other
> > architectures. So I suggest adding a dump_stack() before the
> > "return false" in trigger_all_cpu_backtrace(), as in the patch
> > shown below.
>
> I'm running on x86_64 (inside a KVM VM) so it should be present.

Indeed it should! Is NMI delivery busted or something?

In addition, how about putting a compiler error into
trigger_all_cpu_backtrace() and verifying that your build fails? Just in
case and all that.

> Hmmm - I note that sysrq-l has a fallback implementation that uses
> smp_call_function() should trigger_all_cpu_backtrace() return false.
> I'd bet that's why sysrq-l is working and the rcu stall detection
> isn't. i.e arch_trigger_all_cpu_backtrace() is either broken or for
> some reason not compiled in. I can't tell why - I get lost in all
> the different ways that arch specific code is inlined by
> preprocessor magic...

Interesting.

> > But this is still strange. I -know- I have seen stack dumps for
> > all CPUs when running on Power... But the code has not changed
> > for quite some time.
> >
> > Nevertheless, could you please try out the patch below? It should
> > get you at least the stack dump for the current CPU, which in your
> > case was the offending CPU.
>
> I'll give it a go, though perhaps using the same fallback as sysrq-l
> might be a better idea?

If your architecture really does believe that trigger_all_cpu_backtrace()
is available, my patch will have no effect. Substituting dump_stack()
for trigger_all_cpu_backtrace() would be helpful.

Just out of curiosity, what are you thinking of doing in the code to
figure out that trigger_all_cpu_backtrace() didn't work and that it
was time to fall back on sysrq-l-style processing?

Thanx, Paul