Re: Question concerning RCU

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Mon, Jan 12, 2015 at 11:48:28AM +0000, Stoidner, Christoph wrote:
> Hi Paul,
> 
> > You got stack traces with the stall warnings, correct?  If so, please look
> > at them and at Documentation/RCU/stallwarn.txt and see if the kernel is
> > looping somewhere inappropriate.
> 
> Yes and no. I have a stack trace, but it is not generated by a stall warning. More
> precise: I can never see any stall warning. The reason is that the system freezes 
> when it is about to output such a warning. Instead the stack trace is generated 
> by gdb and JTAG hardware debugging, when freezing has occurred.
> 
> So I am not sure if there is really a CPU-stall condition or it is just a misrepresented
> stall detection. However, outputting a stall warning leads to system freeze. The 
> warning is never seen.

Two things to try:

1.	alt-sysreq-t to get all tasks' stacks, or
2.	disable RCU CPU stall warnings and see if the hangs go away.

Hmmm...  Are you by chance pushing all dmesg through a serial console?

> > I am not familiar with the low-level ARM kernel code, but the stack below
> > leads me to suspect that your kernel is interrupting itself to death or
> > is improperly handling interrupts.
> 
> The stack trace must be read from bottom to top. The repetitive occurrence of
> "__irq_svc () at arch/arm/kernel/entry-armv.S:202" on bottom of stack trace is 
> caused by the stack frame of the interrupt context. This is completely legal and 
> also the case in normal situations. Instead the problem is on the top of the stack 
> trace, in function rcu_print_task_stall(). The loop rcutree_plugin.h in line 528 
> never ends:
> 
> static int rcu_print_task_stall(struct rcu_node *rnp)
> {
> 	...
> 	...
> 
> 	list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
> 		printk(KERN_CONT " P%d", t->pid);
> 		ndetected++;
> 	}
> 
> 	...    
> 	...
> }        
> 
> That means list_for_each_entry_continue () never ends since rcu_node_entry.next 
> seems to point to it-self but not to rnp->blkd_tasks. I have no idea how this can
> happen.

It is not supposed to happen, and I haven't heard of it happening
anywhere else.  I do hold the appropriate lock across that code.

One thing to try would be to add a counter and break out of the loop
after (say) 10 iterations.  Is that a change you are comfortable making?

> One more thing: Just for testing I have now enabled CONFIG_TINY_PREEMPT_RCU. 
> Until now the problem has not occurred anymore. Do you have any idea what makes
> the differences here?

Any number of things, including that I am not sure that your version
of CONFIG_TINY_PREEMPT_RCU correctly detects RCU CPU stalls.  ;-)
Please note that CONFIG_TINY_PREEMPT_RCU was removed a few versions ago.

							Thanx, Paul