From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754791AbbALToc (ORCPT ); Mon, 12 Jan 2015 14:44:32 -0500 Received: from e31.co.us.ibm.com ([32.97.110.149]:57807 "EHLO e31.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752340AbbALToa (ORCPT ); Mon, 12 Jan 2015 14:44:30 -0500 Date: Mon, 12 Jan 2015 11:44:12 -0800 From: "Paul E. McKenney" To: "Stoidner, Christoph" Cc: "linux-kernel@vger.kernel.org" Subject: Re: Question concerning RCU Message-ID: <20150112194412.GE9719@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <94921a144a97457385ae95b838c3c6fa@EX132MBOX1A.de2.local> <20150106194317.GG5280@linux.vnet.ibm.com> <81b94fc89c774b71a967fc93823e9c63@EX132MBOX1A.de2.local> <20150111202604.GC8063@linux.vnet.ibm.com> <249b36ec1afa43a699b281a4192c6afd@EX132MBOX1A.de2.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <249b36ec1afa43a699b281a4192c6afd@EX132MBOX1A.de2.local> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15011219-8236-0000-0000-0000088DE2F4 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jan 12, 2015 at 11:48:28AM +0000, Stoidner, Christoph wrote: > Hi Paul, > > > You got stack traces with the stall warnings, correct? If so, please look > > at them and at Documentation/RCU/stallwarn.txt and see if the kernel is > > looping somewhere inappropriate. > > Yes and no. I have a stack trace, but it is not generated by a stall warning. More > precise: I can never see any stall warning. The reason is that the system freezes > when it is about to output such a warning. Instead the stack trace is generated > by gdb and JTAG hardware debugging, when freezing has occurred. > > So I am not sure if there is really a CPU-stall condition or it is just a misrepresented > stall detection. However, outputting a stall warning leads to system freeze. The > warning is never seen. Two things to try: 1. alt-sysreq-t to get all tasks' stacks, or 2. disable RCU CPU stall warnings and see if the hangs go away. Hmmm... Are you by chance pushing all dmesg through a serial console? > > I am not familiar with the low-level ARM kernel code, but the stack below > > leads me to suspect that your kernel is interrupting itself to death or > > is improperly handling interrupts. > > The stack trace must be read from bottom to top. The repetitive occurrence of > "__irq_svc () at arch/arm/kernel/entry-armv.S:202" on bottom of stack trace is > caused by the stack frame of the interrupt context. This is completely legal and > also the case in normal situations. Instead the problem is on the top of the stack > trace, in function rcu_print_task_stall(). The loop rcutree_plugin.h in line 528 > never ends: > > static int rcu_print_task_stall(struct rcu_node *rnp) > { > ... > ... > > list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) { > printk(KERN_CONT " P%d", t->pid); > ndetected++; > } > > ... > ... > } > > That means list_for_each_entry_continue () never ends since rcu_node_entry.next > seems to point to it-self but not to rnp->blkd_tasks. I have no idea how this can > happen. It is not supposed to happen, and I haven't heard of it happening anywhere else. I do hold the appropriate lock across that code. One thing to try would be to add a counter and break out of the loop after (say) 10 iterations. Is that a change you are comfortable making? > One more thing: Just for testing I have now enabled CONFIG_TINY_PREEMPT_RCU. > Until now the problem has not occurred anymore. Do you have any idea what makes > the differences here? Any number of things, including that I am not sure that your version of CONFIG_TINY_PREEMPT_RCU correctly detects RCU CPU stalls. ;-) Please note that CONFIG_TINY_PREEMPT_RCU was removed a few versions ago. Thanx, Paul