From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751717AbcFCDjv (ORCPT ); Thu, 2 Jun 2016 23:39:51 -0400 Received: from e31.co.us.ibm.com ([32.97.110.149]:32768 "EHLO e31.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750993AbcFCDjt (ORCPT ); Thu, 2 Jun 2016 23:39:49 -0400 X-IBM-Helo: d03dlp01.boulder.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org Date: Thu, 2 Jun 2016 20:39:51 -0700 From: "Paul E. McKenney" To: Daniel Bristot de Oliveira Cc: linux-kernel@vger.kernel.org, Jonathan Corbet , Josh Triplett , Steven Rostedt , Mathieu Desnoyers , Lai Jiangshan , Christian Borntraeger , Arnaldo Carvalho de Melo , "Luis Claudio R. Goncalves" Subject: Re: [PATCH v2] rcu: sysctl: Panic on RCU Stall Message-ID: <20160603033951.GO5231@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <41f0764c4da7e43f2d1f9ead7010a36b27f94744.1464743544.git.bristot@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41f0764c4da7e43f2d1f9ead7010a36b27f94744.1464743544.git.bristot@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16060303-8236-0000-0000-000031DC0173 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Jun 02, 2016 at 01:51:41PM -0300, Daniel Bristot de Oliveira wrote: > It is not always easy to define the cause of an RCU stall just by > analysing the RCU stall messages, mainly when the problem is caused > by the indirect starvation of rcu threads. For example, when preempt_rcu > is not awakened due to the starvation of a timer softirq. > > We have been hard coding panic() in the RCU stall functions for > some time while testing the kernel-rt. But this is not possible in > some scenarios, like when supporting customers. > > This patch implements the sysctl kernel.panic_on_rcu_stall. If > set to 1, the system will panic() when an RCU stall takes place, > enabling the capture of a vmcore. The vmcore provides a way to analyze > all kernel/tasks states, helping out to point to the culprit and the > solution for the stall. > > The kernel.panic_on_rcu_stall sysctl is disabled by default. > > Changes from v1: > - Fixed a typo in the git log > - The if(sysctl_panic_on_rcu_stall) panic() is in a static function > - Fixed the CONFIG_TINY_RCU compilation issue > - The var sysctl_panic_on_rcu_stall is now __read_mostly > > Cc: Jonathan Corbet > Cc: "Paul E. McKenney" > Cc: Josh Triplett > Cc: Steven Rostedt > Cc: Mathieu Desnoyers > Cc: Lai Jiangshan > Acked-by: Christian Borntraeger > Reviewed-by: Josh Triplett > Reviewed-by: Arnaldo Carvalho de Melo > Tested-by: "Luis Claudio R. Goncalves" > Signed-off-by: Daniel Bristot de Oliveira Queued for testing and further review. Thanx, Paul > --- > Documentation/sysctl/kernel.txt | 12 ++++++++++++ > include/linux/kernel.h | 1 + > kernel/rcu/tree.c | 12 ++++++++++++ > kernel/sysctl.c | 11 +++++++++++ > 4 files changed, 36 insertions(+) > > diff --git a/Documentation/sysctl/kernel.txt b/Documentation/sysctl/kernel.txt > index a3683ce..3320460 100644 > --- a/Documentation/sysctl/kernel.txt > +++ b/Documentation/sysctl/kernel.txt > @@ -58,6 +58,7 @@ show up in /proc/sys/kernel: > - panic_on_stackoverflow > - panic_on_unrecovered_nmi > - panic_on_warn > +- panic_on_rcu_stall > - perf_cpu_time_max_percent > - perf_event_paranoid > - perf_event_max_stack > @@ -618,6 +619,17 @@ a kernel rebuild when attempting to kdump at the location of a WARN(). > > ============================================================== > > +panic_on_rcu_stall: > + > +When set to 1, calls panic() after RCU stall detection messages. This > +is useful to define the root cause of RCU stalls using a vmcore. > + > +0: do not panic() when RCU stall takes place, default behavior. > + > +1: panic() after printing RCU stall messages. > + > +============================================================== > + > perf_cpu_time_max_percent: > > Hints to the kernel how much CPU time it should be allowed to > diff --git a/include/linux/kernel.h b/include/linux/kernel.h > index 94aa10f..c420821 100644 > --- a/include/linux/kernel.h > +++ b/include/linux/kernel.h > @@ -451,6 +451,7 @@ extern int panic_on_oops; > extern int panic_on_unrecovered_nmi; > extern int panic_on_io_nmi; > extern int panic_on_warn; > +extern int sysctl_panic_on_rcu_stall; > extern int sysctl_panic_on_stackoverflow; > > extern bool crash_kexec_post_notifiers; > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c > index c7f1bc4..d531988 100644 > --- a/kernel/rcu/tree.c > +++ b/kernel/rcu/tree.c > @@ -125,6 +125,8 @@ int rcu_num_lvls __read_mostly = RCU_NUM_LVLS; > /* Number of rcu_nodes at specified level. */ > static int num_rcu_lvl[] = NUM_RCU_LVL_INIT; > int rcu_num_nodes __read_mostly = NUM_RCU_NODES; /* Total # rcu_nodes in use. */ > +/* panic() on RCU Stall sysctl. */ > +int sysctl_panic_on_rcu_stall __read_mostly; > > /* > * The rcu_scheduler_active variable transitions from zero to one just > @@ -1311,6 +1313,12 @@ static void rcu_stall_kick_kthreads(struct rcu_state *rsp) > } > } > > +static inline void panic_on_rcu_stall(void) > +{ > + if (sysctl_panic_on_rcu_stall) > + panic("RCU Stall\n"); > +} > + > static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum) > { > int cpu; > @@ -1390,6 +1398,8 @@ static void print_other_cpu_stall(struct rcu_state *rsp, unsigned long gpnum) > > rcu_check_gp_kthread_starvation(rsp); > > + panic_on_rcu_stall(); > + > force_quiescent_state(rsp); /* Kick them all. */ > } > > @@ -1430,6 +1440,8 @@ static void print_cpu_stall(struct rcu_state *rsp) > jiffies + 3 * rcu_jiffies_till_stall_check() + 3); > raw_spin_unlock_irqrestore_rcu_node(rnp, flags); > > + panic_on_rcu_stall(); > + > /* > * Attempt to revive the RCU machinery by forcing a context switch. > * > diff --git a/kernel/sysctl.c b/kernel/sysctl.c > index 87b2fc3..35f0dcb 100644 > --- a/kernel/sysctl.c > +++ b/kernel/sysctl.c > @@ -1205,6 +1205,17 @@ static struct ctl_table kern_table[] = { > .extra2 = &one, > }, > #endif > +#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU) > + { > + .procname = "panic_on_rcu_stall", > + .data = &sysctl_panic_on_rcu_stall, > + .maxlen = sizeof(sysctl_panic_on_rcu_stall), > + .mode = 0644, > + .proc_handler = proc_dointvec_minmax, > + .extra1 = &zero, > + .extra2 = &one, > + }, > +#endif > { } > }; > > -- > 2.5.5 >