Re: Mysterious CFQ crash and RCU

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Fri, Jun 03, 2011 at 09:45:14AM -0400, Vivek Goyal wrote:
> On Thu, Jun 02, 2011 at 10:07:24PM -0700, Paul E. McKenney wrote:
> 
> [..]
> > > Thu May 26 10:47:20 CEST 2011
> > > /sys/kernel/debug/rcu/rcugp:
> > > rcu_sched: completed=682249  gpnum=682250
> > 
> > 15 more seconds, a few thousand more grace periods.  About 500 grace
> > periods per second, which is quite reasonable on a single-CPU system.
> 
> PaulB mentioned that crash happened at May 26 10:47:07. I am wondering
> how are we able to sample the data after the crash. I am assuming
> that above data gives information only before crash and does not
> tell us anything about what happened just before crash. What am I missing.
> 
> PaulM, in one of the mails you had mentioned that one could print
> context switch id to make sure we did not block in rcu section. Would
> you have quick pointer where is context switch id stored. May be
> I can write a small patch for PaulB.

>From what I can see, the task_struct nvcsw and nivcsw fields should do
it, though I am not seeing where these are incremented.

So if these don't do what you need, the following (untested but trivial)
patch will provide an rcu_switch_count in the task structure.

							Thanx, Paul

------------------------------------------------------------------------

rcu: add diagnostic per-task context-switch count

Note that this is also incremented by softirqs.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 2a8621c..5ef22e2 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1261,6 +1261,7 @@ struct task_struct {
 #ifdef CONFIG_RCU_BOOST
 	struct rt_mutex *rcu_boost_mutex;
 #endif /* #ifdef CONFIG_RCU_BOOST */
+	unsigned long rcu_switch_count;
 
 #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
 	struct sched_info sched_info;
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 89419ff..080c6eb 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -154,6 +154,7 @@ void rcu_bh_qs(int cpu)
  */
 void rcu_note_context_switch(int cpu)
 {
+	current->rcu_switch_count++;
 	rcu_sched_qs(cpu);
 	rcu_preempt_note_context_switch(cpu);
 }