Re: Mysterious CFQ crash and RCU

Paul E. McKenney
2011-06-03T16:54:24Z

On Fri, Jun 03, 2011 at 08:33:44AM -0700, Paul E. McKenney wrote:
> On Fri, Jun 03, 2011 at 09:45:14AM -0400, Vivek Goyal wrote:
> > On Thu, Jun 02, 2011 at 10:07:24PM -0700, Paul E. McKenney wrote:
> >
> > [..]
> > > > Thu May 26 10:47:20 CEST 2011
> > > > /sys/kernel/debug/rcu/rcugp:
> > > > rcu_sched: completed=682249 gpnum=682250
> > >
> > > 15 more seconds, a few thousand more grace periods. About 500 grace
> > > periods per second, which is quite reasonable on a single-CPU system.
> >
> > PaulB mentioned that crash happened at May 26 10:47:07. I am wondering
> > how are we able to sample the data after the crash. I am assuming
> > that above data gives information only before crash and does not
> > tell us anything about what happened just before crash. What am I missing.
> >
> > PaulM, in one of the mails you had mentioned that one could print
> > context switch id to make sure we did not block in rcu section. Would
> > you have quick pointer where is context switch id stored. May be
> > I can write a small patch for PaulB.
>
> From what I can see, the task_struct nvcsw and nivcsw fields should do
> it, though I am not seeing where these are incremented.

And Milton Miller pointed out that schedule() takes the address of
these fields in a local pointer "switch_count" and then increments
through this pointer. So there you have it.

> So if these don't do what you need, the following (untested but trivial)
> patch will provide an rcu_switch_count in the task structure.

And he also noted that (nvcsw+nivcsw) is incremented only when context
switches from one task to another, while my patch below would count
every call to schedule(), whether or not a context switch occurs.

Either approach should work.

Thanx, Paul

> ------------------------------------------------------------------------
>
> rcu: add diagnostic per-task context-switch count
>
> Note that this is also incremented by softirqs.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index 2a8621c..5ef22e2 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -1261,6 +1261,7 @@ struct task_struct {
> #ifdef CONFIG_RCU_BOOST
> struct rt_mutex *rcu_boost_mutex;
> #endif /* #ifdef CONFIG_RCU_BOOST */
> + unsigned long rcu_switch_count;
>
> #if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
> struct sched_info sched_info;
> diff --git a/kernel/rcutree.c b/kernel/rcutree.c
> index 89419ff..080c6eb 100644
> --- a/kernel/rcutree.c
> +++ b/kernel/rcutree.c
> @@ -154,6 +154,7 @@ void rcu_bh_qs(int cpu)
> */
> void rcu_note_context_switch(int cpu)
> {
> + current->rcu_switch_count++;
> rcu_sched_qs(cpu);
> rcu_preempt_note_context_switch(cpu);
> }