Re: Mysterious CFQ crash and RCU

Paul E. McKenney
2011-05-21T23:54:14Z

On Sun, May 22, 2011 at 12:23:50AM +0200, Paul Bolle wrote:
> Paul,
>
> On Sat, 2011-05-21 at 14:00 -0700, Paul E. McKenney wrote:
> > On Thu, May 19, 2011 at 06:24:04PM -0400, Vivek Goyal wrote:
> > It does look like a tough one!
>
> Thank you!
>
> > > Is it possible? We have looked at the code many a times and we think
> > > that rcu locking around it is fine. Is it possible that a call_rcu()
> > > can fire before rcu grace period is over.
> >
> > If it does, that would be a bug in RCU.
> >
> > > I had put a debug patch in CFQ (details are in bugzilla) and I can
> > > see that after decoupling the object from the hash list, it got
> > > freed while we were still under rcu_read_lock().
> > >
> > > Is there any known issue or is there any quick tip on how can I
> > > go about debugging it further from rcu point of view.
> >
> > First for uses of RCU:
> >
> > o One thing to try would be CONFIG_PROVE_RCU, which could help
> > find missing rcu_read_lock()s and similar. Some years back, it
> > used to be the case that spin_lock() implied rcu_read_lock(),
> > but it no longer does. There might still be some cases where
> > spin_lock() needs to have an rcu_read_lock() added.
> >
> > o There are a few entries in the bugzilla mentioning that elements
> > are being removed more often than expected. There is a config
> > option CONFIG_DEBUG_OBJECTS_RCU_HEAD that complains if the same
> > object is passed to call_rcu() before the grace period ends for
> > the first round.
> >
> > o Try switching between CONFIG_TREE_RCU and CONFIG_TREE_PREEMPT_RCU.
> > These two settings are each sensitive to different forms of abuse.
> > For example, if you have CONFIG_PREEMPT=n and CONFIG_TREE_RCU=y,
> > illegally placing a synchronize_rcu() -- or anything else that
> > blocks -- in an RCU read-side critical section will silently
> > partition that RCU read-side critical section. In contrast,
> > CONFIG_TREE_PREEMPT_RCU=y will complain about this.
> >
> > Second, for RCU itself, CONFIG_RCU_TRACE enables counter-based tracing
> > in RCU. Sampling each of the files in the debugfs directory "rcu"
> > before and after the badness (if possible) could help me see if anything
> > untoward is happening.
>
> Before we go down that route, I'd like to note that I seem to be unable
> to reproduce this Oops under v2.6.39 (either using the first v2.6.39 rpm
> for i686 shipped for Fedora Rawhide, or two versions of that rpm I built
> locally).
>
> Is anyone able to spot one or more commits in v2.6.39-rc7..v2.6.39 that
> might have fixed this Oops? Or did my chance of hitting this Oops,
> somehow, just got a lot smaller in v.2.6.39?

5f45c69589b7d ("read_lock() does not always imply rcu_read_lock()") might
well be a fix.

> Please note that I have tried to reproduce this Oops very often, using
> quite a number of kernels, so there's a non-zero chance I tricked myself
> in seeing a pattern where there actually is none.

Understood -- races can be a bit frustrating. How long should you run
before you conclude that you fixed it? ;-)

Thanx, Paul