Re: [PATCH 08/11 v2] nohz: Allow rcu extended quiescent state handling seperately from tick stop

[Frederic Weisbecker <fweisbec@gmail.com>]

On Thu, Oct 13, 2011 at 03:51:36PM -0700, Paul E. McKenney wrote:
> On Thu, Oct 13, 2011 at 02:50:20PM +0200, Frederic Weisbecker wrote:
> > On Thu, Oct 13, 2011 at 12:03:57AM -0700, Paul E. McKenney wrote:
> > > On Wed, Oct 12, 2011 at 11:57:52PM -0700, Paul E. McKenney wrote:
> > > > On Sat, Oct 08, 2011 at 04:01:00PM +0200, Frederic Weisbecker wrote:
> > > > > It is assumed that rcu won't be used once we switch to tickless
> > > > > mode and until we restart the tick. However this is not always
> > > > > true, as in x86-64 where we dereference the idle notifiers after
> > > > > the tick is stopped.
> > > > > 
> > > > > To prepare for fixing this, add two new APIs:
> > > > > tick_nohz_idle_enter_norcu() and tick_nohz_idle_exit_norcu().
> > > > > 
> > > > > If no use of RCU is made in the idle loop between
> > > > > tick_nohz_enter_idle() and tick_nohz_exit_idle() calls, the arch
> > > > > must instead call the new *_norcu() version such that the arch doesn't
> > > > > need to call rcu_idle_enter() and rcu_idle_exit().
> > > > > 
> > > > > Otherwise the arch must call tick_nohz_enter_idle() and
> > > > > tick_nohz_exit_idle() and also call explicitly:
> > > > > 
> > > > > - rcu_idle_enter() after its last use of RCU before the CPU is put
> > > > > to sleep.
> > > > > - rcu_idle_exit() before the first use of RCU after the CPU is woken
> > > > > up.
> > > > 
> > > > Thank you, Frederic!  I have queued this to replace the earlier
> > > > version.  The set is available on branch rcu/dyntick of
> > > > 
> > > > 	https://github.com/paulmckrcu/linux
> > > 
> > > Which reminds me...  About the ultimate objective, getting tick-free
> > > operation.  (Or, for the guys who want to eliminate the tick entirely,
> > > shutting up the hrtimer stuff that they want to replace it with.)
> > > 
> > > I believe that you will then need to have two levels of not-in-dynticks
> > > for processes, one for idle vs. not and another for when a process
> > > switches from user-space to kernel execution.  Correct, or am I
> > > confused?
> > > 
> > > The reason I ask is that commit e11f5981 currently only allows one
> > > level of not-in-dynticks for processes.  It is easy to add another
> > > level, but thought I should check beforehand.
> > 
> > Hmm, yeah looking at that patch, it's going to be hard to have a nesting
> > that looks like:
> > 
> > rcu_irq_enter();
> > rcu_user_enter();
> > rcu_irq_exit(); <-- with effective extended quiescent state starting there
> 
> OK, so the idea here is that there has been two runnable processes on
> the current CPU, but during the irq handler one of them moves or some
> such?

No it happens when we have an irq in userspace and we stop the tick
from that irq. Noticing we are in userspace, we want to be in extended
quiescent state when we resume from the interrupt to userspace.

> If so, how about a rcu_user_enter_fromirq() that sets the counter
> to 1 so that the rcu_irq_exit() cleans up properly?  If need be, I could
> of course provide an argument to allow you to specify the count offset.

Yeah I think that should work.
 
> > I also need to be able to call rcu_user_enter() from non-irq path.
> 
> Then rcu_user_enter_fromirq() would be for the irq path and
> rcu_user_enter() from the non-irq path.
> 
> Would that work for you?

Yep!

> 
> > I don't truly understand the problem of the usermode helpers that
> > mess up the dynticks counts. May be we can somewhow fix it differently
> > from the offending callsite?
> 
> I tried a few approaches along these lines, but there were way too
> many opportunities for interruption and preemption along the way.
> The problem is that unless the fixup happens under a no-preempt
> region of code that includes the rcu_irq_enter() or rcu_irq_exit()
> call (as the case may be), then you end up messing up the idle-depth
> count of two CPUs rather than just one.  :-(
> 
> But maybe I am missing something -- suggestions more than welcome!

It's rather me missing everything :)
It happens when we call call_usermodehelper()? If so how? We have a
call to rcu_irq_enter() that lacks an rcu_irq_exit() ?