Re: [PATCH RFC tip/core/rcu] Add callback-free CPUs

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Thu, Sep 06, 2012 at 06:58:56PM +0200, Peter Zijlstra wrote:
> On Thu, 2012-09-06 at 09:47 -0700, Paul E. McKenney wrote:
> > The key point is "would simple put RCU into extended quiescent state".
> > This can only happen if the CPU has no callbacks.  If the CPU does have
> > callbacks, then RCU will need to do some work to advance the callbacks.
> > Advancing the callbacks requires that RCU periodically do work on that
> > CPU, resulting in OS jitter.
> 
> But since its then not actually in adaptive-tick mode (the tick is still
> running) who cares? It will only disable the tick once all preconditions
> are met, this includes RCU being in extended qs, so until that time...

The fact that it is then not actually in adaptive-tick mode is exactly
the problem.  In other words, if the grace-period processing is offloaded
along with the callbacks, then no-CBs CPUs can get into adaptive-tick
mode more quickly than CPUs processing their own CBs.  Getting these
CPUs into adaptive-tick mode more quickly reduces OS jitter, which is
one big expected benefit of adaptive-tick mode.

> > > That way you could run the entire state thing from a kthread with random
> > > affinity, all 'per-cpu' data would still be fine since only the one
> > > kthread will access it, even though locality might suffer somewhat.
> > 
> > Well, the current patch set does move much of the grace-period machinery
> > to a kthread.  Much of the remaining work needs to remain on the CPUs
> > (at least those not in an extended quiescent state) in order to keep
> > the overhead of the read-side primitives and scheduler hooks inexpensive. 
> 
> Ah indeed, what you're saying is that the data required by those hooks
> needs to be accessed locally in order to avoid proper atomic ops.

Yep, that is it!

> So then you do indeed need to break the state machine into two parts,
> and I guess that's the bit you're struggling with.

Exactly!  I should be able to work something out without too much trouble,
but it was not going to happen in time for Plumbers, hence the crude
prototype.

> Still I would not make this more complex than it needs to be, if the
> tick is running we can use this to drive the state machine, if its not,
> we are in extended qs and we don't need to drive the tick.

True, but an important goal of no-CBs CPUs is to spend more time in
tickless mode, thus reducing OS jitter.

							Thanx, Paul