Re: [RFC PATCH 5/6] implement per-cpu&per-domain state machine call_srcu()

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Mon, Mar 12, 2012 at 09:25:16PM +0100, Peter Zijlstra wrote:
> On Mon, 2012-03-12 at 11:32 -0700, Paul E. McKenney wrote:
> > And another question I should have asked to begin with...  Would each
> > VMA have its own SRCU domain, or are you thinking in terms of one
> > SRCU domain for all VMAs globally?
> 
> The latter, single domain for all objects.

OK.

> > If the latter, that pushes pretty strongly for per-CPU SRCU callback
> > lists. 
> 
> Agreed. I was under the impression the proposed thing had this, but on
> looking at it again it does not. Shouldn't be hard to add though.

Agreed, but see srcu_barrier()...

> >  Which brings up srcu_barrier() scalability (and yes, I am working
> > on rcu_barrier() scalability).  One way to handle this at least initially
> > is to have srcu_barrier() avoid enqueueing callbacks on CPUs whose
> > callback lists are empty.  In addition, if the loop over all CPUs is
> > preemptible, then there should not be much in the way of realtime issues.
> 
> Why do we have rcu_barrier() and how is it different from
> synchronize_rcu()?

We need rcu_barrier() in order to be able to safely unload modules that
use call_rcu().  If a module fails to invoke rcu_barrier() between its
last call_rcu() and its unloading, then its RCU callbacks can be fatally
disappointed to learn that their callback functions are no longer in
memory.  See http://lwn.net/Articles/202847/ for more info.

While synchronize_rcu() waits only for a grace period, rcu_barrier()
waits for all pre-existing RCU callbacks to be invoked.  There is also
an rcu_barrier_bh() and rcu_barrier_sched().

Of course, if all uses of call_srcu() were to be from the main kernel
(as opposed to from a module), then there would be no need for a
srcu_barrier().  But it seems quite likely that if a call_srcu() is
available, its use from a module won't be far behind -- especially
given that rcutorture is normally built as a kernel module.

							Thanx, Paul