Re: [RFC PATCH 5/6] implement per-cpu&per-domain state machine call_srcu()

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Mon, Mar 12, 2012 at 06:58:17PM +0100, Peter Zijlstra wrote:
> On Mon, 2012-03-12 at 10:54 -0700, Paul E. McKenney wrote:
> > On Sat, Mar 10, 2012 at 11:09:53AM +0100, Peter Zijlstra wrote:
> > > On Thu, 2012-03-08 at 11:58 -0800, Paul E. McKenney wrote:
> > > > 
> > > > But I guess I should ask...  Peter, what do you expect the maximum
> > > > call_srcu() rate to be in your use cases?  If tens of thousands are
> > > > possible, some adjustments will be needed. 
> > > 
> > > The one call-site I currently have is linked to vma lifetimes, so yeah,
> > > I guess that that can be lots.
> > 
> > So the worst case would be if several processes with lots of VMAs were
> > to exit at about the same time?  If so, my guess is that call_srcu()
> > needs to handle several thousand callbacks showing up within a few
> > tens of microseconds.  Is that a reasonable assumption, or am I missing
> > an order of magnitude or two in either direction?
> 
> That or a process is doing mmap/munmap loops (some file scanners are
> known to do this). But yeah, that can be lots.
> 
> My current use case doesn't quite trigger since it needs another syscall
> to attach something to a vma before vma tear-down actually ends up
> calling call_srcu() so in practice it won't be much at all (for now).
> 
> Still I think its prudent to make it (srcu callbacks) deal with plenty
> callbacks even if initially there won't be many -- who knows what other
> people will do while you're not paying attention... :-)

And another question I should have asked to begin with...  Would each
VMA have its own SRCU domain, or are you thinking in terms of one
SRCU domain for all VMAs globally?

If the latter, that pushes pretty strongly for per-CPU SRCU callback
lists.  Which brings up srcu_barrier() scalability (and yes, I am working
on rcu_barrier() scalability).  One way to handle this at least initially
is to have srcu_barrier() avoid enqueueing callbacks on CPUs whose
callback lists are empty.  In addition, if the loop over all CPUs is
preemptible, then there should not be much in the way of realtime issues.

The memory-ordering issues should be OK -- if a given CPU's list was
empty at any time during srcu_barrier()'s execution, then there is no
need for a callback on that CPU.

							Thanx, Paul