Re: [PATCH 2/2] smp_call_function: use rwlocks on queues rather than rcu

["Paul E. McKenney" <paulmck@linux.vnet.ibm.com>]

On Fri, Aug 22, 2008 at 09:01:22AM -0500, Christoph Lameter wrote:
> Pekka Enberg wrote:
> > Hi Ingo,
> > 
> > On Fri, Aug 22, 2008 at 9:28 AM, Ingo Molnar <mingo@elte.hu> wrote:
> >> * Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> >>
> >>> RCU can only control the lifetime of allocated memory blocks, which
> >>> forces all the call structures to be allocated.  This is expensive
> >>> compared to allocating them on the stack, which is the common case for
> >>> synchronous calls.
> >>>
> >>> This patch takes a different approach.  Rather than using RCU, the
> >>> queues are managed under rwlocks.  Adding or removing from the queue
> >>> requires holding the lock for writing, but multiple CPUs can walk the
> >>> queues to process function calls under read locks.  In the common
> >>> case, where the structures are stack allocated, the calling CPU need
> >>> only wait for its call to be done, take the lock for writing and
> >>> remove the call structure.
> >>>
> >>> Lock contention - particularly write vs read - is reduced by using
> >>> multiple queues.
> >> hm, is there any authorative data on what is cheaper on a big box, a
> >> full-blown MESI cache miss that occurs for every reader in this new
> >> fastpath, or a local SLAB/SLUB allocation+free that occurs with the
> >> current RCU approach?
> > 
> > Christoph might have an idea about it.
> 
> Its on the stack which is presumably hot so no cache miss? If its async then
> presumably we do not need to wait so its okay to call an allocator.
> 
> Generally: The larger the box (longer cacheline acquisition latencies) and the
> higher the contention (cannot get cacheline because of contention) the better
> a slab allocation will be compared to a cacheline miss.
> 
> RCU is problematic because it lets cachelines get cold. A hot cacheline that
> is used frequently read and written to by the same cpu is very good thing for
> performace.

So on your these large boxes, read-only cachelines are preferentially
ejected from the cache, so that one should write to per-CPU data
occasionally to keep it resident?  Or is the issue the long RCU grace
periods which allow the structure being freed to age out of all relevant
caches?  (My guess would be the second.)

							Thanx, Paul