Re: [RFC] NUMA futex hashing

[Ravikiran G Thirumalai <kiran@scalex86.org>]

On Tue, Aug 08, 2006 at 12:10:39PM +0200, Eric Dumazet wrote:
> On Tuesday 08 August 2006 11:57, Andi Kleen wrote:
> > Ravikiran G Thirumalai <kiran@scalex86.org> writes:
> > > Current futex hash scheme is not the best for NUMA.   The futex hash
> > > table is an array of struct futex_hash_bucket, which is just a spinlock
> > > and a list_head -- this means multiple spinlocks on the same cacheline
> > > and on NUMA machines, on the same internode cacheline.  If futexes of two
> > > unrelated threads running on two different nodes happen to hash onto
> > > adjacent hash buckets, or buckets on the same internode cacheline, then
> > > we have the internode cacheline bouncing between nodes.
> >
> > When I did some testing with a (arguably far too lock intensive) benchmark
> > on a bigger box I got most bouncing cycles not in the futex locks itself,
> > but in the down_read on the mm semaphore.
> 
> This is true, even with a normal application (not a biased benchmark) and 
> using oprofile. mmap_sem is the killer.

Not if two threads of two different process (so no same mmap_sem) hash onto
futexes on the same cacheline.   But agreed, mmap_sem needs to be fixed too.
If everyone agrees on a per-process hash table for private futexes, then
we will work on that approach.  

Thanks,
Kiran