Re: [patch] Make slab allocator work with SLAB_MUST_HWCACHE_ALIGN

[Dipankar Sarma <dipankar@in.ibm.com>]

On Sat, Sep 13, 2003 at 10:06:08PM +0200, Manfred Spraul wrote:
> Ravikiran G Thirumalai wrote:
> >I wouldn't be using the slab at all because using slabs would mean using
> >NR_CPUs pointers and one extra dereference which is bad as we had found out
> >earlier.  But I guess slab will have to do node local allocations for
> >other applications.
> > 
> >
> Interesting. Slab internally uses lots of large per-cpu arrays. 
> Alltogether something like around 40 kB/cpu. Right now implemented with 
> NR_CPUs pointers. In the long run I'll try to switch to your allocator.
> 
> But back to the patch that started this thread: Do you still need the 
> ability to set an explicit alignment for slab allocations? If yes, then 
> I'd polish my patch, double check all kmem_cache_create callers and then 
> send the patch to akpm. Otherwise I'd wait - the patch is not a bugfix.

This is the problem - the current dynamic per-cpu allocator
(alloc_percpu()) is broken. I uses kmalloc() to allocate each
CPU's data, but kmalloc() doesn't gurantee cache line alignment
(only SLAB_HWCACHE_ALIGN). This may result in some per-CPU statistics
bouncing between CPUs specially on the ones with large L1 cache lines.

We have a number of options -

1. Force kmalloc() to strictly align on cache line boundary, but will
   result in wastage of space elsewhere (with your strict align patch)
   but alloc_percpu() will never result in cache line sharing.
2. Make alloc_percpu() use its own caches for various sizes with your
   strictly align patch. The rest of the kernel is not affected.
3. Let alloc_percpu() use its own allocator which supports NUMA
   and does not use an offset table.

#2 and #3 has less impact in the kernel and we should consider those,
IMO.

Thanks
Dipankar