Re: [PATCH 4/3] Replace dynamic percpu implementation

[Dipankar Sarma <dipankar@in.ibm.com>]

On Tue, May 20, 2003 at 02:32:37PM +1000, Rusty Russell wrote:
> This requires the first three patches.  IA64 untested.  See comment.
> 
> Name: Dynamic per-cpu allocation using static per-cpu mechanism
> Author: Rusty Russell
> Status: Tested on 2.5.69-bk13
> Depends: Misc/kmalloc_percpu-interface.patch.gz
> 
> D: This patch replaces the dynamic per-cpu allocator, alloc_percpu,
> D: to make it use the same mechanism as the static per-cpu variables, ie.
> D: ptr + __per_cpu_offset[smp_processor_id()] gives the variable address.
> D: This allows it to be used in modules (following patch), and hopefully
> D: increases space and time efficiency of reference at the same time.
> D: It gets moved to its own (SMP-only) file: mm/percpu.c.

We will do some measurements with this but based on a large number
of measurements that Kiran had done earlier, we can see a couple of things -

1. Even though a percpu scheme using pointer arithmatic has one less memory
   reference, the globally shared offset table is often in the cache
   and therefore pointer arithmatic offers no added advantage.

2. Increased sharing of cacheline helps by reducing associativity misses.
   We see this by comparing an interlaced allocator where only same
   sized objects share blocks vs. the current static allocator. Sharing of
   blocks by differently sized objects also allow cache lines to be
   kept warm as more subsystems in the kernel access them.

Considering these results, this allocator seems to be a step in the right
direction.

Thanks
Dipankar