Re: [RFC 00/15] x86_64: Optimize percpu accesses

[Mike Travis <travis@sgi.com>]

Jeremy Fitzhardinge wrote:
> Mike Travis wrote:
>> One thing to remember is the eventual goal is implementing the cpu_alloc
>> functions which I think we've agreed has to be "growable".  This means
>> that
>> the addresses will need to be virtual to allow the same offsets for
>> all cpus.
>> The patchset I have uses 2Mb pages.  This "little" twist might figure
>> into the
>> implementation issues that are being discussed.
> 
> You want to virtually map the percpu area?  How and when would it get
> extended?
> 
>    J


CPU_ALLOC(), or some such means.  This is to replace the percpu allocator
in modules.c:

    Subject: cpu alloc: The allocator

    The per cpu allocator allows dynamic allocation of memory on all
    processors simultaneously. A bitmap is used to track used areas.
    The allocator implements tight packing to reduce the cache footprint
    and increase speed since cacheline contention is typically not a concern
    for memory mainly used by a single cpu. Small objects will fill up gaps
    left by larger allocations that required alignments.

    The size of the cpu_alloc area can be changed via make menuconfig.

    Signed-off-by: Christoph Lameter <clameter@sgi.com>

and:

    Subject: cpu alloc: Use cpu allocator instead of the builtin modules per cpu allocator

    Remove the builtin per cpu allocator from modules.c and use cpu_alloc instead.

    The patch also removes PERCPU_ENOUGH_ROOM. The size of the cpu_alloc area is
    determined by CONFIG_CPU_AREA_SIZE. PERCPU_ENOUGH_ROOMs default was 8k.
    CONFIG_CPU_AREA_SIZE defaults to 32k. Thus we have more space to load modules.

    Signed-off-by: Christoph Lameter <clameter@sgi.com>

The discussion that followed was very emphatic that the size of the space should
not be fixed, but instead be dynamically growable.  Since the offset needs to be
fixed for each cpu, then virtual (I think) is the only way to go.  The use of a
2MB page just conserves map entries.  (Of course, if we just reserved 2MB in the
first place it might not need to be virtual...?  But the concern was for systems
with hundreds of (say) network interfaces using even more than 2MB.)

Thanks,
Mike