Re: [patch[ Simple Topology API

[ebiederm@xmission.com (Eric W. Biederman)]

Andi Kleen <ak@suse.de> writes:
> 
> At least on Hammer the latency difference is small enough that 
> caring about the overall bandwidth makes more sense.

I agree.  I will have to look closer but unless there is more
juice than I have seen in Hyper-Transport it is going to become
one of the architectural bottlenecks of the Hammer.

Currently you get 1600MB/s in a single direction.  Not to bad.
But when the memory controllers get out to dual channel DDR-II 400,
the local bandwidth to that memory is 6400MB/s, and the bandwidth to
remote memory 1600MB/s, or 3200MB/s (if reads are as common as
writes).  

So I suspect bandwidth intensive applications will really benefit
from local memory optimization on the Hammer.  I can buy that the
latency is negligible, the fact the links don't appear to scale
in bandwidth as well as the connection to memory may be a bigger
issue.

> > And then you associate that zone-list with the process, and use that
> > zone-list for all process allocations.
> 
> That's the basic idea sure for normal allocations from applications
> that do not care much about NUMA.
> 
> But "numa aware" applications want to do other things like: 
> - put some memory area into every node (e.g. for the numa equivalent of
> per CPU data in the kernel)
> - "stripe" a shared memory segment over all available memory subsystems
> (e.g. to use memory bandwidth fully if you know your interconnect can
> take it; that's e.g. the case on the Hammer)

The latter I really quite believe.  Even dual channel PC2100 can
exceed your interprocessor bandwidth.

And yes I have measured 2000MB/s memory copy with an Athlon MP and
PC2100 memory.

Eric