Re: [RFC PATCH v2 0/3] mm: mempolicy: Multi-tier weighted interleaving

[Gregory Price <gregory.price@memverge.com>]

On Mon, Oct 30, 2023 at 10:20:14AM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
> 
> The extending adds complexity to the kernel code and changes the kernel
> ABI.  So, IMHO, we need some real life use case to prove the added
> complexity is necessary.
> 
> For example, in [1], Johannes showed the use case to support to add
> per-memory-tier interleave weight.
> 
> [1] https://lore.kernel.org/all/20220607171949.85796-1-hannes@cmpxchg.org/
> 
> --
> Best Regards,
> Huang, Ying

Sorry, I misunderstood your question.

The use case is the same as the N:M interleave strategy between tiers,
and in fact the proposal for weights was directly inspired by the patch
you posted. We're searching for the best way to implement weights.

We've discussed placing these weights in:

1) mempolicy :
   https://lore.kernel.org/linux-cxl/20230914235457.482710-1-gregory.price@memverge.com/

2) tiers
   https://lore.kernel.org/linux-cxl/20231009204259.875232-1-gregory.price@memverge.com/

and now
3) the nodes themselves
   RFC not posted yet

The use case is the exact same as the patch you posted, which is to enable
optimal distribution of memory to maximize memory bandwidth usage.

The use case is straight forward - Consider a machine with the following
numa nodes:

1) Socket 0 - DRAM - ~400GB/s bandwidth local, less cross-socket
2) Socket 1 - DRAM - ~400GB/s bandwidth local, less cross socket
3) CXL Memory Attached to Socket 0 with ~64GB/s per link.
4) CXL Memory Attached to Socket 1 with ~64GB/s per link.

The goal is to enable mempolicy to implement weighted interleave such
that a thread running on socket 0 can effectively spread its memory
across each numa node (or some subset there-of) such that it maximizes
its bandwidth usage across the various devices.

For example, lets consider a system with only 1 & 2 (2 sockets w/ DRAM).

On an Intel System with UPI, the "effective" bandwidth available for a
task on Socket 0 is not 800GB/s, it's about 450-500GB/s split about
300/200 between the sockets (you never get the full amount, and UPI limits
cross-socket bandwidth).

Today `numactl --interleave` will split your memory 50:50 between
sockets, which is just blatantly suboptimal.  In this case you would
prefer a 3:2 distribution (literally weights of 3 and 2 respectively).

The extension to CXL becomes obvious then, as each individual node,
respective to its CPU placement, has a different optimal weight.


Of course the question becomes "what if a task uses more threads than a
single socket has to offer", and the answer there is essentially the
same as the answer today:  Then that process must become "numa-aware" to
make the best use of the available resources.

However, for software capable of exhausting bandwidth with from a single
socket (which on intel takes about 16-20 threads with certain access
patterns), then a weighted-interleave system provided via some interface
like `numactl --weighted-interleave` with weights either set in numa
nodes or mempolicy is sufficient.


~Gregory