[PATCH 0/4] mm/mempolicy: introduce package-aware weighted interleave

[Rakie Kim <rakie.kim@sk.com>]

Package-aware weighted interleave places a task's weighted-interleave
pages on the NUMA nodes of its local package, so that interleave traffic
does not have to cross the interconnect to another package. This keeps
each node's weight aligned with the bandwidth the task actually gets
from it, so effective bandwidth holds up on a system that has more than
one package. (A package is a CPU socket together with the memory
attached to it.)

Changes from RFC:
https://lore.kernel.org/all/20260316051258.246-1-rakie.kim@sk.com/
- Added an opt-in sysfs toggle (off by default) and a read-only sysfs
  view of the package topology
- Added topology validation with a clean fallback to plain weighted
  interleave on unsupported topologies
- Hardened the allocation, device-teardown, and node-hotplug paths

Weighted interleave places pages on nodes in proportion to per-node
weights that are set from each node's bandwidth. Within one package the
weight given to a node matches the bandwidth a task sees from it. Across
packages it no longer does: a memory node's physical bandwidth is fixed,
but the bandwidth a task effectively sees depends on which package its
CPU is in, because memory reached from another package, over the
interconnect between them, is slower than the same memory reached within
the package. The weights are set once from device bandwidth and applied
the same way wherever the task runs, so a node in another package is
given a weight higher than the bandwidth it can deliver to that task.
The kernel has no package abstraction and does not record which package
a node belongs to, so it cannot tell which node pairs are separated by
the interconnect.

          node0             node1
        +-------+         +-------+
        | CPU 0 |---------| CPU 1 |
        +-------+         +-------+
        | DRAM0 |         | DRAM1 |
        +---+---+         +---+---+
            |                 |
        +---+---+         +---+---+
        | CXL 0 |         | CXL 1 |
        +-------+         +-------+
          node2             node3

The numbers below are illustrative single-stream bandwidths (GB/s).
Local DRAM sustains 300 and local CXL 150; any path that crosses to
another package, over the interconnect, is capped at 100, so a node in
another package delivers 100 whether it is DRAM or CXL. Note that local
CXL (150) is still faster than any node in another package (100). The
effective bandwidth each CPU sees is therefore:

              node0  node1  node2  node3
from CPU 0:    300    100    150    100
from CPU 1:    100    300    100    150

Since a single per-node weight cannot encode the interconnect penalty,
a reasonable set of global weights is taken from local device bandwidth
(local DRAM : local CXL = 300 : 150 = 2 : 1): node0=2 node1=2 node2=1
node3=1.

Applied the same way to every source, these weights give the map:

              node0  node1  node2  node3
global:         2      2      1      1

A task on CPU 0 gives node1 - remote DRAM, effective 100 - the same
weight 2 as its own local node0 at 300. Worse, node1 is weighted above
node2, the task's local CXL at effective 150, even though node2 is the
faster of the two. The flat weights rank a slower interconnect-bound
node above a faster local one, which is exactly backwards.

This series makes weighted interleave package-aware. When it is on,
weighted interleave prefers the task's current package: while the
package's nodes have room, the task's pages are spread across them by
weight, so allocations stay off the interconnect. The rest of the
policy nodemask is used when the local package cannot serve the request
- when a node in it is under pressure and the page allocator falls back
along the zonelist, or when the policy nodemask happens to exclude every
node of the current package, in which case the package spanned by the
policy's own nodes is used instead. The nodes considered are always
within the policy nodemask, which mempolicy already narrows to the
task's cpuset, so cpusets and the task nodemask stay in control.

              node0  node1  node2  node3
from CPU 0:     2      0      1      0
from CPU 1:     0      2      0      1

A task on CPU 0 now places pages on node0 (weight 2) and node2
(weight 1) at 2:1, which matches their effective bandwidth of 300:150;
a task on CPU 1 places on node1 and node3 the same way. Placement
follows the bandwidth each task actually sees, NUMA locality is
preserved, and interleave traffic stays off the interconnect.

To make this possible the kernel needs a notion of which nodes share a
package. The NUMA distance model offers only relative latencies and no
structural grouping, which is especially limiting for CXL memory nodes
that come online without an explicit package association.

The series adds a package-aware topology layer that groups CPU and
memory-only nodes into a "memory package", built from the physical
package ids firmware reports and, for a memory-only node, an initiator
CPU node or SLIT distances. A package can contain more than one CPU node
or more than one memory-only node, so the layer maps a package to a set
of nodes rather than to a single node or a single CXL device.

The feature is off by default and opt-in through a sysfs toggle. The
package topology itself is exposed read-only under
/sys/devices/system/package/; there is deliberately no writable
override, since a machine whose firmware describes its topology
incorrectly should be fixed in firmware. On a topology that does not
have the symmetric shape the placement relies on, enabling is refused
and any active mode degrades cleanly to the original flat behavior.

Measured results:

System Configuration:
- Processor: Dual-Socket Intel Xeon 6980P (Granite Rapids)

1) Throughput (System Bandwidth)
   - DRAM Only: 966 GB/s
   - Weighted Interleave: 903 GB/s (7% decrease compared to DRAM Only)
   - Package-Aware Weighted Interleave: 1329 GB/s (1.33 TB/s)
     (38% increase compared to DRAM Only,
      47% increase compared to Weighted Interleave)

2) Loaded Latency (Under High Bandwidth)
   - DRAM Only: 544 ns
   - Weighted Interleave: 545 ns
   - Package-Aware Weighted Interleave: 436 ns
     (20% reduction compared to both)

A small CXL driver change registers a CXL memory node into its package
as the node comes online, using the initiator the driver resolves for
the region; this is where the package layer gets the CPU-side
association that plain NUMA distance does not carry.

The memory_package layer offers a broader interface for grouping and
querying package topology - usable by memory tiering as well - and
package-aware weighted interleave uses the subset it needs.

[PATCH 1/4] mm/numa: introduce nearest_nodes_nodemask()
  Add a NUMA helper that returns every node sharing the minimum distance
  from a source node.

[PATCH 2/4] mm/memory-tiers: package-aware topology management
  Group NUMA nodes into memory packages from firmware topology data,
  expose the grouping read-only under /sys/devices/system/package/, and
  validate the symmetric shape that package-aware placement relies on.

[PATCH 3/4] mm/memory-tiers: register CXL nodes to packages
  Bind a CXL memory node to a package using an initiator CPU node.

[PATCH 4/4] mm/mempolicy: package-aware weighted interleave
  Prefer the current package for weighted interleave node selection,
  behind an opt-in package_mode sysfs toggle that is off by default.

Rakie Kim (4):
  mm/numa: introduce nearest_nodes_nodemask()
  mm/memory-tiers: introduce package-aware topology management for NUMA
    nodes
  mm/memory-tiers: register CXL nodes to memory packages via initiator
  mm/mempolicy: enhance weighted interleave with package-aware locality

 .../ABI/testing/sysfs-devices-system-package  |   35 +
 ...fs-kernel-mm-mempolicy-weighted-interleave |   17 +
 drivers/cxl/core/region.c                     |   54 +
 drivers/cxl/cxl.h                             |    1 +
 drivers/dax/kmem.c                            |    3 +
 include/linux/memory-tiers.h                  |  113 ++
 include/linux/numa.h                          |   11 +
 mm/memory-tiers.c                             | 1009 +++++++++++++++++
 mm/mempolicy.c                                |  200 +++-
 9 files changed, 1439 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-system-package


base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
-- 
2.25.1