[PATCH 0/4] mm/mempolicy: introduce package-aware weighted interleave

[PATCH 0/4] mm/mempolicy: introduce package-aware weighted interleave

[Rakie Kim]

Package-aware weighted interleave places a task's weighted-interleave
pages on the NUMA nodes of its local package, so that interleave traffic
does not have to cross the interconnect to another package. This keeps
each node's weight aligned with the bandwidth the task actually gets
from it, so effective bandwidth holds up on a system that has more than
one package. (A package is a CPU socket together with the memory
attached to it.)

Changes from RFC:
https://lore.kernel.org/all/20260316051258.246-1-rakie.kim@sk.com/
- Added an opt-in sysfs toggle (off by default) and a read-only sysfs
  view of the package topology
- Added topology validation with a clean fallback to plain weighted
  interleave on unsupported topologies
- Hardened the allocation, device-teardown, and node-hotplug paths

Weighted interleave places pages on nodes in proportion to per-node
weights that are set from each node's bandwidth. Within one package the
weight given to a node matches the bandwidth a task sees from it. Across
packages it no longer does: a memory node's physical bandwidth is fixed,
but the bandwidth a task effectively sees depends on which package its
CPU is in, because memory reached from another package, over the
interconnect between them, is slower than the same memory reached within
the package. The weights are set once from device bandwidth and applied
the same way wherever the task runs, so a node in another package is
given a weight higher than the bandwidth it can deliver to that task.
The kernel has no package abstraction and does not record which package
a node belongs to, so it cannot tell which node pairs are separated by
the interconnect.

          node0             node1
        +-------+         +-------+
        | CPU 0 |---------| CPU 1 |
        +-------+         +-------+
        | DRAM0 |         | DRAM1 |
        +---+---+         +---+---+
            |                 |
        +---+---+         +---+---+
        | CXL 0 |         | CXL 1 |
        +-------+         +-------+
          node2             node3

The numbers below are illustrative single-stream bandwidths (GB/s).
Local DRAM sustains 300 and local CXL 150; any path that crosses to
another package, over the interconnect, is capped at 100, so a node in
another package delivers 100 whether it is DRAM or CXL. Note that local
CXL (150) is still faster than any node in another package (100). The
effective bandwidth each CPU sees is therefore:

              node0  node1  node2  node3
from CPU 0:    300    100    150    100
from CPU 1:    100    300    100    150

Since a single per-node weight cannot encode the interconnect penalty,
a reasonable set of global weights is taken from local device bandwidth
(local DRAM : local CXL = 300 : 150 = 2 : 1): node0=2 node1=2 node2=1
node3=1.

Applied the same way to every source, these weights give the map:

              node0  node1  node2  node3
global:         2      2      1      1

A task on CPU 0 gives node1 - remote DRAM, effective 100 - the same
weight 2 as its own local node0 at 300. Worse, node1 is weighted above
node2, the task's local CXL at effective 150, even though node2 is the
faster of the two. The flat weights rank a slower interconnect-bound
node above a faster local one, which is exactly backwards.

This series makes weighted interleave package-aware. When it is on,
weighted interleave prefers the task's current package: while the
package's nodes have room, the task's pages are spread across them by
weight, so allocations stay off the interconnect. The rest of the
policy nodemask is used when the local package cannot serve the request
- when a node in it is under pressure and the page allocator falls back
along the zonelist, or when the policy nodemask happens to exclude every
node of the current package, in which case the package spanned by the
policy's own nodes is used instead. The nodes considered are always
within the policy nodemask, which mempolicy already narrows to the
task's cpuset, so cpusets and the task nodemask stay in control.

              node0  node1  node2  node3
from CPU 0:     2      0      1      0
from CPU 1:     0      2      0      1

A task on CPU 0 now places pages on node0 (weight 2) and node2
(weight 1) at 2:1, which matches their effective bandwidth of 300:150;
a task on CPU 1 places on node1 and node3 the same way. Placement
follows the bandwidth each task actually sees, NUMA locality is
preserved, and interleave traffic stays off the interconnect.

To make this possible the kernel needs a notion of which nodes share a
package. The NUMA distance model offers only relative latencies and no
structural grouping, which is especially limiting for CXL memory nodes
that come online without an explicit package association.

The series adds a package-aware topology layer that groups CPU and
memory-only nodes into a "memory package", built from the physical
package ids firmware reports and, for a memory-only node, an initiator
CPU node or SLIT distances. A package can contain more than one CPU node
or more than one memory-only node, so the layer maps a package to a set
of nodes rather than to a single node or a single CXL device.

The feature is off by default and opt-in through a sysfs toggle. The
package topology itself is exposed read-only under
/sys/devices/system/package/; there is deliberately no writable
override, since a machine whose firmware describes its topology
incorrectly should be fixed in firmware. On a topology that does not
have the symmetric shape the placement relies on, enabling is refused
and any active mode degrades cleanly to the original flat behavior.

Measured results:

System Configuration:
- Processor: Dual-Socket Intel Xeon 6980P (Granite Rapids)

1) Throughput (System Bandwidth)
   - DRAM Only: 966 GB/s
   - Weighted Interleave: 903 GB/s (7% decrease compared to DRAM Only)
   - Package-Aware Weighted Interleave: 1329 GB/s (1.33 TB/s)
     (38% increase compared to DRAM Only,
      47% increase compared to Weighted Interleave)

2) Loaded Latency (Under High Bandwidth)
   - DRAM Only: 544 ns
   - Weighted Interleave: 545 ns
   - Package-Aware Weighted Interleave: 436 ns
     (20% reduction compared to both)

A small CXL driver change registers a CXL memory node into its package
as the node comes online, using the initiator the driver resolves for
the region; this is where the package layer gets the CPU-side
association that plain NUMA distance does not carry.

The memory_package layer offers a broader interface for grouping and
querying package topology - usable by memory tiering as well - and
package-aware weighted interleave uses the subset it needs.

[PATCH 1/4] mm/numa: introduce nearest_nodes_nodemask()
  Add a NUMA helper that returns every node sharing the minimum distance
  from a source node.

[PATCH 2/4] mm/memory-tiers: package-aware topology management
  Group NUMA nodes into memory packages from firmware topology data,
  expose the grouping read-only under /sys/devices/system/package/, and
  validate the symmetric shape that package-aware placement relies on.

[PATCH 3/4] mm/memory-tiers: register CXL nodes to packages
  Bind a CXL memory node to a package using an initiator CPU node.

[PATCH 4/4] mm/mempolicy: package-aware weighted interleave
  Prefer the current package for weighted interleave node selection,
  behind an opt-in package_mode sysfs toggle that is off by default.

Rakie Kim (4):
  mm/numa: introduce nearest_nodes_nodemask()
  mm/memory-tiers: introduce package-aware topology management for NUMA
    nodes
  mm/memory-tiers: register CXL nodes to memory packages via initiator
  mm/mempolicy: enhance weighted interleave with package-aware locality

 .../ABI/testing/sysfs-devices-system-package  |   35 +
 ...fs-kernel-mm-mempolicy-weighted-interleave |   17 +
 drivers/cxl/core/region.c                     |   54 +
 drivers/cxl/cxl.h                             |    1 +
 drivers/dax/kmem.c                            |    3 +
 include/linux/memory-tiers.h                  |  113 ++
 include/linux/numa.h                          |   11 +
 mm/memory-tiers.c                             | 1009 +++++++++++++++++
 mm/mempolicy.c                                |  200 +++-
 9 files changed, 1439 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-system-package


base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
-- 
2.25.1

[PATCH 1/4] mm/numa: introduce nearest_nodes_nodemask()

[Rakie Kim]

Add a NUMA helper, nearest_nodes_nodemask(), that returns every node in a
given nodemask located at the minimum distance from a source node.

Unlike nearest_node_nodemask(), which returns only a single node, this
helper reports the complete set of nodes that share the closest distance.
This is needed when several nodes are equally near and all of the nearest
candidates must be considered together.

The helper clears the output nodemask and sets every node that meets the
minimum-distance condition. It returns 0 on success, or -EINVAL when the
output argument is invalid.

Signed-off-by: Rakie Kim <rakie.kim@sk.com>
---
 include/linux/numa.h | 11 +++++++++++
 mm/mempolicy.c       | 41 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/include/linux/numa.h b/include/linux/numa.h
index e6baaf6051bc..4f2a0c344122 100644
--- a/include/linux/numa.h
+++ b/include/linux/numa.h
@@ -33,6 +33,8 @@ int numa_nearest_node(int node, unsigned int state);
 
 int nearest_node_nodemask(int node, nodemask_t *mask);
 
+int nearest_nodes_nodemask(int node, const nodemask_t *mask, nodemask_t *out);
+
 #ifndef memory_add_physaddr_to_nid
 int memory_add_physaddr_to_nid(u64 start);
 #endif
@@ -54,6 +56,15 @@ static inline int nearest_node_nodemask(int node, nodemask_t *mask)
 	return NUMA_NO_NODE;
 }
 
+static inline int nearest_nodes_nodemask(int node, const nodemask_t *mask,
+					 nodemask_t *out)
+{
+	if (!out)
+		return -EINVAL;
+	nodes_clear(*out);
+	return 0;
+}
+
 static inline int memory_add_physaddr_to_nid(u64 start)
 {
 	return 0;
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4e4421b22b59..19417b0afc30 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -337,6 +337,47 @@ int nearest_node_nodemask(int node, nodemask_t *mask)
 }
 EXPORT_SYMBOL_GPL(nearest_node_nodemask);
 
+/**
+ * nearest_nodes_nodemask - Find all nodes in @mask that are nearest to @node
+ * @node: The reference node ID to measure distance from
+ * @mask: The set of candidate nodes to compare against
+ * @out:  Pointer to a nodemask that will store the nearest node(s)
+ *
+ * This function iterates over all nodes in @mask and measures the distance
+ * between each candidate node and the given @node using node_distance().
+ * It finds the minimum distance and then records all nodes in @mask that
+ * share that same minimum distance into the output mask @out.
+ *
+ * For example, if multiple nodes have equal minimal distance to @node, all
+ * of them are included in @out.
+ *
+ * Return: 0 on success, or -EINVAL if @out is NULL.
+ */
+int nearest_nodes_nodemask(int node, const nodemask_t *mask, nodemask_t *out)
+{
+	int dist, n, min_dist = INT_MAX;
+
+	if (!out)
+		return -EINVAL;
+
+	nodes_clear(*out);
+
+	for_each_node_mask(n, *mask) {
+		dist = node_distance(node, n);
+
+		if (dist < min_dist) {
+			min_dist = dist;
+			nodes_clear(*out);
+			node_set(n, *out);
+		} else if (dist == min_dist) {
+			node_set(n, *out);
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(nearest_nodes_nodemask);
+
 struct mempolicy *get_task_policy(struct task_struct *p)
 {
 	struct mempolicy *pol = p->mempolicy;
-- 
2.25.1

Re: [PATCH 1/4] mm/numa: introduce nearest_nodes_nodemask()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The function fails silently if the input and output nodemasks alias, wiping the input.
- [Low] The non-NUMA fallback implementation incorrectly reports an empty mask for valid inputs.
--

--- Patch [1]: [PATCH 1/4] mm/numa: introduce nearest_nodes_nodemask() ---
commit 5ec50dc5c5bbc33fbae64a16296aa9f2dc086600
Author: Rakie Kim <rakie.kim@sk.com>

mm/numa: introduce nearest_nodes_nodemask()

This commit introduces nearest_nodes_nodemask(), a helper function that
finds all candidate nodes in a given nodemask located at the minimum
distance from a source node. It updates an output nodemask with the complete
set of nearest nodes, returning 0 on success.

> diff --git a/include/linux/numa.h b/include/linux/numa.h
> index e6baaf6051bcf..4f2a0c344122a 100644
> --- a/include/linux/numa.h
> +++ b/include/linux/numa.h
[ ... ]
> @@ -54,6 +56,15 @@ static inline int nearest_node_nodemask(int node, nodemask_t *mask)
>  	return NUMA_NO_NODE;
>  }
>  
> +static inline int nearest_nodes_nodemask(int node, const nodemask_t *mask,
> +					 nodemask_t *out)
> +{
> +	if (!out)
> +		return -EINVAL;
> +	nodes_clear(*out);

[Severity: Low]
On systems with CONFIG_NUMA disabled, there is only a single implicit node
(Node 0). If a caller passes an input mask containing Node 0, that node is
trivially the nearest.

Should this fallback stub preserve the valid node from the input mask
instead of unconditionally clearing the output mask and falsely reporting no
nearest nodes?

> +	return 0;
> +}
> +
>  static inline int memory_add_physaddr_to_nid(u64 start)
>  {
>  	return 0;
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 4e4421b22b59f..19417b0afc30b 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
[ ... ]
> +int nearest_nodes_nodemask(int node, const nodemask_t *mask, nodemask_t *out)
> +{
> +	int dist, n, min_dist = INT_MAX;
> +
> +	if (!out)
> +		return -EINVAL;
> +
> +	nodes_clear(*out);

[Severity: Medium]
What happens if a caller attempts to filter a nodemask in-place by passing
the same pointer for both the input and output masks?

In-place filtering is a common pattern in the kernel (for example, with
nodes_and()), but if nearest_nodes_nodemask(node, &my_mask, &my_mask) is
called, this initial nodes_clear(*out) will wipe out the input mask before
the loop even begins iterating.

> +
> +	for_each_node_mask(n, *mask) {
> +		dist = node_distance(node, n);
> +
> +		if (dist < min_dist) {
> +			min_dist = dist;
> +			nodes_clear(*out);

[Severity: Medium]
Similarly, if the input and output pointers alias, clearing the output mask
dynamically here destroys the remaining iteration state for the for_each
loop.

Could this be avoided by using a temporary nodemask on the stack,
and copying it to *out at the end of the function?

> +			node_set(n, *out);
> +		} else if (dist == min_dist) {
> +			node_set(n, *out);
> +		}
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(nearest_nodes_nodemask);
> +

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260806080936.421-1-rakie.kim@sk.com?part=1

[PATCH 2/4] mm/memory-tiers: introduce package-aware topology management for NUMA nodes

[Rakie Kim]

The NUMA distance model provides only relative latency values between nodes
and has no notion of structural grouping. Memory policies based on distance
alone therefore cannot tell which nodes are local to the same physical
package and which belong to a different one, which limits how well they can
keep placement and migration within a package.

Introduce a package-aware topology layer that groups NUMA nodes into a
"memory package": a set of CPU nodes and their local memory-only nodes
(such as CXL or HBM). A subsystem that owns a node registers a resolver to
supply its package, and a policy can query, for any node, the other nodes
that belong to the same package.

The grouping is built from the information firmware already provides, as
nodes come online. A CPU node is placed in the package named by its
firmware physical package id. A memory-only node is placed by an initiator
CPU node when a driver supplies one, or otherwise by its nearest CPU node
under the SLIT distance table; a SLIT-derived entry is provisional and is
upgraded once a driver supplies an initiator. This makes the package
association explicit so that policies can consume it, and the mapping can
be revisited as firmware interfaces expose better information.

A single package may hold more than one CPU node or more than one
memory-only node, so the model does not assume a one-to-one package to node
or package to CXL device relationship.

The current package topology is exposed read-only under
/sys/devices/system/package/packageN/: package_nodes (all NUMA nodes in the
package), package_cpu_nodes (CPU nodes), package_mem_only_nodes
(memory-only nodes), physical_package_id (physical package id).

The interface is read-only by design: it reports the topology the kernel
derived but does not let user space override it. A machine whose firmware
describes the topology incorrectly should be fixed in firmware rather than
papered over through sysfs.

Topology is also validated for the symmetric shape that package-aware
placement relies on - every package having the same number of CPU and
memory-only nodes, at least two packages, and at least two nodes per
package. The verdict is recomputed at boot and on node hotplug and is
published for policies to consult.

On any topology that does not meet these conditions the verdict is simply
false, so a consumer degrades to its original, non-package-aware behavior
instead of acting on a grouping that does not describe the machine.

Signed-off-by: Rakie Kim <rakie.kim@sk.com>
---
 .../ABI/testing/sysfs-devices-system-package  |   35 +
 include/linux/memory-tiers.h                  |  113 ++
 mm/memory-tiers.c                             | 1009 +++++++++++++++++
 3 files changed, 1157 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-devices-system-package

diff --git a/Documentation/ABI/testing/sysfs-devices-system-package b/Documentation/ABI/testing/sysfs-devices-system-package
new file mode 100644
index 000000000000..6500f9e5ff19
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-system-package
@@ -0,0 +1,35 @@
+What:		/sys/devices/system/package/
+Date:		August 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	Memory package topology
+
+		A "memory package" groups the NUMA nodes associated with one
+		physical CPU package (socket): the nodes that have CPUs and
+		the nodes that only have memory (e.g. CXL or HBM).
+
+		All attributes are read-only; the topology cannot be
+		overridden from user space.
+
+What:		/sys/devices/system/package/packageN/package_nodes
+Date:		August 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	All NUMA nodes in this package, in nodelist format
+		(e.g. "0,2").
+
+What:		/sys/devices/system/package/packageN/package_cpu_nodes
+Date:		August 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	The nodes that have CPUs in this package, in nodelist
+		format.
+
+What:		/sys/devices/system/package/packageN/package_mem_only_nodes
+Date:		August 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	The nodes that only have memory (e.g. CXL/HBM) in this
+		package, in nodelist format.
+
+What:		/sys/devices/system/package/packageN/physical_package_id
+Date:		August 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	The physical package id this group corresponds to, as
+		reported by CPU topology.
diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
index 7999c58629ee..f8778c43429f 100644
--- a/include/linux/memory-tiers.h
+++ b/include/linux/memory-tiers.h
@@ -52,10 +52,25 @@ int mt_perf_to_adistance(struct access_coordinate *perf, int *adist);
 struct memory_dev_type *mt_find_alloc_memory_type(int adist,
 						  struct list_head *memory_types);
 void mt_put_memory_types(struct list_head *memory_types);
+
+int register_mp_package_notifier(struct notifier_block *notifier);
+void unregister_mp_package_notifier(struct notifier_block *notifier);
+int mp_probe_package_id(int nid);
+int mp_add_package_node_by_initiator(int nid, int initiator_nid);
+int mp_add_package_node(int nid);
+int mp_get_package_nodes(int nid, nodemask_t *out);
+int mp_get_package_cpu_nodes(int nid, nodemask_t *out);
+int mp_get_package_memory_only_nodes(int nid, nodemask_t *out);
+bool mp_is_topology_symmetric(void);
 #ifdef CONFIG_NUMA_MIGRATION
 int next_demotion_node(int node, const nodemask_t *allowed_mask);
 void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
 bool node_is_toptier(int node);
+
+int mp_next_demotion_nodemask(int nid, nodemask_t *out);
+int mp_next_demotion_node(int nid);
+int mp_next_promotion_nodemask(int nid, nodemask_t *out);
+int mp_next_promotion_node(int nid);
 #else
 static inline int next_demotion_node(int node, const nodemask_t *allowed_mask)
 {
@@ -71,6 +86,30 @@ static inline bool node_is_toptier(int node)
 {
 	return true;
 }
+
+static inline int mp_next_demotion_nodemask(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline int mp_next_demotion_node(int nid)
+{
+	return NUMA_NO_NODE;
+}
+
+static inline int mp_next_promotion_nodemask(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline int mp_next_promotion_node(int nid)
+{
+	return NUMA_NO_NODE;
+}
 #endif
 
 #else
@@ -151,5 +190,79 @@ static inline struct memory_dev_type *mt_find_alloc_memory_type(int adist,
 static inline void mt_put_memory_types(struct list_head *memory_types)
 {
 }
+
+static inline int register_mp_package_notifier(struct notifier_block *notifier)
+{
+	return 0;
+}
+
+static inline void unregister_mp_package_notifier(struct notifier_block *notifier)
+{
+}
+
+static inline int mp_probe_package_id(int nid)
+{
+	return NOTIFY_DONE;
+}
+
+static inline int mp_add_package_node_by_initiator(int nid, int initiator_nid)
+{
+	return 0;
+}
+
+static inline int mp_add_package_node(int nid)
+{
+	return 0;
+}
+
+static inline int mp_get_package_nodes(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline int mp_get_package_cpu_nodes(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline int mp_get_package_memory_only_nodes(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline bool mp_is_topology_symmetric(void)
+{
+	return false;
+}
+
+static inline int mp_next_demotion_nodemask(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline int mp_next_demotion_node(int nid)
+{
+	return NUMA_NO_NODE;
+}
+
+static inline int mp_next_promotion_nodemask(int nid, nodemask_t *out)
+{
+	if (out)
+		nodes_clear(*out);
+	return -ENOENT;
+}
+
+static inline int mp_next_promotion_node(int nid)
+{
+	return NUMA_NO_NODE;
+}
 #endif	/* CONFIG_NUMA */
 #endif  /* _LINUX_MEMORY_TIERS_H */
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b..5932df315604 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+#include <linux/cleanup.h>
 #include <linux/slab.h>
 #include <linux/lockdep.h>
 #include <linux/sysfs.h>
@@ -51,6 +52,11 @@ static const struct bus_type memory_tier_subsys = {
 	.dev_name = "memory_tier",
 };
 
+static const struct bus_type package_subsys = {
+	.name = "package",
+	.dev_name = "package",
+};
+
 #ifdef CONFIG_NUMA_BALANCING
 /**
  * folio_use_access_time - check if a folio reuses cpupid for page access time
@@ -1007,3 +1013,1006 @@ static int __init numa_init_sysfs(void)
 subsys_initcall(numa_init_sysfs);
 #endif /* CONFIG_SYSFS */
 #endif
+
+/**
+ * enum mp_nodes_type - Selector for which subset of a package to return
+ * @MP_NODES_ALL:       All NUMA nodes that belong to the package.
+ * @MP_NODES_CPU:       Only CPU nodes in the package.
+ * @MP_NODES_MEM_ONLY:  Only memory-only nodes (e.g. CXL/HBM) in the package.
+ *
+ * Used internally to choose which nodemask to expose for a given package.
+ */
+enum mp_nodes_type {
+	MP_NODES_ALL,
+	MP_NODES_CPU,
+	MP_NODES_MEM_ONLY
+};
+
+/**
+ * struct memory_package - Per-physical-package container
+ * @package_id:          Physical package id (from topology).
+ * @nodes:               Nodemask of all member nodes in this package.
+ * @cpu_nodes:           Nodemask of CPU nodes in this package.
+ * @memory_only_nodes:   Nodemask of memory-only nodes in this package.
+ * @cpu_list:            List head of CPU-type members.
+ * @memory_only_list:    List head of memory-only members.
+ * @list:                Linkage on the global @memory_packages list.
+ * @dev:                 sysfs device for this package.
+ *
+ * A memory_package groups NUMA nodes that share the same physical CPU package.
+ * The masks are used to implement package-local placement/demotion/promotion.
+ */
+struct memory_package {
+	int package_id;
+	nodemask_t nodes;
+	nodemask_t cpu_nodes;
+	nodemask_t memory_only_nodes;
+	struct list_head cpu_list;
+	struct list_head memory_only_list;
+	struct list_head list;
+	struct device dev;
+};
+
+/**
+ * enum mpn_source_flags - Source used to resolve a node's package membership
+ * @MPN_SRC_UNKNOWN:     Unknown/unspecified.
+ * @MPN_SRC_CPU:         Directly resolved from a CPU node (1:1).
+ * @MPN_SRC_INITIATOR:   Resolved via an initiator CPU node provided by a driver.
+ * @MPN_SRC_SLIT:        Resolved via SLIT/nearest-node.
+ *
+ * These flags are informational; they describe how a given node was bound to
+ * its package and help with policy decisions later.
+ */
+enum mpn_source_flags {
+	MPN_SRC_UNKNOWN		= 0,
+	MPN_SRC_CPU		= BIT(1),
+	MPN_SRC_INITIATOR	= BIT(2),
+	MPN_SRC_SLIT		= BIT(3)
+};
+
+/**
+ * struct memory_package_node - Per-node membership and preferences
+ * @nid:              NUMA node id for this entry.
+ * @initiator_nid:    CPU nid that served as the initiator when resolving @nid.
+ * @package_id:       Resolved package id that @nid belongs to.
+ * @source_flags:     One of &enum mpn_source_flags describing the resolution.
+ * @preferred:        Opposite-type nearest candidates inside the same package.
+ * @package:          Pointer to the owning &struct memory_package (NULL until bound).
+ * @package_entry:    Linkage on the owning package's type list.
+ *
+ * Each NUMA node that participates in package-aware policy gets a wrapper entry
+ * that caches package membership and the precomputed set of preferred targets.
+ */
+struct memory_package_node {
+	int nid;
+	int initiator_nid;
+	int package_id;
+	int source_flags;
+	nodemask_t preferred;
+	struct memory_package *package;
+	struct list_head package_entry;
+};
+
+#define node_is_memory_only(_nid) \
+	(node_state((_nid), N_MEMORY) && !node_state((_nid), N_CPU))
+
+static BLOCKING_NOTIFIER_HEAD(mp_package_algorithms);
+
+static LIST_HEAD(memory_packages);
+static struct memory_package_node *mpns[MAX_NUMNODES];
+static DEFINE_MUTEX(memory_package_lock);
+
+/*
+ * RCU snapshot of the package topology. The allocation path reads it
+ * often, so it is published for lockless reads instead of locking on
+ * every access.
+ */
+struct mp_snapshot {
+	struct rcu_head rcu;
+	int nr_packages;
+	int pkg_of[MAX_NUMNODES];
+	struct {
+		nodemask_t nodes;
+		nodemask_t cpu_nodes;
+		nodemask_t memory_only_nodes;
+	} pkg[];
+};
+
+static struct mp_snapshot __rcu *mp_snapshot;
+
+/**
+ * register_mp_package_notifier - Register a package resolution algorithm
+ * @notifier: Notifier called with the nid to resolve (see mp_probe_package_id()).
+ *
+ * Drivers (e.g., CXL region/decoder code) register here to supply a package
+ * hint for newly appearing nodes. The notifier is invoked during nid->package
+ * resolution.
+ *
+ * Return: 0 on success, negative errno on failure.
+ */
+int register_mp_package_notifier(struct notifier_block *notifier)
+{
+	return blocking_notifier_chain_register(&mp_package_algorithms, notifier);
+}
+EXPORT_SYMBOL_GPL(register_mp_package_notifier);
+
+/**
+ * unregister_mp_package_notifier - Unregister a package resolution algorithm
+ * @notifier: Notifier previously registered with register_mp_package_notifier().
+ */
+void unregister_mp_package_notifier(struct notifier_block *notifier)
+{
+	blocking_notifier_chain_unregister(&mp_package_algorithms, notifier);
+}
+EXPORT_SYMBOL_GPL(unregister_mp_package_notifier);
+
+/**
+ * mp_probe_package_id - Invoke registered notifiers to resolve a node's package
+ * @nid: NUMA node id to resolve.
+ *
+ * Calls the blocking notifier chain to let subsystems provide an initiator or
+ * package id for @nid.
+ *
+ * Return: Notifier return code (>=0 typically); negative errno on failure.
+ */
+int mp_probe_package_id(int nid)
+{
+	return blocking_notifier_call_chain(&mp_package_algorithms, nid, NULL);
+}
+EXPORT_SYMBOL_GPL(mp_probe_package_id);
+
+static int mp_node_to_package_id(int nid)
+{
+	int package_id;
+	unsigned int first_cpu;
+	const struct cpumask *cpu_mask;
+
+	if (nid < 0 || nid >= MAX_NUMNODES)
+		return -EINVAL;
+
+	if (!node_state(nid, N_CPU))
+		return -EINVAL;
+
+	cpu_mask = cpumask_of_node(nid);
+	if (cpumask_empty(cpu_mask))
+		return -EINVAL;
+
+	first_cpu = cpumask_first(cpu_mask);
+	if (first_cpu >= nr_cpu_ids)
+		return -EINVAL;
+
+	package_id = topology_physical_package_id(first_cpu);
+	if (package_id < 0)
+		return -EINVAL;
+
+	return package_id;
+}
+
+static void update_package_preferred(struct memory_package *mp)
+{
+	struct memory_package_node *mpn;
+
+	lockdep_assert_held(&memory_package_lock);
+
+	/*
+	 * For each CPU node, compute its preferred set as the nearest
+	 * memory-only node(s) within the same package. If the package has
+	 * no memory-only nodes, fall back to a self-reference so callers
+	 * never see an empty preferred set.
+	 */
+	list_for_each_entry(mpn, &mp->cpu_list, package_entry) {
+		nodes_clear(mpn->preferred);
+		if (!nodes_empty(mp->memory_only_nodes))
+			nearest_nodes_nodemask(mpn->nid, &mp->memory_only_nodes,
+					       &mpn->preferred);
+		else
+			node_set(mpn->nid, mpn->preferred);
+	}
+
+	/*
+	 * Symmetrically, for each memory-only node, compute its preferred set
+	 * as the nearest CPU node(s) within the same package. If the package
+	 * has no CPU nodes, fall back to a self-reference.
+	 */
+	list_for_each_entry(mpn, &mp->memory_only_list, package_entry) {
+		nodes_clear(mpn->preferred);
+		if (!nodes_empty(mp->cpu_nodes))
+			nearest_nodes_nodemask(mpn->nid, &mp->cpu_nodes,
+					       &mpn->preferred);
+		else
+			node_set(mpn->nid, mpn->preferred);
+	}
+}
+
+static inline bool memory_package_is_empty(struct memory_package *mp)
+{
+	lockdep_assert_held(&memory_package_lock);
+
+	return (nodes_empty(mp->cpu_nodes) && nodes_empty(mp->memory_only_nodes));
+}
+
+static inline bool package_node_is_valid(int nid)
+{
+	if (!mpns[nid])
+		return false;
+
+	if (nodes_empty(mpns[nid]->preferred) || (mpns[nid]->package == NULL))
+		return false;
+
+	return true;
+}
+
+static const struct attribute_group *memory_package_groups[];
+
+/* Freed when the last reference to the package's sysfs device is dropped. */
+static void memory_package_release(struct device *dev)
+{
+	struct memory_package *mp = container_of(dev, struct memory_package, dev);
+
+	kfree(mp);
+}
+
+static struct memory_package *create_memory_package(int package_id)
+{
+	struct memory_package *mempackage;
+	int ret;
+
+	mempackage = kzalloc_obj(*mempackage);
+	if (!mempackage)
+		return ERR_PTR(-ENOMEM);
+
+	mempackage->package_id = package_id;
+	mempackage->nodes = NODE_MASK_NONE;
+	mempackage->cpu_nodes = NODE_MASK_NONE;
+	mempackage->memory_only_nodes = NODE_MASK_NONE;
+	INIT_LIST_HEAD(&mempackage->cpu_list);
+	INIT_LIST_HEAD(&mempackage->memory_only_list);
+	INIT_LIST_HEAD(&mempackage->list);
+	device_initialize(&mempackage->dev);
+	mempackage->dev.release = memory_package_release;
+	dev_set_drvdata(&mempackage->dev, mempackage);
+	mempackage->dev.bus = &package_subsys;
+	mempackage->dev.groups = memory_package_groups;
+	ret = dev_set_name(&mempackage->dev, "package%d", package_id);
+	if (ret) {
+		put_device(&mempackage->dev);
+		return ERR_PTR(ret);
+	}
+
+	return mempackage;
+}
+
+static struct memory_package *find_create_memory_package(int package_id)
+{
+	struct memory_package *mempackage, *existing;
+	int ret;
+
+	mutex_lock(&memory_package_lock);
+	list_for_each_entry(mempackage, &memory_packages, list) {
+		if (mempackage->package_id == package_id) {
+			mutex_unlock(&memory_package_lock);
+			return mempackage;
+		}
+	}
+	mutex_unlock(&memory_package_lock);
+
+	mempackage = create_memory_package(package_id);
+	if (IS_ERR(mempackage))
+		return mempackage;
+
+	mutex_lock(&memory_package_lock);
+	list_for_each_entry(existing, &memory_packages, list) {
+		if (existing->package_id == package_id) {
+			mutex_unlock(&memory_package_lock);
+			put_device(&mempackage->dev);
+			return existing;
+		}
+	}
+	list_add(&mempackage->list, &memory_packages);
+	mutex_unlock(&memory_package_lock);
+
+	ret = device_add(&mempackage->dev);
+	if (ret) {
+		mutex_lock(&memory_package_lock);
+		list_del(&mempackage->list);
+		mutex_unlock(&memory_package_lock);
+		put_device(&mempackage->dev);
+		return ERR_PTR(ret);
+	}
+
+	return mempackage;
+}
+
+static void mp_snapshot_rebuild(void)
+{
+	struct mp_snapshot *new, *old;
+	struct memory_package *mp;
+	int nr = 0, i = 0, nid;
+
+	lockdep_assert_held(&memory_package_lock);
+
+	list_for_each_entry(mp, &memory_packages, list)
+		nr++;
+
+	new = kvzalloc(struct_size(new, pkg, nr), GFP_KERNEL);
+	if (!new)
+		return;
+
+	memset(new->pkg_of, 0xff, sizeof(new->pkg_of));
+
+	list_for_each_entry(mp, &memory_packages, list) {
+		new->pkg[i].nodes = mp->nodes;
+		new->pkg[i].cpu_nodes = mp->cpu_nodes;
+		new->pkg[i].memory_only_nodes = mp->memory_only_nodes;
+		for_each_node_mask(nid, mp->nodes)
+			new->pkg_of[nid] = i;
+		i++;
+	}
+	new->nr_packages = nr;
+
+	old = rcu_replace_pointer(mp_snapshot, new,
+				  lockdep_is_held(&memory_package_lock));
+	if (old)
+		kvfree_rcu(old, rcu);
+}
+
+static int bind_node_to_package(int nid)
+{
+	int package_id, pkg_id;
+	struct memory_package *mp;
+	nodemask_t nodes, cpu, mem;
+
+	mutex_lock(&memory_package_lock);
+	if (!mpns[nid]) {
+		mutex_unlock(&memory_package_lock);
+		return -EINVAL;
+	}
+	package_id = mpns[nid]->package_id;
+	mutex_unlock(&memory_package_lock);
+
+	mp = find_create_memory_package(package_id);
+	if (IS_ERR(mp))
+		return PTR_ERR(mp);
+
+	mutex_lock(&memory_package_lock);
+	if (!mpns[nid]) {
+		mutex_unlock(&memory_package_lock);
+		return -ENOENT;
+	}
+	mpns[nid]->package = mp;
+	node_set(mpns[nid]->nid, mp->nodes);
+	if (node_is_memory_only(mpns[nid]->nid)) {
+		node_set(mpns[nid]->nid, mp->memory_only_nodes);
+		list_add(&mpns[nid]->package_entry, &mp->memory_only_list);
+	} else {
+		node_set(mpns[nid]->nid, mp->cpu_nodes);
+		list_add(&mpns[nid]->package_entry, &mp->cpu_list);
+	}
+	update_package_preferred(mp);
+	mp_snapshot_rebuild();
+	pkg_id = mp->package_id;
+	nodes = mp->nodes;
+	cpu = mp->cpu_nodes;
+	mem = mp->memory_only_nodes;
+	mutex_unlock(&memory_package_lock);
+
+	pr_info("memory_package %d: nodes=%*pbl cpu=%*pbl memory_only=%*pbl\n",
+		pkg_id, nodemask_pr_args(&nodes),
+		nodemask_pr_args(&cpu), nodemask_pr_args(&mem));
+
+	return 0;
+}
+
+static void unbind_node_to_package(struct memory_package *mp, int nid)
+{
+	lockdep_assert_held(&memory_package_lock);
+
+	node_clear(nid, mp->nodes);
+	if (node_state(nid, N_CPU))
+		node_clear(nid, mp->cpu_nodes);
+	else
+		node_clear(nid, mp->memory_only_nodes);
+
+	if (mpns[nid])
+		list_del(&mpns[nid]->package_entry);
+
+	update_package_preferred(mp);
+}
+
+static struct memory_package_node *create_package_node(int nid, int initiator_nid)
+{
+	int cpu_nid, package_id;
+	int source_flags;
+	struct memory_package_node *mpn;
+
+	if (node_state(nid, N_CPU)) {
+		cpu_nid = nid;
+		source_flags = MPN_SRC_CPU;
+	} else {
+		if (initiator_nid >= 0) {
+			cpu_nid = initiator_nid;
+			source_flags = MPN_SRC_INITIATOR;
+		} else {
+			/*
+			 * No driver-supplied initiator: fall back to the
+			 * nearest CPU node (via SLIT/numa_distance).
+			 */
+			cpu_nid = numa_nearest_node(nid, N_CPU);
+			source_flags = MPN_SRC_SLIT;
+		}
+	}
+
+	package_id = mp_node_to_package_id(cpu_nid);
+	if (package_id < 0)
+		return ERR_PTR(-EINVAL);
+
+	mpn = kzalloc_obj(*mpn);
+	if (!mpn)
+		return ERR_PTR(-ENOMEM);
+
+	mpn->nid = nid;
+	mpn->initiator_nid = cpu_nid;
+	mpn->package_id = package_id;
+	mpn->source_flags = source_flags;
+	mpn->preferred = NODE_MASK_NONE;
+	mpn->package = NULL;
+	INIT_LIST_HEAD(&mpn->package_entry);
+
+	return mpn;
+}
+
+/*
+ * Topology symmetry status
+ * Indicates whether all packages have identical node structure
+ * (same number of CPU nodes and memory-only nodes).
+ */
+static bool topology_symmetric;
+
+static void validate_topology_symmetry(void);
+
+static struct memory_package *__destroy_package_node(int nid)
+{
+	struct memory_package_node *mpn;
+	struct memory_package *mp, *unreg_mp = NULL;
+
+	lockdep_assert_held(&memory_package_lock);
+
+	mpn = mpns[nid];
+	if (!mpn)
+		return NULL;
+
+	mp = mpn->package;
+	if (mp) {
+		unbind_node_to_package(mp, nid);
+		mpn->package = NULL;
+
+		if (memory_package_is_empty(mp)) {
+			list_del(&mp->list);
+			unreg_mp = mp;
+		}
+	}
+
+	mpns[nid] = NULL;
+	kfree(mpn);
+	mp_snapshot_rebuild();
+
+	return unreg_mp;
+}
+
+static void destroy_package_node(int nid)
+{
+	struct memory_package *unreg_mp;
+
+	mutex_lock(&memory_package_lock);
+	unreg_mp = __destroy_package_node(nid);
+	mutex_unlock(&memory_package_lock);
+
+	if (unreg_mp)
+		device_unregister(&unreg_mp->dev);
+
+	validate_topology_symmetry();
+}
+
+static int find_package_node(int nid, int initiator_nid)
+{
+	struct memory_package *unreg_mp = NULL;
+	int ret = nid;
+
+	mutex_lock(&memory_package_lock);
+	if (!mpns[nid]) {
+		ret = NUMA_NO_NODE;
+	} else if (mpns[nid]->source_flags == MPN_SRC_SLIT && initiator_nid >= 0) {
+		/*
+		 * SLIT-derived entries are provisional; if a driver later
+		 * provides an explicit initiator, drop the provisional
+		 * entry and rebuild with the stronger hint.
+		 */
+		unreg_mp = __destroy_package_node(nid);
+		ret = NUMA_NO_NODE;
+	}
+	mutex_unlock(&memory_package_lock);
+
+	if (unreg_mp)
+		device_unregister(&unreg_mp->dev);
+
+	return ret;
+}
+
+static int find_create_package_node(int nid, int initiator_nid)
+{
+	int mpn_nid;
+	struct memory_package_node *mpn;
+
+	mpn_nid = find_package_node(nid, initiator_nid);
+	if (mpn_nid != NUMA_NO_NODE)
+		return mpn_nid;
+
+	mpn = create_package_node(nid, initiator_nid);
+	if (IS_ERR(mpn))
+		return PTR_ERR(mpn);
+
+	guard(mutex)(&memory_package_lock);
+	if (mpns[nid]) {
+		kfree(mpn);
+		return nid;
+	}
+	mpns[nid] = mpn;
+
+	return nid;
+}
+
+static int create_node_with_package(int nid)
+{
+	int ret;
+
+	ret = find_create_package_node(nid, NUMA_NO_NODE);
+	if (ret < 0)
+		return ret;
+
+	ret = bind_node_to_package(nid);
+	if (ret)
+		return ret;
+
+	validate_topology_symmetry();
+	return 0;
+}
+
+/**
+ * mp_add_package_node_by_initiator - Add a node with an initiator
+ * @nid:            Target NUMA node to add.
+ * @initiator_nid:  CPU nid used to resolve @nid's package (>=0).
+ *
+ * Ensures that a &struct memory_package_node exists for @nid and that its
+ * package_id is determined using @initiator_nid when provided. Binding to the
+ * package is not performed here.
+ *
+ * Return: 0 on success; negative errno on failure.
+ */
+int mp_add_package_node_by_initiator(int nid, int initiator_nid)
+{
+	int ret;
+
+	ret = find_create_package_node(nid, initiator_nid);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(mp_add_package_node_by_initiator);
+
+/**
+ * mp_add_package_node - Add a node, resolving package automatically
+ * @nid: Target NUMA node to add.
+ *
+ * Wrapper over mp_add_package_node_by_initiator() that requests automatic
+ * initiator resolution (e.g., nearest CPU).
+ *
+ * Return: 0 on success; negative errno on failure.
+ */
+int mp_add_package_node(int nid)
+{
+	return mp_add_package_node_by_initiator(nid, NUMA_NO_NODE);
+}
+EXPORT_SYMBOL_GPL(mp_add_package_node);
+
+static int __mp_get_package_nodemask(int nid, enum mp_nodes_type node_type,
+				     nodemask_t *out)
+{
+	struct mp_snapshot *snap;
+	int pkg;
+
+	if (!out)
+		return -EINVAL;
+
+	nodes_clear(*out);
+
+	if (nid < 0 || nid >= MAX_NUMNODES)
+		return -EINVAL;
+
+	guard(rcu)();
+
+	snap = rcu_dereference(mp_snapshot);
+	if (!snap)
+		return -ENOENT;
+
+	pkg = snap->pkg_of[nid];
+	if (pkg < 0)
+		return -ENOENT;
+
+	switch (node_type) {
+	case MP_NODES_ALL:
+		nodes_copy(*out, snap->pkg[pkg].nodes);
+		break;
+	case MP_NODES_CPU:
+		nodes_copy(*out, snap->pkg[pkg].cpu_nodes);
+		break;
+	case MP_NODES_MEM_ONLY:
+		nodes_copy(*out, snap->pkg[pkg].memory_only_nodes);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+#ifdef CONFIG_NUMA_MIGRATION
+static int __mp_get_preferred_nodemask(int nid, enum mp_nodes_type node_type,
+				    nodemask_t *out)
+{
+	int ret = 0;
+
+	/* No hot-path callers: the mutex is fine here. */
+	guard(mutex)(&memory_package_lock);
+
+	if (!out) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	nodes_clear(*out);
+
+	if (nid < 0 || nid >= MAX_NUMNODES) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (node_type == MP_NODES_CPU) {
+		if (node_is_memory_only(nid)) {
+			ret = -EINVAL;
+			goto out;
+		}
+	} else if (node_type == MP_NODES_MEM_ONLY) {
+		if (!node_is_memory_only(nid)) {
+			ret = -EINVAL;
+			goto out;
+		}
+	} else {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (!package_node_is_valid(nid)) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	nodes_copy(*out, mpns[nid]->preferred);
+
+out:
+	return ret;
+}
+
+/**
+ * mp_next_demotion_nodemask - Demotion candidates within a package
+ * @nid: CPU node from which memory would be demoted.
+ * @out: Output nodemask of nearest memory-only targets in the same package.
+ *
+ * Return: 0 on success; negative errno if @nid is invalid or not initialized.
+ */
+int mp_next_demotion_nodemask(int nid, nodemask_t *out)
+{
+	return __mp_get_preferred_nodemask(nid, MP_NODES_CPU, out);
+}
+EXPORT_SYMBOL_GPL(mp_next_demotion_nodemask);
+
+/**
+ * mp_next_demotion_node - Pick one demotion target
+ * @nid: CPU node from which memory would be demoted.
+ *
+ * Picks one target (random among the nearest) from mp_next_demotion_nodemask().
+ *
+ * Return: target nid on success, or NUMA_NO_NODE if no candidate is available.
+ */
+int mp_next_demotion_node(int nid)
+{
+	int target_nid;
+	nodemask_t target_nodemask;
+
+	if (mp_next_demotion_nodemask(nid, &target_nodemask))
+		return NUMA_NO_NODE;
+	if (nodes_empty(target_nodemask))
+		return NUMA_NO_NODE;
+
+	target_nid = node_random(&target_nodemask);
+
+	return target_nid;
+}
+EXPORT_SYMBOL_GPL(mp_next_demotion_node);
+
+/**
+ * mp_next_promotion_nodemask - Promotion candidates within a package
+ * @nid: Memory-only node towards which promotion seeks CPU locality.
+ * @out: Output nodemask of nearest CPU targets in the same package.
+ *
+ * Return: 0 on success; negative errno if @nid is invalid or not initialized.
+ */
+int mp_next_promotion_nodemask(int nid, nodemask_t *out)
+{
+	return __mp_get_preferred_nodemask(nid, MP_NODES_MEM_ONLY, out);
+}
+EXPORT_SYMBOL_GPL(mp_next_promotion_nodemask);
+
+/**
+ * mp_next_promotion_node - Pick one promotion target
+ * @nid: Memory-only node to be promoted towards CPUs.
+ *
+ * Picks one target (random among the nearest) from mp_next_promotion_nodemask().
+ *
+ * Return: target nid on success, or NUMA_NO_NODE if no candidate is available.
+ */
+int mp_next_promotion_node(int nid)
+{
+	int target_nid;
+	nodemask_t target_nodemask;
+
+	if (mp_next_promotion_nodemask(nid, &target_nodemask))
+		return NUMA_NO_NODE;
+	if (nodes_empty(target_nodemask))
+		return NUMA_NO_NODE;
+
+	target_nid = node_random(&target_nodemask);
+
+	return target_nid;
+}
+EXPORT_SYMBOL_GPL(mp_next_promotion_node);
+#endif /* CONFIG_NUMA_MIGRATION */
+
+/**
+ * mp_get_package_nodes - Return all members of @nid's package
+ * @nid: Any NUMA node in the package.
+ * @out: Output nodemask to receive all members.
+ *
+ * Return: 0 on success; negative errno if @nid is invalid or not initialized.
+ */
+int mp_get_package_nodes(int nid, nodemask_t *out)
+{
+	return __mp_get_package_nodemask(nid, MP_NODES_ALL, out);
+}
+EXPORT_SYMBOL_GPL(mp_get_package_nodes);
+
+/**
+ * mp_get_package_cpu_nodes - Return CPU members of @nid's package
+ * @nid: Any NUMA node in the package.
+ * @out: Output nodemask to receive CPU members.
+ *
+ * Return: 0 on success; negative errno if @nid is invalid or not initialized.
+ */
+int mp_get_package_cpu_nodes(int nid, nodemask_t *out)
+{
+	return __mp_get_package_nodemask(nid, MP_NODES_CPU, out);
+}
+EXPORT_SYMBOL_GPL(mp_get_package_cpu_nodes);
+
+/**
+ * mp_get_package_memory_only_nodes - Return memory-only members of @nid's package
+ * @nid: Any NUMA node in the package.
+ * @out: Output nodemask to receive memory-only members.
+ *
+ * Return: 0 on success; negative errno if @nid is invalid or not initialized.
+ */
+int mp_get_package_memory_only_nodes(int nid, nodemask_t *out)
+{
+	return __mp_get_package_nodemask(nid, MP_NODES_MEM_ONLY, out);
+}
+EXPORT_SYMBOL_GPL(mp_get_package_memory_only_nodes);
+
+static int __meminit mp_hotplug_callback(struct notifier_block *nb,
+		unsigned long action, void *_arg)
+{
+	int nid;
+	struct node_notify *nn = _arg;
+
+	nid = nn->nid;
+	if (nid < 0)
+		return notifier_from_errno(0);
+
+	switch (action) {
+	case NODE_REMOVED_LAST_MEMORY:
+		destroy_package_node(nid);
+		break;
+
+	case NODE_ADDED_FIRST_MEMORY:
+		create_node_with_package(nid);
+		break;
+
+	default:
+		break;
+	}
+
+	return notifier_from_errno(0);
+}
+
+/*
+ * sysfs interface for memory_package topology
+ * Read-only attributes:
+ *   - package_nodes:       All NUMA nodes in this package (node mask)
+ *   - package_cpu_nodes:   CPU nodes in this package (node mask)
+ *   - package_mem_only_nodes: Memory-only nodes in this package (node mask)
+ *   - physical_package_id: Physical package ID
+ */
+
+static ssize_t package_nodes_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	struct memory_package *mp = dev_get_drvdata(dev);
+
+	guard(mutex)(&memory_package_lock);
+	return sysfs_emit(buf, "%*pbl\n", nodemask_pr_args(&mp->nodes));
+}
+static DEVICE_ATTR_RO(package_nodes);
+
+static ssize_t package_cpu_nodes_show(struct device *dev,
+				      struct device_attribute *attr, char *buf)
+{
+	struct memory_package *mp = dev_get_drvdata(dev);
+
+	guard(mutex)(&memory_package_lock);
+	return sysfs_emit(buf, "%*pbl\n", nodemask_pr_args(&mp->cpu_nodes));
+}
+static DEVICE_ATTR_RO(package_cpu_nodes);
+
+static ssize_t package_mem_only_nodes_show(struct device *dev,
+					   struct device_attribute *attr, char *buf)
+{
+	struct memory_package *mp = dev_get_drvdata(dev);
+
+	guard(mutex)(&memory_package_lock);
+	return sysfs_emit(buf, "%*pbl\n", nodemask_pr_args(&mp->memory_only_nodes));
+}
+static DEVICE_ATTR_RO(package_mem_only_nodes);
+
+static ssize_t physical_package_id_show(struct device *dev,
+					struct device_attribute *attr, char *buf)
+{
+	struct memory_package *mp = dev_get_drvdata(dev);
+
+	return sysfs_emit(buf, "%d\n", mp->package_id);
+}
+static DEVICE_ATTR_RO(physical_package_id);
+
+static struct attribute *memory_package_attrs[] = {
+	&dev_attr_package_nodes.attr,
+	&dev_attr_package_cpu_nodes.attr,
+	&dev_attr_package_mem_only_nodes.attr,
+	&dev_attr_physical_package_id.attr,
+	NULL,
+};
+
+static const struct attribute_group memory_package_group = {
+	.attrs = memory_package_attrs,
+};
+
+static const struct attribute_group *memory_package_groups[] = {
+	&memory_package_group,
+	NULL,
+};
+
+static int __init memory_package_sysfs_init(void)
+{
+	int ret;
+
+	ret = subsys_system_register(&package_subsys, NULL);
+	if (ret) {
+		pr_err("memory_package subsys_system_register failed: %d\n", ret);
+		return ret;
+	}
+	return 0;
+}
+core_initcall(memory_package_sysfs_init);
+
+/**
+ * mp_is_topology_symmetric - Check if all packages have identical node structure
+ *
+ * Returns true if all memory packages have the same number of CPU nodes
+ * and memory-only nodes, indicating a symmetric topology.
+ *
+ * The topology_symmetric variable is protected by memory_package_lock for
+ * writes (in validate_topology_symmetry).
+ */
+bool mp_is_topology_symmetric(void)
+{
+	return READ_ONCE(topology_symmetric);
+}
+EXPORT_SYMBOL_GPL(mp_is_topology_symmetric);
+
+/* Walk the package list and compare node counts; the caller holds the lock. */
+static bool __check_topology_symmetry(void)
+{
+	struct memory_package *mp;
+	int ref_cpu_node_count = 0, ref_memory_only_node_count = 0;
+	int package_count = 0;
+
+	lockdep_assert_held(&memory_package_lock);
+
+	list_for_each_entry(mp, &memory_packages, list) {
+		int cpu_node_count = nodes_weight(mp->cpu_nodes);
+		int memory_only_node_count = nodes_weight(mp->memory_only_nodes);
+		int total_node_count = cpu_node_count + memory_only_node_count;
+
+		package_count++;
+
+		/* Each package must have at least 2 nodes */
+		if (total_node_count < 2)
+			return false;
+
+		if (package_count == 1) {
+			ref_cpu_node_count = cpu_node_count;
+			ref_memory_only_node_count = memory_only_node_count;
+		} else if (cpu_node_count != ref_cpu_node_count ||
+			   memory_only_node_count != ref_memory_only_node_count) {
+			return false;
+		}
+	}
+
+	if (package_count < 2)
+		return false;
+
+	return true;
+}
+
+/**
+ * validate_topology_symmetry - Validate topology for package-aware features
+ *
+ * Validates topology symmetry by checking all memory packages have identical
+ * node structure. Updates the global topology_symmetric variable.
+ *
+ * Topology is valid only when all conditions are met:
+ * 1. At least 2 packages exist (multi-package system)
+ * 2. All packages have identical cpu_node and memory_only_node counts
+ * 3. Each package has at least 2 total nodes (cpu_node + memory_only_node >= 2)
+ *
+ * Valid configurations per package:
+ * - 1 cpu_node + 1 memory_only_node = 2 nodes (valid)
+ * - 2 cpu_nodes + 0 memory_only_node = 2 nodes (valid)
+ * - 0 cpu_node + 2 memory_only_nodes = 2 nodes (valid)
+ * - 1 cpu_node + 0 memory_only_node = 1 node (invalid - single node package)
+ */
+static void validate_topology_symmetry(void)
+{
+	guard(mutex)(&memory_package_lock);
+
+	WRITE_ONCE(topology_symmetric, __check_topology_symmetry());
+}
+
+static int __init memory_package_init(void)
+{
+	int ret = 0, nid;
+
+	for_each_online_node(nid) {
+		if (!node_state(nid, N_MEMORY))
+			continue;
+
+		ret = create_node_with_package(nid);
+		if (ret)
+			goto out;
+	}
+
+	hotplug_node_notifier(mp_hotplug_callback, MEMTIER_HOTPLUG_PRI);
+
+	validate_topology_symmetry();
+
+out:
+	return ret;
+}
+late_initcall(memory_package_init);
-- 
2.25.1

Re: [PATCH 2/4] mm/memory-tiers: introduce package-aware topology management for NUMA nodes

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 10 potential issue(s) to consider:
- [High] CPU nodes without local memory are completely excluded from the package topology.
- [High] Upgrading a provisional SLIT entry permanently unbinds and orphans the node.
- [High] TOCTOU race in `bind_node_to_package` binds a newly upgraded node to the wrong package.
- [High] `unbind_node_to_package` clears the node from the wrong nodemask if the node's state changed, permanently leaking the package.
- [High] `memory_package_lock` is acquired in a reclaim path while also being held during a GFP_KERNEL allocation, causing a deadlock.
- [Medium] Initialization loop in memory_package_init() permanently leaks kobjects and sysfs devices on failure.
- [Medium] Mixing goto-based error handling with scope-based guard() violates strict cleanup guidelines.
- [Medium] Global topology symmetry state is not recalculated when a provisional node is dropped.
- [Medium] The package notifier chain is exported but never invoked during node creation.
- [Medium] Exported API `mp_add_package_node_by_initiator` lacks bounds checking on `nid`, causing an out-of-bounds array access.
--

--- Patch [2]: [PATCH 2/4] mm/memory-tiers: introduce package-aware topology management for NUMA nodes ---
commit 0e269cf99bac40f9c00f53199f76f3781bc439c6
Author: Rakie Kim <rakie.kim@sk.com>
Date:   Thu Aug 6 08:09:33 2026 +0000

    mm/memory-tiers: introduce package-aware topology management for NUMA nodes
    
    The NUMA distance model provides only relative latency values between nodes
    and has no notion of structural grouping. Memory policies based on distance
    alone therefore cannot tell which nodes are local to the same physical
    package and which belong to a different one, which limits how well they can
    keep placement and migration within a package.
    
    Introduce a package-aware topology layer that groups NUMA nodes into a
    "memory package": a set of CPU nodes and their local memory-only nodes
    (such as CXL or HBM). A subsystem that owns a node registers a resolver to
    supply its package, and a policy can query, for any node, the other nodes
    that belong to the same package.

> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 54851d8a195b0..5932df315604a 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c

[ ... ]

> +static void mp_snapshot_rebuild(void)
> +{
> +	struct mp_snapshot *new, *old;
> +	struct memory_package *mp;
> +	int nr = 0, i = 0, nid;
> +
> +	lockdep_assert_held(&memory_package_lock);
> +
> +	list_for_each_entry(mp, &memory_packages, list)
> +		nr++;
> +
> +	new = kvzalloc(struct_size(new, pkg, nr), GFP_KERNEL);

[Severity: High]
Could this allocation trigger a deadlock?

The memory_package_lock is held here during a GFP_KERNEL allocation, which
can trigger direct reclaim. However, the same lock is acquired in
__mp_get_preferred_nodemask() during memory demotion, which runs in reclaim
context. 

If direct reclaim attempts to acquire the already held lock, will it
cause a self-deadlock?

> +	if (!new)
> +		return;

[ ... ]

> +static int bind_node_to_package(int nid)
> +{
> +	int package_id, pkg_id;
> +	struct memory_package *mp;
> +	nodemask_t nodes, cpu, mem;
> +
> +	mutex_lock(&memory_package_lock);
> +	if (!mpns[nid]) {
> +		mutex_unlock(&memory_package_lock);
> +		return -EINVAL;
> +	}
> +	package_id = mpns[nid]->package_id;
> +	mutex_unlock(&memory_package_lock);
> +
> +	mp = find_create_memory_package(package_id);
> +	if (IS_ERR(mp))
> +		return PTR_ERR(mp);
> +
> +	mutex_lock(&memory_package_lock);
> +	if (!mpns[nid]) {
> +		mutex_unlock(&memory_package_lock);
> +		return -ENOENT;
> +	}
> +	mpns[nid]->package = mp;

[Severity: High]
Is there a race condition when the lock is dropped and reacquired?

If a concurrent driver probe calls mp_add_package_node_by_initiator() while
the lock is dropped, it might destroy the old mpns[nid] and create a new one
with a different package_id.

When the lock is reacquired, mpns[nid] is not NULL, but could we end up
binding the new node (which wants the new package ID) to the old package 
struct that we just fetched?

> +	node_set(mpns[nid]->nid, mp->nodes);

[ ... ]

> +static void unbind_node_to_package(struct memory_package *mp, int nid)
> +{
> +	lockdep_assert_held(&memory_package_lock);
> +
> +	node_clear(nid, mp->nodes);
> +	if (node_state(nid, N_CPU))
> +		node_clear(nid, mp->cpu_nodes);
> +	else
> +		node_clear(nid, mp->memory_only_nodes);

[Severity: High]
Does this clear the node from the correct nodemask if its state changed?

If a memory-only node was initially added to mp->memory_only_nodes, but CPUs
were later hot-added to it, node_state(nid, N_CPU) will now evaluate to true.

During a subsequent hot-remove, this will clear the bit from mp->cpu_nodes
instead of mp->memory_only_nodes, leaving the bit permanently set and
potentially leaking the kobject.

> +
> +	if (mpns[nid])
> +		list_del(&mpns[nid]->package_entry);
> +
> +	update_package_preferred(mp);
> +}
> +
> +static struct memory_package_node *create_package_node(int nid, int initiator_nid)
> +{
> +	int cpu_nid, package_id;
> +	int source_flags;
> +	struct memory_package_node *mpn;
> +
> +	if (node_state(nid, N_CPU)) {
> +		cpu_nid = nid;
> +		source_flags = MPN_SRC_CPU;
> +	} else {
> +		if (initiator_nid >= 0) {
> +			cpu_nid = initiator_nid;
> +			source_flags = MPN_SRC_INITIATOR;
> +		} else {
> +			/*
> +			 * No driver-supplied initiator: fall back to the
> +			 * nearest CPU node (via SLIT/numa_distance).
> +			 */
> +			cpu_nid = numa_nearest_node(nid, N_CPU);
> +			source_flags = MPN_SRC_SLIT;
> +		}
> +	}

[Severity: Medium]
Are we missing a call to the exported package notifier chain here?

The register_mp_package_notifier() and mp_probe_package_id() APIs are
provided so subsystems can supply package hints, but mp_probe_package_id()
is never called in the node creation flow.

If a node is created without an explicit initiator, shouldn't we invoke the
notifier before falling back directly to numa_nearest_node()?

> +
> +	package_id = mp_node_to_package_id(cpu_nid);

[ ... ]

> +static int find_package_node(int nid, int initiator_nid)
> +{
> +	struct memory_package *unreg_mp = NULL;
> +	int ret = nid;
> +
> +	mutex_lock(&memory_package_lock);
> +	if (!mpns[nid]) {

[Severity: Medium]
Is it safe to index the mpns array with nid here?

The nid is passed directly from mp_add_package_node_by_initiator(), which
is an exported API. If a driver processes malformed or malicious firmware
tables, could an unvalidated nid cause an out-of-bounds array access?

> +		ret = NUMA_NO_NODE;
> +	} else if (mpns[nid]->source_flags == MPN_SRC_SLIT && initiator_nid >= 0) {
> +		/*
> +		 * SLIT-derived entries are provisional; if a driver later
> +		 * provides an explicit initiator, drop the provisional
> +		 * entry and rebuild with the stronger hint.
> +		 */
> +		unreg_mp = __destroy_package_node(nid);
> +		ret = NUMA_NO_NODE;
> +	}
> +	mutex_unlock(&memory_package_lock);
> +
> +	if (unreg_mp)
> +		device_unregister(&unreg_mp->dev);
> +
> +	return ret;
> +}

[Severity: Medium]
Should we recalculate the global topology symmetry state here?

When an existing provisional SLIT-derived node is dropped to make way for an
explicit driver initiator, __destroy_package_node() modifies the package node
counts. However, validate_topology_symmetry() is not called before returning.

Will the system's symmetry verdict remain stale until another completely
unrelated hotplug event triggers a recalculation?

[ ... ]

> +int mp_add_package_node_by_initiator(int nid, int initiator_nid)
> +{
> +	int ret;
> +
> +	ret = find_create_package_node(nid, initiator_nid);
> +	if (ret < 0)
> +		return ret;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(mp_add_package_node_by_initiator);

[Severity: High]
Does upgrading a provisional SLIT entry permanently drop it from the topology?

If a driver calls this function to supply an initiator for a node that was
already initialized at boot, find_create_package_node() destroys the old entry 
and creates a new mpns[nid].

However, we simply return 0 here without calling bind_node_to_package().
Since the node is already online, no hotplug event will fire to bind it, 
potentially leaving mpn->package as NULL and orphaning the node.

[ ... ]

> +static int __mp_get_preferred_nodemask(int nid, enum mp_nodes_type node_type,
> +				    nodemask_t *out)
> +{
> +	int ret = 0;
> +
> +	/* No hot-path callers: the mutex is fine here. */
> +	guard(mutex)(&memory_package_lock);
> +
> +	if (!out) {
> +		ret = -EINVAL;
> +		goto out;
> +	}

[Severity: Medium]
Does this mix goto-based error handling with scope-based guard() blocks?

The kernel's cleanup subsystem guidelines in include/linux/cleanup.h
explicitly forbid mixing goto jumps with guard() cleanup declarations in
the same function, as it can create unsafe ownership semantics.

[ ... ]

> +static int __init memory_package_init(void)
> +{
> +	int ret = 0, nid;
> +
> +	for_each_online_node(nid) {
> +		if (!node_state(nid, N_MEMORY))
> +			continue;

[Severity: High]
Are CPU nodes without local memory completely excluded from the topology?

If a CPU node evaluates to false for N_MEMORY, it will be skipped entirely.
This would cause update_package_preferred() to find an empty CPU mask for
memory-only nodes in that package, potentially breaking memory promotion by
leaving a memory-only node to set itself as its preferred promotion target.

> +
> +		ret = create_node_with_package(nid);
> +		if (ret)
> +			goto out;

[Severity: Medium]
Does this leak prior successfully initialized nodes on failure?

If an allocation or registration fails inside create_node_with_package(),
the code jumps directly to out.

Without a centralized rollback loop to individually clean up the
successfully established memory_package devices and kobjects, will this
result in a permanent unrecoverable memory leak?

> +	}
> +
> +	hotplug_node_notifier(mp_hotplug_callback, MEMTIER_HOTPLUG_PRI);
> +
> +	validate_topology_symmetry();
> +
> +out:
> +	return ret;
> +}
> +late_initcall(memory_package_init);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260806080936.421-1-rakie.kim@sk.com?part=2

[PATCH 3/4] mm/memory-tiers: register CXL nodes to memory packages via initiator

[Rakie Kim]

A CXL memory node comes online without an explicit package association, and
plain NUMA distance does not convey which physical package it belongs to.
Without that association a CXL node cannot be grouped with the CPUs that
front it.

Register a package notifier per CXL region. When the region's memory node
comes online, the notifier resolves an initiator CPU node - the NUMA node
of the first memdev backing the region - and binds the memory node to that
initiator's package. This gives the topology layer the CPU-side association
that plain NUMA distance does not carry.

The initiator nid comes from the firmware and driver description of the
region's endpoint, so the association is only as accurate as that
description; it gives a more direct host-side grouping than flat distance
values.

Signed-off-by: Rakie Kim <rakie.kim@sk.com>
---
 drivers/cxl/core/region.c | 54 +++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h         |  1 +
 drivers/dax/kmem.c        |  3 +++
 3 files changed, 58 insertions(+)

diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index e50dc716d4e8..af66e2e06c62 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -2673,6 +2673,55 @@ static int cxl_region_calculate_adistance(struct notifier_block *nb,
 	return NOTIFY_STOP;
 }
 
+/*
+ * Find a NUMA node to act as the initiator for this region: scan the
+ * region's endpoint targets and return the first one that resolves to a
+ * valid NUMA node.
+ */
+static int cxl_region_find_nearest_node(struct cxl_region *cxlr)
+{
+	struct cxl_region_params *p = &cxlr->params;
+	struct cxl_endpoint_decoder *cxled = NULL;
+	struct cxl_memdev *cxlmd = NULL;
+	int i, numa_node;
+
+	for (i = 0; i < p->nr_targets; i++) {
+		cxled = p->targets[i];
+		cxlmd = cxled_to_memdev(cxled);
+		numa_node = dev_to_node(&cxlmd->dev);
+		if (numa_node != NUMA_NO_NODE)
+			return numa_node;
+	}
+	return NUMA_NO_NODE;
+}
+
+/*
+ * Package notifier callback: when a new memory node is onlined via dax
+ * kmem, bind the node this CXL region backs to its memory package, using
+ * the nearest region target as the initiator. Notifications for other
+ * nodes are ignored.
+ */
+static int cxl_region_add_package_node(struct notifier_block *nb,
+				       unsigned long dax_nid, void *data)
+{
+	int region_nid, nearest_nid, ret;
+	struct cxl_region *cxlr = container_of(nb, struct cxl_region, package_notifier);
+
+	region_nid = phys_to_target_node(cxlr->params.res->start);
+	if (region_nid != dax_nid)
+		return NOTIFY_DONE;
+
+	nearest_nid = cxl_region_find_nearest_node(cxlr);
+	if (nearest_nid == NUMA_NO_NODE)
+		return NOTIFY_DONE;
+
+	ret = mp_add_package_node_by_initiator(dax_nid, nearest_nid);
+	if (ret)
+		return NOTIFY_DONE;
+
+	return NOTIFY_OK;
+}
+
 /**
  * devm_cxl_add_region - Adds a region to a decoder
  * @cxlrd: root decoder
@@ -3852,6 +3901,7 @@ static void shutdown_notifiers(void *_cxlr)
 
 	unregister_node_notifier(&cxlr->node_notifier);
 	unregister_mt_adistance_algorithm(&cxlr->adist_notifier);
+	unregister_mp_package_notifier(&cxlr->package_notifier);
 }
 
 static void remove_debugfs(void *dentry)
@@ -4066,6 +4116,10 @@ static int cxl_region_probe(struct device *dev)
 	cxlr->adist_notifier.priority = 100;
 	register_mt_adistance_algorithm(&cxlr->adist_notifier);
 
+	cxlr->package_notifier.notifier_call = cxl_region_add_package_node;
+	cxlr->package_notifier.priority = 100;
+	register_mp_package_notifier(&cxlr->package_notifier);
+
 	rc = devm_add_action_or_reset(&cxlr->dev, shutdown_notifiers, cxlr);
 	if (rc)
 		return rc;
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 1297594beaec..9281ca3a5f0f 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -477,6 +477,7 @@ struct cxl_region {
 	struct access_coordinate coord[ACCESS_COORDINATE_MAX];
 	struct notifier_block node_notifier;
 	struct notifier_block adist_notifier;
+	struct notifier_block package_notifier;
 };
 
 struct cxl_nvdimm_bridge {
diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index 2cc8749bc871..1de23f196354 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -94,6 +94,9 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
 	if (IS_ERR(mtype))
 		return PTR_ERR(mtype);
 
+	/* Resolve the memory package for this newly onlined kmem node. */
+	mp_probe_package_id(numa_node);
+
 	for (i = 0; i < dev_dax->nr_range; i++) {
 		struct range range;
 
-- 
2.25.1

Re: [PATCH 3/4] mm/memory-tiers: register CXL nodes to memory packages via initiator

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] `cxl_region_find_nearest_node()` incorrectly uses `p->nr_targets` as an upper bound and assumes a dense array, causing a NULL pointer dereference when accessing sparse targets.
- [High] The package notifier callback `cxl_region_add_package_node` accesses region parameters and targets without holding the required `cxl_rwsem.region` lock, allowing race conditions with concurrent region teardown.

Pre-existing issues:
- [High] Preexisting loops iterating over `p->targets` up to `p->nr_targets` fail to check for `NULL` pointers, leading to potential crashes when the array is sparse.
--

--- Patch [3]: [PATCH 3/4] mm/memory-tiers: register CXL nodes to memory packages via initiator ---
commit dff9dccb7111817b944d1f03a5ff003c16d488a5
Author: Rakie Kim <rakie.kim@sk.com>

mm/memory-tiers: register CXL nodes to memory packages via initiator

A CXL memory node comes online without an explicit package association, and
plain NUMA distance does not convey which physical package it belongs to.
Without that association a CXL node cannot be grouped with the CPUs that
front it.

Register a package notifier per CXL region. When the region's memory node
comes online, the notifier resolves an initiator CPU node - the NUMA node
of the first memdev backing the region - and binds the memory node to that
initiator's package. This gives the topology layer the CPU-side association
that plain NUMA distance does not carry.

> diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
> index e50dc716d4e82..af66e2e06c624 100644
> --- a/drivers/cxl/core/region.c
> +++ b/drivers/cxl/core/region.c
> @@ -2673,6 +2673,55 @@ static int cxl_region_calculate_adistance(struct notifier_block *nb,
>  	return NOTIFY_STOP;
>  }
>  
> +/*
> + * Find a NUMA node to act as the initiator for this region: scan the
> + * region's endpoint targets and return the first one that resolves to a
> + * valid NUMA node.
> + */
> +static int cxl_region_find_nearest_node(struct cxl_region *cxlr)
> +{
> +	struct cxl_region_params *p = &cxlr->params;
> +	struct cxl_endpoint_decoder *cxled = NULL;
> +	struct cxl_memdev *cxlmd = NULL;
> +	int i, numa_node;
> +
> +	for (i = 0; i < p->nr_targets; i++) {
> +		cxled = p->targets[i];
> +		cxlmd = cxled_to_memdev(cxled);

[Severity: High]
Does this code handle sparse target arrays correctly?

If __cxl_decoder_detach() clears elements in p->targets[] and decrements
p->nr_targets when a target is detached, the array can become sparse.
When iterating using p->nr_targets as an upper bound, could p->targets[i]
be NULL for detached targets?

Passing a NULL cxled to cxled_to_memdev() will immediately dereference
cxled->cxld.dev.parent, potentially causing a kernel panic.

[Severity: High]
This isn't a bug introduced by this patch, but the same missing NULL check
appears to be a pre-existing issue in other functions iterating p->targets.

For example, in cxl_dpa_to_hpa():

drivers/cxl/core/region.c:cxl_dpa_to_hpa() {
    ...
    for (int i = 0; i < p->nr_targets; i++) {
        if (cxlmd == cxled_to_memdev(p->targets[i])) {
    ...
}

And in region_offset_to_dpa_result():

drivers/cxl/core/region.c:region_offset_to_dpa_result() {
    ...
    for (int i = 0; i < p->nr_targets; i++) {
        cxled = p->targets[i];
        if (cxled->pos != pos)
    ...
}

Should these loops also check if p->targets[i] is NULL before dereferencing it?

> +		numa_node = dev_to_node(&cxlmd->dev);
> +		if (numa_node != NUMA_NO_NODE)
> +			return numa_node;
> +	}
> +	return NUMA_NO_NODE;
> +}
> +
> +/*
> + * Package notifier callback: when a new memory node is onlined via dax
> + * kmem, bind the node this CXL region backs to its memory package, using
> + * the nearest region target as the initiator. Notifications for other
> + * nodes are ignored.
> + */
> +static int cxl_region_add_package_node(struct notifier_block *nb,
> +				       unsigned long dax_nid, void *data)
> +{
> +	int region_nid, nearest_nid, ret;
> +	struct cxl_region *cxlr = container_of(nb, struct cxl_region, package_notifier);
> +
> +	region_nid = phys_to_target_node(cxlr->params.res->start);
> +	if (region_nid != dax_nid)
> +		return NOTIFY_DONE;
> +
> +	nearest_nid = cxl_region_find_nearest_node(cxlr);

[Severity: High]
Does this callback need to hold the cxl_rwsem.region lock when accessing
shared region parameters?

This notifier can be called asynchronously when a DAX kmem node is probed.
It accesses cxlr->params.res->start and p->targets without holding the
topology lock.

Concurrently, cxl_decoder_detach() could modify p->targets, or teardown
logic could free p->res under the cxl_rwsem.region write lock. Could this
race condition result in a use-after-free or NULL pointer dereference?

> +	if (nearest_nid == NUMA_NO_NODE)
> +		return NOTIFY_DONE;
> +
> +	ret = mp_add_package_node_by_initiator(dax_nid, nearest_nid);
> +	if (ret)
> +		return NOTIFY_DONE;
> +
> +	return NOTIFY_OK;
> +}

[ ... ]

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260806080936.421-1-rakie.kim@sk.com?part=3

[PATCH 4/4] mm/mempolicy: enhance weighted interleave with package-aware locality

[Rakie Kim]

Weighted interleave places pages on nodes in proportion to per-node weights
derived from each node's bandwidth. Within one package the weight given to
a node matches the bandwidth a task sees from it; across packages it does
not. The weights are set once from device bandwidth and applied the same
way wherever the task runs, but memory reached over the interconnect to
another package is slower than the same memory reached locally, so a node
in another package is given a weight higher than the bandwidth it can
deliver to the task. Flat weighted interleave then steers allocations onto
the interconnect even when local capacity exists, degrading effective
bandwidth.

          node0             node1
        +-------+         +-------+
        | CPU 0 |---------| CPU 1 |
        +-------+         +-------+
        | DRAM0 |         | DRAM1 |
        +---+---+         +---+---+
            |                 |
        +---+---+         +---+---+
        | CXL 0 |         | CXL 1 |
        +-------+         +-------+
          node2             node3

The numbers below are illustrative single-stream bandwidths (GB/s). Local
DRAM sustains 300 and local CXL 150; any path that crosses to another
package, over the interconnect, is capped at 100, so a node in another
package delivers 100 whether it is DRAM or CXL. Local CXL (150) is still
faster than any node in another package (100). The effective bandwidth each
CPU sees is:

              node0  node1  node2  node3
from CPU 0:    300    100    150    100
from CPU 1:    100    300    100    150

Since a single per-node weight cannot encode the interconnect penalty,
a reasonable set of global weights is taken from local device bandwidth
(local DRAM : local CXL = 300 : 150 = 2 : 1): node0=2 node1=2 node2=1
node3=1.

              node0  node1  node2  node3
global:         2      2      1      1   (same wherever the task runs)

A task on CPU 0 gives node1 - remote DRAM, effective 100 - the same weight
2 as its own local node0 at 300. Worse, node1 is weighted above node2, the
task's local CXL at effective 150, even though node2 is the faster of the
two. The flat weights rank a slower interconnect-bound node above a faster
local one, which is exactly backwards.

Make weighted interleave package-aware. When enabled, node selection is
restricted to the nodes of the task's current package, intersected with the
policy nodemask: the task's pages are spread by weight across the package's
nodes, and nodes outside the package are not part of the selection.

The only mask-level fallback is the empty-intersection case - if the policy
nodemask excludes every node of the current package, selection falls back
to the package spanned by the policy's own nodes, so a misconfiguration
never yields an empty candidate set. There is no spill to a remote package
as a placement preference.

Availability is still preferred over containment at allocation time: the
resolved mask constrains node selection only, and the page allocator is
invoked without a package nodemask, so when the selected node is exhausted
the allocation is served from another node - exactly as plain weighted
interleave already behaves - rather than forcing reclaim on the local
package.

The resolved mask is by construction a subset of the policy nodemask, which
mempolicy already restricts to the task's cpuset; package mode can only
narrow that set, never widen it, so cpusets and the task nodemask remain
authoritative.

              node0  node1  node2  node3
from CPU 0:     2      0      1      0
from CPU 1:     0      2      0      1

Tasks on CPU 0 place pages on DRAM0(2) and CXL0(1) at 2:1, which matches
their effective bandwidth of 300:150; tasks on CPU 1 place on DRAM1(2) and
CXL1(1) the same way. This aligns allocation with per-package bandwidth,
preserves NUMA locality, and keeps interleave traffic off the saturable
cross-socket interconnect.

The behavior is opt-in and off by default. A sysfs toggle at
/sys/kernel/mm/mempolicy/weighted_interleave/package_mode turns it on or
off at runtime; reading it reports the current setting.

Enabling is refused on a topology that is not symmetric (see
/sys/devices/system/package/), and if the topology stops being symmetric
while the mode is on - for example after node hotplug - weighted interleave
transparently degrades to its flat behavior while the configured value is
preserved and takes effect again once the topology is symmetric.

The following are the results with package-aware weighted interleave
applied:

System Configuration:
- Processor: Dual-Socket Intel Xeon 6980P (Granite Rapids)

1) Throughput (System Bandwidth)
   - DRAM Only: 966 GB/s
   - Weighted Interleave: 903 GB/s (7% decrease compared to DRAM Only)
   - Package-Aware Weighted Interleave: 1329 GB/s (1.33 TB/s)
     (38% increase compared to DRAM Only,
      47% increase compared to Weighted Interleave)

2) Loaded Latency (Under High Bandwidth)
   - DRAM Only: 544 ns
   - Weighted Interleave: 545 ns
   - Package-Aware Weighted Interleave: 436 ns
     (20% reduction compared to both)

Signed-off-by: Rakie Kim <rakie.kim@sk.com>
---
 ...fs-kernel-mm-mempolicy-weighted-interleave |  17 ++
 mm/mempolicy.c                                | 159 +++++++++++++++++-
 2 files changed, 172 insertions(+), 4 deletions(-)

diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
index 649c0e9b895c..d2ccba171c5e 100644
--- a/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
+++ b/Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
@@ -52,3 +52,20 @@ Description:	Auto-weighting configuration interface
 
 		Writing a new weight to a node directly via the nodeN interface
 		will also automatically switch the system to manual mode.
+
+What:		/sys/kernel/mm/mempolicy/weighted_interleave/package_mode
+Date:		August 2026
+Contact:	Linux memory management mailing list <linux-mm@kvack.org>
+Description:	Package-aware weighted interleave toggle
+
+		'true' restricts weighted interleave node selection to the
+		NUMA nodes of the package (CPU socket) the allocating task
+		is running on. 'false' (the default) uses the existing
+		weighted interleave behavior.
+
+		Enabling is rejected with -EINVAL while the package topology
+		is not symmetric.
+
+		Writing any true value string (e.g. Y or 1) enables the
+		restriction, any false value string (e.g. N or 0) disables
+		it. All other strings return -EINVAL.
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 19417b0afc30..66bccb9a0a19 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -117,6 +117,7 @@
 #include <asm/tlb.h>
 #include <linux/uaccess.h>
 #include <linux/memory.h>
+#include <linux/memory-tiers.h>
 
 #include "internal.h"
 
@@ -167,6 +168,8 @@ static unsigned int *node_bw_table;
  */
 static DEFINE_MUTEX(wi_state_lock);
 
+static bool package_mode_enabled;
+
 static u8 get_il_weight(int node)
 {
 	struct weighted_interleave_state *state;
@@ -180,6 +183,11 @@ static u8 get_il_weight(int node)
 	return weight;
 }
 
+static bool wi_package_mode_enabled(void)
+{
+	return READ_ONCE(package_mode_enabled) && mp_is_topology_symmetric();
+}
+
 /*
  * Convert bandwidth values into weighted interleave weights.
  * Call with wi_state_lock.
@@ -2138,17 +2146,97 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
 	return zone >= dynamic_policy_zone;
 }
 
+/**
+ * policy_resolve_package_nodes - Restrict policy nodes to the current package
+ * @policy: Target mempolicy whose user-selected nodes are in @policy->nodes.
+ * @mask:   Output nodemask. On success, contains policy->nodes limited to
+ *          the package that should be used for the allocation.
+ *
+ * This helper combines two constraints to decide where within a package
+ * memory may be allocated:
+ *
+ *   1) The caller's package: derived via mp_get_package_nodes(numa_node_id()).
+ *   2) The user's preselected set @policy->nodes (cpusets/mempolicy).
+ *
+ * The function obtains the nodemask of the current CPU's package and
+ * intersects it with @policy->nodes. If the intersection is empty (e.g. the
+ * user excluded every node of the current package), it falls back to the
+ * node in @policy->nodes, derives that node's package, and intersects
+ * again. If the fallback also yields an empty set, @mask stays empty and a
+ * non-zero error is returned.
+ *
+ * Examples (packages: P0={CPU:0, MEM:2}, P1={CPU:1, MEM:3}):
+ *   - policy->nodes = {0,1,2,3}
+ *       on P0: mask = {0,2}; on P1: mask = {1,3}.
+ *   - policy->nodes = {0,1,3}
+ *       on P0: mask = {0}      (only node 0 from P0 is allowed).
+ *   - policy->nodes = {1,2,3}
+ *       on P0: mask = {2}      (only node 2 from P0 is allowed).
+ *   - policy->nodes = {1,3}
+ *       on P0: current package (P0) & policy = NULL -> fallback to policy=1,
+ *               package(1)=P1, mask = {1,3}. (User effectively opted out of P0.)
+ *
+ * If the selected node is low on memory, the allocation may use another node.
+ *
+ * Return:
+ *   0 on success with @mask set as above;
+ *   -EINVAL if @policy/@mask is NULL;
+ *   -ENOENT if even the fallback intersection is empty;
+ *   Propagated error from mp_get_package_nodes() on failure.
+ */
+static int policy_resolve_package_nodes(struct mempolicy *policy, nodemask_t *mask)
+{
+	nodemask_t package_mask;
+	int node, ret;
+
+	if (!policy || !mask)
+		return -EINVAL;
+
+	nodes_clear(*mask);
+
+	node = numa_node_id();
+	ret = mp_get_package_nodes(node, &package_mask);
+	if (ret)
+		return ret;
+
+	nodes_and(*mask, package_mask, policy->nodes);
+	if (!nodes_empty(*mask))
+		return 0;
+
+	/*
+	 * The user's nodemask excludes every node of the current package;
+	 * fall back to the package spanned by the user's own first node.
+	 */
+	node = first_node(policy->nodes);
+	ret = mp_get_package_nodes(node, &package_mask);
+	if (ret)
+		return ret;
+
+	nodes_and(*mask, package_mask, policy->nodes);
+	if (nodes_empty(*mask))
+		return -ENOENT;
+
+	return 0;
+}
+
 static unsigned int weighted_interleave_nodes(struct mempolicy *policy)
 {
 	unsigned int node;
 	unsigned int cpuset_mems_cookie;
+	nodemask_t mask;
 
 retry:
 	/* to prevent miscount use tsk->mems_allowed_seq to detect rebind */
 	cpuset_mems_cookie = read_mems_allowed_begin();
 	node = current->il_prev;
-	if (!current->il_weight || !node_isset(node, policy->nodes)) {
-		node = next_node_in(node, policy->nodes);
+
+	/* Package mode off or unresolved: fall back to the full policy nodemask. */
+	if (!wi_package_mode_enabled() ||
+	    policy_resolve_package_nodes(policy, &mask))
+		mask = policy->nodes;
+
+	if (!current->il_weight || !node_isset(node, mask)) {
+		node = next_node_in(node, mask);
 		if (read_mems_allowed_retry(cpuset_mems_cookie))
 			goto retry;
 		if (node == MAX_NUMNODES)
@@ -2241,6 +2329,30 @@ static unsigned int read_once_policy_nodemask(struct mempolicy *pol,
 	return nodes_weight(*mask);
 }
 
+/*
+ * Package-aware counterpart of read_once_policy_nodemask(): resolve the
+ * current package's nodes intersected with the policy, falling back to the
+ * full policy nodemask when package mode is off or resolution fails.
+ */
+static unsigned int read_once_policy_package_nodemask(struct mempolicy *pol,
+						      nodemask_t *mask)
+{
+	nodemask_t package_mask;
+
+	barrier();
+	if (!wi_package_mode_enabled()) {
+		memcpy(mask, &pol->nodes, sizeof(nodemask_t));
+		return nodes_weight(*mask);
+	}
+	if (policy_resolve_package_nodes(pol, &package_mask))
+		memcpy(mask, &pol->nodes, sizeof(nodemask_t));
+	else
+		memcpy(mask, &package_mask, sizeof(nodemask_t));
+	barrier();
+
+	return nodes_weight(*mask);
+}
+
 static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
 {
 	struct weighted_interleave_state *state;
@@ -2251,7 +2363,7 @@ static unsigned int weighted_interleave_nid(struct mempolicy *pol, pgoff_t ilx)
 	u8 weight;
 	int nid = 0;
 
-	nr_nodes = read_once_policy_nodemask(pol, &nodemask);
+	nr_nodes = read_once_policy_package_nodemask(pol, &nodemask);
 	if (!nr_nodes)
 		return numa_node_id();
 
@@ -2695,7 +2807,7 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
 	/* read the nodes onto the stack, retry if done during rebind */
 	do {
 		cpuset_mems_cookie = read_mems_allowed_begin();
-		nnodes = read_once_policy_nodemask(pol, &nodes);
+		nnodes = read_once_policy_package_nodemask(pol, &nodes);
 	} while (read_mems_allowed_retry(cpuset_mems_cookie));
 
 	/* if the nodemask has become invalid, we cannot do anything */
@@ -3835,7 +3947,42 @@ static struct kobj_attribute wi_auto_attr = {
 	.store = weighted_interleave_auto_store,
 };
 
+static ssize_t package_mode_show(struct kobject *kobj,
+		struct kobj_attribute *attr, char *buf)
+{
+	return sysfs_emit(buf, "%s\n", str_true_false(READ_ONCE(package_mode_enabled)));
+}
+
+static ssize_t package_mode_store(struct kobject *kobj,
+		struct kobj_attribute *attr, const char *buf, size_t count)
+{
+	bool input;
+	int err;
+
+	err = kstrtobool(buf, &input);
+	if (err)
+		return err;
+
+	/*
+	 * Disable package-aware weighted interleave on non-symmetric topologies.
+	 * Non-symmetric topology (e.g., asymmetric CXL memory attachment) can
+	 * cause performance degradation if package-aware allocation is used.
+	 * Reject enable request if topology is not symmetric.
+	 */
+	if (input && !mp_is_topology_symmetric()) {
+		pr_warn("package_mode cannot be enabled on non-symmetric topology\n");
+		return -EINVAL;
+	}
+
+	WRITE_ONCE(package_mode_enabled, input);
+	return count;
+}
+
+static struct kobj_attribute wi_package_mode_attr =
+	__ATTR(package_mode, 0664, package_mode_show, package_mode_store);
+
 static void wi_cleanup(void) {
+	sysfs_remove_file(&wi_group->wi_kobj, &wi_package_mode_attr.attr);
 	sysfs_remove_file(&wi_group->wi_kobj, &wi_auto_attr.attr);
 	sysfs_wi_node_delete_all();
 	wi_state_free();
@@ -3941,6 +4088,10 @@ static int __init add_weighted_interleave_group(struct kobject *mempolicy_kobj)
 	if (err)
 		goto err_put_kobj;
 
+	err = sysfs_create_file(&wi_group->wi_kobj, &wi_package_mode_attr.attr);
+	if (err)
+		goto err_cleanup_kobj;
+
 	for_each_online_node(nid) {
 		if (!node_state(nid, N_MEMORY))
 			continue;
-- 
2.25.1

Re: [PATCH 4/4] mm/mempolicy: enhance weighted interleave with package-aware locality

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Package-aware weighted interleave silently degrades to flat mode on memoryless CPU nodes because they are not registered in the package topology.
- [Medium] Unstabilized TOCTOU access of `policy->nodes` in `policy_resolve_package_nodes()` can cause torn reads and inconsistent node intersection if the policy is concurrently modified.

Pre-existing issues:
- [High] This problem wasn't introduced by this patch, but the function `alloc_pages_bulk_weighted_interleave` unconditionally uses `GFP_KERNEL` for a `kzalloc` call, which can cause a sleep-in-atomic bug if invoked with a non-blocking `gfp` mask.
--

--- Patch [4]: [PATCH 4/4] mm/mempolicy: enhance weighted interleave with package-aware locality ---
commit 9bf854e133ba0fbe547595b9d47a304b6f619169
Author: Rakie Kim <rakie.kim@sk.com>

mm/mempolicy: enhance weighted interleave with package-aware locality

This commit introduces a package-aware mode for weighted interleave, aiming
to prioritize local nodes within the same package over remote nodes to
preserve bandwidth. The mode is opt-in via sysfs and restricts allocations
to the NUMA nodes of the task's current package.

> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 19417b0afc30b..66bccb9a0a191 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c

[ ... ]

> @@ -2138,17 +2146,97 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
>  	return zone >= dynamic_policy_zone;
>  }
>  
> +/**
> + * policy_resolve_package_nodes - Restrict policy nodes to the current package
[ ... ]
> + */
> +static int policy_resolve_package_nodes(struct mempolicy *policy, nodemask_t *mask)
> +{
> +	nodemask_t package_mask;
> +	int node, ret;
> +
> +	if (!policy || !mask)
> +		return -EINVAL;
> +
> +	nodes_clear(*mask);
> +
> +	node = numa_node_id();
> +	ret = mp_get_package_nodes(node, &package_mask);

[Severity: High]
Will mp_get_package_nodes() fail and return an error for memoryless CPU
nodes?

Since memoryless CPU nodes are excluded from being registered into memory
packages by memory_package_init() because they lack N_MEMORY, it appears
this lookup will fail. Does this cause the package-aware interleave feature
to silently degrade and fall back to the unconstrained flat mode when a task
executes on a CPU node with no local memory (such as a compute die accessing
separate CXL memory)?

> +	if (ret)
> +		return ret;
> +
> +	nodes_and(*mask, package_mask, policy->nodes);

[Severity: Medium]
Is it safe to directly read policy->nodes multiple times here without first
stabilizing it into a local copy?

If policy->nodes is concurrently modified (for example, via sys_mbind() on a
shared VMA policy), a modification occurring between this first nodes_and()
check and the fallback first_node() or nodes_and() calls below might cause
the logic to observe mixed states.

Could this result in an inconsistent node intersection or an incorrect
fallback node being selected?

> +	if (!nodes_empty(*mask))
> +		return 0;
> +
> +	/*
> +	 * The user's nodemask excludes every node of the current package;
> +	 * fall back to the package spanned by the user's own first node.
> +	 */
> +	node = first_node(policy->nodes);
> +	ret = mp_get_package_nodes(node, &package_mask);
> +	if (ret)
> +		return ret;
> +
> +	nodes_and(*mask, package_mask, policy->nodes);
> +	if (nodes_empty(*mask))
> +		return -ENOENT;
> +
> +	return 0;
> +}

[ ... ]

> @@ -2695,7 +2807,7 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
>  	/* read the nodes onto the stack, retry if done during rebind */
>  	do {
>  		cpuset_mems_cookie = read_mems_allowed_begin();
> -		nnodes = read_once_policy_nodemask(pol, &nodes);
> +		nnodes = read_once_policy_package_nodemask(pol, &nodes);
>  	} while (read_mems_allowed_retry(cpuset_mems_cookie));
>  
>  	/* if the nodemask has become invalid, we cannot do anything */

[Severity: High]
This isn't a bug introduced by this patch, but does this function safely
handle atomic allocations?

Further down in alloc_pages_bulk_weighted_interleave(), the weights array
is allocated unconditionally with GFP_KERNEL:

    weights = kzalloc(nr_node_ids, GFP_KERNEL);
    if (!weights)
            return total_allocated;

If alloc_pages_bulk() is called from an atomic context with a non-blocking
mask like GFP_ATOMIC or GFP_NOWAIT, will ignoring the passed gfp argument and
using GFP_KERNEL here cause a 'scheduling while atomic' panic or deadlock?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260806080936.421-1-rakie.kim@sk.com?part=4

Re: [PATCH 0/4] mm/mempolicy: introduce package-aware weighted interleave

[Andrew Morton]

On Thu,  6 Aug 2026 17:09:31 +0900 Rakie Kim <rakie.kim@sk.com> wrote:

> Package-aware weighted interleave places a task's weighted-interleave
> pages on the NUMA nodes of its local package, so that interleave traffic
> does not have to cross the interconnect to another package. This keeps
> each node's weight aligned with the bandwidth the task actually gets
> from it, so effective bandwidth holds up on a system that has more than
> one package. (A package is a CPU socket together with the memory
> attached to it.)

"package" is not a familiar term in MM.  It would be helpful if the
[0/N] were to carefully and fully define/describe the new term before
using it 40 times!

> Measured results:
> 
> System Configuration:
> - Processor: Dual-Socket Intel Xeon 6980P (Granite Rapids)
> 
> 1) Throughput (System Bandwidth)
>    - DRAM Only: 966 GB/s
>    - Weighted Interleave: 903 GB/s (7% decrease compared to DRAM Only)
>    - Package-Aware Weighted Interleave: 1329 GB/s (1.33 TB/s)
>      (38% increase compared to DRAM Only,
>       47% increase compared to Weighted Interleave)
> 
> 2) Loaded Latency (Under High Bandwidth)
>    - DRAM Only: 544 ns
>    - Weighted Interleave: 545 ns
>    - Package-Aware Weighted Interleave: 436 ns
>      (20% reduction compared to both)

Well that sounds nice.

>  .../ABI/testing/sysfs-devices-system-package  |   35 +
>  ...fs-kernel-mm-mempolicy-weighted-interleave |   17 +
>  drivers/cxl/core/region.c                     |   54 +
>  drivers/cxl/cxl.h                             |    1 +
>  drivers/dax/kmem.c                            |    3 +
>  include/linux/memory-tiers.h                  |  113 ++
>  include/linux/numa.h                          |   11 +
>  mm/memory-tiers.c                             | 1009 +++++++++++++++++
>  mm/mempolicy.c                                |  200 +++-

Are some user-facing Documentation/ updates appropriate?

The Documentation/ABI things are rather dry and information-free.  How
about some documentation for the operator who is wondering "should I
use this and if so why and how"?


The runtime sysfs on/off tunable is interesting.  I hear from google
operations people that every new feature should have such an "off"
switch so that if development send them a new thing and they think it's
problematic, they can disable it in order to quickly get back to the
old regime.  Perhaps that was your motivation, perhaps not.  Can you
please describe?


I see you've been emailed the Sashiko report, which appears substantial.
	https://sashiko.dev/#/patchset/20260806080936.421-1-rakie.kim@sk.com

Re: [PATCH 0/4] mm/mempolicy: introduce package-aware weighted interleave

[Rakie Kim]

On Thu, 6 Aug 2026 14:38:39 -0700 Andrew Morton <akpm@linux-foundation.org> wrote:
> On Thu,  6 Aug 2026 17:09:31 +0900 Rakie Kim <rakie.kim@sk.com> wrote:
>

Hello Andrew,

Thank you for taking the time to review this series.

> > Package-aware weighted interleave places a task's weighted-interleave
> > pages on the NUMA nodes of its local package, so that interleave traffic
> > does not have to cross the interconnect to another package. This keeps
> > each node's weight aligned with the bandwidth the task actually gets
> > from it, so effective bandwidth holds up on a system that has more than
> > one package. (A package is a CPU socket together with the memory
> > attached to it.)
>
> "package" is not a familiar term in MM.  It would be helpful if the
> [0/N] were to carefully and fully define/describe the new term before
> using it 40 times!
>

You are right. I think my explanation was not sufficient. When I
prepared this series, I went back and forth between "socket" and
other candidate terms, and settled on "package" because modern
processors can contain multiple NUMA nodes and dies within a single
physical socket, so "socket" felt misleading. I did not explain this
reasoning in the cover letter. In the next version, I will define
the term at the top of the cover letter before it is used.

> > Measured results:
> >
> > System Configuration:
> > - Processor: Dual-Socket Intel Xeon 6980P (Granite Rapids)
> >
> > 1) Throughput (System Bandwidth)
> >    - DRAM Only: 966 GB/s
> >    - Weighted Interleave: 903 GB/s (7% decrease compared to DRAM Only)
> >    - Package-Aware Weighted Interleave: 1329 GB/s (1.33 TB/s)
> >      (38% increase compared to DRAM Only,
> >       47% increase compared to Weighted Interleave)
> >
> > 2) Loaded Latency (Under High Bandwidth)
> >    - DRAM Only: 544 ns
> >    - Weighted Interleave: 545 ns
> >    - Package-Aware Weighted Interleave: 436 ns
> >      (20% reduction compared to both)
>
> Well that sounds nice.
>

Thank you.

> >  .../ABI/testing/sysfs-devices-system-package  |   35 +
> >  ...fs-kernel-mm-mempolicy-weighted-interleave |   17 +
> >  drivers/cxl/core/region.c                     |   54 +
> >  drivers/cxl/cxl.h                             |    1 +
> >  drivers/dax/kmem.c                            |    3 +
> >  include/linux/memory-tiers.h                  |  113 ++
> >  include/linux/numa.h                          |   11 +
> >  mm/memory-tiers.c                             | 1009 +++++++++++++++++
> >  mm/mempolicy.c                                |  200 +++-
>
> Are some user-facing Documentation/ updates appropriate?
>
> The Documentation/ABI things are rather dry and information-free.  How
> about some documentation for the operator who is wondering "should I
> use this and if so why and how"?
>

I agree with your point. The current ABI entries only describe the
sysfs files themselves. In the next version, I will strengthen the
documentation content so that it answers exactly those questions for
an operator: whether this feature fits their system, why it helps,
and how to enable and verify it.

>
> The runtime sysfs on/off tunable is interesting.  I hear from google
> operations people that every new feature should have such an "off"
> switch so that if development send them a new thing and they think it's
> problematic, they can disable it in order to quickly get back to the
> old regime.  Perhaps that was your motivation, perhaps not.  Can you
> please describe?
>

You are right that this was one of the purposes. The switch was
provided with two goals in mind. First, as you describe, it is an
"off" switch for the new feature: if it behaves unexpectedly in
production, the operator can disable it at runtime and immediately
return to the old behavior, without a reboot. Second, it is for
users who want to keep using the existing weighted interleave as it
is: the feature is off by default, and nothing changes for them
unless they explicitly turn it on. I will describe this motivation
in the documentation as well.

>
> I see you've been emailed the Sashiko report, which appears substantial.
> 	https://sashiko.dev/#/patchset/20260806080936.421-1-rakie.kim@sk.com
>

Yes, I have received the report, and it seems to provide a lot of
good information. I am reviewing each finding against the code, and
I plan to reflect it in the next version as much as possible.

Thanks again for your time and review.

Rakie Kim