* [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
@ 2026-07-20 19:33 ` Gregory Price
2026-07-21 5:46 ` Balbir Singh
2026-07-20 19:33 ` [PATCH v5 02/36] mm/page_alloc: refactor build_node_zonelist() out of build_zonelists() Gregory Price
` (37 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
find_next_best_node() picks the next-closest node for a fallback list
from the full N_MEMORY set. Refactor it into find_next_best_node_in(),
which takes an explicit candidates nodemask.
This enables building fallback lists with non-N_MEMORY candidates.
No functional change: every caller still selects from N_MEMORY.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/internal.h | 6 ++++--
mm/memory-tiers.c | 7 ++++---
mm/page_alloc.c | 13 ++++++++-----
3 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index f26423de4ca28..96d78a7778e88 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1103,7 +1103,8 @@ extern int node_reclaim_mode;
extern unsigned long node_reclaim(struct pglist_data *pgdat,
gfp_t gfp_mask, unsigned int order);
-extern int find_next_best_node(int node, nodemask_t *used_node_mask);
+extern int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates);
#else
#define node_reclaim_mode 0
@@ -1112,7 +1113,8 @@ static inline unsigned long node_reclaim(struct pglist_data *pgdat,
{
return 0;
}
-static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
+static inline int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates)
{
return NUMA_NO_NODE;
}
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b0..25e121851b586 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -370,7 +370,7 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
* closest demotion target.
*/
nodes_complement(mask, *allowed_mask);
- return find_next_best_node(node, &mask);
+ return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
}
static void disable_all_demotion_targets(void)
@@ -450,7 +450,7 @@ static void establish_demotion_targets(void)
memtier = list_next_entry(memtier, list);
tier_nodes = get_memtier_nodemask(memtier);
/*
- * find_next_best_node, use 'used' nodemask as a skip list.
+ * find_next_best_node_in, use 'used' nodemask as a skip list.
* Add all memory nodes except the selected memory tier
* nodelist to skip list so that we find the best node from the
* memtier nodelist.
@@ -463,7 +463,8 @@ static void establish_demotion_targets(void)
* in the preferred mask when allocating pages during demotion.
*/
do {
- target = find_next_best_node(node, &tier_nodes);
+ target = find_next_best_node_in(node, &tier_nodes,
+ &node_states[N_MEMORY]);
if (target == NUMA_NO_NODE)
break;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f93a6bb9a872d..acf096f525f49 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5743,9 +5743,10 @@ static int numa_zonelist_order_handler(const struct ctl_table *table, int write,
static int node_load[MAX_NUMNODES];
/**
- * find_next_best_node - find the next node that should appear in a given node's fallback list
+ * find_next_best_node_in - find the next node that should appear in a given node's fallback list
* @node: node whose fallback list we're appending
* @used_node_mask: nodemask_t of already used nodes
+ * @candidates: nodemask_t of nodes eligible for selection
*
* We use a number of factors to determine which is the next node that should
* appear on a given node's fallback list. The node should not have appeared
@@ -5757,7 +5758,8 @@ static int node_load[MAX_NUMNODES];
*
* Return: node id of the found node or %NUMA_NO_NODE if no node is found.
*/
-int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates)
{
int n, val;
int min_val = INT_MAX;
@@ -5767,12 +5769,12 @@ int find_next_best_node(int node, nodemask_t *used_node_mask)
* Use the local node if we haven't already, but for memoryless local
* node, we should skip it and fall back to other nodes.
*/
- if (!node_isset(node, *used_node_mask) && node_state(node, N_MEMORY)) {
+ if (!node_isset(node, *used_node_mask) && node_isset(node, *candidates)) {
node_set(node, *used_node_mask);
return node;
}
- for_each_node_state(n, N_MEMORY) {
+ for_each_node_mask(n, *candidates) {
/* Don't want a node to appear more than once */
if (node_isset(n, *used_node_mask))
@@ -5857,7 +5859,8 @@ static void build_zonelists(pg_data_t *pgdat)
prev_node = local_node;
memset(node_order, 0, sizeof(node_order));
- while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
+ while ((node = find_next_best_node_in(local_node, &used_mask,
+ &node_states[N_MEMORY])) >= 0) {
/*
* We don't want to pressure a particular node.
* So adding penalty to the first node in same
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in
2026-07-20 19:33 ` [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in Gregory Price
@ 2026-07-21 5:46 ` Balbir Singh
0 siblings, 0 replies; 82+ messages in thread
From: Balbir Singh @ 2026-07-21 5:46 UTC (permalink / raw)
To: Gregory Price, linux-mm
Cc: Zhigang.Luo, arun.george, brendan.jackman, yuzenghui, apopple,
alucerop, matthew.brost, akpm, david, ljs, liam, vbabka, rppt,
surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On 7/21/26 5:33 AM, Gregory Price wrote:
> find_next_best_node() picks the next-closest node for a fallback list
> from the full N_MEMORY set. Refactor it into find_next_best_node_in(),
> which takes an explicit candidates nodemask.
>
> This enables building fallback lists with non-N_MEMORY candidates.
>
> No functional change: every caller still selects from N_MEMORY.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> mm/internal.h | 6 ++++--
> mm/memory-tiers.c | 7 ++++---
> mm/page_alloc.c | 13 ++++++++-----
> 3 files changed, 16 insertions(+), 10 deletions(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index f26423de4ca28..96d78a7778e88 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -1103,7 +1103,8 @@ extern int node_reclaim_mode;
>
> extern unsigned long node_reclaim(struct pglist_data *pgdat,
> gfp_t gfp_mask, unsigned int order);
> -extern int find_next_best_node(int node, nodemask_t *used_node_mask);
> +extern int find_next_best_node_in(int node, nodemask_t *used_node_mask,
> + const nodemask_t *candidates);
> #else
> #define node_reclaim_mode 0
>
> @@ -1112,7 +1113,8 @@ static inline unsigned long node_reclaim(struct pglist_data *pgdat,
> {
> return 0;
> }
> -static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
> +static inline int find_next_best_node_in(int node, nodemask_t *used_node_mask,
> + const nodemask_t *candidates)
> {
> return NUMA_NO_NODE;
> }
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 54851d8a195b0..25e121851b586 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -370,7 +370,7 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
> * closest demotion target.
> */
> nodes_complement(mask, *allowed_mask);
> - return find_next_best_node(node, &mask);
> + return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
> }
>
> static void disable_all_demotion_targets(void)
> @@ -450,7 +450,7 @@ static void establish_demotion_targets(void)
> memtier = list_next_entry(memtier, list);
> tier_nodes = get_memtier_nodemask(memtier);
> /*
> - * find_next_best_node, use 'used' nodemask as a skip list.
> + * find_next_best_node_in, use 'used' nodemask as a skip list.
> * Add all memory nodes except the selected memory tier
> * nodelist to skip list so that we find the best node from the
> * memtier nodelist.
> @@ -463,7 +463,8 @@ static void establish_demotion_targets(void)
> * in the preferred mask when allocating pages during demotion.
> */
> do {
> - target = find_next_best_node(node, &tier_nodes);
> + target = find_next_best_node_in(node, &tier_nodes,
> + &node_states[N_MEMORY]);
> if (target == NUMA_NO_NODE)
> break;
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index f93a6bb9a872d..acf096f525f49 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -5743,9 +5743,10 @@ static int numa_zonelist_order_handler(const struct ctl_table *table, int write,
> static int node_load[MAX_NUMNODES];
>
> /**
> - * find_next_best_node - find the next node that should appear in a given node's fallback list
> + * find_next_best_node_in - find the next node that should appear in a given node's fallback list
> * @node: node whose fallback list we're appending
> * @used_node_mask: nodemask_t of already used nodes
> + * @candidates: nodemask_t of nodes eligible for selection
> *
> * We use a number of factors to determine which is the next node that should
> * appear on a given node's fallback list. The node should not have appeared
> @@ -5757,7 +5758,8 @@ static int node_load[MAX_NUMNODES];
> *
> * Return: node id of the found node or %NUMA_NO_NODE if no node is found.
> */
> -int find_next_best_node(int node, nodemask_t *used_node_mask)
> +int find_next_best_node_in(int node, nodemask_t *used_node_mask,
> + const nodemask_t *candidates)
> {
> int n, val;
> int min_val = INT_MAX;
> @@ -5767,12 +5769,12 @@ int find_next_best_node(int node, nodemask_t *used_node_mask)
> * Use the local node if we haven't already, but for memoryless local
> * node, we should skip it and fall back to other nodes.
> */
> - if (!node_isset(node, *used_node_mask) && node_state(node, N_MEMORY)) {
> + if (!node_isset(node, *used_node_mask) && node_isset(node, *candidates)) {
> node_set(node, *used_node_mask);
> return node;
> }
>
> - for_each_node_state(n, N_MEMORY) {
> + for_each_node_mask(n, *candidates) {
>
> /* Don't want a node to appear more than once */
> if (node_isset(n, *used_node_mask))
> @@ -5857,7 +5859,8 @@ static void build_zonelists(pg_data_t *pgdat)
> prev_node = local_node;
>
> memset(node_order, 0, sizeof(node_order));
> - while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
> + while ((node = find_next_best_node_in(local_node, &used_mask,
> + &node_states[N_MEMORY])) >= 0) {
> /*
> * We don't want to pressure a particular node.
> * So adding penalty to the first node in same
Looks good!
Acked-by: Balbir Singh <balbirs@nvidia.com>
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 02/36] mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
2026-07-20 19:33 ` [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in Gregory Price
@ 2026-07-20 19:33 ` Gregory Price
2026-07-20 19:33 ` [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags Gregory Price
` (36 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Extract per-node fallback-list construction into build_node_zonelist().
This lets use build zonelists from candidate nodemasks instead of just
the default N_MEMORY node state list.
No functional change: build_zonelists() builds the same FALLBACK list over
N_MEMORY with node_load updates as before.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/page_alloc.c | 44 ++++++++++++++++++++++++++++++--------------
1 file changed, 30 insertions(+), 14 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index acf096f525f49..4c1ef84c02a3d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5813,12 +5813,12 @@ int find_next_best_node_in(int node, nodemask_t *used_node_mask,
* DMA zone, if any--but risks exhausting DMA zone.
*/
static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
- unsigned nr_nodes)
+ unsigned int nr_nodes, int zlidx)
{
struct zoneref *zonerefs;
int i;
- zonerefs = pgdat->node_zonelists[ZONELIST_FALLBACK]._zonerefs;
+ zonerefs = pgdat->node_zonelists[zlidx]._zonerefs;
for (i = 0; i < nr_nodes; i++) {
int nr_zones;
@@ -5847,26 +5847,28 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)
zonerefs->zone_idx = 0;
}
-static void build_zonelists(pg_data_t *pgdat)
+/*
+ * Build one node-ordered fallback list from a candidate nodemask.
+ * update_load round-robins node_load across equidistant nodes.
+ */
+static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
+ int zlidx, bool update_load,
+ int *node_order, int *nr)
{
- static int node_order[MAX_NUMNODES];
- int node, nr_nodes = 0;
nodemask_t used_mask = NODE_MASK_NONE;
- int local_node, prev_node;
-
- /* NUMA-aware ordering of nodes */
- local_node = pgdat->node_id;
- prev_node = local_node;
+ int local_node = pgdat->node_id;
+ int prev_node = local_node;
+ int node, nr_nodes = 0;
- memset(node_order, 0, sizeof(node_order));
while ((node = find_next_best_node_in(local_node, &used_mask,
- &node_states[N_MEMORY])) >= 0) {
+ candidates)) >= 0) {
/*
* We don't want to pressure a particular node.
* So adding penalty to the first node in same
* distance group to make it round-robin.
*/
- if (node_distance(local_node, node) !=
+ if (update_load &&
+ node_distance(local_node, node) !=
node_distance(local_node, prev_node))
node_load[node] += 1;
@@ -5874,8 +5876,22 @@ static void build_zonelists(pg_data_t *pgdat)
prev_node = node;
}
- build_zonelists_in_node_order(pgdat, node_order, nr_nodes);
+ build_zonelists_in_node_order(pgdat, node_order, nr_nodes, zlidx);
+ *nr = nr_nodes;
+}
+
+static void build_zonelists(pg_data_t *pgdat)
+{
+ static int node_order[MAX_NUMNODES];
+ int local_node = pgdat->node_id;
+ int node, nr_nodes = 0;
+
+ memset(node_order, 0, sizeof(node_order));
+
+ build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
+ true, node_order, &nr_nodes);
build_thisnode_zonelists(pgdat);
+
pr_info("Fallback order for Node %d: ", local_node);
for (node = 0; node < nr_nodes; node++)
pr_cont("%d ", node_order[node]);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
2026-07-20 19:33 ` [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in Gregory Price
2026-07-20 19:33 ` [PATCH v5 02/36] mm/page_alloc: refactor build_node_zonelist() out of build_zonelists() Gregory Price
@ 2026-07-20 19:33 ` Gregory Price
2026-07-21 18:26 ` sashiko-bot
2026-07-20 19:33 ` [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE Gregory Price
` (35 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
__alloc_pages_noprof() takes an explicit alloc_flags, but the bulk and
folio entry points do not, so callers cannot select allocator behaviour
(e.g. an alternate zonelist) through them.
Thread alloc_flags through both, matching __alloc_pages_noprof(), and
keep the flag-carrying primitives mm-internal (page_alloc.h) so the
public gfp.h wrappers stay flag-free:
- add __alloc_pages_bulk_noprof(gfp, alloc_flags, ...) in page_alloc.h
alloc_pages_bulk_noprof() becomes a wrapper passing ALLOC_DEFAULT
- give __folio_alloc_noprof() an alloc_flags parameter and moves
__folio_alloc_node_noprof() moves into page_alloc.c
__folio_alloc_noprof() is no longer exported
No functional change: every caller passes ALLOC_DEFAULT.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/gfp.h | 13 +------------
mm/khugepaged.c | 2 +-
mm/migrate.c | 2 +-
mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++---------------
mm/page_alloc.h | 8 ++++++++
5 files changed, 42 insertions(+), 29 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 872bc53f32ec8..a327e58c313f8 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }
static inline void arch_alloc_page(struct page *page, int order) { }
#endif
-struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
- nodemask_t *nodemask);
-#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
-
unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
struct page **page_array);
@@ -252,14 +248,7 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)
dump_stack();
}
-static inline
-struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
-{
- warn_if_node_offline(nid, gfp);
-
- return __folio_alloc_noprof(gfp, order, nid, NULL);
-}
-
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid);
#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__))
/*
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 27e8f3077e80f..89ce6bcbc376b 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1243,7 +1243,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
int node = collapse_find_target_node(cc);
struct folio *folio;
- folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
+ folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
if (!folio) {
*foliop = NULL;
if (is_pmd_order(order))
diff --git a/mm/migrate.c b/mm/migrate.c
index 60355be9f652c..a8a86141dbb54 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2231,7 +2231,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
gfp_mask |= __GFP_HIGHMEM;
- return __folio_alloc(gfp_mask, order, nid, mtc->nmask);
+ return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
}
#ifdef CONFIG_NUMA_MIGRATION
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4c1ef84c02a3d..aa35bbe5d4c3e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5103,8 +5103,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
* @page_array were set to %NULL on entry, the slots from 0 to the return value
* - 1 will be filled.
*/
-unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
- nodemask_t *nodemask, int nr_pages,
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+ int preferred_nid, nodemask_t *nodemask, int nr_pages,
struct page **page_array)
{
struct page *page;
@@ -5112,8 +5112,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct zoneref *z;
struct per_cpu_pages *pcp;
struct list_head *pcp_list;
- struct alloc_context ac;
- unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ struct alloc_context ac = { .alloc_flags = alloc_flags };
+ unsigned int fastpath_alloc_flags = alloc_flags | ALLOC_WMARK_LOW;
int nr_populated = 0, nr_account = 0;
/*
@@ -5153,7 +5153,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
/* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
gfp &= gfp_allowed_mask;
- if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
+ if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &fastpath_alloc_flags))
goto out;
/* Find an allowed local zone that meets the low watermark. */
@@ -5161,7 +5161,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
for_next_zone_zonelist_nodemask(zone, z, ac.highest_zoneidx, ac.nodemask) {
unsigned long mark;
- if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
+ if (cpusets_enabled() && (fastpath_alloc_flags & ALLOC_CPUSET) &&
!__cpuset_zone_allowed(zone, gfp)) {
continue;
}
@@ -5171,16 +5171,17 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
goto failed;
}
- cond_accept_memory(zone, 0, alloc_flags);
+ cond_accept_memory(zone, 0, fastpath_alloc_flags);
retry_this_zone:
- mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages - nr_populated;
+ mark = wmark_pages(zone, fastpath_alloc_flags & ALLOC_WMARK_MASK) +
+ nr_pages - nr_populated;
if (zone_watermark_fast(zone, 0, mark,
zonelist_zone_idx(ac.preferred_zoneref),
- alloc_flags, gfp)) {
+ fastpath_alloc_flags, gfp)) {
break;
}
- if (cond_accept_memory(zone, 0, alloc_flags))
+ if (cond_accept_memory(zone, 0, fastpath_alloc_flags))
goto retry_this_zone;
/* Try again if zone has deferred pages */
@@ -5212,7 +5213,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
continue;
}
- page = __rmqueue_pcplist(zone, 0, ac.migratetype, alloc_flags,
+ page = __rmqueue_pcplist(zone, 0, ac.migratetype, fastpath_alloc_flags,
pcp, pcp_list);
if (unlikely(!page)) {
/* Try and allocate at least one page */
@@ -5238,11 +5239,19 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
return nr_populated;
failed:
- page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, ALLOC_DEFAULT);
+ page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, alloc_flags);
if (page)
page_array[nr_populated++] = page;
goto out;
}
+
+unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array)
+{
+ return __alloc_pages_bulk_noprof(gfp, ALLOC_DEFAULT, preferred_nid,
+ nodemask, nr_pages, page_array);
+}
EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);
/*
@@ -5433,13 +5442,20 @@ struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order
EXPORT_SYMBOL(alloc_pages_node_noprof);
struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
- nodemask_t *nodemask)
+ nodemask_t *nodemask, unsigned int alloc_flags)
{
struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order,
- preferred_nid, nodemask, ALLOC_DEFAULT);
+ preferred_nid, nodemask, alloc_flags);
return page_rmappable_folio(page);
}
-EXPORT_SYMBOL(__folio_alloc_noprof);
+
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
+{
+ warn_if_node_offline(nid, gfp);
+
+ return __folio_alloc_noprof(gfp, order, nid, NULL, ALLOC_DEFAULT);
+}
+EXPORT_SYMBOL(__folio_alloc_node_noprof);
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59d..23fc79ce97b60 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -258,6 +258,14 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_n
nodemask_t *nodemask, unsigned int alloc_flags);
#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))
+struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
+
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+ int preferred_nid, nodemask_t *nodemask, int nr_pages,
+ struct page **page_array);
+
extern void zone_pcp_reset(struct zone *zone);
extern void zone_pcp_disable(struct zone *zone);
extern void zone_pcp_enable(struct zone *zone);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags
2026-07-20 19:33 ` [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags Gregory Price
@ 2026-07-21 18:26 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:26 UTC (permalink / raw)
To: Gregory Price; +Cc: tj, linux-cxl, mkoutny, nvdimm, damon, hannes, kvm
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly added `alloc_flags` parameter is ignored during the bulk allocation fastpath when calling `prep_new_page`.
--
--- Patch [3]: [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags ---
commit 00718b17ea28c0a7382f9831d41a8a45df59e496
Author: Gregory Price <gourry@gourry.net>
mm/page_alloc: let the bulk and folio allocators carry alloc_flags
This commit threads the alloc_flags parameter through the bulk and folio
allocators to allow callers to select specific allocator behavior. It updates
__alloc_pages_bulk_noprof and __folio_alloc_noprof to accept and pass these
flags internally.
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
[ ... ]
> @@ -5238,11 +5239,19 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
> return nr_populated;
>
> failed:
> - page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, ALLOC_DEFAULT);
> + page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, alloc_flags);
> if (page)
> page_array[nr_populated++] = page;
> goto out;
> }
[Severity: Medium]
Will the fast path ignore the caller's new alloc_flags parameter?
In the slow path fallback above, the code correctly passes the new alloc_flags
parameter to __alloc_pages_noprof(). However, in the fast path batch
allocation loop just above this hunk, the pages are still initialized with a
hardcoded ALLOC_DEFAULT:
__alloc_pages_bulk_noprof() {
...
while (nr_populated < nr_pages) {
...
page = __rmqueue_pcplist(...);
...
prep_new_page(page, 0, gfp, ALLOC_DEFAULT);
set_page_refcounted(page);
page_array[nr_populated++] = page;
}
}
If a caller passes custom alloc_flags to __alloc_pages_bulk_noprof(), it
appears the flags will be dropped by the fast path but honored by the slow
path, leading to inconsistent allocator behavior.
Should prep_new_page() in the fast path loop be updated to use the provided
alloc_flags instead of ALLOC_DEFAULT?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=3
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (2 preceding siblings ...)
2026-07-20 19:33 ` [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags Gregory Price
@ 2026-07-20 19:33 ` Gregory Price
2026-07-21 18:18 ` sashiko-bot
2026-07-20 19:33 ` [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes Gregory Price
` (34 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Some devices want to hotplug their memory onto a node, but not have it
exposed as "general purpose". The intent is to re-use portions of mm/
services for management without exposing it for general consumption.
N_MEMORY nodes are intended to contain general System RAM, so they
are unsuited for this purpose.
Create N_MEMORY_PRIVATE for memory nodes whose memory is not intended
for general consumption.
N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive states.
This commit adds basic infrastructure for N_MEMORY_PRIVATE nodes:
- struct node_private:
Per-node container stored in NODE_DATA(nid) storing the owner.
- folio/page_is_private_node():
true if it resides on a private node
- folio/page_is_private_managed():
common predicate for checking private node or zone device
Used to combine common filtering locations.
- Registration API: node_private_register()/unregister()
- sysfs attribute exposing N_MEMORY_PRIVATE node state.
- fix node_state() and helpers to return false for N_MEMORY_PRIVATE
when !CONFIG_NUMA (mutually exclusive w/ N_MEMORY)
Signed-off-by: Gregory Price <gourry@gourry.net>
---
Documentation/ABI/stable/sysfs-devices-node | 10 ++
drivers/base/node.c | 113 ++++++++++++++++++++
include/linux/mmzone.h | 15 +++
include/linux/node_private.h | 82 ++++++++++++++
include/linux/nodemask.h | 7 +-
mm/internal.h | 12 +++
6 files changed, 236 insertions(+), 3 deletions(-)
create mode 100644 include/linux/node_private.h
diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node
index 2d0e023f22a71..8a36fbc3d4028 100644
--- a/Documentation/ABI/stable/sysfs-devices-node
+++ b/Documentation/ABI/stable/sysfs-devices-node
@@ -29,6 +29,16 @@ Description:
Nodes that have regular or high memory.
Depends on CONFIG_HIGHMEM.
+What: /sys/devices/system/node/has_private_memory
+Contact: Linux Memory Management list <linux-mm@kvack.org>
+Description:
+ Nodes that have private (N_MEMORY_PRIVATE) memory: memory
+ hotplugged by a driver onto a CPU-less node and isolated from
+ the page allocator's normal and fallback zonelists. A node is
+ listed here for as long as it holds private memory; it is never
+ listed in has_memory at the same time (the two states are
+ mutually exclusive). See Documentation/mm/numa_private_nodes.rst.
+
What: /sys/devices/system/node/nodeX
Date: October 2002
Contact: Linux Memory Management list <linux-mm@kvack.org>
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 3da91929ad4e3..94cd51f51b7e8 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -22,6 +22,7 @@
#include <linux/swap.h>
#include <linux/slab.h>
#include <linux/memblock.h>
+#include <linux/node_private.h>
static const struct bus_type node_subsys = {
.name = "node",
@@ -868,6 +869,116 @@ void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn,
(void *)&nid, register_mem_block_under_node_hotplug);
return;
}
+
+static DEFINE_MUTEX(node_private_lock);
+
+/**
+ * node_private_register - Register a private node
+ * @nid: Node identifier
+ * @np: The node_private structure (driver-allocated, driver-owned)
+ *
+ * Register an owner for a private node. If an owner is already registered
+ * (different np), return -EBUSY.
+ *
+ * Re-registration with the same np is allowed.
+ *
+ * The caller owns the node_private memory and must ensure it remains valid
+ * until after node_private_unregister() returns.
+ *
+ * Returns 0 on success, negative errno on failure.
+ */
+int node_private_register(int nid, struct node_private *np)
+{
+ struct node_private *existing;
+ pg_data_t *pgdat;
+ int ret = 0;
+
+ if (!np || !node_possible(nid))
+ return -EINVAL;
+
+ mutex_lock(&node_private_lock);
+ mem_hotplug_begin();
+
+ /* N_MEMORY_PRIVATE and N_MEMORY are mutually exclusive */
+ if (node_state(nid, N_MEMORY)) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ pgdat = NODE_DATA(nid);
+ existing = rcu_dereference_protected(pgdat->node_private,
+ lockdep_is_held(&node_private_lock));
+
+ /* If it exists, restrict to single owner */
+ if (existing) {
+ if (existing != np)
+ ret = -EBUSY;
+ goto out;
+ }
+
+ rcu_assign_pointer(pgdat->node_private, np);
+out:
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(node_private_register);
+
+/**
+ * node_private_unregister - Unregister a private node
+ * @nid: Node identifier
+ *
+ * Unregister the driver from a private node.
+ *
+ * Only succeeds if all memory has been offlined (N_MEMORY_PRIVATE cleared).
+ *
+ * N_MEMORY_PRIVATE state is cleared by offline_pages() when the last
+ * memory is offlined, not by this function.
+ *
+ * After this returns, the node_private pointer is no longer visible and
+ * an RCU grace period has elapsed, so the driver may free its context.
+ *
+ * Return: 0 if unregistered, -EBUSY if N_MEMORY_PRIVATE is still set.
+ */
+int node_private_unregister(int nid)
+{
+ struct node_private *np;
+ pg_data_t *pgdat;
+
+ if (!node_possible(nid))
+ return 0;
+
+ mutex_lock(&node_private_lock);
+ mem_hotplug_begin();
+
+ pgdat = NODE_DATA(nid);
+ np = rcu_dereference_protected(pgdat->node_private,
+ lockdep_is_held(&node_private_lock));
+ if (!np) {
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return 0;
+ }
+
+ /*
+ * Only unregister if N_MEMORY_PRIVATE is cleared (only occurs when
+ * the last memory block is offlined in offline_pages())
+ */
+ if (node_is_private(nid)) {
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return -EBUSY;
+ }
+
+ rcu_assign_pointer(pgdat->node_private, NULL);
+
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+
+ synchronize_rcu();
+ return 0;
+}
+EXPORT_SYMBOL_GPL(node_private_unregister);
#endif /* CONFIG_MEMORY_HOTPLUG */
/**
@@ -966,6 +1077,7 @@ static struct node_attr node_state_attr[] = {
[N_HIGH_MEMORY] = _NODE_ATTR(has_high_memory, N_HIGH_MEMORY),
#endif
[N_MEMORY] = _NODE_ATTR(has_memory, N_MEMORY),
+ [N_MEMORY_PRIVATE] = _NODE_ATTR(has_private_memory, N_MEMORY_PRIVATE),
[N_CPU] = _NODE_ATTR(has_cpu, N_CPU),
[N_GENERIC_INITIATOR] = _NODE_ATTR(has_generic_initiator,
N_GENERIC_INITIATOR),
@@ -979,6 +1091,7 @@ static struct attribute *node_state_attrs[] = {
&node_state_attr[N_HIGH_MEMORY].attr.attr,
#endif
&node_state_attr[N_MEMORY].attr.attr,
+ &node_state_attr[N_MEMORY_PRIVATE].attr.attr,
&node_state_attr[N_CPU].attr.attr,
&node_state_attr[N_GENERIC_INITIATOR].attr.attr,
NULL
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0507193b3ae34..9815e48c03b97 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -26,6 +26,8 @@
#include <linux/sizes.h>
#include <asm/page.h>
+struct node_private;
+
/* Free memory management - zoned buddy allocator. */
#ifndef CONFIG_ARCH_FORCE_MAX_ORDER
#define MAX_PAGE_ORDER 10
@@ -1596,12 +1598,25 @@ typedef struct pglist_data {
atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
#ifdef CONFIG_NUMA
struct memory_tier __rcu *memtier;
+ struct node_private __rcu *node_private;
#endif
#ifdef CONFIG_MEMORY_FAILURE
struct memory_failure_stats mf_stats;
#endif
} pg_data_t;
+#ifdef CONFIG_NUMA
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+ return !!pgdat->node_private;
+}
+#else
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+ return false;
+}
+#endif
+
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
#define node_spanned_pages(nid) (NODE_DATA(nid)->node_spanned_pages)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
new file mode 100644
index 0000000000000..475496c84249f
--- /dev/null
+++ b/include/linux/node_private.h
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_NODE_PRIVATE_H
+#define _LINUX_NODE_PRIVATE_H
+
+#include <linux/mm.h>
+#include <linux/nodemask.h>
+
+struct page;
+
+/**
+ * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
+ *
+ * Allocated by the driver and passed to node_private_register().
+ * The driver owns the memory and must ensure it remains valid until after
+ * node_private_unregister() returns.
+ *
+ * @owner: Opaque driver identifier
+ * @caps: NODE_PRIVATE_CAP_* service opt-ins for the node (zero by default;
+ * individual capabilities are defined and consumed by later changes)
+ */
+struct node_private {
+ void *owner;
+ unsigned long caps;
+};
+
+#ifdef CONFIG_NUMA
+#include <linux/mmzone.h>
+
+static inline bool folio_is_private_node(struct folio *folio)
+{
+ return node_state(folio_nid(folio), N_MEMORY_PRIVATE);
+}
+
+static inline bool page_is_private_node(struct page *page)
+{
+ return node_state(page_to_nid(page), N_MEMORY_PRIVATE);
+}
+
+static inline bool node_is_private(int nid)
+{
+ return node_state(nid, N_MEMORY_PRIVATE);
+}
+
+#else /* !CONFIG_NUMA */
+
+static inline bool folio_is_private_node(struct folio *folio)
+{
+ return false;
+}
+
+static inline bool page_is_private_node(struct page *page)
+{
+ return false;
+}
+
+static inline bool node_is_private(int nid)
+{
+ return false;
+}
+
+#endif /* CONFIG_NUMA */
+
+#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
+
+int node_private_register(int nid, struct node_private *np);
+int node_private_unregister(int nid);
+
+#else /* !CONFIG_NUMA || !CONFIG_MEMORY_HOTPLUG */
+
+static inline int node_private_register(int nid, struct node_private *np)
+{
+ return -ENODEV;
+}
+
+static inline int node_private_unregister(int nid)
+{
+ return 0;
+}
+
+#endif /* CONFIG_NUMA && CONFIG_MEMORY_HOTPLUG */
+
+#endif /* _LINUX_NODE_PRIVATE_H */
diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h
index b842aa5255464..ba3e6c570a112 100644
--- a/include/linux/nodemask.h
+++ b/include/linux/nodemask.h
@@ -391,6 +391,7 @@ enum node_states {
N_HIGH_MEMORY = N_NORMAL_MEMORY,
#endif
N_MEMORY, /* The node has memory(regular, high, movable) */
+ N_MEMORY_PRIVATE, /* The node's memory is private */
N_CPU, /* The node has one or more cpus */
N_GENERIC_INITIATOR, /* The node has one or more Generic Initiators */
NR_NODE_STATES
@@ -457,7 +458,7 @@ static __always_inline void node_set_offline(int nid)
static __always_inline int node_state(int node, enum node_states state)
{
- return node == 0;
+ return node == 0 && state != N_MEMORY_PRIVATE;
}
static __always_inline void node_set_state(int node, enum node_states state)
@@ -470,11 +471,11 @@ static __always_inline void node_clear_state(int node, enum node_states state)
static __always_inline int num_node_state(enum node_states state)
{
- return 1;
+ return state == N_MEMORY_PRIVATE ? 0 : 1;
}
#define for_each_node_state(node, __state) \
- for ( (node) = 0; (node) == 0; (node) = 1)
+ for ((node) = 0; (node) == 0 && (__state) != N_MEMORY_PRIVATE; (node) = 1)
#define first_online_node 0
#define first_memory_node 0
diff --git a/mm/internal.h b/mm/internal.h
index 96d78a7778e88..cb9f4a8342e32 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/mm_inline.h>
#include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
#include <linux/pagemap.h>
#include <linux/pagewalk.h>
#include <linux/rmap.h>
@@ -98,6 +99,17 @@ extern int sysctl_min_unmapped_ratio;
extern int sysctl_min_slab_ratio;
#endif
+/* folio_is_private_managed() - folio has special mm management rules */
+static inline bool folio_is_private_managed(struct folio *folio)
+{
+ return folio_is_zone_device(folio) || folio_is_private_node(folio);
+}
+
+static inline bool page_is_private_managed(struct page *page)
+{
+ return folio_is_private_managed(page_folio(page));
+}
+
/*
* Maintains state across a page table move. The operation assumes both source
* and destination VMAs already exist and are specified by the user.
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE
2026-07-20 19:33 ` [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE Gregory Price
@ 2026-07-21 18:18 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:18 UTC (permalink / raw)
To: Gregory Price; +Cc: nvdimm, damon, mkoutny, hannes, tj, kvm, linux-cxl
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Missing bounds validation on `nid` in `node_private_register()` and `node_private_unregister()` leads to out-of-bounds reads.
- [Medium] Lockless read of an `__rcu` pointer in `pgdat_is_private()` without using `rcu_access_pointer()`.
--
--- Patch [4]: [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE ---
commit 69b3cbf2395b52c525233c59376e585ae83c3f31
Author: Gregory Price <gourry@gourry.net>
numa: introduce N_MEMORY_PRIVATE
Some devices want to hotplug their memory onto a node, but not have it
exposed as "general purpose". This commit adds basic infrastructure for
N_MEMORY_PRIVATE nodes, intended for memory isolated from the page
allocator's normal and fallback zonelists.
> diff --git a/drivers/base/node.c b/drivers/base/node.c
> index 3da91929ad4e3..94cd51f51b7e8 100644
> --- a/drivers/base/node.c
> +++ b/drivers/base/node.c
[ ... ]
> @@ -868,6 +869,116 @@ void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn,
> (void *)&nid, register_mem_block_under_node_hotplug);
> return;
> }
> +
> +static DEFINE_MUTEX(node_private_lock);
> +
> +/**
> + * node_private_register - Register a private node
> + * @nid: Node identifier
[ ... ]
> + */
> +int node_private_register(int nid, struct node_private *np)
> +{
> + struct node_private *existing;
> + pg_data_t *pgdat;
> + int ret = 0;
> +
> + if (!np || !node_possible(nid))
> + return -EINVAL;
[Severity: Medium]
Could this lead to an out-of-bounds read if a driver passes an invalid nid
like NUMA_NO_NODE (-1)?
node_possible() and NODE_DATA() do not perform bounds checking internally.
Should we verify nid >= 0 && nid < MAX_NUMNODES before using it?
> +
> + mutex_lock(&node_private_lock);
> + mem_hotplug_begin();
> +
> + /* N_MEMORY_PRIVATE and N_MEMORY are mutually exclusive */
> + if (node_state(nid, N_MEMORY)) {
> + ret = -EBUSY;
> + goto out;
> + }
> +
> + pgdat = NODE_DATA(nid);
[ ... ]
> +/**
> + * node_private_unregister - Unregister a private node
> + * @nid: Node identifier
[ ... ]
> + */
> +int node_private_unregister(int nid)
> +{
> + struct node_private *np;
> + pg_data_t *pgdat;
> +
> + if (!node_possible(nid))
> + return 0;
[Severity: Medium]
Is it possible for a driver to pass NUMA_NO_NODE here as well, leading to
the same out-of-bounds read issue when node_possible() and NODE_DATA()
are evaluated?
> +
> + mutex_lock(&node_private_lock);
> + mem_hotplug_begin();
> +
> + pgdat = NODE_DATA(nid);
[ ... ]
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 0507193b3ae34..9815e48c03b97 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
[ ... ]
> @@ -1596,12 +1598,25 @@ typedef struct pglist_data {
> atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
> #ifdef CONFIG_NUMA
> struct memory_tier __rcu *memtier;
> + struct node_private __rcu *node_private;
> #endif
> #ifdef CONFIG_MEMORY_FAILURE
> struct memory_failure_stats mf_stats;
> #endif
> } pg_data_t;
>
> +#ifdef CONFIG_NUMA
> +static inline bool pgdat_is_private(pg_data_t *pgdat)
> +{
> + return !!pgdat->node_private;
[Severity: Medium]
Does this lockless read of the __rcu pointer require rcu_access_pointer()?
Without it, sparse will likely complain about an __rcu address space
violation, and we might be missing the READ_ONCE() barrier that prevents
compiler load tearing.
> +}
> +#else
> +static inline bool pgdat_is_private(pg_data_t *pgdat)
> +{
> + return false;
> +}
> +#endif
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=4
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (3 preceding siblings ...)
2026-07-20 19:33 ` [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE Gregory Price
@ 2026-07-20 19:33 ` Gregory Price
2026-07-22 13:00 ` Richard Cheng
2026-07-20 19:34 ` [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open) Gregory Price
` (33 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
N_MEMORY_PRIVATE nodes must be invisible to ordinary allocation,
no allocations may reach one unless explicitly called for.
Make this isolation structural in the zonelists: N_MEMORY_PRIVATE
nodes are excluded from FALLBACK and NOFALLBACK entirely, making
them effectively invisible to all normal callers of page_alloc.
Add ZONELIST_PRIVATE, which spans (N_MEMORY | N_MEMORY_PRIVATE).
When !CONFIG_NUMA, it aliases ZONELIST_FALLBACK and compiles out.
Add ZONELIST_PRIVATE_NOFALLBACK as the __GFP_THISNODE target for
private node allocations.
Zonelist selection rides the allocator's alloc_flags. Add
ALLOC_ZONELIST_PRIVATE and resolve it in select_zonelist(), which
prepare_alloc_pages() applies from ac->alloc_flags; the default
(ALLOC_DEFAULT) keeps node_zonelist()'s gfp-based selection.
Add explicit private-node alloc() functions to prevent open-coding
(both ride ALLOC_ZONELIST_PRIVATE through the alloc_flags-carrying
entry points):
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()
alloc_contig_range adds an explicit node_is_private() guard because
it dervices its target zones from PFNs rather than zonelists. All
in-tree callers use N_MEMORY ranges, but the check prevents future
callers from violating node isolation.
Important note: By design this zonelist isolates N_MEMORY_PRIVATE
from unintentional allocations, but does NOT isolate private-node
allocations from falling back to non-private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/gfp.h | 11 ++++++++
include/linux/mmzone.h | 11 +++++++-
mm/mm_init.c | 2 +-
mm/page_alloc.c | 64 ++++++++++++++++++++++++++++++++++++++++--
mm/page_alloc.h | 26 +++++++++++++++++
5 files changed, 109 insertions(+), 5 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index a327e58c313f8..f3e867de744f6 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -204,6 +204,17 @@ static inline void arch_free_page(struct page *page, int order) { }
static inline void arch_alloc_page(struct page *page, int order) { }
#endif
+/* Allocate from an N_MEMORY_PRIVATE node (selects the private zonelist). */
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid);
+#define alloc_pages_node_private(...) \
+ alloc_hooks(alloc_pages_node_private_noprof(__VA_ARGS__))
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid);
+#define folio_alloc_node_private(...) \
+ alloc_hooks(folio_alloc_node_private_noprof(__VA_ARGS__))
+
unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
struct page **page_array);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 9815e48c03b97..654c5111f7cec 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1392,13 +1392,22 @@ enum {
#ifdef CONFIG_NUMA
/*
* The NUMA zonelists are doubled because we need zonelists that
- * restrict the allocations to a single node for __GFP_THISNODE.
+ * restrict the allocations to a single node for __GFP_THISNODE
+ * and N_MEMORY_PRIVATE nodes (isolated from default lists).
*/
ZONELIST_NOFALLBACK, /* zonelist without fallback (__GFP_THISNODE) */
+ ZONELIST_PRIVATE, /* N_MEMORY_PRIVATE access, falls back to DRAM */
+ ZONELIST_PRIVATE_NOFALLBACK, /* N_MEMORY_PRIVATE access, __GFP_THISNODE */
#endif
MAX_ZONELISTS
};
+#ifndef CONFIG_NUMA
+/* Without NUMA only ZONELIST_FALLBACK exists so everything collapses there */
+#define ZONELIST_PRIVATE ZONELIST_FALLBACK
+#define ZONELIST_PRIVATE_NOFALLBACK ZONELIST_FALLBACK
+#endif
+
/*
* This struct contains information about a zone in a zonelist. It is stored
* here to avoid dereferences into large structures and lookups of tables
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 711f821f7b3c7..adb50647d29ca 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2701,7 +2701,7 @@ void __init mm_core_init(void)
init_zero_page_pfn();
/* Initializations relying on SMP setup */
- BUILD_BUG_ON(MAX_ZONELISTS > 2);
+ BUILD_BUG_ON(MAX_ZONELISTS > 4);
build_all_zonelists(NULL);
page_alloc_init_cpuhp();
alloc_tag_sec_init();
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index aa35bbe5d4c3e..78755a55b1540 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5040,7 +5040,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
unsigned int *alloc_flags)
{
ac->highest_zoneidx = gfp_zone(gfp_mask);
- ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
+ ac->zonelist = select_zonelist(preferred_nid, gfp_mask, ac->alloc_flags);
ac->nodemask = nodemask;
ac->migratetype = gfp_migratetype(gfp_mask);
@@ -5346,7 +5346,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
unsigned int fastpath_alloc_flags = alloc_flags;
/* Other flags could be supported later if needed. */
- if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
+ if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG | ALLOC_ZONELIST_PRIVATE)))
return NULL;
if (!alloc_order_allowed(gfp, order, alloc_flags))
@@ -5457,6 +5457,22 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
}
EXPORT_SYMBOL(__folio_alloc_node_noprof);
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid)
+{
+ return __alloc_pages_noprof(gfp, order, nid, NULL,
+ ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(alloc_pages_node_private_noprof);
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid)
+{
+ return __folio_alloc_noprof(gfp, order, nid, NULL,
+ ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(folio_alloc_node_private_noprof);
+
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
* address cannot represent highmem pages. Use alloc_pages and then kmap if
@@ -5854,9 +5870,21 @@ static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
static void build_thisnode_zonelists(pg_data_t *pgdat)
{
struct zoneref *zonerefs;
- int nr_zones;
+ int nr_zones = 0;
+ /*
+ * NOFALLBACK: the node's own zones. EMPTY for private nodes so a
+ * stray __GFP_THISNODE allocation can't violate isolation.
+ */
zonerefs = pgdat->node_zonelists[ZONELIST_NOFALLBACK]._zonerefs;
+ if (!node_is_private(pgdat->node_id))
+ nr_zones = build_zonerefs_node(pgdat, zonerefs);
+ zonerefs += nr_zones;
+ zonerefs->zone = NULL;
+ zonerefs->zone_idx = 0;
+
+ /* PRIVATE_NOFALLBACK: the node's own zones, private nodes included. */
+ zonerefs = pgdat->node_zonelists[ZONELIST_PRIVATE_NOFALLBACK]._zonerefs;
nr_zones = build_zonerefs_node(pgdat, zonerefs);
zonerefs += nr_zones;
zonerefs->zone = NULL;
@@ -5899,11 +5927,28 @@ static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
static void build_zonelists(pg_data_t *pgdat)
{
static int node_order[MAX_NUMNODES];
+ nodemask_t tier_nodes;
int local_node = pgdat->node_id;
int node, nr_nodes = 0;
memset(node_order, 0, sizeof(node_order));
+ /*
+ * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
+ *
+ * FALLBACK: Allocation order for all nodes. Private nodes have lists
+ * but never appear as an entry in any list. Allocations
+ * targeting a private node w/ FALLBACK land on N_MEMORY.
+ *
+ * NOFALLBACK: A list for each node containing only itself.
+ * Allocations targeting a private node w/ NOFALLBACK
+ * will always fail (their NOFALLBACK is empty)
+ *
+ * PRIVATE: (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
+ * private nodes, and falls back to normal memory unless
+ * __GFP_THISNODE otherwise constrains it.
+ */
+
build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
true, node_order, &nr_nodes);
build_thisnode_zonelists(pgdat);
@@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
for (node = 0; node < nr_nodes; node++)
pr_cont("%d ", node_order[node]);
pr_cont("\n");
+
+ nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ build_node_zonelist(pgdat, &tier_nodes, ZONELIST_PRIVATE, false,
+ node_order, &nr_nodes);
}
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
@@ -7282,6 +7331,15 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
if (__alloc_contig_verify_gfp_mask(gfp_mask, (gfp_t *)&cc.gfp_mask))
return -EINVAL;
+ /*
+ * Private nodes are kept unreachable via the zonelists, but
+ * alloc_contig works directly on a zone derived from the caller's
+ * pfn range, bypassing that. Reject this operation explicitly
+ * rather than silently carving memory out of an isolated node.
+ */
+ if (unlikely(node_is_private(zone_to_nid(cc.zone))))
+ return -EBUSY;
+
/*
* What we do here is we mark all pageblocks in range as
* MIGRATE_ISOLATE. Because pageblock and max order pages may
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index 23fc79ce97b60..741448d83ce77 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -57,6 +57,9 @@
*/
#define ALLOC_NO_CODETAG 0x1000
+/* Select the N_MEMORY_PRIVATE zonelist family (see select_zonelist()). */
+#define ALLOC_ZONELIST_PRIVATE 0x2000
+
/* Flags that allow allocations below the min watermark. */
#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
@@ -95,6 +98,29 @@ struct alloc_context {
unsigned int alloc_flags;
};
+#ifdef CONFIG_NUMA
+static_assert(ZONELIST_PRIVATE + 1 == ZONELIST_PRIVATE_NOFALLBACK);
+#endif
+
+/*
+ * select_zonelist - resolve the zonelist for an allocation
+ *
+ * The default matches node_zonelist(); ALLOC_ZONELIST_PRIVATE selects the
+ * private-node family so an allocation may reach an N_MEMORY_PRIVATE node.
+ */
+static inline struct zonelist *
+select_zonelist(int nid, gfp_t gfp, unsigned int alloc_flags)
+{
+#ifdef CONFIG_NUMA
+ if (alloc_flags & ALLOC_ZONELIST_PRIVATE) {
+ int idx = ZONELIST_PRIVATE + !!(gfp & __GFP_THISNODE);
+
+ return &NODE_DATA(nid)->node_zonelists[idx];
+ }
+#endif
+ return node_zonelist(nid, gfp);
+}
+
/*
* This function returns the order of a free page in the buddy system. In
* general, page_zone(page)->lock must be held by the caller to prevent the
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
2026-07-20 19:33 ` [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes Gregory Price
@ 2026-07-22 13:00 ` Richard Cheng
2026-07-22 13:19 ` Gregory Price
0 siblings, 1 reply; 82+ messages in thread
From: Richard Cheng @ 2026-07-22 13:00 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, Jul 20, 2026 at 03:33:59PM +0800, Gregory Price wrote:
> N_MEMORY_PRIVATE nodes must be invisible to ordinary allocation,
> no allocations may reach one unless explicitly called for.
>
> Make this isolation structural in the zonelists: N_MEMORY_PRIVATE
> nodes are excluded from FALLBACK and NOFALLBACK entirely, making
> them effectively invisible to all normal callers of page_alloc.
>
> Add ZONELIST_PRIVATE, which spans (N_MEMORY | N_MEMORY_PRIVATE).
> When !CONFIG_NUMA, it aliases ZONELIST_FALLBACK and compiles out.
>
> Add ZONELIST_PRIVATE_NOFALLBACK as the __GFP_THISNODE target for
> private node allocations.
>
> Zonelist selection rides the allocator's alloc_flags. Add
> ALLOC_ZONELIST_PRIVATE and resolve it in select_zonelist(), which
> prepare_alloc_pages() applies from ac->alloc_flags; the default
> (ALLOC_DEFAULT) keeps node_zonelist()'s gfp-based selection.
>
> Add explicit private-node alloc() functions to prevent open-coding
> (both ride ALLOC_ZONELIST_PRIVATE through the alloc_flags-carrying
> entry points):
> - alloc_pages_node_private_noprof()
> - folio_alloc_node_private_noprof()
>
> alloc_contig_range adds an explicit node_is_private() guard because
> it dervices its target zones from PFNs rather than zonelists. All
> in-tree callers use N_MEMORY ranges, but the check prevents future
> callers from violating node isolation.
>
> Important note: By design this zonelist isolates N_MEMORY_PRIVATE
> from unintentional allocations, but does NOT isolate private-node
> allocations from falling back to non-private nodes.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> include/linux/gfp.h | 11 ++++++++
> include/linux/mmzone.h | 11 +++++++-
> mm/mm_init.c | 2 +-
> mm/page_alloc.c | 64 ++++++++++++++++++++++++++++++++++++++++--
> mm/page_alloc.h | 26 +++++++++++++++++
> 5 files changed, 109 insertions(+), 5 deletions(-)
>
> diff --git a/include/linux/gfp.h b/include/linux/gfp.h
> index a327e58c313f8..f3e867de744f6 100644
> --- a/include/linux/gfp.h
> +++ b/include/linux/gfp.h
> @@ -204,6 +204,17 @@ static inline void arch_free_page(struct page *page, int order) { }
> static inline void arch_alloc_page(struct page *page, int order) { }
> #endif
>
> +/* Allocate from an N_MEMORY_PRIVATE node (selects the private zonelist). */
> +struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
> + int nid);
> +#define alloc_pages_node_private(...) \
> + alloc_hooks(alloc_pages_node_private_noprof(__VA_ARGS__))
> +
> +struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
> + int nid);
> +#define folio_alloc_node_private(...) \
> + alloc_hooks(folio_alloc_node_private_noprof(__VA_ARGS__))
> +
> unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
> nodemask_t *nodemask, int nr_pages,
> struct page **page_array);
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 9815e48c03b97..654c5111f7cec 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -1392,13 +1392,22 @@ enum {
> #ifdef CONFIG_NUMA
> /*
> * The NUMA zonelists are doubled because we need zonelists that
> - * restrict the allocations to a single node for __GFP_THISNODE.
> + * restrict the allocations to a single node for __GFP_THISNODE
> + * and N_MEMORY_PRIVATE nodes (isolated from default lists).
> */
> ZONELIST_NOFALLBACK, /* zonelist without fallback (__GFP_THISNODE) */
> + ZONELIST_PRIVATE, /* N_MEMORY_PRIVATE access, falls back to DRAM */
> + ZONELIST_PRIVATE_NOFALLBACK, /* N_MEMORY_PRIVATE access, __GFP_THISNODE */
> #endif
> MAX_ZONELISTS
> };
>
> +#ifndef CONFIG_NUMA
> +/* Without NUMA only ZONELIST_FALLBACK exists so everything collapses there */
> +#define ZONELIST_PRIVATE ZONELIST_FALLBACK
> +#define ZONELIST_PRIVATE_NOFALLBACK ZONELIST_FALLBACK
> +#endif
> +
> /*
> * This struct contains information about a zone in a zonelist. It is stored
> * here to avoid dereferences into large structures and lookups of tables
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index 711f821f7b3c7..adb50647d29ca 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -2701,7 +2701,7 @@ void __init mm_core_init(void)
> init_zero_page_pfn();
>
> /* Initializations relying on SMP setup */
> - BUILD_BUG_ON(MAX_ZONELISTS > 2);
> + BUILD_BUG_ON(MAX_ZONELISTS > 4);
> build_all_zonelists(NULL);
> page_alloc_init_cpuhp();
> alloc_tag_sec_init();
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index aa35bbe5d4c3e..78755a55b1540 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -5040,7 +5040,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
> unsigned int *alloc_flags)
> {
> ac->highest_zoneidx = gfp_zone(gfp_mask);
> - ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
> + ac->zonelist = select_zonelist(preferred_nid, gfp_mask, ac->alloc_flags);
> ac->nodemask = nodemask;
> ac->migratetype = gfp_migratetype(gfp_mask);
>
> @@ -5346,7 +5346,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
> unsigned int fastpath_alloc_flags = alloc_flags;
>
> /* Other flags could be supported later if needed. */
> - if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
> + if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG | ALLOC_ZONELIST_PRIVATE)))
> return NULL;
>
> if (!alloc_order_allowed(gfp, order, alloc_flags))
> @@ -5457,6 +5457,22 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
> }
> EXPORT_SYMBOL(__folio_alloc_node_noprof);
>
> +struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
> + int nid)
> +{
> + return __alloc_pages_noprof(gfp, order, nid, NULL,
> + ALLOC_ZONELIST_PRIVATE);
> +}
> +EXPORT_SYMBOL_GPL(alloc_pages_node_private_noprof);
> +
> +struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
> + int nid)
> +{
> + return __folio_alloc_noprof(gfp, order, nid, NULL,
> + ALLOC_ZONELIST_PRIVATE);
> +}
> +EXPORT_SYMBOL_GPL(folio_alloc_node_private_noprof);
> +
> /*
> * Common helper functions. Never use with __GFP_HIGHMEM because the returned
> * address cannot represent highmem pages. Use alloc_pages and then kmap if
> @@ -5854,9 +5870,21 @@ static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
> static void build_thisnode_zonelists(pg_data_t *pgdat)
> {
> struct zoneref *zonerefs;
> - int nr_zones;
> + int nr_zones = 0;
>
> + /*
> + * NOFALLBACK: the node's own zones. EMPTY for private nodes so a
> + * stray __GFP_THISNODE allocation can't violate isolation.
> + */
> zonerefs = pgdat->node_zonelists[ZONELIST_NOFALLBACK]._zonerefs;
> + if (!node_is_private(pgdat->node_id))
> + nr_zones = build_zonerefs_node(pgdat, zonerefs);
> + zonerefs += nr_zones;
> + zonerefs->zone = NULL;
> + zonerefs->zone_idx = 0;
> +
> + /* PRIVATE_NOFALLBACK: the node's own zones, private nodes included. */
> + zonerefs = pgdat->node_zonelists[ZONELIST_PRIVATE_NOFALLBACK]._zonerefs;
> nr_zones = build_zonerefs_node(pgdat, zonerefs);
> zonerefs += nr_zones;
> zonerefs->zone = NULL;
> @@ -5899,11 +5927,28 @@ static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
> static void build_zonelists(pg_data_t *pgdat)
> {
> static int node_order[MAX_NUMNODES];
> + nodemask_t tier_nodes;
> int local_node = pgdat->node_id;
> int node, nr_nodes = 0;
>
> memset(node_order, 0, sizeof(node_order));
>
> + /*
> + * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
> + *
> + * FALLBACK: Allocation order for all nodes. Private nodes have lists
> + * but never appear as an entry in any list. Allocations
> + * targeting a private node w/ FALLBACK land on N_MEMORY.
> + *
> + * NOFALLBACK: A list for each node containing only itself.
> + * Allocations targeting a private node w/ NOFALLBACK
> + * will always fail (their NOFALLBACK is empty)
> + *
> + * PRIVATE: (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
> + * private nodes, and falls back to normal memory unless
> + * __GFP_THISNODE otherwise constrains it.
> + */
> +
> build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
> true, node_order, &nr_nodes);
> build_thisnode_zonelists(pgdat);
> @@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
> for (node = 0; node < nr_nodes; node++)
> pr_cont("%d ", node_order[node]);
> pr_cont("\n");
> +
> + nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
Hi Gregory,
A question about this part, if my understanding is right,
ZONELIST_PRIVATE contains all private nodes, should we use some explicit target
nodemask here to make sure only the targeting owner's private node can be touched?
Otherwise it might fall back into a different owner's private node.
--Richard
> + build_node_zonelist(pgdat, &tier_nodes, ZONELIST_PRIVATE, false,
> + node_order, &nr_nodes);
> }
>
> #ifdef CONFIG_HAVE_MEMORYLESS_NODES
> @@ -7282,6 +7331,15 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
> if (__alloc_contig_verify_gfp_mask(gfp_mask, (gfp_t *)&cc.gfp_mask))
> return -EINVAL;
>
> + /*
> + * Private nodes are kept unreachable via the zonelists, but
> + * alloc_contig works directly on a zone derived from the caller's
> + * pfn range, bypassing that. Reject this operation explicitly
> + * rather than silently carving memory out of an isolated node.
> + */
> + if (unlikely(node_is_private(zone_to_nid(cc.zone))))
> + return -EBUSY;
> +
> /*
> * What we do here is we mark all pageblocks in range as
> * MIGRATE_ISOLATE. Because pageblock and max order pages may
> diff --git a/mm/page_alloc.h b/mm/page_alloc.h
> index 23fc79ce97b60..741448d83ce77 100644
> --- a/mm/page_alloc.h
> +++ b/mm/page_alloc.h
> @@ -57,6 +57,9 @@
> */
> #define ALLOC_NO_CODETAG 0x1000
>
> +/* Select the N_MEMORY_PRIVATE zonelist family (see select_zonelist()). */
> +#define ALLOC_ZONELIST_PRIVATE 0x2000
> +
> /* Flags that allow allocations below the min watermark. */
> #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
>
> @@ -95,6 +98,29 @@ struct alloc_context {
> unsigned int alloc_flags;
> };
>
> +#ifdef CONFIG_NUMA
> +static_assert(ZONELIST_PRIVATE + 1 == ZONELIST_PRIVATE_NOFALLBACK);
> +#endif
> +
> +/*
> + * select_zonelist - resolve the zonelist for an allocation
> + *
> + * The default matches node_zonelist(); ALLOC_ZONELIST_PRIVATE selects the
> + * private-node family so an allocation may reach an N_MEMORY_PRIVATE node.
> + */
> +static inline struct zonelist *
> +select_zonelist(int nid, gfp_t gfp, unsigned int alloc_flags)
> +{
> +#ifdef CONFIG_NUMA
> + if (alloc_flags & ALLOC_ZONELIST_PRIVATE) {
> + int idx = ZONELIST_PRIVATE + !!(gfp & __GFP_THISNODE);
> +
> + return &NODE_DATA(nid)->node_zonelists[idx];
> + }
> +#endif
> + return node_zonelist(nid, gfp);
> +}
> +
> /*
> * This function returns the order of a free page in the buddy system. In
> * general, page_zone(page)->lock must be held by the caller to prevent the
> --
> 2.53.0-Meta
>
>
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
2026-07-22 13:00 ` Richard Cheng
@ 2026-07-22 13:19 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-22 13:19 UTC (permalink / raw)
To: Richard Cheng
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 09:00:51PM +0800, Richard Cheng wrote:
> On Mon, Jul 20, 2026 at 03:33:59PM +0800, Gregory Price wrote:
> > static void build_zonelists(pg_data_t *pgdat)
> > {
> > static int node_order[MAX_NUMNODES];
> > + nodemask_t tier_nodes;
> > int local_node = pgdat->node_id;
> > int node, nr_nodes = 0;
> >
> > memset(node_order, 0, sizeof(node_order));
> >
> > + /*
> > + * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
> > + *
> > + * FALLBACK: Allocation order for all nodes. Private nodes have lists
> > + * but never appear as an entry in any list. Allocations
> > + * targeting a private node w/ FALLBACK land on N_MEMORY.
> > + *
> > + * NOFALLBACK: A list for each node containing only itself.
> > + * Allocations targeting a private node w/ NOFALLBACK
> > + * will always fail (their NOFALLBACK is empty)
> > + *
> > + * PRIVATE: (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
> > + * private nodes, and falls back to normal memory unless
> > + * __GFP_THISNODE otherwise constrains it.
> > + */
> > +
> > build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
> > true, node_order, &nr_nodes);
> > build_thisnode_zonelists(pgdat);
> > @@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
> > for (node = 0; node < nr_nodes; node++)
> > pr_cont("%d ", node_order[node]);
> > pr_cont("\n");
> > +
> > + nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
>
> Hi Gregory,
>
> A question about this part, if my understanding is right,
> ZONELIST_PRIVATE contains all private nodes, should we use some explicit target
> nodemask here to make sure only the targeting owner's private node can be touched?
>
> Otherwise it might fall back into a different owner's private node.
>
> --Richard
>
Zonelists and the constructors here is a bit difficult to understand,
and I see I actually did a poor job of explaining what's going on in
the comments here. I will fix that up in v6.
Lets say we have the following nodes:
N_MEMORY : 0 1
N_MEMORY_PRIVATE : 2 3
The following zonelists would exist:
ZONELIST_FALLBACK[0] = {0,1}
ZONELIST_FALLBACK[1] = {1,0}
ZONELIST_FALLBACK[2] = {0,1}
ZONELIST_FALLBACK[3] = {0,1}
ZONELIST_NOFALLBACK[0] = {0}
ZONELIST_NOFALLBACK[1] = {1}
ZONELIST_NOFALLBACK[2] = {}
ZONELIST_NOFALLBACK[3] = {}
ZONELIST_PRIVATE[0] = {0,1}
ZONELIST_PRIVATE[1] = {1,0}
ZONELIST_PRIVATE[2] = {2,0,1} <- no fallback to 3
ZONELIST_PRIVATE[3] = {3,0,1} <- no fallback to 2
ZONELIST_PRIVATE_NOFALLBACK[0] = {0}
ZONELIST_PRIVATE_NOFALLBACK[1] = {1}
ZONELIST_PRIVATE_NOFALLBACK[2] = {2} <- __GFP_THISNODE
ZONELIST_PRIVATE_NOFALLBACK[3] = {3} <- __GFP_THISNODE
In the current design, we actually can never fallback to another private
node, and instead if you wanted to - for example - interleave across
said nodes you'd have to do so manually.
I think it's reasonable to later extend zonelist construction to support
such a think like:
ZONELIST_PRIVATE[2] = {2,0,1,3}
Later if such a need arises, but I would say wait for the use case.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open)
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (4 preceding siblings ...)
2026-07-20 19:33 ` [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:22 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 07/36] mm/memory_hotplug: disallow migration-driven private node hotunplug Gregory Price
` (32 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
N_MEMORY_PRIVATE node access is gated by per-node capabilities and
the zonelist. Isolating by cpuset offers no functionality and
creates issues on rebind when a cpuset loses access to that node
(in particular: it generates migrations that may not be supported).
Do not partition private nodes via cpuset.mem, instead treat them
as globally accessible resources. Do not engage in silent background
operations on these nodes due to cpuset.mem membership changing.
The result: cpuset operations checking for node validity always
allow N_MEMORY_PRIVATE nodes.
On hot-unplug, we still need to rebind memory policies if the
private node has left N_MEMORY_PRIVATE (i.e. no more memory).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
kernel/cgroup/cpuset.c | 26 ++++++++++++++++++++++----
mm/memcontrol.c | 2 ++
2 files changed, 24 insertions(+), 4 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index dfd0f827e3b92..05468f95c10bd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -26,6 +26,7 @@
#include <linux/mempolicy.h>
#include <linux/mm.h>
#include <linux/memory.h>
+#include <linux/node_private.h>
#include <linux/rcupdate.h>
#include <linux/sched.h>
#include <linux/sched/deadline.h>
@@ -3867,7 +3868,8 @@ static void cpuset_handle_hotplug(void)
static DECLARE_WORK(hk_sd_work, hk_sd_workfn);
static cpumask_t new_cpus;
static nodemask_t new_mems;
- bool cpus_updated, mems_updated;
+ static nodemask_t prev_priv_mems;
+ bool cpus_updated, mems_updated, priv_shrank;
bool on_dfl = is_in_v2_mode();
struct tmpmasks tmp, *ptmp = NULL;
@@ -3890,6 +3892,14 @@ static void cpuset_handle_hotplug(void)
!cpumask_empty(subpartitions_cpus);
mems_updated = !nodes_equal(top_cpuset.effective_mems, new_mems);
+ /*
+ * Private nodes are not partitioned by cpuset, but if one leaves
+ * N_MEMORY_PRIVATE we still need to run mpol_rebind_* to clean up
+ * mempolicies that are binding them.
+ */
+ priv_shrank = !nodes_subset(prev_priv_mems, node_states[N_MEMORY_PRIVATE]);
+ prev_priv_mems = node_states[N_MEMORY_PRIVATE];
+
/* For v1, synchronize cpus_allowed to cpu_active_mask */
if (cpus_updated) {
cpuset_force_rebuild();
@@ -3922,9 +3932,12 @@ static void cpuset_handle_hotplug(void)
top_cpuset.mems_allowed = new_mems;
top_cpuset.effective_mems = new_mems;
spin_unlock_irq(&callback_lock);
- cpuset_update_tasks_nodemask(&top_cpuset);
}
+ /* Rebind task mempolicies if any memory node changed state */
+ if (mems_updated || priv_shrank)
+ cpuset_update_tasks_nodemask(&top_cpuset);
+
mutex_unlock(&cpuset_mutex);
/* if cpus or mems changed, we need to propagate to descendants */
@@ -4155,11 +4168,13 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
* cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
* @nodemask: the nodemask to be checked
*
- * Are any of the nodes in the nodemask allowed in current->mems_allowed?
+ * Are any of the nodes in the nodemask usable? N_MEMORY nodes must be in
+ * current->mems_allowed, while N_MEMORY_PRIVATE nodes are always valid.
*/
int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
{
- return nodes_intersects(*nodemask, current->mems_allowed);
+ return nodes_intersects(*nodemask, current->mems_allowed) ||
+ nodes_intersects(*nodemask, node_states[N_MEMORY_PRIVATE]);
}
/*
@@ -4223,6 +4238,9 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)
if (in_interrupt())
return true;
+ /* N_MEMORY_PRIVATE nodes are not partitioned by cpusets.mems */
+ if (node_is_private(node))
+ return true;
if (node_isset(node, current->mems_allowed))
return true;
/*
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 8319ad8c5c23a..f0dde52dc9e0e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6069,6 +6069,8 @@ void mem_cgroup_node_filter_allowed(struct mem_cgroup *memcg, nodemask_t *mask)
* mask is acceptable.
*/
cpuset_nodes_allowed(memcg->css.cgroup, &allowed);
+ /* N_MEMORY_PRIVATE nodes are not partitioned by cpuset, include them */
+ nodes_or(allowed, allowed, node_states[N_MEMORY_PRIVATE]);
nodes_and(*mask, *mask, allowed);
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open)
2026-07-20 19:34 ` [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open) Gregory Price
@ 2026-07-21 18:22 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:22 UTC (permalink / raw)
To: Gregory Price; +Cc: mkoutny, hannes, nvdimm, linux-cxl, damon, kvm, tj
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Tasks in descendant cpusets do not have their mempolicies rebound when a private node is hot-unplugged.
- [High] Mempolicy creation (`mbind`, `set_mempolicy`) and page migration (`sys_migrate_pages`) unconditionally reject private nodes, breaking UAPI.
- [Medium] Valid private nodes are unconditionally stripped from tasks' memory policies during any unrelated cpuset update.
--
--- Patch [6]: [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open) ---
commit 9d0d11725cd4a8e139c634034823f38bcc9b8f9f
Author: Gregory Price <gourry@gourry.net>
cpuset: exclude private nodes from cpuset.mems (default-open)
This commit modifies cpuset handling to treat N_MEMORY_PRIVATE nodes as globally
accessible resources rather than partitioning them via cpuset.mems. It updates
node validation checks and hotplug logic to accommodate private nodes.
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index dfd0f827e3b92..05468f95c10bd 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
[ ... ]
> @@ -3922,9 +3932,12 @@ static void cpuset_handle_hotplug(void)
> top_cpuset.mems_allowed = new_mems;
> top_cpuset.effective_mems = new_mems;
> spin_unlock_irq(&callback_lock);
> - cpuset_update_tasks_nodemask(&top_cpuset);
> }
>
> + /* Rebind task mempolicies if any memory node changed state */
> + if (mems_updated || priv_shrank)
> + cpuset_update_tasks_nodemask(&top_cpuset);
[Severity: Medium]
When cpuset_update_tasks_nodemask() is called here, it passes
&cs->mems_allowed to mpol_rebind_mm():
kernel/cgroup/cpuset.c:cpuset_update_tasks_nodemask() {
...
mpol_rebind_mm(mm, &cs->mems_allowed);
}
Because mems_allowed explicitly excludes private nodes, will this
inadvertently strip valid private nodes from a task's static memory policy
during completely unrelated cpuset operations?
> +
> mutex_unlock(&cpuset_mutex);
>
> /* if cpus or mems changed, we need to propagate to descendants */
> if (cpus_updated || mems_updated) {
[Severity: High]
Does this skip propagating private node hot-unplug events to descendant
cpusets?
If priv_shrank is true but mems_updated is false, this block is skipped.
Could tasks in descendant cpusets retain mempolicies pointing to the newly
offlined private node?
Additionally, if the loop does execute, cpuset_hotplug_update_tasks()
locally checks:
kernel/cgroup/cpuset.c:cpuset_hotplug_update_tasks() {
...
mems_updated = !nodes_equal(new_mems, cs->effective_mems);
if (!cpus_updated && !mems_updated)
goto unlock;
}
Since effective_mems ignores private nodes, it appears descendant updates
would be skipped here as well.
[ ... ]
> @@ -4155,11 +4168,13 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
> * cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
> * @nodemask: the nodemask to be checked
> *
> - * Are any of the nodes in the nodemask allowed in current->mems_allowed?
> + * Are any of the nodes in the nodemask usable? N_MEMORY nodes must be in
> + * current->mems_allowed, while N_MEMORY_PRIVATE nodes are always valid.
> */
> int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
> {
> - return nodes_intersects(*nodemask, current->mems_allowed);
> + return nodes_intersects(*nodemask, current->mems_allowed) ||
> + nodes_intersects(*nodemask, node_states[N_MEMORY_PRIVATE]);
> }
[Severity: High]
Are private nodes unconditionally rejected by sys_migrate_pages and mempolicy
creation?
While cpuset_nodemask_valid_mems_allowed() was updated here, userspace
operations like kernel_migrate_pages() and mpol_set_nodemask() validate
against cpuset_mems_allowed(current), which is defined just above this
function.
cpuset_mems_allowed() delegates to guarantee_online_mems(), which masks
strictly against N_MEMORY.
For example, in mpol_set_nodemask():
mm/mempolicy.c:mpol_set_nodemask() {
...
nodes_and(nsc->mask1, cpuset_current_mems_allowed, node_states[N_MEMORY]);
...
}
Does this mean any mbind() or set_mempolicy() targeting an N_MEMORY_PRIVATE
node will still be stripped and return -EINVAL?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=6
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 07/36] mm/memory_hotplug: disallow migration-driven private node hotunplug
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (5 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open) Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration Gregory Price
` (31 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
By default, device-backed private node memory may not be safe to
migrate without coordination.
Do not attempt to hotunplug private node memory, instead fail the
hotunplug attempt, and require the owner to drain the folios
safely before allowing hotunplug to proceed.
Unlike transient / longterm pins causing failures, this is not a
transient condition - so we can break out of the hotunplug attempt
entirely instead of spinning forever on a failed migration attempt.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/memory_hotplug.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078ad..2b9c0830821e6 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -34,6 +34,7 @@
#include <linux/memblock.h>
#include <linux/compaction.h>
#include <linux/rmap.h>
+#include <linux/node_private.h>
#include <linux/module.h>
#include <linux/node.h>
@@ -1843,11 +1844,12 @@ static int scan_movable_pages(unsigned long start, unsigned long end,
return 0;
}
-static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
+static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
{
struct folio *folio;
unsigned long pfn;
LIST_HEAD(source);
+ int err = 0;
static DEFINE_RATELIMIT_STATE(migrate_rs, DEFAULT_RATELIMIT_INTERVAL,
DEFAULT_RATELIMIT_BURST);
@@ -1884,6 +1886,15 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
goto put_folio;
}
+ /* Private node folios cannot migrate, fail outright */
+ if (folio_is_private_node(folio)) {
+ pr_info_ratelimited("memory offline refused: node %d pfn %lx\n",
+ folio_nid(folio), pfn);
+ folio_put(folio);
+ err = -EBUSY;
+ break;
+ }
+
if (!isolate_folio_to_list(folio, &source)) {
if (__ratelimit(&migrate_rs)) {
pr_warn("failed to isolate pfn %lx\n",
@@ -1931,6 +1942,7 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
putback_movable_pages(&source);
}
}
+ return err;
}
static int __init cmdline_parse_movable_node(char *p)
@@ -2067,10 +2079,11 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
ret = scan_movable_pages(pfn, end_pfn, &pfn);
if (!ret) {
/*
- * TODO: fatal migration failures should bail
- * out
+ * A fatal migration failure (e.g. a folio on a
+ * non-migratable private node) bails out; transient
+ * failures leave ret zero and are retried below.
*/
- do_migrate_range(pfn, end_pfn);
+ ret = do_migrate_range(pfn, end_pfn);
}
} while (!ret);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (6 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 07/36] mm/memory_hotplug: disallow migration-driven private node hotunplug Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:22 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes Gregory Price
` (30 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Private nodes are already kept out of policy nodemasks (only N_MEMORY
nodes are allowed), but an mbind(MPOL_MF_MOVE) walk can still encounter a
private-node folio in the range. Skip such folios so mempolicy-driven
migration never moves private-node memory.
We already do this for ZONE_DEVICE folios - use the same filter points
for ZONE_DEVICE (with an additional point for large pages).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 5720f7f54d942..8e8763f6e5f5b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -111,6 +111,7 @@
#include <linux/mmu_notifier.h>
#include <linux/printk.h>
#include <linux/leafops.h>
+#include <linux/node_private.h>
#include <linux/gcd.h>
#include <asm/tlbflush.h>
@@ -668,6 +669,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
}
if (!queue_folio_required(folio, qp))
return;
+ if (folio_is_private_node(folio))
+ return;
if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma) ||
!migrate_folio_add(folio, qp->pagelist, qp->flags))
@@ -722,7 +725,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
continue;
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
if (folio_test_large(folio) && max_nr != 1)
nr = folio_pte_batch(folio, pte, ptent, max_nr);
@@ -797,6 +800,8 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
folio = pfn_folio(pte_pfn(ptep));
if (!queue_folio_required(folio, qp))
goto unlock;
+ if (folio_is_private_node(folio))
+ goto unlock;
if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma)) {
qp->nr_failed++;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration
2026-07-20 19:34 ` [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration Gregory Price
@ 2026-07-21 18:22 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:22 UTC (permalink / raw)
To: Gregory Price; +Cc: tj, hannes, nvdimm, mkoutny, damon, kvm, linux-cxl
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Pre-existing issues:
- [Critical] Blind conversion of PMD to folio in queue_folios_pmd() causes out-of-bounds dereference on raw PFN mappings.
- [High] queue_folios_pte_range() sets walk->action = ACTION_AGAIN when pte_offset_map_lock() returns NULL, leading to an infinite retry loop for migration entries.
--
--- Patch [8]: [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration ---
commit 0603242bf30dacf881c78a4de8f6d67872a88c83
Author: Gregory Price <gourry@gourry.net>
Subject: mm/mempolicy: skip private node folios when queueing for migration
Private nodes are already kept out of policy nodemasks (only N_MEMORY
nodes are allowed), but an mbind(MPOL_MF_MOVE) walk can still encounter a
private-node folio in the range. Skip such folios so mempolicy-driven
migration never moves private-node memory.
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 5720f7f54d942..8e8763f6e5f5b 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
[ ... ]
> @@ -668,6 +669,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
> }
> if (!queue_folio_required(folio, qp))
> return;
> + if (folio_is_private_node(folio))
> + return;
> if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
> !vma_migratable(walk->vma) ||
> !migrate_folio_add(folio, qp->pagelist, qp->flags))
[Severity: Critical]
This is a pre-existing issue, but can this cause an out-of-bounds
dereference on raw PFN mappings?
Earlier in mm/mempolicy.c:queue_folios_pmd(), the code unconditionally
calls pmd_folio() and dereferences it:
folio = pmd_folio(pmdval);
if (is_huge_zero_folio(folio)) {
If an unprivileged user calls mbind(..., MPOL_MF_STRICT) on a memory
region mapped by a driver using raw PFNs via huge PMDs (e.g.,
vmf_insert_pfn_pmd() in a VM_PFNMAP VMA), pmd_trans_huge_lock() will
succeed. However, treating a raw PFN as a struct folio yields a bogus
pointer, leading to a kernel panic when accessed. Does this need a check
like pmd_special() to filter out raw PFN mappings?
> @@ -722,7 +725,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
> continue;
> }
> folio = vm_normal_folio(vma, addr, ptent);
> - if (!folio || folio_is_zone_device(folio))
> + if (!folio || folio_is_private_managed(folio))
> continue;
> if (folio_test_large(folio) && max_nr != 1)
> nr = folio_pte_batch(folio, pte, ptent, max_nr);
[Severity: High]
This is a pre-existing issue, but can this lead to an infinite retry loop?
Just above this section in mm/mempolicy.c:queue_folios_pte_range(), there
is:
mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
if (!pte) {
walk->action = ACTION_AGAIN;
return 0;
}
Since pte_offset_map_lock() returns NULL non-transiently for migration
entries, setting walk->action = ACTION_AGAIN and returning 0 on this
failure instructs walk_pte_range() to retry the exact same PMD
indefinitely with no limit. Because this early return skips the
cond_resched() at the bottom of the function, it creates a tight CPU
livelock. Should this instead return 0 without setting ACTION_AGAIN to
gracefully skip the entry?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=8
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (7 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:32 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios Gregory Price
` (29 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Use the same filter locations as zone_device folios to disallow
userland driven migration requests for private node memory.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/migrate.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index a8a86141dbb54..d20674c07b947 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2268,7 +2268,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
return -EFAULT;
- if (folio_is_zone_device(folio))
+ if (folio_is_private_managed(folio))
return -ENOENT;
if (folio_nid(folio) == node)
@@ -2477,7 +2477,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
if (folio) {
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
err = -EFAULT;
- else if (folio_is_zone_device(folio))
+ else if (folio_is_private_managed(folio))
err = -ENOENT;
else
err = folio_nid(folio);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes
2026-07-20 19:34 ` [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes Gregory Price
@ 2026-07-21 18:32 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:32 UTC (permalink / raw)
To: Gregory Price; +Cc: nvdimm, linux-cxl, kvm, mkoutny, damon, tj, hannes
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [High] Incomplete replacement of `zone_device` checks allows uncoordinated background migrations of private node memory.
- [High] node_private_unregister() permanently returns -EBUSY for private nodes because offline_pages() never clears N_MEMORY_PRIVATE.
Pre-existing issues:
- [High] User-space addresses are not stripped of memory tags (e.g., ARM64 MTE) before VMA lookup in do_pages_stat_array(), causing valid tagged addresses to fail with -EFAULT.
--
--- Patch [9]: [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes ---
commit 7ed55bccdffe7ca2bf917693b091b54ebe294b02
Author: Gregory Price <gourry@gourry.net>
mm/migrate: disallow userland driven migration for private nodes
Use the same filter locations as zone_device folios to disallow
userland driven migration requests for private node memory.
> diff --git a/mm/migrate.c b/mm/migrate.c
> index a8a86141dbb54..d20674c07b947 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2268,7 +2268,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
> if (is_zero_folio(folio) || is_huge_zero_folio(folio))
> return -EFAULT;
>
> - if (folio_is_zone_device(folio))
> + if (folio_is_private_managed(folio))
[Severity: High]
By updating these filters, do we also need to update similar checks in
uncoordinated background migration paths?
If background tasks encounter a private node folio, they will evaluate
folio_is_zone_device() or is_zone_device_page() as false. Could this allow
NUMA balancing, KSM, or khugepaged to silently migrate private node memory
to general-purpose nodes?
For example, in NUMA balancing:
mm/memory.c:do_numa_page() {
...
if (!folio || folio_is_zone_device(folio))
goto out_map;
...
}
In THP collapse:
mm/khugepaged.c:khugepaged_scan_pmd() {
...
if (WARN_ON_ONCE(page && is_zone_device_page(page)))
page = NULL;
...
}
And in KSM:
mm/ksm.c:get_mergeable_page() {
...
if (!folio_is_zone_device(folio) &&
folio_test_anon(folio)) {
...
}
> return -ENOENT;
>
> if (folio_nid(folio) == node)
> @@ -2477,7 +2477,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
> if (folio) {
> if (is_zero_folio(folio) || is_huge_zero_folio(folio))
> err = -EFAULT;
> - else if (folio_is_zone_device(folio))
> + else if (folio_is_private_managed(folio))
[Severity: High]
This isn't a bug introduced by this patch, but does do_pages_stat_array() fail
to handle tagged pointers?
When a user-space program passes an array of tagged pointers to move_pages
with nodes == NULL, the pointers are cast directly to unsigned long without
being sanitized by untagged_addr_remote():
mm/migrate.c:do_pages_stat_array() {
...
unsigned long addr = (unsigned long)(*pages);
struct vm_area_struct *vma;
// ...
vma = vma_lookup(mm, addr);
...
}
Could this cause vma_lookup() to treat the tag bits as part of the virtual
address, causing valid tagged addresses (like those used with ARM64 MTE) to
fail with -EFAULT?
> err = -ENOENT;
> else
> err = folio_nid(folio);
[Severity: High]
On a related note regarding private node teardown, does offline_pages() need
to clear N_MEMORY_PRIVATE?
If a driver attempts to teardown a device and offline its memory,
offline_pages() clears N_MEMORY but omits N_MEMORY_PRIVATE:
mm/memory_hotplug.c:offline_pages() {
...
if (node_arg.nid >= 0)
node_clear_state(node, N_MEMORY);
...
}
Will this cause node_private_unregister() to evaluate node_is_private(nid)
as true and permanently return -EBUSY?
drivers/base/node.c:node_private_unregister() {
...
if (node_is_private(nid)) {
mem_hotplug_done();
mutex_unlock(&node_private_lock);
return -EBUSY;
}
...
}
Could this permanent -EBUSY prevent drivers using the N_MEMORY_PRIVATE
feature from cleanly unloading?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=9
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (8 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:30 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes Gregory Price
` (28 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Use the same filter locations as zone_device, plus additional filters
for huge pages to avoid madvise operations on private node memory.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/huge_memory.c | 5 +++++
mm/madvise.c | 8 ++++++--
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 5bd8d4f59a7b8..1df91b4e5c2bc 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -37,6 +37,7 @@
#include <linux/page_owner.h>
#include <linux/sched/sysctl.h>
#include <linux/memory-tiers.h>
+#include <linux/node_private.h>
#include <linux/compat.h>
#include <linux/pgalloc.h>
#include <linux/pgalloc_tag.h>
@@ -2338,6 +2339,10 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
}
folio = pmd_folio(orig_pmd);
+
+ if (folio_is_private_node(folio))
+ goto out;
+
/*
* If other processes are mapping this folio, we couldn't discard
* the folio unless they all do MADV_FREE so let's skip the folio.
diff --git a/mm/madvise.c b/mm/madvise.c
index 07a21ca31bad4..29f35a23919a0 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -32,6 +32,7 @@
#include <linux/leafops.h>
#include <linux/shmem_fs.h>
#include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
#include <asm/tlb.h>
@@ -395,6 +396,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
folio = pmd_folio(orig_pmd);
+ if (folio_is_private_node(folio))
+ goto huge_unlock;
+
/* Do not interfere with other mappings of this folio */
if (folio_maybe_mapped_shared(folio))
goto huge_unlock;
@@ -474,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
continue;
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
/*
@@ -703,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
/*
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios
2026-07-20 19:34 ` [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios Gregory Price
@ 2026-07-21 18:30 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:30 UTC (permalink / raw)
To: Gregory Price; +Cc: damon, linux-cxl, kvm, tj, hannes, mkoutny, nvdimm
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] madvise_cold_or_pageout_pte_range() calls folio_get() before blocking on folio_lock() when attempting to split a large PMD-mapped folio, violating the required lock-before-refcount ordering for large folio splits.
--
--- Patch [10]: [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios ---
commit 76f9465747c7c18f9f36e91d7a628706fb9905f9
Author: Gregory Price <gourry@gourry.net>
mm/madvise: disallow madvise operations on private node folios
Use the same filter locations as zone_device, plus additional filters
for huge pages to avoid madvise operations on private node memory.
[ ... ]
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 07a21ca31bad4..29f35a23919a0 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
[ ... ]
> @@ -395,6 +396,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
>
> folio = pmd_folio(orig_pmd);
>
> + if (folio_is_private_node(folio))
> + goto huge_unlock;
> +
> /* Do not interfere with other mappings of this folio */
> if (folio_maybe_mapped_shared(folio))
> goto huge_unlock;
[Severity: High]
This is a pre-existing issue, but does the sequence just below this block
in madvise_cold_or_pageout_pte_range() cause split_folio() to fail under
contention?
A few lines down, the code attempts to split a PMD-mapped large folio:
folio_get(folio);
spin_unlock(ptl);
folio_lock(folio);
err = split_folio(folio);
When multiple tasks concurrently attempt to split the same folio, taking
the reference before blocking on the lock inflates the refcount. This
causes split_folio() to spuriously return -EAGAIN because the expected
refcount does not match.
Could we trylock the folio before raising the reference count here,
similar to how it is handled in madvise_free_huge_pmd() in huge_memory.c?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=10
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (9 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:22 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 12/36] mm/page_alloc: clear private node watermarks and system reserves Gregory Price
` (27 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Skip compaction on private nodes. Compaction requires the ability to
migrate pages, which is not yet supported on private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/compaction.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/mm/compaction.c b/mm/compaction.c
index 9f81055a358ed..8c1351cce7bcc 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2462,6 +2462,9 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
+ if (node_is_private(zone_to_nid(zone)))
+ continue;
+
/*
* Do not consider all the reclaimable memory because we do not
* want to trash just for a single high order allocation which
@@ -2853,6 +2856,9 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
+ if (node_is_private(zone_to_nid(zone)))
+ continue;
+
if (prio > MIN_COMPACT_PRIORITY
&& compaction_deferred(zone, order)) {
rc = max_t(enum compact_result, COMPACT_DEFERRED, rc);
@@ -2922,6 +2928,9 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
.proactive_compaction = proactive,
};
+ if (node_is_private(pgdat->node_id))
+ return 0;
+
for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
zone = &pgdat->node_zones[zoneid];
if (!populated_zone(zone))
@@ -3017,6 +3026,9 @@ static ssize_t compact_store(struct device *dev,
{
int nid = dev->id;
+ if (node_is_private(nid))
+ return -EINVAL;
+
if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
/* Flush pending updates to the LRU lists */
lru_add_drain_all();
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes
2026-07-20 19:34 ` [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes Gregory Price
@ 2026-07-21 18:22 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:22 UTC (permalink / raw)
To: Gregory Price; +Cc: linux-cxl, nvdimm, tj, damon, hannes, kvm, mkoutny
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The kcompactd background compaction daemon is not prevented from compacting private nodes.
--
--- Patch [11]: [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes ---
commit 8f0db5de5c5b2025b3451b6bb90c3cd06411f91d
Author: Gregory Price <gourry@gourry.net>
mm/compaction: disallow compaction on private nodes
This commit skips compaction on private nodes because they do not yet support
page migration. It adds checks to direct compaction and sysfs paths to abort
compaction early for private nodes.
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 9f81055a358ed..8c1351cce7bcc 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
[ ... ]
> @@ -3017,6 +3026,9 @@ static ssize_t compact_store(struct device *dev,
> {
> int nid = dev->id;
>
> + if (node_is_private(nid))
> + return -EINVAL;
> +
> if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
> /* Flush pending updates to the LRU lists */
> lru_add_drain_all();
[Severity: High]
The commit message states we should disallow compaction on private nodes
because migration is not yet supported. Does this patch miss the background
compaction daemon paths?
Under memory pressure, kswapd calls wakeup_kcompactd(), which currently
bypasses private node checks:
mm/compaction.c:wakeup_kcompactd() {
...
if (!kcompactd_node_suitable(pgdat))
return;
wake_up_interruptible(&pgdat->kcompactd_wait);
}
This wakes up the daemon which will then unconditionally execute
compact_zone() for the node without verifying if the node is private:
mm/compaction.c:kcompactd_do_work() {
...
for (zoneid = 0; zoneid <= cc.highest_zoneidx; zoneid++) {
...
status = compact_zone(&cc, NULL);
...
}
Could this result in the background daemon uselessly waking up and
attempting unsupported page migration on private nodes?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=11
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 12/36] mm/page_alloc: clear private node watermarks and system reserves
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (10 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 13/36] mm/mempolicy: disallow NUMA Balancing prot_none on private nodes Gregory Price
` (26 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Set private node watermarks to 0 (reclaim isn't supported yet, so they
are pointless), and do not include private nodes in system reserves.
The oom killer can't actually resolve general system pressure by
releasing private node capacity, so including them in system reserve
calculations dilutes DRAM watermarks and causes poor OOM kill behavior.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/page_alloc.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 78755a55b1540..2b08bea2379a9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6555,6 +6555,10 @@ static void calculate_totalreserve_pages(void)
pgdat->totalreserve_pages = 0;
+ /* private nodes have zero watermarks */
+ if (node_is_private(pgdat->node_id))
+ continue;
+
for (i = 0; i < MAX_NR_ZONES; i++) {
struct zone *zone = pgdat->node_zones + i;
long max = 0;
@@ -6647,9 +6651,15 @@ static void __setup_per_zone_wmarks(void)
struct zone *zone;
unsigned long flags;
- /* Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages */
+ /*
+ * Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages
+ *
+ * Private nodes are excluded because they are not in the regular
+ * allocation fallback path - including them dilutes DRAM watermarks.
+ */
for_each_zone(zone) {
- if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE)
+ if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE &&
+ !node_is_private(zone_to_nid(zone)))
lowmem_pages += zone_managed_pages(zone);
}
@@ -6657,6 +6667,16 @@ static void __setup_per_zone_wmarks(void)
u64 tmp;
spin_lock_irqsave(&zone->lock, flags);
+ if (node_is_private(zone_to_nid(zone))) {
+ zone->_watermark[WMARK_MIN] = 0;
+ zone->_watermark[WMARK_LOW] = 0;
+ zone->_watermark[WMARK_HIGH] = 0;
+ zone->_watermark[WMARK_PROMO] = 0;
+ zone->watermark_boost = 0;
+ spin_unlock_irqrestore(&zone->lock, flags);
+ continue;
+ }
+
tmp = (u64)pages_min * zone_managed_pages(zone);
tmp = div64_ul(tmp, lowmem_pages);
if (is_highmem(zone) || zone_idx(zone) == ZONE_MOVABLE) {
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 13/36] mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (11 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 12/36] mm/page_alloc: clear private node watermarks and system reserves Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout Gregory Price
` (25 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Skip private node memory the same way zone device memory is
skipped, since private nodes do not presently support migration.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 8e8763f6e5f5b..2b76c57a460c9 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -845,7 +845,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
{
int nid;
- if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
+ if (!folio || folio_is_private_managed(folio) || folio_test_ksm(folio))
return false;
/* Also skip shared copy-on-write folios */
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (12 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 13/36] mm/mempolicy: disallow NUMA Balancing prot_none on private nodes Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:18 ` sashiko-bot
2026-07-21 23:46 ` SJ Park
2026-07-20 19:34 ` [PATCH v5 15/36] mm/ksm: skip KSM for managed-memory folios Gregory Price
` (24 subsequent siblings)
38 siblings, 2 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
DAMON operates on physical address ranges, which can cover private
node memory. Skip private-node folios in both DAMON's migration
and reclaim paths.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/damon/paddr.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index e4f98d67461f5..c741a94319750 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -12,6 +12,7 @@
#include <linux/swap.h>
#include <linux/memory-tiers.h>
#include <linux/mm_inline.h>
+#include <linux/node_private.h>
#include "../internal.h"
#include "ops-common.h"
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
continue;
}
+ /* private node memory is not reclaimable by default */
+ if (folio_is_private_node(folio))
+ goto put_folio;
+
if (damos_pa_filter_out(s, folio))
goto put_folio;
else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
else
*sz_filter_passed += folio_size(folio) / addr_unit;
+ /* private nodes do not support migration by default */
+ if (folio_is_private_node(folio))
+ goto put_folio;
+
if (!folio_isolate_lru(folio))
goto put_folio;
list_add(&folio->lru, &folio_list);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-20 19:34 ` [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout Gregory Price
@ 2026-07-21 18:18 ` sashiko-bot
2026-07-21 23:46 ` SJ Park
1 sibling, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:18 UTC (permalink / raw)
To: Gregory Price; +Cc: nvdimm, hannes, damon, kvm, tj, mkoutny, linux-cxl
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Inconsistent placement of the `folio_is_private_node()` check causes incorrect `sz_filter_passed` accounting and unnecessary filter side-effects in `damon_pa_migrate()` compared to `damon_pa_pageout()`.
--
--- Patch [14]: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout ---
commit eb373ee8a3d83be1e35d66f74d2720ee3b1c26fa
Author: Gregory Price <gourry@gourry.net>
mm/damon: skip private node memory in DAMON migration and pageout
This commit updates DAMON's physical address operations to skip private-node
folios during both migration and pageout reclaim paths.
> diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
> index e4f98d67461f5..c741a94319750 100644
> --- a/mm/damon/paddr.c
> +++ b/mm/damon/paddr.c
> @@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
> continue;
> }
>
> + /* private node memory is not reclaimable by default */
> + if (folio_is_private_node(folio))
> + goto put_folio;
> +
> if (damos_pa_filter_out(s, folio))
> goto put_folio;
> else
[ ... ]
> @@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
> else
> *sz_filter_passed += folio_size(folio) / addr_unit;
>
> + /* private nodes do not support migration by default */
> + if (folio_is_private_node(folio))
> + goto put_folio;
> +
[Severity: Medium]
Does placing this check after the filter evaluation cause incorrect DAMON
statistics?
Because the check happens here rather than before the filter evaluation, it
appears unmigratable private node folios will be erroneously counted in
sz_filter_passed. This might also trigger unnecessary filter logic evaluations
before we ultimately skip the folio.
Could we move this up before damos_pa_filter_out() in damon_pa_migrate() so
it matches the logic flow used earlier in damon_pa_pageout()?
> if (!folio_isolate_lru(folio))
> goto put_folio;
> list_add(&folio->lru, &folio_list);
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=14
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-20 19:34 ` [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout Gregory Price
2026-07-21 18:18 ` sashiko-bot
@ 2026-07-21 23:46 ` SJ Park
2026-07-22 12:16 ` Gregory Price
1 sibling, 1 reply; 82+ messages in thread
From: SJ Park @ 2026-07-21 23:46 UTC (permalink / raw)
To: Gregory Price
Cc: SJ Park, linux-mm, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, jgg, jhubbard, peterx,
baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, 20 Jul 2026 15:34:08 -0400 Gregory Price <gourry@gourry.net> wrote:
> DAMON operates on physical address ranges, which can cover private
> node memory. Skip private-node folios in both DAMON's migration
> and reclaim paths.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> mm/damon/paddr.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
> index e4f98d67461f5..c741a94319750 100644
> --- a/mm/damon/paddr.c
> +++ b/mm/damon/paddr.c
> @@ -12,6 +12,7 @@
> #include <linux/swap.h>
> #include <linux/memory-tiers.h>
> #include <linux/mm_inline.h>
> +#include <linux/node_private.h>
>
> #include "../internal.h"
> #include "ops-common.h"
> @@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
> continue;
> }
>
> + /* private node memory is not reclaimable by default */
> + if (folio_is_private_node(folio))
> + goto put_folio;
> +
"by default". Does that mean it could be reclaimable in some situations? If
so, could we check if it is reclaimable?
Also, what happens if we just try paging out the private node memory? Will it
simply fail? Or, make some problems?
> if (damos_pa_filter_out(s, folio))
> goto put_folio;
> else
> @@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
> else
> *sz_filter_passed += folio_size(folio) / addr_unit;
>
> + /* private nodes do not support migration by default */
> + if (folio_is_private_node(folio))
> + goto put_folio;
> +
Same questions.
> if (!folio_isolate_lru(folio))
> goto put_folio;
> list_add(&folio->lru, &folio_list);
> --
> 2.53.0-Meta
Thanks,
SJ
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-21 23:46 ` SJ Park
@ 2026-07-22 12:16 ` Gregory Price
2026-07-23 0:19 ` SJ Park
0 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-22 12:16 UTC (permalink / raw)
To: SJ Park
Cc: linux-mm, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, jgg, jhubbard, peterx,
baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Tue, Jul 21, 2026 at 04:46:43PM -0700, SJ Park wrote:
> On Mon, 20 Jul 2026 15:34:08 -0400 Gregory Price <gourry@gourry.net> wrote:
>
> > DAMON operates on physical address ranges, which can cover private
> > node memory. Skip private-node folios in both DAMON's migration
> > and reclaim paths.
> >
> > Signed-off-by: Gregory Price <gourry@gourry.net>
> > ---
> > mm/damon/paddr.c | 9 +++++++++
> > 1 file changed, 9 insertions(+)
> >
> > diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
> > index e4f98d67461f5..c741a94319750 100644
> > --- a/mm/damon/paddr.c
> > +++ b/mm/damon/paddr.c
> > @@ -12,6 +12,7 @@
> > #include <linux/swap.h>
> > #include <linux/memory-tiers.h>
> > #include <linux/mm_inline.h>
> > +#include <linux/node_private.h>
> >
> > #include "../internal.h"
> > #include "ops-common.h"
> > @@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
> > continue;
> > }
> >
> > + /* private node memory is not reclaimable by default */
> > + if (folio_is_private_node(folio))
> > + goto put_folio;
> > +
>
> "by default". Does that mean it could be reclaimable in some situations? If
> so, could we check if it is reclaimable?
>
See the damon changes in:
https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a
> Also, what happens if we just try paging out the private node memory? Will it
> simply fail? Or, make some problems?
>
user controls for MADV_ commands that affect private node folios would
skip those folios - exactly same as ZONE_DEVICE folios.
Which I'm not sure damon actually handles correctly presently (zone
device). I remember looking around damon and thinking to myself "what
if this is a zone device page?" and not being able to find a satisfying
answer.
> > if (damos_pa_filter_out(s, folio))
> > goto put_folio;
> > else
> > @@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
> > else
> > *sz_filter_passed += folio_size(folio) / addr_unit;
> >
> > + /* private nodes do not support migration by default */
> > + if (folio_is_private_node(folio))
> > + goto put_folio;
> > +
>
> Same questions.
>
Technically there is nothing in migration core to prevent migration
operations, it's done on a service basis - hotunplug, reclaim/demotion,
user numa (mbind, migrate/move_pages) etc.
Operations on private nodes/private node folioes are refused if the
capability bit is not set.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-22 12:16 ` Gregory Price
@ 2026-07-23 0:19 ` SJ Park
2026-07-23 3:25 ` Gregory Price
0 siblings, 1 reply; 82+ messages in thread
From: SJ Park @ 2026-07-23 0:19 UTC (permalink / raw)
To: Gregory Price
Cc: SJ Park, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, jgg, jhubbard, peterx,
baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <gourry@gourry.net> wrote:
> On Tue, Jul 21, 2026 at 04:46:43PM -0700, SJ Park wrote:
> > On Mon, 20 Jul 2026 15:34:08 -0400 Gregory Price <gourry@gourry.net> wrote:
> >
> > > DAMON operates on physical address ranges, which can cover private
> > > node memory. Skip private-node folios in both DAMON's migration
> > > and reclaim paths.
> > >
> > > Signed-off-by: Gregory Price <gourry@gourry.net>
> > > ---
> > > mm/damon/paddr.c | 9 +++++++++
> > > 1 file changed, 9 insertions(+)
> > >
> > > diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
> > > index e4f98d67461f5..c741a94319750 100644
> > > --- a/mm/damon/paddr.c
> > > +++ b/mm/damon/paddr.c
> > > @@ -12,6 +12,7 @@
> > > #include <linux/swap.h>
> > > #include <linux/memory-tiers.h>
> > > #include <linux/mm_inline.h>
> > > +#include <linux/node_private.h>
> > >
> > > #include "../internal.h"
> > > #include "ops-common.h"
> > > @@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
> > > continue;
> > > }
> > >
> > > + /* private node memory is not reclaimable by default */
> > > + if (folio_is_private_node(folio))
> > > + goto put_folio;
> > > +
> >
> > "by default". Does that mean it could be reclaimable in some situations? If
> > so, could we check if it is reclaimable?
> >
>
> See the damon changes in:
>
> https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a
Summarizing what you want to say, quoting something from the patch, or at least
calling it "26th patch of this series" would have made reviewing much easier.
Cc-ing damon@ for only patches that toucing DAMON source files and the cover
letter of the series could also be helpful. Please consider doing some of
these for future replies.
So, I understand later patches will make it optionally reclaimable and update
this restriction by the 26th patch? That sounds fair. But, could we drop "by
default" from the above comment for reducing the confusion?
>
> > Also, what happens if we just try paging out the private node memory? Will it
> > simply fail? Or, make some problems?
> >
>
> user controls for MADV_ commands that affect private node folios would
> skip those folios - exactly same as ZONE_DEVICE folios.
>
> Which I'm not sure damon actually handles correctly presently (zone
> device). I remember looking around damon and thinking to myself "what
> if this is a zone device page?" and not being able to find a satisfying
> answer.
>
> > > if (damos_pa_filter_out(s, folio))
> > > goto put_folio;
> > > else
> > > @@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
> > > else
> > > *sz_filter_passed += folio_size(folio) / addr_unit;
> > >
> > > + /* private nodes do not support migration by default */
> > > + if (folio_is_private_node(folio))
> > > + goto put_folio;
> > > +
> >
> > Same questions.
> >
>
> Technically there is nothing in migration core to prevent migration
> operations, it's done on a service basis - hotunplug, reclaim/demotion,
> user numa (mbind, migrate/move_pages) etc.
>
> Operations on private nodes/private node folioes are refused if the
> capability bit is not set.
I haven't had a chance to read the entire series, sorry about that. So, do you
mean the above blocking is not really needed, or that will conditionally be
allowed by another later patch?
Thanks,
SJ
[...]
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-23 0:19 ` SJ Park
@ 2026-07-23 3:25 ` Gregory Price
2026-07-23 13:36 ` SJ Park
0 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-23 3:25 UTC (permalink / raw)
To: SJ Park
Cc: arun.george, balbirs, brendan.jackman, yuzenghui, apopple,
alucerop, matthew.brost, akpm, david, ljs, liam, vbabka, rppt,
surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, jgg, jhubbard, peterx,
baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 05:19:42PM -0700, SJ Park wrote:
> On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <gourry@gourry.net> wrote:
>
> > > "by default". Does that mean it could be reclaimable in some situations? If
> > > so, could we check if it is reclaimable?
> > >
> >
> > See the damon changes in:
> >
> > https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a
>
> Summarizing what you want to say, quoting something from the patch, or at least
> calling it "26th patch of this series" would have made reviewing much easier.
> Cc-ing damon@ for only patches that toucing DAMON source files and the cover
> letter of the series could also be helpful. Please consider doing some of
> these for future replies.
>
I have been very much discouraged from this kind of trimming because it
removes the context of the series from the individual patches - making it
even more confusing.
But I understand, I will try to remember to do this for you.
> So, I understand later patches will make it optionally reclaimable and update
> this restriction by the 26th patch? That sounds fair. But, could we drop "by
> default" from the above comment for reducing the confusion?
Yes, but you bring up a good point - I think this two-step process is
actually poor form and that I need to redo it.
Maybe this can be redone as node states (features?) such as:
N_MEMORY_RECLAIMABLE
N_MEMORY_DEMOTABLE
N_MEMORY_TIERABLE
so these damon changes will forego this confusing intermediate step of:
if (folio_is_private_node(folio))
and go straight to
if (node_is_reclaimable(folio_nid(folio)))
In the first patch - without ever having to revisit damon again.
> >
> > Technically there is nothing in migration core to prevent migration
> > operations, it's done on a service basis - hotunplug, reclaim/demotion,
> > user numa (mbind, migrate/move_pages) etc.
> >
> > Operations on private nodes/private node folioes are refused if the
> > capability bit is not set.
>
> I haven't had a chance to read the entire series, sorry about that. So, do you
> mean the above blocking is not really needed, or that will conditionally be
> allowed by another later patch?
>
I will reduce this all to a single patch in v6, I see where things can
be improved now based on a few pieces of feedback.
Thanks SJ!
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
2026-07-23 3:25 ` Gregory Price
@ 2026-07-23 13:36 ` SJ Park
0 siblings, 0 replies; 82+ messages in thread
From: SJ Park @ 2026-07-23 13:36 UTC (permalink / raw)
To: Gregory Price
Cc: SJ Park, balbirs, brendan.jackman, yuzenghui, apopple, alucerop,
matthew.brost, akpm, david, ljs, liam, vbabka, rppt, surenb,
mhocko, corbet, skhan, gregkh, rafael, dakr, djbw, vishal.l.verma,
dave.jiang, alison.schofield, osandov, jannh, pfalcato, jackmanb,
hannes, ziy, pbonzini, osalvador, joshua.hahnjy, rakie.kim,
byungchul, ying.huang, kasong, qi.zheng, shakeel.butt, baohua,
axelrasmussen, yuanchu, weixugc, yury.norov, linux, longman,
ridong.chen, tj, mkoutny, jgg, jhubbard, peterx, baolin.wang,
npache, ryan.roberts, dev.jain, lance.yang, usama.arif, xu.xin16,
chengming.zhou, roman.gushchin, muchun.song, linux-kernel,
linux-doc, driver-core, nvdimm, linux-cxl, linux-debuggers,
linux-fsdevel, kvm, cgroups, damon, linux-kselftest, kernel-team
On Wed, 22 Jul 2026 23:25:15 -0400 Gregory Price <gourry@gourry.net> wrote:
> On Wed, Jul 22, 2026 at 05:19:42PM -0700, SJ Park wrote:
> > On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <gourry@gourry.net> wrote:
> >
> > > > "by default". Does that mean it could be reclaimable in some situations? If
> > > > so, could we check if it is reclaimable?
> > > >
> > >
> > > See the damon changes in:
> > >
> > > https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a
> >
> > Summarizing what you want to say, quoting something from the patch, or at least
> > calling it "26th patch of this series" would have made reviewing much easier.
> > Cc-ing damon@ for only patches that toucing DAMON source files and the cover
> > letter of the series could also be helpful. Please consider doing some of
> > these for future replies.
> >
>
> I have been very much discouraged from this kind of trimming because it
> removes the context of the series from the individual patches - making it
> even more confusing.
That's fair and making sense.
>
> But I understand, I will try to remember to do this for you.
Thank you, Gregory. Nonetheless, I wouldn't insist selective Cc-ing. I agree
your point, and I expect doing that would be tedious if you don't have a script
or tool (I use a tool). It is just one of my suggestion to help reviewing in
this context, insted of only lore link. Just one among the other three
suggestions (summarizing your thought, quoting, or simply calling it "26th
patch of this series") should also suffice.
>
> > So, I understand later patches will make it optionally reclaimable and update
> > this restriction by the 26th patch? That sounds fair. But, could we drop "by
> > default" from the above comment for reducing the confusion?
>
> Yes, but you bring up a good point - I think this two-step process is
> actually poor form and that I need to redo it.
>
> Maybe this can be redone as node states (features?) such as:
>
> N_MEMORY_RECLAIMABLE
> N_MEMORY_DEMOTABLE
> N_MEMORY_TIERABLE
>
> so these damon changes will forego this confusing intermediate step of:
>
> if (folio_is_private_node(folio))
>
> and go straight to
>
> if (node_is_reclaimable(folio_nid(folio)))
>
> In the first patch - without ever having to revisit damon again.
Indeed that sounds better than my suggestion.
>
> > >
> > > Technically there is nothing in migration core to prevent migration
> > > operations, it's done on a service basis - hotunplug, reclaim/demotion,
> > > user numa (mbind, migrate/move_pages) etc.
> > >
> > > Operations on private nodes/private node folioes are refused if the
> > > capability bit is not set.
> >
> > I haven't had a chance to read the entire series, sorry about that. So, do you
> > mean the above blocking is not really needed, or that will conditionally be
> > allowed by another later patch?
> >
>
> I will reduce this all to a single patch in v6, I see where things can
> be improved now based on a few pieces of feedback.
Sounds good, thank you Gregory. Looking forward to the next version!
Thanks,
SJ
[...]
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 15/36] mm/ksm: skip KSM for managed-memory folios
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (13 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse Gregory Price
` (23 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Since private memory is intended for explicit placement, allowing
KSM to operate on it by default would completely defeat the purpose.
Skip KSM merge and scan operations for private node folios using
the same filter location as zone_device folios.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/ksm.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index 47006f494fcb1..dcc5c46a29eac 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -827,7 +827,7 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (!folio_is_zone_device(folio) &&
+ if (!folio_is_private_managed(folio) &&
folio_test_anon(folio)) {
folio_get(folio);
page = fw.page;
@@ -2558,7 +2558,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
goto not_found_unlock;
folio = page_folio(page);
- if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+ if (unlikely(folio_is_private_managed(folio)) ||
+ !folio_test_anon(folio))
goto not_found_unlock;
page += ((addr & (PMD_SIZE - 1)) >> PAGE_SHIFT);
@@ -2582,7 +2583,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
continue;
folio = page_folio(page);
- if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+ if (unlikely(folio_is_private_managed(folio)) ||
+ !folio_test_anon(folio))
continue;
goto found_unlock;
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse.
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (14 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 15/36] mm/ksm: skip KSM for managed-memory folios Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:36 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory Gregory Price
` (22 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
A collapse operation causes new THP allocation to occur, and may
migrate memory from one node to another.
Handle this the same as zone_device for now (disallow collapse).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/khugepaged.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 89ce6bcbc376b..fb4378cc17b10 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -700,7 +700,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
goto out;
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+ if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
result = SCAN_PAGE_NULL;
goto out;
}
@@ -1687,7 +1687,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+ if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
result = SCAN_PAGE_NULL;
goto out_unmap;
}
@@ -1944,7 +1944,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
}
page = vm_normal_page(vma, addr, ptent);
- if (WARN_ON_ONCE(page && is_zone_device_page(page)))
+ if (WARN_ON_ONCE(page && page_is_private_managed(page)))
page = NULL;
/*
* Note that uprobe, debugger, or MAP_PRIVATE may change the
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse.
2026-07-20 19:34 ` [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse Gregory Price
@ 2026-07-21 18:36 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:36 UTC (permalink / raw)
To: Gregory Price; +Cc: damon, hannes, nvdimm, kvm, linux-cxl, tj, mkoutny
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] The replacement of `is_zone_device_page()` with `page_is_private_managed()` inside a `WARN_ON_ONCE` in `try_collapse_pte_mapped_thp()` triggers spurious kernel warnings and aborts PMD restoration for valid memory on private NUMA nodes.
Pre-existing issues:
- [Critical] Functions structurally modifying page tables under `mmap_write_lock()` fail to call `vma_start_write()`, leading to critical Use-After-Free race conditions with concurrent per-VMA lock readers (like MADV_DONTNEED).
--
--- Patch [16]: [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse. ---
commit c62053ea9e5edc37182a3d1e8367dfb0ed6fbf08
Author: Gregory Price <gourry@gourry.net>
mm/khugepaged: skip private node folios when trying to collapse.
A collapse operation causes new THP allocation to occur, and may
migrate memory from one node to another. Handle this the same as
zone_device for now (disallow collapse).
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 89ce6bcbc376b..fb4378cc17b10 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
[ ... ]
> @@ -1944,7 +1944,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
> }
>
> page = vm_normal_page(vma, addr, ptent);
> - if (WARN_ON_ONCE(page && is_zone_device_page(page)))
> + if (WARN_ON_ONCE(page && page_is_private_managed(page)))
[Severity: High]
Does replacing is_zone_device_page() with page_is_private_managed() inside
this WARN_ON_ONCE trigger spurious warnings and abort PMD restoration?
When a user maps a file-backed THP on a private NUMA node, temporarily
splits the PMD into PTEs, and then triggers a PMD collapse, we reach
try_collapse_pte_mapped_thp(). Because page_is_private_managed() can be
true for valid memory on private nodes, won't this trigger the warning
and unintentionally set page to NULL?
If panic_on_warn is enabled, this could be used by an unprivileged user
to reliably crash the system.
[Severity: Critical]
This is a pre-existing issue, but does try_collapse_pte_mapped_thp() need
to ensure vma_start_write() is called to prevent a use-after-free with
concurrent per-VMA lock readers?
When a thread registers a uprobe via register_for_each_vma(), it acquires
the mmap_write_lock() and eventually calls into this function:
kernel/events/uprobes.c:register_for_each_vma() {
...
mmap_write_lock(mm);
if (check_stable_address_space(mm))
goto unlock;
vma = find_vma(mm, info->vaddr);
...
}
try_collapse_pte_mapped_thp() then clears PMDs and frees PTE pages under
the assumption of exclusive access:
mm/khugepaged.c:try_collapse_pte_mapped_thp() {
...
result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
...
start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl);
...
}
However, if another thread concurrently executes a VMA-locked operation
like MADV_DONTNEED, it only holds the VMA read lock. Because
vma_start_write() was never called to drain these concurrent readers, the
two threads can race to free or access the same PTE page.
> page = NULL;
> /*
> * Note that uprobe, debugger, or MAP_PRIVATE may change the
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=16
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (15 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:35 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 18/36] mm/gup: disallow longterm pin of private node folios Gregory Price
` (21 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Private nodes do not support reclaim by default, so prevent all
reclaimers from operating on private nodes.
shrink_node() is the common chokepoint for most reclaimers (direct,
kswapd, proactive, mglru, etc), so it's the best place to filter.
DAMON's proactive pageout bypasses shrink_node() entirely, so we
disable DAMON in a separate commit as its own discrete service.
Other reclaim entry points are already covered - e.g. MGLRU's debugfs
interface rejects nids that are not N_MEMORY (N_MEMORY_PRIVATE is
mutually exclusive), and madvise pageout/cold skips private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/vmscan.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index e26c6931f5fde..86b2334c23b98 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6141,6 +6141,13 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
struct lruvec *target_lruvec;
bool reclaimable = false;
+ /*
+ * Private nodes do not support reclaim by default, filtering here
+ * captures all normal reclaim paths that may attempt eviction.
+ */
+ if (node_is_private(pgdat->node_id))
+ return;
+
if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
memset(&sc->nr, 0, sizeof(sc->nr));
lru_gen_shrink_node(pgdat, sc);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory
2026-07-20 19:34 ` [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory Gregory Price
@ 2026-07-21 18:35 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:35 UTC (permalink / raw)
To: Gregory Price; +Cc: damon, tj, hannes, linux-cxl, mkoutny, nvdimm, kvm
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Soft-limit reclaim bypasses the new private node filter, allowing eviction on private nodes.
--
--- Patch [17]: [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory ---
commit 9fc4bff86adde3091fee2924952f9af9700e6308
Author: Gregory Price <gourry@gourry.net>
mm/vmscan: disallow reclaim of private node memory
This commit attempts to prevent reclaim on private nodes by adding a filter
to shrink_node().
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index e26c6931f5fde..86b2334c23b98 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -6141,6 +6141,13 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
> struct lruvec *target_lruvec;
> bool reclaimable = false;
>
> + /*
> + * Private nodes do not support reclaim by default, filtering here
> + * captures all normal reclaim paths that may attempt eviction.
> + */
> + if (node_is_private(pgdat->node_id))
> + return;
> +
[Severity: High]
Does this filter successfully capture all normal reclaim paths?
When the system uses memcg v1 and a cgroup exceeds its soft limit, global
direct reclaim in shrink_zones() or kswapd in balance_pgdat() will invoke
memcg1_soft_limit_reclaim(). This bypasses shrink_node() entirely:
mm/vmscan.c:shrink_zones() {
...
nr_soft_reclaimed = memcg1_soft_limit_reclaim(zone->zone_pgdat,
sc->order, sc->gfp_mask,
&nr_soft_scanned);
...
}
This leads to mem_cgroup_shrink_node() directly calling shrink_lruvec()
without checking if the node is private:
mm/vmscan.c:mem_cgroup_shrink_node() {
...
shrink_lruvec(lruvec, &sc);
...
}
Will this allow unintended eviction on private nodes during soft limit reclaim?
> if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
> memset(&sc->nr, 0, sizeof(sc->nr));
> lru_gen_shrink_node(pgdat, sc);
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=17
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 18/36] mm/gup: disallow longterm pin of private node folios
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (16 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output Gregory Price
` (20 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
FOLL_LONGTERM on a N_MEMORY_PRIVATE node folio is wrong both ways the
GUP longterm path resolves it:
a ZONE_NORMAL folio would be pinned in place
a ZONE_MOVABLE folio would be migrated off first
A private node folio should do neither - the pin should fail in place.
Add folio_longterm_pin_forbidden() (true for any private node folio) and
reject such a pin in check_and_migrate_movable_pages_or_folios() before
any isolation, so nothing is migrated.
The gup-fast path defers a private folio to the slow path via
folio_allows_longterm_pin().
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/gup.c | 27 +++++++++++++++++++++++++--
mm/internal.h | 25 +++++++++++++++++++++++++
2 files changed, 50 insertions(+), 2 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 1d32e9a3dc79c..a7d4de223785c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -547,10 +547,10 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,
/*
* Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
* right zone, so fail and let the caller fall back to the slow
- * path.
+ * path. Fail for private-node folios here so slow path rejects.
*/
if (unlikely((flags & FOLL_LONGTERM) &&
- !folio_is_longterm_pinnable(folio))) {
+ !folio_allows_longterm_pin(folio))) {
folio_put_refs(folio, refs);
return NULL;
}
@@ -2389,12 +2389,35 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list,
return ret;
}
+/*
+ * True if any folio sits on a private node whose folios may not be longterm
+ * pinned. Such folios can neither be pinned nor migrated, so the whole pin
+ * must fail before migration. Checked before any isolation occurs.
+ */
+static bool pofs_has_ltpin_forbidden(struct pages_or_folios *pofs)
+{
+ struct folio *folio;
+ long i = 0;
+
+ for (folio = pofs_get_folio(pofs, i); folio;
+ folio = pofs_next_folio(folio, pofs, &i)) {
+ if (folio_longterm_pin_forbidden(folio))
+ return true;
+ }
+ return false;
+}
+
static long
check_and_migrate_movable_pages_or_folios(struct pages_or_folios *pofs)
{
LIST_HEAD(movable_folio_list);
unsigned long collected;
+ if (pofs_has_ltpin_forbidden(pofs)) {
+ pofs_unpin(pofs);
+ return -EFAULT;
+ }
+
collected = collect_longterm_unpinnable_folios(&movable_folio_list,
pofs);
if (!collected)
diff --git a/mm/internal.h b/mm/internal.h
index cb9f4a8342e32..85c460296cea1 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -110,6 +110,31 @@ static inline bool page_is_private_managed(struct page *page)
return folio_is_private_managed(page_folio(page));
}
+/*
+ * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
+ *
+ * checks folio_is_longterm_pinnable() rules plus private node permissions.
+ * private node permission is checked here to resolve circular header
+ * dependencies in folio_is_longterm_pinnable.
+ */
+static inline bool folio_allows_longterm_pin(struct folio *folio)
+{
+ return folio_is_longterm_pinnable(folio) &&
+ !folio_is_private_node(folio);
+}
+
+/*
+ * folio_longterm_pin_forbidden() - must a longterm pin of this folio fail
+ * outright (neither pinned in place nor migrated off the node)?
+ *
+ * True for any folio on a private node: such memory can be neither pinned
+ * nor migrated, so the pin must be rejected with the folio left in place.
+ */
+static inline bool folio_longterm_pin_forbidden(struct folio *folio)
+{
+ return folio_is_private_node(folio);
+}
+
/*
* Maintains state across a page table move. The operation assumes both source
* and destination VMAs already exist and are specified by the user.
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (17 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 18/36] mm/gup: disallow longterm pin of private node folios Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:30 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats Gregory Price
` (19 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
numa_maps collects per-node page counts in the page-table walkers
and emits them in show_numa_map. All three filtered by N_MEMORY,
so pages on N_MEMORY_PRIVATE nodes were never gathered/printed.
Accept N_MEMORY_PRIVATE pages in both walkers and emit private nodes
in show_numa_map, so private-node mappings are visible in numa_maps.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
fs/proc/task_mmu.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 817e3e0f91943..e116cd5f157b9 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -3389,7 +3389,8 @@ static struct page *can_gather_numa_stats(pte_t pte, struct vm_area_struct *vma,
return NULL;
nid = page_to_nid(page);
- if (!node_isset(nid, node_states[N_MEMORY]))
+ if (!node_isset(nid, node_states[N_MEMORY]) &&
+ !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
return NULL;
return page;
@@ -3414,7 +3415,8 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
return NULL;
nid = page_to_nid(page);
- if (!node_isset(nid, node_states[N_MEMORY]))
+ if (!node_isset(nid, node_states[N_MEMORY]) &&
+ !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
return NULL;
return page;
@@ -3602,6 +3604,10 @@ static int show_numa_map(struct seq_file *m, void *v)
if (md->node[nid])
seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+ for_each_node_state(nid, N_MEMORY_PRIVATE)
+ if (md->node[nid])
+ seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+
seq_printf(m, " kernelpagesize_kB=%lu", vma_kernel_pagesize(vma) >> 10);
out:
seq_putc(m, '\n');
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output
2026-07-20 19:34 ` [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output Gregory Price
@ 2026-07-21 18:30 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:30 UTC (permalink / raw)
To: Gregory Price; +Cc: kvm, hannes, nvdimm, mkoutny, linux-cxl, damon, tj
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Using ACTION_AGAIN when pte_offset_map_lock() fails can create an infinite loop.
--
--- Patch [19]: [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output ---
commit 744e7cd9225bc94011608ebdc2dbdb49cbdf8476
Author: Gregory Price <gourry@gourry.net>
proc: include N_MEMORY_PRIVATE nodes in numa_maps output
numa_maps collects per-node page counts in the page-table walkers
and emits them in show_numa_map. All three filtered by N_MEMORY,
so pages on N_MEMORY_PRIVATE nodes were never gathered/printed.
Accept N_MEMORY_PRIVATE pages in both walkers and emit private nodes
in show_numa_map, so private-node mappings are visible in numa_maps.
> diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
> index 817e3e0f91943..e116cd5f157b9 100644
> --- a/fs/proc/task_mmu.c
> +++ b/fs/proc/task_mmu.c
[ ... ]
> @@ -3414,7 +3415,8 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
> return NULL;
>
> nid = page_to_nid(page);
> - if (!node_isset(nid, node_states[N_MEMORY]))
> + if (!node_isset(nid, node_states[N_MEMORY]) &&
> + !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
> return NULL;
>
> return page;
[Severity: High]
This is a pre-existing issue, but can the error handling in gather_pte_stats()
lead to an infinite loop?
When reading from procfs memory mapping files (like numa_maps),
pte_offset_map_lock() can fail non-transiently for a PMD migration entry and
return NULL:
fs/proc/task_mmu.c:gather_pte_stats() {
...
orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
if (!pte) {
walk->action = ACTION_AGAIN;
return 0;
}
...
}
Because this sets walk->action = ACTION_AGAIN, wouldn't the walk_pmd_range()
core unconditionally retry the same PMD entry indefinitely without making
forward progress?
The same pattern also appears to exist in smaps_pte_range(),
clear_refs_pte_range(), pagemap_pmd_range(), and pagemap_scan_pmd_entry().
[ ... ]
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=19
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (18 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:33 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map Gregory Price
` (18 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Private nodes folios are charged like any other - the node's per-cpu
lruvec counters are increment even for N_MEMORY_PRIVATE.
The memcg-level totals in memory.stat therefore include this memory,
but the per-node views do not:
- mem_cgroup_css_rstat_flush() folds the per-node percpu deltas into the
aggregate only for N_MEMORY nodes, so a private node's lruvec aggregate
is never refreshed.
- memory.numa_stat (v2) and the v1 numa_stat skip private nodes, so they
never emit an Nx= field for one.
So memory.numa_stat does not sum to memory.stat once a cgroup has memory
on a private node.
Include N_MEMORY_PRIVATE in the flush/display process so private-node
memory is folded in and reported per node.
This is pure accounting of memory that already exists and is already
charged, so it is unconditional - regardless of future capabilities.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/memcontrol-v1.c | 8 ++++++--
mm/memcontrol.c | 11 +++++++++--
2 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 2dc599484d006..5e464e52dfb9a 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -2132,14 +2132,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)
const struct numa_stat *stat;
int nid;
struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+ nodemask_t reportable;
mem_cgroup_flush_stats(memcg);
+ /* Private nodes hold cgroup memory too, report them. */
+ nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+
for (stat = stats; stat < ARRAY_END(stats); stat++) {
seq_printf(m, "%s=%lu", stat->name,
mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
false));
- for_each_node_state(nid, N_MEMORY)
+ for_each_node_mask(nid, reportable)
seq_printf(m, " N%d=%lu", nid,
mem_cgroup_node_nr_lru_pages(memcg, nid,
stat->lru_mask, false));
@@ -2151,7 +2155,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)
seq_printf(m, "hierarchical_%s=%lu", stat->name,
mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
true));
- for_each_node_state(nid, N_MEMORY)
+ for_each_node_mask(nid, reportable)
seq_printf(m, " N%d=%lu", nid,
mem_cgroup_node_nr_lru_pages(memcg, nid,
stat->lru_mask, true));
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f0dde52dc9e0e..4d79e238bc57d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4496,6 +4496,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
struct mem_cgroup *parent = parent_mem_cgroup(memcg);
struct memcg_vmstats_percpu *statc;
struct aggregate_control ac;
+ nodemask_t reportable;
int nid;
flush_nmi_stats(memcg, parent);
@@ -4524,7 +4525,9 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
};
mem_cgroup_stat_aggregate(&ac);
- for_each_node_state(nid, N_MEMORY) {
+ /* Private nodes must also be accounted or numa_stat is misleading */
+ nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ for_each_node_mask(nid, reportable) {
struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid];
struct lruvec_stats *lstats = pn->lruvec_stats;
struct lruvec_stats *plstats = NULL;
@@ -4947,9 +4950,13 @@ static int memory_numa_stat_show(struct seq_file *m, void *v)
{
int i;
struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+ nodemask_t reportable;
mem_cgroup_flush_stats(memcg);
+ /* Private nodes hold cgroup memory too, report them. */
+ nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+
for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
int nid;
@@ -4957,7 +4964,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v)
continue;
seq_printf(m, "%s", memory_stats[i].name);
- for_each_node_state(nid, N_MEMORY) {
+ for_each_node_mask(nid, reportable) {
u64 size;
struct lruvec *lruvec;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats
2026-07-20 19:34 ` [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats Gregory Price
@ 2026-07-21 18:33 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:33 UTC (permalink / raw)
To: Gregory Price; +Cc: tj, linux-cxl, damon, hannes, mkoutny, nvdimm, kvm
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The patch updates `mem_cgroup_css_rstat_flush` to include `N_MEMORY_PRIVATE` nodes, but misses `flush_nmi_stats()` which still strictly iterates over `N_MEMORY`, leaking NMI slab stats on private nodes.
- [Medium] `hugetlb_cgroup_read_numa_stat()` in the hugetlb cgroup controller skips `N_MEMORY_PRIVATE` nodes, leading to hidden node memory and incorrect hierarchical totals.
--
--- Patch [20]: [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats ---
commit 923e253e1d279f965b728ad3baafd46c0f7044eb
Author: Gregory Price <gourry@gourry.net>
mm/memcontrol: account private-node memory in per-node stats
Private nodes folios are charged like any other - the node's per-cpu
lruvec counters are increment even for N_MEMORY_PRIVATE.
[ ... ]
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index f0dde52dc9e0e..4d79e238bc57d 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -4496,6 +4496,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
> struct mem_cgroup *parent = parent_mem_cgroup(memcg);
> struct memcg_vmstats_percpu *statc;
> struct aggregate_control ac;
> + nodemask_t reportable;
> int nid;
>
> flush_nmi_stats(memcg, parent);
[Severity: Medium]
Will this still leak NMI slab stats on private nodes?
flush_nmi_stats() still strictly iterates over N_MEMORY instead of including
N_MEMORY_PRIVATE.
If a slab allocation occurs in NMI context on an N_MEMORY_PRIVATE node, the
node-local atomic counter is incremented, but it appears flush_nmi_stats()
will fail to drain it because it only iterates over N_MEMORY.
[ ... ]
[Severity: Medium]
Does mm/hugetlb_cgroup.c:hugetlb_cgroup_read_numa_stat() also need to be
updated to include N_MEMORY_PRIVATE?
It currently only covers N_MEMORY when calculating and printing per-node
hierarchical usage:
for_each_node_state(nid, N_MEMORY) {
usage = 0;
...
If hugetlb memory is allocated on an N_MEMORY_PRIVATE node and charged to a
cgroup, reading the cgroup's NUMA stat file would skip accumulating or
displaying this node's usage.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=20
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (19 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 20:05 ` Omar Sandoval
2026-07-20 19:34 ` [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection Gregory Price
` (17 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
The kcore_ram_list bounds its scan at the highest pfn in N_MEMORY.
As a result, private nodes placed above the last N_MEMORY node, which is
the common case for device/CXL memory - is truncated out of /proc/kcore.
Include N_MEMORY_PRIVATE nodes in the scan boundary. /proc/kcore is
root-only (CAP_SYS_RAWIO), so this exposes nothing new - it just lets
a debugger read private-node memory like any other RAM.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
fs/proc/kcore.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/fs/proc/kcore.c b/fs/proc/kcore.c
index 390547b992acd..f67194dc2a9b8 100644
--- a/fs/proc/kcore.c
+++ b/fs/proc/kcore.c
@@ -251,11 +251,14 @@ static int kcore_ram_list(struct list_head *list)
{
int nid, ret;
unsigned long end_pfn;
+ nodemask_t mem;
/* Not initialized....update now */
/* find out "max pfn" */
end_pfn = 0;
- for_each_node_state(nid, N_MEMORY) {
+ /* Include private node memory in the scan */
+ nodes_or(mem, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ for_each_node_mask(nid, mem) {
unsigned long node_end;
node_end = node_end_pfn(nid);
if (end_pfn < node_end)
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map
2026-07-20 19:34 ` [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map Gregory Price
@ 2026-07-20 20:05 ` Omar Sandoval
0 siblings, 0 replies; 82+ messages in thread
From: Omar Sandoval @ 2026-07-20 20:05 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, Jul 20, 2026 at 03:34:15PM -0400, Gregory Price wrote:
> The kcore_ram_list bounds its scan at the highest pfn in N_MEMORY.
>
> As a result, private nodes placed above the last N_MEMORY node, which is
> the common case for device/CXL memory - is truncated out of /proc/kcore.
>
> Include N_MEMORY_PRIVATE nodes in the scan boundary. /proc/kcore is
> root-only (CAP_SYS_RAWIO), so this exposes nothing new - it just lets
> a debugger read private-node memory like any other RAM.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> fs/proc/kcore.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/fs/proc/kcore.c b/fs/proc/kcore.c
> index 390547b992acd..f67194dc2a9b8 100644
> --- a/fs/proc/kcore.c
> +++ b/fs/proc/kcore.c
> @@ -251,11 +251,14 @@ static int kcore_ram_list(struct list_head *list)
> {
> int nid, ret;
> unsigned long end_pfn;
> + nodemask_t mem;
>
> /* Not initialized....update now */
> /* find out "max pfn" */
> end_pfn = 0;
> - for_each_node_state(nid, N_MEMORY) {
> + /* Include private node memory in the scan */
> + nodes_or(mem, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
> + for_each_node_mask(nid, mem) {
> unsigned long node_end;
> node_end = node_end_pfn(nid);
> if (end_pfn < node_end)
> --
> 2.53.0-Meta
>
I'm sure this is the least of your concern with this series, but FWIW:
Acked-by: Omar Sandoval <osandov@osandov.com>
If everything goes well, kcore.c is quiet enough that this can probably
just be merged through whatever tree takes the rest of this work.
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (20 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 19:28 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds Gregory Price
` (16 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Add MPOL_F_PRIVATE, the internal flag noting that a policy contains
a nodemask with a private node, and making private node memory
reachable via standard mempolicies.
Plumb zonelist selection into the mpol allocator interfaces via the
allocator's alloc_flags, using the alloc_flags-carrying page_alloc
interfaces.
mpol_alloc_flags() maps MPOL_F_PRIVATE to the alloc_flags an
allocation uses:
ALLOC_DEFAULT - normal allocation
ALLOC_ZONELIST_PRIVATE - private node allocation
With this, a VMA with (mpol->flags & MPOL_F_PRIVATE) can successfully
services faults like any other mbind() from the private node:
buf = mmap(..., MAP_ANON);
mbind(buf, ..., {private_node});
buf[0] = 0xdeadbeef; // Page faulted from private node memory
Like any other bind, the nodemask relaxes if it is unsatisfiable.
For example: apply_policy_zone() allows an unmovable allocation
targeting the VMA to fallback to a viable node instead of failing.
cpuset rebinding does not affect private nodes - the nodes in the
original mask are preserved, and we enforce two remap rules:
1) never allow N_MEMORY nodes to remap to N_MEMORY_PRIVATE
2) never remap N_MEMORY_PRIVATE nodes at all (they stay in place)
In the case of an empty nodemask as a result of rebind, revert to
cpuset - which never includes N_MEMORY_PRIVATE and always guarantees
at least one N_MEMORY node.
As of this patch, nothing can actually sets MPOL_F_PRIVATE, so this
is a no-op that simply adds the plumbing throughout mempolicy.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/uapi/linux/mempolicy.h | 1 +
mm/mempolicy.c | 79 +++++++++++++++++++++++-----------
2 files changed, 55 insertions(+), 25 deletions(-)
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index 7f6fc9599693b..87af18c84b947 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -67,6 +67,7 @@ enum mempolicy_mode {
#define MPOL_F_SHARED (1 << 0) /* identify shared policies */
#define MPOL_F_MOF (1 << 3) /* this policy wants migrate on fault */
#define MPOL_F_MORON (1 << 4) /* Migrate On protnone Reference On Node */
+#define MPOL_F_PRIVATE (1 << 5) /* nodemask contains private nodes */
/*
* Enabling zone reclaim means the page allocator will attempt to fulfill
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 2b76c57a460c9..90110e9761122 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -432,6 +432,10 @@ static int mpol_set_nodemask(struct mempolicy *pol,
else
pol->w.cpuset_mems_allowed = cpuset_current_mems_allowed;
+ /* If any private nodes left in the nodemask - add the private flag */
+ if (nodes_intersects(nsc->mask2, node_states[N_MEMORY_PRIVATE]))
+ pol->flags |= MPOL_F_PRIVATE;
+
ret = mpol_ops[pol->mode].create(pol, &nsc->mask2);
return ret;
}
@@ -505,22 +509,33 @@ static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
static void mpol_rebind_nodemask(struct mempolicy *pol, const nodemask_t *nodes)
{
- nodemask_t tmp;
+ nodemask_t tmp, priv;
+
+ /* preserve online private nodes to re-add later */
+ nodes_and(priv, pol->nodes, node_states[N_MEMORY_PRIVATE]);
if (pol->flags & MPOL_F_STATIC_NODES)
nodes_and(tmp, pol->w.user_nodemask, *nodes);
- else if (pol->flags & MPOL_F_RELATIVE_NODES)
- mpol_relative_nodemask(&tmp, &pol->w.user_nodemask, nodes);
- else {
+ else if (pol->flags & MPOL_F_RELATIVE_NODES) {
+ /* fold only the public part: a private node must not take a slot */
+ nodes_and(tmp, pol->w.user_nodemask, node_states[N_MEMORY]);
+ mpol_relative_nodemask(&tmp, &tmp, nodes);
+ } else {
nodes_remap(tmp, pol->nodes, pol->w.cpuset_mems_allowed,
*nodes);
pol->w.cpuset_mems_allowed = *nodes;
}
- if (nodes_empty(tmp))
+ /* private nodes are identity-mapped during remap, drop them here */
+ nodes_and(tmp, tmp, node_states[N_MEMORY]);
+ if (nodes_empty(tmp) && nodes_empty(priv))
tmp = *nodes;
- pol->nodes = tmp;
+ /* If any online private nodes remain, add them back */
+ nodes_or(pol->nodes, tmp, priv);
+ /* If no online private nodes remain, strip the private flag */
+ if (nodes_empty(priv))
+ pol->flags &= ~MPOL_F_PRIVATE;
}
static void mpol_rebind_preferred(struct mempolicy *pol,
@@ -2411,7 +2426,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk,
}
static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
- int nid, nodemask_t *nodemask)
+ int nid, nodemask_t *nodemask,
+ unsigned int aflags)
{
struct page *page;
gfp_t preferred_gfp;
@@ -2425,14 +2441,21 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
preferred_gfp = gfp | __GFP_NOWARN;
preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
page = __alloc_frozen_pages_noprof(preferred_gfp, order, nid, nodemask,
- ALLOC_DEFAULT);
+ aflags);
if (!page)
page = __alloc_frozen_pages_noprof(gfp, order, nid, NULL,
- ALLOC_DEFAULT);
+ aflags);
return page;
}
+/* A private policy allocates from the private zonelist. */
+static inline unsigned int mpol_alloc_flags(struct mempolicy *pol)
+{
+ return (pol->flags & MPOL_F_PRIVATE) ? ALLOC_ZONELIST_PRIVATE :
+ ALLOC_DEFAULT;
+}
+
/**
* alloc_pages_mpol - Allocate pages according to NUMA mempolicy.
* @gfp: GFP flags.
@@ -2448,11 +2471,13 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
{
nodemask_t *nodemask;
struct page *page;
+ unsigned int aflags = mpol_alloc_flags(pol);
nodemask = policy_nodemask(gfp, pol, ilx, &nid);
if (pol->mode == MPOL_PREFERRED_MANY)
- return alloc_pages_preferred_many(gfp, order, nid, nodemask);
+ return alloc_pages_preferred_many(gfp, order, nid, nodemask,
+ aflags);
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
/* filter "hugepage" allocation, unless from alloc_pages() */
@@ -2476,7 +2501,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
*/
page = __alloc_frozen_pages_noprof(
gfp | __GFP_THISNODE | __GFP_NORETRY, order,
- nid, NULL, ALLOC_DEFAULT);
+ nid, NULL, aflags);
if (page || !(gfp & __GFP_DIRECT_RECLAIM))
return page;
/*
@@ -2488,7 +2513,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
}
}
- page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, ALLOC_DEFAULT);
+ page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, aflags);
if (unlikely(pol->mode == MPOL_INTERLEAVE ||
pol->mode == MPOL_WEIGHTED_INTERLEAVE) && page) {
@@ -2597,6 +2622,7 @@ static unsigned long alloc_pages_bulk_interleave(gfp_t gfp,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
{
+ unsigned int aflags = mpol_alloc_flags(pol);
int nodes;
unsigned long nr_pages_per_node;
int delta;
@@ -2610,14 +2636,14 @@ static unsigned long alloc_pages_bulk_interleave(gfp_t gfp,
for (i = 0; i < nodes; i++) {
if (delta) {
- nr_allocated = alloc_pages_bulk_noprof(gfp,
- interleave_nodes(pol), NULL,
+ nr_allocated = __alloc_pages_bulk_noprof(gfp,
+ aflags, interleave_nodes(pol), NULL,
nr_pages_per_node + 1,
page_array);
delta--;
} else {
- nr_allocated = alloc_pages_bulk_noprof(gfp,
- interleave_nodes(pol), NULL,
+ nr_allocated = __alloc_pages_bulk_noprof(gfp,
+ aflags, interleave_nodes(pol), NULL,
nr_pages_per_node, page_array);
}
@@ -2632,6 +2658,7 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
{
+ unsigned int aflags = mpol_alloc_flags(pol);
struct weighted_interleave_state *state;
struct task_struct *me = current;
unsigned int cpuset_mems_cookie;
@@ -2667,8 +2694,8 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
weight = me->il_weight;
if (weight && node_isset(node, nodes)) {
node_pages = min(rem_pages, weight);
- nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
- page_array);
+ nr_allocated = __alloc_pages_bulk_noprof(gfp, aflags, node,
+ NULL, node_pages, page_array);
page_array += nr_allocated;
total_allocated += nr_allocated;
/* if that's all the pages, no need to interleave */
@@ -2732,8 +2759,8 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
/* node_pages can be 0 if an allocation fails and rounds == 0 */
if (!node_pages)
break;
- nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
- page_array);
+ nr_allocated = __alloc_pages_bulk_noprof(gfp, aflags, node,
+ NULL, node_pages, page_array);
page_array += nr_allocated;
total_allocated += nr_allocated;
if (total_allocated == nr_pages)
@@ -2750,17 +2777,19 @@ static unsigned long alloc_pages_bulk_preferred_many(gfp_t gfp, int nid,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
{
+ unsigned int aflags = mpol_alloc_flags(pol);
gfp_t preferred_gfp;
unsigned long nr_allocated = 0;
preferred_gfp = gfp | __GFP_NOWARN;
preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
- nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes,
- nr_pages, page_array);
+ nr_allocated = __alloc_pages_bulk_noprof(preferred_gfp, aflags,
+ nid, &pol->nodes, nr_pages, page_array);
if (nr_allocated < nr_pages)
- nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL,
+ nr_allocated += __alloc_pages_bulk_noprof(gfp, aflags,
+ numa_node_id(), NULL,
nr_pages - nr_allocated,
page_array + nr_allocated);
return nr_allocated;
@@ -2796,8 +2825,8 @@ unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
nid = numa_node_id();
nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid);
- return alloc_pages_bulk_noprof(gfp, nid, nodemask,
- nr_pages, page_array);
+ return __alloc_pages_bulk_noprof(gfp, mpol_alloc_flags(pol), nid,
+ nodemask, nr_pages, page_array);
}
int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
2026-07-20 19:34 ` [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection Gregory Price
@ 2026-07-21 19:28 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 19:28 UTC (permalink / raw)
To: Gregory Price; +Cc: damon, hannes, kvm, tj, mkoutny, linux-cxl, nvdimm
Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:
New issues:
- [High] Private nodes are permanently lost from MPOL_F_STATIC_NODES policies during memory offlining/rebinding.
- [High] Mempolicies targeting private nodes will silently fail to allocate from them due to cpuset restrictions.
- [Medium] Non-atomic clearing of MPOL_F_PRIVATE in pol->flags causes a data race with lockless page allocators.
Pre-existing issues:
- [High] Lockless and unprotected access to pol->nodes in alloc_pages_bulk_interleave can lead to a divide-by-zero crash.
- [High] Unconditional GFP_KERNEL allocation in alloc_pages_bulk_weighted_interleave violates caller's GFP constraints.
--
--- Patch [22]: [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection ---
commit 11224b898bbcf008cfe6641a6cd14bd637edb1d1
Author: Gregory Price <gourry@gourry.net>
mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
This patch adds MPOL_F_PRIVATE to mempolicy flags to identify policies
containing private nodes, allowing private node memory to be reachable.
It also plumbs zonelist selection into the mempolicy allocator interfaces
using the alloc_flags mechanism.
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 2b76c57a460c9..90110e9761122 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
[ ... ]
> @@ -505,22 +509,33 @@ static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
>
> static void mpol_rebind_nodemask(struct mempolicy *pol, const nodemask_t *nodes)
> {
> - nodemask_t tmp;
> + nodemask_t tmp, priv;
> +
> + /* preserve online private nodes to re-add later */
> + nodes_and(priv, pol->nodes, node_states[N_MEMORY_PRIVATE]);
[Severity: High]
Does this evaluate private nodes against pol->nodes instead of
pol->w.user_nodemask?
If a private node targeted by an MPOL_F_STATIC_NODES policy goes offline,
it is removed from pol->nodes. When the node comes back online and
mpol_rebind_nodemask() runs, attempting to restore private nodes from
pol->nodes will find it empty.
Could this result in permanent loss of private nodes from static policies?
> if (pol->flags & MPOL_F_STATIC_NODES)
> nodes_and(tmp, pol->w.user_nodemask, *nodes);
[ ... ]
> - if (nodes_empty(tmp))
> + /* private nodes are identity-mapped during remap, drop them here */
> + nodes_and(tmp, tmp, node_states[N_MEMORY]);
> + if (nodes_empty(tmp) && nodes_empty(priv))
> tmp = *nodes;
>
> - pol->nodes = tmp;
> + /* If any online private nodes remain, add them back */
> + nodes_or(pol->nodes, tmp, priv);
> + /* If no online private nodes remain, strip the private flag */
> + if (nodes_empty(priv))
> + pol->flags &= ~MPOL_F_PRIVATE;
[Severity: Medium]
Does clearing MPOL_F_PRIVATE non-atomically create a data race with lockless
page allocators?
When mpol_alloc_flags() concurrently reads (pol->flags & MPOL_F_PRIVATE)
locklessly in the page allocator fastpath, could this result in torn reads
and trigger KCSAN warnings?
> }
[ ... ]
> @@ -2488,7 +2513,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
> }
> }
>
> - page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, ALLOC_DEFAULT);
> + page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, aflags);
[Severity: High]
For MPOL_BIND policies, will cpuset_nodemask_valid_mems_allowed() return
false because private nodes are excluded from cpuset.mems, setting nodemask
to NULL?
When a NULL nodemask is passed to the allocator, the fastpath defaults to
cpuset_current_mems_allowed with ALLOC_CPUSET. Since private nodes are
absent from the cpuset, won't __cpuset_zone_allowed() skip the private
nodes regardless of ALLOC_ZONELIST_PRIVATE?
Will mempolicies targeting private nodes silently fail to allocate from them
and fall back to standard nodes?
> if (unlikely(pol->mode == MPOL_INTERLEAVE ||
[ ... ]
> @@ -2597,6 +2622,7 @@ static unsigned long alloc_pages_bulk_interleave(gfp_t gfp,
> struct mempolicy *pol, unsigned long nr_pages,
> struct page **page_array)
> {
> + unsigned int aflags = mpol_alloc_flags(pol);
> int nodes;
> unsigned long nr_pages_per_node;
[Severity: High]
This isn't a bug introduced by this patch, but is it safe to read pol->nodes
locklessly in alloc_pages_bulk_interleave()?
alloc_pages_bulk_interleave() reads the nodemask word-by-word via
nodes_weight(pol->nodes) further down in this function.
Concurrently, mpol_rebind_nodemask() mutates pol->nodes word-by-word via
nodes_or(pol->nodes, tmp, priv).
On systems with >64 nodes, could a torn read result in a weight of 0,
causing a divide-by-zero panic when calculating
nr_pages_per_node = nr_pages / nodes?
[ ... ]
> @@ -2632,6 +2658,7 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
> struct mempolicy *pol, unsigned long nr_pages,
> struct page **page_array)
> {
> + unsigned int aflags = mpol_alloc_flags(pol);
> struct weighted_interleave_state *state;
> struct task_struct *me = current;
[Severity: High]
This is a pre-existing issue, but does alloc_pages_bulk_weighted_interleave()
violate the caller's GFP constraints?
This function can be called via alloc_pages_bulk_mempolicy_noprof() from
vmalloc with restrictive flags like GFP_NOFS or GFP_NOIO. However, it
allocates its weights array unconditionally using
kzalloc(nr_node_ids, GFP_KERNEL) later in the function.
Could this strip the __GFP_FS or __GFP_IO protection and allow the
allocation to recurse into the filesystem or block layer, potentially
causing a deadlock?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=22
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (21 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 19:39 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services Gregory Price
` (15 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
apply_policy_zone() raises dynamic_policy_zone to ZONE_MOVABLE when none
of a policy's nodes have non-movable memory, so a bind to movable-only
node does not force kernel-zone allocations onto a zone that cannot
satisfy them - resulting retries forever (livelock).
The current code uses N_HIGH_MEMORY to denote eligibility, but private
nodes are deliberately not in N_HIGH_MEMORY nor N_NORMAL_MEMORY. This
means a private node onlined into ZONE_NORMAL is judged as movable-only.
This means a policy set with set_mempolicy() binding a private node with
ZONE_NORMAL memory will shunt otherwise eligible allocations (like the
task's page tables) back to N_MEMORY nodes.
This also becomes relevant for in-kernel users that want non-movable
memory (such as KVM placing pinned VM memory solely on a private node).
Add policy_private_has_kernel_zone() to consult the actual zones of
the policy's private nodes and keep the policy applied at lower zones
when a kernel-zoned private node can satisfy the allocation.
Only a mempolicy binding a private node pays the zone scan, all other
mempolicies preserve fast-path performance.
Additionally: update the comments in apply_policy_zone for clarity,
because this particular policy mechanism is very confusing.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 48 ++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 42 insertions(+), 6 deletions(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 90110e9761122..e83c2c7a94c1d 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2091,6 +2091,34 @@ bool vma_policy_mof(struct vm_area_struct *vma)
return mof;
}
+/*
+ * true if any policy *private* node can satisfy a !__GFP_MOVABLE allocation.
+ * Policies without private nodes (~MPOL_F_PRIVATE) always return false.
+ *
+ * This provides a way for otherwise-bound kernel allocations to make their
+ * way onto private nodes with ZONE_NORMAL memory without having to audit
+ * every caller. The cost is a zone scan for private nodes in the mask.
+ */
+static bool policy_private_has_kernel_zone(const struct mempolicy *pol)
+{
+ int nid;
+
+ if (!(pol->flags & MPOL_F_PRIVATE))
+ return false;
+
+ for_each_node_mask(nid, pol->nodes) {
+ pg_data_t *pgdat = NODE_DATA(nid);
+ enum zone_type zt;
+
+ if (!node_is_private(nid))
+ continue;
+ for (zt = ZONE_NORMAL; zt < ZONE_MOVABLE; zt++)
+ if (managed_zone(&pgdat->node_zones[zt]))
+ return true;
+ }
+ return false;
+}
+
bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
{
enum zone_type dynamic_policy_zone = policy_zone;
@@ -2098,14 +2126,22 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
BUG_ON(dynamic_policy_zone == ZONE_MOVABLE);
/*
- * if policy->nodes has movable memory only,
- * we apply policy when gfp_zone(gfp) = ZONE_MOVABLE only.
+ * dynamic_policy_zone is the lowest zone this policy is enforced for,
+ * allocations below it ignore the nodemask and fall back freely.
+ *
+ * If all policy-nodes are movable-only (all ZONE_MOVABLE), raise the
+ * dynamic_policy_zone to ZONE_MOVABLE so that only movable allocations
+ * stay bound - otherwise this would force kernel-zone allocations
+ * (e.g. page tables) onto a zone that cannot satisfy them. This
+ * results in allocation failure and retry loop (livelock).
*
- * policy->nodes is intersect with node_states[N_MEMORY].
- * so if the following test fails, it implies
- * policy->nodes has movable memory only.
+ * N_HIGH_MEMORY tells us "has a non-movable zone" for ordinary nodes.
+ * Private nodes are deliberately excluded from N_HIGH_MEMORY, so we
+ * need to check their actual zones - a kernel-zoned private node can
+ * hold the allocation and must keep the policy applied.
*/
- if (!nodes_intersects(policy->nodes, node_states[N_HIGH_MEMORY]))
+ if (!nodes_intersects(policy->nodes, node_states[N_HIGH_MEMORY]) &&
+ !policy_private_has_kernel_zone(policy))
dynamic_policy_zone = ZONE_MOVABLE;
return zone >= dynamic_policy_zone;
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds
2026-07-20 19:34 ` [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds Gregory Price
@ 2026-07-21 19:39 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 19:39 UTC (permalink / raw)
To: Gregory Price; +Cc: nvdimm, kvm, tj, linux-cxl, hannes, mkoutny, damon
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The `policy_private_has_kernel_zone()` function incorrectly skips checking `ZONE_DMA` and `ZONE_DMA32`, which causes mempolicy bypass for private nodes possessing only lower-zone memory.
--
--- Patch [23]: [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds ---
commit 04782bb72086af5fe29a7dbf4184affd04c79df6
Author: Gregory Price <gourry@gourry.net>
mm/mempolicy: apply policy at the kernel zone for private-node binds
This commit introduces policy_private_has_kernel_zone() to prevent mempolicy
bypass when private nodes have non-movable memory.
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 90110e9761122..e83c2c7a94c1d 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
[ ... ]
> @@ -2091,6 +2091,34 @@ bool vma_policy_mof(struct vm_area_struct *vma)
[ ... ]
> +static bool policy_private_has_kernel_zone(const struct mempolicy *pol)
> +{
> + int nid;
> +
> + if (!(pol->flags & MPOL_F_PRIVATE))
> + return false;
> +
> + for_each_node_mask(nid, pol->nodes) {
> + pg_data_t *pgdat = NODE_DATA(nid);
> + enum zone_type zt;
> +
> + if (!node_is_private(nid))
> + continue;
> + for (zt = ZONE_NORMAL; zt < ZONE_MOVABLE; zt++)
[Severity: Medium]
Does this loop miss checking ZONE_DMA and ZONE_DMA32 by starting at
ZONE_NORMAL?
If a user process sets a mempolicy binding to a private NUMA node that only
contains memory in ZONE_DMA or ZONE_DMA32, this loop will return false
because it skips those populated lower zones.
This would cause apply_policy_zone() to incorrectly raise
dynamic_policy_zone to ZONE_MOVABLE, resulting in kernel allocations
falling back to other system nodes and bypassing the policy intent.
> + if (managed_zone(&pgdat->node_zones[zt]))
> + return true;
> + }
> + return false;
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=23
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (22 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:36 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug Gregory Price
` (14 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Add and export interfaces to enable modules to build a mempolicy with
(MPOL_BIND | MPOL_F_PRIVATE) pinned to a private NUMA node, so the
drivers can stamp it onto VMAs they control (via vma->vm_policy).
mpol_bind_node() - acts the same as a userland mbind()
mpol_private_bind() - bypasses the CAP_USER_NUMA check.
Export it to modules (private:kmem, bind:kvm).
Widen __mpol_put()'s module export to kmem so the driver can
release the policy.
Adjust mpol_set_nodemask to check for a pre-set MPOL_F_PRIVATE flag
to allow the mpol_private_bind() check to bypass the N_MEMORY filter.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/mempolicy.h | 14 ++++++
mm/mempolicy.c | 102 +++++++++++++++++++++++++++++++++++++-
2 files changed, 114 insertions(+), 2 deletions(-)
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 65c732d440d2f..715951a5b03c1 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -7,6 +7,7 @@
#define _LINUX_MEMPOLICY_H 1
#include <linux/sched.h>
+#include <linux/err.h>
#include <linux/mmzone.h>
#include <linux/slab.h>
#include <linux/rbtree.h>
@@ -128,6 +129,9 @@ void mpol_free_shared_policy(struct shared_policy *sp);
struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
pgoff_t idx);
+struct mempolicy *mpol_private_bind(int nid);
+struct mempolicy *mpol_bind_node(int nid);
+
struct mempolicy *get_task_policy(struct task_struct *p);
struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
unsigned long addr, pgoff_t *ilx);
@@ -226,6 +230,16 @@ mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx)
return NULL;
}
+static inline struct mempolicy *mpol_private_bind(int nid)
+{
+ return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline struct mempolicy *mpol_bind_node(int nid)
+{
+ return ERR_PTR(-EOPNOTSUPP);
+}
+
static inline struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
unsigned long addr, int order, pgoff_t *ilx)
{
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index e83c2c7a94c1d..a3ffb09897489 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -406,7 +406,7 @@ static int mpol_new_preferred(struct mempolicy *pol, const nodemask_t *nodes)
static int mpol_set_nodemask(struct mempolicy *pol,
const nodemask_t *nodes, struct nodemask_scratch *nsc)
{
- int ret;
+ int ret, nid;
/*
* Default (pol==NULL) resp. local memory policies are not a
@@ -432,6 +432,18 @@ static int mpol_set_nodemask(struct mempolicy *pol,
else
pol->w.cpuset_mems_allowed = cpuset_current_mems_allowed;
+ /*
+ * Private nodes are not in cpuset.mems, so they're always stripped.
+ * Driver-allocated policies will already have MPOL_F_PRIVATE set,
+ * if that's the case, add back in the requested set of private nodes.
+ */
+ for_each_node_mask(nid, *nodes) {
+ if (!node_is_private(nid))
+ continue;
+ if (pol->flags & MPOL_F_PRIVATE)
+ node_set(nid, nsc->mask2);
+ }
+
/* If any private nodes left in the nodemask - add the private flag */
if (nodes_intersects(nsc->mask2, node_states[N_MEMORY_PRIVATE]))
pol->flags |= MPOL_F_PRIVATE;
@@ -501,7 +513,7 @@ void __mpol_put(struct mempolicy *pol)
*/
kfree_rcu(pol, rcu);
}
-EXPORT_SYMBOL_FOR_MODULES(__mpol_put, "kvm");
+EXPORT_SYMBOL_FOR_MODULES(__mpol_put, "kvm,kmem");
static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
{
@@ -1126,6 +1138,92 @@ static long do_set_mempolicy(unsigned short mode, unsigned short flags,
return ret;
}
+/*
+ * Build a refcounted MPOL_BIND policy targeting the single node @nid,
+ * contextualised to the caller's cpuset like a userspace mbind().
+ *
+ * @flags is MPOL_F_PRIVATE for the an explicit in-kernel user, letting
+ * a private node be bound without checking CAP_USER_NUMA. Otherwise,
+ * CAP_USER_NUMA is enforced.
+ *
+ * The caller owns the reference and frees it with mpol_put().
+ */
+static struct mempolicy *__mpol_bind_node(int nid, unsigned short flags)
+{
+ struct mempolicy *pol;
+ nodemask_t nodes;
+ int err;
+
+ NODEMASK_SCRATCH(scratch);
+
+ if (!scratch)
+ return ERR_PTR(-ENOMEM);
+
+ nodes_clear(nodes);
+ node_set(nid, nodes);
+
+ pol = mpol_new(MPOL_BIND, flags, &nodes);
+ if (IS_ERR(pol)) {
+ NODEMASK_SCRATCH_FREE(scratch);
+ return pol;
+ }
+
+ err = mpol_set_nodemask(pol, &nodes, scratch);
+ NODEMASK_SCRATCH_FREE(scratch);
+ if (err) {
+ mpol_put(pol);
+ return ERR_PTR(err);
+ }
+ return pol;
+}
+
+/**
+ * mpol_private_bind - build an MPOL_BIND policy pinned to a private node
+ * @nid: an N_MEMORY_PRIVATE node
+ *
+ * Returns a refcounted mempolicy that binds allocations to @nid with the
+ * private-placement intent (MPOL_F_PRIVATE). This binds to @nid regardless
+ * of the node's CAP_USER_NUMA, providing a privileged way for node-owners
+ * to bind driver/service owned VMAs to the node.
+ *
+ * Like any MPOL_BIND it is relaxable: an unsatisfiable request falls back
+ * rather than failing.
+ *
+ * Must be called while @nid is N_MEMORY_PRIVATE.
+ *
+ * The caller owns the reference and frees it with mpol_put().
+ *
+ * Return: the policy, or an ERR_PTR on failure.
+ */
+struct mempolicy *mpol_private_bind(int nid)
+{
+ if (!node_is_private(nid))
+ return ERR_PTR(-EINVAL);
+ return __mpol_bind_node(nid, MPOL_F_PRIVATE);
+}
+EXPORT_SYMBOL_FOR_MODULES(mpol_private_bind, "kmem");
+
+/**
+ * mpol_bind_node - build an MPOL_BIND policy targeting @nid for in-kernel use
+ * @nid: the node to bind to
+ *
+ * Returns a refcounted MPOL_BIND policy that places allocations on @nid,
+ * contextualised to the caller's cpuset exactly like a userspace mbind().
+ *
+ * This interface should be used by services implenting mempolicy support with
+ * user-provided node bindings. N_MEMORY_PRIVATE node bindings are honored if
+ * the node has CAP_USER_NUMA, otherwise return -EINVAL.
+ *
+ * The caller owns the reference and frees it with mpol_put().
+ *
+ * Return: the policy, or an ERR_PTR on failure.
+ */
+struct mempolicy *mpol_bind_node(int nid)
+{
+ return __mpol_bind_node(nid, 0);
+}
+EXPORT_SYMBOL_FOR_MODULES(mpol_bind_node, "kvm");
+
/*
* Return nodemask for policy for get_mempolicy() query
*
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
2026-07-20 19:34 ` [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services Gregory Price
@ 2026-07-21 18:36 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:36 UTC (permalink / raw)
To: Gregory Price; +Cc: nvdimm, kvm, hannes, linux-cxl, tj, damon, mkoutny
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] `mpol_bind_node()` and `mpol_private_bind()` fail to validate `nid` against bounds, leading to stack buffer overflows and out-of-bounds reads.
- [Medium] `__mpol_bind_node()` calls `mpol_set_nodemask()` without holding `task_lock(current)` or `mmap_lock`, causing a data race and potential torn read.
- [Medium] `mpol_bind_node()` unconditionally rejects private node bindings, breaking the documented API contract.
--
--- Patch [24]: [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services ---
commit 056427d8f98ec56375b0a715f69798d4dfde2ec3
Author: Gregory Price <gourry@gourry.net>
mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
This patch adds and exports interfaces to enable modules to build MPOL_BIND
mempolicies pinned to a private NUMA node. It exports mpol_bind_node() and
mpol_private_bind() for module use, widens the __mpol_put() export, and
adjusts mpol_set_nodemask() to handle MPOL_F_PRIVATE flags.
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index e83c2c7a94c1d..a3ffb09897489 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -432,6 +432,18 @@ static int mpol_set_nodemask(struct mempolicy *pol,
> else
> pol->w.cpuset_mems_allowed = cpuset_current_mems_allowed;
>
> + /*
> + * Private nodes are not in cpuset.mems, so they're always stripped.
> + * Driver-allocated policies will already have MPOL_F_PRIVATE set,
> + * if that's the case, add back in the requested set of private nodes.
> + */
> + for_each_node_mask(nid, *nodes) {
> + if (!node_is_private(nid))
> + continue;
> + if (pol->flags & MPOL_F_PRIVATE)
> + node_set(nid, nsc->mask2);
> + }
[Severity: Medium]
The commit message mentions that N_MEMORY_PRIVATE node bindings are honored
if the node has CAP_USER_NUMA. Does this implementation actually perform that
capability check?
It looks like this only checks for MPOL_F_PRIVATE. If mpol_bind_node() calls
__mpol_bind_node() with flags=0, wouldn't the private node be stripped
unconditionally, causing the nodemask to become empty and returning -EINVAL?
[ ... ]
> +static struct mempolicy *__mpol_bind_node(int nid, unsigned short flags)
> +{
> + struct mempolicy *pol;
> + nodemask_t nodes;
> + int err;
> +
> + NODEMASK_SCRATCH(scratch);
> +
> + if (!scratch)
> + return ERR_PTR(-ENOMEM);
> +
> + nodes_clear(nodes);
> + node_set(nid, nodes);
[Severity: High]
Should nid be validated against MAX_NUMNODES before being used?
If an invalid or unsanitized node ID like NUMA_NO_NODE (-1) or a value
greater than MAX_NUMNODES is passed from a module, could node_set()
perform an out-of-bounds write on the stack-allocated nodemask_t?
> +
> + pol = mpol_new(MPOL_BIND, flags, &nodes);
> + if (IS_ERR(pol)) {
> + NODEMASK_SCRATCH_FREE(scratch);
> + return pol;
> + }
> +
> + err = mpol_set_nodemask(pol, &nodes, scratch);
[Severity: Medium]
Does this call to mpol_set_nodemask() need to hold task_lock(current) or
mmap_lock?
mpol_set_nodemask() reads cpuset_current_mems_allowed (current->mems_allowed).
Without locking, couldn't this race with concurrent cpuset updates and result
in a torn read or corrupted mempolicy nodemask?
> + NODEMASK_SCRATCH_FREE(scratch);
> + if (err) {
> + mpol_put(pol);
> + return ERR_PTR(err);
> + }
> + return pol;
> +}
> +
> +/**
> + * mpol_private_bind - build an MPOL_BIND policy pinned to a private node
> + * @nid: an N_MEMORY_PRIVATE node
> + *
> + * Returns a refcounted mempolicy that binds allocations to @nid with the
> + * private-placement intent (MPOL_F_PRIVATE). This binds to @nid regardless
> + * of the node's CAP_USER_NUMA, providing a privileged way for node-owners
> + * to bind driver/service owned VMAs to the node.
> + *
> + * Like any MPOL_BIND it is relaxable: an unsatisfiable request falls back
> + * rather than failing.
> + *
> + * Must be called while @nid is N_MEMORY_PRIVATE.
> + *
> + * The caller owns the reference and frees it with mpol_put().
> + *
> + * Return: the policy, or an ERR_PTR on failure.
> + */
> +struct mempolicy *mpol_private_bind(int nid)
> +{
> + if (!node_is_private(nid))
[Severity: High]
Similar to the issue in __mpol_bind_node(), if an invalid nid is passed in,
could node_is_private() read out of bounds?
> + return ERR_PTR(-EINVAL);
> + return __mpol_bind_node(nid, MPOL_F_PRIVATE);
> +}
> +EXPORT_SYMBOL_FOR_MODULES(mpol_private_bind, "kmem");
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=24
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (23 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 18:33 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim Gregory Price
` (13 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Add add_private_memory_driver_managed() to let modules hotplug
memory onto an N_MEMORY_PRIVATE node they control.
Ownership is denoted via `struct node_private` and a private node
can only have a single owner.
The common __add_memory_driver_managed function is only exported to
in-tree modules that allow hotplug of N_MEMORY nodes. The new private
wrapper does not allow N_MEMORY hotplug, only N_MEMORY_PRIVATE.
When node_private is NULL hotplug behaviour is unchanged.
When node_private is non-NULL the node owner is registered before any
memory is added. Later during hotplug, when memory is actually onlined,
hotplug marks the node N_MEMORY_PRIVATE instead of N_MEMORY.
If the add fails the registration is undone.
offline_pages() clears N_MEMORY_PRIVATE the same as N_MEMORY, and
offline_and_remove_memory_ranges() drops private node registration
once the last node's memory is removed.
Private nodes are opted out of starting kswapd/kcompactd by default.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
drivers/dax/kmem.c | 2 +-
include/linux/memory_hotplug.h | 5 +-
mm/memory_hotplug.c | 99 ++++++++++++++++++++++++++++++----
3 files changed, 94 insertions(+), 12 deletions(-)
diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
index a48d699cf344c..c3faf18e7f85d 100644
--- a/drivers/dax/kmem.c
+++ b/drivers/dax/kmem.c
@@ -126,7 +126,7 @@ static int dax_kmem_do_hotplug(struct dev_dax *dev_dax,
*/
rc = __add_memory_driver_managed(data->mgid, range.start,
range_len(&range), kmem_name, mhp_flags,
- online_type);
+ online_type, NULL);
if (rc) {
dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n",
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index b39605d308963..fcceb0a423579 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -305,7 +305,10 @@ extern int add_memory_resource(int nid, struct resource *resource,
mhp_t mhp_flags);
int __add_memory_driver_managed(int nid, u64 start, u64 size,
const char *resource_name, mhp_t mhp_flags,
- enum mmop online_type);
+ enum mmop online_type, struct node_private *np);
+int add_private_memory_driver_managed(int nid, u64 start, u64 size,
+ const char *resource_name, mhp_t mhp_flags,
+ enum mmop online_type, struct node_private *np);
extern int add_memory_driver_managed(int nid, u64 start, u64 size,
const char *resource_name,
mhp_t mhp_flags);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 2b9c0830821e6..be230ac9efe5a 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1169,7 +1169,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
move_pfn_range_to_zone(zone, pfn, nr_pages, NULL, MIGRATE_MOVABLE,
true);
- if (!node_state(nid, N_MEMORY)) {
+ if (!node_state(nid, N_MEMORY) && !node_is_private(nid)) {
/* Adding memory to the node for the first time */
node_arg.nid = nid;
ret = node_notify(NODE_ADDING_FIRST_MEMORY, &node_arg);
@@ -1204,13 +1204,20 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
online_pages_range(pfn, nr_pages);
adjust_present_page_count(pfn_to_page(pfn), group, nr_pages);
+ /*
+ * N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive, determine
+ * which is correct based on whether the pgdat->private is set.
+ */
if (node_arg.nid >= 0)
- node_set_state(nid, N_MEMORY);
+ node_set_state(nid, pgdat_is_private(NODE_DATA(nid)) ?
+ N_MEMORY_PRIVATE : N_MEMORY);
/*
* Check whether we are adding normal memory to the node for the first
* time.
*/
- if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+ if (!node_is_private(nid) &&
+ !node_state(nid, N_NORMAL_MEMORY) &&
+ zone_idx(zone) <= ZONE_NORMAL)
node_set_state(nid, N_NORMAL_MEMORY);
if (need_zonelists_rebuild)
@@ -1230,8 +1237,11 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
/* reinitialise watermarks and update pcp limits */
init_per_zone_wmark_min();
- kswapd_run(nid);
- kcompactd_run(nid);
+ /* Private nodes opt-out of reclaim/compaction by default */
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }
if (node_arg.nid >= 0)
/* First memory added successfully. Notify consumers. */
@@ -1649,6 +1659,8 @@ EXPORT_SYMBOL_GPL(add_memory);
* @resource_name: Resource name in format "System RAM ($DRIVER)"
* @mhp_flags: Memory hotplug flags
* @online_type: Auto-Online behavior (offline, online, kernel, movable)
+ * @np: driver-owned node_private to online the node as N_MEMORY_PRIVATE, or
+ * NULL for ordinary system RAM
*
* Add special, driver-managed memory to the system as system RAM. Such
* memory is not exposed via the raw firmware-provided memmap as system
@@ -1675,9 +1687,11 @@ EXPORT_SYMBOL_GPL(add_memory);
*/
int __add_memory_driver_managed(int nid, u64 start, u64 size,
const char *resource_name, mhp_t mhp_flags,
- enum mmop online_type)
+ enum mmop online_type, struct node_private *np)
{
+ int real_nid = nid;
struct resource *res;
+ struct memory_group *group;
int rc;
if (!resource_name ||
@@ -1688,6 +1702,19 @@ int __add_memory_driver_managed(int nid, u64 start, u64 size,
if (online_type < MMOP_OFFLINE || online_type > MMOP_ONLINE_MOVABLE)
return -EINVAL;
+ /* Register a private-node owner before adding memory. */
+ if (np) {
+ if (mhp_flags & MHP_NID_IS_MGID) {
+ group = memory_group_find_by_id(nid);
+ if (!group)
+ return -EINVAL;
+ real_nid = group->nid;
+ }
+ rc = node_private_register(real_nid, np);
+ if (rc)
+ return rc;
+ }
+
lock_device_hotplug();
res = register_memory_resource(start, size, resource_name);
@@ -1702,10 +1729,41 @@ int __add_memory_driver_managed(int nid, u64 start, u64 size,
out_unlock:
unlock_device_hotplug();
+ if (rc < 0 && np)
+ node_private_unregister(real_nid);
return rc;
}
EXPORT_SYMBOL_FOR_MODULES(__add_memory_driver_managed, "kmem");
+/**
+ * add_private_memory_driver_managed - add private driver-managed memory
+ * @nid: NUMA node ID where the memory will be added
+ * @start: Start physical address of the memory range
+ * @size: Size of the memory range in bytes
+ * @resource_name: Resource name in format "System RAM ($DRIVER)"
+ * @mhp_flags: Memory hotplug flags
+ * @online_type: Auto-Online behavior (offline, online, kernel, movable)
+ * @np: private node context for the owner - mandatory.
+ *
+ * Add private driver-managed memory. Private node context is required to
+ * avoid the memory being added as a normal N_MEMORY node.
+ *
+ * See __add_memory_driver_managed for more details.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int add_private_memory_driver_managed(int nid, u64 start, u64 size,
+ const char *resource_name, mhp_t mhp_flags,
+ enum mmop online_type, struct node_private *np)
+{
+ if (!np)
+ return -EINVAL;
+
+ return __add_memory_driver_managed(nid, start, size, resource_name,
+ mhp_flags, online_type, np);
+}
+EXPORT_SYMBOL_GPL(add_private_memory_driver_managed);
+
/**
* add_memory_driver_managed - add driver-managed memory
* @nid: NUMA node ID where the memory will be added
@@ -1726,7 +1784,7 @@ int add_memory_driver_managed(int nid, u64 start, u64 size,
{
return __add_memory_driver_managed(nid, start, size, resource_name,
mhp_flags,
- mhp_get_default_online_type());
+ mhp_get_default_online_type(), NULL);
}
EXPORT_SYMBOL_GPL(add_memory_driver_managed);
@@ -2041,7 +2099,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
/*
* Check whether the node will have no present pages after we offline
* 'nr_pages' more. If so, we know that the node will become empty, and
- * so we will clear N_MEMORY for it.
+ * so we will clear N_MEMORY(_PRIVATE) for it.
*/
if (nr_pages >= pgdat->node_present_pages) {
node_arg.nid = node;
@@ -2146,8 +2204,10 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
* Make sure to mark the node as memory-less before rebuilding the zone
* list. Otherwise this node would still appear in the fallback lists.
*/
- if (node_arg.nid >= 0)
+ if (node_arg.nid >= 0) {
node_clear_state(node, N_MEMORY);
+ node_clear_state(node, N_MEMORY_PRIVATE);
+ }
if (!populated_zone(zone)) {
zone_pcp_reset(zone);
build_all_zonelists(NULL);
@@ -2211,6 +2271,15 @@ static int count_memory_range_altmaps_cb(struct memory_block *mem, void *arg)
return 0;
}
+static int collect_memblock_nodes_cb(struct memory_block *mem, void *arg)
+{
+ nodemask_t *nodes = arg;
+
+ if (mem->nid != NUMA_NO_NODE)
+ node_set(mem->nid, *nodes);
+ return 0;
+}
+
static int check_cpu_on_node(int nid)
{
int cpu;
@@ -2474,10 +2543,11 @@ EXPORT_SYMBOL_GPL(offline_and_remove_memory);
int offline_and_remove_memory_ranges(const struct range *ranges,
unsigned int nr_ranges)
{
+ nodemask_t nodes = NODE_MASK_NONE;
unsigned long mb_count = 0;
uint8_t *online_types, *tmp;
unsigned int i;
- int rc = 0;
+ int nid, rc = 0;
if (!ranges || !nr_ranges)
return -EINVAL;
@@ -2505,6 +2575,11 @@ int offline_and_remove_memory_ranges(const struct range *ranges,
lock_device_hotplug();
+ /* Record the nodes for the blocks to drop private-node data after */
+ for (i = 0; i < nr_ranges; i++)
+ walk_memory_blocks(ranges[i].start, range_len(&ranges[i]),
+ &nodes, collect_memblock_nodes_cb);
+
/*
* Phase 1: offline every block in every range. An already-offline
* block folds to success, so out-of-band offlining never blocks unplug.
@@ -2535,6 +2610,10 @@ int offline_and_remove_memory_ranges(const struct range *ranges,
out_unlock:
unlock_device_hotplug();
+ /* Drop private node registration if they are now memoryless */
+ for_each_node_mask(nid, nodes)
+ node_private_unregister(nid);
+
kfree(online_types);
return rc;
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
2026-07-20 19:34 ` [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug Gregory Price
@ 2026-07-21 18:33 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 18:33 UTC (permalink / raw)
To: Gregory Price; +Cc: mkoutny, nvdimm, tj, kvm, damon, hannes, linux-cxl
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Private node registration and unregistration incorrectly rely on N_MEMORY and N_MEMORY_PRIVATE states, which only track online memory, causing isolation breakage and hijacking of normal memory nodes.
- [High] A TOCTOU race condition exists between memory unplug and private node registration due to lock ordering.
- [Medium] Direct access of __rcu pointer in pgdat_is_private() without using rcu_access_pointer().
--
--- Patch [25]: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug ---
commit 976d363a06939b47fdc2d106fc8f3ec46359abd3
Author: Gregory Price <gourry@gourry.net>
Subject: mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
This commit introduces add_private_memory_driver_managed() to allow
modules to hotplug memory into an N_MEMORY_PRIVATE node. It introduces
logic to register node ownership and opts these private nodes out of
reclaim/compaction by default.
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 2b9c0830821e6..be230ac9efe5a 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
[ ... ]
> @@ -1204,13 +1204,20 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
> online_pages_range(pfn, nr_pages);
> adjust_present_page_count(pfn_to_page(pfn), group, nr_pages);
>
> + /*
> + * N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive, determine
> + * which is correct based on whether the pgdat->private is set.
> + */
> if (node_arg.nid >= 0)
> - node_set_state(nid, N_MEMORY);
> + node_set_state(nid, pgdat_is_private(NODE_DATA(nid)) ?
> + N_MEMORY_PRIVATE : N_MEMORY);
[Severity: Medium]
Does pgdat_is_private() directly evaluate the __rcu annotated
pgdat->node_private pointer without using rcu_access_pointer()? A direct
boolean evaluation of an __rcu pointer can result in compiler load tearing.
[ ... ]
> @@ -1688,6 +1702,19 @@ int __add_memory_driver_managed(int nid, u64 start, u64 size,
> if (online_type < MMOP_OFFLINE || online_type > MMOP_ONLINE_MOVABLE)
> return -EINVAL;
>
> + /* Register a private-node owner before adding memory. */
> + if (np) {
> + if (mhp_flags & MHP_NID_IS_MGID) {
> + group = memory_group_find_by_id(nid);
> + if (!group)
> + return -EINVAL;
> + real_nid = group->nid;
> + }
> + rc = node_private_register(real_nid, np);
> + if (rc)
> + return rc;
> + }
> +
> lock_device_hotplug();
[Severity: High]
Is there a race condition here since node_private_register() is called before
acquiring device_hotplug_lock?
If Thread A drops the lock in offline_and_remove_memory_ranges() before
calling node_private_unregister(), Thread B could successfully register
the node here. Thread B then blocks on device_hotplug_lock, allowing Thread A
to clear the registration, which might cause Thread B to online its newly
added private memory incorrectly as normal system RAM.
[ ... ]
> @@ -1702,10 +1729,41 @@ int __add_memory_driver_managed(int nid, u64 start, u64 size,
>
> out_unlock:
> unlock_device_hotplug();
> + if (rc < 0 && np)
> + node_private_unregister(real_nid);
> return rc;
> }
[Severity: High]
If this error path is hit, does node_private_unregister() unconditionally
drop the private node registration based only on the online state, ignoring
offline memory blocks on the same node? This could break previously added
offline blocks on this node.
[ ... ]
> @@ -2535,6 +2610,10 @@ int offline_and_remove_memory_ranges(const struct range *ranges,
> out_unlock:
> unlock_device_hotplug();
>
> + /* Drop private node registration if they are now memoryless */
> + for_each_node_mask(nid, nodes)
> + node_private_unregister(nid);
> +
> kfree(online_types);
> return rc;
> }
[Severity: High]
Could node_private_unregister() blindly clear pgdat->node_private here
if it only checks the online N_MEMORY state instead of present memory?
If remaining memblocks on the node are offline, they might become normal
system RAM when later onlined.
Additionally, because device_hotplug_lock is dropped before unregistering
the private nodes, does this open a race window where a concurrent
add_private_memory_driver_managed() could have its state wiped out?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=25
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (24 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 20:02 ` sashiko-bot
2026-07-22 13:36 ` Richard Cheng
2026-07-20 19:34 ` [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls Gregory Price
` (12 subsequent siblings)
38 siblings, 2 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Provide a mechanism to opt private nodes into the reclaim process.
Reclaim as a "singular service" is actually made up of:
- kswapd reclaim
- direct reclaim
- kcompactd compaction
- direct compaction
- both mglru / lru paths
- madvise reclaim hints
- damon reclaim operations
CAP_RECLAIM gates whether the kernel may do inter-node placement
(compaction) or swap for the folios on that private node.
node_allows_reclaim() encapsulates the policy as a whole. Ordinary
nodes are always reclaimable, private nodes only when opted in.
With the exception of madvise and DAMON, reclaim operations are
highly integrated with one another, so they are opted in/out of
together - otherwise reclaim becomes unpredictable.
To prevent bisect debugging failures, this stays in a single commit.
For example: reclaim without compaction will OOM on high-order
allocation failure despite a large amount of free memory. Normally
compaction would (potentially) resolve this issue.
If a private node opts into reclaim, we create normal watermarks for
that node - otherwise pgdat_balanced() is always true and reclaim
thinks there is no work to do.
Cross-node operations (demotion, promotion, khugepaged) are NOT
included in CAP_RECLAIM because some workflows may desire different
behaviors for this kind of operation:
- prefer direct-to-swap, do not demote
- remain resident and OOM
- __GFP_THISNODE: fail and let the driver augment reclaim
Normal swap-out is allowed because there are clear userland controls
(not registering swap, cgroup.swap, etc) to control that per-workload.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/node_private.h | 33 +++++++++++++++++++++++++
mm/compaction.c | 9 ++++---
mm/damon/paddr.c | 4 +--
mm/huge_memory.c | 2 +-
mm/internal.h | 13 ++++++++++
mm/madvise.c | 6 ++---
mm/memory_hotplug.c | 3 +--
mm/page_alloc.c | 2 +-
mm/vmscan.c | 48 ++++++++++++++++++++++++++++++------
9 files changed, 100 insertions(+), 20 deletions(-)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
index 475496c84249f..f7cbae1309904 100644
--- a/include/linux/node_private.h
+++ b/include/linux/node_private.h
@@ -7,6 +7,13 @@
struct page;
+/*
+ * Per-node service opt-ins (node_private.caps). A private node is isolated
+ * from all general mm services by default; the registering driver sets these
+ * to let specific services operate on its node.
+ */
+#define NODE_PRIVATE_CAP_RECLAIM (1UL << 0) /* allow mm reclaim */
+
/**
* struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
*
@@ -41,6 +48,27 @@ static inline bool node_is_private(int nid)
return node_state(nid, N_MEMORY_PRIVATE);
}
+/**
+ * node_allows_reclaim - may the mm reclaim from this node?
+ * @nid: the node to test
+ *
+ * Only a private node is ever excluded. Every other node can safely
+ * be operated on by reclaim.
+ */
+static inline bool node_allows_reclaim(int nid)
+{
+ struct node_private *np;
+ bool ret;
+
+ if (!node_state(nid, N_MEMORY_PRIVATE))
+ return true;
+ rcu_read_lock();
+ np = rcu_dereference(NODE_DATA(nid)->node_private);
+ ret = np && (np->caps & NODE_PRIVATE_CAP_RECLAIM);
+ rcu_read_unlock();
+ return ret;
+}
+
#else /* !CONFIG_NUMA */
static inline bool folio_is_private_node(struct folio *folio)
@@ -58,6 +86,11 @@ static inline bool node_is_private(int nid)
return false;
}
+static inline bool node_allows_reclaim(int nid)
+{
+ return true;
+}
+
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
diff --git a/mm/compaction.c b/mm/compaction.c
index 8c1351cce7bcc..b9c25c599732f 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -25,6 +25,7 @@
#include <linux/psi.h>
#include <linux/cpuset.h>
#include "page_alloc.h"
+#include <linux/node_private.h>
#include "internal.h"
#ifdef CONFIG_COMPACTION
@@ -2462,7 +2463,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
- if (node_is_private(zone_to_nid(zone)))
+ if (!node_allows_reclaim(zone_to_nid(zone)))
continue;
/*
@@ -2856,7 +2857,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
- if (node_is_private(zone_to_nid(zone)))
+ if (!node_allows_reclaim(zone_to_nid(zone)))
continue;
if (prio > MIN_COMPACT_PRIORITY
@@ -2928,7 +2929,7 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
.proactive_compaction = proactive,
};
- if (node_is_private(pgdat->node_id))
+ if (!node_allows_reclaim(pgdat->node_id))
return 0;
for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
@@ -3026,7 +3027,7 @@ static ssize_t compact_store(struct device *dev,
{
int nid = dev->id;
- if (node_is_private(nid))
+ if (!node_allows_reclaim(nid))
return -EINVAL;
if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index c741a94319750..b668cf55d1b67 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -251,8 +251,8 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
continue;
}
- /* private node memory is not reclaimable by default */
- if (folio_is_private_node(folio))
+ /* DAMOS pageout is reclaim; gate a private node on CAP_RECLAIM */
+ if (!node_allows_reclaim(folio_nid(folio)))
goto put_folio;
if (damos_pa_filter_out(s, folio))
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 1df91b4e5c2bc..cebc89eb0541f 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -2340,7 +2340,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
folio = pmd_folio(orig_pmd);
- if (folio_is_private_node(folio))
+ if (!node_allows_reclaim(folio_nid(folio)))
goto out;
/*
diff --git a/mm/internal.h b/mm/internal.h
index 85c460296cea1..8329034ae561f 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -110,6 +110,19 @@ static inline bool page_is_private_managed(struct page *page)
return folio_is_private_managed(page_folio(page));
}
+/*
+ * folio_allows_madvise() - may madvise reclaim hints act on this folio?
+ *
+ * madvise reclaim hints (COLD/PAGEOUT/FREE) are userland-driven reclaim, so
+ * they follow reclaim opt-in: false for ZONE_DEVICE and for N_MEMORY_PRIVATE
+ * nodes without CAP_RECLAIM, true for all other normal folios.
+ */
+static inline bool folio_allows_madvise(struct folio *folio)
+{
+ return !folio_is_zone_device(folio) &&
+ node_allows_reclaim(folio_nid(folio));
+}
+
/*
* folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
*
diff --git a/mm/madvise.c b/mm/madvise.c
index 29f35a23919a0..56ca974542707 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -396,7 +396,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
folio = pmd_folio(orig_pmd);
- if (folio_is_private_node(folio))
+ if (!node_allows_reclaim(folio_nid(folio)))
goto huge_unlock;
/* Do not interfere with other mappings of this folio */
@@ -478,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
continue;
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_private_managed(folio))
+ if (!folio || !folio_allows_madvise(folio))
continue;
/*
@@ -707,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_private_managed(folio))
+ if (!folio || !folio_allows_madvise(folio))
continue;
/*
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index be230ac9efe5a..1f42ed303366c 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1237,8 +1237,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
/* reinitialise watermarks and update pcp limits */
init_per_zone_wmark_min();
- /* Private nodes opt-out of reclaim/compaction by default */
- if (!node_is_private(nid)) {
+ if (node_allows_reclaim(nid)) {
kswapd_run(nid);
kcompactd_run(nid);
}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2b08bea2379a9..2667a4564b7ac 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6667,7 +6667,7 @@ static void __setup_per_zone_wmarks(void)
u64 tmp;
spin_lock_irqsave(&zone->lock, flags);
- if (node_is_private(zone_to_nid(zone))) {
+ if (!node_allows_reclaim(zone_to_nid(zone))) {
zone->_watermark[WMARK_MIN] = 0;
zone->_watermark[WMARK_LOW] = 0;
zone->_watermark[WMARK_HIGH] = 0;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 86b2334c23b98..f1722693ac2db 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -5396,6 +5396,21 @@ static const struct attribute_group lru_gen_attr_group = {
* debugfs interface
******************************************************************************/
+/*
+ * Nodes the lru_gen debugfs interface lists: ordinary memory nodes plus any
+ * N_MEMORY_PRIVATE nodes opted into reclaim. run_cmd() already accepts the
+ * latter, so keep the listing in sync with what it accepts.
+ */
+static void lru_gen_seq_nodes(nodemask_t *nodes)
+{
+ int nid;
+
+ *nodes = node_states[N_MEMORY];
+ for_each_node_state(nid, N_MEMORY_PRIVATE)
+ if (node_allows_reclaim(nid))
+ node_set(nid, *nodes);
+}
+
static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos)
{
struct mem_cgroup *memcg;
@@ -5407,9 +5422,11 @@ static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos)
memcg = mem_cgroup_iter(NULL, NULL, NULL);
do {
+ nodemask_t nodes;
int nid;
- for_each_node_state(nid, N_MEMORY) {
+ lru_gen_seq_nodes(&nodes);
+ for_each_node_mask(nid, nodes) {
if (!nr_to_skip--)
return get_lruvec(memcg, nid);
}
@@ -5431,16 +5448,18 @@ static void *lru_gen_seq_next(struct seq_file *m, void *v, loff_t *pos)
{
int nid = lruvec_pgdat(v)->node_id;
struct mem_cgroup *memcg = lruvec_memcg(v);
+ nodemask_t nodes;
++*pos;
- nid = next_memory_node(nid);
+ lru_gen_seq_nodes(&nodes);
+ nid = next_node(nid, nodes);
if (nid == MAX_NUMNODES) {
memcg = mem_cgroup_iter(NULL, memcg, NULL);
if (!memcg)
return NULL;
- nid = first_memory_node;
+ nid = first_node(nodes);
}
return get_lruvec(memcg, nid);
@@ -5509,10 +5528,12 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
struct lru_gen_folio *lrugen = &lruvec->lrugen;
int nid = lruvec_pgdat(lruvec)->node_id;
struct mem_cgroup *memcg = lruvec_memcg(lruvec);
+ nodemask_t nodes;
DEFINE_MAX_SEQ(lruvec);
DEFINE_MIN_SEQ(lruvec);
- if (nid == first_memory_node) {
+ lru_gen_seq_nodes(&nodes);
+ if (nid == first_node(nodes)) {
const char *path = memcg ? m->private : "";
#ifdef CONFIG_MEMCG
@@ -5612,7 +5633,9 @@ static int run_cmd(char cmd, u64 memcg_id, int nid, unsigned long seq,
int err = -EINVAL;
struct mem_cgroup *memcg = NULL;
- if (nid < 0 || nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
+ if (nid < 0 || nid >= MAX_NUMNODES ||
+ !(node_state(nid, N_MEMORY) ||
+ (node_is_private(nid) && node_allows_reclaim(nid))))
return -EINVAL;
if (!mem_cgroup_disabled()) {
@@ -6145,7 +6168,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
* Private nodes do not support reclaim by default, filtering here
* captures all normal reclaim paths that may attempt eviction.
*/
- if (node_is_private(pgdat->node_id))
+ if (!node_allows_reclaim(pgdat->node_id))
return;
if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
@@ -6758,6 +6781,16 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg,
return sc.nr_reclaimed;
}
+static struct zonelist *memcg_reclaim_zonelist(int nid, gfp_t gfp_mask)
+{
+ unsigned int aflags = ALLOC_DEFAULT;
+
+ if (unlikely(!nodes_empty(node_states[N_MEMORY_PRIVATE])))
+ aflags = ALLOC_ZONELIST_PRIVATE;
+
+ return select_zonelist(nid, gfp_mask, aflags);
+}
+
unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
unsigned long nr_pages,
gfp_t gfp_mask,
@@ -6784,7 +6817,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
* equal pressure on all the nodes. This is based on the assumption that
* the reclaim does not bail out early.
*/
- struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask);
+ struct zonelist *zonelist = memcg_reclaim_zonelist(numa_node_id(),
+ sc.gfp_mask);
set_task_reclaim_state(current, &sc.reclaim_state);
trace_mm_vmscan_memcg_reclaim_begin(sc.gfp_mask, 0, memcg);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
2026-07-20 19:34 ` [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim Gregory Price
@ 2026-07-21 20:02 ` sashiko-bot
2026-07-22 13:36 ` Richard Cheng
1 sibling, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 20:02 UTC (permalink / raw)
To: Gregory Price; +Cc: kvm, tj, mkoutny, damon, hannes, linux-cxl, nvdimm
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] calculate_totalreserve_pages() unconditionally skips private nodes, failing to account for their watermarks in totalreserve_pages.
- [Medium] Incorrect zone watermark calculation for private nodes allowing reclaim.
--
--- Patch [26]: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim ---
commit 29ab758bf14800171bcfa7488632ce55ed680b45
Author: Gregory Price <gourry@gourry.net>
Subject: mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
This commit introduces NODE_PRIVATE_CAP_RECLAIM to allow private memory
nodes to opt into reclaim processes. It replaces various node_is_private()
checks with node_allows_reclaim() across the memory management subsystem.
Private nodes that opt in will now have normal watermarks calculated for
them.
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 2b08bea2379a9..2667a4564b7ac 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -6667,7 +6667,7 @@ static void __setup_per_zone_wmarks(void)
> u64 tmp;
>
> spin_lock_irqsave(&zone->lock, flags);
> - if (node_is_private(zone_to_nid(zone))) {
> + if (!node_allows_reclaim(zone_to_nid(zone))) {
> zone->_watermark[WMARK_MIN] = 0;
> zone->_watermark[WMARK_LOW] = 0;
> zone->_watermark[WMARK_HIGH] = 0;
[Severity: Medium]
Will this lead to an incorrect watermark calculation for private nodes that
allow reclaim?
Earlier in __setup_per_zone_wmarks(), lowmem_pages is calculated by
explicitly excluding private memory:
for_each_zone(zone) {
if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE &&
!node_is_private(zone_to_nid(zone)))
lowmem_pages += zone_managed_pages(zone);
}
Because lowmem_pages excludes private memory, when we calculate WMARK_MIN
for a private node later in this function using:
tmp = (u64)pages_min * zone_managed_pages(zone);
tmp = div64_ul(tmp, lowmem_pages);
does this mean the private node will reserve a disproportionately high
percentage of its memory if the system has a small amount of normal memory
and a large private node?
[Severity: Medium]
Also, does this miss updating the totalreserve_pages counter?
At the end of __setup_per_zone_wmarks(), we call
calculate_totalreserve_pages(). However, that function unconditionally
skips all private nodes:
for_each_online_pgdat(pgdat) {
/* private nodes have zero watermarks */
if (node_is_private(pgdat->node_id))
continue;
If a private node has CAP_RECLAIM and non-zero watermarks, does skipping it
here cause totalreserve_pages to remain artificially low?
This might cause node_dirtyable_memory() and global_dirtyable_memory() to
overestimate available memory, which could inflate dirty limits and
writeback thresholds.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=26
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
2026-07-20 19:34 ` [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim Gregory Price
2026-07-21 20:02 ` sashiko-bot
@ 2026-07-22 13:36 ` Richard Cheng
2026-07-22 13:48 ` Gregory Price
1 sibling, 1 reply; 82+ messages in thread
From: Richard Cheng @ 2026-07-22 13:36 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, Jul 20, 2026 at 03:34:20PM +0800, Gregory Price wrote:
> Provide a mechanism to opt private nodes into the reclaim process.
>
> Reclaim as a "singular service" is actually made up of:
>
> - kswapd reclaim
> - direct reclaim
> - kcompactd compaction
> - direct compaction
> - both mglru / lru paths
> - madvise reclaim hints
> - damon reclaim operations
>
> CAP_RECLAIM gates whether the kernel may do inter-node placement
> (compaction) or swap for the folios on that private node.
>
> node_allows_reclaim() encapsulates the policy as a whole. Ordinary
> nodes are always reclaimable, private nodes only when opted in.
>
> With the exception of madvise and DAMON, reclaim operations are
> highly integrated with one another, so they are opted in/out of
> together - otherwise reclaim becomes unpredictable.
>
> To prevent bisect debugging failures, this stays in a single commit.
>
> For example: reclaim without compaction will OOM on high-order
> allocation failure despite a large amount of free memory. Normally
> compaction would (potentially) resolve this issue.
>
> If a private node opts into reclaim, we create normal watermarks for
> that node - otherwise pgdat_balanced() is always true and reclaim
> thinks there is no work to do.
>
> Cross-node operations (demotion, promotion, khugepaged) are NOT
> included in CAP_RECLAIM because some workflows may desire different
> behaviors for this kind of operation:
>
> - prefer direct-to-swap, do not demote
> - remain resident and OOM
> - __GFP_THISNODE: fail and let the driver augment reclaim
>
> Normal swap-out is allowed because there are clear userland controls
> (not registering swap, cgroup.swap, etc) to control that per-workload.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> include/linux/node_private.h | 33 +++++++++++++++++++++++++
> mm/compaction.c | 9 ++++---
> mm/damon/paddr.c | 4 +--
> mm/huge_memory.c | 2 +-
> mm/internal.h | 13 ++++++++++
> mm/madvise.c | 6 ++---
> mm/memory_hotplug.c | 3 +--
> mm/page_alloc.c | 2 +-
> mm/vmscan.c | 48 ++++++++++++++++++++++++++++++------
> 9 files changed, 100 insertions(+), 20 deletions(-)
>
> diff --git a/include/linux/node_private.h b/include/linux/node_private.h
> index 475496c84249f..f7cbae1309904 100644
> --- a/include/linux/node_private.h
> +++ b/include/linux/node_private.h
> @@ -7,6 +7,13 @@
>
> struct page;
>
> +/*
> + * Per-node service opt-ins (node_private.caps). A private node is isolated
> + * from all general mm services by default; the registering driver sets these
> + * to let specific services operate on its node.
> + */
> +#define NODE_PRIVATE_CAP_RECLAIM (1UL << 0) /* allow mm reclaim */
> +
> /**
> * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
> *
> @@ -41,6 +48,27 @@ static inline bool node_is_private(int nid)
> return node_state(nid, N_MEMORY_PRIVATE);
> }
>
> +/**
> + * node_allows_reclaim - may the mm reclaim from this node?
> + * @nid: the node to test
> + *
> + * Only a private node is ever excluded. Every other node can safely
> + * be operated on by reclaim.
> + */
> +static inline bool node_allows_reclaim(int nid)
> +{
> + struct node_private *np;
> + bool ret;
> +
> + if (!node_state(nid, N_MEMORY_PRIVATE))
> + return true;
> + rcu_read_lock();
> + np = rcu_dereference(NODE_DATA(nid)->node_private);
> + ret = np && (np->caps & NODE_PRIVATE_CAP_RECLAIM);
> + rcu_read_unlock();
> + return ret;
> +}
> +
> #else /* !CONFIG_NUMA */
>
> static inline bool folio_is_private_node(struct folio *folio)
> @@ -58,6 +86,11 @@ static inline bool node_is_private(int nid)
> return false;
> }
>
> +static inline bool node_allows_reclaim(int nid)
> +{
> + return true;
> +}
> +
> #endif /* CONFIG_NUMA */
>
> #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 8c1351cce7bcc..b9c25c599732f 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -25,6 +25,7 @@
> #include <linux/psi.h>
> #include <linux/cpuset.h>
> #include "page_alloc.h"
> +#include <linux/node_private.h>
> #include "internal.h"
>
> #ifdef CONFIG_COMPACTION
> @@ -2462,7 +2463,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
> !__cpuset_zone_allowed(zone, gfp_mask))
> continue;
>
> - if (node_is_private(zone_to_nid(zone)))
> + if (!node_allows_reclaim(zone_to_nid(zone)))
> continue;
>
> /*
> @@ -2856,7 +2857,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
> !__cpuset_zone_allowed(zone, gfp_mask))
> continue;
>
> - if (node_is_private(zone_to_nid(zone)))
> + if (!node_allows_reclaim(zone_to_nid(zone)))
> continue;
>
> if (prio > MIN_COMPACT_PRIORITY
> @@ -2928,7 +2929,7 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
> .proactive_compaction = proactive,
> };
>
> - if (node_is_private(pgdat->node_id))
> + if (!node_allows_reclaim(pgdat->node_id))
> return 0;
>
> for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
> @@ -3026,7 +3027,7 @@ static ssize_t compact_store(struct device *dev,
> {
> int nid = dev->id;
>
> - if (node_is_private(nid))
> + if (!node_allows_reclaim(nid))
> return -EINVAL;
>
> if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
> diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
> index c741a94319750..b668cf55d1b67 100644
> --- a/mm/damon/paddr.c
> +++ b/mm/damon/paddr.c
> @@ -251,8 +251,8 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
> continue;
> }
>
> - /* private node memory is not reclaimable by default */
> - if (folio_is_private_node(folio))
> + /* DAMOS pageout is reclaim; gate a private node on CAP_RECLAIM */
> + if (!node_allows_reclaim(folio_nid(folio)))
> goto put_folio;
>
> if (damos_pa_filter_out(s, folio))
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index 1df91b4e5c2bc..cebc89eb0541f 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -2340,7 +2340,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
>
> folio = pmd_folio(orig_pmd);
>
> - if (folio_is_private_node(folio))
> + if (!node_allows_reclaim(folio_nid(folio)))
> goto out;
>
> /*
> diff --git a/mm/internal.h b/mm/internal.h
> index 85c460296cea1..8329034ae561f 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -110,6 +110,19 @@ static inline bool page_is_private_managed(struct page *page)
> return folio_is_private_managed(page_folio(page));
> }
>
> +/*
> + * folio_allows_madvise() - may madvise reclaim hints act on this folio?
> + *
> + * madvise reclaim hints (COLD/PAGEOUT/FREE) are userland-driven reclaim, so
> + * they follow reclaim opt-in: false for ZONE_DEVICE and for N_MEMORY_PRIVATE
> + * nodes without CAP_RECLAIM, true for all other normal folios.
> + */
> +static inline bool folio_allows_madvise(struct folio *folio)
> +{
> + return !folio_is_zone_device(folio) &&
> + node_allows_reclaim(folio_nid(folio));
> +}
> +
> /*
> * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
> *
> diff --git a/mm/madvise.c b/mm/madvise.c
> index 29f35a23919a0..56ca974542707 100644
> --- a/mm/madvise.c
> +++ b/mm/madvise.c
> @@ -396,7 +396,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
>
> folio = pmd_folio(orig_pmd);
>
> - if (folio_is_private_node(folio))
> + if (!node_allows_reclaim(folio_nid(folio)))
> goto huge_unlock;
>
> /* Do not interfere with other mappings of this folio */
> @@ -478,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
> continue;
>
> folio = vm_normal_folio(vma, addr, ptent);
> - if (!folio || folio_is_private_managed(folio))
> + if (!folio || !folio_allows_madvise(folio))
> continue;
>
> /*
> @@ -707,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
> }
>
> folio = vm_normal_folio(vma, addr, ptent);
> - if (!folio || folio_is_private_managed(folio))
> + if (!folio || !folio_allows_madvise(folio))
> continue;
>
> /*
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index be230ac9efe5a..1f42ed303366c 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1237,8 +1237,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
> /* reinitialise watermarks and update pcp limits */
> init_per_zone_wmark_min();
>
> - /* Private nodes opt-out of reclaim/compaction by default */
> - if (!node_is_private(nid)) {
> + if (node_allows_reclaim(nid)) {
> kswapd_run(nid);
> kcompactd_run(nid);
> }
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 2b08bea2379a9..2667a4564b7ac 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -6667,7 +6667,7 @@ static void __setup_per_zone_wmarks(void)
> u64 tmp;
>
> spin_lock_irqsave(&zone->lock, flags);
> - if (node_is_private(zone_to_nid(zone))) {
> + if (!node_allows_reclaim(zone_to_nid(zone))) {
> zone->_watermark[WMARK_MIN] = 0;
> zone->_watermark[WMARK_LOW] = 0;
> zone->_watermark[WMARK_HIGH] = 0;
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index 86b2334c23b98..f1722693ac2db 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -5396,6 +5396,21 @@ static const struct attribute_group lru_gen_attr_group = {
> * debugfs interface
> ******************************************************************************/
>
> +/*
> + * Nodes the lru_gen debugfs interface lists: ordinary memory nodes plus any
> + * N_MEMORY_PRIVATE nodes opted into reclaim. run_cmd() already accepts the
> + * latter, so keep the listing in sync with what it accepts.
> + */
> +static void lru_gen_seq_nodes(nodemask_t *nodes)
> +{
> + int nid;
> +
> + *nodes = node_states[N_MEMORY];
> + for_each_node_state(nid, N_MEMORY_PRIVATE)
> + if (node_allows_reclaim(nid))
> + node_set(nid, *nodes);
> +}
> +
> static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos)
> {
> struct mem_cgroup *memcg;
> @@ -5407,9 +5422,11 @@ static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos)
>
> memcg = mem_cgroup_iter(NULL, NULL, NULL);
> do {
> + nodemask_t nodes;
> int nid;
>
> - for_each_node_state(nid, N_MEMORY) {
> + lru_gen_seq_nodes(&nodes);
> + for_each_node_mask(nid, nodes) {
> if (!nr_to_skip--)
> return get_lruvec(memcg, nid);
> }
> @@ -5431,16 +5448,18 @@ static void *lru_gen_seq_next(struct seq_file *m, void *v, loff_t *pos)
> {
> int nid = lruvec_pgdat(v)->node_id;
> struct mem_cgroup *memcg = lruvec_memcg(v);
> + nodemask_t nodes;
>
> ++*pos;
>
> - nid = next_memory_node(nid);
> + lru_gen_seq_nodes(&nodes);
> + nid = next_node(nid, nodes);
> if (nid == MAX_NUMNODES) {
> memcg = mem_cgroup_iter(NULL, memcg, NULL);
> if (!memcg)
> return NULL;
>
> - nid = first_memory_node;
> + nid = first_node(nodes);
> }
>
> return get_lruvec(memcg, nid);
> @@ -5509,10 +5528,12 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
> struct lru_gen_folio *lrugen = &lruvec->lrugen;
> int nid = lruvec_pgdat(lruvec)->node_id;
> struct mem_cgroup *memcg = lruvec_memcg(lruvec);
> + nodemask_t nodes;
> DEFINE_MAX_SEQ(lruvec);
> DEFINE_MIN_SEQ(lruvec);
>
> - if (nid == first_memory_node) {
> + lru_gen_seq_nodes(&nodes);
> + if (nid == first_node(nodes)) {
> const char *path = memcg ? m->private : "";
>
> #ifdef CONFIG_MEMCG
> @@ -5612,7 +5633,9 @@ static int run_cmd(char cmd, u64 memcg_id, int nid, unsigned long seq,
> int err = -EINVAL;
> struct mem_cgroup *memcg = NULL;
>
> - if (nid < 0 || nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
> + if (nid < 0 || nid >= MAX_NUMNODES ||
> + !(node_state(nid, N_MEMORY) ||
> + (node_is_private(nid) && node_allows_reclaim(nid))))
> return -EINVAL;
>
> if (!mem_cgroup_disabled()) {
> @@ -6145,7 +6168,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
> * Private nodes do not support reclaim by default, filtering here
> * captures all normal reclaim paths that may attempt eviction.
> */
> - if (node_is_private(pgdat->node_id))
> + if (!node_allows_reclaim(pgdat->node_id))
> return;
>
> if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
> @@ -6758,6 +6781,16 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg,
> return sc.nr_reclaimed;
> }
>
> +static struct zonelist *memcg_reclaim_zonelist(int nid, gfp_t gfp_mask)
> +{
> + unsigned int aflags = ALLOC_DEFAULT;
> +
> + if (unlikely(!nodes_empty(node_states[N_MEMORY_PRIVATE])))
> + aflags = ALLOC_ZONELIST_PRIVATE;
> +
> + return select_zonelist(nid, gfp_mask, aflags);
> +}
I applied this code change and notice in mem_cgroup_shrink_node(), it calls
shrink_lruvec() directly and never passes through shrink_node().
This patch makes private nodes reachable through memcg reclaim zonelist, but
what prevents the direct per-node memcg path from reclaiming a private node
that has not set NODE_PRIVATE_CAP_RECLAIM ?
--Richard
> +
> unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
> unsigned long nr_pages,
> gfp_t gfp_mask,
> @@ -6784,7 +6817,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
> * equal pressure on all the nodes. This is based on the assumption that
> * the reclaim does not bail out early.
> */
> - struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask);
> + struct zonelist *zonelist = memcg_reclaim_zonelist(numa_node_id(),
> + sc.gfp_mask);
>
> set_task_reclaim_state(current, &sc.reclaim_state);
> trace_mm_vmscan_memcg_reclaim_begin(sc.gfp_mask, 0, memcg);
> --
> 2.53.0-Meta
>
>
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
2026-07-22 13:36 ` Richard Cheng
@ 2026-07-22 13:48 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-22 13:48 UTC (permalink / raw)
To: Richard Cheng
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 09:36:14PM +0800, Richard Cheng wrote:
> On Mon, Jul 20, 2026 at 03:34:20PM +0800, Gregory Price wrote:
> > +static struct zonelist *memcg_reclaim_zonelist(int nid, gfp_t gfp_mask)
> > +{
> > + unsigned int aflags = ALLOC_DEFAULT;
> > +
> > + if (unlikely(!nodes_empty(node_states[N_MEMORY_PRIVATE])))
> > + aflags = ALLOC_ZONELIST_PRIVATE;
> > +
> > + return select_zonelist(nid, gfp_mask, aflags);
> > +}
>
>
> I applied this code change and notice in mem_cgroup_shrink_node(), it calls
> shrink_lruvec() directly and never passes through shrink_node().
>
> This patch makes private nodes reachable through memcg reclaim zonelist, but
> what prevents the direct per-node memcg path from reclaiming a private node
> that has not set NODE_PRIVATE_CAP_RECLAIM ?
>
I legitimately missed this.
Sashiko also picked it up on the filter patch:
https://sashiko.dev/#/message/20260721183528.B4B9C1F00A3A%40smtp.kernel.org
If nothing else - implementing this feature has taught me some
sub-components of mm/ have very well centralized logic, and others (like
vmscan.c) have 3 or 4 different paths in with a lot of strange quirks
carved out.
I think some of the vmscan paths can be improved to simplify this
interaction, but i will fix this particular miss in v6 and come back
around to see whether we can clean up these different paths as a general
improvement.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (25 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 20:13 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 28/36] mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes Gregory Price
` (11 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Provide a mechanism to opt private nodes into userland numa management.
Add node_allows_user_numa() to encapsulate whether a node supports
userland NUMA controls (always true for normal nodes).
Placement - setting mempolicy via:
- mbind()
- set_mempolicy()
- set_mempolicy_home_node()
For mempolicy, enforcement lives in one place: mpol_set_nodemask()
Private nodes are not N_MEMORY, so they are trimmed from a nodemask
like a cpuset-trimmed node. All-private nodemasks without a valid
node collapse to empty and the mempolicy fails.
home_node is not special-cased - it is only a preferred-nid hint, and
placement is governed by the bind nodemask, so a home node pointed at
an invalid node simply falls back via the normal fallback zonelists.
(All the same behavior as a node/mask not intersecting cpuset.mems).
Migration - relocation of pages to/from a private node:
- mbind(MPOL_MF_MOVE)
- move_pages()
- migrate_pages()
mbind(MPOL_MF_MOVE) is a migration, so mempolicy and migration share
a single opt-in control.
The migration interfaces all check node eligibility and use
ALLOC_ZONELIST_PRIVATE to allow eligible migration requests to
move a folio to a private node..
alloc_migration_target() carries mtc->zlsel into the allocator
via __folio_alloc_zonelist().
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/node_private.h | 33 +++++++++++++++++++++++++++++++++
mm/internal.h | 1 +
mm/mempolicy.c | 27 ++++++++++++++++++++-------
mm/migrate.c | 19 ++++++++++++++-----
4 files changed, 68 insertions(+), 12 deletions(-)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
index f7cbae1309904..655fe9ec5cb61 100644
--- a/include/linux/node_private.h
+++ b/include/linux/node_private.h
@@ -13,6 +13,7 @@ struct page;
* to let specific services operate on its node.
*/
#define NODE_PRIVATE_CAP_RECLAIM (1UL << 0) /* allow mm reclaim */
+#define NODE_PRIVATE_CAP_USER_NUMA (1UL << 1) /* allow mempolicy */
/**
* struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -69,6 +70,33 @@ static inline bool node_allows_reclaim(int nid)
return ret;
}
+/**
+ * node_allows_user_numa - may userspace place or migrate memory here?
+ * @nid: the node to test
+ *
+ * Gate all userspace-directed memory operations on a private node.
+ * - mbind()/set_mempolicy()
+ * - move_pages()/migrate_pages()
+ *
+ * return: true for N_MEMORY and N_MEMORY_PRIVATE with CAP_USER_NUMA.
+ * false for memoryless or opted-out private node.
+ */
+static inline bool node_allows_user_numa(int nid)
+{
+ struct node_private *np;
+ bool ret;
+
+ if (node_state(nid, N_MEMORY))
+ return true;
+ if (!node_state(nid, N_MEMORY_PRIVATE))
+ return false;
+ rcu_read_lock();
+ np = rcu_dereference(NODE_DATA(nid)->node_private);
+ ret = np && (np->caps & NODE_PRIVATE_CAP_USER_NUMA);
+ rcu_read_unlock();
+ return ret;
+}
+
#else /* !CONFIG_NUMA */
static inline bool folio_is_private_node(struct folio *folio)
@@ -91,6 +119,11 @@ static inline bool node_allows_reclaim(int nid)
return true;
}
+static inline bool node_allows_user_numa(int nid)
+{
+ return true;
+}
+
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
diff --git a/mm/internal.h b/mm/internal.h
index 8329034ae561f..62e68acae08a3 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1250,6 +1250,7 @@ struct migration_target_control {
nodemask_t *nmask;
gfp_t gfp_mask;
enum migrate_reason reason;
+ unsigned int alloc_flags;
};
/*
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index a3ffb09897489..fe42a510590a2 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -434,13 +434,14 @@ static int mpol_set_nodemask(struct mempolicy *pol,
/*
* Private nodes are not in cpuset.mems, so they're always stripped.
- * Driver-allocated policies will already have MPOL_F_PRIVATE set,
- * if that's the case, add back in the requested set of private nodes.
+ * Driver-allocated policies (MPOL_F_PRIVATE) and CAP_USER_NUMA private
+ * nodes should be added back into the nodemask.
*/
for_each_node_mask(nid, *nodes) {
if (!node_is_private(nid))
continue;
- if (pol->flags & MPOL_F_PRIVATE)
+ if ((pol->flags & MPOL_F_PRIVATE) ||
+ node_allows_user_numa(nid))
node_set(nid, nsc->mask2);
}
@@ -696,7 +697,7 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
}
if (!queue_folio_required(folio, qp))
return;
- if (folio_is_private_node(folio))
+ if (!node_allows_user_numa(folio_nid(folio)))
return;
if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma) ||
@@ -752,7 +753,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
continue;
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_private_managed(folio))
+ if (!folio || folio_is_zone_device(folio) ||
+ !node_allows_user_numa(folio_nid(folio)))
continue;
if (folio_test_large(folio) && max_nr != 1)
nr = folio_pte_batch(folio, pte, ptent, max_nr);
@@ -827,7 +829,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
folio = pfn_folio(pte_pfn(ptep));
if (!queue_folio_required(folio, qp))
goto unlock;
- if (folio_is_private_node(folio))
+ if (!node_allows_user_numa(folio_nid(folio)))
goto unlock;
if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma)) {
@@ -1412,6 +1414,8 @@ static long migrate_to_node(struct mm_struct *mm, int source, int dest,
.nid = dest,
.gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
.reason = MR_SYSCALL,
+ .alloc_flags = node_is_private(dest) ?
+ ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT,
};
nodes_clear(nmask);
@@ -1985,9 +1989,10 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
struct mm_struct *mm = NULL;
struct task_struct *task;
nodemask_t task_nodes;
- int err;
+ nodemask_t priv_ok;
nodemask_t *old;
nodemask_t *new;
+ int err, nid;
NODEMASK_SCRATCH(scratch);
if (!scratch)
@@ -2027,7 +2032,14 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
}
rcu_read_unlock();
+ /* Private nodes are stripped by cpuset checks. Allow eligible ones. */
+ nodes_clear(priv_ok);
+ for_each_node_mask(nid, *new)
+ if (node_is_private(nid) && node_allows_user_numa(nid))
+ node_set(nid, priv_ok);
+
task_nodes = cpuset_mems_allowed(task);
+ nodes_or(task_nodes, task_nodes, priv_ok);
/* Is the user allowed to access the target nodes? */
if (!nodes_subset(*new, task_nodes) && !capable(CAP_SYS_NICE)) {
err = -EPERM;
@@ -2035,6 +2047,7 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
}
task_nodes = cpuset_mems_allowed(current);
+ nodes_or(task_nodes, task_nodes, priv_ok);
if (!nodes_and(*new, *new, task_nodes))
goto out_put;
diff --git a/mm/migrate.c b/mm/migrate.c
index d20674c07b947..b548d79352a38 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2231,7 +2231,8 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
gfp_mask |= __GFP_HIGHMEM;
- return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
+ return __folio_alloc(gfp_mask, order, nid, mtc->nmask,
+ mtc->alloc_flags);
}
#ifdef CONFIG_NUMA_MIGRATION
@@ -2253,6 +2254,8 @@ static int do_move_pages_to_node(struct list_head *pagelist, int node)
.nid = node,
.gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
.reason = MR_SYSCALL,
+ .alloc_flags = node_is_private(node) ?
+ ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT,
};
err = migrate_pages(pagelist, alloc_migration_target, NULL,
@@ -2268,7 +2271,8 @@ static int __add_folio_for_migration(struct folio *folio, int node,
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
return -EFAULT;
- if (folio_is_private_managed(folio))
+ if (folio_is_zone_device(folio) ||
+ !node_allows_user_numa(folio_nid(folio)))
return -ENOENT;
if (folio_nid(folio) == node)
@@ -2392,11 +2396,14 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
err = -ENODEV;
if (node < 0 || node >= MAX_NUMNODES)
goto out_flush;
- if (!node_state(node, N_MEMORY))
+
+ if (!node_allows_user_numa(node))
goto out_flush;
err = -EACCES;
- if (!node_isset(node, task_nodes))
+ /* Private nodes are not partitioned by cpuset.mem */
+ if (!node_is_private(node) &&
+ !node_isset(node, task_nodes))
goto out_flush;
if (current_node == NUMA_NO_NODE) {
@@ -2477,7 +2484,9 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
if (folio) {
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
err = -EFAULT;
- else if (folio_is_private_managed(folio))
+ else if (folio_is_zone_device(folio) ||
+ (folio_is_private_node(folio) &&
+ !node_allows_user_numa(folio_nid(folio))))
err = -ENOENT;
else
err = folio_nid(folio);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
2026-07-20 19:34 ` [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls Gregory Price
@ 2026-07-21 20:13 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 20:13 UTC (permalink / raw)
To: Gregory Price; +Cc: nvdimm, damon, linux-cxl, tj, mkoutny, hannes, kvm
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
New issues:
- [Medium] Relative node indices are incorrectly treated as absolute node IDs for private node checking in mpol_set_nodemask.
- [Medium] Hugetlb folio migration to private nodes unconditionally fails because the hugetlb allocator ignores ALLOC_ZONELIST_PRIVATE.
Pre-existing issues:
- [High] Returning ACTION_AGAIN on pte_offset_map_lock() failure causes an infinite loop in the page table walker.
--
--- Patch [27]: [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls ---
commit 90c010ac54e1befaa9a30ca8c81fdac4b3524857
Author: Gregory Price <gourry@gourry.net>
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
This commit provides a mechanism to opt private nodes into userland numa
management. It adds node_allows_user_numa() to encapsulate whether a node
supports userland NUMA controls and modifies the memory policy and migration
interfaces to support these nodes.
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index a3ffb09897489..fe42a510590a2 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -434,13 +434,14 @@ static int mpol_set_nodemask(struct mempolicy *pol,
>
> /*
> * Private nodes are not in cpuset.mems, so they're always stripped.
> - * Driver-allocated policies will already have MPOL_F_PRIVATE set,
> - * if that's the case, add back in the requested set of private nodes.
> + * Driver-allocated policies (MPOL_F_PRIVATE) and CAP_USER_NUMA private
> + * nodes should be added back into the nodemask.
> */
> for_each_node_mask(nid, *nodes) {
> if (!node_is_private(nid))
> continue;
> - if (pol->flags & MPOL_F_PRIVATE)
> + if ((pol->flags & MPOL_F_PRIVATE) ||
> + node_allows_user_numa(nid))
> node_set(nid, nsc->mask2);
> }
[Severity: Medium]
When MPOL_F_RELATIVE_NODES is set, the *nodes mask contains relative indices
instead of absolute node IDs. Does this loop incorrectly evaluate those
relative indices as absolute node IDs when calling node_is_private(nid)?
If the relative index happens to match an absolute private node ID, it appears
that node could be erroneously added to the final policy mask.
[ ... ]
> @@ -752,7 +753,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
> continue;
> }
> folio = vm_normal_folio(vma, addr, ptent);
> - if (!folio || folio_is_private_managed(folio))
> + if (!folio || folio_is_zone_device(folio) ||
> + !node_allows_user_numa(folio_nid(folio)))
> continue;
> if (folio_test_large(folio) && max_nr != 1)
> nr = folio_pte_batch(folio, pte, ptent, max_nr);
[Severity: High]
This is a pre-existing issue, but earlier in queue_folios_pte_range() we
return ACTION_AGAIN if pte_offset_map_lock() fails:
queue_folios_pte_range()
...
mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
if (!pte) {
walk->action = ACTION_AGAIN;
return 0;
}
Does setting ACTION_AGAIN cause an infinite retry loop in the pagewalker
when pte_offset_map_lock() fails non-transiently, such as when encountering
migration entries?
> diff --git a/mm/migrate.c b/mm/migrate.c
> index d20674c07b947..b548d79352a38 100644
> --- a/mm/migrate.c
> +++ b/mm/migrate.c
> @@ -2231,7 +2231,8 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
> if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
> gfp_mask |= __GFP_HIGHMEM;
>
> - return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
> + return __folio_alloc(gfp_mask, order, nid, mtc->nmask,
> + mtc->alloc_flags);
> }
[Severity: Medium]
This correctly updates the standard folio allocation to use mtc->alloc_flags,
but earlier in alloc_migration_target(), the hugetlb allocation path does
not pass this flag:
if (folio_test_hugetlb(src)) {
...
return alloc_hugetlb_folio_nodemask(h, nid, mtc->nmask, gfp_mask,
htlb_allow_alloc_fallback(mtc->reason));
}
Will hugetlb migration to private nodes fail unconditionally because it misses
the ALLOC_ZONELIST_PRIVATE flag needed by the hugetlb allocator?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=27
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 28/36] mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (26 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-22 14:04 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion Gregory Price
` (10 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Add CAP_HOTUNPLUG, which allows memory_hotplug to migrate a private
node's folios for the purpose of hotunplugging memory.
Without this, hotunplug fails if any folio on the node is allocated.
Add node_allows_hotunplug() - which returns true for any normal node
and private nodes with CAP_HOTUNPLUG.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/node_private.h | 26 ++++++++++++++++++++++++++
mm/memory_hotplug.c | 4 ++--
2 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
index 655fe9ec5cb61..7b617b1fa9c28 100644
--- a/include/linux/node_private.h
+++ b/include/linux/node_private.h
@@ -14,6 +14,7 @@ struct page;
*/
#define NODE_PRIVATE_CAP_RECLAIM (1UL << 0) /* allow mm reclaim */
#define NODE_PRIVATE_CAP_USER_NUMA (1UL << 1) /* allow mempolicy */
+#define NODE_PRIVATE_CAP_HOTUNPLUG (1UL << 2) /* allow hot-unplug */
/**
* struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -97,6 +98,26 @@ static inline bool node_allows_user_numa(int nid)
return ret;
}
+/**
+ * node_allows_hotunplug - may hot-unplug migrate this node's folios?
+ * @nid: the node to test
+ *
+ * True for normal nodes and private nodes opted into CAP_HOTUNPLUG.
+ */
+static inline bool node_allows_hotunplug(int nid)
+{
+ struct node_private *np;
+ bool ret;
+
+ if (!node_state(nid, N_MEMORY_PRIVATE))
+ return true;
+ rcu_read_lock();
+ np = rcu_dereference(NODE_DATA(nid)->node_private);
+ ret = np && (np->caps & NODE_PRIVATE_CAP_HOTUNPLUG);
+ rcu_read_unlock();
+ return ret;
+}
+
#else /* !CONFIG_NUMA */
static inline bool folio_is_private_node(struct folio *folio)
@@ -124,6 +145,11 @@ static inline bool node_allows_user_numa(int nid)
return true;
}
+static inline bool node_allows_hotunplug(int nid)
+{
+ return true;
+}
+
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 1f42ed303366c..e9573f90e0dcd 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1943,8 +1943,8 @@ static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
goto put_folio;
}
- /* Private node folios cannot migrate, fail outright */
- if (folio_is_private_node(folio)) {
+ /* Fail outright on private nodes w/o hotunplug support */
+ if (!node_allows_hotunplug(folio_nid(folio))) {
pr_info_ratelimited("memory offline refused: node %d pfn %lx\n",
folio_nid(folio), pfn);
folio_put(folio);
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 28/36] mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
2026-07-20 19:34 ` [PATCH v5 28/36] mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes Gregory Price
@ 2026-07-22 14:04 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-22 14:04 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, Jul 20, 2026 at 03:34:22PM -0400, Gregory Price wrote:
> Add CAP_HOTUNPLUG, which allows memory_hotplug to migrate a private
> node's folios for the purpose of hotunplugging memory.
>
> Without this, hotunplug fails if any folio on the node is allocated.
>
> Add node_allows_hotunplug() - which returns true for any normal node
> and private nodes with CAP_HOTUNPLUG.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
after this question:
https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#m5b8166212a276fece2380f2dd03d6864375a576b
and some off-list feedback about limiting initial cap bits to active
use cases, I realized that this CAP bit can be entirely eliminated.
hotplug notifier callbacks already allow a driver to prevent memory
block state transitions - so that control already exists.
v6 will remove both the filtering here and the cap bit.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (27 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 28/36] mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 20:28 ` sashiko-bot
2026-07-20 19:34 ` [PATCH v5 30/36] mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing Gregory Price
` (9 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
A private node is invisible to the tiering/demotion hierarchy by default:
memory-tiers.c never includes it, so reclaim never demotes onto it.
Add NODE_PRIVATE_CAP_DEMOTION to opt a private node into demotion.
When set, memory-tiers adds the node to the demotion set and reclaim's
demote path may target it (allocating from the private zonelist).
Demotion is driven by reclaim, so CAP_DEMOTION requires CAP_RECLAIM.
Otherwise the node can either fill up and drive odd system-wide OOM
behavior, or demotion doesn't work (nothing can demote from the node).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
drivers/base/node.c | 5 +++++
include/linux/node_private.h | 30 +++++++++++++++++++++++++++++
mm/memory-tiers.c | 37 +++++++++++++++++++++++++++---------
mm/vmscan.c | 4 ++++
4 files changed, 67 insertions(+), 9 deletions(-)
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 94cd51f51b7e8..3d61ca1b805dc 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -896,6 +896,11 @@ int node_private_register(int nid, struct node_private *np)
if (!np || !node_possible(nid))
return -EINVAL;
+ /* Demotion is driven by reclaim, so it requires reclaim. */
+ if ((np->caps & NODE_PRIVATE_CAP_DEMOTION) &&
+ !(np->caps & NODE_PRIVATE_CAP_RECLAIM))
+ return -EINVAL;
+
mutex_lock(&node_private_lock);
mem_hotplug_begin();
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
index 7b617b1fa9c28..87b03444b2c97 100644
--- a/include/linux/node_private.h
+++ b/include/linux/node_private.h
@@ -15,6 +15,7 @@ struct page;
#define NODE_PRIVATE_CAP_RECLAIM (1UL << 0) /* allow mm reclaim */
#define NODE_PRIVATE_CAP_USER_NUMA (1UL << 1) /* allow mempolicy */
#define NODE_PRIVATE_CAP_HOTUNPLUG (1UL << 2) /* allow hot-unplug */
+#define NODE_PRIVATE_CAP_DEMOTION (1UL << 3) /* allow tiering demotion */
/**
* struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -118,6 +119,30 @@ static inline bool node_allows_hotunplug(int nid)
return ret;
}
+/**
+ * node_allows_demotion - may kernel tiering demote to this node?
+ * @nid: the node to test
+ *
+ * Governs whether a private node participates in the demotion hierarchy.
+ * Demotion accumulates pages on the node, so CAP_DEMOTION requires CAP_RECLAIM
+ * (enforced at registration) as a safety valve.
+ *
+ * return: true for normal nodes and private nodes opted into CAP_DEMOTION.
+ */
+static inline bool node_allows_demotion(int nid)
+{
+ struct node_private *np;
+ bool ret;
+
+ if (!node_state(nid, N_MEMORY_PRIVATE))
+ return true;
+ rcu_read_lock();
+ np = rcu_dereference(NODE_DATA(nid)->node_private);
+ ret = np && (np->caps & NODE_PRIVATE_CAP_DEMOTION);
+ rcu_read_unlock();
+ return ret;
+}
+
#else /* !CONFIG_NUMA */
static inline bool folio_is_private_node(struct folio *folio)
@@ -150,6 +175,11 @@ static inline bool node_allows_hotunplug(int nid)
return true;
}
+static inline bool node_allows_demotion(int nid)
+{
+ return true;
+}
+
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 25e121851b586..c673080d153e4 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -7,6 +7,7 @@
#include <linux/memory-tiers.h>
#include <linux/notifier.h>
#include <linux/sched/sysctl.h>
+#include <linux/node_private.h>
#include "internal.h"
@@ -317,6 +318,21 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
rcu_read_unlock();
}
+/* Tiering set: N_MEMORY | (N_MEMORY_PRIVATE w/ CAP_DEMOTION) */
+static nodemask_t tierable_nodes;
+
+static void update_tierable_nodes(void)
+{
+ int node;
+
+ lockdep_assert_held_once(&memory_tier_lock);
+
+ tierable_nodes = node_states[N_MEMORY];
+ for_each_node_state(node, N_MEMORY_PRIVATE)
+ if (node_allows_demotion(node))
+ node_set(node, tierable_nodes);
+}
+
/**
* next_demotion_node() - Get the next node in the demotion path
* @node: The starting node to lookup the next node
@@ -330,7 +346,7 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
int next_demotion_node(int node, const nodemask_t *allowed_mask)
{
struct demotion_nodes *nd;
- nodemask_t mask;
+ nodemask_t mask, tierable;
if (!node_demotion)
return NUMA_NO_NODE;
@@ -370,7 +386,8 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
* closest demotion target.
*/
nodes_complement(mask, *allowed_mask);
- return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
+ tierable = tierable_nodes;
+ return find_next_best_node_in(node, &mask, &tierable);
}
static void disable_all_demotion_targets(void)
@@ -378,7 +395,7 @@ static void disable_all_demotion_targets(void)
struct memory_tier *memtier;
int node;
- for_each_node_state(node, N_MEMORY) {
+ for_each_node_mask(node, tierable_nodes) {
node_demotion[node].preferred = NODE_MASK_NONE;
/*
* We are holding memory_tier_lock, it is safe
@@ -401,7 +418,7 @@ static void dump_demotion_targets(void)
{
int node;
- for_each_node_state(node, N_MEMORY) {
+ for_each_node_mask(node, tierable_nodes) {
struct memory_tier *memtier = __node_get_memory_tier(node);
nodemask_t preferred = node_demotion[node].preferred;
@@ -435,9 +452,10 @@ static void establish_demotion_targets(void)
if (!node_demotion)
return;
+ update_tierable_nodes();
disable_all_demotion_targets();
- for_each_node_state(node, N_MEMORY) {
+ for_each_node_mask(node, tierable_nodes) {
best_distance = -1;
nd = &node_demotion[node];
@@ -455,7 +473,7 @@ static void establish_demotion_targets(void)
* nodelist to skip list so that we find the best node from the
* memtier nodelist.
*/
- nodes_andnot(tier_nodes, node_states[N_MEMORY], tier_nodes);
+ nodes_andnot(tier_nodes, tierable_nodes, tier_nodes);
/*
* Find all the nodes in the memory tier node list of same best distance.
@@ -464,7 +482,7 @@ static void establish_demotion_targets(void)
*/
do {
target = find_next_best_node_in(node, &tier_nodes,
- &node_states[N_MEMORY]);
+ &tierable_nodes);
if (target == NUMA_NO_NODE)
break;
@@ -503,7 +521,7 @@ static void establish_demotion_targets(void)
* allocation to a set of nodes that is closer the above selected
* preferred node.
*/
- lower_tier = node_states[N_MEMORY];
+ lower_tier = tierable_nodes;
list_for_each_entry(memtier, &memory_tiers, list) {
/*
* Keep removing current tier from lower_tier nodes,
@@ -550,7 +568,8 @@ static struct memory_tier *set_node_memory_tier(int node)
lockdep_assert_held_once(&memory_tier_lock);
- if (!node_state(node, N_MEMORY))
+ /* Include N_MEMORY and N_MEMORY_PRIVATE with CAP_DEMOTION */
+ if (!node_state(node, N_MEMORY) && !node_allows_demotion(node))
return ERR_PTR(-EINVAL);
mt_calc_adistance(node, &adist);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index f1722693ac2db..b617f7cd1e716 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -963,6 +963,10 @@ static struct folio *alloc_demote_folio(struct folio *src,
mtc = (struct migration_target_control *)private;
+ if (mtc->nmask &&
+ nodes_intersects(*mtc->nmask, node_states[N_MEMORY_PRIVATE]))
+ mtc->alloc_flags = ALLOC_ZONELIST_PRIVATE;
+
/*
* make sure we allocate from the target node first also trying to
* demote or reclaim pages from the target node via kswapd if we are
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
2026-07-20 19:34 ` [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion Gregory Price
@ 2026-07-21 20:28 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 20:28 UTC (permalink / raw)
To: Gregory Price; +Cc: damon, mkoutny, kvm, tj, nvdimm, hannes, linux-cxl
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent, unprotected access to the global `tierable_nodes` nodemask can result in a torn read.
--
--- Patch [29]: [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion ---
commit e4de60011eb318a533982e5d4b896e77cc0ad141
Author: Gregory Price <gourry@gourry.net>
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
This commit introduces NODE_PRIVATE_CAP_DEMOTION, allowing private memory
nodes to be included in the tiering and demotion hierarchy. By opting into
this capability, memory-tiers will include the private node in demotion
paths, allowing page reclaim to target it.
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 25e121851b586..c673080d153e4 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -317,6 +318,21 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
> rcu_read_unlock();
> }
>
> +/* Tiering set: N_MEMORY | (N_MEMORY_PRIVATE w/ CAP_DEMOTION) */
> +static nodemask_t tierable_nodes;
> +
> +static void update_tierable_nodes(void)
> +{
> + int node;
> +
> + lockdep_assert_held_once(&memory_tier_lock);
> +
> + tierable_nodes = node_states[N_MEMORY];
> + for_each_node_state(node, N_MEMORY_PRIVATE)
> + if (node_allows_demotion(node))
> + node_set(node, tierable_nodes);
> +}
> +
[ ... ]
> @@ -330,7 +346,7 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
> int next_demotion_node(int node, const nodemask_t *allowed_mask)
> {
> struct demotion_nodes *nd;
> - nodemask_t mask;
> + nodemask_t mask, tierable;
>
> if (!node_demotion)
> return NUMA_NO_NODE;
[ ... ]
> @@ -370,7 +386,8 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
> * closest demotion target.
> */
> nodes_complement(mask, *allowed_mask);
> - return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
> + tierable = tierable_nodes;
[Severity: Medium]
Could this lockless copy result in a torn read of the tierable_nodes nodemask?
In update_tierable_nodes(), the global tierable_nodes mask is updated via a
non-atomic struct assignment and iterative node_set() modifications. Here in
next_demotion_node(), the mask is copied locklessly to the stack during the
reclaim hotpath.
Since nodemask_t can be up to 128 bytes depending on MAX_NUMNODES, concurrent
updates from memory hotplug could cause next_demotion_node() to see a
partially updated mask. This could cause page reclaim to transiently skip valid
private nodes or behave suboptimally during demotion.
> + return find_next_best_node_in(node, &mask, &tierable);
> }
>
> static void disable_all_demotion_targets(void)
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=29
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 30/36] mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (28 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 31/36] mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning Gregory Price
` (8 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
By default NUMA balancing does not scan or prot_none private-node
folios, so the kernel never migrates them via access sampling.
Add NODE_PRIVATE_CAP_NUMA_BALANCING to opt a private node in.
Add folio_allows_numa_balance() to gate change_prot_numa() scans on
node eligibility. Opted-in private node participate like normal.
Unlike demotion, NUMA balancing is not reclaim-driven, so this
capability stands alone and does not require CAP_RECLAIM.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/node_private.h | 29 +++++++++++++++++++++++++++++
mm/internal.h | 13 +++++++++++++
mm/mempolicy.c | 2 +-
3 files changed, 43 insertions(+), 1 deletion(-)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
index 87b03444b2c97..5c3e070ed0deb 100644
--- a/include/linux/node_private.h
+++ b/include/linux/node_private.h
@@ -16,6 +16,7 @@ struct page;
#define NODE_PRIVATE_CAP_USER_NUMA (1UL << 1) /* allow mempolicy */
#define NODE_PRIVATE_CAP_HOTUNPLUG (1UL << 2) /* allow hot-unplug */
#define NODE_PRIVATE_CAP_DEMOTION (1UL << 3) /* allow tiering demotion */
+#define NODE_PRIVATE_CAP_NUMA_BALANCING (1UL << 4) /* allow NUMA balancing */
/**
* struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -143,6 +144,29 @@ static inline bool node_allows_demotion(int nid)
return ret;
}
+/**
+ * node_allows_numa_balancing - may NUMA balancing scan/migrate this node?
+ * @nid: the node to test
+ *
+ * Access-based promotion/migration. Unlike demotion this is not reclaim-driven,
+ * so CAP_NUMA_BALANCING stands alone (no CAP_RECLAIM dependency).
+ *
+ * return: true for normal nodes and private nodes opted into CAP_NUMA_BALANCING.
+ */
+static inline bool node_allows_numa_balancing(int nid)
+{
+ struct node_private *np;
+ bool ret;
+
+ if (!node_state(nid, N_MEMORY_PRIVATE))
+ return true;
+ rcu_read_lock();
+ np = rcu_dereference(NODE_DATA(nid)->node_private);
+ ret = np && (np->caps & NODE_PRIVATE_CAP_NUMA_BALANCING);
+ rcu_read_unlock();
+ return ret;
+}
+
#else /* !CONFIG_NUMA */
static inline bool folio_is_private_node(struct folio *folio)
@@ -180,6 +204,11 @@ static inline bool node_allows_demotion(int nid)
return true;
}
+static inline bool node_allows_numa_balancing(int nid)
+{
+ return true;
+}
+
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
diff --git a/mm/internal.h b/mm/internal.h
index 62e68acae08a3..01ab8b32b0bd8 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -123,6 +123,19 @@ static inline bool folio_allows_madvise(struct folio *folio)
node_allows_reclaim(folio_nid(folio));
}
+/*
+ * folio_allows_numa_balance() - may NUMA balancing scan/migrate this folio?
+ *
+ * NUMA balancing is access-aware tiering migration, so it follows the tiering
+ * opt-in: false for ZONE_DEVICE and for N_MEMORY_PRIVATE nodes without
+ * CAP_NUMA_BALANCING, true for all other folios.
+ */
+static inline bool folio_allows_numa_balance(struct folio *folio)
+{
+ return !folio_is_zone_device(folio) &&
+ node_allows_numa_balancing(folio_nid(folio));
+}
+
/*
* folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
*
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index fe42a510590a2..4daba81fff7c7 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -874,7 +874,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
{
int nid;
- if (!folio || folio_is_private_managed(folio) || folio_test_ksm(folio))
+ if (!folio || !folio_allows_numa_balance(folio) || folio_test_ksm(folio))
return false;
/* Also skip shared copy-on-write folios */
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 31/36] mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (29 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 30/36] mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits Gregory Price
` (7 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Add NODE_PRIVATE_CAP_LTPIN so a node can opt back into gup pins.
An opted-in node is then pinned exactly like ordinary memory, or
migrated and pinned if in ZONE_MOVABLE.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/node_private.h | 29 +++++++++++++++++++++++++++++
mm/gup.c | 3 ++-
mm/internal.h | 9 +++++----
3 files changed, 36 insertions(+), 5 deletions(-)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
index 5c3e070ed0deb..6f568bc0d46c7 100644
--- a/include/linux/node_private.h
+++ b/include/linux/node_private.h
@@ -17,6 +17,7 @@ struct page;
#define NODE_PRIVATE_CAP_HOTUNPLUG (1UL << 2) /* allow hot-unplug */
#define NODE_PRIVATE_CAP_DEMOTION (1UL << 3) /* allow tiering demotion */
#define NODE_PRIVATE_CAP_NUMA_BALANCING (1UL << 4) /* allow NUMA balancing */
+#define NODE_PRIVATE_CAP_LTPIN (1UL << 5) /* allow GUP pins */
/**
* struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -167,6 +168,29 @@ static inline bool node_allows_numa_balancing(int nid)
return ret;
}
+/**
+ * node_allows_ltpin - may a folio on this node be long-term GUP-pinned?
+ * @nid: the node to test
+ *
+ * Opted-out private nodes cause longterm pins to outright fail regardless
+ * of ZONE placement (NORMAL would allow, MOVABLE would migrate first).
+ *
+ * Opted-in private nodes allow longterm pins to operate normally.
+ */
+static inline bool node_allows_ltpin(int nid)
+{
+ struct node_private *np;
+ bool ret;
+
+ if (!node_state(nid, N_MEMORY_PRIVATE))
+ return true;
+ rcu_read_lock();
+ np = rcu_dereference(NODE_DATA(nid)->node_private);
+ ret = np && (np->caps & NODE_PRIVATE_CAP_LTPIN);
+ rcu_read_unlock();
+ return ret;
+}
+
#else /* !CONFIG_NUMA */
static inline bool folio_is_private_node(struct folio *folio)
@@ -209,6 +233,11 @@ static inline bool node_allows_numa_balancing(int nid)
return true;
}
+static inline bool node_allows_ltpin(int nid)
+{
+ return true;
+}
+
#endif /* CONFIG_NUMA */
#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
diff --git a/mm/gup.c b/mm/gup.c
index a7d4de223785c..1e313aa590bc1 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -547,7 +547,8 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,
/*
* Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
* right zone, so fail and let the caller fall back to the slow
- * path. Fail for private-node folios here so slow path rejects.
+ * path. Fail for non-opted-in private node folios here so the
+ * slow path can reject the pin entirely.
*/
if (unlikely((flags & FOLL_LONGTERM) &&
!folio_allows_longterm_pin(folio))) {
diff --git a/mm/internal.h b/mm/internal.h
index 01ab8b32b0bd8..9621eb127c28c 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -146,19 +146,20 @@ static inline bool folio_allows_numa_balance(struct folio *folio)
static inline bool folio_allows_longterm_pin(struct folio *folio)
{
return folio_is_longterm_pinnable(folio) &&
- !folio_is_private_node(folio);
+ node_allows_ltpin(folio_nid(folio));
}
/*
* folio_longterm_pin_forbidden() - must a longterm pin of this folio fail
* outright (neither pinned in place nor migrated off the node)?
*
- * True for any folio on a private node: such memory can be neither pinned
- * nor migrated, so the pin must be rejected with the folio left in place.
+ * True only for a folio on a private node that did not opt into longterm
+ * pinning (NODE_PRIVATE_CAP_LTPIN); node_allows_ltpin() is true for ordinary
+ * nodes and for opted-in private nodes, so this never trips them.
*/
static inline bool folio_longterm_pin_forbidden(struct folio *folio)
{
- return folio_is_private_node(folio);
+ return !node_allows_ltpin(folio_nid(folio));
}
/*
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (30 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 31/36] mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 20:49 ` sashiko-bot
2026-07-22 13:24 ` Richard Cheng
2026-07-20 19:34 ` [PATCH v5 33/36] Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes Gregory Price
` (6 subsequent siblings)
38 siblings, 2 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Collapse can move base pages across nodes. The khugepaged daemon does
this transparently, while MADV_COLLAPSE is a userland construct.
Isolate private nodes from khugepaged exactly like ZONE_DEVICE.
Never collapse private node folios silently (the owner may not
support migration).
MADV_COLLAPSE is user-initiated, so it is allowed on a private node
opted into CAP_USER_NUMA, consistent with other userspace interfaces.
folio_allows_collapse() encodes this: false for khugepaged on a private
node, CAP_USER_NUMA for MADV_COLLAPSE.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/internal.h | 26 ++++++++++++++++++++++++++
mm/khugepaged.c | 16 +++++++++++++---
2 files changed, 39 insertions(+), 3 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index 9621eb127c28c..9dbc5752cd3fe 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -136,6 +136,32 @@ static inline bool folio_allows_numa_balance(struct folio *folio)
node_allows_numa_balancing(folio_nid(folio));
}
+/*
+ * folio_allows_collapse() - may collapse fold this folio into a THP?
+ * @is_khugepaged: true for the khugepaged, false for MADV_COLLAPSE.
+ *
+ * Collapse is a residency operation gated by the actor calling it:
+ * - khugepaged never operates on private-node folios (like ZONE_DEVICE)
+ * - MADV_COLLAPSE is gated by CAP_USER_NUMA
+ *
+ * Never true for ZONE_DEVICE.
+ */
+static inline bool folio_allows_collapse(struct folio *folio, bool is_khugepaged)
+{
+ int nid = folio_nid(folio);
+
+ if (folio_is_zone_device(folio))
+ return false;
+ if (is_khugepaged)
+ return !node_is_private(nid);
+ return node_allows_user_numa(nid);
+}
+
+static inline bool page_allows_collapse(struct page *page, bool is_khugepaged)
+{
+ return folio_allows_collapse(page_folio(page), is_khugepaged);
+}
+
/*
* folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
*
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index fb4378cc17b10..5f20839857738 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -700,7 +700,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
goto out;
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+ if (unlikely(!page) ||
+ unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
result = SCAN_PAGE_NULL;
goto out;
}
@@ -1241,9 +1242,17 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
GFP_TRANSHUGE);
int node = collapse_find_target_node(cc);
+ unsigned int aflags;
struct folio *folio;
+ bool allow;
- folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
+ /* Private node access: khugepaged never, madvise with CAP_USER_NUMA */
+ allow = cc->is_khugepaged ? !node_is_private(node)
+ : node_allows_user_numa(node);
+ aflags = (allow && node_is_private(node)) ?
+ ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT;
+
+ folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, aflags);
if (!folio) {
*foliop = NULL;
if (is_pmd_order(order))
@@ -1687,7 +1696,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+ if (unlikely(!page) ||
+ unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
result = SCAN_PAGE_NULL;
goto out_unmap;
}
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits
2026-07-20 19:34 ` [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits Gregory Price
@ 2026-07-21 20:49 ` sashiko-bot
2026-07-22 13:24 ` Richard Cheng
1 sibling, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 20:49 UTC (permalink / raw)
To: Gregory Price; +Cc: kvm, linux-cxl, hannes, nvdimm, tj, mkoutny, damon
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing `folio_allows_collapse()` checks in `collapse_scan_file` allows khugepaged to silently migrate file-backed/shmem folios off private nodes.
--
--- Patch [32]: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits ---
commit e336e94fb86a5e84bcd43a7f17af3bf83d2b9126
Author: Gregory Price <gourry@gourry.net>
mm/khugepaged: base private node collapse eligiblity on actor/cap bits
Collapse can move base pages across nodes. The khugepaged daemon does
this transparently, while MADV_COLLAPSE is a userland construct.
Isolate private nodes from khugepaged exactly like ZONE_DEVICE.
Never collapse private node folios silently (the owner may not
support migration).
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index fb4378cc17b10..5f20839857738 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
[ ... ]
> @@ -1687,7 +1696,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> }
>
> page = vm_normal_page(vma, addr, pteval);
> - if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
> + if (unlikely(!page) ||
> + unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
> result = SCAN_PAGE_NULL;
> goto out_unmap;
> }
[Severity: High]
Since this patch updates collapse_scan_pmd() to check
page_allows_collapse(), does collapse_scan_file() need a similar check?
Looking at collapse_scan_file(), the loop scans folios and tallies their
residency without verifying collapse eligibility:
mm/khugepaged.c:collapse_scan_file() {
...
node = folio_nid(folio);
if (collapse_scan_abort(node, cc)) {
result = SCAN_SCAN_ABORT;
folio_put(folio);
break;
}
cc->node_load[node]++;
...
}
If an unprivileged process maps file-backed memory to a private node, this
allows the scan to proceed. alloc_charge_folio() will then deny allocation
on the private node (since is_khugepaged is true) and fall back to a normal
node via ALLOC_DEFAULT.
Once the fallback THP is allocated, collapse_file() copies the memory from
the private node to the newly allocated THP on the normal node. Could this
result in a silent migration that bypasses the intended isolation guarantees?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=32
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits
2026-07-20 19:34 ` [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits Gregory Price
2026-07-21 20:49 ` sashiko-bot
@ 2026-07-22 13:24 ` Richard Cheng
2026-07-22 13:43 ` Gregory Price
1 sibling, 1 reply; 82+ messages in thread
From: Richard Cheng @ 2026-07-22 13:24 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, Jul 20, 2026 at 03:34:26PM +0800, Gregory Price wrote:
> Collapse can move base pages across nodes. The khugepaged daemon does
> this transparently, while MADV_COLLAPSE is a userland construct.
>
> Isolate private nodes from khugepaged exactly like ZONE_DEVICE.
> Never collapse private node folios silently (the owner may not
> support migration).
>
> MADV_COLLAPSE is user-initiated, so it is allowed on a private node
> opted into CAP_USER_NUMA, consistent with other userspace interfaces.
>
> folio_allows_collapse() encodes this: false for khugepaged on a private
> node, CAP_USER_NUMA for MADV_COLLAPSE.
>
> Signed-off-by: Gregory Price <gourry@gourry.net>
> ---
> mm/internal.h | 26 ++++++++++++++++++++++++++
> mm/khugepaged.c | 16 +++++++++++++---
> 2 files changed, 39 insertions(+), 3 deletions(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index 9621eb127c28c..9dbc5752cd3fe 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -136,6 +136,32 @@ static inline bool folio_allows_numa_balance(struct folio *folio)
> node_allows_numa_balancing(folio_nid(folio));
> }
>
> +/*
> + * folio_allows_collapse() - may collapse fold this folio into a THP?
> + * @is_khugepaged: true for the khugepaged, false for MADV_COLLAPSE.
> + *
> + * Collapse is a residency operation gated by the actor calling it:
> + * - khugepaged never operates on private-node folios (like ZONE_DEVICE)
> + * - MADV_COLLAPSE is gated by CAP_USER_NUMA
> + *
> + * Never true for ZONE_DEVICE.
> + */
> +static inline bool folio_allows_collapse(struct folio *folio, bool is_khugepaged)
> +{
> + int nid = folio_nid(folio);
> +
> + if (folio_is_zone_device(folio))
> + return false;
> + if (is_khugepaged)
> + return !node_is_private(nid);
> + return node_allows_user_numa(nid);
> +}
> +
> +static inline bool page_allows_collapse(struct page *page, bool is_khugepaged)
> +{
> + return folio_allows_collapse(page_folio(page), is_khugepaged);
> +}
> +
> /*
> * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
> *
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index fb4378cc17b10..5f20839857738 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -700,7 +700,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
> goto out;
> }
> page = vm_normal_page(vma, addr, pteval);
> - if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
> + if (unlikely(!page) ||
> + unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
> result = SCAN_PAGE_NULL;
> goto out;
> }
> @@ -1241,9 +1242,17 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
> gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
> GFP_TRANSHUGE);
> int node = collapse_find_target_node(cc);
> + unsigned int aflags;
> struct folio *folio;
> + bool allow;
>
> - folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
> + /* Private node access: khugepaged never, madvise with CAP_USER_NUMA */
> + allow = cc->is_khugepaged ? !node_is_private(node)
> + : node_allows_user_numa(node);
> + aflags = (allow && node_is_private(node)) ?
> + ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT;
> +
> + folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, aflags);
> if (!folio) {
> *foliop = NULL;
> if (is_pmd_order(order))
> @@ -1687,7 +1696,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
> }
>
> page = vm_normal_page(vma, addr, pteval);
> - if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
> + if (unlikely(!page) ||
> + unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
I think collapse_scan_file() need something like folio_allows_collapse() check as well.
--Richard
> result = SCAN_PAGE_NULL;
> goto out_unmap;
> }
> --
> 2.53.0-Meta
>
>
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits
2026-07-22 13:24 ` Richard Cheng
@ 2026-07-22 13:43 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-22 13:43 UTC (permalink / raw)
To: Richard Cheng
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 09:24:37PM +0800, Richard Cheng wrote:
> On Mon, Jul 20, 2026 at 03:34:26PM +0800, Gregory Price wrote:
> >
> > page = vm_normal_page(vma, addr, pteval);
> > - if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
> > + if (unlikely(!page) ||
> > + unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
>
> I think collapse_scan_file() need something like folio_allows_collapse() check as well.
>
I believe you are correct, sashiko up some stuff here too.
Thanks!
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* [PATCH v5 33/36] Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (31 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 34/36] mm/mempolicy: add mpol_set_shared_policy_range() Gregory Price
` (5 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
Add a design overview of private memory nodes:
- the isolation model (structural zonelist exclusion)
- ZONELIST_PRIVATE
- driver provisioning API
- capability opt-in model and its dependency rules
- observability surfaces
Signed-off-by: Gregory Price <gourry@gourry.net>
---
Documentation/mm/index.rst | 1 +
Documentation/mm/numa_private_nodes.rst | 160 ++++++++++++++++++++++++
2 files changed, 161 insertions(+)
create mode 100644 Documentation/mm/numa_private_nodes.rst
diff --git a/Documentation/mm/index.rst b/Documentation/mm/index.rst
index 13a79f5d092c0..f60704df6104c 100644
--- a/Documentation/mm/index.rst
+++ b/Documentation/mm/index.rst
@@ -65,6 +65,7 @@ documentation, or deleted if it has served its purpose.
mmu_notifier
multigen_lru
numa
+ numa_private_nodes
overcommit-accounting
page_migration
page_frags
diff --git a/Documentation/mm/numa_private_nodes.rst b/Documentation/mm/numa_private_nodes.rst
new file mode 100644
index 0000000000000..3b27a2e24b086
--- /dev/null
+++ b/Documentation/mm/numa_private_nodes.rst
@@ -0,0 +1,160 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+====================
+Private memory nodes
+====================
+
+A *private memory node* is a NUMA node whose memory is hotplugged by a driver
+and deliberately hidden from the kernel's normal memory management. Such a
+node is marked ``N_MEMORY_PRIVATE`` instead of ``N_MEMORY``; the two states
+are mutually exclusive, so a private node is never considered by the page
+allocator's normal or fallback paths.
+
+The intent is to give a driver a block of NUMA-addressable memory that the rest
+of the kernel will not allocate from on its own, while still letting that memory
+be mapped into processes as ordinary, struct-page, LRU-managed folios -- and to
+let the driver re-enable individual mm services it is capable of allowing.
+
+Preconditions
+=============
+
+``N_MEMORY_PRIVATE`` and ``N_MEMORY`` are mutually exclusive, so the backing
+memory must come up on a node that has no DRAM of its own (otherwise the node
+would already be ``N_MEMORY``).
+
+In practice the memory is provided by a device driver or a DAX device whose
+target node has no other memory, and usually no CPUs.
+
+Isolation model
+===============
+
+Isolation is *opt-in by exclusion* and is **structural**: by default nothing in
+the kernel can place memory on a private node because the node is absent from the
+zonelists an ordinary allocation walks.
+
+Zonelist exclusion
+ The kernel page allocator depends on the ``FALLBACK`` and ``NOFALLBACK``
+ zonelists to allocate memory. A normal ``N_MEMORY`` node's zones (except
+ ``ZONE_DEVICE``) appear in these lists and allow allocations to fall-back
+ to less preferable locations if the preferred location is pressured.
+
+ ``__GFP_THISNODE`` is used during normal operation to switch between
+ ``FALLBACK`` and ``NOFALLBACK``, where ``NOFALLBACK`` only contains the
+ zonelists of the preferred node.
+
+ ``N_MEMORY_PRIVATE`` nodes are **excluded** from both ``FALLBACK`` and
+ ``NOFALLBACK`` zonelists. Instead they are added to ``ZONELIST_PRIVATE``,
+ which includes both ``N_MEMORY`` and ``N_MEMORY_PRIVATE`` nodes. This is
+ the only zonelist that contains private-node zones, and so the only way
+ to acquire private node allocations is to explicitly request that zonelist.
+
+ Even an allocation carrying ``__GFP_THISNODE`` cannot access the node's
+ memory without also explicitly passing the private zonelist. This prevents
+ incidental allocation of private memory by users of possible/online
+ nodelists.
+
+ When ``CONFIG_NUMA`` is disabled ``ZONELIST_PRIVATE`` aliases
+ ``ZONELIST_FALLBACK`` and is never selected.
+
+The user_numa path
+
+ ``MPOL_F_PRIVATE`` is an internal user_numa flag (never accepted from
+ userspace) marking that a mempolicy has a private node in its nodemask.
+
+ When ``CAP_USER_NUMA`` for a private node is set, user-sourced mempolicy
+ (``set_mempolicy(2)``) and migration (``move_pages(2)``) operations are
+ allowed to include that node in nodemasks and targets respectively.
+
+ ``mbind(MPOL_MF_MOVE)`` is both a mempolicy and a migration operation,
+ so placement and migration share the same capability.
+
+ The mempolicy component uses ``MPOL_F_PRIVATE`` at fault-time to select
+ ``ZONELIST_PRIVATE`` and makes the node's memory available for allocation.
+ It is otherwise an ordinary, relaxable mempolicy: an unsatisfiable request
+ (an unmovable allocation on a movable-only private node) simply falls back.
+
+
+cpuset interaction
+==================
+
+cpuset.mems does **not** partition private nodes. cpuset neither grants nor
+denies access, and rebinding cpuset.mems nodemasks do not affect a private node's
+residency in any nodemask.
+
+Likewise, a private node's inclusion in a nodemask does not affect cpuset.mems'
+filtering of any ``N_MEMORY`` - they remain partitioned according to cpuset.
+
+
+Provisioning
+============
+
+A driver brings memory up as private with::
+
+ add_private_memory_driver_managed(nid, start, size, resource_name,
+ mhp_flags, online_type, np)
+
+which onlines the range and registers the driver-owned ``struct node_private``
+(``np``) describing the node, including its capability bitmap (see below).
+
+Only one driver/service may register a ``struct node_private``, which
+heavily implies a "one-node-per-device" design of the system.
+
+The node leaves ``N_MEMORY_PRIVATE`` only when the last range is offlined.
+
+.. kernel-doc:: mm/memory_hotplug.c
+ :identifiers: __add_memory_driver_managed
+
+.. kernel-doc:: drivers/base/node.c
+ :identifiers: node_private_register node_private_unregister
+
+Capabilities (per-service opt-ins)
+==================================
+
+Because the default is "no mm service touches the node", each service a driver
+wants back is requested explicitly through a capability bit in
+``np->caps``. The mm side checks the matching ``node_allows_*()`` /
+``folio_allows_*()`` predicate before acting:
+
+.. list-table::
+ :header-rows: 1
+ :widths: 35 65
+
+ * - Capability
+ - Re-enables
+ * - ``NODE_PRIVATE_CAP_RECLAIM``
+ - reclaim of the node's folios, by the mm and by userspace
+ ``MADV_COLD`` / ``PAGEOUT`` / ``FREE`` (userland-driven reclaim)
+ * - ``NODE_PRIVATE_CAP_USER_NUMA``
+ - all userspace-directed placement and migration: ``mbind()`` /
+ ``set_mempolicy()`` / home node, and ``move_pages()`` /
+ ``migrate_pages()`` to/from the node
+ * - ``NODE_PRIVATE_CAP_HOTUNPLUG``
+ - hot-unplug via migration
+ * - ``NODE_PRIVATE_CAP_DEMOTION``
+ - reclaim-driven tiering demotion onto the node (the node joins the
+ demotion hierarchy)
+ * - ``NODE_PRIVATE_CAP_NUMA_BALANCING``
+ - access-based NUMA balancing scan/migration of the node's folios
+ * - ``NODE_PRIVATE_CAP_LTPIN``
+ - ``FOLL_LONGTERM`` GUP pins
+
+khugepaged never operates on private-node folios (like ZONE_DEVICE), and DAMON
+does not act on them; ``MADV_COLLAPSE`` is covered by ``CAP_USER_NUMA``.
+
+Dependencies between capabilities are enforced **once**, by
+``node_private_register()`` at hotplug, rather than by whatever sets the bits:
+
+* ``DEMOTION`` requires ``RECLAIM`` (a demotion target accumulates demoted
+ pages, so without reclaim as a safety valve it would just fill up).
+
+Capability flags are expected to be stable at runtime.
+
+Observability
+=============
+
+A private node is reported through:
+
+* ``/sys/devices/system/node/has_private_memory``
+* ``/proc/<pid>/numa_maps`` -- per-node residency includes private nodes
+* ``/proc/kcore`` -- private-node RAM appears in the kcore RAM map
+* memcg per-node statistics account private-node memory.
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 34/36] mm/mempolicy: add mpol_set_shared_policy_range()
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (32 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 33/36] Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-20 19:34 ` [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation Gregory Price
` (4 subsequent siblings)
38 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
mpol_set_shared_policy() installs a policy over a VMA's page-offset
range, which is the only programmatic way to populate an inode's
shared policy after init.
Two limitations make it unusable for binding an entire backing inode
from in-kernel code:
- It requires a VMA, so it cannot cover unmapped offsets.
(e.g. unmapped file folios faulted by pagecache)
- mpol_shared_policy_init(), the only no-VMA installer, reconstructs
the policy from mpol->w.user_nodemask. That field is only populated
for static/relative or mount-string policies.
a policy built programmatically (e.g. by mpol_bind_node()) leaves it
empty and stores w.cpuset_mems_allowed instead, so _init mangles it.
Add mpol_set_shared_policy_range(), which installs an already-built,
fully contextualised policy verbatim over an arbitrary [start, end)
page range with no VMA.
Reimplement mpol_set_shared_policy() as a thin wrapper that derives
the range from the VMA, so both share a single underlying path.
Suggested-by: Dave Jiang <dave.jiang@intel.com>
Co-developed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-8
---
include/linux/mempolicy.h | 2 ++
mm/mempolicy.c | 35 +++++++++++++++++++++++++++++------
2 files changed, 31 insertions(+), 6 deletions(-)
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index 715951a5b03c1..1348e9f5f2cf9 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -125,6 +125,8 @@ int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst);
void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol);
int mpol_set_shared_policy(struct shared_policy *sp,
struct vm_area_struct *vma, struct mempolicy *mpol);
+int mpol_set_shared_policy_range(struct shared_policy *sp, pgoff_t start,
+ pgoff_t end, struct mempolicy *mpol);
void mpol_free_shared_policy(struct shared_policy *sp);
struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
pgoff_t idx);
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 4daba81fff7c7..23c4c25097450 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -3424,24 +3424,47 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
}
EXPORT_SYMBOL_FOR_MODULES(mpol_shared_policy_init, "kvm");
-int mpol_set_shared_policy(struct shared_policy *sp,
- struct vm_area_struct *vma, struct mempolicy *pol)
+/**
+ * mpol_set_shared_policy_range - install @pol over [@start, @end) of @sp
+ * @sp: the shared policy tree
+ * @start: first page offset (inclusive)
+ * @end: last page offset (exclusive)
+ * @pol: a fully-built, validated policy, or NULL to clear the range
+ *
+ * Installs @pol over the given range, replacing any overlapping policy.
+ * @sp takes its own reference, the caller retains its reference on @pol.
+ *
+ * The policy is not reconstructed, so the policy is preserved exactly.
+ *
+ * Unlike mpol_set_shared_policy(), no VMA is required, so a range that
+ * is never mapped into a VMA can be covered, including the whole file.
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+int mpol_set_shared_policy_range(struct shared_policy *sp, pgoff_t start,
+ pgoff_t end, struct mempolicy *pol)
{
- const pgoff_t pgoff = vma_start_pgoff(vma);
- const pgoff_t pgoff_end = vma_end_pgoff(vma);
struct sp_node *new = NULL;
int err;
if (pol) {
- new = sp_alloc(pgoff, pgoff_end, pol);
+ new = sp_alloc(start, end, pol);
if (!new)
return -ENOMEM;
}
- err = shared_policy_replace(sp, pgoff, pgoff_end, new);
+ err = shared_policy_replace(sp, start, end, new);
if (err && new)
sp_free(new);
return err;
}
+EXPORT_SYMBOL_FOR_MODULES(mpol_set_shared_policy_range, "kvm");
+
+int mpol_set_shared_policy(struct shared_policy *sp,
+ struct vm_area_struct *vma, struct mempolicy *pol)
+{
+ return mpol_set_shared_policy_range(sp, vma->vm_pgoff,
+ vma->vm_pgoff + vma_pages(vma), pol);
+}
EXPORT_SYMBOL_FOR_MODULES(mpol_set_shared_policy, "kvm");
/* Free a backing policy store on inode delete. */
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (33 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 34/36] mm/mempolicy: add mpol_set_shared_policy_range() Gregory Price
@ 2026-07-20 19:34 ` Gregory Price
2026-07-21 21:11 ` sashiko-bot
2026-07-21 3:46 ` [PATCH v5 00/36] Private Memory NUMA Nodes Balbir Singh
` (3 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
guest_memfd presently allocates its page-cache folios through a
per-inode shared mempolicy (kvm_gmem_get_folio()).
Today that policy can only be set after the fact, via mbind() on
a host mmap of the fd. This requires the fd to be mmap-able and
cannot reach folios that are only ever guest-faulted (no host VMA).
Neither holds for a non-mappable (confidential) guest_memfd.
Add GUEST_MEMFD_FLAG_BIND_NODE.
When set, KVM builds an MPOL_BIND policy for the requested node and
installs it over the whole inode's shared policy, so every folio
is allocated on the requested node with no userspace mbind().
The flag is advertised through KVM_CAP_GUEST_MEMFD_FLAGS only when
CONFIG_NUMA is enabled.
Suggested-by: Dave Jiang <dave.jiang@intel.com>
Co-developed-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Dave Jiang <dave.jiang@intel.com>
Signed-off-by: Gregory Price <gourry@gourry.net>
Assisted-by: Claude:claude-opus-4-8
---
include/linux/kvm_host.h | 3 +++
include/uapi/linux/kvm.h | 5 ++++-
virt/kvm/guest_memfd.c | 39 +++++++++++++++++++++++++++++++++++++--
3 files changed, 44 insertions(+), 3 deletions(-)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d31..f6448a73d464c 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -739,6 +739,9 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
+ if (IS_ENABLED(CONFIG_NUMA))
+ flags |= GUEST_MEMFD_FLAG_BIND_NODE;
+
return flags;
}
#endif
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 419011097fa8e..a8ff150e2cbb7 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1654,11 +1654,14 @@ struct kvm_memory_attributes {
#define KVM_CREATE_GUEST_MEMFD _IOWR(KVMIO, 0xd4, struct kvm_create_guest_memfd)
#define GUEST_MEMFD_FLAG_MMAP (1ULL << 0)
#define GUEST_MEMFD_FLAG_INIT_SHARED (1ULL << 1)
+#define GUEST_MEMFD_FLAG_BIND_NODE (1ULL << 2)
struct kvm_create_guest_memfd {
__u64 size;
__u64 flags;
- __u64 reserved[6];
+ __u32 node;
+ __u32 pad;
+ __u64 reserved[5];
};
#define KVM_PRE_FAULT_MEMORY _IOWR(KVMIO, 0xd5, struct kvm_pre_fault_memory)
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index f0e5da4908660..de320df08e0ff 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -456,6 +456,26 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
*/
return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff);
}
+
+static int kvm_gmem_bind_node(struct inode *inode, int node)
+{
+ struct mempolicy *pol;
+ int err;
+
+ pol = mpol_bind_node(node);
+ if (IS_ERR(pol))
+ return PTR_ERR(pol);
+
+ err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, 0,
+ MAX_LFS_FILESIZE >> PAGE_SHIFT, pol);
+ mpol_put(pol);
+ return err;
+}
+#else
+static int kvm_gmem_bind_node(struct inode *inode, int node)
+{
+ return -EINVAL;
+}
#endif /* CONFIG_NUMA */
static const struct vm_operations_struct kvm_gmem_vm_ops = {
@@ -557,7 +577,7 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
return true;
}
-static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int node)
{
static const char *name = "[kvm-gmem]";
struct gmem_file *f;
@@ -598,6 +618,12 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
GMEM_I(inode)->flags = flags;
+ if (flags & GUEST_MEMFD_FLAG_BIND_NODE) {
+ err = kvm_gmem_bind_node(inode, node);
+ if (err)
+ goto err_inode;
+ }
+
file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops);
if (IS_ERR(file)) {
err = PTR_ERR(file);
@@ -630,6 +656,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
{
loff_t size = args->size;
u64 flags = args->flags;
+ int node = NUMA_NO_NODE;
if (flags & ~kvm_gmem_get_supported_flags(kvm))
return -EINVAL;
@@ -637,7 +664,15 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
if (size <= 0 || !PAGE_ALIGNED(size))
return -EINVAL;
- return __kvm_gmem_create(kvm, size, flags);
+ if (flags & GUEST_MEMFD_FLAG_BIND_NODE) {
+ if (args->pad || args->node >= MAX_NUMNODES)
+ return -EINVAL;
+ node = args->node;
+ } else if (args->node || args->pad) {
+ return -EINVAL;
+ }
+
+ return __kvm_gmem_create(kvm, size, flags, node);
}
int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 82+ messages in thread* Re: [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation
2026-07-20 19:34 ` [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation Gregory Price
@ 2026-07-21 21:11 ` sashiko-bot
0 siblings, 0 replies; 82+ messages in thread
From: sashiko-bot @ 2026-07-21 21:11 UTC (permalink / raw)
To: Gregory Price; +Cc: mkoutny, linux-cxl, nvdimm, damon, tj, hannes, kvm
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Unconditional validation of flag-gated syscall parameters outside their flag scope breaks ABI compatibility for existing userspace.
- [Medium] Missing lock when calling `mpol_set_nodemask` leads to data race and torn read of `current->mems_allowed`.
--
--- Patch [35]: [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation ---
commit 45130876f6e87292f19cee2648796441ea4622ce
Author: Gregory Price <gourry@gourry.net>
KVM: guest_memfd: bind backing memory to a NUMA node at creation
guest_memfd presently allocates its page-cache folios through a per-inode
shared mempolicy. This patch adds GUEST_MEMFD_FLAG_BIND_NODE to set an
MPOL_BIND policy for the requested node and install it over the whole
inode's shared policy.
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index f0e5da4908660..de320df08e0ff 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
[ ... ]
> @@ -456,6 +456,26 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
> */
> return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff);
> }
> +
> +static int kvm_gmem_bind_node(struct inode *inode, int node)
> +{
> + struct mempolicy *pol;
> + int err;
> +
> + pol = mpol_bind_node(node);
[Severity: Medium]
Can calling mpol_bind_node() here without holding task_lock(current)
cause a data race?
Looking at mpol_bind_node(), it calls __mpol_bind_node() which then calls
mpol_set_nodemask(). This accesses current->mems_allowed locklessly.
If a task's cpuset is concurrently modified, this could lead to a torn read
of current->mems_allowed on systems with >64 NUMA nodes, potentially
failing or creating a policy with an incorrect or empty nodemask.
> + if (IS_ERR(pol))
> + return PTR_ERR(pol);
> +
> + err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, 0,
> + MAX_LFS_FILESIZE >> PAGE_SHIFT, pol);
> + mpol_put(pol);
> + return err;
> +}
[ ... ]
> @@ -630,6 +656,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
> {
> loff_t size = args->size;
> u64 flags = args->flags;
> + int node = NUMA_NO_NODE;
>
> if (flags & ~kvm_gmem_get_supported_flags(kvm))
> return -EINVAL;
> @@ -637,7 +664,15 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
> if (size <= 0 || !PAGE_ALIGNED(size))
> return -EINVAL;
>
> - return __kvm_gmem_create(kvm, size, flags);
> + if (flags & GUEST_MEMFD_FLAG_BIND_NODE) {
> + if (args->pad || args->node >= MAX_NUMNODES)
> + return -EINVAL;
> + node = args->node;
> + } else if (args->node || args->pad) {
[Severity: Critical]
Does unconditionally checking args->node and args->pad when the
GUEST_MEMFD_FLAG_BIND_NODE flag is absent break ABI compatibility?
Since these fields were previously part of the unused reserved[] array,
existing userspace binaries might allocate struct kvm_create_guest_memfd
on the stack without zeroing them.
If an older binary calls the ioctl without the new flag, and the stack
garbage happens to be non-zero, this will now fail with -EINVAL.
> + return -EINVAL;
> + }
> +
> + return __kvm_gmem_create(kvm, size, flags, node);
> }
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260720193431.3841992-1-gourry@gourry.net?part=35
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (34 preceding siblings ...)
2026-07-20 19:34 ` [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation Gregory Price
@ 2026-07-21 3:46 ` Balbir Singh
2026-07-21 18:16 ` Gregory Price
2026-07-21 13:26 ` Zenghui Yu
` (2 subsequent siblings)
38 siblings, 1 reply; 82+ messages in thread
From: Balbir Singh @ 2026-07-21 3:46 UTC (permalink / raw)
To: Gregory Price, linux-mm
Cc: Zhigang.Luo, arun.george, brendan.jackman, yuzenghui, apopple,
alucerop, matthew.brost, akpm, david, ljs, liam, vbabka, rppt,
surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On 7/21/26 5:33 AM, Gregory Price wrote:
> This series introduces the concept of "Private Memory Nodes", which
> are opted into two basic functionalities by default:
> - page allocation (mm/page_alloc.c)
> - OOM killing
>
> All other features of mm/ are opted out of managing these NUMA
> nodes and its memory. Then we add capability bits to allow node
> owners to opt back into those services (if supported).
>
> NUMA-hosted memory is presently a most-privilege system (any memory on
> any node, except ZONE_DEVICE, is 100% fungible and accessible). We
> then slap restrictions on top of it (cpuset, mempolicy, ZONE type,
> page/folio flags, etc).
>
> The goal here is to flip that dynamic, isolate by default and then
> opt-in to specific services that the device says is safe.
>
> Isolation at the NUMA/Zonelist layer provides a powerful mechanism for
> memory hosted on accelerators - re-use of the kernel mm/ code.
>
> - Accelerators (GPUs) can use demotion, numactl, and reclaim.
> - Special memory devices (Compressed RAM) with special access controls
> (promote-on-write) can have generic services written for them.
> - Network devices with large memory regions intended for ring buffers
> can use the buddy and standard networking stack.
> - Slow, disaggregated memory pools which aren't suitable as general
> purpose memory get cleaner interfaces (no need to re-write the buddy
> in userland, can use migration interface, etc).
> - Per-workload dedicated memory nodes (disaggregated VM memory)
>
> And more use cases I have collected over the past few years.
>
> Not included here is a dax-extension [1] that exposes all the internal
> bits as userland controls for testing - along with a pile of selftests
> that prove correctness.
>
> Changes Since V4
> ================
> - Massive reduction in complexity.
> - no ops struct
> - no callback functions
> - no __GFP_PRIVATE
> - no __GFP_THISNODE requirement
> - no task flags (no PF_MEMALLOC_* in the alloc path)
Very happy to see this
> - isolation via a dedicated zonelist:
> - private nodes are omitted from FALLBACK/NOFALLBACK
> - added ZONELIST_PRIVATE(_NOFALLBACK)
> - ALLOC_ZONELIST_PRIVATE alloc_flag
> - on top of Brendan Jackman's recent mm/page_alloc.h work [3]
> - Zonelist selection rides the allocator's alloc_flags
> - rename OPS -> CAPS (capabilities)
> - split base functionality (isolation) from opt-ins (CAPS)
> - first half of series can be merged without CAPS
> - dropped compressed ram example from series
> - will submit separately if this moves forward
> - Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
> - fully functional dax-kmem extension and huge suite of selftests
> located at my github, to be discussed separately [1]
>
> Patch Layout
> ============
> The series is broken into two sections:
>
> 1) N_MEMORY_PRIVATE Introduction.
> Introduce the node state.
> Opt those nodes out of mm/ services.
>
> 2) NODE_PRIVATE_CAP_* features
> A set of mm/ service opt-in flags that augment private
> nodes to make them more useful (i.e. reclaim = overcommit).
>
> NODE_PRIVATE_CAP_LTPIN for private node folio pinning
> NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
> NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
> NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
> NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
> NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
>
Looks reasonable, I wonder why USER_NUMA/HOTUNPUG is an opt-in?
> My hope is to merge at least #1 pulled ahead while #2 is debated.
>
> Allocation Isolation
> ====================
> page_alloc presently controls whether a node's memory can be allocated
> on a given call by 4 things (in order of authority)
>
> 1) ZONELIST membership
> If a node is not in the walked zonelist, it's unreachable.
>
> 2) __GFP_THISNODE
> If this flag is set and the only node in the ZONELIST is the
> singular preferred node (or the local node, for -1)
>
> 3) cpuset.mems membership
> cpuset trims any node in its allowed list
>
> 4) mempolicy nodemask
> the allocator will skip any node not in the nodemask.
>
> Except for #1 (Zonelist membership) there are all kinds of weird corner
> conditions in which 2-4 can be completely ignored (interrupt context,
> empty set because cpuset doesn't intersect mempolicy, shared vma, ...)
>
> But ZONELIST membership is *absolute*. If a zone is not in the
> zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.
>
> The existing zonelists are constructed like so:
> ZONELIST_FALLBACK : All N_MEMORY nodes
> ZONELIST_NOFALLBACK : A singleton N_MEMORY node
>
> Private node isolation is implemented via ZONELIST isolation:
> ZONELIST_PRIVATE : The private node + N_MEMORY
> ZONELIST_PRIVATE_NOFALLBACK : The private node alone.
>
> (mirrors exactly the fallback/nofallback for __GFP_THISNODE)
>
> Private nodes:
> 1) Never appear in any ZONELIST_FALLBACK
> 2) Have an empty ZONELIST_NOFALLBACK
> 3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)
>
> 1 & 2 mean all existing in-tree callers to page_alloc can NEVER
> accidentally allocate from a private node.
>
> An allocation must explicitly ask via a zonelist and a nodemask.
>
> alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
> __alloc_pages(..., nodemask); /* with the private node set */
>
> The page allocator keeps all its original interfaces which only
> ever touch the default zonelists - avoiding churn.
>
> Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
> =================================================================
> The vast majority of the kernel will never need to know about
> ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.
>
> When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces
> do the zonelist selection for the source of the allocation.
>
> That really is the whole explanation of the mechanism:
>
> alloc_flags = mpol_alloc_flags(pol);
> page = __alloc_frozen_pages_noprof(..., alloc_flags);
>
> On mm-new this rides the allocator's existing alloc_flags plumbing:
> ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new
> parameter, enum, or alloc_context change is required.
>
> For modules that want to implement their own special handling, they
> get the _private variants for the page allocator. This lets modules
> re-use the buddy instead of rewriting it.
>
> - alloc_pages_node_private_noprof()
> - folio_alloc_node_private_noprof()
>
> This keeps ALLOC_ flags mm/ internal (these functions add the flags).
>
> Isolating private node folios from kernel services
> ==================================================
> We implement filter points in mm/ to prevent operations on
> private node memory. Where possible, we even re-use existing
> filter points from ZONE_DEVICE.
>
> Most filter points are one or two lines of code:
>
> Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
> - if (folio_is_zone_device(folio))
> + if (unlikely(folio_is_private_managed(folio)))
>
> Disabling a service:
> + if (!node_is_private(nid)) {
> + kswapd_run(nid);
> + kcompactd_run(nid);
> + }
>
> Disallowing a uapi interaction:
> + if (node_state(nid, N_MEMORY_PRIVATE))
> + return -EINVAL;
>
> In the second half of the series, we replace blanket N_MEMORY_PRIVATE
> filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that
> interaction if CAP is set.
>
> We abstract this with a nice clean interface to make it really clear
> what is happening (nodes have features!)
>
> - if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
> + if (!node_allows_reclaim(pgdat->node_id))
>
> NODE_PRIVATE_CAP_* features
> ===========================
> This series of commits opts private nodes into various mm/ services.
>
> Capabilities:
> NODE_PRIVATE_CAP_RECLAIM - direct and kswapd reclaim
> NODE_PRIVATE_CAP_USER_NUMA - userland numa controls
> NODE_PRIVATE_CAP_DEMOTION - node is a demotion target
> NODE_PRIVATE_CAP_HOTUNPLUG - hotunplug may migrate
> NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
> NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally
>
> Some opt-in support is more intensive than others, so these features
> are broken out in a way that we can defer them as future work streams.
>
> NODE_PRIVATE_CAP_RECLAIM:
> Enabling reclaim for these nodes is actually surprisingly trivial.
>
> Without CAP_RECLAIM, when an allocation failure occurs, the system
> will not attempt to swap the memory - and instead will OOM (typically
> whatever task is using the most memory on *that* private node).
>
> This capability consists of:
> 1) enabling kswapd and kcompactd for that node at hotplug time.
> 2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
> 3) Sets normal watermarks for these nodes.
> 4) Allow madvise operations on that node (PAGEOUT).
> 5) A small tweak to how LRU decides which zones to visit.
>
> NODE_PRIVATE_CAP_USER_NUMA
> Enables the following userland interfaces to accept the node:
> mbind()
> set_mempolicy()
> set_mempolicy_home_node()
> move_pages()
> migrate_pages()
>
> example:
> buf = mmap(..., MAP_ANON);
> mbind(buf, ..., {private_node});
> buf[0] = 0xDEADBEEF; /* Page faults onto the private node */
>
> Later - the KVM example shows how in-kernel mempolicies can
> also be bound by CAP_USER_NUMA.
>
> Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.
>
> NODE_PRIVATE_CAP_HOTUNPLUG
> This is simple: allow hotunplug to migrate this nodes folios.
>
> Some devices may not be able to tolerate unexpected migrations,
> so we prevent hotunplug from engaging in migration by default.
>
> Some devices may have an mmu_notifier in their driver that can
> manage the migration and subsequent refault.
>
> CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.
>
> NODE_PRIVATE_CAP_DEMOTION
> This adds the private node as a valid demotion target, and allows
> reclaim to demote memory from a private node to a demotion target.
>
> Requires: NODE_PRIVATE_CAP_RECLAIM
>
> NODE_PRIVATE_CAP_NUMA_BALANCING
> This enables numa balancing to inject prot_none on private node
> folio mappings and promote them when faults are taken.
>
> NODE_PRIVATE_CAP_LTPIN
> This allows GUP Longterm Pinning to operate normally.
>
> Normally, longterm pinning determines folio eligibility based
> on its ZONE_* membership (among other things).
>
> ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require
> migration to ZONE_NORMAL before pinning.
>
> Neither operation is preferable by default on a private node,
> so the base behavior of FOLL_LONGTERM is to FAIL.
>
> This capability allows longterm pinning to operate normally
> based on the ZONE membership. Private node memory may be
> hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.
>
> In-tree User: KVM
> =================
> Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of
> enabling disaggregated memory pools to host dedicated KVM memory.
>
> With private nodes, this is trivial (with a bit of basic plumbing):
>
> static int kvm_gmem_bind_node(struct inode *inode, int node)
> {
> ...
> /* Bind to a private node - gated on CAP_USER_NUMA */
> pol = mpol_bind_node(node);
> if (IS_ERR(pol))
> return PTR_ERR(pol);
>
> /* Set the shared policy */
> err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
> ...
> }
>
> KVM doesn't even need to know about private nodes at all, all it
> does is ask mempolicy whether the requested node is a valid bind.
>
> mm/ component testing with dax driver
> =====================================
> The dax driver extensions[1] implements a simple interface to create
> a private node from a dax device created by any source.
>
> I left the dax driver extensions out of this feature set because
> it locks in the CAP_ bits before anyone has input. It's there
> primarily for testing at this point.
>
> The simplest way to get a dax device is with the memmap= boot arg.
> e.g.: "memmap=0x40000000!0x140000000"
>
> The dax driver extension has the following sysfs entries:
> dax0.0/private - set the node to private
> dax0.0/dax_file - make /dev/dax0.0 mmap'able in kmem mode
> dax0.0/adistance - dictate memory_tierN membership
> dax0.0/reclaim - CAP_RECLAIM
> dax0.0/demotion - CAP_DEMOTION
> dax0.0/user_numa - CAP_USER_NUMA
> dax0.0/hotunplug - CAP_HOTUNPLUG
> dax0.0/numa_balancing - CAP_NUMA_BALANCING
> dax0.0/ltpin - CAP_LTPIN
>
> Now consider the following...
>
> Single node reclaim + mbind support:
> echo 1 > dax0.0/private
> echo 1 > dax0.0/reclaim
> echo 1 > dax0.0/user_numa
> echo online_movable > dax0.0/state
>
> Test program:
> /* node1: 1GB Private Memory Node, 4GB swap */
> buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
> memset(buf, 0xff, TWO_GB);
>
> We can *guarantee* the ONLY reclaiming tasks are exactly:
> - kswapdN (in theory we can even make this optional!)
> - the memset task faulting pages in
>
> This also means these are the only tasks capable of becoming
> locked up and OOMing (as long as it is not under pressure).
>
> The rest of the system remains entirely functional for debugging.
>
> It now becomes possible to micro-benchmark and A/B test reclaim
> changes with different scenarios (number of tasks, amount of
> memory, watermark targets, etc), because we have hard controls
> over exactly which tasks can access that node memory and how.
>
> If we broke CAP_RECLAIM into subflags:
> - CAP_RECLAIM_KSWAPD
> - CAP_RECLAIM_DIRECT
> - CAP_COMPACTION_KCOMPACTD
> - CAP_COMPACTION_DIRECT
>
> We can actually test the efficacy of each of these mechanisms in
> isolation to each other - something that is strictly impossible today.
>
> Bonus Configuration: HBM device memory tiering
> ==============================================
> echo 1 > dax0.0/private # make the node private
> echo 0 > dax0.0/adistance # highest tier
> echo 1 > dax0.0/reclaim # reclaim active
> echo 1 > dax0.0/demotion # may demote from the node
> echo 1 > dax0.0/user_numa # mbind()
> echo online_movable > dax0.0/state
> echo 1 > numa/demotion_enabled
>
> This is an HBM device which is treated as the top-tier in the
> system but for which memory can only enter via explicit mbind().
>
> It can be overcommitted because it can be reclaimed (demotions
> go to CPU DRAM, and reclaim can swap from it).
>
> If the HBM is managed by an accelerator (GPU), the mmu_notifier
> allows it to know when reclaim is moving memory out to do
> device-mmu invalidation prior to migration.
>
> Prereqs, base commit, references
> ================================
> akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]
>
> Prereqs (all already in akpm/mm-new; listed for out-of-tree application):
>
> page_alloc.h split + alloc_flags plumbing this series rides on:
> commit e81fae43cd69 ("mm: split out internal page_alloc.h")
> commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")
>
> dax atomic whole-device hotplug (used by the dax extension [1]):
> commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper")
> commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")
>
> [1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
> [2] https://lore.kernel.org/all/20260423170219.281618-1-dave.jiang@intel.com/
> [3] https://lore.kernel.org/all/20260702-alloc-trylock-v4-0-0af8ff387e80@google.com/
>
> base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e
>
> Gregory Price (36):
> mm: refactor find_next_best_node to find_next_best_node_in
> mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
> mm/page_alloc: let the bulk and folio allocators carry alloc_flags
> numa: introduce N_MEMORY_PRIVATE
> mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
> cpuset: exclude private nodes from cpuset.mems (default-open)
> mm/memory_hotplug: disallow migration-driven private node hotunplug
> mm/mempolicy: skip private node folios when queueing for migration
> mm/migrate: disallow userland driven migration for private nodes
> mm/madvise: disallow madvise operations on private node folios
> mm/compaction: disallow compaction on private nodes
> mm/page_alloc: clear private node watermarks and system reserves
> mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
> mm/damon: skip private node memory in DAMON migration and pageout
> mm/ksm: skip KSM for managed-memory folios
> mm/khugepaged: skip private node folios when trying to collapse.
> mm/vmscan: disallow reclaim of private node memory
> mm/gup: disallow longterm pin of private node folios
> proc: include N_MEMORY_PRIVATE nodes in numa_maps output
> mm/memcontrol: account private-node memory in per-node stats
> proc/kcore: include private-node RAM in the kcore RAM map
> mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
> mm/mempolicy: apply policy at the kernel zone for private-node binds
> mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
> mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
> mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
> mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
> mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
> mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
> mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
> balancing
> mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning
> mm/khugepaged: base private node collapse eligiblity on actor/cap bits
> Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes
> mm/mempolicy: add mpol_set_shared_policy_range()
> KVM: guest_memfd: bind backing memory to a NUMA node at creation
> KVM: selftests: add a guest_memfd FLAG_BIND_NODE test
>
> Documentation/ABI/stable/sysfs-devices-node | 10 +
> Documentation/mm/index.rst | 1 +
> Documentation/mm/numa_private_nodes.rst | 160 ++++++++++
> drivers/base/node.c | 118 +++++++
> drivers/dax/kmem.c | 2 +-
> fs/proc/kcore.c | 5 +-
> fs/proc/task_mmu.c | 10 +-
> include/linux/gfp.h | 22 +-
> include/linux/kvm_host.h | 3 +
> include/linux/memory_hotplug.h | 5 +-
> include/linux/mempolicy.h | 16 +
> include/linux/mmzone.h | 26 +-
> include/linux/node_private.h | 262 ++++++++++++++++
> include/linux/nodemask.h | 7 +-
> include/uapi/linux/kvm.h | 5 +-
> include/uapi/linux/mempolicy.h | 1 +
> kernel/cgroup/cpuset.c | 26 +-
> mm/compaction.c | 13 +
> mm/damon/paddr.c | 9 +
> mm/gup.c | 28 +-
> mm/huge_memory.c | 5 +
> mm/internal.h | 97 +++++-
> mm/khugepaged.c | 18 +-
> mm/ksm.c | 8 +-
> mm/madvise.c | 8 +-
> mm/memcontrol-v1.c | 8 +-
> mm/memcontrol.c | 13 +-
> mm/memory-tiers.c | 40 ++-
> mm/memory_hotplug.c | 119 +++++++-
> mm/mempolicy.c | 288 +++++++++++++++---
> mm/migrate.c | 19 +-
> mm/mm_init.c | 2 +-
> mm/page_alloc.c | 189 +++++++++---
> mm/page_alloc.h | 34 +++
> mm/vmscan.c | 57 +++-
> tools/testing/selftests/kvm/Makefile.kvm | 2 +
> .../kvm/guest_memfd_bind_node_test.c | 213 +++++++++++++
> virt/kvm/guest_memfd.c | 39 ++-
> 38 files changed, 1728 insertions(+), 160 deletions(-)
> create mode 100644 Documentation/mm/numa_private_nodes.rst
> create mode 100644 include/linux/node_private.h
> create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
>
Thanks,
Balbir
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-21 3:46 ` [PATCH v5 00/36] Private Memory NUMA Nodes Balbir Singh
@ 2026-07-21 18:16 ` Gregory Price
2026-07-22 8:29 ` Balbir Singh
0 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-21 18:16 UTC (permalink / raw)
To: Balbir Singh
Cc: linux-mm, Zhigang.Luo, arun.george, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Tue, Jul 21, 2026 at 01:46:01PM +1000, Balbir Singh wrote:
> On 7/21/26 5:33 AM, Gregory Price wrote:
> >
> > Patch Layout
> > ============
> > The series is broken into two sections:
> >
> > 1) N_MEMORY_PRIVATE Introduction.
> > Introduce the node state.
> > Opt those nodes out of mm/ services.
> >
> > 2) NODE_PRIVATE_CAP_* features
> > A set of mm/ service opt-in flags that augment private
> > nodes to make them more useful (i.e. reclaim = overcommit).
> >
> > NODE_PRIVATE_CAP_LTPIN for private node folio pinning
> > NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
> > NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
> > NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
> > NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
> > NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
> >
>
> Looks reasonable, I wonder why USER_NUMA/HOTUNPUG is an opt-in?
>
Ideologically: Private nodes default to fully isolated, why should any
given feature be special?
Concretely:
User-numa
Some devices don't want the user to have control over placement.
I have been working on compressed memory, for example, which only
ever wants to be used as a reclaim-demotion target.
(The reasoning for this is another thread, i plan on publishing
my research on this this year)
Hot-unplug:
Some devices can't necessarily handle unexpected migration, and
hot-unplug is fundamentally a migration. So the HOTUNPLUG cap
actually means "hot-unplug can execute migrations".
If the entire device has pre-drained the memory (all memory is free)
then unplug works - it's just not very hot (no migrations) :]
Maybe a naming issue?
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-21 18:16 ` Gregory Price
@ 2026-07-22 8:29 ` Balbir Singh
2026-07-22 12:28 ` Gregory Price
0 siblings, 1 reply; 82+ messages in thread
From: Balbir Singh @ 2026-07-22 8:29 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On 7/22/26 4:16 AM, Gregory Price wrote:
> On Tue, Jul 21, 2026 at 01:46:01PM +1000, Balbir Singh wrote:
>> On 7/21/26 5:33 AM, Gregory Price wrote:
>>>
>>> Patch Layout
>>> ============
>>> The series is broken into two sections:
>>>
>>> 1) N_MEMORY_PRIVATE Introduction.
>>> Introduce the node state.
>>> Opt those nodes out of mm/ services.
>>>
>>> 2) NODE_PRIVATE_CAP_* features
>>> A set of mm/ service opt-in flags that augment private
>>> nodes to make them more useful (i.e. reclaim = overcommit).
>>>
>>> NODE_PRIVATE_CAP_LTPIN for private node folio pinning
>>> NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
>>> NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
>>> NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
>>> NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
>>> NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
>>>
>>
>> Looks reasonable, I wonder why USER_NUMA/HOTUNPUG is an opt-in?
>>
>
> Ideologically: Private nodes default to fully isolated, why should any
> given feature be special?
>
> Concretely:
>
> User-numa
>
> Some devices don't want the user to have control over placement.
> I have been working on compressed memory, for example, which only
> ever wants to be used as a reclaim-demotion target.
>
> (The reasoning for this is another thread, i plan on publishing
> my research on this this year)
>
I don't fully understand, how do we allocate memory on these devices then? Is
the driver expected to allocate and map via vm_insert_page()?
> Hot-unplug:
>
> Some devices can't necessarily handle unexpected migration, and
> hot-unplug is fundamentally a migration. So the HOTUNPLUG cap
> actually means "hot-unplug can execute migrations".
>
> If the entire device has pre-drained the memory (all memory is free)
> then unplug works - it's just not very hot (no migrations) :]
>
> Maybe a naming issue?
>
Yes, I would prefer MIGRATION in the name, HOTUNPLUG made me wonder how these
devices come online and because it is a device, it can go offline while the
system is still online
> ~Gregory
>
Balbir
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-22 8:29 ` Balbir Singh
@ 2026-07-22 12:28 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-22 12:28 UTC (permalink / raw)
To: Balbir Singh
Cc: linux-mm, Zhigang.Luo, arun.george, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 06:29:53PM +1000, Balbir Singh wrote:
> On 7/22/26 4:16 AM, Gregory Price wrote:
> >
> > User-numa
> >
> > Some devices don't want the user to have control over placement.
> > I have been working on compressed memory, for example, which only
> > ever wants to be used as a reclaim-demotion target.
> >
> > (The reasoning for this is another thread, i plan on publishing
> > my research on this this year)
> >
>
> I don't fully understand, how do we allocate memory on these devices then? Is
> the driver expected to allocate and map via vm_insert_page()?
>
It depends on your use case, but yes that's one possibility.
In another use case, you could enable (CAP_RECLAIM | CAP_DEMOTION) and
pages are allocated via alloc_demotion_target() in the reclaim path.
For general "device hosted memory", you can do a variety of things:
Direct alloc
ioctl(...) -> explicit alloc call
mmap(/dev/my_device) + mm_fault handler
Or a kernel-internal mempolicy
ioctl(...) -> set kernel-internal mempolicy on a vma
mmap(/dev/my_device) + kernel-internal only mempolicy on the vma
The KVM guest_memfd patch included at the end of the series uses the
kernel-internal mempolicy mechanism:
https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#me04a7e3556677babd82bfe3947b1a83b7f0017e1
This pattern is needed because the memory is never mapped into userland,
so userland actually *can't* set a mempolicy on this memory itself.
and then of course: CAP_USER_NUMA + mbind/set_mempolicy directly
This is kind of the point - the source of the memory has some control
over how it can be used.
> > Hot-unplug:
> >
> > Some devices can't necessarily handle unexpected migration, and
> > hot-unplug is fundamentally a migration. So the HOTUNPLUG cap
> > actually means "hot-unplug can execute migrations".
> >
> > If the entire device has pre-drained the memory (all memory is free)
> > then unplug works - it's just not very hot (no migrations) :]
> >
> > Maybe a naming issue?
> >
>
> Yes, I would prefer MIGRATION in the name, HOTUNPLUG made me wonder how these
> devices come online and because it is a device, it can go offline while the
> system is still online
>
There's a difference between device hotplug and memory hotplug. The
kernel's memory hotunplug system can at best be described as "best
effort", and if hotplugged as ZONE_NORMAL (on a non-private node) you're
unlikely to ever be able to hotunplug.
It's possible that this CAP bit should just go away and if the device
doesn't want its memory to be run migratable then it needs to hold extra
references on the folios. That's probably reasonable and looks a lot
like a long term pin.
(note: I have to respin for Sashiko fixes, i'm likely going to drop most
of the CAP bits except USER_NUMA in v6 and have them come in with
specific use cases).
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (35 preceding siblings ...)
2026-07-21 3:46 ` [PATCH v5 00/36] Private Memory NUMA Nodes Balbir Singh
@ 2026-07-21 13:26 ` Zenghui Yu
2026-07-21 17:18 ` Gregory Price
2026-07-22 14:20 ` Richard Cheng
2026-07-23 8:38 ` Arun George/Arun George
38 siblings, 1 reply; 82+ messages in thread
From: Zenghui Yu @ 2026-07-21 13:26 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On 2026/7/21 3:33, Gregory Price wrote:
> KVM: guest_memfd: bind backing memory to a NUMA node at creation
> KVM: selftests: add a guest_memfd FLAG_BIND_NODE test
I can't find the last patch in my inbox or on the lore. Not sure what went
wrong.
Thanks,
Zenghui
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-21 13:26 ` Zenghui Yu
@ 2026-07-21 17:18 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-21 17:18 UTC (permalink / raw)
To: Zenghui Yu
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Tue, Jul 21, 2026 at 09:26:09PM +0800, Zenghui Yu wrote:
> On 2026/7/21 3:33, Gregory Price wrote:
> > KVM: guest_memfd: bind backing memory to a NUMA node at creation
> > KVM: selftests: add a guest_memfd FLAG_BIND_NODE test
>
> I can't find the last patch in my inbox or on the lore. Not sure what went
> wrong.
>
> Thanks,
> Zenghui
Apologies, my email account got rate-limited to the number of recipients.
It should be in everyone's box shortly.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (36 preceding siblings ...)
2026-07-21 13:26 ` Zenghui Yu
@ 2026-07-22 14:20 ` Richard Cheng
2026-07-22 15:40 ` Gregory Price
2026-07-23 8:38 ` Arun George/Arun George
38 siblings, 1 reply; 82+ messages in thread
From: Richard Cheng @ 2026-07-22 14:20 UTC (permalink / raw)
To: Gregory Price
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Mon, Jul 20, 2026 at 03:33:54PM +0800, Gregory Price wrote:
> This series introduces the concept of "Private Memory Nodes", which
> are opted into two basic functionalities by default:
> - page allocation (mm/page_alloc.c)
> - OOM killing
>
> All other features of mm/ are opted out of managing these NUMA
> nodes and its memory. Then we add capability bits to allow node
> owners to opt back into those services (if supported).
>
> NUMA-hosted memory is presently a most-privilege system (any memory on
> any node, except ZONE_DEVICE, is 100% fungible and accessible). We
> then slap restrictions on top of it (cpuset, mempolicy, ZONE type,
> page/folio flags, etc).
>
Hi Gregory,
I applied your series on mm-next and give a quick review,
not thoroughly, still I have some questions below regarding the design.
> The goal here is to flip that dynamic, isolate by default and then
> opt-in to specific services that the device says is safe.
>
> Isolation at the NUMA/Zonelist layer provides a powerful mechanism for
> memory hosted on accelerators - re-use of the kernel mm/ code.
>
> - Accelerators (GPUs) can use demotion, numactl, and reclaim.
> - Special memory devices (Compressed RAM) with special access controls
> (promote-on-write) can have generic services written for them.
> - Network devices with large memory regions intended for ring buffers
> can use the buddy and standard networking stack.
> - Slow, disaggregated memory pools which aren't suitable as general
> purpose memory get cleaner interfaces (no need to re-write the buddy
> in userland, can use migration interface, etc).
> - Per-workload dedicated memory nodes (disaggregated VM memory)
>
For accelerator part, take CXL Type-2 as example, it has its own protocol
CXL.cache, CXL.mem which is the rule they need to obey during memory transaction,
Adding more rules for them since they are NUMA node confused me, I wonder the reason ?
Because NUMA node, at least for me, representing topology/locality rather than
something with ownership or capability or rules.
> And more use cases I have collected over the past few years.
>
> Not included here is a dax-extension [1] that exposes all the internal
> bits as userland controls for testing - along with a pile of selftests
> that prove correctness.
>
> Changes Since V4
> ================
> - Massive reduction in complexity.
> - no ops struct
> - no callback functions
> - no __GFP_PRIVATE
> - no __GFP_THISNODE requirement
> - no task flags (no PF_MEMALLOC_* in the alloc path)
> - isolation via a dedicated zonelist:
> - private nodes are omitted from FALLBACK/NOFALLBACK
> - added ZONELIST_PRIVATE(_NOFALLBACK)
I saw the reply in patch 5, so in fact there's not only one
dedicate zonelist, but numerous ?
I raise the question because zonelist was supposed to be a
global, unbypasssable thing in MM design, but now what you
are trying to do is to seperate the whole global list into several
parts ?
Can you explain why in current design you don't consider to support
something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination
of what a global zonelist should look like, no matter private or non.
> - ALLOC_ZONELIST_PRIVATE alloc_flag
> - on top of Brendan Jackman's recent mm/page_alloc.h work [3]
> - Zonelist selection rides the allocator's alloc_flags
> - rename OPS -> CAPS (capabilities)
> - split base functionality (isolation) from opt-ins (CAPS)
> - first half of series can be merged without CAPS
> - dropped compressed ram example from series
> - will submit separately if this moves forward
> - Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
> - fully functional dax-kmem extension and huge suite of selftests
> located at my github, to be discussed separately [1]
>
> Patch Layout
> ============
> The series is broken into two sections:
>
> 1) N_MEMORY_PRIVATE Introduction.
> Introduce the node state.
> Opt those nodes out of mm/ services.
>
> 2) NODE_PRIVATE_CAP_* features
> A set of mm/ service opt-in flags that augment private
> nodes to make them more useful (i.e. reclaim = overcommit).
>
> NODE_PRIVATE_CAP_LTPIN for private node folio pinning
> NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
> NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
> NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
> NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
> NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
>
> My hope is to merge at least #1 pulled ahead while #2 is debated.
>
> Allocation Isolation
> ====================
> page_alloc presently controls whether a node's memory can be allocated
> on a given call by 4 things (in order of authority)
>
> 1) ZONELIST membership
> If a node is not in the walked zonelist, it's unreachable.
>
So a device gets hotplugged in the system will get a dedicated zonelist here ?
And make sure it obeys the device's own protocol if it has one ?
> 2) __GFP_THISNODE
> If this flag is set and the only node in the ZONELIST is the
> singular preferred node (or the local node, for -1)
>
> 3) cpuset.mems membership
> cpuset trims any node in its allowed list
>
Sounds sane to me.
> 4) mempolicy nodemask
> the allocator will skip any node not in the nodemask.
>
> Except for #1 (Zonelist membership) there are all kinds of weird corner
> conditions in which 2-4 can be completely ignored (interrupt context,
> empty set because cpuset doesn't intersect mempolicy, shared vma, ...)
>
> But ZONELIST membership is *absolute*. If a zone is not in the
> zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.
>
> The existing zonelists are constructed like so:
> ZONELIST_FALLBACK : All N_MEMORY nodes
> ZONELIST_NOFALLBACK : A singleton N_MEMORY node
>
> Private node isolation is implemented via ZONELIST isolation:
> ZONELIST_PRIVATE : The private node + N_MEMORY
> ZONELIST_PRIVATE_NOFALLBACK : The private node alone.
>
> (mirrors exactly the fallback/nofallback for __GFP_THISNODE)
>
> Private nodes:
> 1) Never appear in any ZONELIST_FALLBACK
> 2) Have an empty ZONELIST_NOFALLBACK
> 3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)
>
> 1 & 2 mean all existing in-tree callers to page_alloc can NEVER
> accidentally allocate from a private node.
>
> An allocation must explicitly ask via a zonelist and a nodemask.
>
> alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
> __alloc_pages(..., nodemask); /* with the private node set */
>
As I stated above, NUMA concept was quite naive at first glance for my
limited knowledge.
This is quite alot to add for NUMA node concept, I'll want to see
more explanation in v6 and learn from it, thanks.
> The page allocator keeps all its original interfaces which only
> ever touch the default zonelists - avoiding churn.
>
> Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
> =================================================================
> The vast majority of the kernel will never need to know about
> ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.
>
> When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces
> do the zonelist selection for the source of the allocation.
>
> That really is the whole explanation of the mechanism:
>
> alloc_flags = mpol_alloc_flags(pol);
> page = __alloc_frozen_pages_noprof(..., alloc_flags);
>
> On mm-new this rides the allocator's existing alloc_flags plumbing:
> ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new
> parameter, enum, or alloc_context change is required.
>
> For modules that want to implement their own special handling, they
> get the _private variants for the page allocator. This lets modules
> re-use the buddy instead of rewriting it.
>
> - alloc_pages_node_private_noprof()
> - folio_alloc_node_private_noprof()
>
> This keeps ALLOC_ flags mm/ internal (these functions add the flags).
>
> Isolating private node folios from kernel services
> ==================================================
> We implement filter points in mm/ to prevent operations on
> private node memory. Where possible, we even re-use existing
> filter points from ZONE_DEVICE.
>
> Most filter points are one or two lines of code:
>
> Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
> - if (folio_is_zone_device(folio))
> + if (unlikely(folio_is_private_managed(folio)))
>
> Disabling a service:
> + if (!node_is_private(nid)) {
> + kswapd_run(nid);
> + kcompactd_run(nid);
> + }
>
> Disallowing a uapi interaction:
> + if (node_state(nid, N_MEMORY_PRIVATE))
> + return -EINVAL;
>
I'm not sure of why do we re-implement more filter and basically
doing the same thing ? Any unavoidable scenario ?
re-using the exisintg filter would be nice if that's possible.
> In the second half of the series, we replace blanket N_MEMORY_PRIVATE
> filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that
> interaction if CAP is set.
>
> We abstract this with a nice clean interface to make it really clear
> what is happening (nodes have features!)
>
> - if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
> + if (!node_allows_reclaim(pgdat->node_id))
>
> NODE_PRIVATE_CAP_* features
> ===========================
> This series of commits opts private nodes into various mm/ services.
>
> Capabilities:
> NODE_PRIVATE_CAP_RECLAIM - direct and kswapd reclaim
> NODE_PRIVATE_CAP_USER_NUMA - userland numa controls
> NODE_PRIVATE_CAP_DEMOTION - node is a demotion target
> NODE_PRIVATE_CAP_HOTUNPLUG - hotunplug may migrate
> NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
> NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally
>
> Some opt-in support is more intensive than others, so these features
> are broken out in a way that we can defer them as future work streams.
>
> NODE_PRIVATE_CAP_RECLAIM:
> Enabling reclaim for these nodes is actually surprisingly trivial.
>
> Without CAP_RECLAIM, when an allocation failure occurs, the system
> will not attempt to swap the memory - and instead will OOM (typically
> whatever task is using the most memory on *that* private node).
>
> This capability consists of:
> 1) enabling kswapd and kcompactd for that node at hotplug time.
> 2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
> 3) Sets normal watermarks for these nodes.
> 4) Allow madvise operations on that node (PAGEOUT).
> 5) A small tweak to how LRU decides which zones to visit.
>
> NODE_PRIVATE_CAP_USER_NUMA
> Enables the following userland interfaces to accept the node:
> mbind()
> set_mempolicy()
> set_mempolicy_home_node()
> move_pages()
> migrate_pages()
>
> example:
> buf = mmap(..., MAP_ANON);
> mbind(buf, ..., {private_node});
> buf[0] = 0xDEADBEEF; /* Page faults onto the private node */
>
> Later - the KVM example shows how in-kernel mempolicies can
> also be bound by CAP_USER_NUMA.
>
> Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.
>
> NODE_PRIVATE_CAP_HOTUNPLUG
> This is simple: allow hotunplug to migrate this nodes folios.
>
> Some devices may not be able to tolerate unexpected migrations,
> so we prevent hotunplug from engaging in migration by default.
>
> Some devices may have an mmu_notifier in their driver that can
> manage the migration and subsequent refault.
>
> CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.
>
> NODE_PRIVATE_CAP_DEMOTION
> This adds the private node as a valid demotion target, and allows
> reclaim to demote memory from a private node to a demotion target.
>
> Requires: NODE_PRIVATE_CAP_RECLAIM
>
> NODE_PRIVATE_CAP_NUMA_BALANCING
> This enables numa balancing to inject prot_none on private node
> folio mappings and promote them when faults are taken.
>
> NODE_PRIVATE_CAP_LTPIN
> This allows GUP Longterm Pinning to operate normally.
>
> Normally, longterm pinning determines folio eligibility based
> on its ZONE_* membership (among other things).
>
> ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require
> migration to ZONE_NORMAL before pinning.
>
> Neither operation is preferable by default on a private node,
> so the base behavior of FOLL_LONGTERM is to FAIL.
>
> This capability allows longterm pinning to operate normally
> based on the ZONE membership. Private node memory may be
> hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.
>
> In-tree User: KVM
> =================
> Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of
> enabling disaggregated memory pools to host dedicated KVM memory.
>
> With private nodes, this is trivial (with a bit of basic plumbing):
>
> static int kvm_gmem_bind_node(struct inode *inode, int node)
> {
> ...
> /* Bind to a private node - gated on CAP_USER_NUMA */
> pol = mpol_bind_node(node);
> if (IS_ERR(pol))
> return PTR_ERR(pol);
>
> /* Set the shared policy */
> err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
> ...
> }
>
> KVM doesn't even need to know about private nodes at all, all it
> does is ask mempolicy whether the requested node is a valid bind.
>
> mm/ component testing with dax driver
> =====================================
> The dax driver extensions[1] implements a simple interface to create
> a private node from a dax device created by any source.
>
> I left the dax driver extensions out of this feature set because
> it locks in the CAP_ bits before anyone has input. It's there
> primarily for testing at this point.
>
> The simplest way to get a dax device is with the memmap= boot arg.
> e.g.: "memmap=0x40000000!0x140000000"
>
> The dax driver extension has the following sysfs entries:
> dax0.0/private - set the node to private
> dax0.0/dax_file - make /dev/dax0.0 mmap'able in kmem mode
> dax0.0/adistance - dictate memory_tierN membership
> dax0.0/reclaim - CAP_RECLAIM
> dax0.0/demotion - CAP_DEMOTION
> dax0.0/user_numa - CAP_USER_NUMA
> dax0.0/hotunplug - CAP_HOTUNPLUG
> dax0.0/numa_balancing - CAP_NUMA_BALANCING
> dax0.0/ltpin - CAP_LTPIN
>
> Now consider the following...
>
> Single node reclaim + mbind support:
> echo 1 > dax0.0/private
> echo 1 > dax0.0/reclaim
> echo 1 > dax0.0/user_numa
> echo online_movable > dax0.0/state
>
> Test program:
> /* node1: 1GB Private Memory Node, 4GB swap */
> buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
> MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
> memset(buf, 0xff, TWO_GB);
>
> We can *guarantee* the ONLY reclaiming tasks are exactly:
> - kswapdN (in theory we can even make this optional!)
> - the memset task faulting pages in
>
> This also means these are the only tasks capable of becoming
> locked up and OOMing (as long as it is not under pressure).
>
> The rest of the system remains entirely functional for debugging.
>
> It now becomes possible to micro-benchmark and A/B test reclaim
> changes with different scenarios (number of tasks, amount of
> memory, watermark targets, etc), because we have hard controls
> over exactly which tasks can access that node memory and how.
>
> If we broke CAP_RECLAIM into subflags:
> - CAP_RECLAIM_KSWAPD
> - CAP_RECLAIM_DIRECT
> - CAP_COMPACTION_KCOMPACTD
> - CAP_COMPACTION_DIRECT
>
> We can actually test the efficacy of each of these mechanisms in
> isolation to each other - something that is strictly impossible today.
>
> Bonus Configuration: HBM device memory tiering
> ==============================================
> echo 1 > dax0.0/private # make the node private
> echo 0 > dax0.0/adistance # highest tier
> echo 1 > dax0.0/reclaim # reclaim active
> echo 1 > dax0.0/demotion # may demote from the node
> echo 1 > dax0.0/user_numa # mbind()
> echo online_movable > dax0.0/state
> echo 1 > numa/demotion_enabled
>
> This is an HBM device which is treated as the top-tier in the
> system but for which memory can only enter via explicit mbind().
>
> It can be overcommitted because it can be reclaimed (demotions
> go to CPU DRAM, and reclaim can swap from it).
>
> If the HBM is managed by an accelerator (GPU), the mmu_notifier
> allows it to know when reclaim is moving memory out to do
> device-mmu invalidation prior to migration.
>
> Prereqs, base commit, references
> ================================
> akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]
>
Still thanks for the work, I learn alot from your work as well, thanks.
Best regards,
Richard Cheng.
> Prereqs (all already in akpm/mm-new; listed for out-of-tree application):
>
> page_alloc.h split + alloc_flags plumbing this series rides on:
> commit e81fae43cd69 ("mm: split out internal page_alloc.h")
> commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")
>
> dax atomic whole-device hotplug (used by the dax extension [1]):
> commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper")
> commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")
>
> [1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
> [2] https://lore.kernel.org/all/20260423170219.281618-1-dave.jiang@intel.com/
> [3] https://lore.kernel.org/all/20260702-alloc-trylock-v4-0-0af8ff387e80@google.com/
>
> base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e
>
> Gregory Price (36):
> mm: refactor find_next_best_node to find_next_best_node_in
> mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
> mm/page_alloc: let the bulk and folio allocators carry alloc_flags
> numa: introduce N_MEMORY_PRIVATE
> mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
> cpuset: exclude private nodes from cpuset.mems (default-open)
> mm/memory_hotplug: disallow migration-driven private node hotunplug
> mm/mempolicy: skip private node folios when queueing for migration
> mm/migrate: disallow userland driven migration for private nodes
> mm/madvise: disallow madvise operations on private node folios
> mm/compaction: disallow compaction on private nodes
> mm/page_alloc: clear private node watermarks and system reserves
> mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
> mm/damon: skip private node memory in DAMON migration and pageout
> mm/ksm: skip KSM for managed-memory folios
> mm/khugepaged: skip private node folios when trying to collapse.
> mm/vmscan: disallow reclaim of private node memory
> mm/gup: disallow longterm pin of private node folios
> proc: include N_MEMORY_PRIVATE nodes in numa_maps output
> mm/memcontrol: account private-node memory in per-node stats
> proc/kcore: include private-node RAM in the kcore RAM map
> mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
> mm/mempolicy: apply policy at the kernel zone for private-node binds
> mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
> mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
> mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
> mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
> mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
> mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
> mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
> balancing
> mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning
> mm/khugepaged: base private node collapse eligiblity on actor/cap bits
> Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes
> mm/mempolicy: add mpol_set_shared_policy_range()
> KVM: guest_memfd: bind backing memory to a NUMA node at creation
> KVM: selftests: add a guest_memfd FLAG_BIND_NODE test
>
> Documentation/ABI/stable/sysfs-devices-node | 10 +
> Documentation/mm/index.rst | 1 +
> Documentation/mm/numa_private_nodes.rst | 160 ++++++++++
> drivers/base/node.c | 118 +++++++
> drivers/dax/kmem.c | 2 +-
> fs/proc/kcore.c | 5 +-
> fs/proc/task_mmu.c | 10 +-
> include/linux/gfp.h | 22 +-
> include/linux/kvm_host.h | 3 +
> include/linux/memory_hotplug.h | 5 +-
> include/linux/mempolicy.h | 16 +
> include/linux/mmzone.h | 26 +-
> include/linux/node_private.h | 262 ++++++++++++++++
> include/linux/nodemask.h | 7 +-
> include/uapi/linux/kvm.h | 5 +-
> include/uapi/linux/mempolicy.h | 1 +
> kernel/cgroup/cpuset.c | 26 +-
> mm/compaction.c | 13 +
> mm/damon/paddr.c | 9 +
> mm/gup.c | 28 +-
> mm/huge_memory.c | 5 +
> mm/internal.h | 97 +++++-
> mm/khugepaged.c | 18 +-
> mm/ksm.c | 8 +-
> mm/madvise.c | 8 +-
> mm/memcontrol-v1.c | 8 +-
> mm/memcontrol.c | 13 +-
> mm/memory-tiers.c | 40 ++-
> mm/memory_hotplug.c | 119 +++++++-
> mm/mempolicy.c | 288 +++++++++++++++---
> mm/migrate.c | 19 +-
> mm/mm_init.c | 2 +-
> mm/page_alloc.c | 189 +++++++++---
> mm/page_alloc.h | 34 +++
> mm/vmscan.c | 57 +++-
> tools/testing/selftests/kvm/Makefile.kvm | 2 +
> .../kvm/guest_memfd_bind_node_test.c | 213 +++++++++++++
> virt/kvm/guest_memfd.c | 39 ++-
> 38 files changed, 1728 insertions(+), 160 deletions(-)
> create mode 100644 Documentation/mm/numa_private_nodes.rst
> create mode 100644 include/linux/node_private.h
> create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
>
> --
> 2.53.0-Meta
>
>
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-22 14:20 ` Richard Cheng
@ 2026-07-22 15:40 ` Gregory Price
2026-07-22 20:56 ` Gregory Price
0 siblings, 1 reply; 82+ messages in thread
From: Gregory Price @ 2026-07-22 15:40 UTC (permalink / raw)
To: Richard Cheng
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 10:20:00PM +0800, Richard Cheng wrote:
> On Mon, Jul 20, 2026 at 03:33:54PM +0800, Gregory Price wrote:
> Hi Gregory,
>
> I applied your series on mm-next and give a quick review,
> not thoroughly, still I have some questions below regarding the design.
>
Hi Richard,
Thank you for the read.
Just a heads up, this was based on mm-new because of some recent work
on Brendan's page_alloc.h cleanup, but if you managed to get it applied
then maybe that's made its way forward already.
> > The goal here is to flip that dynamic, isolate by default and then
> > opt-in to specific services that the device says is safe.
> >
> > Isolation at the NUMA/Zonelist layer provides a powerful mechanism for
> > memory hosted on accelerators - re-use of the kernel mm/ code.
> >
> > - Accelerators (GPUs) can use demotion, numactl, and reclaim.
> > - Special memory devices (Compressed RAM) with special access controls
> > (promote-on-write) can have generic services written for them.
> > - Network devices with large memory regions intended for ring buffers
> > can use the buddy and standard networking stack.
> > - Slow, disaggregated memory pools which aren't suitable as general
> > purpose memory get cleaner interfaces (no need to re-write the buddy
> > in userland, can use migration interface, etc).
> > - Per-workload dedicated memory nodes (disaggregated VM memory)
> >
>
> For accelerator part, take CXL Type-2 as example, it has its own protocol
> CXL.cache, CXL.mem which is the rule they need to obey during memory transaction,
> Adding more rules for them since they are NUMA node confused me, I wonder the reason ?
>
CXL protocols simply state how to do the memory transaction at a
physical / transport level. It does not make any statement on how the
operating systems are to make sense of these devices or what constructs
/ abstractions to build to actually manage the devices themselves.
This series is not necessarily attached to CXL in particular, you could
just as easily carve out memory from the general pool onto a private
node to ensure it only gets used for a particular use.
In fact, that is how I have been testing this with dax/kmem:
https://github.com/gourryinverse/linux/commit/f279e741d9c3643597525bfab916b29b95cb635b
> Because NUMA node, at least for me, representing topology/locality rather than
> something with ownership or capability or rules.
>
The NUMA abstraction representing topology/locality is a construct we
(the OS developers) have decided on historically - but there's nothing
that dictates we can never create new useful abstractions with it.
Consider:
N_NORMAL_MEMORY, /* The node has regular memory */
N_HIGH_MEMORY, /* The node has regular or high memory */
These have nothing to do with topology or locality, they are node states
that only have meaning in the context of linux mm/.
To make something clear - there is no *requirement* for any particular
device to use this abstraction. It simply enables a cleaner way for
devices carrying memory to re-utilize mm/ services while ensuring their
memory does not silently get used under system pressure (or some vagrant
in userspace doing `numactl --interleave all`).
Ignoring all the CAP bits entirely, if you just took the base series
your driver could re-use the buddy allocator without any special logic
AND have confidence that your driver is the only possible user of that
memory (barring some truly obscene bug).
> > - isolation via a dedicated zonelist:
> > - private nodes are omitted from FALLBACK/NOFALLBACK
> > - added ZONELIST_PRIVATE(_NOFALLBACK)
>
> I saw the reply in patch 5, so in fact there's not only one
> dedicate zonelist, but numerous ?
> I raise the question because zonelist was supposed to be a
> global, unbypasssable thing in MM design, but now what you
> are trying to do is to seperate the whole global list into several
> parts ?
>
> Can you explain why in current design you don't consider to support
> something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination
> of what a global zonelist should look like, no matter private or non.
>
First let me say that there's nothing that prevents us from doing this,
and we *could* make this the default case - but this decision was
intentional by me.
Having them all present in each-other's zonelists by default creates a
number of implicit opt-ins that are unclear:
- any direct zonelist iterator now iterates all zones on all private
nodes, even those nodes do not opt into the same services
e.g. zonelist iteration in reclaim that targets Private Node A would
attempt to reclaim Private Node B as well. That would require and
extra explicit filter.
I ask: Why do this? Just isolate in the zonelist, and if there is
a desire for intersections - make it explicit, not implicit.
- fallback allocations can now occur across private nodes, even
if those nodes are intended for different purposes.
Obviously you can use nodemask to tighten the allocation target, but
I use `numactl --interleave --all` as an example of a clear case
where intersected nodelists may not give you the behavior you want.
So for consistency - everything including zonelists have full isolation.
This way all interactions with a private node must be explicit - always.
This is actually one of the problems with ZONE_DEVICE - and you can see
it in this patch set. Some of the hooks in mm/ for zone_device only
apply to PTE cases, and are absent from PMD cases - only because PMD
mappings in ZONE_DEVICE aren't supported.
That kind of implicit behavior is quite bad.
That said, future improvements could include something like:
for_reclaimable_zone(ZONELIST_PRIVATE) {}
where we loosen this isolation, and formalize a filter, but I would
like to see the usecase for it first. Loosening the isolation defeats
the entire purpose of the series, so there should be a strong reason to
do so.
> > Allocation Isolation
> > ====================
> > page_alloc presently controls whether a node's memory can be allocated
> > on a given call by 4 things (in order of authority)
> >
> > 1) ZONELIST membership
> > If a node is not in the walked zonelist, it's unreachable.
> >
>
> So a device gets hotplugged in the system will get a dedicated zonelist here ?
Yes.
> And make sure it obeys the device's own protocol if it has one ?
>
I'm not sure i follow this question, can you help me understand?
If by protocol you mean the CAP bits (opting into reclaim, demotion,
etc), then yes. If you mean something else (CXL) then I think that's
orthogonal and unrelated.
> > Private nodes:
> > 1) Never appear in any ZONELIST_FALLBACK
> > 2) Have an empty ZONELIST_NOFALLBACK
> > 3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)
> >
> > 1 & 2 mean all existing in-tree callers to page_alloc can NEVER
> > accidentally allocate from a private node.
> >
> > An allocation must explicitly ask via a zonelist and a nodemask.
> >
> > alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
> > __alloc_pages(..., nodemask); /* with the private node set */
> >
>
> As I stated above, NUMA concept was quite naive at first glance for my
> limited knowledge.
>
> This is quite alot to add for NUMA node concept, I'll want to see
> more explanation in v6 and learn from it, thanks.
>
Sure, I can expand on it. I think it's not as much to add as you think
though - it's simply adding the concept of isolation to a NUMA node.
Some more explanation below, but if you think there is something i
should explicit spell out in v6 cover, please let me know.
---
I agree that NUMA as a concept was best-effort for, comically enough,
somewhat *Uniform* memory access - instead of *Non*-Uniform memory access.
The current abstraction quite nicely handles the case where all memory
on the system is roughly of the same calibre (DDR4, DDR5, etc) and
roughly for the same purpose (general system memory).
But it is quite incapable of handling truly heterogeneous memory systems
(precious HBM attached to the CPU, GPU's with HBM over a coherent link,
hardware-compressed memory expansion, network devices w/ memory, etc).
If you look at the history of ZONE_DEVICE, what it fundamentally does is
slaps an isolation mechanism on top of NUMA nodes because the NUMA
abstraction doesn't provide one.
The problem with that approach is now you have to reason about a node
having both fungible and non-fungible memory. It creates the need for
something like migrate_device.c when a properly isolated NUMA node could
just use migrate.c directly (with a coherent link).
One example of what isolation on the node enables:
A Private Node can hotplug memory in ZONE_NORMAL - which means it
can be GUP pinned. That means driver support for GPU direct storage
is simply `alloc_pages_node() + pin()`. The driver doesn't have to
worry about something like SLAB accidentally using that same memory.
All without having to rewrite a bunch of mm/ in a driver, and with
having to do some kind of heroics with ZONE_DEVICE that causes even
more special mm/ interactions.
That only comes from adding an isolation primitive to a NUMA node.
> > Isolating private node folios from kernel services
> > ==================================================
> > We implement filter points in mm/ to prevent operations on
> > private node memory. Where possible, we even re-use existing
> > filter points from ZONE_DEVICE.
> >
> > Most filter points are one or two lines of code:
> >
> > Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
> > - if (folio_is_zone_device(folio))
> > + if (unlikely(folio_is_private_managed(folio)))
> >
> > Disabling a service:
> > + if (!node_is_private(nid)) {
> > + kswapd_run(nid);
> > + kcompactd_run(nid);
> > + }
> >
> > Disallowing a uapi interaction:
> > + if (node_state(nid, N_MEMORY_PRIVATE))
> > + return -EINVAL;
> >
>
> I'm not sure of why do we re-implement more filter and basically
> doing the same thing ? Any unavoidable scenario ?
>
> re-using the exisintg filter would be nice if that's possible.
>
We re-use (combine) existing filters where possible.
We add new ones where ZONE_DEVICE did not implement filters due to some
*implicit* filter already existing.
Two clear examples:
- reclaim does not target ZONE_DEVICE
- ZONE_DEVICE does not support PMD
Both cases result in private-node filters that otherwise would have to
exist for ZONE_DEVICE (and if ZONE_DEVICE ever grows PMD support, those
filters will have to be added).
> > Bonus Configuration: HBM device memory tiering
> > ==============================================
> > echo 1 > dax0.0/private # make the node private
> > echo 0 > dax0.0/adistance # highest tier
> > echo 1 > dax0.0/reclaim # reclaim active
> > echo 1 > dax0.0/demotion # may demote from the node
> > echo 1 > dax0.0/user_numa # mbind()
> > echo online_movable > dax0.0/state
> > echo 1 > numa/demotion_enabled
> >
> > This is an HBM device which is treated as the top-tier in the
> > system but for which memory can only enter via explicit mbind().
> >
> > It can be overcommitted because it can be reclaimed (demotions
> > go to CPU DRAM, and reclaim can swap from it).
> >
> > If the HBM is managed by an accelerator (GPU), the mmu_notifier
> > allows it to know when reclaim is moving memory out to do
> > device-mmu invalidation prior to migration.
> >
> > Prereqs, base commit, references
> > ================================
> > akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]
> >
>
> Still thanks for the work, I learn alot from your work as well, thanks.
>
Of course, and thank you for reading.
If nothing else I hope this series helps folks understand the page
allocator better (it certainly has helped me). If there is anything
you think I can improve or better explain, I am happy to discuss.
I am planning a larger publication of all my research sometime in the
future, but I think code is more impactful and useful - so I am
prioritizing that for now :].
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-22 15:40 ` Gregory Price
@ 2026-07-22 20:56 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-22 20:56 UTC (permalink / raw)
To: Richard Cheng
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
On Wed, Jul 22, 2026 at 11:40:06AM -0400, Gregory Price wrote:
> On Wed, Jul 22, 2026 at 10:20:00PM +0800, Richard Cheng wrote:
> >
> > I saw the reply in patch 5, so in fact there's not only one
> > dedicate zonelist, but numerous ?
> > I raise the question because zonelist was supposed to be a
> > global, unbypasssable thing in MM design, but now what you
> > are trying to do is to seperate the whole global list into several
> > parts ?
> >
> > Can you explain why in current design you don't consider to support
> > something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination
> > of what a global zonelist should look like, no matter private or non.
> >
>
> First let me say that there's nothing that prevents us from doing this,
> and we *could* make this the default case - but this decision was
> intentional by me.
>
You caught me on something here - I am wrong about my own code.
I described to you how I originally implemented zonelist isolation,
which I walked back when I was testing mempolicy and demotion.
The current code and the comments in the code are correct, we get
the following zonelists.
N_MEMORY : 0,1
N_MEMORY_PRIVATE: 2,3
ZONELIST_PRIVATE[0] = {0,1,2,3}
ZONELIST_PRIVATE[1] = {1,0,2,3}
ZONELIST_PRIVATE[2] = {2,0,1,3}
ZONELIST_PRIVATE[3] = {3,0,1,2}
Meaning everything except for MPOL_F_RELATIVE_NODES works (because
of cpuset isolation, but more on this in a moment).
So what I shipped here is what you are describing.
Looking back at my notes - I realized a few things that lead me to
walk back that isolation:
1) mempolicy makes more sense this way.
2) reclaim/demotion is easier to implement without isolation.
3) The individual service checks and nodelists handle the rest.
4) It overally just makes more sense. If something breaks isolation
it's because there's a bug, not because of a structural issue.
For example in demotion we do this:
demote_folio_list():
/* Get demotion targets, find the best one, and demote */
node_get_allowed_targets(pgdat, &allowed_mask);
target_nid = next_demotion_node(pgdat->node_id, &allowed_mask);
migrate_pages(demote_folios, alloc_demote_folio, ...);
Where the first step already filters on CAP_DEMOTION.
The result is still clean - no special iterators needed.
What I realized from your questions here is that since i walked back the
zonelist isolation, I think i can actually include any CAP_USER_NUMA
node in cpuset.mems and allow parititioning fairly trivially.
I think this might be the last majority complexity falling out.
Thank you for the questions - this has helped.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread
* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-20 19:33 ` [PATCH v5 00/36] Private Memory NUMA Nodes Gregory Price
` (37 preceding siblings ...)
2026-07-22 14:20 ` Richard Cheng
@ 2026-07-23 8:38 ` Arun George/Arun George
2026-07-23 16:53 ` Gregory Price
38 siblings, 1 reply; 82+ messages in thread
From: Arun George/Arun George @ 2026-07-23 8:38 UTC (permalink / raw)
To: Gregory Price, linux-mm
Cc: Zhigang.Luo, balbirs, brendan.jackman, yuzenghui, apopple,
alucerop, matthew.brost, akpm, david, ljs, liam, vbabka, rppt,
surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team, cpgs
On 21-07-2026 01:03 am, Gregory Price wrote:
> Changes Since V4
> ================
> - dropped compressed ram example from series
> - will submit separately if this moves forward
We intend to test this series on a compression capable CXL expander
hardware. Since compressed ram example (cram) is not part of this
series, how do you suggest to do that? Do you have a version of cram
module compatible with this series?
~Arun
^ permalink raw reply [flat|nested] 82+ messages in thread* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
2026-07-23 8:38 ` Arun George/Arun George
@ 2026-07-23 16:53 ` Gregory Price
0 siblings, 0 replies; 82+ messages in thread
From: Gregory Price @ 2026-07-23 16:53 UTC (permalink / raw)
To: Arun George/Arun George
Cc: linux-mm, Zhigang.Luo, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team, cpgs
On Thu, Jul 23, 2026 at 02:08:31PM +0530, Arun George/Arun George wrote:
> On 21-07-2026 01:03 am, Gregory Price wrote:
> > Changes Since V4
> > ================
> > - dropped compressed ram example from series
> > - will submit separately if this moves forward
>
> We intend to test this series on a compression capable CXL expander
> hardware. Since compressed ram example (cram) is not part of this
> series, how do you suggest to do that? Do you have a version of cram
> module compatible with this series?
>
> ~Arun
Hi Arun,
I dropped it from the base series because i want to try to move forward
with this without RFC. The compressed service itself will be a separate
submission after I get decent signal on whether this will be accepted.
I plan to RFC the new setup for compressed ram this a bit later, but I
will share my working branch with you for testing.
~Gregory
^ permalink raw reply [flat|nested] 82+ messages in thread