* [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Add MPOL_F_PRIVATE, the internal flag noting that a policy contains
a nodemask with a private node, and making private node memory
reachable via standard mempolicies.
Plumb zonelist selection into the mpol allocator interfaces via the
allocator's alloc_flags, using the alloc_flags-carrying page_alloc
interfaces.
mpol_alloc_flags() maps MPOL_F_PRIVATE to the alloc_flags an
allocation uses:
ALLOC_DEFAULT - normal allocation
ALLOC_ZONELIST_PRIVATE - private node allocation
With this, a VMA with (mpol->flags & MPOL_F_PRIVATE) can successfully
services faults like any other mbind() from the private node:
buf = mmap(..., MAP_ANON);
mbind(buf, ..., {private_node});
buf[0] = 0xdeadbeef; // Page faulted from private node memory
Like any other bind, the nodemask relaxes if it is unsatisfiable.
For example: apply_policy_zone() allows an unmovable allocation
targeting the VMA to fallback to a viable node instead of failing.
cpuset rebinding does not affect private nodes - the nodes in the
original mask are preserved, and we enforce two remap rules:
1) never allow N_MEMORY nodes to remap to N_MEMORY_PRIVATE
2) never remap N_MEMORY_PRIVATE nodes at all (they stay in place)
In the case of an empty nodemask as a result of rebind, revert to
cpuset - which never includes N_MEMORY_PRIVATE and always guarantees
at least one N_MEMORY node.
As of this patch, nothing can actually sets MPOL_F_PRIVATE, so this
is a no-op that simply adds the plumbing throughout mempolicy.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/uapi/linux/mempolicy.h | 1 +
mm/mempolicy.c | 79 +++++++++++++++++++++++-----------
2 files changed, 55 insertions(+), 25 deletions(-)
diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h
index 7f6fc9599693b..87af18c84b947 100644
--- a/include/uapi/linux/mempolicy.h
+++ b/include/uapi/linux/mempolicy.h
@@ -67,6 +67,7 @@ enum mempolicy_mode {
#define MPOL_F_SHARED (1 << 0) /* identify shared policies */
#define MPOL_F_MOF (1 << 3) /* this policy wants migrate on fault */
#define MPOL_F_MORON (1 << 4) /* Migrate On protnone Reference On Node */
+#define MPOL_F_PRIVATE (1 << 5) /* nodemask contains private nodes */
/*
* Enabling zone reclaim means the page allocator will attempt to fulfill
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 2b76c57a460c9..90110e9761122 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -432,6 +432,10 @@ static int mpol_set_nodemask(struct mempolicy *pol,
else
pol->w.cpuset_mems_allowed = cpuset_current_mems_allowed;
+ /* If any private nodes left in the nodemask - add the private flag */
+ if (nodes_intersects(nsc->mask2, node_states[N_MEMORY_PRIVATE]))
+ pol->flags |= MPOL_F_PRIVATE;
+
ret = mpol_ops[pol->mode].create(pol, &nsc->mask2);
return ret;
}
@@ -505,22 +509,33 @@ static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
static void mpol_rebind_nodemask(struct mempolicy *pol, const nodemask_t *nodes)
{
- nodemask_t tmp;
+ nodemask_t tmp, priv;
+
+ /* preserve online private nodes to re-add later */
+ nodes_and(priv, pol->nodes, node_states[N_MEMORY_PRIVATE]);
if (pol->flags & MPOL_F_STATIC_NODES)
nodes_and(tmp, pol->w.user_nodemask, *nodes);
- else if (pol->flags & MPOL_F_RELATIVE_NODES)
- mpol_relative_nodemask(&tmp, &pol->w.user_nodemask, nodes);
- else {
+ else if (pol->flags & MPOL_F_RELATIVE_NODES) {
+ /* fold only the public part: a private node must not take a slot */
+ nodes_and(tmp, pol->w.user_nodemask, node_states[N_MEMORY]);
+ mpol_relative_nodemask(&tmp, &tmp, nodes);
+ } else {
nodes_remap(tmp, pol->nodes, pol->w.cpuset_mems_allowed,
*nodes);
pol->w.cpuset_mems_allowed = *nodes;
}
- if (nodes_empty(tmp))
+ /* private nodes are identity-mapped during remap, drop them here */
+ nodes_and(tmp, tmp, node_states[N_MEMORY]);
+ if (nodes_empty(tmp) && nodes_empty(priv))
tmp = *nodes;
- pol->nodes = tmp;
+ /* If any online private nodes remain, add them back */
+ nodes_or(pol->nodes, tmp, priv);
+ /* If no online private nodes remain, strip the private flag */
+ if (nodes_empty(priv))
+ pol->flags &= ~MPOL_F_PRIVATE;
}
static void mpol_rebind_preferred(struct mempolicy *pol,
@@ -2411,7 +2426,8 @@ bool mempolicy_in_oom_domain(struct task_struct *tsk,
}
static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
- int nid, nodemask_t *nodemask)
+ int nid, nodemask_t *nodemask,
+ unsigned int aflags)
{
struct page *page;
gfp_t preferred_gfp;
@@ -2425,14 +2441,21 @@ static struct page *alloc_pages_preferred_many(gfp_t gfp, unsigned int order,
preferred_gfp = gfp | __GFP_NOWARN;
preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
page = __alloc_frozen_pages_noprof(preferred_gfp, order, nid, nodemask,
- ALLOC_DEFAULT);
+ aflags);
if (!page)
page = __alloc_frozen_pages_noprof(gfp, order, nid, NULL,
- ALLOC_DEFAULT);
+ aflags);
return page;
}
+/* A private policy allocates from the private zonelist. */
+static inline unsigned int mpol_alloc_flags(struct mempolicy *pol)
+{
+ return (pol->flags & MPOL_F_PRIVATE) ? ALLOC_ZONELIST_PRIVATE :
+ ALLOC_DEFAULT;
+}
+
/**
* alloc_pages_mpol - Allocate pages according to NUMA mempolicy.
* @gfp: GFP flags.
@@ -2448,11 +2471,13 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
{
nodemask_t *nodemask;
struct page *page;
+ unsigned int aflags = mpol_alloc_flags(pol);
nodemask = policy_nodemask(gfp, pol, ilx, &nid);
if (pol->mode == MPOL_PREFERRED_MANY)
- return alloc_pages_preferred_many(gfp, order, nid, nodemask);
+ return alloc_pages_preferred_many(gfp, order, nid, nodemask,
+ aflags);
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
/* filter "hugepage" allocation, unless from alloc_pages() */
@@ -2476,7 +2501,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
*/
page = __alloc_frozen_pages_noprof(
gfp | __GFP_THISNODE | __GFP_NORETRY, order,
- nid, NULL, ALLOC_DEFAULT);
+ nid, NULL, aflags);
if (page || !(gfp & __GFP_DIRECT_RECLAIM))
return page;
/*
@@ -2488,7 +2513,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
}
}
- page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, ALLOC_DEFAULT);
+ page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, aflags);
if (unlikely(pol->mode == MPOL_INTERLEAVE ||
pol->mode == MPOL_WEIGHTED_INTERLEAVE) && page) {
@@ -2597,6 +2622,7 @@ static unsigned long alloc_pages_bulk_interleave(gfp_t gfp,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
{
+ unsigned int aflags = mpol_alloc_flags(pol);
int nodes;
unsigned long nr_pages_per_node;
int delta;
@@ -2610,14 +2636,14 @@ static unsigned long alloc_pages_bulk_interleave(gfp_t gfp,
for (i = 0; i < nodes; i++) {
if (delta) {
- nr_allocated = alloc_pages_bulk_noprof(gfp,
- interleave_nodes(pol), NULL,
+ nr_allocated = __alloc_pages_bulk_noprof(gfp,
+ aflags, interleave_nodes(pol), NULL,
nr_pages_per_node + 1,
page_array);
delta--;
} else {
- nr_allocated = alloc_pages_bulk_noprof(gfp,
- interleave_nodes(pol), NULL,
+ nr_allocated = __alloc_pages_bulk_noprof(gfp,
+ aflags, interleave_nodes(pol), NULL,
nr_pages_per_node, page_array);
}
@@ -2632,6 +2658,7 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
{
+ unsigned int aflags = mpol_alloc_flags(pol);
struct weighted_interleave_state *state;
struct task_struct *me = current;
unsigned int cpuset_mems_cookie;
@@ -2667,8 +2694,8 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
weight = me->il_weight;
if (weight && node_isset(node, nodes)) {
node_pages = min(rem_pages, weight);
- nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
- page_array);
+ nr_allocated = __alloc_pages_bulk_noprof(gfp, aflags, node,
+ NULL, node_pages, page_array);
page_array += nr_allocated;
total_allocated += nr_allocated;
/* if that's all the pages, no need to interleave */
@@ -2732,8 +2759,8 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
/* node_pages can be 0 if an allocation fails and rounds == 0 */
if (!node_pages)
break;
- nr_allocated = __alloc_pages_bulk(gfp, node, NULL, node_pages,
- page_array);
+ nr_allocated = __alloc_pages_bulk_noprof(gfp, aflags, node,
+ NULL, node_pages, page_array);
page_array += nr_allocated;
total_allocated += nr_allocated;
if (total_allocated == nr_pages)
@@ -2750,17 +2777,19 @@ static unsigned long alloc_pages_bulk_preferred_many(gfp_t gfp, int nid,
struct mempolicy *pol, unsigned long nr_pages,
struct page **page_array)
{
+ unsigned int aflags = mpol_alloc_flags(pol);
gfp_t preferred_gfp;
unsigned long nr_allocated = 0;
preferred_gfp = gfp | __GFP_NOWARN;
preferred_gfp &= ~(__GFP_DIRECT_RECLAIM | __GFP_NOFAIL);
- nr_allocated = alloc_pages_bulk_noprof(preferred_gfp, nid, &pol->nodes,
- nr_pages, page_array);
+ nr_allocated = __alloc_pages_bulk_noprof(preferred_gfp, aflags,
+ nid, &pol->nodes, nr_pages, page_array);
if (nr_allocated < nr_pages)
- nr_allocated += alloc_pages_bulk_noprof(gfp, numa_node_id(), NULL,
+ nr_allocated += __alloc_pages_bulk_noprof(gfp, aflags,
+ numa_node_id(), NULL,
nr_pages - nr_allocated,
page_array + nr_allocated);
return nr_allocated;
@@ -2796,8 +2825,8 @@ unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
nid = numa_node_id();
nodemask = policy_nodemask(gfp, pol, NO_INTERLEAVE_INDEX, &nid);
- return alloc_pages_bulk_noprof(gfp, nid, nodemask,
- nr_pages, page_array);
+ return __alloc_pages_bulk_noprof(gfp, mpol_alloc_flags(pol), nid,
+ nodemask, nr_pages, page_array);
}
int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
The kcore_ram_list bounds its scan at the highest pfn in N_MEMORY.
As a result, private nodes placed above the last N_MEMORY node, which is
the common case for device/CXL memory - is truncated out of /proc/kcore.
Include N_MEMORY_PRIVATE nodes in the scan boundary. /proc/kcore is
root-only (CAP_SYS_RAWIO), so this exposes nothing new - it just lets
a debugger read private-node memory like any other RAM.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
fs/proc/kcore.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/fs/proc/kcore.c b/fs/proc/kcore.c
index 390547b992acd..f67194dc2a9b8 100644
--- a/fs/proc/kcore.c
+++ b/fs/proc/kcore.c
@@ -251,11 +251,14 @@ static int kcore_ram_list(struct list_head *list)
{
int nid, ret;
unsigned long end_pfn;
+ nodemask_t mem;
/* Not initialized....update now */
/* find out "max pfn" */
end_pfn = 0;
- for_each_node_state(nid, N_MEMORY) {
+ /* Include private node memory in the scan */
+ nodes_or(mem, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ for_each_node_mask(nid, mem) {
unsigned long node_end;
node_end = node_end_pfn(nid);
if (end_pfn < node_end)
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Private nodes folios are charged like any other - the node's per-cpu
lruvec counters are increment even for N_MEMORY_PRIVATE.
The memcg-level totals in memory.stat therefore include this memory,
but the per-node views do not:
- mem_cgroup_css_rstat_flush() folds the per-node percpu deltas into the
aggregate only for N_MEMORY nodes, so a private node's lruvec aggregate
is never refreshed.
- memory.numa_stat (v2) and the v1 numa_stat skip private nodes, so they
never emit an Nx= field for one.
So memory.numa_stat does not sum to memory.stat once a cgroup has memory
on a private node.
Include N_MEMORY_PRIVATE in the flush/display process so private-node
memory is folded in and reported per node.
This is pure accounting of memory that already exists and is already
charged, so it is unconditional - regardless of future capabilities.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/memcontrol-v1.c | 8 ++++++--
mm/memcontrol.c | 11 +++++++++--
2 files changed, 15 insertions(+), 4 deletions(-)
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 2dc599484d006..5e464e52dfb9a 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -2132,14 +2132,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)
const struct numa_stat *stat;
int nid;
struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+ nodemask_t reportable;
mem_cgroup_flush_stats(memcg);
+ /* Private nodes hold cgroup memory too, report them. */
+ nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+
for (stat = stats; stat < ARRAY_END(stats); stat++) {
seq_printf(m, "%s=%lu", stat->name,
mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
false));
- for_each_node_state(nid, N_MEMORY)
+ for_each_node_mask(nid, reportable)
seq_printf(m, " N%d=%lu", nid,
mem_cgroup_node_nr_lru_pages(memcg, nid,
stat->lru_mask, false));
@@ -2151,7 +2155,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)
seq_printf(m, "hierarchical_%s=%lu", stat->name,
mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
true));
- for_each_node_state(nid, N_MEMORY)
+ for_each_node_mask(nid, reportable)
seq_printf(m, " N%d=%lu", nid,
mem_cgroup_node_nr_lru_pages(memcg, nid,
stat->lru_mask, true));
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index f0dde52dc9e0e..4d79e238bc57d 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -4496,6 +4496,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
struct mem_cgroup *parent = parent_mem_cgroup(memcg);
struct memcg_vmstats_percpu *statc;
struct aggregate_control ac;
+ nodemask_t reportable;
int nid;
flush_nmi_stats(memcg, parent);
@@ -4524,7 +4525,9 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
};
mem_cgroup_stat_aggregate(&ac);
- for_each_node_state(nid, N_MEMORY) {
+ /* Private nodes must also be accounted or numa_stat is misleading */
+ nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ for_each_node_mask(nid, reportable) {
struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid];
struct lruvec_stats *lstats = pn->lruvec_stats;
struct lruvec_stats *plstats = NULL;
@@ -4947,9 +4950,13 @@ static int memory_numa_stat_show(struct seq_file *m, void *v)
{
int i;
struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+ nodemask_t reportable;
mem_cgroup_flush_stats(memcg);
+ /* Private nodes hold cgroup memory too, report them. */
+ nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+
for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
int nid;
@@ -4957,7 +4964,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v)
continue;
seq_printf(m, "%s", memory_stats[i].name);
- for_each_node_state(nid, N_MEMORY) {
+ for_each_node_mask(nid, reportable) {
u64 size;
struct lruvec *lruvec;
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
numa_maps collects per-node page counts in the page-table walkers
and emits them in show_numa_map. All three filtered by N_MEMORY,
so pages on N_MEMORY_PRIVATE nodes were never gathered/printed.
Accept N_MEMORY_PRIVATE pages in both walkers and emit private nodes
in show_numa_map, so private-node mappings are visible in numa_maps.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
fs/proc/task_mmu.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 817e3e0f91943..e116cd5f157b9 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -3389,7 +3389,8 @@ static struct page *can_gather_numa_stats(pte_t pte, struct vm_area_struct *vma,
return NULL;
nid = page_to_nid(page);
- if (!node_isset(nid, node_states[N_MEMORY]))
+ if (!node_isset(nid, node_states[N_MEMORY]) &&
+ !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
return NULL;
return page;
@@ -3414,7 +3415,8 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
return NULL;
nid = page_to_nid(page);
- if (!node_isset(nid, node_states[N_MEMORY]))
+ if (!node_isset(nid, node_states[N_MEMORY]) &&
+ !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
return NULL;
return page;
@@ -3602,6 +3604,10 @@ static int show_numa_map(struct seq_file *m, void *v)
if (md->node[nid])
seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+ for_each_node_state(nid, N_MEMORY_PRIVATE)
+ if (md->node[nid])
+ seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+
seq_printf(m, " kernelpagesize_kB=%lu", vma_kernel_pagesize(vma) >> 10);
out:
seq_putc(m, '\n');
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 18/36] mm/gup: disallow longterm pin of private node folios
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
FOLL_LONGTERM on a N_MEMORY_PRIVATE node folio is wrong both ways the
GUP longterm path resolves it:
a ZONE_NORMAL folio would be pinned in place
a ZONE_MOVABLE folio would be migrated off first
A private node folio should do neither - the pin should fail in place.
Add folio_longterm_pin_forbidden() (true for any private node folio) and
reject such a pin in check_and_migrate_movable_pages_or_folios() before
any isolation, so nothing is migrated.
The gup-fast path defers a private folio to the slow path via
folio_allows_longterm_pin().
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/gup.c | 27 +++++++++++++++++++++++++--
mm/internal.h | 25 +++++++++++++++++++++++++
2 files changed, 50 insertions(+), 2 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 1d32e9a3dc79c..a7d4de223785c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -547,10 +547,10 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,
/*
* Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
* right zone, so fail and let the caller fall back to the slow
- * path.
+ * path. Fail for private-node folios here so slow path rejects.
*/
if (unlikely((flags & FOLL_LONGTERM) &&
- !folio_is_longterm_pinnable(folio))) {
+ !folio_allows_longterm_pin(folio))) {
folio_put_refs(folio, refs);
return NULL;
}
@@ -2389,12 +2389,35 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list,
return ret;
}
+/*
+ * True if any folio sits on a private node whose folios may not be longterm
+ * pinned. Such folios can neither be pinned nor migrated, so the whole pin
+ * must fail before migration. Checked before any isolation occurs.
+ */
+static bool pofs_has_ltpin_forbidden(struct pages_or_folios *pofs)
+{
+ struct folio *folio;
+ long i = 0;
+
+ for (folio = pofs_get_folio(pofs, i); folio;
+ folio = pofs_next_folio(folio, pofs, &i)) {
+ if (folio_longterm_pin_forbidden(folio))
+ return true;
+ }
+ return false;
+}
+
static long
check_and_migrate_movable_pages_or_folios(struct pages_or_folios *pofs)
{
LIST_HEAD(movable_folio_list);
unsigned long collected;
+ if (pofs_has_ltpin_forbidden(pofs)) {
+ pofs_unpin(pofs);
+ return -EFAULT;
+ }
+
collected = collect_longterm_unpinnable_folios(&movable_folio_list,
pofs);
if (!collected)
diff --git a/mm/internal.h b/mm/internal.h
index cb9f4a8342e32..85c460296cea1 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -110,6 +110,31 @@ static inline bool page_is_private_managed(struct page *page)
return folio_is_private_managed(page_folio(page));
}
+/*
+ * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
+ *
+ * checks folio_is_longterm_pinnable() rules plus private node permissions.
+ * private node permission is checked here to resolve circular header
+ * dependencies in folio_is_longterm_pinnable.
+ */
+static inline bool folio_allows_longterm_pin(struct folio *folio)
+{
+ return folio_is_longterm_pinnable(folio) &&
+ !folio_is_private_node(folio);
+}
+
+/*
+ * folio_longterm_pin_forbidden() - must a longterm pin of this folio fail
+ * outright (neither pinned in place nor migrated off the node)?
+ *
+ * True for any folio on a private node: such memory can be neither pinned
+ * nor migrated, so the pin must be rejected with the folio left in place.
+ */
+static inline bool folio_longterm_pin_forbidden(struct folio *folio)
+{
+ return folio_is_private_node(folio);
+}
+
/*
* Maintains state across a page table move. The operation assumes both source
* and destination VMAs already exist and are specified by the user.
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Private nodes do not support reclaim by default, so prevent all
reclaimers from operating on private nodes.
shrink_node() is the common chokepoint for most reclaimers (direct,
kswapd, proactive, mglru, etc), so it's the best place to filter.
DAMON's proactive pageout bypasses shrink_node() entirely, so we
disable DAMON in a separate commit as its own discrete service.
Other reclaim entry points are already covered - e.g. MGLRU's debugfs
interface rejects nids that are not N_MEMORY (N_MEMORY_PRIVATE is
mutually exclusive), and madvise pageout/cold skips private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/vmscan.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index e26c6931f5fde..86b2334c23b98 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6141,6 +6141,13 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
struct lruvec *target_lruvec;
bool reclaimable = false;
+ /*
+ * Private nodes do not support reclaim by default, filtering here
+ * captures all normal reclaim paths that may attempt eviction.
+ */
+ if (node_is_private(pgdat->node_id))
+ return;
+
if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
memset(&sc->nr, 0, sizeof(sc->nr));
lru_gen_shrink_node(pgdat, sc);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse.
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
A collapse operation causes new THP allocation to occur, and may
migrate memory from one node to another.
Handle this the same as zone_device for now (disallow collapse).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/khugepaged.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 89ce6bcbc376b..fb4378cc17b10 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -700,7 +700,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
goto out;
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+ if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
result = SCAN_PAGE_NULL;
goto out;
}
@@ -1687,7 +1687,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+ if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
result = SCAN_PAGE_NULL;
goto out_unmap;
}
@@ -1944,7 +1944,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
}
page = vm_normal_page(vma, addr, ptent);
- if (WARN_ON_ONCE(page && is_zone_device_page(page)))
+ if (WARN_ON_ONCE(page && page_is_private_managed(page)))
page = NULL;
/*
* Note that uprobe, debugger, or MAP_PRIVATE may change the
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 15/36] mm/ksm: skip KSM for managed-memory folios
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Since private memory is intended for explicit placement, allowing
KSM to operate on it by default would completely defeat the purpose.
Skip KSM merge and scan operations for private node folios using
the same filter location as zone_device folios.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/ksm.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index 47006f494fcb1..dcc5c46a29eac 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -827,7 +827,7 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (!folio_is_zone_device(folio) &&
+ if (!folio_is_private_managed(folio) &&
folio_test_anon(folio)) {
folio_get(folio);
page = fw.page;
@@ -2558,7 +2558,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
goto not_found_unlock;
folio = page_folio(page);
- if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+ if (unlikely(folio_is_private_managed(folio)) ||
+ !folio_test_anon(folio))
goto not_found_unlock;
page += ((addr & (PMD_SIZE - 1)) >> PAGE_SHIFT);
@@ -2582,7 +2583,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
continue;
folio = page_folio(page);
- if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+ if (unlikely(folio_is_private_managed(folio)) ||
+ !folio_test_anon(folio))
continue;
goto found_unlock;
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
DAMON operates on physical address ranges, which can cover private
node memory. Skip private-node folios in both DAMON's migration
and reclaim paths.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/damon/paddr.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index e4f98d67461f5..c741a94319750 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -12,6 +12,7 @@
#include <linux/swap.h>
#include <linux/memory-tiers.h>
#include <linux/mm_inline.h>
+#include <linux/node_private.h>
#include "../internal.h"
#include "ops-common.h"
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
continue;
}
+ /* private node memory is not reclaimable by default */
+ if (folio_is_private_node(folio))
+ goto put_folio;
+
if (damos_pa_filter_out(s, folio))
goto put_folio;
else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
else
*sz_filter_passed += folio_size(folio) / addr_unit;
+ /* private nodes do not support migration by default */
+ if (folio_is_private_node(folio))
+ goto put_folio;
+
if (!folio_isolate_lru(folio))
goto put_folio;
list_add(&folio->lru, &folio_list);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 13/36] mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Skip private node memory the same way zone device memory is
skipped, since private nodes do not presently support migration.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 8e8763f6e5f5b..2b76c57a460c9 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -845,7 +845,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
{
int nid;
- if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
+ if (!folio || folio_is_private_managed(folio) || folio_test_ksm(folio))
return false;
/* Also skip shared copy-on-write folios */
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 12/36] mm/page_alloc: clear private node watermarks and system reserves
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Set private node watermarks to 0 (reclaim isn't supported yet, so they
are pointless), and do not include private nodes in system reserves.
The oom killer can't actually resolve general system pressure by
releasing private node capacity, so including them in system reserve
calculations dilutes DRAM watermarks and causes poor OOM kill behavior.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/page_alloc.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 78755a55b1540..2b08bea2379a9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6555,6 +6555,10 @@ static void calculate_totalreserve_pages(void)
pgdat->totalreserve_pages = 0;
+ /* private nodes have zero watermarks */
+ if (node_is_private(pgdat->node_id))
+ continue;
+
for (i = 0; i < MAX_NR_ZONES; i++) {
struct zone *zone = pgdat->node_zones + i;
long max = 0;
@@ -6647,9 +6651,15 @@ static void __setup_per_zone_wmarks(void)
struct zone *zone;
unsigned long flags;
- /* Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages */
+ /*
+ * Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages
+ *
+ * Private nodes are excluded because they are not in the regular
+ * allocation fallback path - including them dilutes DRAM watermarks.
+ */
for_each_zone(zone) {
- if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE)
+ if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE &&
+ !node_is_private(zone_to_nid(zone)))
lowmem_pages += zone_managed_pages(zone);
}
@@ -6657,6 +6667,16 @@ static void __setup_per_zone_wmarks(void)
u64 tmp;
spin_lock_irqsave(&zone->lock, flags);
+ if (node_is_private(zone_to_nid(zone))) {
+ zone->_watermark[WMARK_MIN] = 0;
+ zone->_watermark[WMARK_LOW] = 0;
+ zone->_watermark[WMARK_HIGH] = 0;
+ zone->_watermark[WMARK_PROMO] = 0;
+ zone->watermark_boost = 0;
+ spin_unlock_irqrestore(&zone->lock, flags);
+ continue;
+ }
+
tmp = (u64)pages_min * zone_managed_pages(zone);
tmp = div64_ul(tmp, lowmem_pages);
if (is_highmem(zone) || zone_idx(zone) == ZONE_MOVABLE) {
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Skip compaction on private nodes. Compaction requires the ability to
migrate pages, which is not yet supported on private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/compaction.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/mm/compaction.c b/mm/compaction.c
index 9f81055a358ed..8c1351cce7bcc 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2462,6 +2462,9 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
+ if (node_is_private(zone_to_nid(zone)))
+ continue;
+
/*
* Do not consider all the reclaimable memory because we do not
* want to trash just for a single high order allocation which
@@ -2853,6 +2856,9 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
+ if (node_is_private(zone_to_nid(zone)))
+ continue;
+
if (prio > MIN_COMPACT_PRIORITY
&& compaction_deferred(zone, order)) {
rc = max_t(enum compact_result, COMPACT_DEFERRED, rc);
@@ -2922,6 +2928,9 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
.proactive_compaction = proactive,
};
+ if (node_is_private(pgdat->node_id))
+ return 0;
+
for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
zone = &pgdat->node_zones[zoneid];
if (!populated_zone(zone))
@@ -3017,6 +3026,9 @@ static ssize_t compact_store(struct device *dev,
{
int nid = dev->id;
+ if (node_is_private(nid))
+ return -EINVAL;
+
if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
/* Flush pending updates to the LRU lists */
lru_add_drain_all();
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Use the same filter locations as zone_device, plus additional filters
for huge pages to avoid madvise operations on private node memory.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/huge_memory.c | 5 +++++
mm/madvise.c | 8 ++++++--
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 5bd8d4f59a7b8..1df91b4e5c2bc 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -37,6 +37,7 @@
#include <linux/page_owner.h>
#include <linux/sched/sysctl.h>
#include <linux/memory-tiers.h>
+#include <linux/node_private.h>
#include <linux/compat.h>
#include <linux/pgalloc.h>
#include <linux/pgalloc_tag.h>
@@ -2338,6 +2339,10 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
}
folio = pmd_folio(orig_pmd);
+
+ if (folio_is_private_node(folio))
+ goto out;
+
/*
* If other processes are mapping this folio, we couldn't discard
* the folio unless they all do MADV_FREE so let's skip the folio.
diff --git a/mm/madvise.c b/mm/madvise.c
index 07a21ca31bad4..29f35a23919a0 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -32,6 +32,7 @@
#include <linux/leafops.h>
#include <linux/shmem_fs.h>
#include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
#include <asm/tlb.h>
@@ -395,6 +396,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
folio = pmd_folio(orig_pmd);
+ if (folio_is_private_node(folio))
+ goto huge_unlock;
+
/* Do not interfere with other mappings of this folio */
if (folio_maybe_mapped_shared(folio))
goto huge_unlock;
@@ -474,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
continue;
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
/*
@@ -703,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Use the same filter locations as zone_device folios to disallow
userland driven migration requests for private node memory.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/migrate.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index a8a86141dbb54..d20674c07b947 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2268,7 +2268,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
return -EFAULT;
- if (folio_is_zone_device(folio))
+ if (folio_is_private_managed(folio))
return -ENOENT;
if (folio_nid(folio) == node)
@@ -2477,7 +2477,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
if (folio) {
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
err = -EFAULT;
- else if (folio_is_zone_device(folio))
+ else if (folio_is_private_managed(folio))
err = -ENOENT;
else
err = folio_nid(folio);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Private nodes are already kept out of policy nodemasks (only N_MEMORY
nodes are allowed), but an mbind(MPOL_MF_MOVE) walk can still encounter a
private-node folio in the range. Skip such folios so mempolicy-driven
migration never moves private-node memory.
We already do this for ZONE_DEVICE folios - use the same filter points
for ZONE_DEVICE (with an additional point for large pages).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 5720f7f54d942..8e8763f6e5f5b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -111,6 +111,7 @@
#include <linux/mmu_notifier.h>
#include <linux/printk.h>
#include <linux/leafops.h>
+#include <linux/node_private.h>
#include <linux/gcd.h>
#include <asm/tlbflush.h>
@@ -668,6 +669,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
}
if (!queue_folio_required(folio, qp))
return;
+ if (folio_is_private_node(folio))
+ return;
if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma) ||
!migrate_folio_add(folio, qp->pagelist, qp->flags))
@@ -722,7 +725,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
continue;
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
if (folio_test_large(folio) && max_nr != 1)
nr = folio_pte_batch(folio, pte, ptent, max_nr);
@@ -797,6 +800,8 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
folio = pfn_folio(pte_pfn(ptep));
if (!queue_folio_required(folio, qp))
goto unlock;
+ if (folio_is_private_node(folio))
+ goto unlock;
if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma)) {
qp->nr_failed++;
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 07/36] mm/memory_hotplug: disallow migration-driven private node hotunplug
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
By default, device-backed private node memory may not be safe to
migrate without coordination.
Do not attempt to hotunplug private node memory, instead fail the
hotunplug attempt, and require the owner to drain the folios
safely before allowing hotunplug to proceed.
Unlike transient / longterm pins causing failures, this is not a
transient condition - so we can break out of the hotunplug attempt
entirely instead of spinning forever on a failed migration attempt.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/memory_hotplug.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078ad..2b9c0830821e6 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -34,6 +34,7 @@
#include <linux/memblock.h>
#include <linux/compaction.h>
#include <linux/rmap.h>
+#include <linux/node_private.h>
#include <linux/module.h>
#include <linux/node.h>
@@ -1843,11 +1844,12 @@ static int scan_movable_pages(unsigned long start, unsigned long end,
return 0;
}
-static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
+static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
{
struct folio *folio;
unsigned long pfn;
LIST_HEAD(source);
+ int err = 0;
static DEFINE_RATELIMIT_STATE(migrate_rs, DEFAULT_RATELIMIT_INTERVAL,
DEFAULT_RATELIMIT_BURST);
@@ -1884,6 +1886,15 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
goto put_folio;
}
+ /* Private node folios cannot migrate, fail outright */
+ if (folio_is_private_node(folio)) {
+ pr_info_ratelimited("memory offline refused: node %d pfn %lx\n",
+ folio_nid(folio), pfn);
+ folio_put(folio);
+ err = -EBUSY;
+ break;
+ }
+
if (!isolate_folio_to_list(folio, &source)) {
if (__ratelimit(&migrate_rs)) {
pr_warn("failed to isolate pfn %lx\n",
@@ -1931,6 +1942,7 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
putback_movable_pages(&source);
}
}
+ return err;
}
static int __init cmdline_parse_movable_node(char *p)
@@ -2067,10 +2079,11 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
ret = scan_movable_pages(pfn, end_pfn, &pfn);
if (!ret) {
/*
- * TODO: fatal migration failures should bail
- * out
+ * A fatal migration failure (e.g. a folio on a
+ * non-migratable private node) bails out; transient
+ * failures leave ret zero and are retried below.
*/
- do_migrate_range(pfn, end_pfn);
+ ret = do_migrate_range(pfn, end_pfn);
}
} while (!ret);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open)
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
N_MEMORY_PRIVATE node access is gated by per-node capabilities and
the zonelist. Isolating by cpuset offers no functionality and
creates issues on rebind when a cpuset loses access to that node
(in particular: it generates migrations that may not be supported).
Do not partition private nodes via cpuset.mem, instead treat them
as globally accessible resources. Do not engage in silent background
operations on these nodes due to cpuset.mem membership changing.
The result: cpuset operations checking for node validity always
allow N_MEMORY_PRIVATE nodes.
On hot-unplug, we still need to rebind memory policies if the
private node has left N_MEMORY_PRIVATE (i.e. no more memory).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
kernel/cgroup/cpuset.c | 26 ++++++++++++++++++++++----
mm/memcontrol.c | 2 ++
2 files changed, 24 insertions(+), 4 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index dfd0f827e3b92..05468f95c10bd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -26,6 +26,7 @@
#include <linux/mempolicy.h>
#include <linux/mm.h>
#include <linux/memory.h>
+#include <linux/node_private.h>
#include <linux/rcupdate.h>
#include <linux/sched.h>
#include <linux/sched/deadline.h>
@@ -3867,7 +3868,8 @@ static void cpuset_handle_hotplug(void)
static DECLARE_WORK(hk_sd_work, hk_sd_workfn);
static cpumask_t new_cpus;
static nodemask_t new_mems;
- bool cpus_updated, mems_updated;
+ static nodemask_t prev_priv_mems;
+ bool cpus_updated, mems_updated, priv_shrank;
bool on_dfl = is_in_v2_mode();
struct tmpmasks tmp, *ptmp = NULL;
@@ -3890,6 +3892,14 @@ static void cpuset_handle_hotplug(void)
!cpumask_empty(subpartitions_cpus);
mems_updated = !nodes_equal(top_cpuset.effective_mems, new_mems);
+ /*
+ * Private nodes are not partitioned by cpuset, but if one leaves
+ * N_MEMORY_PRIVATE we still need to run mpol_rebind_* to clean up
+ * mempolicies that are binding them.
+ */
+ priv_shrank = !nodes_subset(prev_priv_mems, node_states[N_MEMORY_PRIVATE]);
+ prev_priv_mems = node_states[N_MEMORY_PRIVATE];
+
/* For v1, synchronize cpus_allowed to cpu_active_mask */
if (cpus_updated) {
cpuset_force_rebuild();
@@ -3922,9 +3932,12 @@ static void cpuset_handle_hotplug(void)
top_cpuset.mems_allowed = new_mems;
top_cpuset.effective_mems = new_mems;
spin_unlock_irq(&callback_lock);
- cpuset_update_tasks_nodemask(&top_cpuset);
}
+ /* Rebind task mempolicies if any memory node changed state */
+ if (mems_updated || priv_shrank)
+ cpuset_update_tasks_nodemask(&top_cpuset);
+
mutex_unlock(&cpuset_mutex);
/* if cpus or mems changed, we need to propagate to descendants */
@@ -4155,11 +4168,13 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
* cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
* @nodemask: the nodemask to be checked
*
- * Are any of the nodes in the nodemask allowed in current->mems_allowed?
+ * Are any of the nodes in the nodemask usable? N_MEMORY nodes must be in
+ * current->mems_allowed, while N_MEMORY_PRIVATE nodes are always valid.
*/
int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
{
- return nodes_intersects(*nodemask, current->mems_allowed);
+ return nodes_intersects(*nodemask, current->mems_allowed) ||
+ nodes_intersects(*nodemask, node_states[N_MEMORY_PRIVATE]);
}
/*
@@ -4223,6 +4238,9 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)
if (in_interrupt())
return true;
+ /* N_MEMORY_PRIVATE nodes are not partitioned by cpusets.mems */
+ if (node_is_private(node))
+ return true;
if (node_isset(node, current->mems_allowed))
return true;
/*
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 8319ad8c5c23a..f0dde52dc9e0e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6069,6 +6069,8 @@ void mem_cgroup_node_filter_allowed(struct mem_cgroup *memcg, nodemask_t *mask)
* mask is acceptable.
*/
cpuset_nodes_allowed(memcg->css.cgroup, &allowed);
+ /* N_MEMORY_PRIVATE nodes are not partitioned by cpuset, include them */
+ nodes_or(allowed, allowed, node_states[N_MEMORY_PRIVATE]);
nodes_and(*mask, *mask, allowed);
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
N_MEMORY_PRIVATE nodes must be invisible to ordinary allocation,
no allocations may reach one unless explicitly called for.
Make this isolation structural in the zonelists: N_MEMORY_PRIVATE
nodes are excluded from FALLBACK and NOFALLBACK entirely, making
them effectively invisible to all normal callers of page_alloc.
Add ZONELIST_PRIVATE, which spans (N_MEMORY | N_MEMORY_PRIVATE).
When !CONFIG_NUMA, it aliases ZONELIST_FALLBACK and compiles out.
Add ZONELIST_PRIVATE_NOFALLBACK as the __GFP_THISNODE target for
private node allocations.
Zonelist selection rides the allocator's alloc_flags. Add
ALLOC_ZONELIST_PRIVATE and resolve it in select_zonelist(), which
prepare_alloc_pages() applies from ac->alloc_flags; the default
(ALLOC_DEFAULT) keeps node_zonelist()'s gfp-based selection.
Add explicit private-node alloc() functions to prevent open-coding
(both ride ALLOC_ZONELIST_PRIVATE through the alloc_flags-carrying
entry points):
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()
alloc_contig_range adds an explicit node_is_private() guard because
it dervices its target zones from PFNs rather than zonelists. All
in-tree callers use N_MEMORY ranges, but the check prevents future
callers from violating node isolation.
Important note: By design this zonelist isolates N_MEMORY_PRIVATE
from unintentional allocations, but does NOT isolate private-node
allocations from falling back to non-private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/gfp.h | 11 ++++++++
include/linux/mmzone.h | 11 +++++++-
mm/mm_init.c | 2 +-
mm/page_alloc.c | 64 ++++++++++++++++++++++++++++++++++++++++--
mm/page_alloc.h | 26 +++++++++++++++++
5 files changed, 109 insertions(+), 5 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index a327e58c313f8..f3e867de744f6 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -204,6 +204,17 @@ static inline void arch_free_page(struct page *page, int order) { }
static inline void arch_alloc_page(struct page *page, int order) { }
#endif
+/* Allocate from an N_MEMORY_PRIVATE node (selects the private zonelist). */
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid);
+#define alloc_pages_node_private(...) \
+ alloc_hooks(alloc_pages_node_private_noprof(__VA_ARGS__))
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid);
+#define folio_alloc_node_private(...) \
+ alloc_hooks(folio_alloc_node_private_noprof(__VA_ARGS__))
+
unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
struct page **page_array);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 9815e48c03b97..654c5111f7cec 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1392,13 +1392,22 @@ enum {
#ifdef CONFIG_NUMA
/*
* The NUMA zonelists are doubled because we need zonelists that
- * restrict the allocations to a single node for __GFP_THISNODE.
+ * restrict the allocations to a single node for __GFP_THISNODE
+ * and N_MEMORY_PRIVATE nodes (isolated from default lists).
*/
ZONELIST_NOFALLBACK, /* zonelist without fallback (__GFP_THISNODE) */
+ ZONELIST_PRIVATE, /* N_MEMORY_PRIVATE access, falls back to DRAM */
+ ZONELIST_PRIVATE_NOFALLBACK, /* N_MEMORY_PRIVATE access, __GFP_THISNODE */
#endif
MAX_ZONELISTS
};
+#ifndef CONFIG_NUMA
+/* Without NUMA only ZONELIST_FALLBACK exists so everything collapses there */
+#define ZONELIST_PRIVATE ZONELIST_FALLBACK
+#define ZONELIST_PRIVATE_NOFALLBACK ZONELIST_FALLBACK
+#endif
+
/*
* This struct contains information about a zone in a zonelist. It is stored
* here to avoid dereferences into large structures and lookups of tables
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 711f821f7b3c7..adb50647d29ca 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2701,7 +2701,7 @@ void __init mm_core_init(void)
init_zero_page_pfn();
/* Initializations relying on SMP setup */
- BUILD_BUG_ON(MAX_ZONELISTS > 2);
+ BUILD_BUG_ON(MAX_ZONELISTS > 4);
build_all_zonelists(NULL);
page_alloc_init_cpuhp();
alloc_tag_sec_init();
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index aa35bbe5d4c3e..78755a55b1540 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5040,7 +5040,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
unsigned int *alloc_flags)
{
ac->highest_zoneidx = gfp_zone(gfp_mask);
- ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
+ ac->zonelist = select_zonelist(preferred_nid, gfp_mask, ac->alloc_flags);
ac->nodemask = nodemask;
ac->migratetype = gfp_migratetype(gfp_mask);
@@ -5346,7 +5346,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
unsigned int fastpath_alloc_flags = alloc_flags;
/* Other flags could be supported later if needed. */
- if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
+ if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG | ALLOC_ZONELIST_PRIVATE)))
return NULL;
if (!alloc_order_allowed(gfp, order, alloc_flags))
@@ -5457,6 +5457,22 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
}
EXPORT_SYMBOL(__folio_alloc_node_noprof);
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid)
+{
+ return __alloc_pages_noprof(gfp, order, nid, NULL,
+ ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(alloc_pages_node_private_noprof);
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid)
+{
+ return __folio_alloc_noprof(gfp, order, nid, NULL,
+ ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(folio_alloc_node_private_noprof);
+
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
* address cannot represent highmem pages. Use alloc_pages and then kmap if
@@ -5854,9 +5870,21 @@ static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
static void build_thisnode_zonelists(pg_data_t *pgdat)
{
struct zoneref *zonerefs;
- int nr_zones;
+ int nr_zones = 0;
+ /*
+ * NOFALLBACK: the node's own zones. EMPTY for private nodes so a
+ * stray __GFP_THISNODE allocation can't violate isolation.
+ */
zonerefs = pgdat->node_zonelists[ZONELIST_NOFALLBACK]._zonerefs;
+ if (!node_is_private(pgdat->node_id))
+ nr_zones = build_zonerefs_node(pgdat, zonerefs);
+ zonerefs += nr_zones;
+ zonerefs->zone = NULL;
+ zonerefs->zone_idx = 0;
+
+ /* PRIVATE_NOFALLBACK: the node's own zones, private nodes included. */
+ zonerefs = pgdat->node_zonelists[ZONELIST_PRIVATE_NOFALLBACK]._zonerefs;
nr_zones = build_zonerefs_node(pgdat, zonerefs);
zonerefs += nr_zones;
zonerefs->zone = NULL;
@@ -5899,11 +5927,28 @@ static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
static void build_zonelists(pg_data_t *pgdat)
{
static int node_order[MAX_NUMNODES];
+ nodemask_t tier_nodes;
int local_node = pgdat->node_id;
int node, nr_nodes = 0;
memset(node_order, 0, sizeof(node_order));
+ /*
+ * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
+ *
+ * FALLBACK: Allocation order for all nodes. Private nodes have lists
+ * but never appear as an entry in any list. Allocations
+ * targeting a private node w/ FALLBACK land on N_MEMORY.
+ *
+ * NOFALLBACK: A list for each node containing only itself.
+ * Allocations targeting a private node w/ NOFALLBACK
+ * will always fail (their NOFALLBACK is empty)
+ *
+ * PRIVATE: (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
+ * private nodes, and falls back to normal memory unless
+ * __GFP_THISNODE otherwise constrains it.
+ */
+
build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
true, node_order, &nr_nodes);
build_thisnode_zonelists(pgdat);
@@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
for (node = 0; node < nr_nodes; node++)
pr_cont("%d ", node_order[node]);
pr_cont("\n");
+
+ nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ build_node_zonelist(pgdat, &tier_nodes, ZONELIST_PRIVATE, false,
+ node_order, &nr_nodes);
}
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
@@ -7282,6 +7331,15 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
if (__alloc_contig_verify_gfp_mask(gfp_mask, (gfp_t *)&cc.gfp_mask))
return -EINVAL;
+ /*
+ * Private nodes are kept unreachable via the zonelists, but
+ * alloc_contig works directly on a zone derived from the caller's
+ * pfn range, bypassing that. Reject this operation explicitly
+ * rather than silently carving memory out of an isolated node.
+ */
+ if (unlikely(node_is_private(zone_to_nid(cc.zone))))
+ return -EBUSY;
+
/*
* What we do here is we mark all pageblocks in range as
* MIGRATE_ISOLATE. Because pageblock and max order pages may
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index 23fc79ce97b60..741448d83ce77 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -57,6 +57,9 @@
*/
#define ALLOC_NO_CODETAG 0x1000
+/* Select the N_MEMORY_PRIVATE zonelist family (see select_zonelist()). */
+#define ALLOC_ZONELIST_PRIVATE 0x2000
+
/* Flags that allow allocations below the min watermark. */
#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
@@ -95,6 +98,29 @@ struct alloc_context {
unsigned int alloc_flags;
};
+#ifdef CONFIG_NUMA
+static_assert(ZONELIST_PRIVATE + 1 == ZONELIST_PRIVATE_NOFALLBACK);
+#endif
+
+/*
+ * select_zonelist - resolve the zonelist for an allocation
+ *
+ * The default matches node_zonelist(); ALLOC_ZONELIST_PRIVATE selects the
+ * private-node family so an allocation may reach an N_MEMORY_PRIVATE node.
+ */
+static inline struct zonelist *
+select_zonelist(int nid, gfp_t gfp, unsigned int alloc_flags)
+{
+#ifdef CONFIG_NUMA
+ if (alloc_flags & ALLOC_ZONELIST_PRIVATE) {
+ int idx = ZONELIST_PRIVATE + !!(gfp & __GFP_THISNODE);
+
+ return &NODE_DATA(nid)->node_zonelists[idx];
+ }
+#endif
+ return node_zonelist(nid, gfp);
+}
+
/*
* This function returns the order of a free page in the buddy system. In
* general, page_zone(page)->lock must be held by the caller to prevent the
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Some devices want to hotplug their memory onto a node, but not have it
exposed as "general purpose". The intent is to re-use portions of mm/
services for management without exposing it for general consumption.
N_MEMORY nodes are intended to contain general System RAM, so they
are unsuited for this purpose.
Create N_MEMORY_PRIVATE for memory nodes whose memory is not intended
for general consumption.
N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive states.
This commit adds basic infrastructure for N_MEMORY_PRIVATE nodes:
- struct node_private:
Per-node container stored in NODE_DATA(nid) storing the owner.
- folio/page_is_private_node():
true if it resides on a private node
- folio/page_is_private_managed():
common predicate for checking private node or zone device
Used to combine common filtering locations.
- Registration API: node_private_register()/unregister()
- sysfs attribute exposing N_MEMORY_PRIVATE node state.
- fix node_state() and helpers to return false for N_MEMORY_PRIVATE
when !CONFIG_NUMA (mutually exclusive w/ N_MEMORY)
Signed-off-by: Gregory Price <gourry@gourry.net>
---
Documentation/ABI/stable/sysfs-devices-node | 10 ++
drivers/base/node.c | 113 ++++++++++++++++++++
include/linux/mmzone.h | 15 +++
include/linux/node_private.h | 82 ++++++++++++++
include/linux/nodemask.h | 7 +-
mm/internal.h | 12 +++
6 files changed, 236 insertions(+), 3 deletions(-)
create mode 100644 include/linux/node_private.h
diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node
index 2d0e023f22a71..8a36fbc3d4028 100644
--- a/Documentation/ABI/stable/sysfs-devices-node
+++ b/Documentation/ABI/stable/sysfs-devices-node
@@ -29,6 +29,16 @@ Description:
Nodes that have regular or high memory.
Depends on CONFIG_HIGHMEM.
+What: /sys/devices/system/node/has_private_memory
+Contact: Linux Memory Management list <linux-mm@kvack.org>
+Description:
+ Nodes that have private (N_MEMORY_PRIVATE) memory: memory
+ hotplugged by a driver onto a CPU-less node and isolated from
+ the page allocator's normal and fallback zonelists. A node is
+ listed here for as long as it holds private memory; it is never
+ listed in has_memory at the same time (the two states are
+ mutually exclusive). See Documentation/mm/numa_private_nodes.rst.
+
What: /sys/devices/system/node/nodeX
Date: October 2002
Contact: Linux Memory Management list <linux-mm@kvack.org>
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 3da91929ad4e3..94cd51f51b7e8 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -22,6 +22,7 @@
#include <linux/swap.h>
#include <linux/slab.h>
#include <linux/memblock.h>
+#include <linux/node_private.h>
static const struct bus_type node_subsys = {
.name = "node",
@@ -868,6 +869,116 @@ void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn,
(void *)&nid, register_mem_block_under_node_hotplug);
return;
}
+
+static DEFINE_MUTEX(node_private_lock);
+
+/**
+ * node_private_register - Register a private node
+ * @nid: Node identifier
+ * @np: The node_private structure (driver-allocated, driver-owned)
+ *
+ * Register an owner for a private node. If an owner is already registered
+ * (different np), return -EBUSY.
+ *
+ * Re-registration with the same np is allowed.
+ *
+ * The caller owns the node_private memory and must ensure it remains valid
+ * until after node_private_unregister() returns.
+ *
+ * Returns 0 on success, negative errno on failure.
+ */
+int node_private_register(int nid, struct node_private *np)
+{
+ struct node_private *existing;
+ pg_data_t *pgdat;
+ int ret = 0;
+
+ if (!np || !node_possible(nid))
+ return -EINVAL;
+
+ mutex_lock(&node_private_lock);
+ mem_hotplug_begin();
+
+ /* N_MEMORY_PRIVATE and N_MEMORY are mutually exclusive */
+ if (node_state(nid, N_MEMORY)) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ pgdat = NODE_DATA(nid);
+ existing = rcu_dereference_protected(pgdat->node_private,
+ lockdep_is_held(&node_private_lock));
+
+ /* If it exists, restrict to single owner */
+ if (existing) {
+ if (existing != np)
+ ret = -EBUSY;
+ goto out;
+ }
+
+ rcu_assign_pointer(pgdat->node_private, np);
+out:
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(node_private_register);
+
+/**
+ * node_private_unregister - Unregister a private node
+ * @nid: Node identifier
+ *
+ * Unregister the driver from a private node.
+ *
+ * Only succeeds if all memory has been offlined (N_MEMORY_PRIVATE cleared).
+ *
+ * N_MEMORY_PRIVATE state is cleared by offline_pages() when the last
+ * memory is offlined, not by this function.
+ *
+ * After this returns, the node_private pointer is no longer visible and
+ * an RCU grace period has elapsed, so the driver may free its context.
+ *
+ * Return: 0 if unregistered, -EBUSY if N_MEMORY_PRIVATE is still set.
+ */
+int node_private_unregister(int nid)
+{
+ struct node_private *np;
+ pg_data_t *pgdat;
+
+ if (!node_possible(nid))
+ return 0;
+
+ mutex_lock(&node_private_lock);
+ mem_hotplug_begin();
+
+ pgdat = NODE_DATA(nid);
+ np = rcu_dereference_protected(pgdat->node_private,
+ lockdep_is_held(&node_private_lock));
+ if (!np) {
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return 0;
+ }
+
+ /*
+ * Only unregister if N_MEMORY_PRIVATE is cleared (only occurs when
+ * the last memory block is offlined in offline_pages())
+ */
+ if (node_is_private(nid)) {
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return -EBUSY;
+ }
+
+ rcu_assign_pointer(pgdat->node_private, NULL);
+
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+
+ synchronize_rcu();
+ return 0;
+}
+EXPORT_SYMBOL_GPL(node_private_unregister);
#endif /* CONFIG_MEMORY_HOTPLUG */
/**
@@ -966,6 +1077,7 @@ static struct node_attr node_state_attr[] = {
[N_HIGH_MEMORY] = _NODE_ATTR(has_high_memory, N_HIGH_MEMORY),
#endif
[N_MEMORY] = _NODE_ATTR(has_memory, N_MEMORY),
+ [N_MEMORY_PRIVATE] = _NODE_ATTR(has_private_memory, N_MEMORY_PRIVATE),
[N_CPU] = _NODE_ATTR(has_cpu, N_CPU),
[N_GENERIC_INITIATOR] = _NODE_ATTR(has_generic_initiator,
N_GENERIC_INITIATOR),
@@ -979,6 +1091,7 @@ static struct attribute *node_state_attrs[] = {
&node_state_attr[N_HIGH_MEMORY].attr.attr,
#endif
&node_state_attr[N_MEMORY].attr.attr,
+ &node_state_attr[N_MEMORY_PRIVATE].attr.attr,
&node_state_attr[N_CPU].attr.attr,
&node_state_attr[N_GENERIC_INITIATOR].attr.attr,
NULL
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0507193b3ae34..9815e48c03b97 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -26,6 +26,8 @@
#include <linux/sizes.h>
#include <asm/page.h>
+struct node_private;
+
/* Free memory management - zoned buddy allocator. */
#ifndef CONFIG_ARCH_FORCE_MAX_ORDER
#define MAX_PAGE_ORDER 10
@@ -1596,12 +1598,25 @@ typedef struct pglist_data {
atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
#ifdef CONFIG_NUMA
struct memory_tier __rcu *memtier;
+ struct node_private __rcu *node_private;
#endif
#ifdef CONFIG_MEMORY_FAILURE
struct memory_failure_stats mf_stats;
#endif
} pg_data_t;
+#ifdef CONFIG_NUMA
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+ return !!pgdat->node_private;
+}
+#else
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+ return false;
+}
+#endif
+
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
#define node_spanned_pages(nid) (NODE_DATA(nid)->node_spanned_pages)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
new file mode 100644
index 0000000000000..475496c84249f
--- /dev/null
+++ b/include/linux/node_private.h
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_NODE_PRIVATE_H
+#define _LINUX_NODE_PRIVATE_H
+
+#include <linux/mm.h>
+#include <linux/nodemask.h>
+
+struct page;
+
+/**
+ * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
+ *
+ * Allocated by the driver and passed to node_private_register().
+ * The driver owns the memory and must ensure it remains valid until after
+ * node_private_unregister() returns.
+ *
+ * @owner: Opaque driver identifier
+ * @caps: NODE_PRIVATE_CAP_* service opt-ins for the node (zero by default;
+ * individual capabilities are defined and consumed by later changes)
+ */
+struct node_private {
+ void *owner;
+ unsigned long caps;
+};
+
+#ifdef CONFIG_NUMA
+#include <linux/mmzone.h>
+
+static inline bool folio_is_private_node(struct folio *folio)
+{
+ return node_state(folio_nid(folio), N_MEMORY_PRIVATE);
+}
+
+static inline bool page_is_private_node(struct page *page)
+{
+ return node_state(page_to_nid(page), N_MEMORY_PRIVATE);
+}
+
+static inline bool node_is_private(int nid)
+{
+ return node_state(nid, N_MEMORY_PRIVATE);
+}
+
+#else /* !CONFIG_NUMA */
+
+static inline bool folio_is_private_node(struct folio *folio)
+{
+ return false;
+}
+
+static inline bool page_is_private_node(struct page *page)
+{
+ return false;
+}
+
+static inline bool node_is_private(int nid)
+{
+ return false;
+}
+
+#endif /* CONFIG_NUMA */
+
+#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
+
+int node_private_register(int nid, struct node_private *np);
+int node_private_unregister(int nid);
+
+#else /* !CONFIG_NUMA || !CONFIG_MEMORY_HOTPLUG */
+
+static inline int node_private_register(int nid, struct node_private *np)
+{
+ return -ENODEV;
+}
+
+static inline int node_private_unregister(int nid)
+{
+ return 0;
+}
+
+#endif /* CONFIG_NUMA && CONFIG_MEMORY_HOTPLUG */
+
+#endif /* _LINUX_NODE_PRIVATE_H */
diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h
index b842aa5255464..ba3e6c570a112 100644
--- a/include/linux/nodemask.h
+++ b/include/linux/nodemask.h
@@ -391,6 +391,7 @@ enum node_states {
N_HIGH_MEMORY = N_NORMAL_MEMORY,
#endif
N_MEMORY, /* The node has memory(regular, high, movable) */
+ N_MEMORY_PRIVATE, /* The node's memory is private */
N_CPU, /* The node has one or more cpus */
N_GENERIC_INITIATOR, /* The node has one or more Generic Initiators */
NR_NODE_STATES
@@ -457,7 +458,7 @@ static __always_inline void node_set_offline(int nid)
static __always_inline int node_state(int node, enum node_states state)
{
- return node == 0;
+ return node == 0 && state != N_MEMORY_PRIVATE;
}
static __always_inline void node_set_state(int node, enum node_states state)
@@ -470,11 +471,11 @@ static __always_inline void node_clear_state(int node, enum node_states state)
static __always_inline int num_node_state(enum node_states state)
{
- return 1;
+ return state == N_MEMORY_PRIVATE ? 0 : 1;
}
#define for_each_node_state(node, __state) \
- for ( (node) = 0; (node) == 0; (node) = 1)
+ for ((node) = 0; (node) == 0 && (__state) != N_MEMORY_PRIVATE; (node) = 1)
#define first_online_node 0
#define first_memory_node 0
diff --git a/mm/internal.h b/mm/internal.h
index 96d78a7778e88..cb9f4a8342e32 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/mm_inline.h>
#include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
#include <linux/pagemap.h>
#include <linux/pagewalk.h>
#include <linux/rmap.h>
@@ -98,6 +99,17 @@ extern int sysctl_min_unmapped_ratio;
extern int sysctl_min_slab_ratio;
#endif
+/* folio_is_private_managed() - folio has special mm management rules */
+static inline bool folio_is_private_managed(struct folio *folio)
+{
+ return folio_is_zone_device(folio) || folio_is_private_node(folio);
+}
+
+static inline bool page_is_private_managed(struct page *page)
+{
+ return folio_is_private_managed(page_folio(page));
+}
+
/*
* Maintains state across a page table move. The operation assumes both source
* and destination VMAs already exist and are specified by the user.
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
__alloc_pages_noprof() takes an explicit alloc_flags, but the bulk and
folio entry points do not, so callers cannot select allocator behaviour
(e.g. an alternate zonelist) through them.
Thread alloc_flags through both, matching __alloc_pages_noprof(), and
keep the flag-carrying primitives mm-internal (page_alloc.h) so the
public gfp.h wrappers stay flag-free:
- add __alloc_pages_bulk_noprof(gfp, alloc_flags, ...) in page_alloc.h
alloc_pages_bulk_noprof() becomes a wrapper passing ALLOC_DEFAULT
- give __folio_alloc_noprof() an alloc_flags parameter and moves
__folio_alloc_node_noprof() moves into page_alloc.c
__folio_alloc_noprof() is no longer exported
No functional change: every caller passes ALLOC_DEFAULT.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/gfp.h | 13 +------------
mm/khugepaged.c | 2 +-
mm/migrate.c | 2 +-
mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++---------------
mm/page_alloc.h | 8 ++++++++
5 files changed, 42 insertions(+), 29 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 872bc53f32ec8..a327e58c313f8 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }
static inline void arch_alloc_page(struct page *page, int order) { }
#endif
-struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
- nodemask_t *nodemask);
-#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
-
unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
struct page **page_array);
@@ -252,14 +248,7 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)
dump_stack();
}
-static inline
-struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
-{
- warn_if_node_offline(nid, gfp);
-
- return __folio_alloc_noprof(gfp, order, nid, NULL);
-}
-
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid);
#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__))
/*
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 27e8f3077e80f..89ce6bcbc376b 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1243,7 +1243,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
int node = collapse_find_target_node(cc);
struct folio *folio;
- folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
+ folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
if (!folio) {
*foliop = NULL;
if (is_pmd_order(order))
diff --git a/mm/migrate.c b/mm/migrate.c
index 60355be9f652c..a8a86141dbb54 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2231,7 +2231,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
gfp_mask |= __GFP_HIGHMEM;
- return __folio_alloc(gfp_mask, order, nid, mtc->nmask);
+ return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
}
#ifdef CONFIG_NUMA_MIGRATION
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4c1ef84c02a3d..aa35bbe5d4c3e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5103,8 +5103,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
* @page_array were set to %NULL on entry, the slots from 0 to the return value
* - 1 will be filled.
*/
-unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
- nodemask_t *nodemask, int nr_pages,
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+ int preferred_nid, nodemask_t *nodemask, int nr_pages,
struct page **page_array)
{
struct page *page;
@@ -5112,8 +5112,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct zoneref *z;
struct per_cpu_pages *pcp;
struct list_head *pcp_list;
- struct alloc_context ac;
- unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ struct alloc_context ac = { .alloc_flags = alloc_flags };
+ unsigned int fastpath_alloc_flags = alloc_flags | ALLOC_WMARK_LOW;
int nr_populated = 0, nr_account = 0;
/*
@@ -5153,7 +5153,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
/* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
gfp &= gfp_allowed_mask;
- if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
+ if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &fastpath_alloc_flags))
goto out;
/* Find an allowed local zone that meets the low watermark. */
@@ -5161,7 +5161,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
for_next_zone_zonelist_nodemask(zone, z, ac.highest_zoneidx, ac.nodemask) {
unsigned long mark;
- if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
+ if (cpusets_enabled() && (fastpath_alloc_flags & ALLOC_CPUSET) &&
!__cpuset_zone_allowed(zone, gfp)) {
continue;
}
@@ -5171,16 +5171,17 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
goto failed;
}
- cond_accept_memory(zone, 0, alloc_flags);
+ cond_accept_memory(zone, 0, fastpath_alloc_flags);
retry_this_zone:
- mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages - nr_populated;
+ mark = wmark_pages(zone, fastpath_alloc_flags & ALLOC_WMARK_MASK) +
+ nr_pages - nr_populated;
if (zone_watermark_fast(zone, 0, mark,
zonelist_zone_idx(ac.preferred_zoneref),
- alloc_flags, gfp)) {
+ fastpath_alloc_flags, gfp)) {
break;
}
- if (cond_accept_memory(zone, 0, alloc_flags))
+ if (cond_accept_memory(zone, 0, fastpath_alloc_flags))
goto retry_this_zone;
/* Try again if zone has deferred pages */
@@ -5212,7 +5213,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
continue;
}
- page = __rmqueue_pcplist(zone, 0, ac.migratetype, alloc_flags,
+ page = __rmqueue_pcplist(zone, 0, ac.migratetype, fastpath_alloc_flags,
pcp, pcp_list);
if (unlikely(!page)) {
/* Try and allocate at least one page */
@@ -5238,11 +5239,19 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
return nr_populated;
failed:
- page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, ALLOC_DEFAULT);
+ page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, alloc_flags);
if (page)
page_array[nr_populated++] = page;
goto out;
}
+
+unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array)
+{
+ return __alloc_pages_bulk_noprof(gfp, ALLOC_DEFAULT, preferred_nid,
+ nodemask, nr_pages, page_array);
+}
EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);
/*
@@ -5433,13 +5442,20 @@ struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order
EXPORT_SYMBOL(alloc_pages_node_noprof);
struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
- nodemask_t *nodemask)
+ nodemask_t *nodemask, unsigned int alloc_flags)
{
struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order,
- preferred_nid, nodemask, ALLOC_DEFAULT);
+ preferred_nid, nodemask, alloc_flags);
return page_rmappable_folio(page);
}
-EXPORT_SYMBOL(__folio_alloc_noprof);
+
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
+{
+ warn_if_node_offline(nid, gfp);
+
+ return __folio_alloc_noprof(gfp, order, nid, NULL, ALLOC_DEFAULT);
+}
+EXPORT_SYMBOL(__folio_alloc_node_noprof);
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59d..23fc79ce97b60 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -258,6 +258,14 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_n
nodemask_t *nodemask, unsigned int alloc_flags);
#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))
+struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
+
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+ int preferred_nid, nodemask_t *nodemask, int nr_pages,
+ struct page **page_array);
+
extern void zone_pcp_reset(struct zone *zone);
extern void zone_pcp_disable(struct zone *zone);
extern void zone_pcp_enable(struct zone *zone);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 02/36] mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Extract per-node fallback-list construction into build_node_zonelist().
This lets use build zonelists from candidate nodemasks instead of just
the default N_MEMORY node state list.
No functional change: build_zonelists() builds the same FALLBACK list over
N_MEMORY with node_load updates as before.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/page_alloc.c | 44 ++++++++++++++++++++++++++++++--------------
1 file changed, 30 insertions(+), 14 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index acf096f525f49..4c1ef84c02a3d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5813,12 +5813,12 @@ int find_next_best_node_in(int node, nodemask_t *used_node_mask,
* DMA zone, if any--but risks exhausting DMA zone.
*/
static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
- unsigned nr_nodes)
+ unsigned int nr_nodes, int zlidx)
{
struct zoneref *zonerefs;
int i;
- zonerefs = pgdat->node_zonelists[ZONELIST_FALLBACK]._zonerefs;
+ zonerefs = pgdat->node_zonelists[zlidx]._zonerefs;
for (i = 0; i < nr_nodes; i++) {
int nr_zones;
@@ -5847,26 +5847,28 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)
zonerefs->zone_idx = 0;
}
-static void build_zonelists(pg_data_t *pgdat)
+/*
+ * Build one node-ordered fallback list from a candidate nodemask.
+ * update_load round-robins node_load across equidistant nodes.
+ */
+static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
+ int zlidx, bool update_load,
+ int *node_order, int *nr)
{
- static int node_order[MAX_NUMNODES];
- int node, nr_nodes = 0;
nodemask_t used_mask = NODE_MASK_NONE;
- int local_node, prev_node;
-
- /* NUMA-aware ordering of nodes */
- local_node = pgdat->node_id;
- prev_node = local_node;
+ int local_node = pgdat->node_id;
+ int prev_node = local_node;
+ int node, nr_nodes = 0;
- memset(node_order, 0, sizeof(node_order));
while ((node = find_next_best_node_in(local_node, &used_mask,
- &node_states[N_MEMORY])) >= 0) {
+ candidates)) >= 0) {
/*
* We don't want to pressure a particular node.
* So adding penalty to the first node in same
* distance group to make it round-robin.
*/
- if (node_distance(local_node, node) !=
+ if (update_load &&
+ node_distance(local_node, node) !=
node_distance(local_node, prev_node))
node_load[node] += 1;
@@ -5874,8 +5876,22 @@ static void build_zonelists(pg_data_t *pgdat)
prev_node = node;
}
- build_zonelists_in_node_order(pgdat, node_order, nr_nodes);
+ build_zonelists_in_node_order(pgdat, node_order, nr_nodes, zlidx);
+ *nr = nr_nodes;
+}
+
+static void build_zonelists(pg_data_t *pgdat)
+{
+ static int node_order[MAX_NUMNODES];
+ int local_node = pgdat->node_id;
+ int node, nr_nodes = 0;
+
+ memset(node_order, 0, sizeof(node_order));
+
+ build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
+ true, node_order, &nr_nodes);
build_thisnode_zonelists(pgdat);
+
pr_info("Fallback order for Node %d: ", local_node);
for (node = 0; node < nr_nodes; node++)
pr_cont("%d ", node_order[node]);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
find_next_best_node() picks the next-closest node for a fallback list
from the full N_MEMORY set. Refactor it into find_next_best_node_in(),
which takes an explicit candidates nodemask.
This enables building fallback lists with non-N_MEMORY candidates.
No functional change: every caller still selects from N_MEMORY.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/internal.h | 6 ++++--
mm/memory-tiers.c | 7 ++++---
mm/page_alloc.c | 13 ++++++++-----
3 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index f26423de4ca28..96d78a7778e88 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1103,7 +1103,8 @@ extern int node_reclaim_mode;
extern unsigned long node_reclaim(struct pglist_data *pgdat,
gfp_t gfp_mask, unsigned int order);
-extern int find_next_best_node(int node, nodemask_t *used_node_mask);
+extern int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates);
#else
#define node_reclaim_mode 0
@@ -1112,7 +1113,8 @@ static inline unsigned long node_reclaim(struct pglist_data *pgdat,
{
return 0;
}
-static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
+static inline int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates)
{
return NUMA_NO_NODE;
}
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b0..25e121851b586 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -370,7 +370,7 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
* closest demotion target.
*/
nodes_complement(mask, *allowed_mask);
- return find_next_best_node(node, &mask);
+ return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
}
static void disable_all_demotion_targets(void)
@@ -450,7 +450,7 @@ static void establish_demotion_targets(void)
memtier = list_next_entry(memtier, list);
tier_nodes = get_memtier_nodemask(memtier);
/*
- * find_next_best_node, use 'used' nodemask as a skip list.
+ * find_next_best_node_in, use 'used' nodemask as a skip list.
* Add all memory nodes except the selected memory tier
* nodelist to skip list so that we find the best node from the
* memtier nodelist.
@@ -463,7 +463,8 @@ static void establish_demotion_targets(void)
* in the preferred mask when allocating pages during demotion.
*/
do {
- target = find_next_best_node(node, &tier_nodes);
+ target = find_next_best_node_in(node, &tier_nodes,
+ &node_states[N_MEMORY]);
if (target == NUMA_NO_NODE)
break;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f93a6bb9a872d..acf096f525f49 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5743,9 +5743,10 @@ static int numa_zonelist_order_handler(const struct ctl_table *table, int write,
static int node_load[MAX_NUMNODES];
/**
- * find_next_best_node - find the next node that should appear in a given node's fallback list
+ * find_next_best_node_in - find the next node that should appear in a given node's fallback list
* @node: node whose fallback list we're appending
* @used_node_mask: nodemask_t of already used nodes
+ * @candidates: nodemask_t of nodes eligible for selection
*
* We use a number of factors to determine which is the next node that should
* appear on a given node's fallback list. The node should not have appeared
@@ -5757,7 +5758,8 @@ static int node_load[MAX_NUMNODES];
*
* Return: node id of the found node or %NUMA_NO_NODE if no node is found.
*/
-int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates)
{
int n, val;
int min_val = INT_MAX;
@@ -5767,12 +5769,12 @@ int find_next_best_node(int node, nodemask_t *used_node_mask)
* Use the local node if we haven't already, but for memoryless local
* node, we should skip it and fall back to other nodes.
*/
- if (!node_isset(node, *used_node_mask) && node_state(node, N_MEMORY)) {
+ if (!node_isset(node, *used_node_mask) && node_isset(node, *candidates)) {
node_set(node, *used_node_mask);
return node;
}
- for_each_node_state(n, N_MEMORY) {
+ for_each_node_mask(n, *candidates) {
/* Don't want a node to appear more than once */
if (node_isset(n, *used_node_mask))
@@ -5857,7 +5859,8 @@ static void build_zonelists(pg_data_t *pgdat)
prev_node = local_node;
memset(node_order, 0, sizeof(node_order));
- while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
+ while ((node = find_next_best_node_in(local_node, &used_mask,
+ &node_states[N_MEMORY])) >= 0) {
/*
* We don't want to pressure a particular node.
* So adding penalty to the first node in same
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 00/36] Private Memory NUMA Nodes
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
This series introduces the concept of "Private Memory Nodes", which
are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing
All other features of mm/ are opted out of managing these NUMA
nodes and its memory. Then we add capability bits to allow node
owners to opt back into those services (if supported).
NUMA-hosted memory is presently a most-privilege system (any memory on
any node, except ZONE_DEVICE, is 100% fungible and accessible). We
then slap restrictions on top of it (cpuset, mempolicy, ZONE type,
page/folio flags, etc).
The goal here is to flip that dynamic, isolate by default and then
opt-in to specific services that the device says is safe.
Isolation at the NUMA/Zonelist layer provides a powerful mechanism for
memory hosted on accelerators - re-use of the kernel mm/ code.
- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls
(promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers
can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general
purpose memory get cleaner interfaces (no need to re-write the buddy
in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)
And more use cases I have collected over the past few years.
Not included here is a dax-extension [1] that exposes all the internal
bits as userland controls for testing - along with a pile of selftests
that prove correctness.
Changes Since V4
================
- Massive reduction in complexity.
- no ops struct
- no callback functions
- no __GFP_PRIVATE
- no __GFP_THISNODE requirement
- no task flags (no PF_MEMALLOC_* in the alloc path)
- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)
- ALLOC_ZONELIST_PRIVATE alloc_flag
- on top of Brendan Jackman's recent mm/page_alloc.h work [3]
- Zonelist selection rides the allocator's alloc_flags
- rename OPS -> CAPS (capabilities)
- split base functionality (isolation) from opt-ins (CAPS)
- first half of series can be merged without CAPS
- dropped compressed ram example from series
- will submit separately if this moves forward
- Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
- fully functional dax-kmem extension and huge suite of selftests
located at my github, to be discussed separately [1]
Patch Layout
============
The series is broken into two sections:
1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.
2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private
nodes to make them more useful (i.e. reclaim = overcommit).
NODE_PRIVATE_CAP_LTPIN for private node folio pinning
NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
My hope is to merge at least #1 pulled ahead while #2 is debated.
Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated
on a given call by 4 things (in order of authority)
1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.
2) __GFP_THISNODE
If this flag is set and the only node in the ZONELIST is the
singular preferred node (or the local node, for -1)
3) cpuset.mems membership
cpuset trims any node in its allowed list
4) mempolicy nodemask
the allocator will skip any node not in the nodemask.
Except for #1 (Zonelist membership) there are all kinds of weird corner
conditions in which 2-4 can be completely ignored (interrupt context,
empty set because cpuset doesn't intersect mempolicy, shared vma, ...)
But ZONELIST membership is *absolute*. If a zone is not in the
zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.
The existing zonelists are constructed like so:
ZONELIST_FALLBACK : All N_MEMORY nodes
ZONELIST_NOFALLBACK : A singleton N_MEMORY node
Private node isolation is implemented via ZONELIST isolation:
ZONELIST_PRIVATE : The private node + N_MEMORY
ZONELIST_PRIVATE_NOFALLBACK : The private node alone.
(mirrors exactly the fallback/nofallback for __GFP_THISNODE)
Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)
1 & 2 mean all existing in-tree callers to page_alloc can NEVER
accidentally allocate from a private node.
An allocation must explicitly ask via a zonelist and a nodemask.
alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */
The page allocator keeps all its original interfaces which only
ever touch the default zonelists - avoiding churn.
Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
=================================================================
The vast majority of the kernel will never need to know about
ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.
When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces
do the zonelist selection for the source of the allocation.
That really is the whole explanation of the mechanism:
alloc_flags = mpol_alloc_flags(pol);
page = __alloc_frozen_pages_noprof(..., alloc_flags);
On mm-new this rides the allocator's existing alloc_flags plumbing:
ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new
parameter, enum, or alloc_context change is required.
For modules that want to implement their own special handling, they
get the _private variants for the page allocator. This lets modules
re-use the buddy instead of rewriting it.
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()
This keeps ALLOC_ flags mm/ internal (these functions add the flags).
Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on
private node memory. Where possible, we even re-use existing
filter points from ZONE_DEVICE.
Most filter points are one or two lines of code:
Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))
Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }
Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;
In the second half of the series, we replace blanket N_MEMORY_PRIVATE
filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that
interaction if CAP is set.
We abstract this with a nice clean interface to make it really clear
what is happening (nodes have features!)
- if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
+ if (!node_allows_reclaim(pgdat->node_id))
NODE_PRIVATE_CAP_* features
===========================
This series of commits opts private nodes into various mm/ services.
Capabilities:
NODE_PRIVATE_CAP_RECLAIM - direct and kswapd reclaim
NODE_PRIVATE_CAP_USER_NUMA - userland numa controls
NODE_PRIVATE_CAP_DEMOTION - node is a demotion target
NODE_PRIVATE_CAP_HOTUNPLUG - hotunplug may migrate
NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally
Some opt-in support is more intensive than others, so these features
are broken out in a way that we can defer them as future work streams.
NODE_PRIVATE_CAP_RECLAIM:
Enabling reclaim for these nodes is actually surprisingly trivial.
Without CAP_RECLAIM, when an allocation failure occurs, the system
will not attempt to swap the memory - and instead will OOM (typically
whatever task is using the most memory on *that* private node).
This capability consists of:
1) enabling kswapd and kcompactd for that node at hotplug time.
2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
3) Sets normal watermarks for these nodes.
4) Allow madvise operations on that node (PAGEOUT).
5) A small tweak to how LRU decides which zones to visit.
NODE_PRIVATE_CAP_USER_NUMA
Enables the following userland interfaces to accept the node:
mbind()
set_mempolicy()
set_mempolicy_home_node()
move_pages()
migrate_pages()
example:
buf = mmap(..., MAP_ANON);
mbind(buf, ..., {private_node});
buf[0] = 0xDEADBEEF; /* Page faults onto the private node */
Later - the KVM example shows how in-kernel mempolicies can
also be bound by CAP_USER_NUMA.
Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.
NODE_PRIVATE_CAP_HOTUNPLUG
This is simple: allow hotunplug to migrate this nodes folios.
Some devices may not be able to tolerate unexpected migrations,
so we prevent hotunplug from engaging in migration by default.
Some devices may have an mmu_notifier in their driver that can
manage the migration and subsequent refault.
CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.
NODE_PRIVATE_CAP_DEMOTION
This adds the private node as a valid demotion target, and allows
reclaim to demote memory from a private node to a demotion target.
Requires: NODE_PRIVATE_CAP_RECLAIM
NODE_PRIVATE_CAP_NUMA_BALANCING
This enables numa balancing to inject prot_none on private node
folio mappings and promote them when faults are taken.
NODE_PRIVATE_CAP_LTPIN
This allows GUP Longterm Pinning to operate normally.
Normally, longterm pinning determines folio eligibility based
on its ZONE_* membership (among other things).
ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require
migration to ZONE_NORMAL before pinning.
Neither operation is preferable by default on a private node,
so the base behavior of FOLL_LONGTERM is to FAIL.
This capability allows longterm pinning to operate normally
based on the ZONE membership. Private node memory may be
hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.
In-tree User: KVM
=================
Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of
enabling disaggregated memory pools to host dedicated KVM memory.
With private nodes, this is trivial (with a bit of basic plumbing):
static int kvm_gmem_bind_node(struct inode *inode, int node)
{
...
/* Bind to a private node - gated on CAP_USER_NUMA */
pol = mpol_bind_node(node);
if (IS_ERR(pol))
return PTR_ERR(pol);
/* Set the shared policy */
err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
...
}
KVM doesn't even need to know about private nodes at all, all it
does is ask mempolicy whether the requested node is a valid bind.
mm/ component testing with dax driver
=====================================
The dax driver extensions[1] implements a simple interface to create
a private node from a dax device created by any source.
I left the dax driver extensions out of this feature set because
it locks in the CAP_ bits before anyone has input. It's there
primarily for testing at this point.
The simplest way to get a dax device is with the memmap= boot arg.
e.g.: "memmap=0x40000000!0x140000000"
The dax driver extension has the following sysfs entries:
dax0.0/private - set the node to private
dax0.0/dax_file - make /dev/dax0.0 mmap'able in kmem mode
dax0.0/adistance - dictate memory_tierN membership
dax0.0/reclaim - CAP_RECLAIM
dax0.0/demotion - CAP_DEMOTION
dax0.0/user_numa - CAP_USER_NUMA
dax0.0/hotunplug - CAP_HOTUNPLUG
dax0.0/numa_balancing - CAP_NUMA_BALANCING
dax0.0/ltpin - CAP_LTPIN
Now consider the following...
Single node reclaim + mbind support:
echo 1 > dax0.0/private
echo 1 > dax0.0/reclaim
echo 1 > dax0.0/user_numa
echo online_movable > dax0.0/state
Test program:
/* node1: 1GB Private Memory Node, 4GB swap */
buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
memset(buf, 0xff, TWO_GB);
We can *guarantee* the ONLY reclaiming tasks are exactly:
- kswapdN (in theory we can even make this optional!)
- the memset task faulting pages in
This also means these are the only tasks capable of becoming
locked up and OOMing (as long as it is not under pressure).
The rest of the system remains entirely functional for debugging.
It now becomes possible to micro-benchmark and A/B test reclaim
changes with different scenarios (number of tasks, amount of
memory, watermark targets, etc), because we have hard controls
over exactly which tasks can access that node memory and how.
If we broke CAP_RECLAIM into subflags:
- CAP_RECLAIM_KSWAPD
- CAP_RECLAIM_DIRECT
- CAP_COMPACTION_KCOMPACTD
- CAP_COMPACTION_DIRECT
We can actually test the efficacy of each of these mechanisms in
isolation to each other - something that is strictly impossible today.
Bonus Configuration: HBM device memory tiering
==============================================
echo 1 > dax0.0/private # make the node private
echo 0 > dax0.0/adistance # highest tier
echo 1 > dax0.0/reclaim # reclaim active
echo 1 > dax0.0/demotion # may demote from the node
echo 1 > dax0.0/user_numa # mbind()
echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled
This is an HBM device which is treated as the top-tier in the
system but for which memory can only enter via explicit mbind().
It can be overcommitted because it can be reclaimed (demotions
go to CPU DRAM, and reclaim can swap from it).
If the HBM is managed by an accelerator (GPU), the mmu_notifier
allows it to know when reclaim is moving memory out to do
device-mmu invalidation prior to migration.
Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]
Prereqs (all already in akpm/mm-new; listed for out-of-tree application):
page_alloc.h split + alloc_flags plumbing this series rides on:
commit e81fae43cd69 ("mm: split out internal page_alloc.h")
commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")
dax atomic whole-device hotplug (used by the dax extension [1]):
commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper")
commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")
[1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
[2] https://lore.kernel.org/all/20260423170219.281618-1-dave.jiang@intel.com/
[3] https://lore.kernel.org/all/20260702-alloc-trylock-v4-0-0af8ff387e80@google.com/
base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e
Gregory Price (36):
mm: refactor find_next_best_node to find_next_best_node_in
mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
mm/page_alloc: let the bulk and folio allocators carry alloc_flags
numa: introduce N_MEMORY_PRIVATE
mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
cpuset: exclude private nodes from cpuset.mems (default-open)
mm/memory_hotplug: disallow migration-driven private node hotunplug
mm/mempolicy: skip private node folios when queueing for migration
mm/migrate: disallow userland driven migration for private nodes
mm/madvise: disallow madvise operations on private node folios
mm/compaction: disallow compaction on private nodes
mm/page_alloc: clear private node watermarks and system reserves
mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
mm/damon: skip private node memory in DAMON migration and pageout
mm/ksm: skip KSM for managed-memory folios
mm/khugepaged: skip private node folios when trying to collapse.
mm/vmscan: disallow reclaim of private node memory
mm/gup: disallow longterm pin of private node folios
proc: include N_MEMORY_PRIVATE nodes in numa_maps output
mm/memcontrol: account private-node memory in per-node stats
proc/kcore: include private-node RAM in the kcore RAM map
mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
mm/mempolicy: apply policy at the kernel zone for private-node binds
mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
balancing
mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning
mm/khugepaged: base private node collapse eligiblity on actor/cap bits
Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes
mm/mempolicy: add mpol_set_shared_policy_range()
KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test
Documentation/ABI/stable/sysfs-devices-node | 10 +
Documentation/mm/index.rst | 1 +
Documentation/mm/numa_private_nodes.rst | 160 ++++++++++
drivers/base/node.c | 118 +++++++
drivers/dax/kmem.c | 2 +-
fs/proc/kcore.c | 5 +-
fs/proc/task_mmu.c | 10 +-
include/linux/gfp.h | 22 +-
include/linux/kvm_host.h | 3 +
include/linux/memory_hotplug.h | 5 +-
include/linux/mempolicy.h | 16 +
include/linux/mmzone.h | 26 +-
include/linux/node_private.h | 262 ++++++++++++++++
include/linux/nodemask.h | 7 +-
include/uapi/linux/kvm.h | 5 +-
include/uapi/linux/mempolicy.h | 1 +
kernel/cgroup/cpuset.c | 26 +-
mm/compaction.c | 13 +
mm/damon/paddr.c | 9 +
mm/gup.c | 28 +-
mm/huge_memory.c | 5 +
mm/internal.h | 97 +++++-
mm/khugepaged.c | 18 +-
mm/ksm.c | 8 +-
mm/madvise.c | 8 +-
mm/memcontrol-v1.c | 8 +-
mm/memcontrol.c | 13 +-
mm/memory-tiers.c | 40 ++-
mm/memory_hotplug.c | 119 +++++++-
mm/mempolicy.c | 288 +++++++++++++++---
mm/migrate.c | 19 +-
mm/mm_init.c | 2 +-
mm/page_alloc.c | 189 +++++++++---
mm/page_alloc.h | 34 +++
mm/vmscan.c | 57 +++-
tools/testing/selftests/kvm/Makefile.kvm | 2 +
.../kvm/guest_memfd_bind_node_test.c | 213 +++++++++++++
virt/kvm/guest_memfd.c | 39 ++-
38 files changed, 1728 insertions(+), 160 deletions(-)
create mode 100644 Documentation/mm/numa_private_nodes.rst
create mode 100644 include/linux/node_private.h
create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
--
2.53.0-Meta
^ permalink raw reply
* Re: [PATCH bpf-next v5 8/8] selftests: net: add test for XDP_PASS skb checksum invalidation
From: Stanislav Fomichev @ 2026-07-20 19:32 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: Donald Hunter, Jakub Kicinski, David S. Miller, Eric Dumazet,
Paolo Abeni, Simon Horman, Alexei Starovoitov, Daniel Borkmann,
Jesper Dangaard Brouer, John Fastabend, Stanislav Fomichev,
Andrew Lunn, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Hao Luo, Jiri Olsa, Shuah Khan,
Maciej Fijalkowski, Jonathan Corbet, Shuah Khan,
Kumar Kartikeya Dwivedi, Emil Tsalapatis, Vladimir Vdovin,
Jakub Sitnicki, netdev, bpf, intel-wired-lan, linux-kselftest,
linux-doc
In-Reply-To: <alkBnI3XcH0XGjEo@lore-desk>
On 07/16, Lorenzo Bianconi wrote:
> On Jul 16, Stanislav Fomichev wrote:
> > On 07/15, Lorenzo Bianconi wrote:
> > > Add a test that verifies skb->ip_summed is set to CHECKSUM_NONE
> > > when a device running in XDP mode creates an skb from a xdp_buff
> > > if the attached ebpf program returns an XDP_PASS.
> > > The test attaches an XDP program returning XDP_PASS, and a TC
> > > ingress program that runs the bpf_skb_rx_checksum() kfunc to
> > > inspect the resulting skb. After XDP_PASS the driver must invalidate
> > > any previously computed hardware RX checksum since XDP may have
> > > modified the packet data.
> > > The BPF program counts packets per checksum type in a map, and the
> > > test runner verifies that after sending traffic the CHECKSUM_NONE
> > > counter is non-zero while CHECKSUM_UNNECESSARY and CHECKSUM_COMPLETE
> > > counters are zero.
> > >
> > > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> > > ---
> > > Documentation/networking/xdp-rx-metadata.rst | 5 ++
> > > .../selftests/drivers/net/hw/xdp_metadata.py | 55 +++++++++++++++-
> > > .../selftests/net/lib/skb_metadata_csum.bpf.c | 73 ++++++++++++++++++++++
> > > 3 files changed, 132 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/Documentation/networking/xdp-rx-metadata.rst b/Documentation/networking/xdp-rx-metadata.rst
> > > index 93918b3769a3..7434ac98242a 100644
> > > --- a/Documentation/networking/xdp-rx-metadata.rst
> > > +++ b/Documentation/networking/xdp-rx-metadata.rst
> > > @@ -90,6 +90,11 @@ conversion, and the XDP metadata is not used by the kernel when building
> > > ``skbs``. However, TC-BPF programs can access the XDP metadata area using
> > > the ``data_meta`` pointer.
> >
> > [..]
> >
> > > +If a driver is running in XDP mode, any existing hardware RX checksum
> > > +(``CHECKSUM_UNNECESSARY`` or ``CHECKSUM_COMPLETE``) must be invalidated
> > > +by setting ``skb->ip_summed`` to ``CHECKSUM_NONE`` before passing the
> > > +skb to the kernel, since XDP may have modified the packet data.
> > > +
> > > In the future, we'd like to support a case where an XDP program
> > > can override some of the metadata used for building ``skbs``.
> >
> > Sorry for keeping nitpicking on this, but I'm still not convinced that
> > it is what we currently do. From my previous reply:
>
> no worries :)
> My current take-away from the previous discussion is we just need to document
> what would be the driver expected behaviour adding a kselftest for it (without
> modifying any driver).
>
> >
> > > > Looking at a few drivers:
> > > > - bnxt (bnxt_rx_pkt) does UNNECESSARY - ok
> > > > - mlx5 (mlx5e_handle_csum) does UNNECESSARY and skips COMPLETE if there is
> > > > bpf prog attached
> > > > - fbnic (fbnic_rx_csum) - can do COMPLETE even with xdp attached?
> > > > - gve (gve_rx) - can do COMPLETE even with xdp attached?
> >
> > (although for gve I might be wrong, there is also gve_rx_skb_csum that only
> > does UNNECESSARY).
> >
> > I'd wait for Jakub to chime in, but it feels like we should just document
> > what we currently do as a recommended approach: for the drivers
> > that support COMPLETE, do not report it when the bpf program is attached.
> > Both NONE and UNNECESSARY are ok.
>
> I am not completely sure the UNNECESSARY case is different from the COMPLETE
> one. What are we supposed to do if the driver reports UNNECESSARY and the ebpf
> program modifies some fields covered by the rx-checksum?
For unnecessary, I think the safe expectation is that the bpf program
will update the value of the checksum in the packet if it touches the data?
> > Also, did you run this test on real HW? NIPA now has HW tests, maybe it
> > makes sense to route this series via net-next to get the real coverage?
>
> What about splitting this series and have two different series:
> - bpf-next: add xdp rx kfunc and related selftest
> - net-next: add kselftest for the driver expected behaviour.
>
> What do you think?
I'd post everything to net-next to get the HW coverage. Once you get all
the acks we can ask the maintainers' guidance.
^ permalink raw reply
* [RFC PATCH 2/2] KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
From: Luka Absandze @ 2026-07-20 19:22 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, kvm
Cc: linux-kernel, linux-doc, Alexander Graf, David Woodhouse,
Luka Absandze
In-Reply-To: <20260720192221.72912-1-absandze@amazon.de>
Describe the new VM-scoped capability: its parameter (a bitmask of
PERF_COUNT_HW_* event IDs), the value returned by KVM_CHECK_EXTENSION
(the set of events that can be disabled), the accuracy trade-off, and
why an emulated-vPMU host wants it.
Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Luka Absandze <absandze@amazon.de>
---
Documentation/virt/kvm/api.rst | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index a5f9ee92f43e..231a5ba7567c 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -8949,6 +8949,38 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
interpretation are disabled. If cmma has already been enabled or the
hpage_2g module parameter is not set to 1, -EINVAL is returned.
+7.48 KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
+------------------------------------------
+
+:Architectures: x86
+:Type: vm
+:Parameters: args[0] - bitmask of PERF_COUNT_HW_* event IDs
+:Returns: 0 on success; -EINVAL if args[0] contains an unsupported event or
+ if the VM already has vCPUs.
+
+By default, KVM software-accounts instructions it emulates against a guest
+PMU counter programmed for retired instructions (PERF_COUNT_HW_INSTRUCTIONS)
+or retired branches (PERF_COUNT_HW_BRANCH_INSTRUCTIONS), by walking the vPMU
+counters and requesting a counter reprogram on the next VM-entry. On hosts
+with an emulated vPMU this reprogram is expensive and, under some workloads,
+inflates timer-interrupt handling enough to trigger guest CSD lockups.
+
+This capability lets userspace disable that software accounting for the given
+events. Setting a bit for an event ID makes KVM skip the accounting for that
+event: instructions KVM emulates in host context go uncounted. Hardware-
+executed guest instructions are still counted by the backing perf_event, and
+its overflow/PMI path is unaffected.
+
+The capability only affects the emulated (perf-based) vPMU. A mediated vPMU
+does not incur the reprogram cost and increments the guest counter directly,
+so accounting is never skipped for it regardless of this setting.
+
+This can only be invoked on a VM prior to the creation of vCPUs; attempting to
+set it once any vCPU exists returns -EINVAL.
+
+KVM_CHECK_EXTENSION returns the bitmask of event IDs that can be disabled.
+args[0] must be a subset of that mask.
+
8. Other capabilities.
======================
--
2.47.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox