* Re: [patch 00/18] entry: Consolidate and rework syscall entry handling
From: Thomas Gleixner @ 2026-07-20 19:21 UTC (permalink / raw)
To: Magnus Lindholm
Cc: LKML, Peter Zijlstra, Michael Ellerman, Shrikanth Hegde,
linuxppc-dev, Kees Cook, Huacai Chen, loongarch, Paul Walmsley,
Palmer Dabbelt, linux-riscv, Sven Schnelle, linux-s390, x86,
Mark Rutland, Jinjie Ruan, Andy Lutomirski, Oleg Nesterov,
Richard Henderson, Russell King, Catalin Marinas, Guo Ren,
Geert Uytterhoeven, Thomas Bogendoerfer, Helge Deller,
Yoshinori Sato, Richard Weinberger, Chris Zankel,
linux-arm-kernel, linux-alpha, linux-csky, linux-m68k, linux-mips,
linux-parisc, linux-sh, linux-um, Arnd Bergmann, Vineet Gupta,
Will Deacon, Brian Cain, Michal Simek, Dinh Nguyen,
David S. Miller, Andreas Larsson, linux-snps-arc, linux-hexagon,
linux-openrisc, sparclinux, linux-arch, Michal Suchánek,
Jonathan Corbet, linux-doc
In-Reply-To: <CA+=Fv5TDNH8+JAojV9_W36kUkEE68uiA00FQ7i0vJunebqs1sA@mail.gmail.com>
Magnus!
On Sun, Jul 19 2026 at 13:25, Magnus Lindholm wrote:
> One Alpha-specific wrinkle I hit while rebasing the GENERIC_ENTRY patch is
> that Alpha cannot unconditionally pre-seed the normal -ENOSYS return state
> before calling into the generic entry code: syscall_set_return_value() also
> updates r19/a3, which is still syscall argument 4 on the normal entry path.
> Doing that too early broke early userspace mmap() during boot.
Similar to what Peter and me suggested in the related discussion
vs. s390 which has a similar issue, you can just have a dedicated
syscall_return member in your pt_regs struct, which is preset to -ENOSYS
and operate on that. You have an unused padding entry there which means
it won't even change the size.
> Are you planning to send a v2 with the seccomp recheck fix and any other
> review fixups folded in? Is the intention to route this upstream
> during the next merge window?
You were on CC for the V2, no?
The whole pile including Jinjie's seccomp bypass fix is in
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/entry
and is targeted for the 7.3 merge window.
If you want to base your stuff on that for 7.3, I can add a tag which
makes it immutable so it can be pulled into the alpha tree.
Thanks,
tglx
^ permalink raw reply
* [RFC PATCH 0/2] KVM: x86/pmu: Let userspace disable SW accounting of emulated instructions
From: Luka Absandze @ 2026-07-20 19:22 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, kvm
Cc: linux-kernel, linux-doc, Alexander Graf, David Woodhouse,
Luka Absandze
On hosts that run guests with an *emulated* vPMU -- guest PMU MSR
accesses trap into KVM and each guest counter is backed by a host
perf_event -- the per-emulated-instruction PMU accounting added by commit
9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") is
pathologically expensive.
Whenever a guest programs a counter for retired instructions (0xc0) or
retired branches (0xc2) -- routine for any profiler-style workload --
every instruction KVM emulates triggers a software increment plus a
batched KVM_REQ_PMU reprogram. The reprogram is drained on the next
VM-entry regardless of which exit preceded it, so ordinary exits (most
importantly the guest's 1kHz timer tick) absorb a full
reprogram_counter() cycle: ctx->mutex, ctx_resched(), and a burst of
serialized PMU-MSR writes. That inflates timer-interrupt handling into
the hundreds of microseconds and, under load, can escalate to a guest
CSD lockup.
This series adds KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING, a VM-scoped
capability that takes a bitmask of PERF_COUNT_HW_* event IDs and makes
KVM skip the software accounting of emulated instructions for those
events. It is opt-in and defaults off; the only cost to an opted-in
guest is that instructions-retired / branches-retired counters
undercount by the instructions KVM emulates in host context, i.e. the
behavior that predates that accounting. Hardware-executed guest
instructions are still counted by the backing perf_event and its
overflow/PMI path is untouched.
Patch 1 adds the capability and the accounting short-circuit.
Patch 2 documents it in Documentation/virt/kvm/api.rst.
Measured benefit
================
Host: AMD EPYC 7R13 "Milan", KVM, v7.2-rc4 + this series. Guest: QEMU,
-cpu host,pmu=on, 8 vCPUs. The guest workload keeps an instructions-
retired counter resident on every vCPU and periodically reprograms the
PMU, driving the accounting storm above, while the vCPUs are kept busy
so the 1kHz tick fires. The guest's local-APIC timer interrupt handler
(__sysvec_apic_timer_interrupt) is traced with ftrace function_graph
for 30s (~160k samples per run). The capability is toggled per-VM via
KVM_ENABLE_CAP; nothing else changes between runs.
Timer-interrupt handler duration:
metric accounting ON (baseline) accounting OFF (CAP) speedup
------ ------------------------ -------------------- -------
mean 209.8 us 3.16 us 66x
p50 256.4 us 3.14 us 82x
p99 463.3 us 4.45 us 104x
p99.9 915.2 us 5.03 us 182x
max 929.7 us 22.55 us 41x
Some workloads genuinely need the extra precision that the emulated-
instruction accounting provides, and for them the default behavior is
the right trade-off. But for many others the accuracy gained is not
worth the cost it imposes on every exit, and this capability lets those
users opt out of paying it.
Luka Absandze (2):
KVM: x86/pmu: Add CAP to disable SW accounting of emulated
instructions
KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
Documentation/virt/kvm/api.rst | 24 ++++++++++++++++++++++++
arch/x86/include/asm/kvm_host.h | 7 +++++++
arch/x86/kvm/pmu.c | 6 ++++++
arch/x86/kvm/pmu.h | 9 +++++++++
arch/x86/kvm/x86.c | 10 ++++++++++
include/uapi/linux/kvm.h | 1 +
6 files changed, 57 insertions(+)
--
2.47.3
^ permalink raw reply
* [RFC PATCH 1/2] KVM: x86/pmu: Add CAP to disable SW accounting of emulated instructions
From: Luka Absandze @ 2026-07-20 19:22 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, kvm
Cc: linux-kernel, linux-doc, Alexander Graf, David Woodhouse,
Luka Absandze
In-Reply-To: <20260720192221.72912-1-absandze@amazon.de>
On a host with an emulated vPMU (guest PMU MSR accesses trap and each
guest counter is backed by a host perf_event), the per-emulated-
instruction PMU accounting added by commit 9cd803d496e7 ("KVM: x86: Update
vPMCs when retiring instructions") is catastrophically expensive.
For every instruction KVM emulates, kvm_skip_emulated_instruction(),
the emulator writeback path, and nested VMLAUNCH/VMRESUME call into
the retired-instruction accounting for PERF_COUNT_HW_INSTRUCTIONS and
PERF_COUNT_HW_BRANCH_INSTRUCTIONS. If the guest has a counter
programmed for retired instructions (0xc0) or retired branches (0xc2)
-- e.g. any guest running a profiler with an instructions-retired
event -- this walks the counters, software-increments the matching
one, and requests a full reprogram (KVM_REQ_PMU). The reprogram is
drained on the vCPU's next VM-entry, where reprogram_counter() runs
the full pmc_pause_counter() + perf_event_period() + perf_event_
enable() sequence: ctx->mutex, a ctx_resched() of the PMU context,
and a burst of serialized PMU-MSR writes.
Because the batched reprogram is serviced on the next VM-entry
regardless of which exit preceded it, ordinary exits -- notably the
guest's 1kHz timer tick -- absorb the cost, inflating timer
interrupts into the hundreds of microseconds. With certain workloads
this can escalate to a CSD lockup in the guest.
Add KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING, a VM-scoped capability that
takes a bitmask of PERF_COUNT_HW_* event IDs and makes KVM skip the
software accounting of KVM-emulated instructions for those events.
KVM_CHECK_EXTENSION returns the set of events that can be disabled
(PERF_COUNT_HW_INSTRUCTIONS and PERF_COUNT_HW_BRANCH_INSTRUCTIONS --
the only events the retired-instruction path ever triggers).
The only functional change for an opted-in VM is reduced accuracy: a
guest counting instructions-retired or branches-retired undercounts by
the instructions KVM emulates in host context, i.e. the behavior that
predates the accounting cited above. Hardware-executed guest
instructions continue to be counted by the backing perf_event, and its
overflow/PMI path is unchanged. Default behavior (mask 0) is unchanged.
Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Luka Absandze <absandze@amazon.de>
---
arch/x86/include/asm/kvm_host.h | 7 +++++++
arch/x86/kvm/pmu.c | 21 +++++++++++++++++++++
arch/x86/kvm/pmu.h | 9 +++++++++
arch/x86/kvm/x86.c | 15 +++++++++++++++
include/uapi/linux/kvm.h | 1 +
5 files changed, 53 insertions(+)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 5f6c1ce9673b..522f66c44e6a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1560,6 +1560,13 @@ struct kvm_arch {
bool enable_pmu;
bool created_mediated_pmu;
+ /*
+ * Bitmask of PERF_COUNT_HW_* event IDs for which software accounting
+ * of KVM-emulated instructions is disabled (KVM_CAP_X86_DISABLE_PMU_
+ * SW_ACCOUNTING). See kvm_pmu_trigger_event().
+ */
+ u64 pmu_disable_sw_accounting;
+
u32 notify_window;
u32 notify_vmexit_flags;
/*
diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c
index dd1c57593f48..fb3f2df068dc 100644
--- a/arch/x86/kvm/pmu.c
+++ b/arch/x86/kvm/pmu.c
@@ -1145,14 +1145,35 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *vcpu,
srcu_read_unlock(&vcpu->kvm->srcu, idx);
}
+/*
+ * Whether software accounting of KVM-emulated instructions for @perf_hw_id is
+ * disabled via KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING. The capability only
+ * targets the emulated (perf-based) vPMU, where the accounting drives an
+ * expensive counter reprogram; a mediated vPMU increments pmc->counter
+ * directly (and may raise a PMI on overflow), so it is never skipped.
+ */
+static bool kvm_pmu_skip_sw_accounting(struct kvm_vcpu *vcpu, u64 perf_hw_id)
+{
+ if (kvm_vcpu_has_mediated_pmu(vcpu))
+ return false;
+
+ return vcpu->kvm->arch.pmu_disable_sw_accounting & BIT_ULL(perf_hw_id);
+}
+
void kvm_pmu_instruction_retired(struct kvm_vcpu *vcpu)
{
+ if (kvm_pmu_skip_sw_accounting(vcpu, PERF_COUNT_HW_INSTRUCTIONS))
+ return;
+
kvm_pmu_trigger_event(vcpu, vcpu_to_pmu(vcpu)->pmc_counting_instructions);
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_pmu_instruction_retired);
void kvm_pmu_branch_retired(struct kvm_vcpu *vcpu)
{
+ if (kvm_pmu_skip_sw_accounting(vcpu, PERF_COUNT_HW_BRANCH_INSTRUCTIONS))
+ return;
+
kvm_pmu_trigger_event(vcpu, vcpu_to_pmu(vcpu)->pmc_counting_branches);
}
EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_pmu_branch_retired);
diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h
index a5821d7c87f9..e34b9e814355 100644
--- a/arch/x86/kvm/pmu.h
+++ b/arch/x86/kvm/pmu.h
@@ -23,6 +23,15 @@
#define KVM_FIXED_PMC_BASE_IDX INTEL_PMC_IDX_FIXED
+/*
+ * Events for which KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING can disable software
+ * accounting of emulated instructions. These are the only events KVM ever
+ * passes to kvm_pmu_trigger_event() (from the retired-instruction path).
+ */
+#define KVM_PMU_SW_ACCOUNTING_VALID_MASK \
+ (BIT_ULL(PERF_COUNT_HW_INSTRUCTIONS) | \
+ BIT_ULL(PERF_COUNT_HW_BRANCH_INSTRUCTIONS))
+
struct kvm_pmu_ops {
struct kvm_pmc *(*rdpmc_ecx_to_pmc)(struct kvm_vcpu *vcpu,
unsigned int idx, u64 *mask);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index afcac1042947..cc8a36f28aea 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4964,6 +4964,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_DISABLE_QUIRKS2:
r = kvm_caps.supported_quirks;
break;
+ case KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING:
+ r = KVM_PMU_SW_ACCOUNTING_VALID_MASK;
+ break;
case KVM_CAP_X86_NOTIFY_VMEXIT:
r = kvm_caps.has_notify_vmexit;
break;
@@ -6730,6 +6733,18 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
return -EINVAL;
switch (cap->cap) {
+ case KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING:
+ r = -EINVAL;
+ if (cap->args[0] & ~KVM_PMU_SW_ACCOUNTING_VALID_MASK)
+ break;
+
+ mutex_lock(&kvm->lock);
+ if (!kvm->created_vcpus) {
+ kvm->arch.pmu_disable_sw_accounting = cap->args[0];
+ r = 0;
+ }
+ mutex_unlock(&kvm->lock);
+ break;
case KVM_CAP_DISABLE_QUIRKS2:
r = -EINVAL;
if (cap->args[0] & ~kvm_caps.supported_quirks)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 419011097fa8..186054245b80 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -997,6 +997,7 @@ struct kvm_enable_cap {
#define KVM_CAP_S390_KEYOP 247
#define KVM_CAP_S390_VSIE_ESAMODE 248
#define KVM_CAP_S390_HPAGE_2G 249
+#define KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING 250
struct kvm_irq_routing_irqchip {
__u32 irqchip;
--
2.47.3
^ permalink raw reply related
* [RFC PATCH 2/2] KVM: Documentation: Document KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
From: Luka Absandze @ 2026-07-20 19:22 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, kvm
Cc: linux-kernel, linux-doc, Alexander Graf, David Woodhouse,
Luka Absandze
In-Reply-To: <20260720192221.72912-1-absandze@amazon.de>
Describe the new VM-scoped capability: its parameter (a bitmask of
PERF_COUNT_HW_* event IDs), the value returned by KVM_CHECK_EXTENSION
(the set of events that can be disabled), the accuracy trade-off, and
why an emulated-vPMU host wants it.
Assisted-by: Claude:claude-opus-4.8
Signed-off-by: Luka Absandze <absandze@amazon.de>
---
Documentation/virt/kvm/api.rst | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index a5f9ee92f43e..231a5ba7567c 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -8949,6 +8949,38 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
interpretation are disabled. If cmma has already been enabled or the
hpage_2g module parameter is not set to 1, -EINVAL is returned.
+7.48 KVM_CAP_X86_DISABLE_PMU_SW_ACCOUNTING
+------------------------------------------
+
+:Architectures: x86
+:Type: vm
+:Parameters: args[0] - bitmask of PERF_COUNT_HW_* event IDs
+:Returns: 0 on success; -EINVAL if args[0] contains an unsupported event or
+ if the VM already has vCPUs.
+
+By default, KVM software-accounts instructions it emulates against a guest
+PMU counter programmed for retired instructions (PERF_COUNT_HW_INSTRUCTIONS)
+or retired branches (PERF_COUNT_HW_BRANCH_INSTRUCTIONS), by walking the vPMU
+counters and requesting a counter reprogram on the next VM-entry. On hosts
+with an emulated vPMU this reprogram is expensive and, under some workloads,
+inflates timer-interrupt handling enough to trigger guest CSD lockups.
+
+This capability lets userspace disable that software accounting for the given
+events. Setting a bit for an event ID makes KVM skip the accounting for that
+event: instructions KVM emulates in host context go uncounted. Hardware-
+executed guest instructions are still counted by the backing perf_event, and
+its overflow/PMI path is unaffected.
+
+The capability only affects the emulated (perf-based) vPMU. A mediated vPMU
+does not incur the reprogram cost and increments the guest counter directly,
+so accounting is never skipped for it regardless of this setting.
+
+This can only be invoked on a VM prior to the creation of vCPUs; attempting to
+set it once any vCPU exists returns -EINVAL.
+
+KVM_CHECK_EXTENSION returns the bitmask of event IDs that can be disabled.
+args[0] must be a subset of that mask.
+
8. Other capabilities.
======================
--
2.47.3
^ permalink raw reply related
* Re: [PATCH bpf-next v5 8/8] selftests: net: add test for XDP_PASS skb checksum invalidation
From: Stanislav Fomichev @ 2026-07-20 19:32 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: Donald Hunter, Jakub Kicinski, David S. Miller, Eric Dumazet,
Paolo Abeni, Simon Horman, Alexei Starovoitov, Daniel Borkmann,
Jesper Dangaard Brouer, John Fastabend, Stanislav Fomichev,
Andrew Lunn, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Hao Luo, Jiri Olsa, Shuah Khan,
Maciej Fijalkowski, Jonathan Corbet, Shuah Khan,
Kumar Kartikeya Dwivedi, Emil Tsalapatis, Vladimir Vdovin,
Jakub Sitnicki, netdev, bpf, intel-wired-lan, linux-kselftest,
linux-doc
In-Reply-To: <alkBnI3XcH0XGjEo@lore-desk>
On 07/16, Lorenzo Bianconi wrote:
> On Jul 16, Stanislav Fomichev wrote:
> > On 07/15, Lorenzo Bianconi wrote:
> > > Add a test that verifies skb->ip_summed is set to CHECKSUM_NONE
> > > when a device running in XDP mode creates an skb from a xdp_buff
> > > if the attached ebpf program returns an XDP_PASS.
> > > The test attaches an XDP program returning XDP_PASS, and a TC
> > > ingress program that runs the bpf_skb_rx_checksum() kfunc to
> > > inspect the resulting skb. After XDP_PASS the driver must invalidate
> > > any previously computed hardware RX checksum since XDP may have
> > > modified the packet data.
> > > The BPF program counts packets per checksum type in a map, and the
> > > test runner verifies that after sending traffic the CHECKSUM_NONE
> > > counter is non-zero while CHECKSUM_UNNECESSARY and CHECKSUM_COMPLETE
> > > counters are zero.
> > >
> > > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> > > ---
> > > Documentation/networking/xdp-rx-metadata.rst | 5 ++
> > > .../selftests/drivers/net/hw/xdp_metadata.py | 55 +++++++++++++++-
> > > .../selftests/net/lib/skb_metadata_csum.bpf.c | 73 ++++++++++++++++++++++
> > > 3 files changed, 132 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/Documentation/networking/xdp-rx-metadata.rst b/Documentation/networking/xdp-rx-metadata.rst
> > > index 93918b3769a3..7434ac98242a 100644
> > > --- a/Documentation/networking/xdp-rx-metadata.rst
> > > +++ b/Documentation/networking/xdp-rx-metadata.rst
> > > @@ -90,6 +90,11 @@ conversion, and the XDP metadata is not used by the kernel when building
> > > ``skbs``. However, TC-BPF programs can access the XDP metadata area using
> > > the ``data_meta`` pointer.
> >
> > [..]
> >
> > > +If a driver is running in XDP mode, any existing hardware RX checksum
> > > +(``CHECKSUM_UNNECESSARY`` or ``CHECKSUM_COMPLETE``) must be invalidated
> > > +by setting ``skb->ip_summed`` to ``CHECKSUM_NONE`` before passing the
> > > +skb to the kernel, since XDP may have modified the packet data.
> > > +
> > > In the future, we'd like to support a case where an XDP program
> > > can override some of the metadata used for building ``skbs``.
> >
> > Sorry for keeping nitpicking on this, but I'm still not convinced that
> > it is what we currently do. From my previous reply:
>
> no worries :)
> My current take-away from the previous discussion is we just need to document
> what would be the driver expected behaviour adding a kselftest for it (without
> modifying any driver).
>
> >
> > > > Looking at a few drivers:
> > > > - bnxt (bnxt_rx_pkt) does UNNECESSARY - ok
> > > > - mlx5 (mlx5e_handle_csum) does UNNECESSARY and skips COMPLETE if there is
> > > > bpf prog attached
> > > > - fbnic (fbnic_rx_csum) - can do COMPLETE even with xdp attached?
> > > > - gve (gve_rx) - can do COMPLETE even with xdp attached?
> >
> > (although for gve I might be wrong, there is also gve_rx_skb_csum that only
> > does UNNECESSARY).
> >
> > I'd wait for Jakub to chime in, but it feels like we should just document
> > what we currently do as a recommended approach: for the drivers
> > that support COMPLETE, do not report it when the bpf program is attached.
> > Both NONE and UNNECESSARY are ok.
>
> I am not completely sure the UNNECESSARY case is different from the COMPLETE
> one. What are we supposed to do if the driver reports UNNECESSARY and the ebpf
> program modifies some fields covered by the rx-checksum?
For unnecessary, I think the safe expectation is that the bpf program
will update the value of the checksum in the packet if it touches the data?
> > Also, did you run this test on real HW? NIPA now has HW tests, maybe it
> > makes sense to route this series via net-next to get the real coverage?
>
> What about splitting this series and have two different series:
> - bpf-next: add xdp rx kfunc and related selftest
> - net-next: add kselftest for the driver expected behaviour.
>
> What do you think?
I'd post everything to net-next to get the HW coverage. Once you get all
the acks we can ask the maintainers' guidance.
^ permalink raw reply
* [PATCH v5 00/36] Private Memory NUMA Nodes
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
This series introduces the concept of "Private Memory Nodes", which
are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing
All other features of mm/ are opted out of managing these NUMA
nodes and its memory. Then we add capability bits to allow node
owners to opt back into those services (if supported).
NUMA-hosted memory is presently a most-privilege system (any memory on
any node, except ZONE_DEVICE, is 100% fungible and accessible). We
then slap restrictions on top of it (cpuset, mempolicy, ZONE type,
page/folio flags, etc).
The goal here is to flip that dynamic, isolate by default and then
opt-in to specific services that the device says is safe.
Isolation at the NUMA/Zonelist layer provides a powerful mechanism for
memory hosted on accelerators - re-use of the kernel mm/ code.
- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls
(promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers
can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general
purpose memory get cleaner interfaces (no need to re-write the buddy
in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)
And more use cases I have collected over the past few years.
Not included here is a dax-extension [1] that exposes all the internal
bits as userland controls for testing - along with a pile of selftests
that prove correctness.
Changes Since V4
================
- Massive reduction in complexity.
- no ops struct
- no callback functions
- no __GFP_PRIVATE
- no __GFP_THISNODE requirement
- no task flags (no PF_MEMALLOC_* in the alloc path)
- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)
- ALLOC_ZONELIST_PRIVATE alloc_flag
- on top of Brendan Jackman's recent mm/page_alloc.h work [3]
- Zonelist selection rides the allocator's alloc_flags
- rename OPS -> CAPS (capabilities)
- split base functionality (isolation) from opt-ins (CAPS)
- first half of series can be merged without CAPS
- dropped compressed ram example from series
- will submit separately if this moves forward
- Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
- fully functional dax-kmem extension and huge suite of selftests
located at my github, to be discussed separately [1]
Patch Layout
============
The series is broken into two sections:
1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.
2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private
nodes to make them more useful (i.e. reclaim = overcommit).
NODE_PRIVATE_CAP_LTPIN for private node folio pinning
NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing
NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
My hope is to merge at least #1 pulled ahead while #2 is debated.
Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated
on a given call by 4 things (in order of authority)
1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.
2) __GFP_THISNODE
If this flag is set and the only node in the ZONELIST is the
singular preferred node (or the local node, for -1)
3) cpuset.mems membership
cpuset trims any node in its allowed list
4) mempolicy nodemask
the allocator will skip any node not in the nodemask.
Except for #1 (Zonelist membership) there are all kinds of weird corner
conditions in which 2-4 can be completely ignored (interrupt context,
empty set because cpuset doesn't intersect mempolicy, shared vma, ...)
But ZONELIST membership is *absolute*. If a zone is not in the
zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.
The existing zonelists are constructed like so:
ZONELIST_FALLBACK : All N_MEMORY nodes
ZONELIST_NOFALLBACK : A singleton N_MEMORY node
Private node isolation is implemented via ZONELIST isolation:
ZONELIST_PRIVATE : The private node + N_MEMORY
ZONELIST_PRIVATE_NOFALLBACK : The private node alone.
(mirrors exactly the fallback/nofallback for __GFP_THISNODE)
Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)
1 & 2 mean all existing in-tree callers to page_alloc can NEVER
accidentally allocate from a private node.
An allocation must explicitly ask via a zonelist and a nodemask.
alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */
The page allocator keeps all its original interfaces which only
ever touch the default zonelists - avoiding churn.
Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
=================================================================
The vast majority of the kernel will never need to know about
ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.
When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces
do the zonelist selection for the source of the allocation.
That really is the whole explanation of the mechanism:
alloc_flags = mpol_alloc_flags(pol);
page = __alloc_frozen_pages_noprof(..., alloc_flags);
On mm-new this rides the allocator's existing alloc_flags plumbing:
ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new
parameter, enum, or alloc_context change is required.
For modules that want to implement their own special handling, they
get the _private variants for the page allocator. This lets modules
re-use the buddy instead of rewriting it.
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()
This keeps ALLOC_ flags mm/ internal (these functions add the flags).
Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on
private node memory. Where possible, we even re-use existing
filter points from ZONE_DEVICE.
Most filter points are one or two lines of code:
Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))
Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }
Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;
In the second half of the series, we replace blanket N_MEMORY_PRIVATE
filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that
interaction if CAP is set.
We abstract this with a nice clean interface to make it really clear
what is happening (nodes have features!)
- if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
+ if (!node_allows_reclaim(pgdat->node_id))
NODE_PRIVATE_CAP_* features
===========================
This series of commits opts private nodes into various mm/ services.
Capabilities:
NODE_PRIVATE_CAP_RECLAIM - direct and kswapd reclaim
NODE_PRIVATE_CAP_USER_NUMA - userland numa controls
NODE_PRIVATE_CAP_DEMOTION - node is a demotion target
NODE_PRIVATE_CAP_HOTUNPLUG - hotunplug may migrate
NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally
Some opt-in support is more intensive than others, so these features
are broken out in a way that we can defer them as future work streams.
NODE_PRIVATE_CAP_RECLAIM:
Enabling reclaim for these nodes is actually surprisingly trivial.
Without CAP_RECLAIM, when an allocation failure occurs, the system
will not attempt to swap the memory - and instead will OOM (typically
whatever task is using the most memory on *that* private node).
This capability consists of:
1) enabling kswapd and kcompactd for that node at hotplug time.
2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
3) Sets normal watermarks for these nodes.
4) Allow madvise operations on that node (PAGEOUT).
5) A small tweak to how LRU decides which zones to visit.
NODE_PRIVATE_CAP_USER_NUMA
Enables the following userland interfaces to accept the node:
mbind()
set_mempolicy()
set_mempolicy_home_node()
move_pages()
migrate_pages()
example:
buf = mmap(..., MAP_ANON);
mbind(buf, ..., {private_node});
buf[0] = 0xDEADBEEF; /* Page faults onto the private node */
Later - the KVM example shows how in-kernel mempolicies can
also be bound by CAP_USER_NUMA.
Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.
NODE_PRIVATE_CAP_HOTUNPLUG
This is simple: allow hotunplug to migrate this nodes folios.
Some devices may not be able to tolerate unexpected migrations,
so we prevent hotunplug from engaging in migration by default.
Some devices may have an mmu_notifier in their driver that can
manage the migration and subsequent refault.
CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.
NODE_PRIVATE_CAP_DEMOTION
This adds the private node as a valid demotion target, and allows
reclaim to demote memory from a private node to a demotion target.
Requires: NODE_PRIVATE_CAP_RECLAIM
NODE_PRIVATE_CAP_NUMA_BALANCING
This enables numa balancing to inject prot_none on private node
folio mappings and promote them when faults are taken.
NODE_PRIVATE_CAP_LTPIN
This allows GUP Longterm Pinning to operate normally.
Normally, longterm pinning determines folio eligibility based
on its ZONE_* membership (among other things).
ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require
migration to ZONE_NORMAL before pinning.
Neither operation is preferable by default on a private node,
so the base behavior of FOLL_LONGTERM is to FAIL.
This capability allows longterm pinning to operate normally
based on the ZONE membership. Private node memory may be
hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.
In-tree User: KVM
=================
Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of
enabling disaggregated memory pools to host dedicated KVM memory.
With private nodes, this is trivial (with a bit of basic plumbing):
static int kvm_gmem_bind_node(struct inode *inode, int node)
{
...
/* Bind to a private node - gated on CAP_USER_NUMA */
pol = mpol_bind_node(node);
if (IS_ERR(pol))
return PTR_ERR(pol);
/* Set the shared policy */
err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
...
}
KVM doesn't even need to know about private nodes at all, all it
does is ask mempolicy whether the requested node is a valid bind.
mm/ component testing with dax driver
=====================================
The dax driver extensions[1] implements a simple interface to create
a private node from a dax device created by any source.
I left the dax driver extensions out of this feature set because
it locks in the CAP_ bits before anyone has input. It's there
primarily for testing at this point.
The simplest way to get a dax device is with the memmap= boot arg.
e.g.: "memmap=0x40000000!0x140000000"
The dax driver extension has the following sysfs entries:
dax0.0/private - set the node to private
dax0.0/dax_file - make /dev/dax0.0 mmap'able in kmem mode
dax0.0/adistance - dictate memory_tierN membership
dax0.0/reclaim - CAP_RECLAIM
dax0.0/demotion - CAP_DEMOTION
dax0.0/user_numa - CAP_USER_NUMA
dax0.0/hotunplug - CAP_HOTUNPLUG
dax0.0/numa_balancing - CAP_NUMA_BALANCING
dax0.0/ltpin - CAP_LTPIN
Now consider the following...
Single node reclaim + mbind support:
echo 1 > dax0.0/private
echo 1 > dax0.0/reclaim
echo 1 > dax0.0/user_numa
echo online_movable > dax0.0/state
Test program:
/* node1: 1GB Private Memory Node, 4GB swap */
buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
memset(buf, 0xff, TWO_GB);
We can *guarantee* the ONLY reclaiming tasks are exactly:
- kswapdN (in theory we can even make this optional!)
- the memset task faulting pages in
This also means these are the only tasks capable of becoming
locked up and OOMing (as long as it is not under pressure).
The rest of the system remains entirely functional for debugging.
It now becomes possible to micro-benchmark and A/B test reclaim
changes with different scenarios (number of tasks, amount of
memory, watermark targets, etc), because we have hard controls
over exactly which tasks can access that node memory and how.
If we broke CAP_RECLAIM into subflags:
- CAP_RECLAIM_KSWAPD
- CAP_RECLAIM_DIRECT
- CAP_COMPACTION_KCOMPACTD
- CAP_COMPACTION_DIRECT
We can actually test the efficacy of each of these mechanisms in
isolation to each other - something that is strictly impossible today.
Bonus Configuration: HBM device memory tiering
==============================================
echo 1 > dax0.0/private # make the node private
echo 0 > dax0.0/adistance # highest tier
echo 1 > dax0.0/reclaim # reclaim active
echo 1 > dax0.0/demotion # may demote from the node
echo 1 > dax0.0/user_numa # mbind()
echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled
This is an HBM device which is treated as the top-tier in the
system but for which memory can only enter via explicit mbind().
It can be overcommitted because it can be reclaimed (demotions
go to CPU DRAM, and reclaim can swap from it).
If the HBM is managed by an accelerator (GPU), the mmu_notifier
allows it to know when reclaim is moving memory out to do
device-mmu invalidation prior to migration.
Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]
Prereqs (all already in akpm/mm-new; listed for out-of-tree application):
page_alloc.h split + alloc_flags plumbing this series rides on:
commit e81fae43cd69 ("mm: split out internal page_alloc.h")
commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")
dax atomic whole-device hotplug (used by the dax extension [1]):
commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper")
commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")
[1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
[2] https://lore.kernel.org/all/20260423170219.281618-1-dave.jiang@intel.com/
[3] https://lore.kernel.org/all/20260702-alloc-trylock-v4-0-0af8ff387e80@google.com/
base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e
Gregory Price (36):
mm: refactor find_next_best_node to find_next_best_node_in
mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
mm/page_alloc: let the bulk and folio allocators carry alloc_flags
numa: introduce N_MEMORY_PRIVATE
mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
cpuset: exclude private nodes from cpuset.mems (default-open)
mm/memory_hotplug: disallow migration-driven private node hotunplug
mm/mempolicy: skip private node folios when queueing for migration
mm/migrate: disallow userland driven migration for private nodes
mm/madvise: disallow madvise operations on private node folios
mm/compaction: disallow compaction on private nodes
mm/page_alloc: clear private node watermarks and system reserves
mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
mm/damon: skip private node memory in DAMON migration and pageout
mm/ksm: skip KSM for managed-memory folios
mm/khugepaged: skip private node folios when trying to collapse.
mm/vmscan: disallow reclaim of private node memory
mm/gup: disallow longterm pin of private node folios
proc: include N_MEMORY_PRIVATE nodes in numa_maps output
mm/memcontrol: account private-node memory in per-node stats
proc/kcore: include private-node RAM in the kcore RAM map
mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection
mm/mempolicy: apply policy at the kernel zone for private-node binds
mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services
mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
balancing
mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning
mm/khugepaged: base private node collapse eligiblity on actor/cap bits
Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes
mm/mempolicy: add mpol_set_shared_policy_range()
KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test
Documentation/ABI/stable/sysfs-devices-node | 10 +
Documentation/mm/index.rst | 1 +
Documentation/mm/numa_private_nodes.rst | 160 ++++++++++
drivers/base/node.c | 118 +++++++
drivers/dax/kmem.c | 2 +-
fs/proc/kcore.c | 5 +-
fs/proc/task_mmu.c | 10 +-
include/linux/gfp.h | 22 +-
include/linux/kvm_host.h | 3 +
include/linux/memory_hotplug.h | 5 +-
include/linux/mempolicy.h | 16 +
include/linux/mmzone.h | 26 +-
include/linux/node_private.h | 262 ++++++++++++++++
include/linux/nodemask.h | 7 +-
include/uapi/linux/kvm.h | 5 +-
include/uapi/linux/mempolicy.h | 1 +
kernel/cgroup/cpuset.c | 26 +-
mm/compaction.c | 13 +
mm/damon/paddr.c | 9 +
mm/gup.c | 28 +-
mm/huge_memory.c | 5 +
mm/internal.h | 97 +++++-
mm/khugepaged.c | 18 +-
mm/ksm.c | 8 +-
mm/madvise.c | 8 +-
mm/memcontrol-v1.c | 8 +-
mm/memcontrol.c | 13 +-
mm/memory-tiers.c | 40 ++-
mm/memory_hotplug.c | 119 +++++++-
mm/mempolicy.c | 288 +++++++++++++++---
mm/migrate.c | 19 +-
mm/mm_init.c | 2 +-
mm/page_alloc.c | 189 +++++++++---
mm/page_alloc.h | 34 +++
mm/vmscan.c | 57 +++-
tools/testing/selftests/kvm/Makefile.kvm | 2 +
.../kvm/guest_memfd_bind_node_test.c | 213 +++++++++++++
virt/kvm/guest_memfd.c | 39 ++-
38 files changed, 1728 insertions(+), 160 deletions(-)
create mode 100644 Documentation/mm/numa_private_nodes.rst
create mode 100644 include/linux/node_private.h
create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
--
2.53.0-Meta
^ permalink raw reply
* [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
find_next_best_node() picks the next-closest node for a fallback list
from the full N_MEMORY set. Refactor it into find_next_best_node_in(),
which takes an explicit candidates nodemask.
This enables building fallback lists with non-N_MEMORY candidates.
No functional change: every caller still selects from N_MEMORY.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/internal.h | 6 ++++--
mm/memory-tiers.c | 7 ++++---
mm/page_alloc.c | 13 ++++++++-----
3 files changed, 16 insertions(+), 10 deletions(-)
diff --git a/mm/internal.h b/mm/internal.h
index f26423de4ca28..96d78a7778e88 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -1103,7 +1103,8 @@ extern int node_reclaim_mode;
extern unsigned long node_reclaim(struct pglist_data *pgdat,
gfp_t gfp_mask, unsigned int order);
-extern int find_next_best_node(int node, nodemask_t *used_node_mask);
+extern int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates);
#else
#define node_reclaim_mode 0
@@ -1112,7 +1113,8 @@ static inline unsigned long node_reclaim(struct pglist_data *pgdat,
{
return 0;
}
-static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
+static inline int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates)
{
return NUMA_NO_NODE;
}
diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
index 54851d8a195b0..25e121851b586 100644
--- a/mm/memory-tiers.c
+++ b/mm/memory-tiers.c
@@ -370,7 +370,7 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
* closest demotion target.
*/
nodes_complement(mask, *allowed_mask);
- return find_next_best_node(node, &mask);
+ return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
}
static void disable_all_demotion_targets(void)
@@ -450,7 +450,7 @@ static void establish_demotion_targets(void)
memtier = list_next_entry(memtier, list);
tier_nodes = get_memtier_nodemask(memtier);
/*
- * find_next_best_node, use 'used' nodemask as a skip list.
+ * find_next_best_node_in, use 'used' nodemask as a skip list.
* Add all memory nodes except the selected memory tier
* nodelist to skip list so that we find the best node from the
* memtier nodelist.
@@ -463,7 +463,8 @@ static void establish_demotion_targets(void)
* in the preferred mask when allocating pages during demotion.
*/
do {
- target = find_next_best_node(node, &tier_nodes);
+ target = find_next_best_node_in(node, &tier_nodes,
+ &node_states[N_MEMORY]);
if (target == NUMA_NO_NODE)
break;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index f93a6bb9a872d..acf096f525f49 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5743,9 +5743,10 @@ static int numa_zonelist_order_handler(const struct ctl_table *table, int write,
static int node_load[MAX_NUMNODES];
/**
- * find_next_best_node - find the next node that should appear in a given node's fallback list
+ * find_next_best_node_in - find the next node that should appear in a given node's fallback list
* @node: node whose fallback list we're appending
* @used_node_mask: nodemask_t of already used nodes
+ * @candidates: nodemask_t of nodes eligible for selection
*
* We use a number of factors to determine which is the next node that should
* appear on a given node's fallback list. The node should not have appeared
@@ -5757,7 +5758,8 @@ static int node_load[MAX_NUMNODES];
*
* Return: node id of the found node or %NUMA_NO_NODE if no node is found.
*/
-int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+ const nodemask_t *candidates)
{
int n, val;
int min_val = INT_MAX;
@@ -5767,12 +5769,12 @@ int find_next_best_node(int node, nodemask_t *used_node_mask)
* Use the local node if we haven't already, but for memoryless local
* node, we should skip it and fall back to other nodes.
*/
- if (!node_isset(node, *used_node_mask) && node_state(node, N_MEMORY)) {
+ if (!node_isset(node, *used_node_mask) && node_isset(node, *candidates)) {
node_set(node, *used_node_mask);
return node;
}
- for_each_node_state(n, N_MEMORY) {
+ for_each_node_mask(n, *candidates) {
/* Don't want a node to appear more than once */
if (node_isset(n, *used_node_mask))
@@ -5857,7 +5859,8 @@ static void build_zonelists(pg_data_t *pgdat)
prev_node = local_node;
memset(node_order, 0, sizeof(node_order));
- while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
+ while ((node = find_next_best_node_in(local_node, &used_mask,
+ &node_states[N_MEMORY])) >= 0) {
/*
* We don't want to pressure a particular node.
* So adding penalty to the first node in same
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 02/36] mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Extract per-node fallback-list construction into build_node_zonelist().
This lets use build zonelists from candidate nodemasks instead of just
the default N_MEMORY node state list.
No functional change: build_zonelists() builds the same FALLBACK list over
N_MEMORY with node_load updates as before.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/page_alloc.c | 44 ++++++++++++++++++++++++++++++--------------
1 file changed, 30 insertions(+), 14 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index acf096f525f49..4c1ef84c02a3d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5813,12 +5813,12 @@ int find_next_best_node_in(int node, nodemask_t *used_node_mask,
* DMA zone, if any--but risks exhausting DMA zone.
*/
static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
- unsigned nr_nodes)
+ unsigned int nr_nodes, int zlidx)
{
struct zoneref *zonerefs;
int i;
- zonerefs = pgdat->node_zonelists[ZONELIST_FALLBACK]._zonerefs;
+ zonerefs = pgdat->node_zonelists[zlidx]._zonerefs;
for (i = 0; i < nr_nodes; i++) {
int nr_zones;
@@ -5847,26 +5847,28 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)
zonerefs->zone_idx = 0;
}
-static void build_zonelists(pg_data_t *pgdat)
+/*
+ * Build one node-ordered fallback list from a candidate nodemask.
+ * update_load round-robins node_load across equidistant nodes.
+ */
+static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
+ int zlidx, bool update_load,
+ int *node_order, int *nr)
{
- static int node_order[MAX_NUMNODES];
- int node, nr_nodes = 0;
nodemask_t used_mask = NODE_MASK_NONE;
- int local_node, prev_node;
-
- /* NUMA-aware ordering of nodes */
- local_node = pgdat->node_id;
- prev_node = local_node;
+ int local_node = pgdat->node_id;
+ int prev_node = local_node;
+ int node, nr_nodes = 0;
- memset(node_order, 0, sizeof(node_order));
while ((node = find_next_best_node_in(local_node, &used_mask,
- &node_states[N_MEMORY])) >= 0) {
+ candidates)) >= 0) {
/*
* We don't want to pressure a particular node.
* So adding penalty to the first node in same
* distance group to make it round-robin.
*/
- if (node_distance(local_node, node) !=
+ if (update_load &&
+ node_distance(local_node, node) !=
node_distance(local_node, prev_node))
node_load[node] += 1;
@@ -5874,8 +5876,22 @@ static void build_zonelists(pg_data_t *pgdat)
prev_node = node;
}
- build_zonelists_in_node_order(pgdat, node_order, nr_nodes);
+ build_zonelists_in_node_order(pgdat, node_order, nr_nodes, zlidx);
+ *nr = nr_nodes;
+}
+
+static void build_zonelists(pg_data_t *pgdat)
+{
+ static int node_order[MAX_NUMNODES];
+ int local_node = pgdat->node_id;
+ int node, nr_nodes = 0;
+
+ memset(node_order, 0, sizeof(node_order));
+
+ build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
+ true, node_order, &nr_nodes);
build_thisnode_zonelists(pgdat);
+
pr_info("Fallback order for Node %d: ", local_node);
for (node = 0; node < nr_nodes; node++)
pr_cont("%d ", node_order[node]);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
__alloc_pages_noprof() takes an explicit alloc_flags, but the bulk and
folio entry points do not, so callers cannot select allocator behaviour
(e.g. an alternate zonelist) through them.
Thread alloc_flags through both, matching __alloc_pages_noprof(), and
keep the flag-carrying primitives mm-internal (page_alloc.h) so the
public gfp.h wrappers stay flag-free:
- add __alloc_pages_bulk_noprof(gfp, alloc_flags, ...) in page_alloc.h
alloc_pages_bulk_noprof() becomes a wrapper passing ALLOC_DEFAULT
- give __folio_alloc_noprof() an alloc_flags parameter and moves
__folio_alloc_node_noprof() moves into page_alloc.c
__folio_alloc_noprof() is no longer exported
No functional change: every caller passes ALLOC_DEFAULT.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/gfp.h | 13 +------------
mm/khugepaged.c | 2 +-
mm/migrate.c | 2 +-
mm/page_alloc.c | 46 ++++++++++++++++++++++++++++++---------------
mm/page_alloc.h | 8 ++++++++
5 files changed, 42 insertions(+), 29 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 872bc53f32ec8..a327e58c313f8 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }
static inline void arch_alloc_page(struct page *page, int order) { }
#endif
-struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
- nodemask_t *nodemask);
-#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
-
unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
struct page **page_array);
@@ -252,14 +248,7 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)
dump_stack();
}
-static inline
-struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
-{
- warn_if_node_offline(nid, gfp);
-
- return __folio_alloc_noprof(gfp, order, nid, NULL);
-}
-
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid);
#define __folio_alloc_node(...) alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__))
/*
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 27e8f3077e80f..89ce6bcbc376b 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -1243,7 +1243,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
int node = collapse_find_target_node(cc);
struct folio *folio;
- folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
+ folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
if (!folio) {
*foliop = NULL;
if (is_pmd_order(order))
diff --git a/mm/migrate.c b/mm/migrate.c
index 60355be9f652c..a8a86141dbb54 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2231,7 +2231,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
gfp_mask |= __GFP_HIGHMEM;
- return __folio_alloc(gfp_mask, order, nid, mtc->nmask);
+ return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
}
#ifdef CONFIG_NUMA_MIGRATION
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4c1ef84c02a3d..aa35bbe5d4c3e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5103,8 +5103,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
* @page_array were set to %NULL on entry, the slots from 0 to the return value
* - 1 will be filled.
*/
-unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
- nodemask_t *nodemask, int nr_pages,
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+ int preferred_nid, nodemask_t *nodemask, int nr_pages,
struct page **page_array)
{
struct page *page;
@@ -5112,8 +5112,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct zoneref *z;
struct per_cpu_pages *pcp;
struct list_head *pcp_list;
- struct alloc_context ac;
- unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ struct alloc_context ac = { .alloc_flags = alloc_flags };
+ unsigned int fastpath_alloc_flags = alloc_flags | ALLOC_WMARK_LOW;
int nr_populated = 0, nr_account = 0;
/*
@@ -5153,7 +5153,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
/* May set ALLOC_NOFRAGMENT, fragmentation will return 1 page. */
gfp &= gfp_allowed_mask;
- if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &alloc_flags))
+ if (!prepare_alloc_pages(gfp, 0, preferred_nid, nodemask, &ac, &gfp, &fastpath_alloc_flags))
goto out;
/* Find an allowed local zone that meets the low watermark. */
@@ -5161,7 +5161,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
for_next_zone_zonelist_nodemask(zone, z, ac.highest_zoneidx, ac.nodemask) {
unsigned long mark;
- if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
+ if (cpusets_enabled() && (fastpath_alloc_flags & ALLOC_CPUSET) &&
!__cpuset_zone_allowed(zone, gfp)) {
continue;
}
@@ -5171,16 +5171,17 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
goto failed;
}
- cond_accept_memory(zone, 0, alloc_flags);
+ cond_accept_memory(zone, 0, fastpath_alloc_flags);
retry_this_zone:
- mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK) + nr_pages - nr_populated;
+ mark = wmark_pages(zone, fastpath_alloc_flags & ALLOC_WMARK_MASK) +
+ nr_pages - nr_populated;
if (zone_watermark_fast(zone, 0, mark,
zonelist_zone_idx(ac.preferred_zoneref),
- alloc_flags, gfp)) {
+ fastpath_alloc_flags, gfp)) {
break;
}
- if (cond_accept_memory(zone, 0, alloc_flags))
+ if (cond_accept_memory(zone, 0, fastpath_alloc_flags))
goto retry_this_zone;
/* Try again if zone has deferred pages */
@@ -5212,7 +5213,7 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
continue;
}
- page = __rmqueue_pcplist(zone, 0, ac.migratetype, alloc_flags,
+ page = __rmqueue_pcplist(zone, 0, ac.migratetype, fastpath_alloc_flags,
pcp, pcp_list);
if (unlikely(!page)) {
/* Try and allocate at least one page */
@@ -5238,11 +5239,19 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
return nr_populated;
failed:
- page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, ALLOC_DEFAULT);
+ page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, alloc_flags);
if (page)
page_array[nr_populated++] = page;
goto out;
}
+
+unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
+ nodemask_t *nodemask, int nr_pages,
+ struct page **page_array)
+{
+ return __alloc_pages_bulk_noprof(gfp, ALLOC_DEFAULT, preferred_nid,
+ nodemask, nr_pages, page_array);
+}
EXPORT_SYMBOL_GPL(alloc_pages_bulk_noprof);
/*
@@ -5433,13 +5442,20 @@ struct page *alloc_pages_node_noprof(int nid, gfp_t gfp_mask, unsigned int order
EXPORT_SYMBOL(alloc_pages_node_noprof);
struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
- nodemask_t *nodemask)
+ nodemask_t *nodemask, unsigned int alloc_flags)
{
struct page *page = __alloc_pages_noprof(gfp | __GFP_COMP, order,
- preferred_nid, nodemask, ALLOC_DEFAULT);
+ preferred_nid, nodemask, alloc_flags);
return page_rmappable_folio(page);
}
-EXPORT_SYMBOL(__folio_alloc_noprof);
+
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
+{
+ warn_if_node_offline(nid, gfp);
+
+ return __folio_alloc_noprof(gfp, order, nid, NULL, ALLOC_DEFAULT);
+}
+EXPORT_SYMBOL(__folio_alloc_node_noprof);
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index b9259deddb59d..23fc79ce97b60 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -258,6 +258,14 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_n
nodemask_t *nodemask, unsigned int alloc_flags);
#define __alloc_pages(...) alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))
+struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
+ nodemask_t *nodemask, unsigned int alloc_flags);
+#define __folio_alloc(...) alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
+
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+ int preferred_nid, nodemask_t *nodemask, int nr_pages,
+ struct page **page_array);
+
extern void zone_pcp_reset(struct zone *zone);
extern void zone_pcp_disable(struct zone *zone);
extern void zone_pcp_enable(struct zone *zone);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Some devices want to hotplug their memory onto a node, but not have it
exposed as "general purpose". The intent is to re-use portions of mm/
services for management without exposing it for general consumption.
N_MEMORY nodes are intended to contain general System RAM, so they
are unsuited for this purpose.
Create N_MEMORY_PRIVATE for memory nodes whose memory is not intended
for general consumption.
N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive states.
This commit adds basic infrastructure for N_MEMORY_PRIVATE nodes:
- struct node_private:
Per-node container stored in NODE_DATA(nid) storing the owner.
- folio/page_is_private_node():
true if it resides on a private node
- folio/page_is_private_managed():
common predicate for checking private node or zone device
Used to combine common filtering locations.
- Registration API: node_private_register()/unregister()
- sysfs attribute exposing N_MEMORY_PRIVATE node state.
- fix node_state() and helpers to return false for N_MEMORY_PRIVATE
when !CONFIG_NUMA (mutually exclusive w/ N_MEMORY)
Signed-off-by: Gregory Price <gourry@gourry.net>
---
Documentation/ABI/stable/sysfs-devices-node | 10 ++
drivers/base/node.c | 113 ++++++++++++++++++++
include/linux/mmzone.h | 15 +++
include/linux/node_private.h | 82 ++++++++++++++
include/linux/nodemask.h | 7 +-
mm/internal.h | 12 +++
6 files changed, 236 insertions(+), 3 deletions(-)
create mode 100644 include/linux/node_private.h
diff --git a/Documentation/ABI/stable/sysfs-devices-node b/Documentation/ABI/stable/sysfs-devices-node
index 2d0e023f22a71..8a36fbc3d4028 100644
--- a/Documentation/ABI/stable/sysfs-devices-node
+++ b/Documentation/ABI/stable/sysfs-devices-node
@@ -29,6 +29,16 @@ Description:
Nodes that have regular or high memory.
Depends on CONFIG_HIGHMEM.
+What: /sys/devices/system/node/has_private_memory
+Contact: Linux Memory Management list <linux-mm@kvack.org>
+Description:
+ Nodes that have private (N_MEMORY_PRIVATE) memory: memory
+ hotplugged by a driver onto a CPU-less node and isolated from
+ the page allocator's normal and fallback zonelists. A node is
+ listed here for as long as it holds private memory; it is never
+ listed in has_memory at the same time (the two states are
+ mutually exclusive). See Documentation/mm/numa_private_nodes.rst.
+
What: /sys/devices/system/node/nodeX
Date: October 2002
Contact: Linux Memory Management list <linux-mm@kvack.org>
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 3da91929ad4e3..94cd51f51b7e8 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -22,6 +22,7 @@
#include <linux/swap.h>
#include <linux/slab.h>
#include <linux/memblock.h>
+#include <linux/node_private.h>
static const struct bus_type node_subsys = {
.name = "node",
@@ -868,6 +869,116 @@ void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn,
(void *)&nid, register_mem_block_under_node_hotplug);
return;
}
+
+static DEFINE_MUTEX(node_private_lock);
+
+/**
+ * node_private_register - Register a private node
+ * @nid: Node identifier
+ * @np: The node_private structure (driver-allocated, driver-owned)
+ *
+ * Register an owner for a private node. If an owner is already registered
+ * (different np), return -EBUSY.
+ *
+ * Re-registration with the same np is allowed.
+ *
+ * The caller owns the node_private memory and must ensure it remains valid
+ * until after node_private_unregister() returns.
+ *
+ * Returns 0 on success, negative errno on failure.
+ */
+int node_private_register(int nid, struct node_private *np)
+{
+ struct node_private *existing;
+ pg_data_t *pgdat;
+ int ret = 0;
+
+ if (!np || !node_possible(nid))
+ return -EINVAL;
+
+ mutex_lock(&node_private_lock);
+ mem_hotplug_begin();
+
+ /* N_MEMORY_PRIVATE and N_MEMORY are mutually exclusive */
+ if (node_state(nid, N_MEMORY)) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ pgdat = NODE_DATA(nid);
+ existing = rcu_dereference_protected(pgdat->node_private,
+ lockdep_is_held(&node_private_lock));
+
+ /* If it exists, restrict to single owner */
+ if (existing) {
+ if (existing != np)
+ ret = -EBUSY;
+ goto out;
+ }
+
+ rcu_assign_pointer(pgdat->node_private, np);
+out:
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(node_private_register);
+
+/**
+ * node_private_unregister - Unregister a private node
+ * @nid: Node identifier
+ *
+ * Unregister the driver from a private node.
+ *
+ * Only succeeds if all memory has been offlined (N_MEMORY_PRIVATE cleared).
+ *
+ * N_MEMORY_PRIVATE state is cleared by offline_pages() when the last
+ * memory is offlined, not by this function.
+ *
+ * After this returns, the node_private pointer is no longer visible and
+ * an RCU grace period has elapsed, so the driver may free its context.
+ *
+ * Return: 0 if unregistered, -EBUSY if N_MEMORY_PRIVATE is still set.
+ */
+int node_private_unregister(int nid)
+{
+ struct node_private *np;
+ pg_data_t *pgdat;
+
+ if (!node_possible(nid))
+ return 0;
+
+ mutex_lock(&node_private_lock);
+ mem_hotplug_begin();
+
+ pgdat = NODE_DATA(nid);
+ np = rcu_dereference_protected(pgdat->node_private,
+ lockdep_is_held(&node_private_lock));
+ if (!np) {
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return 0;
+ }
+
+ /*
+ * Only unregister if N_MEMORY_PRIVATE is cleared (only occurs when
+ * the last memory block is offlined in offline_pages())
+ */
+ if (node_is_private(nid)) {
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+ return -EBUSY;
+ }
+
+ rcu_assign_pointer(pgdat->node_private, NULL);
+
+ mem_hotplug_done();
+ mutex_unlock(&node_private_lock);
+
+ synchronize_rcu();
+ return 0;
+}
+EXPORT_SYMBOL_GPL(node_private_unregister);
#endif /* CONFIG_MEMORY_HOTPLUG */
/**
@@ -966,6 +1077,7 @@ static struct node_attr node_state_attr[] = {
[N_HIGH_MEMORY] = _NODE_ATTR(has_high_memory, N_HIGH_MEMORY),
#endif
[N_MEMORY] = _NODE_ATTR(has_memory, N_MEMORY),
+ [N_MEMORY_PRIVATE] = _NODE_ATTR(has_private_memory, N_MEMORY_PRIVATE),
[N_CPU] = _NODE_ATTR(has_cpu, N_CPU),
[N_GENERIC_INITIATOR] = _NODE_ATTR(has_generic_initiator,
N_GENERIC_INITIATOR),
@@ -979,6 +1091,7 @@ static struct attribute *node_state_attrs[] = {
&node_state_attr[N_HIGH_MEMORY].attr.attr,
#endif
&node_state_attr[N_MEMORY].attr.attr,
+ &node_state_attr[N_MEMORY_PRIVATE].attr.attr,
&node_state_attr[N_CPU].attr.attr,
&node_state_attr[N_GENERIC_INITIATOR].attr.attr,
NULL
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0507193b3ae34..9815e48c03b97 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -26,6 +26,8 @@
#include <linux/sizes.h>
#include <asm/page.h>
+struct node_private;
+
/* Free memory management - zoned buddy allocator. */
#ifndef CONFIG_ARCH_FORCE_MAX_ORDER
#define MAX_PAGE_ORDER 10
@@ -1596,12 +1598,25 @@ typedef struct pglist_data {
atomic_long_t vm_stat[NR_VM_NODE_STAT_ITEMS];
#ifdef CONFIG_NUMA
struct memory_tier __rcu *memtier;
+ struct node_private __rcu *node_private;
#endif
#ifdef CONFIG_MEMORY_FAILURE
struct memory_failure_stats mf_stats;
#endif
} pg_data_t;
+#ifdef CONFIG_NUMA
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+ return !!pgdat->node_private;
+}
+#else
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+ return false;
+}
+#endif
+
#define node_present_pages(nid) (NODE_DATA(nid)->node_present_pages)
#define node_spanned_pages(nid) (NODE_DATA(nid)->node_spanned_pages)
diff --git a/include/linux/node_private.h b/include/linux/node_private.h
new file mode 100644
index 0000000000000..475496c84249f
--- /dev/null
+++ b/include/linux/node_private.h
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_NODE_PRIVATE_H
+#define _LINUX_NODE_PRIVATE_H
+
+#include <linux/mm.h>
+#include <linux/nodemask.h>
+
+struct page;
+
+/**
+ * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
+ *
+ * Allocated by the driver and passed to node_private_register().
+ * The driver owns the memory and must ensure it remains valid until after
+ * node_private_unregister() returns.
+ *
+ * @owner: Opaque driver identifier
+ * @caps: NODE_PRIVATE_CAP_* service opt-ins for the node (zero by default;
+ * individual capabilities are defined and consumed by later changes)
+ */
+struct node_private {
+ void *owner;
+ unsigned long caps;
+};
+
+#ifdef CONFIG_NUMA
+#include <linux/mmzone.h>
+
+static inline bool folio_is_private_node(struct folio *folio)
+{
+ return node_state(folio_nid(folio), N_MEMORY_PRIVATE);
+}
+
+static inline bool page_is_private_node(struct page *page)
+{
+ return node_state(page_to_nid(page), N_MEMORY_PRIVATE);
+}
+
+static inline bool node_is_private(int nid)
+{
+ return node_state(nid, N_MEMORY_PRIVATE);
+}
+
+#else /* !CONFIG_NUMA */
+
+static inline bool folio_is_private_node(struct folio *folio)
+{
+ return false;
+}
+
+static inline bool page_is_private_node(struct page *page)
+{
+ return false;
+}
+
+static inline bool node_is_private(int nid)
+{
+ return false;
+}
+
+#endif /* CONFIG_NUMA */
+
+#if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
+
+int node_private_register(int nid, struct node_private *np);
+int node_private_unregister(int nid);
+
+#else /* !CONFIG_NUMA || !CONFIG_MEMORY_HOTPLUG */
+
+static inline int node_private_register(int nid, struct node_private *np)
+{
+ return -ENODEV;
+}
+
+static inline int node_private_unregister(int nid)
+{
+ return 0;
+}
+
+#endif /* CONFIG_NUMA && CONFIG_MEMORY_HOTPLUG */
+
+#endif /* _LINUX_NODE_PRIVATE_H */
diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h
index b842aa5255464..ba3e6c570a112 100644
--- a/include/linux/nodemask.h
+++ b/include/linux/nodemask.h
@@ -391,6 +391,7 @@ enum node_states {
N_HIGH_MEMORY = N_NORMAL_MEMORY,
#endif
N_MEMORY, /* The node has memory(regular, high, movable) */
+ N_MEMORY_PRIVATE, /* The node's memory is private */
N_CPU, /* The node has one or more cpus */
N_GENERIC_INITIATOR, /* The node has one or more Generic Initiators */
NR_NODE_STATES
@@ -457,7 +458,7 @@ static __always_inline void node_set_offline(int nid)
static __always_inline int node_state(int node, enum node_states state)
{
- return node == 0;
+ return node == 0 && state != N_MEMORY_PRIVATE;
}
static __always_inline void node_set_state(int node, enum node_states state)
@@ -470,11 +471,11 @@ static __always_inline void node_clear_state(int node, enum node_states state)
static __always_inline int num_node_state(enum node_states state)
{
- return 1;
+ return state == N_MEMORY_PRIVATE ? 0 : 1;
}
#define for_each_node_state(node, __state) \
- for ( (node) = 0; (node) == 0; (node) = 1)
+ for ((node) = 0; (node) == 0 && (__state) != N_MEMORY_PRIVATE; (node) = 1)
#define first_online_node 0
#define first_memory_node 0
diff --git a/mm/internal.h b/mm/internal.h
index 96d78a7778e88..cb9f4a8342e32 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/mm_inline.h>
#include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
#include <linux/pagemap.h>
#include <linux/pagewalk.h>
#include <linux/rmap.h>
@@ -98,6 +99,17 @@ extern int sysctl_min_unmapped_ratio;
extern int sysctl_min_slab_ratio;
#endif
+/* folio_is_private_managed() - folio has special mm management rules */
+static inline bool folio_is_private_managed(struct folio *folio)
+{
+ return folio_is_zone_device(folio) || folio_is_private_node(folio);
+}
+
+static inline bool page_is_private_managed(struct page *page)
+{
+ return folio_is_private_managed(page_folio(page));
+}
+
/*
* Maintains state across a page table move. The operation assumes both source
* and destination VMAs already exist and are specified by the user.
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
From: Gregory Price @ 2026-07-20 19:33 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
N_MEMORY_PRIVATE nodes must be invisible to ordinary allocation,
no allocations may reach one unless explicitly called for.
Make this isolation structural in the zonelists: N_MEMORY_PRIVATE
nodes are excluded from FALLBACK and NOFALLBACK entirely, making
them effectively invisible to all normal callers of page_alloc.
Add ZONELIST_PRIVATE, which spans (N_MEMORY | N_MEMORY_PRIVATE).
When !CONFIG_NUMA, it aliases ZONELIST_FALLBACK and compiles out.
Add ZONELIST_PRIVATE_NOFALLBACK as the __GFP_THISNODE target for
private node allocations.
Zonelist selection rides the allocator's alloc_flags. Add
ALLOC_ZONELIST_PRIVATE and resolve it in select_zonelist(), which
prepare_alloc_pages() applies from ac->alloc_flags; the default
(ALLOC_DEFAULT) keeps node_zonelist()'s gfp-based selection.
Add explicit private-node alloc() functions to prevent open-coding
(both ride ALLOC_ZONELIST_PRIVATE through the alloc_flags-carrying
entry points):
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()
alloc_contig_range adds an explicit node_is_private() guard because
it dervices its target zones from PFNs rather than zonelists. All
in-tree callers use N_MEMORY ranges, but the check prevents future
callers from violating node isolation.
Important note: By design this zonelist isolates N_MEMORY_PRIVATE
from unintentional allocations, but does NOT isolate private-node
allocations from falling back to non-private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
include/linux/gfp.h | 11 ++++++++
include/linux/mmzone.h | 11 +++++++-
mm/mm_init.c | 2 +-
mm/page_alloc.c | 64 ++++++++++++++++++++++++++++++++++++++++--
mm/page_alloc.h | 26 +++++++++++++++++
5 files changed, 109 insertions(+), 5 deletions(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index a327e58c313f8..f3e867de744f6 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -204,6 +204,17 @@ static inline void arch_free_page(struct page *page, int order) { }
static inline void arch_alloc_page(struct page *page, int order) { }
#endif
+/* Allocate from an N_MEMORY_PRIVATE node (selects the private zonelist). */
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid);
+#define alloc_pages_node_private(...) \
+ alloc_hooks(alloc_pages_node_private_noprof(__VA_ARGS__))
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid);
+#define folio_alloc_node_private(...) \
+ alloc_hooks(folio_alloc_node_private_noprof(__VA_ARGS__))
+
unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
nodemask_t *nodemask, int nr_pages,
struct page **page_array);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 9815e48c03b97..654c5111f7cec 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1392,13 +1392,22 @@ enum {
#ifdef CONFIG_NUMA
/*
* The NUMA zonelists are doubled because we need zonelists that
- * restrict the allocations to a single node for __GFP_THISNODE.
+ * restrict the allocations to a single node for __GFP_THISNODE
+ * and N_MEMORY_PRIVATE nodes (isolated from default lists).
*/
ZONELIST_NOFALLBACK, /* zonelist without fallback (__GFP_THISNODE) */
+ ZONELIST_PRIVATE, /* N_MEMORY_PRIVATE access, falls back to DRAM */
+ ZONELIST_PRIVATE_NOFALLBACK, /* N_MEMORY_PRIVATE access, __GFP_THISNODE */
#endif
MAX_ZONELISTS
};
+#ifndef CONFIG_NUMA
+/* Without NUMA only ZONELIST_FALLBACK exists so everything collapses there */
+#define ZONELIST_PRIVATE ZONELIST_FALLBACK
+#define ZONELIST_PRIVATE_NOFALLBACK ZONELIST_FALLBACK
+#endif
+
/*
* This struct contains information about a zone in a zonelist. It is stored
* here to avoid dereferences into large structures and lookups of tables
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 711f821f7b3c7..adb50647d29ca 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -2701,7 +2701,7 @@ void __init mm_core_init(void)
init_zero_page_pfn();
/* Initializations relying on SMP setup */
- BUILD_BUG_ON(MAX_ZONELISTS > 2);
+ BUILD_BUG_ON(MAX_ZONELISTS > 4);
build_all_zonelists(NULL);
page_alloc_init_cpuhp();
alloc_tag_sec_init();
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index aa35bbe5d4c3e..78755a55b1540 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5040,7 +5040,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
unsigned int *alloc_flags)
{
ac->highest_zoneidx = gfp_zone(gfp_mask);
- ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
+ ac->zonelist = select_zonelist(preferred_nid, gfp_mask, ac->alloc_flags);
ac->nodemask = nodemask;
ac->migratetype = gfp_migratetype(gfp_mask);
@@ -5346,7 +5346,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
unsigned int fastpath_alloc_flags = alloc_flags;
/* Other flags could be supported later if needed. */
- if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
+ if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG | ALLOC_ZONELIST_PRIVATE)))
return NULL;
if (!alloc_order_allowed(gfp, order, alloc_flags))
@@ -5457,6 +5457,22 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
}
EXPORT_SYMBOL(__folio_alloc_node_noprof);
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid)
+{
+ return __alloc_pages_noprof(gfp, order, nid, NULL,
+ ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(alloc_pages_node_private_noprof);
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+ int nid)
+{
+ return __folio_alloc_noprof(gfp, order, nid, NULL,
+ ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(folio_alloc_node_private_noprof);
+
/*
* Common helper functions. Never use with __GFP_HIGHMEM because the returned
* address cannot represent highmem pages. Use alloc_pages and then kmap if
@@ -5854,9 +5870,21 @@ static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
static void build_thisnode_zonelists(pg_data_t *pgdat)
{
struct zoneref *zonerefs;
- int nr_zones;
+ int nr_zones = 0;
+ /*
+ * NOFALLBACK: the node's own zones. EMPTY for private nodes so a
+ * stray __GFP_THISNODE allocation can't violate isolation.
+ */
zonerefs = pgdat->node_zonelists[ZONELIST_NOFALLBACK]._zonerefs;
+ if (!node_is_private(pgdat->node_id))
+ nr_zones = build_zonerefs_node(pgdat, zonerefs);
+ zonerefs += nr_zones;
+ zonerefs->zone = NULL;
+ zonerefs->zone_idx = 0;
+
+ /* PRIVATE_NOFALLBACK: the node's own zones, private nodes included. */
+ zonerefs = pgdat->node_zonelists[ZONELIST_PRIVATE_NOFALLBACK]._zonerefs;
nr_zones = build_zonerefs_node(pgdat, zonerefs);
zonerefs += nr_zones;
zonerefs->zone = NULL;
@@ -5899,11 +5927,28 @@ static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
static void build_zonelists(pg_data_t *pgdat)
{
static int node_order[MAX_NUMNODES];
+ nodemask_t tier_nodes;
int local_node = pgdat->node_id;
int node, nr_nodes = 0;
memset(node_order, 0, sizeof(node_order));
+ /*
+ * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
+ *
+ * FALLBACK: Allocation order for all nodes. Private nodes have lists
+ * but never appear as an entry in any list. Allocations
+ * targeting a private node w/ FALLBACK land on N_MEMORY.
+ *
+ * NOFALLBACK: A list for each node containing only itself.
+ * Allocations targeting a private node w/ NOFALLBACK
+ * will always fail (their NOFALLBACK is empty)
+ *
+ * PRIVATE: (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
+ * private nodes, and falls back to normal memory unless
+ * __GFP_THISNODE otherwise constrains it.
+ */
+
build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
true, node_order, &nr_nodes);
build_thisnode_zonelists(pgdat);
@@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
for (node = 0; node < nr_nodes; node++)
pr_cont("%d ", node_order[node]);
pr_cont("\n");
+
+ nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+ build_node_zonelist(pgdat, &tier_nodes, ZONELIST_PRIVATE, false,
+ node_order, &nr_nodes);
}
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
@@ -7282,6 +7331,15 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
if (__alloc_contig_verify_gfp_mask(gfp_mask, (gfp_t *)&cc.gfp_mask))
return -EINVAL;
+ /*
+ * Private nodes are kept unreachable via the zonelists, but
+ * alloc_contig works directly on a zone derived from the caller's
+ * pfn range, bypassing that. Reject this operation explicitly
+ * rather than silently carving memory out of an isolated node.
+ */
+ if (unlikely(node_is_private(zone_to_nid(cc.zone))))
+ return -EBUSY;
+
/*
* What we do here is we mark all pageblocks in range as
* MIGRATE_ISOLATE. Because pageblock and max order pages may
diff --git a/mm/page_alloc.h b/mm/page_alloc.h
index 23fc79ce97b60..741448d83ce77 100644
--- a/mm/page_alloc.h
+++ b/mm/page_alloc.h
@@ -57,6 +57,9 @@
*/
#define ALLOC_NO_CODETAG 0x1000
+/* Select the N_MEMORY_PRIVATE zonelist family (see select_zonelist()). */
+#define ALLOC_ZONELIST_PRIVATE 0x2000
+
/* Flags that allow allocations below the min watermark. */
#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
@@ -95,6 +98,29 @@ struct alloc_context {
unsigned int alloc_flags;
};
+#ifdef CONFIG_NUMA
+static_assert(ZONELIST_PRIVATE + 1 == ZONELIST_PRIVATE_NOFALLBACK);
+#endif
+
+/*
+ * select_zonelist - resolve the zonelist for an allocation
+ *
+ * The default matches node_zonelist(); ALLOC_ZONELIST_PRIVATE selects the
+ * private-node family so an allocation may reach an N_MEMORY_PRIVATE node.
+ */
+static inline struct zonelist *
+select_zonelist(int nid, gfp_t gfp, unsigned int alloc_flags)
+{
+#ifdef CONFIG_NUMA
+ if (alloc_flags & ALLOC_ZONELIST_PRIVATE) {
+ int idx = ZONELIST_PRIVATE + !!(gfp & __GFP_THISNODE);
+
+ return &NODE_DATA(nid)->node_zonelists[idx];
+ }
+#endif
+ return node_zonelist(nid, gfp);
+}
+
/*
* This function returns the order of a free page in the buddy system. In
* general, page_zone(page)->lock must be held by the caller to prevent the
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open)
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
N_MEMORY_PRIVATE node access is gated by per-node capabilities and
the zonelist. Isolating by cpuset offers no functionality and
creates issues on rebind when a cpuset loses access to that node
(in particular: it generates migrations that may not be supported).
Do not partition private nodes via cpuset.mem, instead treat them
as globally accessible resources. Do not engage in silent background
operations on these nodes due to cpuset.mem membership changing.
The result: cpuset operations checking for node validity always
allow N_MEMORY_PRIVATE nodes.
On hot-unplug, we still need to rebind memory policies if the
private node has left N_MEMORY_PRIVATE (i.e. no more memory).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
kernel/cgroup/cpuset.c | 26 ++++++++++++++++++++++----
mm/memcontrol.c | 2 ++
2 files changed, 24 insertions(+), 4 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index dfd0f827e3b92..05468f95c10bd 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -26,6 +26,7 @@
#include <linux/mempolicy.h>
#include <linux/mm.h>
#include <linux/memory.h>
+#include <linux/node_private.h>
#include <linux/rcupdate.h>
#include <linux/sched.h>
#include <linux/sched/deadline.h>
@@ -3867,7 +3868,8 @@ static void cpuset_handle_hotplug(void)
static DECLARE_WORK(hk_sd_work, hk_sd_workfn);
static cpumask_t new_cpus;
static nodemask_t new_mems;
- bool cpus_updated, mems_updated;
+ static nodemask_t prev_priv_mems;
+ bool cpus_updated, mems_updated, priv_shrank;
bool on_dfl = is_in_v2_mode();
struct tmpmasks tmp, *ptmp = NULL;
@@ -3890,6 +3892,14 @@ static void cpuset_handle_hotplug(void)
!cpumask_empty(subpartitions_cpus);
mems_updated = !nodes_equal(top_cpuset.effective_mems, new_mems);
+ /*
+ * Private nodes are not partitioned by cpuset, but if one leaves
+ * N_MEMORY_PRIVATE we still need to run mpol_rebind_* to clean up
+ * mempolicies that are binding them.
+ */
+ priv_shrank = !nodes_subset(prev_priv_mems, node_states[N_MEMORY_PRIVATE]);
+ prev_priv_mems = node_states[N_MEMORY_PRIVATE];
+
/* For v1, synchronize cpus_allowed to cpu_active_mask */
if (cpus_updated) {
cpuset_force_rebuild();
@@ -3922,9 +3932,12 @@ static void cpuset_handle_hotplug(void)
top_cpuset.mems_allowed = new_mems;
top_cpuset.effective_mems = new_mems;
spin_unlock_irq(&callback_lock);
- cpuset_update_tasks_nodemask(&top_cpuset);
}
+ /* Rebind task mempolicies if any memory node changed state */
+ if (mems_updated || priv_shrank)
+ cpuset_update_tasks_nodemask(&top_cpuset);
+
mutex_unlock(&cpuset_mutex);
/* if cpus or mems changed, we need to propagate to descendants */
@@ -4155,11 +4168,13 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
* cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
* @nodemask: the nodemask to be checked
*
- * Are any of the nodes in the nodemask allowed in current->mems_allowed?
+ * Are any of the nodes in the nodemask usable? N_MEMORY nodes must be in
+ * current->mems_allowed, while N_MEMORY_PRIVATE nodes are always valid.
*/
int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
{
- return nodes_intersects(*nodemask, current->mems_allowed);
+ return nodes_intersects(*nodemask, current->mems_allowed) ||
+ nodes_intersects(*nodemask, node_states[N_MEMORY_PRIVATE]);
}
/*
@@ -4223,6 +4238,9 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)
if (in_interrupt())
return true;
+ /* N_MEMORY_PRIVATE nodes are not partitioned by cpusets.mems */
+ if (node_is_private(node))
+ return true;
if (node_isset(node, current->mems_allowed))
return true;
/*
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 8319ad8c5c23a..f0dde52dc9e0e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6069,6 +6069,8 @@ void mem_cgroup_node_filter_allowed(struct mem_cgroup *memcg, nodemask_t *mask)
* mask is acceptable.
*/
cpuset_nodes_allowed(memcg->css.cgroup, &allowed);
+ /* N_MEMORY_PRIVATE nodes are not partitioned by cpuset, include them */
+ nodes_or(allowed, allowed, node_states[N_MEMORY_PRIVATE]);
nodes_and(*mask, *mask, allowed);
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 07/36] mm/memory_hotplug: disallow migration-driven private node hotunplug
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
By default, device-backed private node memory may not be safe to
migrate without coordination.
Do not attempt to hotunplug private node memory, instead fail the
hotunplug attempt, and require the owner to drain the folios
safely before allowing hotunplug to proceed.
Unlike transient / longterm pins causing failures, this is not a
transient condition - so we can break out of the hotunplug attempt
entirely instead of spinning forever on a failed migration attempt.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/memory_hotplug.c | 21 +++++++++++++++++----
1 file changed, 17 insertions(+), 4 deletions(-)
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 226ab9cb078ad..2b9c0830821e6 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -34,6 +34,7 @@
#include <linux/memblock.h>
#include <linux/compaction.h>
#include <linux/rmap.h>
+#include <linux/node_private.h>
#include <linux/module.h>
#include <linux/node.h>
@@ -1843,11 +1844,12 @@ static int scan_movable_pages(unsigned long start, unsigned long end,
return 0;
}
-static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
+static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
{
struct folio *folio;
unsigned long pfn;
LIST_HEAD(source);
+ int err = 0;
static DEFINE_RATELIMIT_STATE(migrate_rs, DEFAULT_RATELIMIT_INTERVAL,
DEFAULT_RATELIMIT_BURST);
@@ -1884,6 +1886,15 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
goto put_folio;
}
+ /* Private node folios cannot migrate, fail outright */
+ if (folio_is_private_node(folio)) {
+ pr_info_ratelimited("memory offline refused: node %d pfn %lx\n",
+ folio_nid(folio), pfn);
+ folio_put(folio);
+ err = -EBUSY;
+ break;
+ }
+
if (!isolate_folio_to_list(folio, &source)) {
if (__ratelimit(&migrate_rs)) {
pr_warn("failed to isolate pfn %lx\n",
@@ -1931,6 +1942,7 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
putback_movable_pages(&source);
}
}
+ return err;
}
static int __init cmdline_parse_movable_node(char *p)
@@ -2067,10 +2079,11 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
ret = scan_movable_pages(pfn, end_pfn, &pfn);
if (!ret) {
/*
- * TODO: fatal migration failures should bail
- * out
+ * A fatal migration failure (e.g. a folio on a
+ * non-migratable private node) bails out; transient
+ * failures leave ret zero and are retried below.
*/
- do_migrate_range(pfn, end_pfn);
+ ret = do_migrate_range(pfn, end_pfn);
}
} while (!ret);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Private nodes are already kept out of policy nodemasks (only N_MEMORY
nodes are allowed), but an mbind(MPOL_MF_MOVE) walk can still encounter a
private-node folio in the range. Skip such folios so mempolicy-driven
migration never moves private-node memory.
We already do this for ZONE_DEVICE folios - use the same filter points
for ZONE_DEVICE (with an additional point for large pages).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 5720f7f54d942..8e8763f6e5f5b 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -111,6 +111,7 @@
#include <linux/mmu_notifier.h>
#include <linux/printk.h>
#include <linux/leafops.h>
+#include <linux/node_private.h>
#include <linux/gcd.h>
#include <asm/tlbflush.h>
@@ -668,6 +669,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
}
if (!queue_folio_required(folio, qp))
return;
+ if (folio_is_private_node(folio))
+ return;
if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma) ||
!migrate_folio_add(folio, qp->pagelist, qp->flags))
@@ -722,7 +725,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
continue;
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
if (folio_test_large(folio) && max_nr != 1)
nr = folio_pte_batch(folio, pte, ptent, max_nr);
@@ -797,6 +800,8 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
folio = pfn_folio(pte_pfn(ptep));
if (!queue_folio_required(folio, qp))
goto unlock;
+ if (folio_is_private_node(folio))
+ goto unlock;
if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
!vma_migratable(walk->vma)) {
qp->nr_failed++;
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Use the same filter locations as zone_device folios to disallow
userland driven migration requests for private node memory.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/migrate.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/migrate.c b/mm/migrate.c
index a8a86141dbb54..d20674c07b947 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2268,7 +2268,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
return -EFAULT;
- if (folio_is_zone_device(folio))
+ if (folio_is_private_managed(folio))
return -ENOENT;
if (folio_nid(folio) == node)
@@ -2477,7 +2477,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
if (folio) {
if (is_zero_folio(folio) || is_huge_zero_folio(folio))
err = -EFAULT;
- else if (folio_is_zone_device(folio))
+ else if (folio_is_private_managed(folio))
err = -ENOENT;
else
err = folio_nid(folio);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Use the same filter locations as zone_device, plus additional filters
for huge pages to avoid madvise operations on private node memory.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/huge_memory.c | 5 +++++
mm/madvise.c | 8 ++++++--
2 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 5bd8d4f59a7b8..1df91b4e5c2bc 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -37,6 +37,7 @@
#include <linux/page_owner.h>
#include <linux/sched/sysctl.h>
#include <linux/memory-tiers.h>
+#include <linux/node_private.h>
#include <linux/compat.h>
#include <linux/pgalloc.h>
#include <linux/pgalloc_tag.h>
@@ -2338,6 +2339,10 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
}
folio = pmd_folio(orig_pmd);
+
+ if (folio_is_private_node(folio))
+ goto out;
+
/*
* If other processes are mapping this folio, we couldn't discard
* the folio unless they all do MADV_FREE so let's skip the folio.
diff --git a/mm/madvise.c b/mm/madvise.c
index 07a21ca31bad4..29f35a23919a0 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -32,6 +32,7 @@
#include <linux/leafops.h>
#include <linux/shmem_fs.h>
#include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
#include <asm/tlb.h>
@@ -395,6 +396,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
folio = pmd_folio(orig_pmd);
+ if (folio_is_private_node(folio))
+ goto huge_unlock;
+
/* Do not interfere with other mappings of this folio */
if (folio_maybe_mapped_shared(folio))
goto huge_unlock;
@@ -474,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
continue;
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
/*
@@ -703,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
}
folio = vm_normal_folio(vma, addr, ptent);
- if (!folio || folio_is_zone_device(folio))
+ if (!folio || folio_is_private_managed(folio))
continue;
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Skip compaction on private nodes. Compaction requires the ability to
migrate pages, which is not yet supported on private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/compaction.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/mm/compaction.c b/mm/compaction.c
index 9f81055a358ed..8c1351cce7bcc 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2462,6 +2462,9 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
+ if (node_is_private(zone_to_nid(zone)))
+ continue;
+
/*
* Do not consider all the reclaimable memory because we do not
* want to trash just for a single high order allocation which
@@ -2853,6 +2856,9 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
+ if (node_is_private(zone_to_nid(zone)))
+ continue;
+
if (prio > MIN_COMPACT_PRIORITY
&& compaction_deferred(zone, order)) {
rc = max_t(enum compact_result, COMPACT_DEFERRED, rc);
@@ -2922,6 +2928,9 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
.proactive_compaction = proactive,
};
+ if (node_is_private(pgdat->node_id))
+ return 0;
+
for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
zone = &pgdat->node_zones[zoneid];
if (!populated_zone(zone))
@@ -3017,6 +3026,9 @@ static ssize_t compact_store(struct device *dev,
{
int nid = dev->id;
+ if (node_is_private(nid))
+ return -EINVAL;
+
if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
/* Flush pending updates to the LRU lists */
lru_add_drain_all();
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 12/36] mm/page_alloc: clear private node watermarks and system reserves
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Set private node watermarks to 0 (reclaim isn't supported yet, so they
are pointless), and do not include private nodes in system reserves.
The oom killer can't actually resolve general system pressure by
releasing private node capacity, so including them in system reserve
calculations dilutes DRAM watermarks and causes poor OOM kill behavior.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/page_alloc.c | 24 ++++++++++++++++++++++--
1 file changed, 22 insertions(+), 2 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 78755a55b1540..2b08bea2379a9 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6555,6 +6555,10 @@ static void calculate_totalreserve_pages(void)
pgdat->totalreserve_pages = 0;
+ /* private nodes have zero watermarks */
+ if (node_is_private(pgdat->node_id))
+ continue;
+
for (i = 0; i < MAX_NR_ZONES; i++) {
struct zone *zone = pgdat->node_zones + i;
long max = 0;
@@ -6647,9 +6651,15 @@ static void __setup_per_zone_wmarks(void)
struct zone *zone;
unsigned long flags;
- /* Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages */
+ /*
+ * Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages
+ *
+ * Private nodes are excluded because they are not in the regular
+ * allocation fallback path - including them dilutes DRAM watermarks.
+ */
for_each_zone(zone) {
- if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE)
+ if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE &&
+ !node_is_private(zone_to_nid(zone)))
lowmem_pages += zone_managed_pages(zone);
}
@@ -6657,6 +6667,16 @@ static void __setup_per_zone_wmarks(void)
u64 tmp;
spin_lock_irqsave(&zone->lock, flags);
+ if (node_is_private(zone_to_nid(zone))) {
+ zone->_watermark[WMARK_MIN] = 0;
+ zone->_watermark[WMARK_LOW] = 0;
+ zone->_watermark[WMARK_HIGH] = 0;
+ zone->_watermark[WMARK_PROMO] = 0;
+ zone->watermark_boost = 0;
+ spin_unlock_irqrestore(&zone->lock, flags);
+ continue;
+ }
+
tmp = (u64)pages_min * zone_managed_pages(zone);
tmp = div64_ul(tmp, lowmem_pages);
if (is_highmem(zone) || zone_idx(zone) == ZONE_MOVABLE) {
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 13/36] mm/mempolicy: disallow NUMA Balancing prot_none on private nodes
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Skip private node memory the same way zone device memory is
skipped, since private nodes do not presently support migration.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/mempolicy.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 8e8763f6e5f5b..2b76c57a460c9 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -845,7 +845,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
{
int nid;
- if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
+ if (!folio || folio_is_private_managed(folio) || folio_test_ksm(folio))
return false;
/* Also skip shared copy-on-write folios */
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
DAMON operates on physical address ranges, which can cover private
node memory. Skip private-node folios in both DAMON's migration
and reclaim paths.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/damon/paddr.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index e4f98d67461f5..c741a94319750 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -12,6 +12,7 @@
#include <linux/swap.h>
#include <linux/memory-tiers.h>
#include <linux/mm_inline.h>
+#include <linux/node_private.h>
#include "../internal.h"
#include "ops-common.h"
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
continue;
}
+ /* private node memory is not reclaimable by default */
+ if (folio_is_private_node(folio))
+ goto put_folio;
+
if (damos_pa_filter_out(s, folio))
goto put_folio;
else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
else
*sz_filter_passed += folio_size(folio) / addr_unit;
+ /* private nodes do not support migration by default */
+ if (folio_is_private_node(folio))
+ goto put_folio;
+
if (!folio_isolate_lru(folio))
goto put_folio;
list_add(&folio->lru, &folio_list);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 15/36] mm/ksm: skip KSM for managed-memory folios
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Since private memory is intended for explicit placement, allowing
KSM to operate on it by default would completely defeat the purpose.
Skip KSM merge and scan operations for private node folios using
the same filter location as zone_device folios.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/ksm.c | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/mm/ksm.c b/mm/ksm.c
index 47006f494fcb1..dcc5c46a29eac 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -827,7 +827,7 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
folio = folio_walk_start(&fw, vma, addr, 0);
if (folio) {
- if (!folio_is_zone_device(folio) &&
+ if (!folio_is_private_managed(folio) &&
folio_test_anon(folio)) {
folio_get(folio);
page = fw.page;
@@ -2558,7 +2558,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
goto not_found_unlock;
folio = page_folio(page);
- if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+ if (unlikely(folio_is_private_managed(folio)) ||
+ !folio_test_anon(folio))
goto not_found_unlock;
page += ((addr & (PMD_SIZE - 1)) >> PAGE_SHIFT);
@@ -2582,7 +2583,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
continue;
folio = page_folio(page);
- if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+ if (unlikely(folio_is_private_managed(folio)) ||
+ !folio_test_anon(folio))
continue;
goto found_unlock;
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse.
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
A collapse operation causes new THP allocation to occur, and may
migrate memory from one node to another.
Handle this the same as zone_device for now (disallow collapse).
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/khugepaged.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 89ce6bcbc376b..fb4378cc17b10 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -700,7 +700,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
goto out;
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+ if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
result = SCAN_PAGE_NULL;
goto out;
}
@@ -1687,7 +1687,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
}
page = vm_normal_page(vma, addr, pteval);
- if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+ if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
result = SCAN_PAGE_NULL;
goto out_unmap;
}
@@ -1944,7 +1944,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
}
page = vm_normal_page(vma, addr, ptent);
- if (WARN_ON_ONCE(page && is_zone_device_page(page)))
+ if (WARN_ON_ONCE(page && page_is_private_managed(page)))
page = NULL;
/*
* Note that uprobe, debugger, or MAP_PRIVATE may change the
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
Private nodes do not support reclaim by default, so prevent all
reclaimers from operating on private nodes.
shrink_node() is the common chokepoint for most reclaimers (direct,
kswapd, proactive, mglru, etc), so it's the best place to filter.
DAMON's proactive pageout bypasses shrink_node() entirely, so we
disable DAMON in a separate commit as its own discrete service.
Other reclaim entry points are already covered - e.g. MGLRU's debugfs
interface rejects nids that are not N_MEMORY (N_MEMORY_PRIVATE is
mutually exclusive), and madvise pageout/cold skips private nodes.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/vmscan.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index e26c6931f5fde..86b2334c23b98 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -6141,6 +6141,13 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
struct lruvec *target_lruvec;
bool reclaimable = false;
+ /*
+ * Private nodes do not support reclaim by default, filtering here
+ * captures all normal reclaim paths that may attempt eviction.
+ */
+ if (node_is_private(pgdat->node_id))
+ return;
+
if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
memset(&sc->nr, 0, sizeof(sc->nr));
lru_gen_shrink_node(pgdat, sc);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 18/36] mm/gup: disallow longterm pin of private node folios
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
FOLL_LONGTERM on a N_MEMORY_PRIVATE node folio is wrong both ways the
GUP longterm path resolves it:
a ZONE_NORMAL folio would be pinned in place
a ZONE_MOVABLE folio would be migrated off first
A private node folio should do neither - the pin should fail in place.
Add folio_longterm_pin_forbidden() (true for any private node folio) and
reject such a pin in check_and_migrate_movable_pages_or_folios() before
any isolation, so nothing is migrated.
The gup-fast path defers a private folio to the slow path via
folio_allows_longterm_pin().
Signed-off-by: Gregory Price <gourry@gourry.net>
---
mm/gup.c | 27 +++++++++++++++++++++++++--
mm/internal.h | 25 +++++++++++++++++++++++++
2 files changed, 50 insertions(+), 2 deletions(-)
diff --git a/mm/gup.c b/mm/gup.c
index 1d32e9a3dc79c..a7d4de223785c 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -547,10 +547,10 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,
/*
* Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
* right zone, so fail and let the caller fall back to the slow
- * path.
+ * path. Fail for private-node folios here so slow path rejects.
*/
if (unlikely((flags & FOLL_LONGTERM) &&
- !folio_is_longterm_pinnable(folio))) {
+ !folio_allows_longterm_pin(folio))) {
folio_put_refs(folio, refs);
return NULL;
}
@@ -2389,12 +2389,35 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list,
return ret;
}
+/*
+ * True if any folio sits on a private node whose folios may not be longterm
+ * pinned. Such folios can neither be pinned nor migrated, so the whole pin
+ * must fail before migration. Checked before any isolation occurs.
+ */
+static bool pofs_has_ltpin_forbidden(struct pages_or_folios *pofs)
+{
+ struct folio *folio;
+ long i = 0;
+
+ for (folio = pofs_get_folio(pofs, i); folio;
+ folio = pofs_next_folio(folio, pofs, &i)) {
+ if (folio_longterm_pin_forbidden(folio))
+ return true;
+ }
+ return false;
+}
+
static long
check_and_migrate_movable_pages_or_folios(struct pages_or_folios *pofs)
{
LIST_HEAD(movable_folio_list);
unsigned long collected;
+ if (pofs_has_ltpin_forbidden(pofs)) {
+ pofs_unpin(pofs);
+ return -EFAULT;
+ }
+
collected = collect_longterm_unpinnable_folios(&movable_folio_list,
pofs);
if (!collected)
diff --git a/mm/internal.h b/mm/internal.h
index cb9f4a8342e32..85c460296cea1 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -110,6 +110,31 @@ static inline bool page_is_private_managed(struct page *page)
return folio_is_private_managed(page_folio(page));
}
+/*
+ * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
+ *
+ * checks folio_is_longterm_pinnable() rules plus private node permissions.
+ * private node permission is checked here to resolve circular header
+ * dependencies in folio_is_longterm_pinnable.
+ */
+static inline bool folio_allows_longterm_pin(struct folio *folio)
+{
+ return folio_is_longterm_pinnable(folio) &&
+ !folio_is_private_node(folio);
+}
+
+/*
+ * folio_longterm_pin_forbidden() - must a longterm pin of this folio fail
+ * outright (neither pinned in place nor migrated off the node)?
+ *
+ * True for any folio on a private node: such memory can be neither pinned
+ * nor migrated, so the pin must be rejected with the folio left in place.
+ */
+static inline bool folio_longterm_pin_forbidden(struct folio *folio)
+{
+ return folio_is_private_node(folio);
+}
+
/*
* Maintains state across a page table move. The operation assumes both source
* and destination VMAs already exist and are specified by the user.
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output
From: Gregory Price @ 2026-07-20 19:34 UTC (permalink / raw)
To: linux-mm
Cc: Zhigang.Luo, arun.george, balbirs, brendan.jackman, yuzenghui,
apopple, alucerop, matthew.brost, akpm, david, ljs, liam, vbabka,
rppt, surenb, mhocko, corbet, skhan, gregkh, rafael, dakr, djbw,
vishal.l.verma, dave.jiang, alison.schofield, osandov, jannh,
pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, gourry, ying.huang, kasong,
qi.zheng, shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc,
yury.norov, linux, longman, ridong.chen, tj, mkoutny, sj, jgg,
jhubbard, peterx, baolin.wang, npache, ryan.roberts, dev.jain,
lance.yang, usama.arif, xu.xin16, chengming.zhou, roman.gushchin,
muchun.song, linux-kernel, linux-doc, driver-core, nvdimm,
linux-cxl, linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <20260720193431.3841992-1-gourry@gourry.net>
numa_maps collects per-node page counts in the page-table walkers
and emits them in show_numa_map. All three filtered by N_MEMORY,
so pages on N_MEMORY_PRIVATE nodes were never gathered/printed.
Accept N_MEMORY_PRIVATE pages in both walkers and emit private nodes
in show_numa_map, so private-node mappings are visible in numa_maps.
Signed-off-by: Gregory Price <gourry@gourry.net>
---
fs/proc/task_mmu.c | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 817e3e0f91943..e116cd5f157b9 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -3389,7 +3389,8 @@ static struct page *can_gather_numa_stats(pte_t pte, struct vm_area_struct *vma,
return NULL;
nid = page_to_nid(page);
- if (!node_isset(nid, node_states[N_MEMORY]))
+ if (!node_isset(nid, node_states[N_MEMORY]) &&
+ !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
return NULL;
return page;
@@ -3414,7 +3415,8 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
return NULL;
nid = page_to_nid(page);
- if (!node_isset(nid, node_states[N_MEMORY]))
+ if (!node_isset(nid, node_states[N_MEMORY]) &&
+ !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
return NULL;
return page;
@@ -3602,6 +3604,10 @@ static int show_numa_map(struct seq_file *m, void *v)
if (md->node[nid])
seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+ for_each_node_state(nid, N_MEMORY_PRIVATE)
+ if (md->node[nid])
+ seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+
seq_printf(m, " kernelpagesize_kB=%lu", vma_kernel_pagesize(vma) >> 10);
out:
seq_putc(m, '\n');
--
2.53.0-Meta
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox