* [PATCH v10 03/11] selftests: tracing: Add a basic testcase for wprobe
From: Masami Hiramatsu (Google) @ 2026-07-22 23:03 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add 'add_remove_wprobe.tc' testcase for testing wprobe event that
tests adding and removing operations of the wprobe event.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Fix command check logic to prevent early exit under 'set -e'
(errexit) when grep or test fails.
- Simplify enable/disable status checks by removing cat pipes.
Changes in v8:
- Fixed silently test failure path.
---
tools/testing/selftests/ftrace/config | 1
.../ftrace/test.d/dynevent/add_remove_wprobe.tc | 63 ++++++++++++++++++++
2 files changed, 64 insertions(+)
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
diff --git a/tools/testing/selftests/ftrace/config b/tools/testing/selftests/ftrace/config
index 544de0db5f58..d2f503722020 100644
--- a/tools/testing/selftests/ftrace/config
+++ b/tools/testing/selftests/ftrace/config
@@ -27,3 +27,4 @@ CONFIG_STACK_TRACER=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y
+CONFIG_WPROBE_EVENTS=y
diff --git a/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc b/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
new file mode 100644
index 000000000000..647c37d5e4c8
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
@@ -0,0 +1,63 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Generic dynamic event - add/remove wprobe events
+# requires: dynamic_events "w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]":README
+
+echo 0 > events/enable
+echo > dynamic_events
+
+# Use jiffies as a variable that is frequently written to.
+TARGET=jiffies
+
+echo "w:my_wprobe w@$TARGET" >> dynamic_events
+
+if ! grep -q my_wprobe dynamic_events; then
+ echo "Failed to create wprobe event"
+ exit_fail
+fi
+
+if [ ! -d events/wprobes/my_wprobe ]; then
+ echo "Failed to create wprobe event directory"
+ exit_fail
+fi
+
+echo 1 > events/wprobes/my_wprobe/enable
+
+# Check if the event is enabled
+if ! grep -q 1 events/wprobes/my_wprobe/enable; then
+ echo "Failed to enable wprobe event"
+ exit_fail
+fi
+
+# Let some time pass to trigger the breakpoint
+sleep 1
+
+# Check if we got any trace output
+if ! grep -q my_wprobe trace; then
+ echo "wprobe event was not triggered"
+ exit_fail
+fi
+
+echo 0 > events/wprobes/my_wprobe/enable
+
+# Check if the event is disabled
+if ! grep -q 0 events/wprobes/my_wprobe/enable; then
+ echo "Failed to disable wprobe event"
+ exit_fail
+fi
+
+echo "-:my_wprobe" >> dynamic_events
+
+if grep -q my_wprobe dynamic_events; then
+ echo "Failed to remove wprobe event"
+ exit_fail
+fi
+
+if [ -d events/wprobes/my_wprobe ]; then
+ echo "Failed to remove wprobe event directory"
+ exit_fail
+fi
+
+clear_trace
+
+exit 0
^ permalink raw reply related
* [PATCH v10 04/11] selftests: tracing: Add syntax testcase for wprobe
From: Masami Hiramatsu (Google) @ 2026-07-22 23:03 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add "wprobe_syntax_errors.tc" testcase for testing syntax errors
of the watch probe events.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
.../test.d/dynevent/wprobes_syntax_errors.tc | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
diff --git a/tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc b/tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
new file mode 100644
index 000000000000..56ac579d60ae
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
@@ -0,0 +1,20 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Watch probe event parser error log check
+# requires: dynamic_events "w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]":README
+
+check_error() { # command-with-error-pos-by-^
+ ftrace_errlog_check 'wprobe' "$1" 'dynamic_events'
+}
+
+check_error 'w ^symbol' # BAD_ACCESS_FMT
+check_error 'w ^a@symbol' # BAD_ACCESS_TYPE
+check_error 'w w@^symbol' # BAD_ACCESS_ADDR
+check_error 'w w@jiffies^+offset' # BAD_ACCESS_ADDR
+check_error 'w w@jiffies:^100' # BAD_ACCESS_LEN
+check_error 'w w@jiffies ^$arg1' # BAD_VAR
+check_error 'w w@jiffies ^$retval' # BAD_VAR
+check_error 'w w@jiffies ^$stack' # BAD_VAR
+check_error 'w w@jiffies ^%ax' # BAD_VAR
+
+exit 0
^ permalink raw reply related
* [PATCH v10 05/11] x86/hw_breakpoint: Unify breakpoint install/uninstall
From: Masami Hiramatsu (Google) @ 2026-07-22 23:03 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Jinchao Wang <wangjinchao600@gmail.com>
Consolidate breakpoint management to reduce code duplication.
The diffstat was misleading, so the stripped code size is compared instead.
After refactoring, it is reduced from 11976 bytes to 11448 bytes on my
x86_64 system built with clang.
This also makes it easier to introduce arch_reinstall_hw_breakpoint().
In addition, including linux/types.h to fix a missing build dependency.
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Jinchao Wang <wangjinchao600@gmail.com>
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v8:
- Add missing barrier() on the disable path of setup_hwbp() to prevent
the compiler from reordering this_cpu_write(cpu_dr7, ...) before
set_debugreg(dr7, 7).
- Clear existing slot control and enable bits in cpu_dr7 inside
setup_hwbp() using __encode_dr7() before setting new ones to prevent
register state corruption on update/reinstall.
---
arch/x86/include/asm/hw_breakpoint.h | 6 +
arch/x86/kernel/hw_breakpoint.c | 144 +++++++++++++++++++---------------
2 files changed, 86 insertions(+), 64 deletions(-)
diff --git a/arch/x86/include/asm/hw_breakpoint.h b/arch/x86/include/asm/hw_breakpoint.h
index 0bc931cd0698..aa6adac6c3a2 100644
--- a/arch/x86/include/asm/hw_breakpoint.h
+++ b/arch/x86/include/asm/hw_breakpoint.h
@@ -5,6 +5,7 @@
#include <uapi/asm/hw_breakpoint.h>
#define __ARCH_HW_BREAKPOINT_H
+#include <linux/types.h>
/*
* The name should probably be something dealt in
@@ -18,6 +19,11 @@ struct arch_hw_breakpoint {
u8 type;
};
+enum bp_slot_action {
+ BP_SLOT_ACTION_INSTALL,
+ BP_SLOT_ACTION_UNINSTALL,
+};
+
#include <linux/kdebug.h>
#include <linux/percpu.h>
#include <linux/list.h>
diff --git a/arch/x86/kernel/hw_breakpoint.c b/arch/x86/kernel/hw_breakpoint.c
index f846c15f21ca..c323c2aab2af 100644
--- a/arch/x86/kernel/hw_breakpoint.c
+++ b/arch/x86/kernel/hw_breakpoint.c
@@ -49,7 +49,6 @@ static DEFINE_PER_CPU(unsigned long, cpu_debugreg[HBP_NUM]);
*/
static DEFINE_PER_CPU(struct perf_event *, bp_per_reg[HBP_NUM]);
-
static inline unsigned long
__encode_dr7(int drnum, unsigned int len, unsigned int type)
{
@@ -86,96 +85,113 @@ int decode_dr7(unsigned long dr7, int bpnum, unsigned *len, unsigned *type)
}
/*
- * Install a perf counter breakpoint.
- *
- * We seek a free debug address register and use it for this
- * breakpoint. Eventually we enable it in the debug control register.
- *
- * Atomic: we hold the counter->ctx->lock and we only handle variables
- * and registers local to this cpu.
+ * We seek a slot and change it or keep it based on the action.
+ * Returns slot number on success, negative error on failure.
+ * Must be called with IRQs disabled.
*/
-int arch_install_hw_breakpoint(struct perf_event *bp)
+static int manage_bp_slot(struct perf_event *bp, enum bp_slot_action action)
{
- struct arch_hw_breakpoint *info = counter_arch_bp(bp);
- unsigned long *dr7;
- int i;
-
- lockdep_assert_irqs_disabled();
+ struct perf_event *old_bp;
+ struct perf_event *new_bp;
+ int slot;
+
+ switch (action) {
+ case BP_SLOT_ACTION_INSTALL:
+ old_bp = NULL;
+ new_bp = bp;
+ break;
+ case BP_SLOT_ACTION_UNINSTALL:
+ old_bp = bp;
+ new_bp = NULL;
+ break;
+ default:
+ return -EINVAL;
+ }
- for (i = 0; i < HBP_NUM; i++) {
- struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
+ for (slot = 0; slot < HBP_NUM; slot++) {
+ struct perf_event **curr = this_cpu_ptr(&bp_per_reg[slot]);
- if (!*slot) {
- *slot = bp;
- break;
+ if (*curr == old_bp) {
+ *curr = new_bp;
+ return slot;
}
}
- if (WARN_ONCE(i == HBP_NUM, "Can't find any breakpoint slot"))
- return -EBUSY;
+ if (old_bp) {
+ WARN_ONCE(1, "Can't find matching breakpoint slot");
+ return -EINVAL;
+ }
- set_debugreg(info->address, i);
- __this_cpu_write(cpu_debugreg[i], info->address);
+ WARN_ONCE(1, "No free breakpoint slots");
+ return -EBUSY;
+}
- dr7 = this_cpu_ptr(&cpu_dr7);
- *dr7 |= encode_dr7(i, info->len, info->type);
+static void setup_hwbp(struct arch_hw_breakpoint *info, int slot, bool enable)
+{
+ unsigned long dr7;
+
+ set_debugreg(info->address, slot);
+ __this_cpu_write(cpu_debugreg[slot], info->address);
+
+ dr7 = this_cpu_read(cpu_dr7);
+ dr7 &= ~(__encode_dr7(slot, 0xc, 0x3) |
+ (DR_LOCAL_ENABLE << (slot * DR_ENABLE_SIZE)));
+ if (enable)
+ dr7 |= encode_dr7(slot, info->len, info->type);
/*
- * Ensure we first write cpu_dr7 before we set the DR7 register.
- * This ensures an NMI never see cpu_dr7 0 when DR7 is not.
+ * Enabling:
+ * Ensure we first write cpu_dr7 before we set the DR7 register.
+ * This ensures an NMI never see cpu_dr7 0 when DR7 is not.
*/
+ if (enable)
+ this_cpu_write(cpu_dr7, dr7);
+
barrier();
- set_debugreg(*dr7, 7);
- if (info->mask)
- amd_set_dr_addr_mask(info->mask, i);
+ set_debugreg(dr7, 7);
+
+ amd_set_dr_addr_mask(enable ? info->mask : 0, slot);
- return 0;
+ barrier();
+
+ /*
+ * Disabling:
+ * Ensure the write to cpu_dr7 is after we've set the DR7 register.
+ * This ensures an NMI never see cpu_dr7 0 when DR7 is not.
+ */
+ if (!enable)
+ this_cpu_write(cpu_dr7, dr7);
}
/*
- * Uninstall the breakpoint contained in the given counter.
- *
- * First we search the debug address register it uses and then we disable
- * it.
- *
- * Atomic: we hold the counter->ctx->lock and we only handle variables
- * and registers local to this cpu.
+ * find suitable breakpoint slot and set it up based on the action
*/
-void arch_uninstall_hw_breakpoint(struct perf_event *bp)
+static int arch_manage_bp(struct perf_event *bp, enum bp_slot_action action)
{
- struct arch_hw_breakpoint *info = counter_arch_bp(bp);
- unsigned long dr7;
- int i;
+ struct arch_hw_breakpoint *info;
+ int slot;
lockdep_assert_irqs_disabled();
- for (i = 0; i < HBP_NUM; i++) {
- struct perf_event **slot = this_cpu_ptr(&bp_per_reg[i]);
-
- if (*slot == bp) {
- *slot = NULL;
- break;
- }
- }
+ slot = manage_bp_slot(bp, action);
+ if (slot < 0)
+ return slot;
- if (WARN_ONCE(i == HBP_NUM, "Can't find any breakpoint slot"))
- return;
+ info = counter_arch_bp(bp);
+ setup_hwbp(info, slot, action != BP_SLOT_ACTION_UNINSTALL);
- dr7 = this_cpu_read(cpu_dr7);
- dr7 &= ~__encode_dr7(i, info->len, info->type);
-
- set_debugreg(dr7, 7);
- if (info->mask)
- amd_set_dr_addr_mask(0, i);
+ return 0;
+}
- /*
- * Ensure the write to cpu_dr7 is after we've set the DR7 register.
- * This ensures an NMI never see cpu_dr7 0 when DR7 is not.
- */
- barrier();
+int arch_install_hw_breakpoint(struct perf_event *bp)
+{
+ return arch_manage_bp(bp, BP_SLOT_ACTION_INSTALL);
+}
- this_cpu_write(cpu_dr7, dr7);
+void arch_uninstall_hw_breakpoint(struct perf_event *bp)
+{
+ arch_manage_bp(bp, BP_SLOT_ACTION_UNINSTALL);
}
static int arch_bp_generic_len(int x86_len)
^ permalink raw reply related
* [PATCH v10 06/11] x86/hw_breakpoint: Add arch_reinstall_hw_breakpoint
From: Masami Hiramatsu (Google) @ 2026-07-22 23:03 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Jinchao Wang <wangjinchao600@gmail.com>
The new arch_reinstall_hw_breakpoint() function can be used in an
atomic context, unlike the more expensive free and re-allocation path.
This allows callers to efficiently re-establish an existing breakpoint.
Signed-off-by: Jinchao Wang <wangjinchao600@gmail.com>
Reviewed-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Update commit message.
- Temporarily disable the active slot in setup_hwbp() before updating
the address register to avoid spurious debug exceptions.
---
arch/x86/include/asm/hw_breakpoint.h | 2 ++
arch/x86/kernel/hw_breakpoint.c | 34 ++++++++++++++++++++++++++++------
2 files changed, 30 insertions(+), 6 deletions(-)
diff --git a/arch/x86/include/asm/hw_breakpoint.h b/arch/x86/include/asm/hw_breakpoint.h
index aa6adac6c3a2..c22cc4e87fc5 100644
--- a/arch/x86/include/asm/hw_breakpoint.h
+++ b/arch/x86/include/asm/hw_breakpoint.h
@@ -21,6 +21,7 @@ struct arch_hw_breakpoint {
enum bp_slot_action {
BP_SLOT_ACTION_INSTALL,
+ BP_SLOT_ACTION_REINSTALL,
BP_SLOT_ACTION_UNINSTALL,
};
@@ -65,6 +66,7 @@ extern int hw_breakpoint_exceptions_notify(struct notifier_block *unused,
int arch_install_hw_breakpoint(struct perf_event *bp);
+int arch_reinstall_hw_breakpoint(struct perf_event *bp);
void arch_uninstall_hw_breakpoint(struct perf_event *bp);
void hw_breakpoint_pmu_read(struct perf_event *bp);
void hw_breakpoint_pmu_unthrottle(struct perf_event *bp);
diff --git a/arch/x86/kernel/hw_breakpoint.c b/arch/x86/kernel/hw_breakpoint.c
index c323c2aab2af..0df3ff556f47 100644
--- a/arch/x86/kernel/hw_breakpoint.c
+++ b/arch/x86/kernel/hw_breakpoint.c
@@ -100,6 +100,10 @@ static int manage_bp_slot(struct perf_event *bp, enum bp_slot_action action)
old_bp = NULL;
new_bp = bp;
break;
+ case BP_SLOT_ACTION_REINSTALL:
+ old_bp = bp;
+ new_bp = bp;
+ break;
case BP_SLOT_ACTION_UNINSTALL:
old_bp = bp;
new_bp = NULL;
@@ -129,23 +133,36 @@ static int manage_bp_slot(struct perf_event *bp, enum bp_slot_action action)
static void setup_hwbp(struct arch_hw_breakpoint *info, int slot, bool enable)
{
unsigned long dr7;
-
- set_debugreg(info->address, slot);
- __this_cpu_write(cpu_debugreg[slot], info->address);
+ bool enabled;
dr7 = this_cpu_read(cpu_dr7);
+ enabled = dr7 & ((DR_LOCAL_ENABLE | DR_GLOBAL_ENABLE) << (slot * DR_ENABLE_SIZE));
dr7 &= ~(__encode_dr7(slot, 0xc, 0x3) |
(DR_LOCAL_ENABLE << (slot * DR_ENABLE_SIZE)));
- if (enable)
- dr7 |= encode_dr7(slot, info->len, info->type);
+
+ /*
+ * If the slot is currently enabled, disable it first before updating
+ * the address register to prevent spurious debug exceptions.
+ */
+ if (enable && enabled) {
+ barrier();
+ set_debugreg(dr7, 7);
+ barrier();
+ this_cpu_write(cpu_dr7, dr7);
+ }
+
+ set_debugreg(info->address, slot);
+ __this_cpu_write(cpu_debugreg[slot], info->address);
/*
* Enabling:
* Ensure we first write cpu_dr7 before we set the DR7 register.
* This ensures an NMI never see cpu_dr7 0 when DR7 is not.
*/
- if (enable)
+ if (enable) {
+ dr7 |= encode_dr7(slot, info->len, info->type);
this_cpu_write(cpu_dr7, dr7);
+ }
barrier();
@@ -189,6 +206,11 @@ int arch_install_hw_breakpoint(struct perf_event *bp)
return arch_manage_bp(bp, BP_SLOT_ACTION_INSTALL);
}
+int arch_reinstall_hw_breakpoint(struct perf_event *bp)
+{
+ return arch_manage_bp(bp, BP_SLOT_ACTION_REINSTALL);
+}
+
void arch_uninstall_hw_breakpoint(struct perf_event *bp)
{
arch_manage_bp(bp, BP_SLOT_ACTION_UNINSTALL);
^ permalink raw reply related
* [PATCH v10 07/11] HWBP: Add modify_wide_hw_breakpoint_local() API
From: Masami Hiramatsu (Google) @ 2026-07-22 23:03 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add modify_wide_hw_breakpoint_local() arch-wide interface which allows
hwbp users to update watch address on-line. This is available if the
arch supports CONFIG_HAVE_REINSTALL_HW_BREAKPOINT.
Note that this allows to change the type only for compatible types,
because it does not release and reserve the hwbp slot based on type.
For instance, you can not change HW_BREAKPOINT_W to HW_BREAKPOINT_X.
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Add lockdep_assert_irqs_disabled() to enforce and document the
interrupt-disabled calling context requirement.
- Implement state rollback logic in modify_wide_hw_breakpoint_local()
to prevent inconsistent state on architecture update failure.
Changes in v8:
- Parse new attributes into a temporary struct arch_hw_breakpoint
copy to prevent in-place mutation and corruption on parse error paths.
- Synchronize logical perf_event attributes by updating
bp->attr.bp_type and bp->attr.bp_len in modify_wide_hw_breakpoint_local().
Changes in v7:
- Update bp->attr.bp_attr so that we can correctly check the
address on it.
- Use -EOPNOTSUPP instead of -ENOSYS.
Changes in v4:
- Update kerneldoc comment about modify_wide_hw_breakpoint_local
according to Randy's comment.
Changes in v2:
- Check type compatibility by checking slot. (Thanks Jinchao!)
---
arch/Kconfig | 10 +++++++
arch/x86/Kconfig | 1 +
include/linux/hw_breakpoint.h | 6 ++++
kernel/events/hw_breakpoint.c | 61 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 78 insertions(+)
diff --git a/arch/Kconfig b/arch/Kconfig
index 959aee9568ff..4a87e843bb4c 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -467,6 +467,16 @@ config HAVE_POST_BREAKPOINT_HOOK
Select this option if your arch implements breakpoints overflow
handler hooks after the target memory is modified.
+config HAVE_REINSTALL_HW_BREAKPOINT
+ bool
+ depends on HAVE_HW_BREAKPOINT
+ help
+ Depending on the arch implementation of hardware breakpoints,
+ some of them are able to update the breakpoint configuration
+ without release and reserve the hardware breakpoint register.
+ What configuration is able to update depends on hardware and
+ software implementation.
+
config HAVE_USER_RETURN_NOTIFIER
bool
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 6b7e14ef8cfb..588218da8f41 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -247,6 +247,7 @@ config X86
select HAVE_GCC_PLUGINS
select HAVE_HW_BREAKPOINT
select HAVE_POST_BREAKPOINT_HOOK
+ select HAVE_REINSTALL_HW_BREAKPOINT
select HAVE_IOREMAP_PROT
select HAVE_IRQ_EXIT_ON_IRQ_STACK if X86_64
select HAVE_IRQ_TIME_ACCOUNTING
diff --git a/include/linux/hw_breakpoint.h b/include/linux/hw_breakpoint.h
index db199d653dd1..6754ffbee9ed 100644
--- a/include/linux/hw_breakpoint.h
+++ b/include/linux/hw_breakpoint.h
@@ -81,6 +81,9 @@ register_wide_hw_breakpoint(struct perf_event_attr *attr,
perf_overflow_handler_t triggered,
void *context);
+extern int modify_wide_hw_breakpoint_local(struct perf_event *bp,
+ struct perf_event_attr *attr);
+
extern int register_perf_hw_breakpoint(struct perf_event *bp);
extern void unregister_hw_breakpoint(struct perf_event *bp);
extern void unregister_wide_hw_breakpoint(struct perf_event * __percpu *cpu_events);
@@ -124,6 +127,9 @@ register_wide_hw_breakpoint(struct perf_event_attr *attr,
perf_overflow_handler_t triggered,
void *context) { return NULL; }
static inline int
+modify_wide_hw_breakpoint_local(struct perf_event *bp,
+ struct perf_event_attr *attr) { return -EOPNOTSUPP; }
+static inline int
register_perf_hw_breakpoint(struct perf_event *bp) { return -ENOSYS; }
static inline void unregister_hw_breakpoint(struct perf_event *bp) { }
static inline void
diff --git a/kernel/events/hw_breakpoint.c b/kernel/events/hw_breakpoint.c
index 789add0c185a..63570a5079bf 100644
--- a/kernel/events/hw_breakpoint.c
+++ b/kernel/events/hw_breakpoint.c
@@ -888,6 +888,67 @@ void unregister_wide_hw_breakpoint(struct perf_event * __percpu *cpu_events)
}
EXPORT_SYMBOL_GPL(unregister_wide_hw_breakpoint);
+/**
+ * modify_wide_hw_breakpoint_local - update breakpoint config for local CPU
+ * @bp: the hwbp perf event for this CPU
+ * @attr: the new attribute for @bp
+ *
+ * This does not release and reserve the slot of a HWBP; it just reuses the
+ * current slot on local CPU. So the users must update the other CPUs by
+ * themselves.
+ * Also, since this does not release/reserve the slot, this can not change the
+ * type to incompatible type of the HWBP.
+ * Return err if attr is invalid or the CPU fails to update debug register
+ * for new @attr.
+ */
+#ifdef CONFIG_HAVE_REINSTALL_HW_BREAKPOINT
+int modify_wide_hw_breakpoint_local(struct perf_event *bp,
+ struct perf_event_attr *attr)
+{
+ struct arch_hw_breakpoint info, old_info;
+ unsigned long old_addr;
+ unsigned int old_type, old_len;
+ int ret;
+
+ lockdep_assert_irqs_disabled();
+
+ if (find_slot_idx(bp->attr.bp_type) != find_slot_idx(attr->bp_type))
+ return -EINVAL;
+
+ ret = hw_breakpoint_arch_parse(bp, attr, &info);
+ if (ret)
+ return ret;
+
+ old_info = *counter_arch_bp(bp);
+ old_addr = bp->attr.bp_addr;
+ old_type = bp->attr.bp_type;
+ old_len = bp->attr.bp_len;
+
+ *counter_arch_bp(bp) = info;
+ bp->attr.bp_addr = attr->bp_addr;
+ bp->attr.bp_type = attr->bp_type;
+ bp->attr.bp_len = attr->bp_len;
+
+ ret = arch_reinstall_hw_breakpoint(bp);
+ if (ret) {
+ /* Rollback to the original state */
+ *counter_arch_bp(bp) = old_info;
+ bp->attr.bp_addr = old_addr;
+ bp->attr.bp_type = old_type;
+ bp->attr.bp_len = old_len;
+ }
+
+ return ret;
+}
+#else
+int modify_wide_hw_breakpoint_local(struct perf_event *bp,
+ struct perf_event_attr *attr)
+{
+ return -EOPNOTSUPP;
+}
+#endif
+EXPORT_SYMBOL_GPL(modify_wide_hw_breakpoint_local);
+
/**
* hw_breakpoint_is_used - check if breakpoints are currently used
*
^ permalink raw reply related
* [PATCH v10 08/11] tracing: wprobe: Add wprobe event trigger
From: Masami Hiramatsu (Google) @ 2026-07-22 23:04 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add wprobe event trigger to set and clear the watch event dynamically.
This allows us to set an watchpoint on a given local variables and
a slab object instead of static objects.
The trigger syntax is below:
- set_wprobe:WPROBE:FIELD[+OFFSET] [if FILTER]
- clear_wprobe:WPROBE[:FIELD[+OFFSET]] [if FILTER]
set_wprobe sets the address pointed by FIELD[+offset] to the WPROBE
event. The FIELD is the field name of trigger event.
clear_wprobe clears the watch address of WPROBE event. If the FIELD
option is specified, it clears only if the current watch address is
same as the given FIELD[+OFFSET] value.
The set_wprobe trigger does not change the type and length, these
must be set when creating a new wprobe.
Also, the WPROBE event must be disabled when setting the new trigger
and it will be busy afterwards. Recommended usage is to add a new
wprobe at NULL address and keep disabled.
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Make event_trigger_free() non-static to solve build dependency.
- Sync irq_work and work inside __unregister_trace_wprobe() before
clearing tw->bp_event to prevent concurrent NULL pointer dereference.
- Introduce private_free destructor in struct event_trigger_data to
safely release wprobe_trigger_data after tracepoint readers exit.
- Fix filter memory leak in wprobe_trigger_cmd_parse() on the error
handling path of trace_event_try_get_ref() failure.
- Avoid overwriting tw->addr on clear_wprobe trigger registration to
prevent active watchpoint corruption and hardware breakpoint leak.
Changes in v8:
- Redesign wprobe_trigger() to be safe in NMI/hardirq contexts by
deferring register updates to a workqueue via irq_work.
- Skip trigger execution and increment an atomic missed count
(tw->missed) if the tracepoint runs in NMI context to prevent
recursive spinlock deadlocks. (this is currently hidden counter)
- Prohibit attaching wprobe triggers to kprobe_events by checking
TRACE_EVENT_FL_KPROBE in wprobe_trigger_cmd_parse().
- Use call_rcu() in trigger deactivation path and add
synchronize_rcu() in parse failure path to ensure safe RCU
lifetime cleanup.
- Acquire the target tracepoint's module reference via
trace_event_try_get_ref() to prevent module refcount underflows.
- Fix event_trigger_data memory leak by properly freeing the
initial refcount in wprobe_trigger_cmd_parse() on success.
- Call on_each_cpu() with wait=true in wprobe_work_func() to
prevent use-after-free during trigger unregistration.
- Synchronize concurrent wprobe triggers on the same event by
using a shared raw spinlock (tw->lock).
- Drop the support of kprobe events (that should be done later).
Changes in v7:
- Use kzalloc_obj().
- Update sample code in document.
Changes in v6:
- Update according to the latest change of trigger ops.
Changes in v5:
- Following the suggestions, the documentation was revised to suit rst.
Changes in v3:
- Add FIELD option support for clear_wprobe and update document.
- Fix to unregister/free event_trigger_data on file correctly.
- Fix syntax comments.
Changes in v2:
- Getting local cpu perf_event from trace_wprobe directly.
- Remove trace_wprobe_local_perf() because it is conditionally unused.
- Make CONFIG_WPROBE_TRIGGERS a hidden config.
---
Documentation/trace/wprobetrace.rst | 93 +++++++
include/linux/trace_events.h | 1
kernel/trace/Kconfig | 10 +
kernel/trace/trace.h | 2
kernel/trace/trace_events_trigger.c | 12 +
kernel/trace/trace_wprobe.c | 438 +++++++++++++++++++++++++++++++++++
6 files changed, 553 insertions(+), 3 deletions(-)
diff --git a/Documentation/trace/wprobetrace.rst b/Documentation/trace/wprobetrace.rst
index eb4f10607530..a4c0f0e676fd 100644
--- a/Documentation/trace/wprobetrace.rst
+++ b/Documentation/trace/wprobetrace.rst
@@ -68,3 +68,96 @@ Here is an example to add a wprobe event on a variable `jiffies`.
<idle>-0 [000] d.Z1. 717.026373: my_jiffies: (tick_do_update_jiffies64+0xbe/0x130)
You can see the code which writes to `jiffies` is `tick_do_update_jiffies64()`.
+
+Combination with trigger action
+-------------------------------
+The event trigger action can extend the utilization of this wprobe.
+
+- set_wprobe:WPEVENT:FIELD[+|-ADJUST]
+- clear_wprobe:WPEVENT[:FIELD[+|-]ADJUST]
+
+Set these triggers to the target event, then the WPROBE event will be
+setup to trace the memory access at FIELD[+|-ADJUST] address.
+When clear_wprobe is hit, if FIELD is NOT specified, the WPEVENT is
+forcibly cleared. If FIELD[[+|-]ADJUST] is set, it clears WPEVENT only
+if its watching address is the same as the FIELD[[+|-]ADJUST] value.
+
+Notes:
+The set_wprobe trigger does not change the type and length, these
+must be set when creating a new wprobe.
+
+The WPROBE event must be disabled when setting the new trigger
+and it will be busy afterwards. Recommended usage is to add a new
+wprobe at NULL address and keep disabled.
+
+Wprobe triggers are not supported on kprobe_events, because kprobes
+themselves can use software breakpoints which conflicts with wprobe
+operation.
+
+
+For example, trace the first 8 bytes of the dentry data structure passed
+to do_truncate() until it is deleted by dentry_kill().
+(Note: all tracefs setup uses '>>' so that it does not kick do_truncate())
+::
+
+ # echo 'w:watch rw@0:8 address=$addr value=+0($addr)' >> dynamic_events
+ # echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
+ # echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
+ # echo 'f:dentry_kill dentry_kill dentry=$arg1' >> dynamic_events
+ # echo 'clear_wprobe:watch:dentry' >> events/fprobes/dentry_kill/trigger
+ # echo 1 >> events/fprobes/truncate/enable
+ # echo 1 >> events/fprobes/dentry_kill/enable
+
+ # echo aaa > /tmp/hoge
+ # echo bbb > /tmp/hoge
+ # echo ccc > /tmp/hoge
+ # rm /tmp/hoge
+
+Then, the trace data will show::
+
+ # tracer: nop
+ #
+ # entries-in-buffer/entries-written: 32/32 #P:8
+ #
+ # _-----=> irqs-off/BH-disabled
+ # / _----=> need-resched
+ # | / _---=> hardirq/softirq
+ # || / _--=> preempt-depth
+ # ||| / _-=> migrate-disable
+ # |||| / delay
+ # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
+ # | | | ||||| | |
+ sh-107 [004] ...1. 9.990418: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618
+ sh-107 [004] ...1. 9.990914: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b3de78
+ sh-107 [004] ...1. 9.993175: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049ddd40
+ sh-107 [004] ..... 9.995198: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880048083a8
+ sh-107 [004] ...1. 9.995389: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db998
+ sh-107 [004] ..Zff 9.997503: watch: (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080
+ sh-107 [004] ..Zff 9.997509: watch: (path_openat+0x211/0xda0) address=0xffff8880048083a8 value=0x8200080
+ sh-107 [004] ..Zff 9.997514: watch: (path_openat+0xa56/0xda0) address=0xffff8880048083a8 value=0x8200080
+ sh-107 [004] ..Zff 9.997518: watch: (path_openat+0xae2/0xda0) address=0xffff8880048083a8 value=0x8200080
+ sh-107 [004] ..... 9.997521: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880048083a8
+ sh-107 [004] ...1. 9.997582: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004808270
+ sh-107 [004] ...1. 9.999365: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db728
+ sh-107 [004] ...1. 9.999388: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b1c000
+ rm-113 [005] ..Zff 10.000965: watch: (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.000971: watch: (path_lookupat+0x97/0x1e0) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.000984: watch: (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.000988: watch: (path_lookupat+0x97/0x1e0) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.001010: watch: (lookup_one_qstr_excl+0x28/0x140) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.001014: watch: (lookup_one_qstr_excl+0xd1/0x140) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.001018: watch: (may_delete_dentry+0x1c/0x200) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.001021: watch: (may_delete_dentry+0x195/0x200) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] ..Zff 10.001031: watch: (vfs_unlink+0x5e/0x260) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] d.Z.. 10.001067: watch: (d_make_discardable+0x1b/0x40) address=0xffff8880048083a8 value=0x8200080
+ rm-113 [005] d.Z.. 10.001071: watch: (d_make_discardable+0x29/0x40) address=0xffff8880048083a8 value=0x200080
+ rm-113 [005] ...1. 10.001072: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880048083a8
+ rm-113 [005] ...1. 10.001218: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880048083a8
+ sh-107 [004] ...1. 10.001416: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db110
+ sh-107 [004] ...1. 10.001444: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db248
+ sh-107 [004] ...1. 10.001500: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618
+ sh-107 [004] ...1. 10.002067: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b41e78
+ sh-107 [004] ...1. 10.904920: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b41e78
+ sh-107 [004] ...1. 10.905129: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618
+
+You can see the watch event is correctly configured on the dentry.
diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
index d1e5ab71d928..e6f3bbcbb9af 100644
--- a/include/linux/trace_events.h
+++ b/include/linux/trace_events.h
@@ -729,6 +729,7 @@ enum event_trigger_type {
ETT_EVENT_HIST = (1 << 4),
ETT_HIST_ENABLE = (1 << 5),
ETT_EVENT_EPROBE = (1 << 6),
+ ETT_EVENT_WPROBE = (1 << 7),
};
extern int filter_match_preds(struct event_filter *filter, void *rec);
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index d9b6fa5c35d9..af570fa2a882 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -876,6 +876,16 @@ config WPROBE_EVENTS
Those events can be inserted wherever hardware breakpoints can be
set, and record accessed memory address and values.
+config WPROBE_TRIGGERS
+ depends on WPROBE_EVENTS
+ depends on HAVE_REINSTALL_HW_BREAKPOINT
+ bool
+ default y
+ help
+ This adds an event trigger which will set the wprobe on a specific
+ field of an event. This allows user to trace the memory access of
+ an address pointed by the event field.
+
config BPF_EVENTS
depends on BPF_SYSCALL
depends on (KPROBE_EVENTS || UPROBE_EVENTS) && PERF_EVENTS
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 2f07c5c4ffc8..e5964a69057c 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -1939,6 +1939,7 @@ struct event_trigger_data {
struct event_filter __rcu *filter;
char *filter_str;
void *private_data;
+ void (*private_free)(void *private_data);
bool paused;
bool paused_tmp;
struct list_head list;
@@ -1982,6 +1983,7 @@ trigger_data_alloc(struct event_command *cmd_ops, char *cmd, char *param,
void *private_data);
extern void trigger_data_free(struct event_trigger_data *data);
extern int event_trigger_init(struct event_trigger_data *data);
+extern void event_trigger_free(struct event_trigger_data *data);
extern int trace_event_trigger_enable_disable(struct trace_event_file *file,
int trigger_enable);
extern void update_cond_flag(struct trace_event_file *file);
diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
index 655db2e82513..1e3b667a2edd 100644
--- a/kernel/trace/trace_events_trigger.c
+++ b/kernel/trace/trace_events_trigger.c
@@ -51,8 +51,11 @@ static void trigger_data_free_queued_locked(void)
tracepoint_synchronize_unregister();
- llist_for_each_entry_safe(data, tmp, llnodes, llist)
+ llist_for_each_entry_safe(data, tmp, llnodes, llist) {
+ if (data->private_free)
+ data->private_free(data->private_data);
kfree(data);
+ }
}
/* Bulk garbage collection of event_trigger_data elements */
@@ -74,8 +77,11 @@ static int trigger_kthread_fn(void *ignore)
/* make sure current triggers exit before free */
tracepoint_synchronize_unregister();
- llist_for_each_entry_safe(data, tmp, llnodes, llist)
+ llist_for_each_entry_safe(data, tmp, llnodes, llist) {
+ if (data->private_free)
+ data->private_free(data->private_data);
kfree(data);
+ }
}
return 0;
@@ -582,7 +588,7 @@ int event_trigger_init(struct event_trigger_data *data)
* Usually used directly as the @free method in event trigger
* implementations.
*/
-static void
+void
event_trigger_free(struct event_trigger_data *data)
{
if (WARN_ON_ONCE(data->ref <= 0))
diff --git a/kernel/trace/trace_wprobe.c b/kernel/trace/trace_wprobe.c
index 4c23c310a383..1edfdddf241d 100644
--- a/kernel/trace/trace_wprobe.c
+++ b/kernel/trace/trace_wprobe.c
@@ -6,7 +6,9 @@
*/
#define pr_fmt(fmt) "trace_wprobe: " fmt
+#include <linux/atomic.h>
#include <linux/compiler.h>
+#include <linux/errno.h>
#include <linux/hw_breakpoint.h>
#include <linux/kallsyms.h>
#include <linux/list.h>
@@ -15,11 +17,16 @@
#include <linux/perf_event.h>
#include <linux/rculist.h>
#include <linux/security.h>
+#include <linux/spinlock.h>
#include <linux/tracepoint.h>
#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+#include <linux/irq_work.h>
+#include <linux/preempt.h>
#include <asm/ptrace.h>
+#include "trace.h"
#include "trace_dynevent.h"
#include "trace_probe.h"
#include "trace_probe_kernel.h"
@@ -50,6 +57,10 @@ struct trace_wprobe {
int len;
int type;
const char *symbol;
+ raw_spinlock_t lock;
+ struct irq_work irq_work;
+ struct work_struct work;
+ atomic_t missed;
struct trace_probe tp;
};
@@ -198,14 +209,49 @@ static int __register_trace_wprobe(struct trace_wprobe *tw)
static void __unregister_trace_wprobe(struct trace_wprobe *tw)
{
if (tw->bp_event) {
+ irq_work_sync(&tw->irq_work);
+ cancel_work_sync(&tw->work);
unregister_wide_hw_breakpoint(tw->bp_event);
tw->bp_event = NULL;
}
}
+static int trace_wprobe_update_local(struct trace_wprobe *tw, unsigned long addr)
+{
+ struct perf_event *bp = *this_cpu_ptr(tw->bp_event);
+ struct perf_event_attr attr = bp->attr;
+
+ attr.bp_addr = addr;
+ return modify_wide_hw_breakpoint_local(bp, &attr);
+}
+
+static void wprobe_smp_update_func(void *info)
+{
+ struct trace_wprobe *tw = info;
+ unsigned long addr = READ_ONCE(tw->addr);
+
+ trace_wprobe_update_local(tw, addr);
+}
+
+static void wprobe_work_func(struct work_struct *work)
+{
+ struct trace_wprobe *tw = container_of(work, struct trace_wprobe, work);
+
+ on_each_cpu(wprobe_smp_update_func, tw, true);
+}
+
+static void wprobe_irq_work_func(struct irq_work *irq_work)
+{
+ struct trace_wprobe *tw = container_of(irq_work, struct trace_wprobe, irq_work);
+
+ schedule_work(&tw->work);
+}
+
static void free_trace_wprobe(struct trace_wprobe *tw)
{
if (tw) {
+ irq_work_sync(&tw->irq_work);
+ cancel_work_sync(&tw->work);
trace_probe_cleanup(&tw->tp);
kfree(tw->symbol);
kfree(tw);
@@ -237,6 +283,10 @@ static struct trace_wprobe *alloc_trace_wprobe(const char *group,
tw->addr = addr;
tw->len = len;
tw->type = type;
+ raw_spin_lock_init(&tw->lock);
+ init_irq_work(&tw->irq_work, wprobe_irq_work_func);
+ INIT_WORK(&tw->work, wprobe_work_func);
+ atomic_set(&tw->missed, 0);
ret = trace_probe_init(&tw->tp, event, group, false, nargs);
if (ret < 0)
@@ -744,3 +794,391 @@ static __init int init_wprobe_trace(void)
}
fs_initcall(init_wprobe_trace);
+#ifdef CONFIG_WPROBE_TRIGGERS
+
+static int wprobe_trigger_global_enabled;
+
+#define SET_WPROBE_STR "set_wprobe"
+#define CLEAR_WPROBE_STR "clear_wprobe"
+#define WPROBE_DEFAULT_CLEAR_ADDRESS ((unsigned long)&wprobe_trigger_global_enabled)
+
+struct wprobe_trigger_data {
+ struct rcu_head rcu;
+ struct trace_event_file *file;
+ struct trace_wprobe *tw;
+ unsigned int offset;
+ long adjust;
+ const char *field;
+ bool clear;
+};
+
+static void wprobe_trigger(struct event_trigger_data *data,
+ struct trace_buffer *buffer, void *rec,
+ struct ring_buffer_event *event)
+{
+ struct wprobe_trigger_data *wprobe_data = data->private_data;
+ struct trace_wprobe *tw = wprobe_data->tw;
+ unsigned long addr, flags;
+ bool changed = false;
+
+ addr = *(unsigned long *)(rec + wprobe_data->offset);
+ addr += wprobe_data->adjust;
+
+ if (in_nmi()) {
+ atomic_inc(&tw->missed);
+ return;
+ }
+
+ raw_spin_lock_irqsave(&tw->lock, flags);
+
+ if (!wprobe_data->clear) {
+ if (tw->addr == WPROBE_DEFAULT_CLEAR_ADDRESS) {
+ tw->addr = addr;
+ changed = true;
+ clear_bit(EVENT_FILE_FL_SOFT_DISABLED_BIT, &wprobe_data->file->flags);
+ }
+ } else {
+ if (tw->addr != WPROBE_DEFAULT_CLEAR_ADDRESS) {
+ if (!wprobe_data->field || tw->addr == addr) {
+ tw->addr = WPROBE_DEFAULT_CLEAR_ADDRESS;
+ changed = true;
+ set_bit(EVENT_FILE_FL_SOFT_DISABLED_BIT, &wprobe_data->file->flags);
+ }
+ }
+ }
+
+ if (changed)
+ irq_work_queue(&tw->irq_work);
+
+ raw_spin_unlock_irqrestore(&tw->lock, flags);
+}
+
+static void free_wprobe_trigger_data(struct wprobe_trigger_data *wprobe_data)
+{
+ if (wprobe_data) {
+ kfree(wprobe_data->field);
+ kfree(wprobe_data);
+ }
+}
+DEFINE_FREE(free_wprobe_trigger_data, struct wprobe_trigger_data *, free_wprobe_trigger_data(_T));
+
+static void free_private_wprobe_trigger_data(void *data)
+{
+ free_wprobe_trigger_data(data);
+}
+
+static int wprobe_trigger_print(struct seq_file *m,
+ struct event_trigger_data *data)
+{
+ struct wprobe_trigger_data *wprobe_data = data->private_data;
+
+ if (wprobe_data->clear) {
+ seq_printf(m, "%s:%s", CLEAR_WPROBE_STR,
+ trace_event_name(wprobe_data->file->event_call));
+ if (wprobe_data->field) {
+ seq_printf(m, ":%s%+ld",
+ wprobe_data->field, wprobe_data->adjust);
+ }
+ } else
+ seq_printf(m, "%s:%s:%s%+ld", SET_WPROBE_STR,
+ trace_event_name(wprobe_data->file->event_call),
+ wprobe_data->field, wprobe_data->adjust);
+
+ if (data->filter_str)
+ seq_printf(m, " if %s\n", data->filter_str);
+ else
+ seq_putc(m, '\n');
+
+ return 0;
+}
+
+static struct wprobe_trigger_data *
+wprobe_trigger_alloc(struct trace_wprobe *tw, struct trace_event_file *file,
+ bool clear)
+{
+ struct wprobe_trigger_data *wprobe_data;
+
+ wprobe_data = kzalloc_obj(*wprobe_data);
+ if (!wprobe_data)
+ return NULL;
+
+ wprobe_data->tw = tw;
+ wprobe_data->clear = clear;
+ wprobe_data->file = file;
+
+ return wprobe_data;
+}
+
+static void wprobe_trigger_free(struct event_trigger_data *data)
+{
+ struct wprobe_trigger_data *wprobe_data = data->private_data;
+
+ if (WARN_ON_ONCE(data->ref <= 0))
+ return;
+
+ data->ref--;
+ if (!data->ref) {
+ /* Remove the SOFT_MODE flag */
+ trace_event_enable_disable(wprobe_data->file, 0, 1);
+ trace_event_put_ref(wprobe_data->file->event_call);
+ trigger_data_free(data);
+ }
+}
+
+static int wprobe_trigger_cmd_parse(struct event_command *cmd_ops,
+ struct trace_event_file *file,
+ char *glob, char *cmd,
+ char *param_and_filter)
+{
+ /*
+ * set_wprobe:EVENT:FIELD[+OFFS]
+ * clear_wprobe:EVENT[:FIELD[+OFFS]]
+ */
+ struct wprobe_trigger_data *wprobe_data __free(free_wprobe_trigger_data) = NULL;
+ struct event_trigger_data *trigger_data __free(kfree) = NULL;
+ struct ftrace_event_field *field = NULL;
+ struct trace_event_file *wprobe_file;
+ struct trace_array *tr = file->tr;
+ struct trace_event_call *event;
+ char *event_str, *field_str;
+ bool remove, clear = false;
+ struct trace_wprobe *tw;
+ char *param, *filter;
+ int ret;
+
+ remove = event_trigger_check_remove(glob);
+
+ if (!strcmp(cmd, CLEAR_WPROBE_STR))
+ clear = true;
+
+ if (event_trigger_empty_param(param_and_filter))
+ return -EINVAL;
+
+ ret = event_trigger_separate_filter(param_and_filter, ¶m, &filter, true);
+ if (ret)
+ return ret;
+
+ if (file->event_call->flags & TRACE_EVENT_FL_KPROBE)
+ return -EOPNOTSUPP;
+
+ event_str = strsep(¶m, ":");
+
+ /* Find target wprobe */
+ tw = find_trace_wprobe(event_str, WPROBE_EVENT_SYSTEM);
+ if (!tw)
+ return -ENOENT;
+ /* The target wprobe must not be used (unless clear) */
+ if (!remove && !clear && trace_probe_is_enabled(&tw->tp))
+ return -EBUSY;
+
+ wprobe_file = find_event_file(tr, WPROBE_EVENT_SYSTEM, event_str);
+ if (!wprobe_file)
+ return -EINVAL;
+
+ wprobe_data = wprobe_trigger_alloc(tw, wprobe_file, clear);
+ if (!wprobe_data)
+ return -ENOMEM;
+
+ /* Find target field, which must be equivarent to "void *" */
+ field_str = strsep(¶m, ":");
+ /* trigger removing or clear_wprobe does not need field. */
+ if (!remove && !clear && !field_str)
+ return -EINVAL;
+
+ if (field_str) {
+ char *offs;
+
+ offs = strpbrk(field_str, "+-");
+ if (offs) {
+ long val;
+
+ if (kstrtol(offs, 0, &val) < 0)
+ return -EINVAL;
+ wprobe_data->adjust = val;
+ *offs = '\0';
+ }
+
+ event = file->event_call;
+ field = trace_find_event_field(event, field_str);
+ if (!field)
+ return -ENOENT;
+
+ if (field->size != sizeof(void *))
+ return -ENOEXEC;
+ wprobe_data->offset = field->offset;
+ wprobe_data->field = kstrdup(field_str, GFP_KERNEL);
+ if (!wprobe_data->field)
+ return -ENOMEM;
+ }
+
+ trigger_data = trigger_data_alloc(cmd_ops, cmd, param, wprobe_data);
+ if (!trigger_data)
+ return -ENOMEM;
+
+ trigger_data->private_free = free_private_wprobe_trigger_data;
+
+ /* Up the trigger_data count to make sure nothing frees it on failure */
+ event_trigger_init(trigger_data);
+
+ if (remove) {
+ event_trigger_unregister(cmd_ops, file, glob+1, trigger_data);
+ return 0;
+ }
+
+ ret = event_trigger_parse_num(param, trigger_data);
+ if (ret)
+ return ret;
+
+ ret = event_trigger_set_filter(cmd_ops, file, filter, trigger_data);
+ if (ret < 0)
+ return ret;
+
+ /* Soft-enable (register) wprobe event on WPROBE_DEFAULT_CLEAR_ADDRESS */
+ if (!trace_event_try_get_ref(wprobe_file->event_call)) {
+ event_trigger_reset_filter(cmd_ops, trigger_data);
+ return -ENODEV;
+ }
+
+ if (!clear)
+ WRITE_ONCE(tw->addr, WPROBE_DEFAULT_CLEAR_ADDRESS);
+ ret = trace_event_enable_disable(wprobe_file, 1, 1);
+ if (ret < 0) {
+ trace_event_put_ref(wprobe_file->event_call);
+ event_trigger_reset_filter(cmd_ops, trigger_data);
+ return ret;
+ }
+ ret = event_trigger_register(cmd_ops, file, glob, trigger_data);
+ if (ret) {
+ event_trigger_reset_filter(cmd_ops, trigger_data);
+ trace_event_enable_disable(wprobe_file, 0, 1);
+ trace_event_put_ref(wprobe_file->event_call);
+ synchronize_rcu();
+ return ret;
+ }
+ /* Make it NULL to avoid freeing trigger_data and wprobe_data by __free() */
+ wprobe_data = NULL;
+ event_trigger_free(trigger_data);
+ trigger_data = NULL;
+
+ return 0;
+}
+
+/* Return event_trigger_data if there is a trigger which points the same wprobe */
+static struct event_trigger_data *
+wprobe_trigger_find_same(struct event_trigger_data *test,
+ struct trace_event_file *file)
+{
+ struct wprobe_trigger_data *test_wprobe_data = test->private_data;
+ struct wprobe_trigger_data *wprobe_data;
+ struct event_trigger_data *iter;
+
+ list_for_each_entry(iter, &file->triggers, list) {
+ wprobe_data = iter->private_data;
+ if (!wprobe_data ||
+ iter->cmd_ops->trigger_type !=
+ test->cmd_ops->trigger_type)
+ continue;
+ if (wprobe_data->tw == test_wprobe_data->tw)
+ return iter;
+ }
+ return NULL;
+}
+
+static int wprobe_register_trigger(char *glob,
+ struct event_trigger_data *data,
+ struct trace_event_file *file)
+{
+ int ret = 0;
+
+ lockdep_assert_held(&event_mutex);
+
+ /* The same wprobe is not accept on the same file (event) */
+ if (wprobe_trigger_find_same(data, file))
+ return -EEXIST;
+
+ if (data->cmd_ops->init) {
+ ret = data->cmd_ops->init(data);
+ if (ret < 0)
+ return ret;
+ }
+
+ list_add_rcu(&data->list, &file->triggers);
+
+ update_cond_flag(file);
+ ret = trace_event_trigger_enable_disable(file, 1);
+ if (ret < 0) {
+ list_del_rcu(&data->list);
+ update_cond_flag(file);
+ }
+ return ret;
+}
+
+static void wprobe_unregister_trigger(char *glob,
+ struct event_trigger_data *test,
+ struct trace_event_file *file)
+{
+ struct event_trigger_data *data;
+
+ lockdep_assert_held(&event_mutex);
+
+ data = wprobe_trigger_find_same(test, file);
+ if (!data)
+ return;
+
+ list_del_rcu(&data->list);
+ trace_event_trigger_enable_disable(file, 0);
+ update_cond_flag(file);
+ if (data->cmd_ops->free)
+ data->cmd_ops->free(data);
+}
+
+static struct event_command trigger_wprobe_set_cmd = {
+ .name = SET_WPROBE_STR,
+ .trigger_type = ETT_EVENT_WPROBE,
+ /* This triggers after when the event is recorded. */
+ .flags = EVENT_CMD_FL_NEEDS_REC,
+ .parse = wprobe_trigger_cmd_parse,
+ .reg = wprobe_register_trigger,
+ .unreg = wprobe_unregister_trigger,
+ .set_filter = set_trigger_filter,
+ .trigger = wprobe_trigger,
+ .count_func = event_trigger_count,
+ .print = wprobe_trigger_print,
+ .init = event_trigger_init,
+ .free = wprobe_trigger_free,
+};
+
+static struct event_command trigger_wprobe_clear_cmd = {
+ .name = CLEAR_WPROBE_STR,
+ .trigger_type = ETT_EVENT_WPROBE,
+ /* This triggers after when the event is recorded. */
+ .flags = EVENT_CMD_FL_NEEDS_REC,
+ .parse = wprobe_trigger_cmd_parse,
+ .reg = wprobe_register_trigger,
+ .unreg = wprobe_unregister_trigger,
+ .set_filter = set_trigger_filter,
+ .trigger = wprobe_trigger,
+ .count_func = event_trigger_count,
+ .print = wprobe_trigger_print,
+ .init = event_trigger_init,
+ .free = wprobe_trigger_free,
+};
+
+static __init int init_trigger_wprobe_cmds(void)
+{
+ int ret;
+
+ ret = register_event_command(&trigger_wprobe_set_cmd);
+ if (WARN_ON(ret < 0))
+ return ret;
+ ret = register_event_command(&trigger_wprobe_clear_cmd);
+ if (WARN_ON(ret < 0))
+ unregister_event_command(&trigger_wprobe_set_cmd);
+
+ if (!ret)
+ wprobe_trigger_global_enabled = 1;
+
+ return ret;
+}
+fs_initcall(init_trigger_wprobe_cmds);
+#endif /* CONFIG_WPROBE_TRIGGERS */
^ permalink raw reply related
* [PATCH v10 09/11] selftests: ftrace: Add wprobe trigger testcase
From: Masami Hiramatsu (Google) @ 2026-07-22 23:04 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add a testcase for checking wprobe trigger. This sets set_wprobe and
clear_wprobe triggers on fprobe events to watch dentry access.
So this depends on both wprobe and fprobe.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Drop dentry cache after removing tempfile for on-disk
filesystem.
Changes in v8:
- Use TMPDIR for making a test file.
- Ensure the fprobe and target functions exists, if not, exit
with UNRESOLVED (== skip).
Changes in v7:
- Add a newline at the end of file.(style fix)
- Use dentry_kill instead of __dentry_kill.
Changes in v5:
- Enable CONFIG_WPROBE_TRIGGERS in the config for ftrace test.
Changes in v3:
- Newly added.
---
tools/testing/selftests/ftrace/config | 1
.../ftrace/test.d/trigger/trigger-wprobe.tc | 70 ++++++++++++++++++++
2 files changed, 71 insertions(+)
create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe.tc
diff --git a/tools/testing/selftests/ftrace/config b/tools/testing/selftests/ftrace/config
index d2f503722020..ecdee77f360f 100644
--- a/tools/testing/selftests/ftrace/config
+++ b/tools/testing/selftests/ftrace/config
@@ -28,3 +28,4 @@ CONFIG_TRACER_SNAPSHOT=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y
CONFIG_WPROBE_EVENTS=y
+CONFIG_WPROBE_TRIGGERS=y
diff --git a/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe.tc b/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe.tc
new file mode 100644
index 000000000000..0634259e2c65
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe.tc
@@ -0,0 +1,70 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event trigger - test wprobe trigger
+# requires: dynamic_events "w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]":README events/sched/sched_process_fork/trigger
+
+echo 0 > tracing_on
+rm -f $TMPDIR/hoge
+
+:;: "Add a wprobe event used by trigger" ;:
+echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events
+
+# we will skip this test if fprobe is not supported.
+if ! grep -Fq "f[:[<group>/][<event>]] <func-name>[%return] [<args>]" README; then
+ echo "UNRESOLVED: fprobe is not supported"
+ exit_unresolved
+fi
+
+# we will skip this test if the target function does not exist.
+if ! grep -wq "do_truncate" /proc/kallsyms; then
+ echo "UNRESOLVED: do_truncate not found"
+ exit_unresolved
+fi
+if ! grep -wq "dentry_kill" /proc/kallsyms; then
+ echo "UNRESOLVED: dentry_kill not found"
+ exit_unresolved
+fi
+
+:;: "Add events for triggering wprobe" ;:
+echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
+echo 'f:dentry_kill dentry_kill dentry=$arg1' >> dynamic_events
+
+:;: "Add wprobe triggers" ;:
+echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
+echo 'clear_wprobe:watch:dentry' >> events/fprobes/dentry_kill/trigger
+cat events/fprobes/truncate/trigger | grep ^set_wprobe
+cat events/fprobes/dentry_kill/trigger | grep ^clear_wprobe
+
+:;: "Ensure wprobe is still disabled" ;:
+cat events/wprobes/watch/enable | grep 0
+
+:;: "Enable events for triggers" ;:
+echo 1 >> events/fprobes/truncate/enable
+echo 1 >> events/fprobes/dentry_kill/enable
+
+:;: "Start test workload" ;:
+echo 1 >> tracing_on
+
+echo aaa > $TMPDIR/hoge
+echo bbb > $TMPDIR/hoge
+echo ccc > $TMPDIR/hoge
+rm $TMPDIR/hoge
+
+:;: "Drop dentry caches (for dentry_kill)" ;:
+# Use append (>>) to avoid calling do_truncate again.
+sync && echo 2 >> /proc/sys/vm/drop_caches
+
+:;: "Check trace results" ;:
+cat trace | grep watch
+
+
+:;: "Ensure wprobe becomes disabled again" ;:
+cat events/wprobes/watch/enable | grep 0
+
+:;: "Remove wprobe triggers" ;:
+echo '!set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
+echo '!clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger
+! grep ^set_wprobe events/fprobes/truncate/trigger
+! grep ^clear_wprobe events/fprobes/dentry_kill/trigger
+
+exit 0
^ permalink raw reply related
* [PATCH v10 10/11] tracing/wprobe: Support BTF typecast in fetchargs
From: Masami Hiramatsu (Google) @ 2026-07-22 23:04 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Allow BTF typecast syntax (STRUCT)FETCHARG->MEMBER in wprobe event
fetchargs. Previously, handle_typecast() rejected any probe context
that was not a function entry/return or tracepoint event probe.
Wprobe events use (the accessed address) and (the value
at that address). By enabling BTF typecast, users can now cast these
to a concrete struct type and access its fields directly. For example:
echo 'w:watch rw@0:8 dflag=(struct dentry)$addr->d_flags' >> dynamic_events
With a set_wprobe trigger pointing the watchpoint at a dentry address,
the resulting trace shows d_flags being accessed at that location.
Note that and are restricted to kernel-space memory,
which is consistent with the existing TPARG_FL_KERNEL flag used when
parsing wprobe fetchargs.
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Newly added.
---
kernel/trace/trace_probe.c | 3 +
kernel/trace/trace_probe.h | 5 +
.../test.d/trigger/trigger-wprobe-btf-typecast.tc | 80 ++++++++++++++++++++
3 files changed, 87 insertions(+), 1 deletion(-)
create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-typecast.tc
diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c
index 5d5e9b477b86..8332ff1bb4ff 100644
--- a/kernel/trace/trace_probe.c
+++ b/kernel/trace/trace_probe.c
@@ -967,7 +967,8 @@ static int handle_typecast(char *arg, struct traceprobe_parse_context *ctx)
if (!(tparg_is_event_probe(ctx->flags) ||
tparg_is_function_entry(ctx->flags) ||
- tparg_is_function_return(ctx->flags))) {
+ tparg_is_function_return(ctx->flags) ||
+ tparg_is_wprobe(ctx->flags))) {
trace_probe_log_err(ctx->offset, NOSUP_BTFARG);
return -EOPNOTSUPP;
}
diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h
index 7380502a85af..0a83b3fb6128 100644
--- a/kernel/trace/trace_probe.h
+++ b/kernel/trace/trace_probe.h
@@ -439,6 +439,11 @@ static inline bool tparg_is_event_probe(unsigned int flags)
return !!(flags & TPARG_FL_TEVENT);
}
+static inline bool tparg_is_wprobe(unsigned int flags)
+{
+ return !!(flags & TPARG_FL_WPROBE);
+}
+
/* Each typecast consumes nested level. So the max number of typecast is 8. */
#define TRACEPROBE_MAX_NESTED_LEVEL 8
diff --git a/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-typecast.tc b/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-typecast.tc
new file mode 100644
index 000000000000..8962c91d8428
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-typecast.tc
@@ -0,0 +1,80 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event trigger - test wprobe trigger with BTF typecast fetchargs
+# requires: dynamic_events "w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]":README events/sched/sched_process_fork/trigger "[(structname[,field])]<argname>[->field[->field|.field...]]":README
+
+echo 0 >> tracing_on
+
+rm -f $TMPDIR/hoge
+
+# we will skip this test if fprobe is not supported.
+if ! grep -Fq "f[:[<group>/][<event>]] <func-name>[%return] [<args>]" README; then
+ echo "UNRESOLVED: fprobe is not supported"
+ exit_unresolved
+fi
+
+# we will skip this test if the target function does not exist.
+if ! grep -wq "do_truncate" /proc/kallsyms; then
+ echo "UNRESOLVED: do_truncate not found"
+ exit_unresolved
+fi
+if ! grep -wq "dentry_kill" /proc/kallsyms; then
+ echo "UNRESOLVED: dentry_kill not found"
+ exit_unresolved
+fi
+
+:;: "Add a wprobe event with BTF typecast fetchargs" ;:
+# $addr is the address being accessed (= dentry pointer when watching dentry)
+# (dentry)$addr->d_flags reads d_flags from the dentry struct via BTF typecast
+# Note: BTF typecast uses (STRUCT) without the 'struct' keyword, matching
+# the fetcharg syntax used in fprobe/tprobe events.
+echo 'w:watch rw@0:8 address=$addr dflag=(dentry)$addr->d_flags' >> dynamic_events
+
+:;: "Check the wprobe event is registered with dflag field" ;:
+grep -q "dflag" dynamic_events
+
+:;: "Add events for triggering wprobe" ;:
+echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
+echo 'f:dentry_kill dentry_kill dentry=$arg1' >> dynamic_events
+
+:;: "Add wprobe triggers" ;:
+echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
+echo 'clear_wprobe:watch:dentry' >> events/fprobes/dentry_kill/trigger
+cat events/fprobes/truncate/trigger | grep ^set_wprobe
+cat events/fprobes/dentry_kill/trigger | grep ^clear_wprobe
+
+:;: "Ensure wprobe is still disabled" ;:
+cat events/wprobes/watch/enable | grep 0
+
+:;: "Enable events for triggers" ;:
+echo 1 >> events/fprobes/truncate/enable
+echo 1 >> events/fprobes/dentry_kill/enable
+
+:;: "Start test workload" ;:
+echo 1 >> tracing_on
+
+echo aaa > $TMPDIR/hoge
+sleep 1
+echo bbb > $TMPDIR/hoge
+sleep 1
+echo ccc > $TMPDIR/hoge
+sleep 1
+rm $TMPDIR/hoge
+
+:;: "Drop dentry caches (for dentry_kill)" ;:
+sync && echo 2 >> /proc/sys/vm/drop_caches
+
+:;: "Check trace results include BTF typecast field dflag" ;:
+cat trace > /tmp/test-trace-typecast.log
+cat trace | grep "watch.*dflag="
+
+:;: "Ensure wprobe becomes disabled again" ;:
+cat events/wprobes/watch/enable | grep 0
+
+:;: "Remove wprobe triggers" ;:
+echo '!set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
+echo '!clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger
+! grep ^set_wprobe events/fprobes/truncate/trigger
+! grep ^clear_wprobe events/fprobes/dentry_kill/trigger
+
+exit 0
^ permalink raw reply related
* [PATCH v10 11/11] tracing/wprobe: Support BTF typecast in wprobe trigger command
From: Masami Hiramatsu (Google) @ 2026-07-22 23:04 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Extend the set_wprobe trigger syntax to support automatic BTF-based
offset calculation using the form:
set_wprobe:WPEVENT:(TYPE[,ASGN])FIELD->MEMBER[.SUBMEMBER[...]]
Previously, the ADJUST value in FIELD[+/-ADJUST] had to be a numeric
literal, requiring the user to know the exact byte offset of the
target struct member.
With this change, if the FIELD portion starts with (STRUCTTYPE), the
offset of MEMBER within STRUCTTYPE is automatically resolved via BTF.
This allows symbolic, kernel-version-independent watchpoint placement
at specific struct fields.
For example, to watch when the d_inode pointer inside a dentry is
modified (not just when the dentry itself is accessed):
echo 'w:watch rw@0:8 address=$addr value=$value' >> dynamic_events
echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
echo 'set_wprobe:watch:(dentry)dentry->d_inode' \
>> events/fprobes/truncate/trigger
Here, "dentry" is a field in the fprobe event record. The BTF
lookup resolves offsetof(struct dentry, d_inode) at set_wprobe parse
time, so no manual numeric offset is needed.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v10:
- Newly added.
---
kernel/trace/trace_wprobe.c | 166 +++++++++++++++++---
.../test.d/trigger/trigger-wprobe-btf-offset.tc | 73 +++++++++
2 files changed, 214 insertions(+), 25 deletions(-)
create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-offset.tc
diff --git a/kernel/trace/trace_wprobe.c b/kernel/trace/trace_wprobe.c
index 1edfdddf241d..2926eabfe3c1 100644
--- a/kernel/trace/trace_wprobe.c
+++ b/kernel/trace/trace_wprobe.c
@@ -27,6 +27,7 @@
#include <asm/ptrace.h>
#include "trace.h"
+#include "trace_btf.h"
#include "trace_dynevent.h"
#include "trace_probe.h"
#include "trace_probe_kernel.h"
@@ -925,6 +926,143 @@ static void wprobe_trigger_free(struct event_trigger_data *data)
}
}
+#ifdef CONFIG_PROBE_EVENTS_BTF_ARGS
+
+static int get_offset_of_field(struct btf *btf, const struct btf_type *type, char *field_name)
+{
+ const struct btf_member *field;
+ int bitoffs = 0;
+ u32 anon_offs;
+ char *next;
+
+ do {
+ next = strchr(field_name, '.');
+ if (next)
+ *next++ = '\0';
+
+ field = btf_find_struct_member(btf, type, field_name, &anon_offs);
+ if (IS_ERR_OR_NULL(field))
+ return -ENOENT;
+ type = btf_type_by_id(btf, field->type);
+
+ /* Reject bitfield member access */
+ if (btf_type_kflag(type))
+ return -EINVAL;
+
+ /* add offset for anonymous struct type */
+ bitoffs += anon_offs + field->offset;
+ field_name = next;
+ if (next) {
+ type = btf_type_skip_modifiers(btf, field->type, NULL);
+ if (!type)
+ return -ENOENT;
+ }
+ } while (next);
+ return bitoffs / BITS_PER_BYTE;
+}
+
+/* btf_put(NULL) is acceptable. */
+DEFINE_FREE(btf_put, struct btf *, btf_put(_T))
+
+/* parse typecast: (TYPE[,ASGN])EVENT_FIELD->FIELD[.SUBFIELD...] and set adjust. */
+static int wprobe_trigger_typecast_parse(char **field_str_ptr,
+ struct wprobe_trigger_data *wprobe_data)
+{
+ struct btf *btf __free(btf_put) = NULL;
+ const struct btf_type *type;
+ char *assign_field;
+ char *event_field;
+ char *type_field;
+ char *type_name;
+ int id;
+ int adjust;
+
+ type_name = *field_str_ptr + 1;
+ event_field = strchr(type_name, ')');
+ if (!event_field)
+ return -EINVAL;
+ *event_field++ = '\0';
+
+ /* Check the optional assign field. */
+ assign_field = strchr(type_name, ',');
+ if (assign_field)
+ *assign_field++ = '\0';
+
+ /* Get the type field name. */
+ type_field = strstr(event_field, "->");
+ if (!type_field)
+ return -EINVAL;
+ *type_field = '\0';
+ type_field += 2;
+
+ /* find type from BTF */
+ id = bpf_find_btf_id(type_name, BTF_KIND_STRUCT, &btf);
+ if (id < 0)
+ return id;
+
+ type = btf_type_by_id(btf, id);
+ if (!type)
+ return -EINVAL;
+
+ adjust = get_offset_of_field(btf, type, type_field);
+ if (adjust < 0)
+ return adjust;
+ wprobe_data->adjust = adjust;
+
+ if (assign_field) {
+ /* assign_field should be a struct field */
+ adjust = get_offset_of_field(btf, type, assign_field);
+ if (adjust < 0)
+ return adjust;
+ wprobe_data->adjust -= adjust;
+ }
+
+ *field_str_ptr = event_field;
+ return 0;
+}
+#else
+static int wprobe_trigger_typecast_parse(char **field_str_ptr, struct wprobe_trigger_data *wprobe_data)
+{
+ return -EOPNOTSUPP;
+}
+#endif /* CONFIG_PROBE_EVENTS_BTF_ARGS */
+
+static int wprobe_trigger_field_parse(char *field_str, struct trace_event_file *file,
+ struct wprobe_trigger_data *wprobe_data)
+{
+ struct ftrace_event_field *field;
+ char *offs;
+
+ if (field_str[0] == '(') {
+ int ret = wprobe_trigger_typecast_parse(&field_str, wprobe_data);
+
+ if (ret < 0)
+ return ret;
+ } else {
+ offs = strpbrk(field_str, "+-");
+ if (offs) {
+ long val;
+
+ if (kstrtol(offs, 0, &val) < 0)
+ return -EINVAL;
+ wprobe_data->adjust = val;
+ *offs = '\0';
+ }
+ }
+
+ field = trace_find_event_field(file->event_call, field_str);
+ if (!field)
+ return -ENOENT;
+ if (field->size != sizeof(void *))
+ return -ENOEXEC;
+ wprobe_data->offset = field->offset;
+ wprobe_data->field = kstrdup(field_str, GFP_KERNEL);
+ if (!wprobe_data->field)
+ return -ENOMEM;
+
+ return 0;
+}
+
static int wprobe_trigger_cmd_parse(struct event_command *cmd_ops,
struct trace_event_file *file,
char *glob, char *cmd,
@@ -936,10 +1074,8 @@ static int wprobe_trigger_cmd_parse(struct event_command *cmd_ops,
*/
struct wprobe_trigger_data *wprobe_data __free(free_wprobe_trigger_data) = NULL;
struct event_trigger_data *trigger_data __free(kfree) = NULL;
- struct ftrace_event_field *field = NULL;
struct trace_event_file *wprobe_file;
struct trace_array *tr = file->tr;
- struct trace_event_call *event;
char *event_str, *field_str;
bool remove, clear = false;
struct trace_wprobe *tw;
@@ -986,29 +1122,9 @@ static int wprobe_trigger_cmd_parse(struct event_command *cmd_ops,
return -EINVAL;
if (field_str) {
- char *offs;
-
- offs = strpbrk(field_str, "+-");
- if (offs) {
- long val;
-
- if (kstrtol(offs, 0, &val) < 0)
- return -EINVAL;
- wprobe_data->adjust = val;
- *offs = '\0';
- }
-
- event = file->event_call;
- field = trace_find_event_field(event, field_str);
- if (!field)
- return -ENOENT;
-
- if (field->size != sizeof(void *))
- return -ENOEXEC;
- wprobe_data->offset = field->offset;
- wprobe_data->field = kstrdup(field_str, GFP_KERNEL);
- if (!wprobe_data->field)
- return -ENOMEM;
+ ret = wprobe_trigger_field_parse(field_str, file, wprobe_data);
+ if (ret < 0)
+ return ret;
}
trigger_data = trigger_data_alloc(cmd_ops, cmd, param, wprobe_data);
diff --git a/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-offset.tc b/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-offset.tc
new file mode 100644
index 000000000000..b79427caf974
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-offset.tc
@@ -0,0 +1,73 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event trigger - test set_wprobe trigger with BTF struct offset
+# requires: dynamic_events "w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]":README events/sched/sched_process_fork/trigger "[(structname[,field])]<argname>[->field[->field|.field...]]":README
+
+echo 0 >> tracing_on
+
+rm -f $TMPDIR/hoge
+
+# we will skip this test if fprobe is not supported.
+if ! grep -Fq "f[:[<group>/][<event>]] <func-name>[%return] [<args>]" README; then
+ echo "UNRESOLVED: fprobe is not supported"
+ exit_unresolved
+fi
+
+# we will skip this test if the target function does not exist.
+if ! grep -wq "do_truncate" /proc/kallsyms; then
+ echo "UNRESOLVED: do_truncate not found"
+ exit_unresolved
+fi
+if ! grep -wq "dentry_kill" /proc/kallsyms; then
+ echo "UNRESOLVED: dentry_kill not found"
+ exit_unresolved
+fi
+
+:;: "Add a wprobe event watching 8 bytes" ;:
+echo 'w:watch rw@0:8 address=$addr value=$value' >> dynamic_events
+
+:;: "Add fprobe events for do_truncate and dentry_kill" ;:
+echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
+echo 'f:dentry_kill dentry_kill dentry=$arg1' >> dynamic_events
+
+:;: "Add set_wprobe trigger using BTF struct offset resolution" ;:
+# Syntax: set_wprobe:WPEVENT:(STRUCT)EVENT_FIELD->MEMBER
+# (dentry) is the BTF struct type (without 'struct' keyword, matching fetcharg syntax)
+# dentry is the event record field name holding the pointer
+# d_inode is the struct member whose offset is resolved automatically via BTF
+# This sets the watchpoint at the resolved d_inode pointer.
+echo 'set_wprobe:watch:(dentry)dentry->d_inode' >> events/fprobes/truncate/trigger
+echo 'clear_wprobe:watch:dentry' >> events/fprobes/dentry_kill/trigger
+
+:;: "Verify triggers are set" ;:
+cat events/fprobes/truncate/trigger | grep ^set_wprobe
+cat events/fprobes/dentry_kill/trigger | grep ^clear_wprobe
+
+:;: "Ensure wprobe is still disabled" ;:
+cat events/wprobes/watch/enable | grep 0
+
+:;: "Enable fprobe events" ;:
+echo 1 >> events/fprobes/truncate/enable
+echo 1 >> events/fprobes/dentry_kill/enable
+
+:;: "Start test workload" ;:
+echo 1 >> tracing_on
+
+echo aaa > $TMPDIR/hoge
+echo bbb > $TMPDIR/hoge
+echo ccc > $TMPDIR/hoge
+rm $TMPDIR/hoge
+
+:;: "Drop dentry caches (for dentry_kill)" ;:
+sync && echo 2 >> /proc/sys/vm/drop_caches
+
+:;: "Check trace results" ;:
+cat trace | grep watch
+
+:;: "Remove wprobe triggers" ;:
+echo '!set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
+echo '!clear_wprobe:watch' >> events/fprobes/dentry_kill/trigger
+! grep ^set_wprobe events/fprobes/truncate/trigger
+! grep ^clear_wprobe events/fprobes/dentry_kill/trigger
+
+exit 0
^ permalink raw reply related
* [PATCH v4 00/16] Fix bugs on HugeTLB folio allocation failure paths
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
This series fixes a few bugs on HugeTLB folio allocation failure paths.
Some of these issues were pointed out by Sashiko while I was working on
[1], and others were revealed while fixing the original issues.
1. subpool usage leak on allocation failure (Sashiko pointed this out
in [2])
When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
hugetlb cgroup charging failure) and gbl_chg == 1, the error path skips
restoring the page to the subpool, leaking the subpool's used_hpages
counter. Fix this by calling hugepage_subpool_put_pages()
unconditionally if map_chg is true.
2. Folio refcount mismatch on memcg charge failure leading to
VM_BUG_ON_FOLIO() (Sashiko pointed this out in [3])
The error path in alloc_hugetlb_folio() calls free_huge_folio() directly
on a folio with a refcount of 1 (set via folio_ref_unfreeze() earlier).
This triggers VM_BUG_ON_FOLIO(folio_ref_count(folio), folio) if
CONFIG_DEBUG_VM is enabled, and can corrupt allocator state otherwise.
Fix this by using folio_put() instead of free_huge_folio() to correctly
drop the refcount before freeing.
3. Returning -ENOMEM on memcg charge failure causes infinite loop
alloc_hugetlb_folio() propagates -ENOMEM on charge failure, which maps
to VM_FAULT_OOM. Because HugeTLB physical allocations are high-order and
use __GFP_RETRY_MAYFAIL, the OOM killer is bypassed. Returning
VM_FAULT_OOM leaks to the #PF handler, which cannot make progress and
retries the faulting instruction indefinitely. Fix this by returning
-ENOSPC instead of -ENOMEM on charge failure, which maps to
VM_FAULT_SIGBUS, terminating the process cleanly.
4. vma reservation leak on mem_cgroup_charge_hugetlb() failure
When mem_cgroup_charge_hugetlb() fails, the error path historically
bypassed vma_end_reservation(). Since the reservation had already been
committed via vma_commit_reservation(), this left the reservation map in
an inconsistent state, leaking resv_huge_pages when the process
exited. Fix this by moving mem_cgroup_charge_hugetlb() earlier in
alloc_hugetlb_folio(), before vma_commit_reservation() is called.
Cleaned up patches, updated documentation, and fixed Sashiko's comments on
v3:
1. subpool cleanup in alloc_hugetlb_folio() was incomplete - I folded the
fix in.
2. More issues (UAF, and another accounting issue) reported [5] - I added
two more patches.
Keeping this section since I think people didn't have much time to review
v3.
1. subpool page tracking
The subpool now tracks every page allocated via
the subpool through used_hpages, regardless of whether min_hpages or
max_hpages is requested by the user.
When min_hpages is set, rsv_hpages is initialized to be equal to
min_hpages. As reserved pages are used, the count is transferred from
rsv_hpages to used_hpages. When reserved pages run out, as long as
max_hpages is not requested or not yet met, used_hpages continues to be
incremented. The following invariant holds true while used_hpages <=
min_hpages:
used_hpages + rsv_hpages = min_hpages
when used_hpages exceeds min_hpages, rsv_hpages must always be 0.
With this, pages can and must always be returned to the subpool if there's
some reservation or allocation failure after getting a page from the
subpool. One thing I missed in v2 was in processing the returned page. In
v2, the number of pages returned was compared to min_hpages, ignoring
used_hpages if it exists. v3 limits the number of pages returned to
min_hpages - used_hpages, upholding the invariant above.
2. (Re-)introduction of the try-commit-cancel protocol for memcg charging.
In v2, I took the shortcut where I moved mem_cgroup_charge_hugetlb() up
above committing vma reservations, and jumped to subpool cleanup when the
charging failed. With that, Sashiko pointed out that there are still issues
with subpool tracking.
In [4] I thought the issue was that gbl_chg == 0 (I flipped the condition
in [4], omg) doesn't always mean a reservation was actually used, but after
digging more, that doesn't seem to be true, since if gbl_chg == 0,
restore_reserve is set on the folio, and free_huge_folio() would have done
a h->resv_huge_pages++.
In my opinion, calling free_huge_folio() in the middle of the cleanup does
too much, and I think the cleaner approach would be to do all the charging
symmetrically, so that the subpool cleanup will also be symmetric
and (hopefully) easier to understand.
+ I would like reviews on all the non-WIP patches. Those are the ones that
I'd like merged first.
+ There are reproducers (not meant to be merged) to help prove
effectiveness of fixes.
+ There are WIP patches for subpool-related refactoring. Those helped me
check that the subpool stuff is working and is posted to show how it
works. I think the subpool-related refactoring and tests would be good to
merge, but perhaps in another patch series? The refactoring was heavily
AI-driven and should be considered RFC. Would like to have your feedback
on that refactoring too.
+ libhugetlbfs tests pass
+ ./tools/testing/selftests/mm/ksft_hugetlb.sh passes
+ Reproducers all pass
v1: https://lore.kernel.org/r/20260707-hugetlb-alloc-failure-fixes-v1-0-5bbd3a4b836d@google.com
v2: https://lore.kernel.org/r/20260708-hugetlb-alloc-failure-fixes-v2-0-c7f27cbb462b@google.com
v3: https://lore.kernel.org/r/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee@google.com
The series [1] has some changes that are dependent on these fixes, so I'll wait
for your reviews before continuing on [1].
Thank you!
[1] https://lore.kernel.org/all/20260702-hugetlb-open-up-v4-0-d53cefcccf34@google.com/T/
[2] https://sashiko.dev/#/patchset/20260518-hugetlb-open-up-v3-0-e14b302477f8%40google.com?part=5
[3] https://sashiko.dev/#/patchset/20260702-hugetlb-open-up-v4-0-d53cefcccf34%40google.com?part=6
[4] https://lore.kernel.org/all/CAEvNRgGN0HSJ2iLSDD2haSKOxifa-uhkO9Hwossh0+Q_d9fzOw@mail.gmail.com/
[5] https://sashiko.dev/#/patchset/20260720-hugetlb-alloc-failure-fixes-v3-0-7d2a169aa9ee%40google.com?part=8
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Ackerley Tng (16):
mm: hugetlb: Track used_hpages when getting/putting pages from subpool
mm: hugetlb: Return -ENOSPC on memcg charge failure
mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios
mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function
mm: hugetlb: Fix subpool usage leak on allocation failure
mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages()
mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool()
fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super()
WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c
WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API
WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c
WIP: tools: testing: Add unit tests for HugeTLB subpool functions
WIP: Reproducer for allocation failure due to cgroup v2 memory limits
WIP: Reproducer for subpool usage leak
WIP: Reproducer for false restoration on shared HugeTLB mappings
WIP: Reproducer for out_put_pages subpool reserve leakage
Documentation/mm/hugetlbfs_reserv.rst | 17 +-
.../translations/zh_CN/mm/hugetlbfs_reserv.rst | 11 +-
cgroup_v2_allocation_failure.c | 169 +++++++++
fs/hugetlbfs/inode.c | 31 +-
hugetlb_reserve_pages_out_put_pages.c | 49 +++
hugetlb_reserve_pages_out_put_pages.sh | 153 ++++++++
include/linux/hugetlb.h | 17 +-
include/linux/memcontrol.h | 30 +-
mm/Makefile | 2 +-
mm/hugetlb.c | 260 +++-----------
mm/hugetlb_subpool.c | 230 ++++++++++++
mm/hugetlb_subpool.h | 21 ++
mm/memcontrol.c | 116 +++++-
subpool_leak_max_size.sh | 72 ++++
subpool_shared_leak.c | 43 +++
subpool_shared_leak.sh | 87 +++++
tools/testing/hugetlb_subpool/.gitignore | 1 +
tools/testing/hugetlb_subpool/Makefile | 18 +
tools/testing/hugetlb_subpool/test_subpool.c | 400 +++++++++++++++++++++
19 files changed, 1445 insertions(+), 282 deletions(-)
---
base-commit: 4539944e515183668109bdf4d0c3d7d228383d88
change-id: 20260706-hugetlb-alloc-failure-fixes-c7f775eca29f
Best regards,
--
Ackerley Tng <ackerleytng@google.com>
^ permalink raw reply
* [PATCH v4 02/16] mm: hugetlb: Return -ENOSPC on memcg charge failure
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
When mem_cgroup_charge_hugetlb() fails with -ENOMEM, alloc_hugetlb_folio()
currently propagates this error. This results in the page fault handler
returning VM_FAULT_OOM.
Because HugeTLB allocations are high-order and use __GFP_RETRY_MAYFAIL,
they bypass the OOM killer. Returning VM_FAULT_OOM to the #PF handler
without triggering the OOM killer (or having it make progress) leads to
an infinite loop of retrying the fault.
Avoid this loop by returning -ENOSPC when charging fails, which maps to
VM_FAULT_SIGBUS, terminating the process cleanly.
Make mem_cgroup_charge_hugetlb() fault handling use a common error handling
path, the same handling used for hugetlb_cgroup_uncharge_cgroup{,_rsvd}(),
which also don't trigger the OOM killer and hence opt to terminate the
process with a SIGBUS.
Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Tested-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/hugetlb.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 36fa3fb3945d8..15f9c5a9f75e2 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3010,7 +3010,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
if (ret == -ENOMEM) {
free_huge_folio(folio);
- return ERR_PTR(-ENOMEM);
+ goto err;
}
return folio;
@@ -3035,6 +3035,16 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
out_end_reservation:
if (map_chg != MAP_CHG_ENFORCED)
vma_end_reservation(h, vma, addr);
+err:
+ /*
+ * Return -ENOSPC when this function fails to allocate or charge a huge
+ * page. If a standard (PAGE_SIZE) page allocation fails, the OOM killer
+ * is given a chance to run, which may resolve the failure on
+ * retry. However, for HugeTLB allocations, the OOM killer is not
+ * triggered. Returning -ENOMEM (or anything resulting in VM_FAULT_OOM)
+ * would leak to the #PF handler, causing it to loop indefinitely
+ * retrying the fault.
+ */
return ERR_PTR(-ENOSPC);
}
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 01/16] mm: hugetlb: Track used_hpages when getting/putting pages from subpool
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
hugepage_subpool_put_pages() currently has two distinct responsibilities
that conflict:
1. When size is specified for the mount, max_hpages != -1: Keep track of
total active pages (allocated + reserved) and decrement this count
(used_hpages) when a page is freed or allocation fails.
2. When min_size is specified for the mount, min_hpages != -1: Ensure we
don't drop below the guaranteed minimum, and restore a reservation
(rsv_hpages) if we do.
This causes trouble because when allocation fails (refer to
alloc_hugetlb_folio()) if gbl_chg = 1 (i.e. no subpool reservation was
taken):
+ To keep used_hpages consistent, HugeTLB needs to call
hugepage_subpool_put_pages() to restore undo used_hpages being
incremented
+ But can't call hugepage_subpool_put_pages() if no reservation was
consumed.
One option would be to conditionally do subpool tracking updates outside of
the hugepage_subpool_put_pages() function, but that would spread logic all
over.
Instead, always track used_hpages, regardless of whether a max_size was
requested for the mount, so that the subpool always knows how many pages
were allocated through it. Every page allocated through the subpool
increments used_hpages, regardless of whether a reservation was taken from
it.
Conceptually, now, every allocation involving a subpool uses a page from
the subpool, which must be returned to the subpool. Every page taken from
the subpool tries to use a subpool reservation. Restoring a page to the
subpool reservations only if the page was taken from subpool
reservations. (If used_hpages >= min_hpages, the page must have not have
been taken from the reservations.)
Always tracking used_hpages provides the subpool with information of both
used and reserved counts to make the correct decision for both max_size and
min_size correctly.
With used_hpages always tracked,
+ subpool_is_free() can be simplified, such that the subpool can be
declared free if there are no more pages in use.
+ open-coding in hugetlb_reserve_pages() can be removed.
Also update the
+ Documentation for used_hpages in the subpool struct, since it no longer
matters whether the used pages count against the maximum.
+ Docstring for hugepage_subpool_{get,put}_pages
+ Documentation to use active voice, and remove some details in favor of
having details documented in the docstring
Also update statfs reporting. Previously, if max_hpages is negative,
used_hpages is static at 0, so returning max_hpages - used_hpages returns
-1 and is always correct. Now, if the subpool doesn't have a maximum
requested size, indicate no limit for free pages (-1). If it does have a
maximum size, report the difference between the requested size and the
number of used pages. This difference is always positive, because if the
mount does have a maximum size, hugepage_subpool_get_pages() ensures that
the subpool usage never exceeds the maximum.
This fixes a bug in hugetlb_unreserve_pages(), where pages are returned to
the subpool regardless of whether it consumed a reservation. The
corresponding bug in the failure handling path of alloc_hugetlb_folio() was
fixed in a833a693a490e.
Fixes: 1c5ecae3a93fa ("hugetlbfs: add minimum size accounting to subpools")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
Documentation/mm/hugetlbfs_reserv.rst | 17 +--
.../translations/zh_CN/mm/hugetlbfs_reserv.rst | 11 +-
fs/hugetlbfs/inode.c | 8 +-
include/linux/hugetlb.h | 4 +-
mm/hugetlb.c | 118 +++++++++++----------
5 files changed, 75 insertions(+), 83 deletions(-)
diff --git a/Documentation/mm/hugetlbfs_reserv.rst b/Documentation/mm/hugetlbfs_reserv.rst
index a49115db18c76..d244583fdcbc3 100644
--- a/Documentation/mm/hugetlbfs_reserv.rst
+++ b/Documentation/mm/hugetlbfs_reserv.rst
@@ -314,21 +314,8 @@ huge pages. If they can not be reserved, the mount fails.
The routines hugepage_subpool_get/put_pages() are called when pages are
obtained from or released back to a subpool. They perform all subpool
accounting, and track any reservations associated with the subpool.
-hugepage_subpool_get/put_pages are passed the number of huge pages by which
-to adjust the subpool 'used page' count (down for get, up for put). Normally,
-they return the same value that was passed or an error if not enough pages
-exist in the subpool.
-
-However, if reserves are associated with the subpool a return value less
-than the passed value may be returned. This return value indicates the
-number of additional global pool adjustments which must be made. For example,
-suppose a subpool contains 3 reserved huge pages and someone asks for 5.
-The 3 reserved pages associated with the subpool can be used to satisfy part
-of the request. But, 2 pages must be obtained from the global pools. To
-relay this information to the caller, the value 2 is returned. The caller
-is then responsible for attempting to obtain the additional two pages from
-the global pools.
-
+hugepage_subpool_get/put_pages() use the number of huge pages passed to adjust
+the subpool 'used page' count.
COW and Reservations
====================
diff --git a/Documentation/translations/zh_CN/mm/hugetlbfs_reserv.rst b/Documentation/translations/zh_CN/mm/hugetlbfs_reserv.rst
index 20947f8bd0654..ae1f1f31477fc 100644
--- a/Documentation/translations/zh_CN/mm/hugetlbfs_reserv.rst
+++ b/Documentation/translations/zh_CN/mm/hugetlbfs_reserv.rst
@@ -246,15 +246,8 @@ hugepage_subpool的min_hpages字段中被跟踪。在挂载时，hugetlb_acct_me
被调用以预留指定数量的巨页。如果它们不能被预留，挂载就会失败。
当从子池中获取或释放页面时，会调用hugepage_subpool_get/put_pages()函数。
-hugepage_subpool_get/put_pages被传递给巨页数量，以此来调整子池的 “已用页面” 计数
-（get为下降，put为上升）。通常情况下，如果子池中没有足够的页面，它们会返回与传递的相同的值或
-一个错误。
-
-然而，如果预留与子池相关联，可能会返回一个小于传递值的返回值。这个返回值表示必须进行的额外全局
-池调整的数量。例如，假设一个子池包含3个预留的巨页，有人要求5个。与子池相关的3个预留页可以用来
-满足部分请求。但是，必须从全局池中获得2个页面。为了向调用者转达这一信息，将返回值2。然后，调用
-者要负责从全局池中获取另外两个页面。
-
+它们负责所有子池的统计核算，并跟踪与子池相关联的预留。
+hugepage_subpool_get/put_pages()函数使用传入的巨页数量来调整子池的“已用页面”计数。
COW和预留
==========
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 216e1a0dd0b23..26c0187340636 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1109,8 +1109,12 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
spin_lock_irq(&sbinfo->spool->lock);
buf->f_blocks = sbinfo->spool->max_hpages;
- free_pages = sbinfo->spool->max_hpages
- - sbinfo->spool->used_hpages;
+ if (sbinfo->spool->max_hpages == -1) {
+ free_pages = -1;
+ } else {
+ free_pages = sbinfo->spool->max_hpages -
+ sbinfo->spool->used_hpages;
+ }
buf->f_bavail = buf->f_bfree = free_pages;
spin_unlock_irq(&sbinfo->spool->lock);
buf->f_files = sbinfo->max_inodes;
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 2abaf99321e90..34b9a3e1be0fa 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -38,8 +38,8 @@ struct hugepage_subpool {
spinlock_t lock;
long count;
long max_hpages; /* Maximum huge pages or -1 if no maximum. */
- long used_hpages; /* Used count against maximum, includes */
- /* both allocated and reserved pages. */
+ long used_hpages; /* Used page count, includes both */
+ /* allocated and reserved pages. */
struct hstate *hstate;
long min_hpages; /* Minimum huge pages or -1 if no minimum. */
long rsv_hpages; /* Pages reserved against global pool to */
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 571212b80835e..36fa3fb3945d8 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -129,12 +129,8 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool)
{
if (spool->count)
return false;
- if (spool->max_hpages != -1)
- return spool->used_hpages == 0;
- if (spool->min_hpages != -1)
- return spool->rsv_hpages == spool->min_hpages;
- return true;
+ return spool->used_hpages == 0;
}
static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
@@ -187,13 +183,18 @@ void hugepage_put_subpool(struct hugepage_subpool *spool)
unlock_or_release_subpool(spool, flags);
}
-/*
- * Subpool accounting for allocating and reserving pages.
- * Return -ENOMEM if there are not enough resources to satisfy the
- * request. Otherwise, return the number of pages by which the
- * global pools must be adjusted (upward). The returned value may
- * only be different than the passed value (delta) in the case where
- * a subpool minimum size must be maintained.
+/**
+ * hugepage_subpool_get_pages - Get pages from a subpool
+ * @spool: pointer to subpool structure (may be NULL)
+ * @delta: number of pages to allocate or reserve
+ *
+ * Check and update subpool page usage counts when allocating or
+ * reserving @delta hugepages.
+ *
+ * Context: Takes spool->lock using spin_lock_irq().
+ * Return: Non-negative number of reservations that cannot be
+ * satisfied by the subpool, or -ENOMEM if the subpool maximum
+ * limit would be exceeded.
*/
static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
long delta)
@@ -205,15 +206,14 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
spin_lock_irq(&spool->lock);
- if (spool->max_hpages != -1) { /* maximum size accounting */
- if ((spool->used_hpages + delta) <= spool->max_hpages)
- spool->used_hpages += delta;
- else {
- ret = -ENOMEM;
- goto unlock_ret;
- }
+ if (spool->max_hpages != -1 &&
+ spool->used_hpages + delta > spool->max_hpages) {
+ ret = -ENOMEM;
+ goto unlock_ret;
}
+ spool->used_hpages += delta;
+
/* minimum size accounting */
if (spool->min_hpages != -1 && spool->rsv_hpages) {
if (delta > spool->rsv_hpages) {
@@ -234,11 +234,19 @@ static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
return ret;
}
-/*
- * Subpool accounting for freeing and unreserving pages.
- * Return the number of global page reservations that must be dropped.
- * The return value may only be different than the passed value (delta)
- * in the case where a subpool minimum size must be maintained.
+/**
+ * hugepage_subpool_put_pages - Release pages back to a subpool
+ * @spool: pointer to subpool structure (may be NULL)
+ * @delta: number of pages to free or unreserve
+ *
+ * Check and update subpool page usage counts when freeing or
+ * unreserving @delta hugepages.
+ *
+ * Context: Takes spool->lock using spin_lock_irqsave(). May release
+ * and free @spool if its usage count and references reach
+ * zero.
+ * Return: Non-negative number of reservations that the subpool cannot
+ * absorb.
*/
static long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
long delta)
@@ -251,19 +259,24 @@ static long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
spin_lock_irqsave(&spool->lock, flags);
- if (spool->max_hpages != -1) /* maximum size accounting */
- spool->used_hpages -= delta;
+ spool->used_hpages -= delta;
/* minimum size accounting */
if (spool->min_hpages != -1 && spool->used_hpages < spool->min_hpages) {
- if (spool->rsv_hpages + delta <= spool->min_hpages)
+ /*
+ * limit is the maximum number of reservations that
+ * can be restored to this subpool.
+ */
+ long limit = spool->min_hpages - spool->used_hpages;
+
+ if (spool->rsv_hpages + delta <= limit)
ret = 0;
else
- ret = spool->rsv_hpages + delta - spool->min_hpages;
+ ret = spool->rsv_hpages + delta - limit;
spool->rsv_hpages += delta;
- if (spool->rsv_hpages > spool->min_hpages)
- spool->rsv_hpages = spool->min_hpages;
+ if (spool->rsv_hpages > limit)
+ spool->rsv_hpages = limit;
}
/*
@@ -6542,7 +6555,7 @@ long hugetlb_reserve_pages(struct inode *inode,
struct vm_area_struct *vma,
vma_flags_t vma_flags)
{
- long chg = -1, add = -1, spool_resv, gbl_resv;
+ long chg = -1, add = -1, gbl_resv;
struct hstate *h = hstate_inode(inode);
struct hugepage_subpool *spool = subpool_inode(inode);
struct resv_map *resv_map;
@@ -6622,9 +6635,9 @@ long hugetlb_reserve_pages(struct inode *inode,
* the subpool has a minimum size, there may be some global
* reservations already in place (gbl_reserve).
*/
- gbl_reserve = hugepage_subpool_get_pages(spool, chg);
- if (gbl_reserve < 0) {
- err = gbl_reserve;
+ gbl_resv = hugepage_subpool_get_pages(spool, chg);
+ if (gbl_resv < 0) {
+ err = gbl_resv;
goto out_uncharge_cgroup;
}
@@ -6632,7 +6645,7 @@ long hugetlb_reserve_pages(struct inode *inode,
* Check enough hugepages are available for the reservation.
* Hand the pages back to the subpool if there are not
*/
- err = hugetlb_acct_memory(h, gbl_reserve);
+ err = hugetlb_acct_memory(h, gbl_resv);
if (err < 0)
goto out_put_pages;
@@ -6651,7 +6664,7 @@ long hugetlb_reserve_pages(struct inode *inode,
add = region_add(resv_map, from, to, regions_needed, h, h_cg);
if (unlikely(add < 0)) {
- hugetlb_acct_memory(h, -gbl_reserve);
+ hugetlb_acct_memory(h, -gbl_resv);
err = add;
goto out_put_pages;
} else if (unlikely(chg > add)) {
@@ -6687,26 +6700,21 @@ long hugetlb_reserve_pages(struct inode *inode,
}
return chg;
-out_put_pages:
- spool_resv = chg - gbl_reserve;
- if (spool_resv) {
- /* put sub pool's reservation back, chg - gbl_reserve */
- gbl_resv = hugepage_subpool_put_pages(spool, spool_resv);
- /*
- * subpool's reserved pages can not be put back due to race,
- * return to hstate.
- */
- hugetlb_acct_memory(h, -gbl_resv);
- }
- /* Restore used_hpages for pages that failed global reservation */
- if (gbl_reserve && spool) {
- unsigned long flags;
+ out_put_pages:
+ /*
+ * Return all that was requested from the subpool, let subpool
+ * tell us the new number of reservations that need to be
+ * returned to the global pool.
+ */
+ gbl_reserve = hugepage_subpool_put_pages(spool, chg);
+ /*
+ * There may be a difference between the number of
+ * reservations to consume and the number to restore now if
+ * there are multiple threads interacting with the subpool -
+ * restore the difference.
+ */
+ hugetlb_acct_memory(h, gbl_resv - gbl_reserve);
- spin_lock_irqsave(&spool->lock, flags);
- if (spool->max_hpages != -1)
- spool->used_hpages -= gbl_reserve;
- unlock_or_release_subpool(spool, flags);
- }
out_uncharge_cgroup:
hugetlb_cgroup_uncharge_cgroup_rsvd(hstate_index(h),
chg * pages_per_huge_page(h), h_cg);
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 03/16] mm: hugetlb: Use try-commit-cancel protocol for memcg charge of folios
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
Using mem_cgroup_charge_hugetlb() to charge a HugeTLB folio during page
fault creates a reservation leak bug if the task hits its memory cgroup
limit.
When alloc_hugetlb_folio() commits the VMA reservation, the reserved
page is removed from the reserve map. If a subsequent call to
mem_cgroup_charge_hugetlb() returns -ENOMEM, the allocation is aborted
and the physical folio is disposed of via free_huge_page(). However,
because the VMA reservation was already consumed, the reservation count
in the reserve map is lost. This causes subsequent faults in the VMA
address range to fail with premature reservation exhaustion.
Additionally, dropping the use of free_huge_folio() on the failure path
fixes an issue where free_huge_folio() was incorrectly invoked on a
folio with a refcount of 1, triggering refcount mismatches and kernel
warnings.
To fix this, introduce a try-commit-cancel protocol for memory cgroup
charging of HugeTLB folios, matching the architecture used by the
hugetlb cgroup controller. Invoking mem_cgroup_hugetlb_try_charge()
before consuming the VMA reservation ensures that if the memory cgroup
limit is reached, the allocation is aborted cleanly without leaking
the reservation entry or having to dispose of a partially initialized
folio.
An alternative would be to retain the current usage of
mem_cgroup_charge_hugetlb() and free_huge_page(), but freeing the folio
performs reservation management for subpools and global hstate, which
complicates rollback in alloc_hugetlb_folio(). Using a try-commit-cancel
protocol is more consistent with the other charging performed in
alloc_hugetlb_folio() and easier to understand.
Fixes: 991135774c0e0 ("memcg/hugetlb: introduce mem_cgroup_charge_hugetlb")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
include/linux/memcontrol.h | 30 ++++++++++++
mm/hugetlb.c | 30 ++++++------
mm/memcontrol.c | 114 +++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 159 insertions(+), 15 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index e1f46a0016fcf..2c3e2c62f4176 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -642,6 +642,15 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm,
}
int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
+int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
+ struct mem_cgroup **memcg_p,
+ struct obj_cgroup **objcg_p);
+void mem_cgroup_hugetlb_commit_charge(struct folio *folio,
+ struct mem_cgroup *memcg,
+ struct obj_cgroup *objcg);
+void mem_cgroup_hugetlb_cancel_charge(unsigned int nr_pages,
+ struct mem_cgroup *memcg,
+ struct obj_cgroup *objcg);
int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
struct mm_struct *mm, gfp_t gfp);
@@ -1133,6 +1142,27 @@ static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp)
return 0;
}
+static inline int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
+ struct mem_cgroup **memcg_p,
+ struct obj_cgroup **objcg_p)
+{
+ *memcg_p = NULL;
+ *objcg_p = NULL;
+ return 0;
+}
+
+static inline void mem_cgroup_hugetlb_commit_charge(struct folio *folio,
+ struct mem_cgroup *memcg,
+ struct obj_cgroup *objcg)
+{
+}
+
+static inline void mem_cgroup_hugetlb_cancel_charge(unsigned int nr_pages,
+ struct mem_cgroup *memcg,
+ struct obj_cgroup *objcg)
+{
+}
+
static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
unsigned short id, struct mm_struct *mm, gfp_t gfp)
{
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 15f9c5a9f75e2..5ee1bc5c00bfe 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -38,6 +38,7 @@
#include <linux/mm_inline.h>
#include <linux/padata.h>
#include <linux/pgalloc.h>
+#include <linux/memcontrol.h>
#include <asm/page.h>
#include <asm/tlb.h>
@@ -2876,6 +2877,8 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
int ret, idx;
struct hugetlb_cgroup *h_cg = NULL;
struct hugetlb_cgroup *h_cg_rsvd = NULL;
+ struct mem_cgroup *mem_cg = NULL;
+ struct obj_cgroup *obj_cg = NULL;
gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
idx = hstate_index(h);
@@ -2935,6 +2938,11 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
if (ret)
goto out_uncharge_cgroup_reservation;
+ ret = mem_cgroup_hugetlb_try_charge(pages_per_huge_page(h), gfp,
+ &mem_cg, &obj_cg);
+ if (ret)
+ goto out_uncharge_cgroup;
+
spin_lock_irq(&hugetlb_lock);
/*
* glb_chg is passed to indicate whether or not a page must be taken
@@ -2946,7 +2954,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
spin_unlock_irq(&hugetlb_lock);
folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, addr);
if (!folio)
- goto out_uncharge_cgroup;
+ goto out_uncharge_cgroup_memcg;
spin_lock_irq(&hugetlb_lock);
list_add(&folio->lru, &h->hugepage_activelist);
folio_ref_unfreeze(folio, 1);
@@ -2973,6 +2981,9 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
spin_unlock_irq(&hugetlb_lock);
+ mem_cgroup_hugetlb_commit_charge(folio, mem_cg, obj_cg);
+ lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
+
hugetlb_set_folio_subpool(folio, spool);
if (map_chg != MAP_CHG_ENFORCED) {
@@ -3000,21 +3011,10 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
}
}
- ret = mem_cgroup_charge_hugetlb(folio, gfp);
- /*
- * Unconditionally increment NR_HUGETLB here. If it turns out that
- * mem_cgroup_charge_hugetlb failed, then immediately free the page and
- * decrement NR_HUGETLB.
- */
- lruvec_stat_mod_folio(folio, NR_HUGETLB, pages_per_huge_page(h));
-
- if (ret == -ENOMEM) {
- free_huge_folio(folio);
- goto err;
- }
-
return folio;
+out_uncharge_cgroup_memcg:
+ mem_cgroup_hugetlb_cancel_charge(pages_per_huge_page(h), mem_cg, obj_cg);
out_uncharge_cgroup:
hugetlb_cgroup_uncharge_cgroup(idx, pages_per_huge_page(h), h_cg);
out_uncharge_cgroup_reservation:
@@ -3035,7 +3035,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
out_end_reservation:
if (map_chg != MAP_CHG_ENFORCED)
vma_end_reservation(h, vma, addr);
-err:
+
/*
* Return -ENOSPC when this function fails to allocate or charge a huge
* page. If a standard (PAGE_SIZE) page allocation fails, the OOM killer
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6dc4888a90f3f..0beee5c0ce93b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5180,6 +5180,120 @@ int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
return ret;
}
+/**
+ * mem_cgroup_hugetlb_try_charge - Try to charge the memcg for a hugetlb folio
+ * @nr_pages: number of base pages to charge
+ * @gfp: reclaim mode
+ * @memcg_p: Output pointer to the charged mem_cgroup (if successful and enabled)
+ * @objcg_p: Output pointer to the charged obj_cgroup (if successful and enabled)
+ *
+ * Prepares and tries to reserve the memory counter for the folio from the current
+ * task's memcg. If successful, both *memcg_p and *objcg_p are populated and their
+ * references are pinned until a subsequent call to mem_cgroup_hugetlb_commit_charge
+ * or mem_cgroup_hugetlb_cancel_charge.
+ *
+ * Returns ENOMEM if the memcg is already full.
+ * Returns 0 if either the charge was successful, or if we skip charging.
+ */
+int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
+ struct mem_cgroup **memcg_p,
+ struct obj_cgroup **objcg_p)
+{
+ struct mem_cgroup *memcg;
+ struct obj_cgroup *objcg;
+ int ret = 0;
+
+ *memcg_p = NULL;
+ *objcg_p = NULL;
+
+ if (mem_cgroup_disabled() || !memcg_accounts_hugetlb() ||
+ !cgroup_subsys_on_dfl(memory_cgrp_subsys))
+ return 0;
+
+ memcg = get_mem_cgroup_from_current();
+ if (!memcg)
+ return 0;
+
+ objcg = get_obj_cgroup_from_memcg(memcg);
+ if (!objcg)
+ goto put_memcg;
+
+ if (!obj_cgroup_is_root(objcg)) {
+ ret = try_charge_memcg(memcg, gfp, nr_pages);
+ if (ret)
+ goto put_objcg;
+ }
+
+ *memcg_p = memcg;
+ *objcg_p = objcg;
+ return 0;
+
+put_objcg:
+ obj_cgroup_put(objcg);
+put_memcg:
+ mem_cgroup_put(memcg);
+ return ret;
+}
+
+/**
+ * mem_cgroup_hugetlb_commit_charge - Commit the memcg charge for a hugetlb folio
+ * @folio: folio being charged
+ * @memcg: Target mem_cgroup obtained from mem_cgroup_hugetlb_try_charge
+ * @objcg: Target obj_cgroup obtained from mem_cgroup_hugetlb_try_charge
+ *
+ * Finalizes the memory and statistics charging for the folio in the specified memcg.
+ * Transfers the pinned objcg reference to the folio structure (for automatic
+ * uncharging upon freeing via mem_cgroup_uncharge). Releases the try-commit reference
+ * on memcg.
+ */
+void mem_cgroup_hugetlb_commit_charge(struct folio *folio,
+ struct mem_cgroup *memcg,
+ struct obj_cgroup *objcg)
+{
+ if (!memcg || !objcg)
+ return;
+
+ commit_charge(folio, objcg);
+ memcg1_commit_charge(folio, memcg);
+
+ /*
+ * Drop our try-commit-cancel protocol reference on memcg.
+ * The objcg reference is TRANSFERRED to the folio by commit_charge,
+ * so it will be put automatically by __mem_cgroup_uncharge() when
+ * the folio is freed.
+ */
+ mem_cgroup_put(memcg);
+}
+
+/**
+ * mem_cgroup_hugetlb_cancel_charge - Cancel and undo a hugetlb folio memcg charge
+ * @nr_pages: number of base pages to uncharge
+ * @memcg: Target mem_cgroup obtained from mem_cgroup_hugetlb_try_charge
+ * @objcg: Target obj_cgroup obtained from mem_cgroup_hugetlb_try_charge
+ *
+ * Cancels and safely rolls back the prepared memory charge for the folio in the
+ * specified memcg. Releases the try-commit pinned references on both memcg and objcg.
+ */
+void mem_cgroup_hugetlb_cancel_charge(unsigned int nr_pages,
+ struct mem_cgroup *memcg,
+ struct obj_cgroup *objcg)
+{
+ if (!memcg || !objcg)
+ return;
+
+ if (!obj_cgroup_is_root(objcg))
+ refill_stock(memcg, nr_pages);
+
+ /*
+ * Drop our try-commit-cancel protocol references on both objcg
+ * and memcg, since this mapping attempt was aborted and the folio
+ * was never committed.
+ */
+ obj_cgroup_put(objcg);
+ mem_cgroup_put(memcg);
+}
+
+
/**
* mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
* @folio: the folio to charge
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 04/16] mm: hugetlb: Remove unused mem_cgroup_charge_hugetlb function
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
Now that the alloc_hugetlb_folio path has been successfully migrated to
the new try-commit-cancel memcg charging protocol, the old
mem_cgroup_charge_hugetlb function and its associated header and
static inline declarations are completely unused. Remove them to clean
up the memory controller's codebase.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
include/linux/memcontrol.h | 6 ------
mm/memcontrol.c | 34 ----------------------------------
2 files changed, 40 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 2c3e2c62f4176..3bdf35dc2d88f 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -641,7 +641,6 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm,
return __mem_cgroup_charge(folio, mm, gfp);
}
-int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
int mem_cgroup_hugetlb_try_charge(unsigned int nr_pages, gfp_t gfp,
struct mem_cgroup **memcg_p,
struct obj_cgroup **objcg_p);
@@ -1133,11 +1132,6 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target,
static inline int mem_cgroup_charge(struct folio *folio,
struct mm_struct *mm, gfp_t gfp)
-{
- return 0;
-}
-
-static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp)
{
return 0;
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 0beee5c0ce93b..6764ff041c196 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5146,40 +5146,6 @@ int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp)
return ret;
}
-/**
- * mem_cgroup_charge_hugetlb - charge the memcg for a hugetlb folio
- * @folio: folio being charged
- * @gfp: reclaim mode
- *
- * This function is called when allocating a huge page folio, after the page has
- * already been obtained and charged to the appropriate hugetlb cgroup
- * controller (if it is enabled).
- *
- * Returns ENOMEM if the memcg is already full.
- * Returns 0 if either the charge was successful, or if we skip the charging.
- */
-int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
-{
- struct mem_cgroup *memcg = get_mem_cgroup_from_current();
- int ret = 0;
-
- /*
- * Even memcg does not account for hugetlb, we still want to update
- * system-level stats via lruvec_stat_mod_folio. Return 0, and skip
- * charging the memcg.
- */
- if (mem_cgroup_disabled() || !memcg_accounts_hugetlb() ||
- !memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys))
- goto out;
-
- if (charge_memcg(folio, memcg, gfp))
- ret = -ENOMEM;
-
-out:
- mem_cgroup_put(memcg);
- return ret;
-}
-
/**
* mem_cgroup_hugetlb_try_charge - Try to charge the memcg for a hugetlb folio
* @nr_pages: number of base pages to charge
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 05/16] mm: hugetlb: Fix subpool usage leak on allocation failure
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
When alloc_hugetlb_folio() fails early (e.g. buddy allocation failure or
hugetlb cgroup charging failure) and gbl_chg == 1 (meaning a reservation
was not used, but a global page was allocated instead), the subpool page
acquired via hugepage_subpool_get_pages() must still be returned.
Currently, the error path out_subpool_put: only calls
hugepage_subpool_put_pages() if !gbl_chg is true. If gbl_chg is 1, it
skips it, permanently leaking the subpool's used_hpages counter.
With the earlier patch to always track used_hpages in the subpool, always
call hugepage_subpool_put_pages() if map_chg is true to consistently
restore the page to the subpool. Condition on map_chg because map_chg also
gates hugepage_subpool_get_pages().
Opportunistically rename gbl_chg to gbl_resv_get, which is the number of
global pages needed (because neither resv_map nor subpool reservations
could be used).
If the number of global pages needed is 0, this allocation uses a
reservation somewhere, hence proceed to consume a reservation by
decrementing h->resv_huge_pages. If the number of global pages needed is 1,
reservations are neither created nor consumed.
Also rename gbl_reserve to gbl_resv_put, which is the number of
pages the subpool could not absorb into its reservations.
Adjust global reservations using hugetlb_acct_memory() with the difference
between gbl_resv_get and gbl_resv_put to take care of possible races where
another thread might have performed get or put with the same subpool, hence
perhaps requiring updates to global reservation counts. (If gbl_resv_get ==
0 because a resv_map reservation was used, map_chg == 0 so the entire
subpool returning is skipped - still correct.)
Fixes: a833a693a490e ("mm: hugetlb: fix incorrect fallback for subpool")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
mm/hugetlb.c | 23 ++++++++++-------------
1 file changed, 10 insertions(+), 13 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 5ee1bc5c00bfe..879e4640dc50d 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2872,7 +2872,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
struct hugepage_subpool *spool = subpool_vma(vma);
struct hstate *h = hstate_vma(vma);
struct folio *folio;
- long retval, gbl_chg, gbl_reserve;
+ long retval, gbl_resv_get;
map_chg_state map_chg;
int ret, idx;
struct hugetlb_cgroup *h_cg = NULL;
@@ -2912,15 +2912,15 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
* Or if it can get one from the pool reservation directly.
*/
if (map_chg) {
- gbl_chg = hugepage_subpool_get_pages(spool, 1);
- if (gbl_chg < 0)
+ gbl_resv_get = hugepage_subpool_get_pages(spool, 1);
+ if (gbl_resv_get < 0)
goto out_end_reservation;
} else {
/*
* If we have the vma reservation ready, no need for extra
* global reservation.
*/
- gbl_chg = 0;
+ gbl_resv_get = 0;
}
/*
@@ -2949,7 +2949,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
* from the global free pool (global change). gbl_chg == 0 indicates
* a reservation exists for the allocation.
*/
- folio = dequeue_hugetlb_folio_vma(h, vma, addr, gbl_chg);
+ folio = dequeue_hugetlb_folio_vma(h, vma, addr, gbl_resv_get);
if (!folio) {
spin_unlock_irq(&hugetlb_lock);
folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, addr);
@@ -2965,7 +2965,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
* Either dequeued or buddy-allocated folio needs to add special
* mark to the folio when it consumes a global reservation.
*/
- if (!gbl_chg) {
+ if (!gbl_resv_get) {
folio_set_hugetlb_restore_reserve(folio);
h->resv_huge_pages--;
}
@@ -3022,13 +3022,10 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
h_cg_rsvd);
out_subpool_put:
- /*
- * put page to subpool iff the quota of subpool's rsv_hpages is used
- * during hugepage_subpool_get_pages.
- */
- if (map_chg && !gbl_chg) {
- gbl_reserve = hugepage_subpool_put_pages(spool, 1);
- hugetlb_acct_memory(h, -gbl_reserve);
+ if (map_chg) {
+ long gbl_resv_put = hugepage_subpool_put_pages(spool, 1);
+
+ hugetlb_acct_memory(h, gbl_resv_get - gbl_resv_put);
}
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 06/16] mm: hugetlb: Rename local variables for clarity in hugetlb_reserve_pages()
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
In hugetlb_reserve_pages(), the return value of
hugepage_subpool_get_pages() was named gbl_resv, while the return value of
hugepage_subpool_put_pages() was named gbl_reserve. These variable names
were inconsistent with the naming conventions established in
alloc_hugetlb_folio().
Rename local variables in hugetlb_reserve_pages() to match naming in
alloc_hugetlb_folio().
+ Rename gbl_resv to gbl_resv_get, which is the number of global
reservations required from the global pool (because neither reservation
maps nor pre-existing subpool reservations could cover the request).
If gbl_resv_get is 0, the reservation request is fully covered by
pre-existing subpool reservations, so no additional global reservations
are requested. If gbl_resv_get > 0, it represents the number of
additional global reservations required, which are charged to global
memory via hugetlb_acct_memory(h, gbl_resv_get).
Note: gbl_resv_get in hugetlb_reserve_pages() is conceptually slightly
different from gbl_resv_get in alloc_hugetlb_folio():
+ In hugetlb_reserve_pages(): additional _reservations_ required,
accounted using hugetlb_acct_memory()
+ In alloc_hugetlb_folio(): additional _pages_ required, no change to
h->resv_huge_pages if gbl_resv_get > 0
+ Rename gbl_reserve to gbl_resv_put, which is the number of pages the
subpool could not absorb into its reservations during rollback (and are
thus returned to the global pool).
In error rollback paths, hugetlb_acct_memory(h, gbl_resv_get -
gbl_resv_put) adjusts global reservations using the difference between
global pages requested (gbl_resv_get) and global pages returned
(gbl_resv_put).
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
mm/hugetlb.c | 19 ++++++++++---------
1 file changed, 10 insertions(+), 9 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 879e4640dc50d..8431c00d48267 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -6562,12 +6562,13 @@ long hugetlb_reserve_pages(struct inode *inode,
struct vm_area_struct *vma,
vma_flags_t vma_flags)
{
- long chg = -1, add = -1, gbl_resv;
+ long chg = -1, add = -1;
struct hstate *h = hstate_inode(inode);
struct hugepage_subpool *spool = subpool_inode(inode);
struct resv_map *resv_map;
struct hugetlb_cgroup *h_cg = NULL;
- long gbl_reserve, regions_needed = 0;
+ long gbl_resv_get, gbl_resv_put;
+ long regions_needed = 0;
int err;
/* This should never happen */
@@ -6642,9 +6643,9 @@ long hugetlb_reserve_pages(struct inode *inode,
* the subpool has a minimum size, there may be some global
* reservations already in place (gbl_reserve).
*/
- gbl_resv = hugepage_subpool_get_pages(spool, chg);
- if (gbl_resv < 0) {
- err = gbl_resv;
+ gbl_resv_get = hugepage_subpool_get_pages(spool, chg);
+ if (gbl_resv_get < 0) {
+ err = gbl_resv_get;
goto out_uncharge_cgroup;
}
@@ -6652,7 +6653,7 @@ long hugetlb_reserve_pages(struct inode *inode,
* Check enough hugepages are available for the reservation.
* Hand the pages back to the subpool if there are not
*/
- err = hugetlb_acct_memory(h, gbl_resv);
+ err = hugetlb_acct_memory(h, gbl_resv_get);
if (err < 0)
goto out_put_pages;
@@ -6671,7 +6672,7 @@ long hugetlb_reserve_pages(struct inode *inode,
add = region_add(resv_map, from, to, regions_needed, h, h_cg);
if (unlikely(add < 0)) {
- hugetlb_acct_memory(h, -gbl_resv);
+ hugetlb_acct_memory(h, -gbl_resv_get);
err = add;
goto out_put_pages;
} else if (unlikely(chg > add)) {
@@ -6713,14 +6714,14 @@ long hugetlb_reserve_pages(struct inode *inode,
* tell us the new number of reservations that need to be
* returned to the global pool.
*/
- gbl_reserve = hugepage_subpool_put_pages(spool, chg);
+ gbl_resv_put = hugepage_subpool_put_pages(spool, chg);
/*
* There may be a difference between the number of
* reservations to consume and the number to restore now if
* there are multiple threads interacting with the subpool -
* restore the difference.
*/
- hugetlb_acct_memory(h, gbl_resv - gbl_reserve);
+ hugetlb_acct_memory(h, gbl_resv_get - gbl_resv_put);
out_uncharge_cgroup:
hugetlb_cgroup_uncharge_cgroup_rsvd(hstate_index(h),
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 07/16] mm: hugetlb: Fix Use-After-Free in unlock_or_release_subpool()
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
In unlock_or_release_subpool(), spool->lock was released before calling
subpool_is_free(spool). Because subpool_is_free() accesses spool->count and
spool->used_hpages locklessly, concurrent threads calling
hugepage_subpool_put_pages() or hugepage_put_subpool() could race when
spool->count is 0.
If Thread 1 drops spool->lock and is preempted before subpool_is_free(),
Thread 2 can acquire spool->lock, decrement spool->used_hpages to 0,
evaluate subpool_is_free() as true, and free spool via kfree(). When Thread
1 resumes, subpool_is_free() dereferences the freed spool pointer, causing
a Use-After-Free (UAF) or double free.
Fix this race by evaluating subpool_is_free() while holding spool->lock
prior to unlocking.
Fixes: 1d88433bb0085 ("mm/hugetlb: fix use after free when subpool max_hpages accounting is not enabled")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
mm/hugetlb.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 8431c00d48267..b759748468734 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -137,12 +137,11 @@ static inline bool subpool_is_free(struct hugepage_subpool *spool)
static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
unsigned long irq_flags)
{
+ bool is_free = subpool_is_free(spool);
+
spin_unlock_irqrestore(&spool->lock, irq_flags);
- /* If no pages are used, and no other handles to the subpool
- * remain, give up any reservations based on minimum size and
- * free the subpool */
- if (subpool_is_free(spool)) {
+ if (is_free) {
if (spool->min_hpages != -1)
hugetlb_acct_memory(spool->hstate,
-spool->min_hpages);
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 11/16] WIP: mm: hugetlb: Make struct hugepage_subpool private to hugetlb_subpool.c
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
Now that all filesystem and memory management subpool functions across
fs/hugetlbfs/inode.c and mm/ use getters, transition struct
hugepage_subpool out of the public header include/linux/hugetlb.h and
encapsulate it privately inside mm/hugetlb_subpool.c.
Replace the header definition with a forward declaration.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
include/linux/hugetlb.h | 13 ++-----------
mm/hugetlb_subpool.c | 12 ++++++++++++
2 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index f36be371c6e88..074a45903973e 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -18,6 +18,7 @@
struct mmu_gather;
struct node;
+struct hstate;
void free_huge_folio(struct folio *folio);
@@ -34,17 +35,7 @@ void free_huge_folio(struct folio *folio);
*/
#define __NR_USED_SUBPAGE 3
-struct hugepage_subpool {
- spinlock_t lock;
- long count;
- long max_hpages; /* Maximum huge pages or -1 if no maximum. */
- long used_hpages; /* Used page count, includes both */
- /* allocated and reserved pages. */
- struct hstate *hstate;
- long min_hpages; /* Minimum huge pages or -1 if no minimum. */
- long rsv_hpages; /* Pages reserved against global pool to */
- /* satisfy minimum size. */
-};
+struct hugepage_subpool;
struct resv_map {
struct kref refs;
diff --git a/mm/hugetlb_subpool.c b/mm/hugetlb_subpool.c
index ac0f9057b4921..39413515a2722 100644
--- a/mm/hugetlb_subpool.c
+++ b/mm/hugetlb_subpool.c
@@ -14,6 +14,18 @@
#include <linux/bug.h>
#include "hugetlb_subpool.h"
+struct hugepage_subpool {
+ spinlock_t lock;
+ long count;
+ long max_hpages; /* Maximum huge pages or -1 if no maximum. */
+ long used_hpages; /* Used page count, includes both */
+ /* allocated and reserved pages. */
+ struct hstate *hstate;
+ long min_hpages; /* Minimum huge pages or -1 if no minimum. */
+ long rsv_hpages; /* Pages reserved against global pool to */
+ /* satisfy minimum size. */
+};
+
static inline bool subpool_is_free(struct hugepage_subpool *spool)
{
if (spool->count)
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 08/16] fs: hugetlbfs: Fix global reservation leak in hugetlbfs_fill_super()
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng, stable
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
In hugetlbfs_fill_super(), if hugepage_new_subpool() succeeds in
allocating a subpool (which reserves global huge pages when
min_hpages != -1), but a subsequent initialization step like
d_make_root() fails, the error path directly invoked
kfree(sbinfo->spool).
Directly freeing the subpool structure with kfree() bypasses
hugepage_put_subpool() and its underlying unlock_or_release_subpool()
destructor. Consequently, hugetlb_acct_memory() is never called to
release the global page reservations allocated for min_hpages,
permanently leaking global huge page reservations.
Fix this leak by calling hugepage_put_subpool(sbinfo->spool) on the
error cleanup path instead of kfree(sbinfo->spool).
sbinfo->spool must be checked before dereferencing the subpool in
hugepage_put_subpool(). Add a NULL guard to hugepage_put_subpool() instead
of checking it in the caller to align it with other subpool helpers like
hugepage_subpool_get_pages() and hugepage_subpool_put_pages() that
gracefully handle NULL subpools. This allows callers to safely invoke
hugepage_put_subpool() without requiring explicit NULL checks. This is also
aligned with how kfree() can be called on NULL.
With the NULL guard in hugepage_put_subpool(), the NULL check in the only
other caller can also be removed.
Fixes: 7ca02d0ae586f ("hugetlbfs: accept subpool min_size mount option and setup accordingly")
Cc: stable@vger.kernel.org
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
fs/hugetlbfs/inode.c | 6 ++----
mm/hugetlb.c | 3 +++
2 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 26c0187340636..e5d86f31eba5b 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1133,9 +1133,7 @@ static void hugetlbfs_put_super(struct super_block *sb)
if (sbi) {
sb->s_fs_info = NULL;
- if (sbi->spool)
- hugepage_put_subpool(sbi->spool);
-
+ hugepage_put_subpool(sbi->spool);
kfree(sbi);
}
}
@@ -1423,7 +1421,7 @@ hugetlbfs_fill_super(struct super_block *sb, struct fs_context *fc)
goto out_free;
return 0;
out_free:
- kfree(sbinfo->spool);
+ hugepage_put_subpool(sbinfo->spool);
kfree(sbinfo);
return -ENOMEM;
}
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index b759748468734..90ec015a11181 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -177,6 +177,9 @@ void hugepage_put_subpool(struct hugepage_subpool *spool)
{
unsigned long flags;
+ if (!spool)
+ return;
+
spin_lock_irqsave(&spool->lock, flags);
BUG_ON(!spool->count);
spool->count--;
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 09/16] WIP: mm: hugetlb: Move subpool functions to hugetlb_subpool.c
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
Move all HugeTLB subpool lifecycle, page reservation, and accounting
routines out of mm/hugetlb.c and into their own dedicated, encapsulated
translation unit at mm/hugetlb_subpool.c.
Also introduces the internal mm/hugetlb_subpool.h header for holding the
subpool-local APIs, allowing fs/hugetlbfs and mm/ to access the subpool
functions cleanly. The subpool internal layout structures remain in
include/linux/hugetlb.h until getters are introduced.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
fs/hugetlbfs/inode.c | 1 +
include/linux/hugetlb.h | 4 +-
mm/Makefile | 2 +-
mm/hugetlb.c | 166 +-------------------------------------------
mm/hugetlb_subpool.c | 181 ++++++++++++++++++++++++++++++++++++++++++++++++
mm/hugetlb_subpool.h | 17 +++++
6 files changed, 202 insertions(+), 169 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index e5d86f31eba5b..86c21f8272470 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -25,6 +25,7 @@
#include <linux/ctype.h>
#include <linux/backing-dev.h>
#include <linux/hugetlb.h>
+#include "../../mm/hugetlb_subpool.h"
#include <linux/folio_batch.h>
#include <linux/fs_parser.h>
#include <linux/mman.h>
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 34b9a3e1be0fa..f36be371c6e88 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -114,9 +114,7 @@ extern int hugetlb_max_hstate __read_mostly;
#define for_each_hstate(h) \
for ((h) = hstates; (h) < &hstates[hugetlb_max_hstate]; (h)++)
-struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
- long min_hpages);
-void hugepage_put_subpool(struct hugepage_subpool *spool);
+int hugetlb_acct_memory(struct hstate *h, long delta);
void hugetlb_dup_vma_private(struct vm_area_struct *vma);
void clear_vma_resv_huge_pages(struct vm_area_struct *vma);
diff --git a/mm/Makefile b/mm/Makefile
index eff9f9e7e061c..3965c959e5099 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -78,7 +78,7 @@ endif
obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o
obj-$(CONFIG_ZSWAP) += zswap.o
obj-$(CONFIG_HAS_DMA) += dmapool.o
-obj-$(CONFIG_HUGETLBFS) += hugetlb.o hugetlb_sysfs.o hugetlb_sysctl.o
+obj-$(CONFIG_HUGETLBFS) += hugetlb.o hugetlb_subpool.o hugetlb_sysfs.o hugetlb_sysctl.o
ifdef CONFIG_CMA
obj-$(CONFIG_HUGETLBFS) += hugetlb_cma.o
endif
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 90ec015a11181..4d44a9720a971 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -51,6 +51,7 @@
#include "hugetlb_vmemmap.h"
#include "hugetlb_cma.h"
#include "hugetlb_internal.h"
+#include "hugetlb_subpool.h"
#include <linux/page-isolation.h>
int hugetlb_max_hstate __read_mostly;
@@ -126,171 +127,6 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
unsigned long start, unsigned long end, bool take_locks);
static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
-static inline bool subpool_is_free(struct hugepage_subpool *spool)
-{
- if (spool->count)
- return false;
-
- return spool->used_hpages == 0;
-}
-
-static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
- unsigned long irq_flags)
-{
- bool is_free = subpool_is_free(spool);
-
- spin_unlock_irqrestore(&spool->lock, irq_flags);
-
- if (is_free) {
- if (spool->min_hpages != -1)
- hugetlb_acct_memory(spool->hstate,
- -spool->min_hpages);
- kfree(spool);
- }
-}
-
-struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
- long min_hpages)
-{
- struct hugepage_subpool *spool;
-
- spool = kzalloc_obj(*spool);
- if (!spool)
- return NULL;
-
- spin_lock_init(&spool->lock);
- spool->count = 1;
- spool->max_hpages = max_hpages;
- spool->hstate = h;
- spool->min_hpages = min_hpages;
-
- if (min_hpages != -1 && hugetlb_acct_memory(h, min_hpages)) {
- kfree(spool);
- return NULL;
- }
- spool->rsv_hpages = min_hpages;
-
- return spool;
-}
-
-void hugepage_put_subpool(struct hugepage_subpool *spool)
-{
- unsigned long flags;
-
- if (!spool)
- return;
-
- spin_lock_irqsave(&spool->lock, flags);
- BUG_ON(!spool->count);
- spool->count--;
- unlock_or_release_subpool(spool, flags);
-}
-
-/**
- * hugepage_subpool_get_pages - Get pages from a subpool
- * @spool: pointer to subpool structure (may be NULL)
- * @delta: number of pages to allocate or reserve
- *
- * Check and update subpool page usage counts when allocating or
- * reserving @delta hugepages.
- *
- * Context: Takes spool->lock using spin_lock_irq().
- * Return: Non-negative number of reservations that cannot be
- * satisfied by the subpool, or -ENOMEM if the subpool maximum
- * limit would be exceeded.
- */
-static long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
- long delta)
-{
- long ret = delta;
-
- if (!spool)
- return ret;
-
- spin_lock_irq(&spool->lock);
-
- if (spool->max_hpages != -1 &&
- spool->used_hpages + delta > spool->max_hpages) {
- ret = -ENOMEM;
- goto unlock_ret;
- }
-
- spool->used_hpages += delta;
-
- /* minimum size accounting */
- if (spool->min_hpages != -1 && spool->rsv_hpages) {
- if (delta > spool->rsv_hpages) {
- /*
- * Asking for more reserves than those already taken on
- * behalf of subpool. Return difference.
- */
- ret = delta - spool->rsv_hpages;
- spool->rsv_hpages = 0;
- } else {
- ret = 0; /* reserves already accounted for */
- spool->rsv_hpages -= delta;
- }
- }
-
-unlock_ret:
- spin_unlock_irq(&spool->lock);
- return ret;
-}
-
-/**
- * hugepage_subpool_put_pages - Release pages back to a subpool
- * @spool: pointer to subpool structure (may be NULL)
- * @delta: number of pages to free or unreserve
- *
- * Check and update subpool page usage counts when freeing or
- * unreserving @delta hugepages.
- *
- * Context: Takes spool->lock using spin_lock_irqsave(). May release
- * and free @spool if its usage count and references reach
- * zero.
- * Return: Non-negative number of reservations that the subpool cannot
- * absorb.
- */
-static long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
- long delta)
-{
- long ret = delta;
- unsigned long flags;
-
- if (!spool)
- return delta;
-
- spin_lock_irqsave(&spool->lock, flags);
-
- spool->used_hpages -= delta;
-
- /* minimum size accounting */
- if (spool->min_hpages != -1 && spool->used_hpages < spool->min_hpages) {
- /*
- * limit is the maximum number of reservations that
- * can be restored to this subpool.
- */
- long limit = spool->min_hpages - spool->used_hpages;
-
- if (spool->rsv_hpages + delta <= limit)
- ret = 0;
- else
- ret = spool->rsv_hpages + delta - limit;
-
- spool->rsv_hpages += delta;
- if (spool->rsv_hpages > limit)
- spool->rsv_hpages = limit;
- }
-
- /*
- * If hugetlbfs_put_super couldn't free spool due to an outstanding
- * quota reference, free it now.
- */
- unlock_or_release_subpool(spool, flags);
-
- return ret;
-}
-
static inline struct hugepage_subpool *subpool_vma(struct vm_area_struct *vma)
{
return subpool_inode(file_inode(vma->vm_file));
diff --git a/mm/hugetlb_subpool.c b/mm/hugetlb_subpool.c
new file mode 100644
index 0000000000000..6184860ed7374
--- /dev/null
+++ b/mm/hugetlb_subpool.c
@@ -0,0 +1,181 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Subpool and reserve accounting for HugeTLB folios.
+ * Extracted from mm/hugetlb.c
+ */
+
+#include <linux/types.h>
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#ifdef __KERNEL__
+#include <linux/hugetlb.h>
+#endif
+#include <linux/spinlock.h>
+#include <linux/bug.h>
+
+#include "hugetlb_subpool.h"
+
+static inline bool subpool_is_free(struct hugepage_subpool *spool)
+{
+ if (spool->count)
+ return false;
+
+ return spool->used_hpages == 0;
+}
+
+static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
+ unsigned long irq_flags)
+{
+ bool is_free = subpool_is_free(spool);
+
+ spin_unlock_irqrestore(&spool->lock, irq_flags);
+
+ if (is_free) {
+ if (spool->min_hpages != -1)
+ hugetlb_acct_memory(spool->hstate,
+ -spool->min_hpages);
+ kfree(spool);
+ }
+}
+
+struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
+ long min_hpages)
+{
+ struct hugepage_subpool *spool;
+
+ spool = kzalloc_obj(*spool);
+ if (!spool)
+ return NULL;
+
+ spin_lock_init(&spool->lock);
+ spool->count = 1;
+ spool->max_hpages = max_hpages;
+ spool->hstate = h;
+ spool->min_hpages = min_hpages;
+
+ if (min_hpages != -1 && hugetlb_acct_memory(h, min_hpages)) {
+ kfree(spool);
+ return NULL;
+ }
+ spool->rsv_hpages = min_hpages;
+
+ return spool;
+}
+
+void hugepage_put_subpool(struct hugepage_subpool *spool)
+{
+ unsigned long flags;
+
+ if (!spool)
+ return;
+
+ spin_lock_irqsave(&spool->lock, flags);
+ BUG_ON(!spool->count);
+ spool->count--;
+ unlock_or_release_subpool(spool, flags);
+}
+
+/**
+ * hugepage_subpool_get_pages - Get pages from a subpool
+ * @spool: pointer to subpool structure (may be NULL)
+ * @delta: number of pages to allocate or reserve
+ *
+ * Check and update subpool page usage counts when allocating or
+ * reserving @delta hugepages.
+ *
+ * Context: Takes spool->lock using spin_lock_irq().
+ * Return: Non-negative number of reservations that cannot be
+ * satisfied by the subpool, or -ENOMEM if the subpool maximum
+ * limit would be exceeded.
+ */
+long hugepage_subpool_get_pages(struct hugepage_subpool *spool,
+ long delta)
+{
+ long ret = delta;
+
+ if (!spool)
+ return ret;
+
+ spin_lock_irq(&spool->lock);
+
+ if (spool->max_hpages != -1 &&
+ spool->used_hpages + delta > spool->max_hpages) {
+ ret = -ENOMEM;
+ goto unlock_ret;
+ }
+
+ spool->used_hpages += delta;
+
+ /* minimum size accounting */
+ if (spool->min_hpages != -1 && spool->rsv_hpages) {
+ if (delta > spool->rsv_hpages) {
+ /*
+ * Asking for more reserves than those already taken on
+ * behalf of subpool. Return difference.
+ */
+ ret = delta - spool->rsv_hpages;
+ spool->rsv_hpages = 0;
+ } else {
+ ret = 0; /* reserves already accounted for */
+ spool->rsv_hpages -= delta;
+ }
+ }
+
+unlock_ret:
+ spin_unlock_irq(&spool->lock);
+ return ret;
+}
+
+/**
+ * hugepage_subpool_put_pages - Release pages back to a subpool
+ * @spool: pointer to subpool structure (may be NULL)
+ * @delta: number of pages to free or unreserve
+ *
+ * Check and update subpool page usage counts when freeing or
+ * unreserving @delta hugepages.
+ *
+ * Context: Takes spool->lock using spin_lock_irqsave(). May release
+ * and free @spool if its usage count and references reach
+ * zero.
+ * Return: Non-negative number of reservations that the subpool cannot
+ * absorb.
+ */
+long hugepage_subpool_put_pages(struct hugepage_subpool *spool,
+ long delta)
+{
+ long ret = delta;
+ unsigned long flags;
+
+ if (!spool)
+ return delta;
+
+ spin_lock_irqsave(&spool->lock, flags);
+
+ spool->used_hpages -= delta;
+
+ /* minimum size accounting */
+ if (spool->min_hpages != -1 && spool->used_hpages < spool->min_hpages) {
+ /*
+ * limit is the maximum number of reservations that
+ * can be restored to this subpool.
+ */
+ long limit = spool->min_hpages - spool->used_hpages;
+
+ if (spool->rsv_hpages + delta <= limit)
+ ret = 0;
+ else
+ ret = spool->rsv_hpages + delta - limit;
+
+ spool->rsv_hpages += delta;
+ if (spool->rsv_hpages > limit)
+ spool->rsv_hpages = limit;
+ }
+
+ /*
+ * If hugetlbfs_put_super couldn't free spool due to an outstanding
+ * quota reference, free it now.
+ */
+ unlock_or_release_subpool(spool, flags);
+
+ return ret;
+}
diff --git a/mm/hugetlb_subpool.h b/mm/hugetlb_subpool.h
new file mode 100644
index 0000000000000..be1f1cf012c9c
--- /dev/null
+++ b/mm/hugetlb_subpool.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _MM_HUGETLB_SUBPOOL_H
+#define _MM_HUGETLB_SUBPOOL_H
+
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+struct hstate;
+struct hugepage_subpool;
+
+struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
+ long min_hpages);
+void hugepage_put_subpool(struct hugepage_subpool *spool);
+long hugepage_subpool_get_pages(struct hugepage_subpool *spool, long delta);
+long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta);
+
+#endif /* _MM_HUGETLB_SUBPOOL_H */
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 10/16] WIP: fs: hugetlbfs: Refactor subpool getters and integrate with hugetlb_subpool API
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
Refactor the direct field accesses to spool->max_hpages, spool->min_hpages,
and calculate subpool properties using getters inside
mm/hugetlb_subpool.c.
This will allow the definition of struct hugepage_subpool to be
encapsulated and private to mm/hugetlb_subpool.c
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
fs/hugetlbfs/inode.c | 28 +++++++++-------------------
mm/hugetlb_subpool.c | 39 ++++++++++++++++++++++++++++++++++++++-
mm/hugetlb_subpool.h | 4 ++++
3 files changed, 51 insertions(+), 20 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 86c21f8272470..b424afdedb3ee 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -1060,7 +1060,6 @@ static int hugetlbfs_show_options(struct seq_file *m, struct dentry *root)
struct hugetlbfs_sb_info *sbinfo = HUGETLBFS_SB(root->d_sb);
struct hugepage_subpool *spool = sbinfo->spool;
unsigned long hpage_size = huge_page_size(sbinfo->hstate);
- unsigned hpage_shift = huge_page_shift(sbinfo->hstate);
char mod;
if (!uid_eq(sbinfo->uid, GLOBAL_ROOT_UID))
@@ -1082,12 +1081,13 @@ static int hugetlbfs_show_options(struct seq_file *m, struct dentry *root)
}
seq_printf(m, ",pagesize=%lu%c", hpage_size, mod);
if (spool) {
- if (spool->max_hpages != -1)
- seq_printf(m, ",size=%llu",
- (unsigned long long)spool->max_hpages << hpage_shift);
- if (spool->min_hpages != -1)
- seq_printf(m, ",min_size=%llu",
- (unsigned long long)spool->min_hpages << hpage_shift);
+ unsigned long long max_size = hugepage_subpool_max_size(spool);
+ unsigned long long min_size = hugepage_subpool_min_size(spool);
+
+ if (max_size != -1ULL)
+ seq_printf(m, ",size=%llu", max_size);
+ if (min_size != -1ULL)
+ seq_printf(m, ",min_size=%llu", min_size);
}
return 0;
}
@@ -1106,18 +1106,8 @@ static int hugetlbfs_statfs(struct dentry *dentry, struct kstatfs *buf)
/* If no limits set, just report 0 or -1 for max/free/used
* blocks, like simple_statfs() */
if (sbinfo->spool) {
- long free_pages;
-
- spin_lock_irq(&sbinfo->spool->lock);
- buf->f_blocks = sbinfo->spool->max_hpages;
- if (sbinfo->spool->max_hpages == -1) {
- free_pages = -1;
- } else {
- free_pages = sbinfo->spool->max_hpages -
- sbinfo->spool->used_hpages;
- }
- buf->f_bavail = buf->f_bfree = free_pages;
- spin_unlock_irq(&sbinfo->spool->lock);
+ buf->f_blocks = hugepage_subpool_max_hpages(sbinfo->spool);
+ buf->f_bavail = buf->f_bfree = hugepage_subpool_free_hpages(sbinfo->spool);
buf->f_files = sbinfo->max_inodes;
buf->f_ffree = sbinfo->free_inodes;
}
diff --git a/mm/hugetlb_subpool.c b/mm/hugetlb_subpool.c
index 6184860ed7374..ac0f9057b4921 100644
--- a/mm/hugetlb_subpool.c
+++ b/mm/hugetlb_subpool.c
@@ -12,7 +12,6 @@
#endif
#include <linux/spinlock.h>
#include <linux/bug.h>
-
#include "hugetlb_subpool.h"
static inline bool subpool_is_free(struct hugepage_subpool *spool)
@@ -38,6 +37,44 @@ static inline void unlock_or_release_subpool(struct hugepage_subpool *spool,
}
}
+long hugepage_subpool_free_hpages(struct hugepage_subpool *spool)
+{
+ long free_pages;
+
+ spin_lock_irq(&spool->lock);
+ if (spool->max_hpages == -1)
+ free_pages = -1;
+ else
+ free_pages = spool->max_hpages - spool->used_hpages;
+ spin_unlock_irq(&spool->lock);
+
+ return free_pages;
+}
+
+static unsigned int hugepage_subpool_hpage_shift(struct hugepage_subpool *spool)
+{
+ return huge_page_shift(spool->hstate);
+}
+
+unsigned long long hugepage_subpool_max_size(struct hugepage_subpool *spool)
+{
+ if (spool->max_hpages == -1)
+ return -1ULL;
+ return (unsigned long long)spool->max_hpages << hugepage_subpool_hpage_shift(spool);
+}
+
+unsigned long long hugepage_subpool_min_size(struct hugepage_subpool *spool)
+{
+ if (spool->min_hpages == -1)
+ return -1ULL;
+ return (unsigned long long)spool->min_hpages << hugepage_subpool_hpage_shift(spool);
+}
+
+long hugepage_subpool_max_hpages(struct hugepage_subpool *spool)
+{
+ return spool->max_hpages;
+}
+
struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
long min_hpages)
{
diff --git a/mm/hugetlb_subpool.h b/mm/hugetlb_subpool.h
index be1f1cf012c9c..41d22239f2c3e 100644
--- a/mm/hugetlb_subpool.h
+++ b/mm/hugetlb_subpool.h
@@ -13,5 +13,9 @@ struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
void hugepage_put_subpool(struct hugepage_subpool *spool);
long hugepage_subpool_get_pages(struct hugepage_subpool *spool, long delta);
long hugepage_subpool_put_pages(struct hugepage_subpool *spool, long delta);
+long hugepage_subpool_free_hpages(struct hugepage_subpool *spool);
+long hugepage_subpool_max_hpages(struct hugepage_subpool *spool);
+unsigned long long hugepage_subpool_max_size(struct hugepage_subpool *spool);
+unsigned long long hugepage_subpool_min_size(struct hugepage_subpool *spool);
#endif /* _MM_HUGETLB_SUBPOOL_H */
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 12/16] WIP: tools: testing: Add unit tests for HugeTLB subpool functions
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
Introduce unit tests for HugeTLB subpool functions to exercise subpool
functions. Set testing up so that tests can be run directly from userspace.
Reuses the private kernel struct hugepage_subpool struct layout natively by
embedding the implementation directly, avoiding structural definition drift
between implementation and testing.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
tools/testing/hugetlb_subpool/.gitignore | 1 +
tools/testing/hugetlb_subpool/Makefile | 18 ++
tools/testing/hugetlb_subpool/test_subpool.c | 400 +++++++++++++++++++++++++++
3 files changed, 419 insertions(+)
diff --git a/tools/testing/hugetlb_subpool/.gitignore b/tools/testing/hugetlb_subpool/.gitignore
new file mode 100644
index 0000000000000..7348c2c72f1e1
--- /dev/null
+++ b/tools/testing/hugetlb_subpool/.gitignore
@@ -0,0 +1 @@
+test_subpool
diff --git a/tools/testing/hugetlb_subpool/Makefile b/tools/testing/hugetlb_subpool/Makefile
new file mode 100644
index 0000000000000..1bdb7e2635614
--- /dev/null
+++ b/tools/testing/hugetlb_subpool/Makefile
@@ -0,0 +1,18 @@
+# SPDX-License-Identifier: GPL-2.0
+.PHONY: all clean test
+
+CC = gcc
+CFLAGS = -Wall -O2 -I../shared -I. -I../../include -I../../arch/x86/include -pthread
+KERNEL_SUBPOOL_H = ../../../mm/hugetlb_subpool.h
+KERNEL_SUBPOOL_C = ../../../mm/hugetlb_subpool.c
+
+all: test
+
+test_subpool: test_subpool.c $(KERNEL_SUBPOOL_C) $(KERNEL_SUBPOOL_H)
+ $(CC) $(CFLAGS) test_subpool.c -o test_subpool
+
+test: test_subpool
+ ./test_subpool
+
+clean:
+ rm -f test_subpool
diff --git a/tools/testing/hugetlb_subpool/test_subpool.c b/tools/testing/hugetlb_subpool/test_subpool.c
new file mode 100644
index 0000000000000..78900274cc264
--- /dev/null
+++ b/tools/testing/hugetlb_subpool/test_subpool.c
@@ -0,0 +1,400 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <assert.h>
+#include <stdlib.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/bug.h>
+
+/* Mocked Userspace implementation for Kernel Subpool allocation dependencies */
+struct hstate {
+ int dummy;
+};
+
+#undef kzalloc_obj
+#undef kzalloc_objs
+#define kzalloc_obj(P, ...) malloc(sizeof(P))
+#define kzalloc_objs(P, COUNT, ...) malloc(sizeof(P) * (COUNT))
+
+#define kfree free
+#define kmalloc malloc
+
+#define huge_page_shift(h) (21 + (0 * ((unsigned long)(h) & 0)))
+#define huge_page_size(h) (1UL << huge_page_shift(h))
+
+static bool hugetlb_acct_memory_called;
+static struct hstate *hugetlb_acct_memory_h;
+static long hugetlb_acct_memory_delta;
+
+static int hugetlb_acct_memory(struct hstate *h, long delta)
+{
+ hugetlb_acct_memory_called = true;
+ hugetlb_acct_memory_h = h;
+ hugetlb_acct_memory_delta = delta;
+ return 0;
+}
+
+static void reset_hugetlb_acct_memory_mock(void)
+{
+ hugetlb_acct_memory_called = false;
+ hugetlb_acct_memory_h = NULL;
+ hugetlb_acct_memory_delta = 0;
+}
+
+static void assert_hugetlb_acct_memory_called(struct hstate *h, long delta)
+{
+ assert(hugetlb_acct_memory_called);
+ assert(hugetlb_acct_memory_h == h);
+ assert(hugetlb_acct_memory_delta == delta);
+
+ reset_hugetlb_acct_memory_mock();
+}
+
+static void assert_hugetlb_acct_memory_not_called(void)
+{
+ assert(!hugetlb_acct_memory_called);
+}
+
+#include "../../../mm/hugetlb_subpool.h"
+#include "../../../mm/hugetlb_subpool.c"
+
+static void test_subpool_new_put_no_min_limit(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+
+ spool = hugepage_new_subpool(&h, 10, -1);
+ assert(spool != NULL);
+ assert(spool->max_hpages == 10);
+ assert(spool->min_hpages == -1);
+ assert(spool->rsv_hpages == -1);
+ assert(spool->count == 1);
+ assert_hugetlb_acct_memory_not_called();
+
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_not_called();
+}
+
+static void test_subpool_new_put_with_min_limit(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+
+ spool = hugepage_new_subpool(&h, 20, 5);
+ assert(spool != NULL);
+ assert(spool->max_hpages == 20);
+ assert(spool->min_hpages == 5);
+ assert(spool->rsv_hpages == 5);
+ assert(spool->count == 1);
+ assert_hugetlb_acct_memory_called(&h, 5);
+
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+}
+
+static void test_subpool_get_pages_below_min(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+ long ret;
+
+ /* Let's initialize: min_hpages = 10, used_hpages = 9, rsv_hpages = 1 */
+ spool = hugepage_new_subpool(&h, -1, 10);
+ assert_hugetlb_acct_memory_called(&h, 10);
+
+ ret = hugepage_subpool_get_pages(spool, 9);
+ assert(ret == 0);
+ assert(spool->used_hpages == 9);
+ assert(spool->rsv_hpages == 1);
+
+ /* Invoke Get (Consumes the remaining 1 subpool reserve!) */
+ ret = hugepage_subpool_get_pages(spool, 1);
+ assert(ret == 0); /* Covered by subpool reserve! */
+ assert(spool->used_hpages == 10);
+ assert(spool->rsv_hpages == 0);
+
+ /* Invoke Put (Replenishes the subpool reserve!) */
+ ret = hugepage_subpool_put_pages(spool, 1);
+ assert(ret == 0); /* Kept by subpool reserve! */
+ assert(spool->used_hpages == 9);
+ assert(spool->rsv_hpages == 1);
+
+ /* Cleanup: Return used_hpages to 0 so the subpool frees symmetrically! */
+ hugepage_subpool_put_pages(spool, 9);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -10);
+}
+
+static void test_subpool_get_pages_crossing_min(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+ long ret;
+
+ /* Let's initialize: min_hpages = 10, used_hpages = 10, rsv_hpages = 0 */
+ spool = hugepage_new_subpool(&h, -1, 10);
+ assert_hugetlb_acct_memory_called(&h, 10);
+
+ hugepage_subpool_get_pages(spool, 10);
+ assert(spool->used_hpages == 10);
+ assert(spool->rsv_hpages == 0);
+
+ /* Invoke Get (Triggers a request for a Global Buddy/Surplus page!) */
+ ret = hugepage_subpool_get_pages(spool, 1);
+ assert(ret == 1); /* Requires global page! */
+ assert(spool->used_hpages == 11);
+ assert(spool->rsv_hpages == 0);
+
+ /* Invoke Put (Above minimum, so it releases the page to the Global Pool!) */
+ ret = hugepage_subpool_put_pages(spool, 1);
+ assert(ret == 1); /* Dropped to global pool! */
+ assert(spool->used_hpages == 10);
+ assert(spool->rsv_hpages == 0);
+
+ /* Cleanup */
+ hugepage_subpool_put_pages(spool, 10);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -10);
+}
+
+static void test_subpool_get_pages_crossing_min_multi(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+ long ret;
+
+ /* Scenario 1: Crossing entirely into surplus territory by a delta > 1 */
+ /* Let's initialize: min_hpages = 10, used_hpages = 8, rsv_hpages = 2 */
+ spool = hugepage_new_subpool(&h, -1, 10);
+ assert_hugetlb_acct_memory_called(&h, 10);
+
+ ret = hugepage_subpool_get_pages(spool, 8);
+ assert(ret == 0);
+ assert(spool->used_hpages == 8);
+ assert(spool->rsv_hpages == 2);
+
+ /* Invoke Get with delta = 5 (Crosses min limit of 10 up to 13) */
+ ret = hugepage_subpool_get_pages(spool, 5);
+ assert(ret == 3); /* (8 + 5) - 10 = 3 global pages required! */
+ assert(spool->used_hpages == 13);
+ assert(spool->rsv_hpages == 0);
+
+ /* Invoke Put with delta = 5 (Drops from 13 down to 8) */
+ ret = hugepage_subpool_put_pages(spool, 5);
+ assert(ret == 3); /* 3 surplus pages released to the global pool! */
+ assert(spool->used_hpages == 8);
+ assert(spool->rsv_hpages == 2); /* 2 subpool reserves perfectly restored! */
+
+ /* Scenario 2: Landing exactly on the min_hpages boundary with delta > 1 */
+ ret = hugepage_subpool_get_pages(spool, 2);
+ assert(ret == 0); /* Perfectly covered by remaining 2 subpool reserves! */
+ assert(spool->used_hpages == 10);
+ assert(spool->rsv_hpages == 0);
+
+ ret = hugepage_subpool_put_pages(spool, 2);
+ assert(ret == 0); /* Swallowed perfectly to replenish the 2 subpool reserves! */
+ assert(spool->used_hpages == 8);
+ assert(spool->rsv_hpages == 2);
+
+ /* Cleanup */
+ hugepage_subpool_put_pages(spool, 8);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -10);
+}
+
+static void test_subpool_get_pages_max_limit(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+ long ret;
+
+ spool = hugepage_new_subpool(&h, 5, -1);
+ assert_hugetlb_acct_memory_not_called();
+
+ ret = hugepage_subpool_get_pages(spool, 5);
+ assert(ret == 5);
+ assert(spool->used_hpages == 5);
+ assert(spool->rsv_hpages == -1);
+
+ /* Invoke Get (Should trigger -ENOMEM due to max cap limit exceeded!) */
+ ret = hugepage_subpool_get_pages(spool, 1);
+ assert(ret == -ENOMEM);
+ assert(spool->used_hpages == 5); /* Unchanged */
+
+ /* Cleanup */
+ hugepage_subpool_put_pages(spool, 5);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_not_called();
+}
+
+static void test_subpool_get_pages_no_limits(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+ long ret;
+
+ spool = hugepage_new_subpool(&h, -1, -1);
+ assert_hugetlb_acct_memory_not_called();
+
+ hugepage_subpool_get_pages(spool, 5);
+ assert(spool->used_hpages == 5);
+
+ /* Invoke Get (Surplus Global Territory) */
+ ret = hugepage_subpool_get_pages(spool, 2);
+ assert(ret == 2);
+ assert(spool->used_hpages == 7);
+
+ /* Invoke Put */
+ ret = hugepage_subpool_put_pages(spool, 2);
+ assert(ret == 2);
+ assert(spool->used_hpages == 5);
+
+ /* Cleanup */
+ hugepage_subpool_put_pages(spool, 5);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_not_called();
+}
+
+static void test_subpool_free_hpages(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+
+ /* Test that free_hpages with NO min_size works perfectly */
+ spool = hugepage_new_subpool(&h, 15, -1);
+ hugepage_subpool_get_pages(spool, 3);
+ assert(hugepage_subpool_free_hpages(spool) == 12);
+ hugepage_subpool_put_pages(spool, 3);
+ hugepage_put_subpool(spool);
+
+ /* Test that free_hpages with a min_size configured is COMPLETELY UNAFFECTED by it */
+ spool = hugepage_new_subpool(&h, 15, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ hugepage_subpool_get_pages(spool, 3);
+ assert(hugepage_subpool_free_hpages(spool) == 12); /* Should still be 15 - 3 = 12! */
+ hugepage_subpool_put_pages(spool, 3);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ spool = hugepage_new_subpool(&h, -1, -1);
+ hugepage_subpool_get_pages(spool, 3);
+ assert(hugepage_subpool_free_hpages(spool) == -1);
+ hugepage_subpool_put_pages(spool, 3);
+ hugepage_put_subpool(spool);
+
+ /* Test that free_hpages with a min_size configured and NO max size returns -1 */
+ spool = hugepage_new_subpool(&h, -1, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ hugepage_subpool_get_pages(spool, 3);
+ assert(hugepage_subpool_free_hpages(spool) == -1);
+ hugepage_subpool_put_pages(spool, 3);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ spool = hugepage_new_subpool(&h, 3, -1);
+ hugepage_subpool_get_pages(spool, 3);
+ assert(hugepage_subpool_free_hpages(spool) == 0);
+ hugepage_subpool_put_pages(spool, 3);
+ hugepage_put_subpool(spool);
+}
+
+static void test_subpool_max_hpages(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+
+ spool = hugepage_new_subpool(&h, 123, -1);
+ assert(hugepage_subpool_max_hpages(spool) == 123);
+ hugepage_put_subpool(spool);
+
+ /* Test that max_hpages with a min_size configured is COMPLETELY UNAFFECTED by it */
+ spool = hugepage_new_subpool(&h, 123, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ assert(hugepage_subpool_max_hpages(spool) == 123);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ spool = hugepage_new_subpool(&h, -1, -1);
+ assert(hugepage_subpool_max_hpages(spool) == -1);
+ hugepage_put_subpool(spool);
+
+ spool = hugepage_new_subpool(&h, -1, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ assert(hugepage_subpool_max_hpages(spool) == -1);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ spool = hugepage_new_subpool(&h, 0, -1);
+ assert(hugepage_subpool_max_hpages(spool) == 0);
+ hugepage_put_subpool(spool);
+}
+
+static void test_subpool_max_size(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+
+ spool = hugepage_new_subpool(&h, 10, -1);
+ assert(hugepage_subpool_max_size(spool) == (10ULL << 21));
+ hugepage_put_subpool(spool);
+
+ /* Test that max_size with a min_size configured is COMPLETELY UNAFFECTED by it */
+ spool = hugepage_new_subpool(&h, 10, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ assert(hugepage_subpool_max_size(spool) == (10ULL << 21));
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ spool = hugepage_new_subpool(&h, -1, -1);
+ assert(hugepage_subpool_max_size(spool) == -1ULL);
+ hugepage_put_subpool(spool);
+
+ spool = hugepage_new_subpool(&h, 0, -1);
+ assert(hugepage_subpool_max_size(spool) == 0ULL);
+ hugepage_put_subpool(spool);
+}
+
+static void test_subpool_min_size(void)
+{
+ struct hstate h;
+ struct hugepage_subpool *spool;
+
+ spool = hugepage_new_subpool(&h, -1, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ assert(hugepage_subpool_min_size(spool) == (5ULL << 21));
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ /* Test that min_size with a max_size configured is COMPLETELY UNAFFECTED by it */
+ spool = hugepage_new_subpool(&h, 20, 5);
+ assert_hugetlb_acct_memory_called(&h, 5);
+ assert(hugepage_subpool_min_size(spool) == (5ULL << 21));
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, -5);
+
+ spool = hugepage_new_subpool(&h, -1, -1);
+ assert(hugepage_subpool_min_size(spool) == -1ULL);
+ hugepage_put_subpool(spool);
+
+ spool = hugepage_new_subpool(&h, -1, 0);
+ assert_hugetlb_acct_memory_called(&h, 0);
+ assert(hugepage_subpool_min_size(spool) == 0ULL);
+ hugepage_put_subpool(spool);
+ assert_hugetlb_acct_memory_called(&h, 0);
+}
+
+int main(void)
+{
+ test_subpool_new_put_no_min_limit();
+ test_subpool_new_put_with_min_limit();
+ test_subpool_get_pages_below_min();
+ test_subpool_get_pages_crossing_min();
+ test_subpool_get_pages_crossing_min_multi();
+ test_subpool_get_pages_max_limit();
+ test_subpool_get_pages_no_limits();
+ test_subpool_free_hpages();
+ test_subpool_max_hpages();
+ test_subpool_max_size();
+ test_subpool_min_size();
+
+ return 0;
+}
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 13/16] WIP: Reproducer for allocation failure due to cgroup v2 memory limits
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
(This reproducer was hacked up and not meant to be merged.)
cgroup_v2_allocation_failure.c triggers HugeTLB allocation failure by exploiting
cgroup v2 memory limits. This allows testing the error paths in the kernel when
memory control charging fails, even when physical huge pages are available.
The program performs the following steps to trigger the failure:
1. Enable hugetlb accounting in cgroup v2.
+ The program checks if memory_hugetlb_accounting is enabled in the cgroup2
mount options. If not, it remounts /sys/fs/cgroup with this option
enabled. This ensures that HugeTLB allocations are charged against the
cgroup memory limits.
2. Create a test cgroup and set limits.
+ The program creates a new cgroup subdirectory named test_reproducer under
/sys/fs/cgroup.
+ It sets the memory.max limit of this cgroup to 1MB (which is less than
the 2MB huge page size).
3. Fork a child process and move it to the test cgroup.
+ The program forks a child process.
+ The child process moves itself into the test_reproducer cgroup by writing
its PID (using 0 for current process) to cgroup.procs in the test cgroup
directory.
4. Attempt to allocate and touch a 2MB huge page.
+ The child process maps a 2MB anonymous huge page using mmap with
MAP_PRIVATE, MAP_ANONYMOUS, and MAP_HUGETLB.
+ The child process writes to the mapped address, triggering a page fault.
5. Triggering the kernel bugs.
+ The page fault handler calls alloc_hugetlb_folio to allocate the huge
page.
+ The allocation of the physical page from buddy allocator succeeds
(assuming nr_hugepages is sufficient).
+ The kernel then attempts to charge this allocation to the child process's
cgroup by calling mem_cgroup_charge_hugetlb.
+ Since the child's cgroup memory limit is 1MB and the page is 2MB, the
charge fails and mem_cgroup_charge_hugetlb returns -ENOMEM.
+ This triggers the error path in alloc_hugetlb_folio where the bugs (folio
refcount mismatch, infinite loop on ENOMEM, and reservation leaks) are
handled.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
cgroup_v2_allocation_failure.c | 169 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 169 insertions(+)
diff --git a/cgroup_v2_allocation_failure.c b/cgroup_v2_allocation_failure.c
new file mode 100644
index 0000000000000..1a813678d8cd2
--- /dev/null
+++ b/cgroup_v2_allocation_failure.c
@@ -0,0 +1,169 @@
+// SPDX-License-Identifier: GPL-2.0
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+#include <string.h>
+#include <errno.h>
+
+#define CGROUP_PATH "/sys/fs/cgroup"
+#define TEST_CGROUP "test_reproducer"
+#define TEST_CGROUP_PATH CGROUP_PATH "/" TEST_CGROUP
+
+static void write_file_val(const char *path, const char *val)
+{
+ int fd = open(path, O_WRONLY);
+
+ if (fd < 0) {
+ fprintf(stderr, "Failed to open %s: %s\n", path, strerror(errno));
+ exit(1);
+ }
+ if (write(fd, val, strlen(val)) < 0) {
+ fprintf(stderr, "Failed to write %s to %s: %s\n", val, path, strerror(errno));
+ close(fd);
+ exit(1);
+ }
+ close(fd);
+}
+
+static int is_hugetlb_accounting_enabled(void)
+{
+ char spec[256], file[256], type[256], opts[512];
+ char line[1024];
+ int enabled = 0;
+ FILE *fp;
+
+ fp = fopen("/proc/mounts", "r");
+ if (!fp) {
+ perror("fopen /proc/mounts");
+ return -1;
+ }
+
+ while (fgets(line, sizeof(line), fp)) {
+ if (sscanf(line, "%255s %255s %255s %511s", spec, file, type, opts) == 4) {
+ if (strcmp(file, CGROUP_PATH) == 0 && strcmp(type, "cgroup2") == 0) {
+ if (strstr(opts, "memory_hugetlb_accounting") != NULL)
+ enabled = 1;
+ break;
+ }
+ }
+ }
+ fclose(fp);
+ return enabled;
+}
+
+static int enable_hugetlb_accounting(void)
+{
+ int ret;
+
+ printf("Attempting to remount cgroup2 with memory_hugetlb_accounting...\n");
+ ret = system("mount -o remount,memory_hugetlb_accounting " CGROUP_PATH);
+ if (ret != 0) {
+ fprintf(stderr, "Failed to remount: system() returned %d\n", ret);
+ return -1;
+ }
+ return 0;
+}
+
+int main(int argc, char **argv)
+{
+ struct stat st;
+ size_t size;
+ void *addr;
+ pid_t pid;
+ int enabled;
+ int status;
+ int fd;
+
+ if (stat(CGROUP_PATH, &st) != 0 || !S_ISDIR(st.st_mode)) {
+ fprintf(stderr, "cgroup v2 not mounted at %s\n", CGROUP_PATH);
+ return 1;
+ }
+
+ enabled = is_hugetlb_accounting_enabled();
+ if (enabled < 0)
+ return 1;
+
+ if (!enabled) {
+ if (enable_hugetlb_accounting() != 0) {
+ fprintf(stderr, "Could not enable memory_hugetlb_accounting\n");
+ return 1;
+ }
+ /* Re-check */
+ enabled = is_hugetlb_accounting_enabled();
+ if (enabled <= 0) {
+ fprintf(stderr, "Failed to enable memory_hugetlb_accounting (re-check failed)\n");
+ return 1;
+ }
+ printf("Successfully enabled memory_hugetlb_accounting\n");
+ } else {
+ printf("memory_hugetlb_accounting is already enabled\n");
+ }
+
+ /* Enable memory controller in subtree */
+ fd = open(CGROUP_PATH "/cgroup.subtree_control", O_WRONLY);
+ if (fd >= 0) {
+ (void)write(fd, "+memory", 7);
+ close(fd);
+ }
+
+ if (mkdir(TEST_CGROUP_PATH, 0755) != 0) {
+ if (errno != EEXIST) {
+ perror("mkdir test_reproducer");
+ return 1;
+ }
+ }
+
+ /* Set memory limit to 1MB (less than 2MB hugepage) */
+ write_file_val(TEST_CGROUP_PATH "/memory.max", "1M");
+
+ pid = fork();
+ if (pid < 0) {
+ perror("fork");
+ return 1;
+ }
+
+ if (pid == 0) {
+ /* Child: Move to cgroup */
+ write_file_val(TEST_CGROUP_PATH "/cgroup.procs", "0");
+
+ printf("Child: Attempting to allocate and touch 2MB hugepage...\n");
+ /* Allocate 2MB hugepage */
+ size = 2 * 1024 * 1024;
+ addr = mmap(NULL, size, PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB, -1, 0);
+ if (addr == MAP_FAILED) {
+ perror("Child: mmap MAP_HUGETLB");
+ exit(1);
+ }
+
+ printf("Child: mmap succeeded at %p, touching it now...\n", addr);
+ *(char *)addr = 1;
+
+ printf("Child: Successfully touched page (bug not triggered?).\n");
+ munmap(addr, size);
+ exit(0);
+ }
+
+ /* Parent */
+ waitpid(pid, &status, 0);
+
+ printf("Parent: Child exited. Cleaning up.\n");
+ rmdir(TEST_CGROUP_PATH);
+
+ if (WIFSIGNALED(status)) {
+ printf("Parent: Child killed by signal %d (%s)\n",
+ WTERMSIG(status), strsignal(WTERMSIG(status)));
+ if (WTERMSIG(status) == SIGBUS)
+ printf("Parent: Child got SIGBUS as expected.\n");
+ } else if (WIFEXITED(status)) {
+ printf("Parent: Child exited with status %d\n", WEXITSTATUS(status));
+ }
+
+ return 0;
+}
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 14/16] WIP: Reproducer for subpool usage leak
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
(This reproducer was hacked up and not meant to be merged.)
The kernel leaks subpool usage and the subpool structure itself if a HugeTLBfs
mount specifying size (which sets max_hpages on the subpool) is created.
subpool_leak_max_size.sh reproduces this with the following steps:
1. Create mount, specifying size=2M (1 page). This sets max_hpages = 1 on the
subpool, but does not reserve any pages.
2. Set nr_hugepages = 0 and nr_overcommit_hugepages = 0 so that physical
allocations will fail.
3. Run fallocate -l 2M on a file in the mount.
+ This calls hugetlbfs_fallocate, which attempts to allocate a page by
calling alloc_hugetlb_folio.
+ alloc_hugetlb_folio calls hugepage_subpool_get_pages to track the
allocation against the subpool limit. This increments used_hpages to 1.
+ Physical allocation fails because nr_hugepages is 0.
+ Before patch (Buggy):
+ The error path in alloc_hugetlb_folio sees gbl_chg is 1 (indicating
we tried to allocate a global page) and incorrectly skips calling
hugepage_subpool_put_pages.
+ fallocate fails and returns to userspace, but the subpool used_hpages
counter remains leaked at 1.
+ After patch:
+ The error path always calls hugepage_subpool_put_pages if map_chg is
true, restoring used_hpages to 0.
4. Unmount the filesystem.
+ During unmount, the kernel calls unlock_or_release_subpool to clean up
the subpool.
+ It checks if the subpool is free using subpool_is_free, which returns
whether used_hpages is 0.
+ Before patch (Buggy):
+ Since used_hpages leaked and is 1, subpool_is_free returns false.
+ The kernel skips freeing the subpool structure, leaking the
hugepage_subpool structure in kernel memory.
+ After patch:
+ Since used_hpages is 0, subpool_is_free returns true, and the subpool
structure is correctly freed.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
subpool_leak_max_size.sh | 72 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 72 insertions(+)
diff --git a/subpool_leak_max_size.sh b/subpool_leak_max_size.sh
new file mode 100755
index 0000000000000..226fd2766c4d0
--- /dev/null
+++ b/subpool_leak_max_size.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+if [ "$EUID" -ne 0 ]; then
+ echo "Please run as root"
+ exit 1
+fi
+
+MNT_PATH="/tmp/mnt_hugetlb"
+FILE_PATH="$MNT_PATH/test_file"
+
+# Save original values
+orig_nr=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
+orig_overcommit=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages)
+
+cleanup() {
+ echo "Cleaning up..."
+ rm -f "$FILE_PATH"
+ umount "$MNT_PATH" 2>/dev/null
+ rmdir "$MNT_PATH" 2>/dev/null
+ echo "$orig_nr" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+ echo "$orig_overcommit" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+ echo "Cleanup done."
+}
+trap cleanup EXIT
+
+# 1. Mount hugetlbfs with size=2M (1 page)
+mkdir -p "$MNT_PATH"
+if ! mount -t hugetlbfs -o size=2M none "$MNT_PATH"; then
+ echo "Failed to mount hugetlbfs"
+ exit 1
+fi
+
+# 2. Set nr_hugepages to 0, overcommit to 0
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+echo 0 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_overcommit_hugepages
+
+# Check subpool usage before running
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used_before=$((total - free))
+echo "Before test - Subpool total blocks: $total"
+echo "Before test - Subpool free blocks: $free"
+echo "Before test - Subpool used blocks: $used_before"
+if [ "$used_before" -ne 0 ]; then
+ echo "ERROR: Subpool is not clean before test starts!"
+ exit 1
+fi
+
+# Run fallocate (expecting failure)
+echo "Running fallocate (expecting failure)..."
+if fallocate -l 2M "$FILE_PATH" 2>/dev/null; then
+ echo "ERROR: fallocate succeeded but should have failed (nr_hugepages is 0)"
+ exit 1
+fi
+
+# Check subpool usage via statfs
+# %b: Total blocks
+# %f: Free blocks
+read total free < <(stat -f -c "%b %f" "$MNT_PATH")
+used=$((total - free))
+
+echo "Subpool total blocks: $total"
+echo "Subpool free blocks: $free"
+echo "Subpool used blocks (leaked if > 0): $used"
+
+if [ "$used" -gt 0 ]; then
+ echo "RESULT: LEAK DETECTED (FAIL)"
+ exit 1
+else
+ echo "RESULT: NO LEAK (PASS)"
+ exit 0
+fi
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
* [PATCH v4 15/16] WIP: Reproducer for false restoration on shared HugeTLB mappings
From: Ackerley Tng via B4 Relay @ 2026-07-22 23:41 UTC (permalink / raw)
To: Muchun Song, Oscar Salvador, David Hildenbrand, Joshua Hahn,
Shakeel Butt, Nhat Pham, Andrew Morton, Peter Xu, Wupeng Ma, fvdl,
rientjes, jthoughton, Mike Kravetz, Johannes Weiner, Michal Hocko,
Roman Gushchin, Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka,
Mike Rapoport, Suren Baghdasaryan, Michal Hocko, Jonathan Corbet,
Shuah Khan, Alex Shi, Yanteng Si, Dongliang Mu, Hongxiang Lou,
Miaohe Lin
Cc: vannapurve, erdemaktas, linux-mm, linux-kernel, cgroups,
linux-doc, Ackerley Tng
In-Reply-To: <20260722-hugetlb-alloc-failure-fixes-v4-0-88e8b81970dc@google.com>
From: Ackerley Tng <ackerleytng@google.com>
(This reproducer was hacked up and not meant to be merged.)
hugetlb_unreserve_pages() unconditionally returns reservations to the subpool
(via hugepage_subpool_put_pages()). This means that regardless of whether a
subpool reservation was actually used, the reservation is processed by the
subpool structure.
To create a false restoration, the reproducer performs these steps:
1. Mount with min_size=2M (1 page). Global resv_hugepages becomes 1.
2. The program maps 4MB (2 pages) shared (which also grows the file to
4MB). Global resv_hugepages becomes 2 (1 from the mount, 1 new global
reservation).
3. The program populates only the first page. Global resv_hugepages decrements
to 1 (reservation consumed by allocation).
4. The program exits (closing VMAs/fds). For shared mappings, reservations are
associated with the file inode, so they remain active. Global resv_hugepages
remains 1.
5. The script truncates the file to 2MB (truncate -s 2M).
+ This synchronously triggers hugetlb_unreserve_pages() to release the
reservation of the truncated range (the unallocated 2nd page).
+ It calls hugepage_subpool_put_pages(spool, 1).
+ On Vanilla Kernel (Buggy):
+ used_hpages is 0 (not tracked).
+ used_hpages (0) < min_hpages (1) is TRUE.
+ The subpool incorrectly restores the reservation (spool->rsv_hpages
becomes 1), even though Page 0 is still allocated and satisfies the
mount's minimum guarantee.
+ hugepage_subpool_put_pages() returns 0, skipping
hugetlb_acct_memory(h, -1).
+ Result: Global resv_hugepages remains stuck at 1 (Leak).
+ On Fixed Kernel:
+ used_hpages is tracked and is initially 2.
+ hugepage_subpool_put_pages(1) decrements used_hpages to 1.
+ used_hpages (1) < min_hpages (1) is FALSE.
+ The subpool does not restore the reservation.
+ hugepage_subpool_put_pages() returns 1.
+ hugetlb_acct_memory(h, -1) is called.
+ Result: Global resv_hugepages decrements to 0 (No leak).
When the filesystem is unmounted, hugetlbfs_put_super drops the subpool
reference. Since the filesystem is being unmounted, the reference count drops to
0, triggering unlock_or_release_subpool.
Inside unlock_or_release_subpool, the kernel checks if the subpool is free using
subpool_is_free.
+ On the buggy kernel, subpool_is_free checks if spool->rsv_hpages is equal to
spool->min_hpages. Because of the phantom reservation, spool->rsv_hpages was
restored to 1. Since min_hpages is 1, the check (1 == 1) returns true.
+ Since the subpool is considered free, the kernel releases the initial
mount-time reservation by calling hugetlb_acct_memory to decrement
resv_huge_pages by spool->min_hpages (which is 1).
+ This decrement reduces resv_huge_pages from 1 (the leaked state) to 0.
As a result, the leaked reservation is cleaned up during unmount and does not
persist afterward.
Signed-off-by: Ackerley Tng <ackerleytng@google.com>
---
subpool_shared_leak.c | 43 +++++++++++++++++++++++++
subpool_shared_leak.sh | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 130 insertions(+)
diff --git a/subpool_shared_leak.c b/subpool_shared_leak.c
new file mode 100644
index 0000000000000..80b972af1c5ce
--- /dev/null
+++ b/subpool_shared_leak.c
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/mman.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+
+#define HPAGE_SIZE (2 * 1024 * 1024)
+
+int main(int argc, char **argv)
+{
+ const char *file_path;
+ void *addr;
+ int fd;
+
+ if (argc < 2) {
+ fprintf(stderr, "Usage: %s <file_path>\n", argv[0]);
+ return 1;
+ }
+ file_path = argv[1];
+
+ fd = open(file_path, O_CREAT | O_RDWR, 0666);
+ if (fd < 0) {
+ perror("open");
+ return 1;
+ }
+
+ addr = mmap(NULL, 2 * HPAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+ if (addr == MAP_FAILED) {
+ perror("mmap");
+ close(fd);
+ return 1;
+ }
+
+ /* Allocate 1st page only. 2nd page remains unallocated (but reserved). */
+ *(char *)addr = 1;
+
+ munmap(addr, 2 * HPAGE_SIZE);
+ close(fd);
+
+ return 0;
+}
diff --git a/subpool_shared_leak.sh b/subpool_shared_leak.sh
new file mode 100755
index 0000000000000..619036145c523
--- /dev/null
+++ b/subpool_shared_leak.sh
@@ -0,0 +1,87 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+if [ "$EUID" -ne 0 ]; then
+ echo "Please run as root"
+ exit 1
+fi
+
+MNT_PATH="/tmp/mnt_hugetlb_shared_leak"
+FILE_PATH="$MNT_PATH/test_file"
+
+# Save original values
+orig_nr=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages)
+
+cleanup() {
+ echo "Cleaning up..."
+ rm -f "$FILE_PATH"
+ umount "$MNT_PATH" 2>/dev/null
+ rmdir "$MNT_PATH" 2>/dev/null
+ echo "$orig_nr" > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+ echo "Cleanup done."
+}
+trap cleanup EXIT
+
+# 1. Set nr_hugepages to 2
+echo 2 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
+
+# 2. Mount hugetlbfs with min_size=2M (1 page)
+mkdir -p "$MNT_PATH"
+if ! mount -t hugetlbfs -o min_size=2M none "$MNT_PATH"; then
+ echo "Failed to mount hugetlbfs"
+ exit 1
+fi
+
+# Check resv_hugepages after mount (should be 1)
+initial_resv=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages)
+echo "Initial resv_hugepages (after mount): $initial_resv"
+if [ "$initial_resv" -ne 1 ]; then
+ echo "ERROR: Initial resv_hugepages is not 1!"
+ exit 1
+fi
+
+# Verify reproducer binary exists
+if [ ! -x ./subpool_shared_leak ]; then
+ echo "reproducer binary './subpool_shared_leak' not found or not executable."
+ echo "Please compile it first: gcc -static -o subpool_shared_leak subpool_shared_leak.c"
+ exit 1
+fi
+
+# 3. Run helper to map 4MB, allocate 2MB, and close.
+# This creates 2 reservations, consumes 1 (by allocating Page 0).
+# The unallocated Page 1 reservation remains active in the inode's resv_map.
+echo "Running helper..."
+./subpool_shared_leak "$FILE_PATH"
+
+resv_after_helper=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages)
+echo "resv_hugepages after helper (should be 1): $resv_after_helper"
+# Page 0 is allocated (no longer reserved). Page 1 is reserved.
+# So resv_hugepages should be 1.
+if [ "$resv_after_helper" -ne 1 ]; then
+ echo "ERROR: resv_hugepages is not 1 after helper run!"
+ exit 1
+fi
+
+# 4. Truncate file to 2MB (releases Page 1 reservation)
+echo "Truncating file to 2MB (releasing 1 page reservation)..."
+truncate -s 2M "$FILE_PATH"
+
+# Check resv_hugepages after truncate.
+# Since Page 0 is still allocated (and in page cache), and satisfies the
+# min_size=2M guarantee, we should have 0 reservations remaining.
+# If the bug is present, the truncate path will incorrectly restore the
+# reservation to the subpool and skip releasing it globally, leaving
+# resv_hugepages at 1.
+final_resv=$(cat /sys/kernel/mm/hugepages/hugepages-2048kB/resv_hugepages)
+echo "Final resv_hugepages (after 2MB truncate): $final_resv"
+
+if [ "$final_resv" -eq 1 ]; then
+ echo "RESULT: LEAK DETECTED (FAIL)"
+ exit 1
+elif [ "$final_resv" -eq 0 ]; then
+ echo "RESULT: NO LEAK (PASS)"
+ exit 0
+else
+ echo "RESULT: UNEXPECTED STATE ($final_resv)"
+ exit 2
+fi
--
2.55.0.229.g6434b31f56-goog
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox