* [PATCH v10 03/11] selftests: tracing: Add a basic testcase for wprobe
From: Masami Hiramatsu (Google) @ 2026-07-22 23:03 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add 'add_remove_wprobe.tc' testcase for testing wprobe event that
tests adding and removing operations of the wprobe event.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v9:
- Fix command check logic to prevent early exit under 'set -e'
(errexit) when grep or test fails.
- Simplify enable/disable status checks by removing cat pipes.
Changes in v8:
- Fixed silently test failure path.
---
tools/testing/selftests/ftrace/config | 1
.../ftrace/test.d/dynevent/add_remove_wprobe.tc | 63 ++++++++++++++++++++
2 files changed, 64 insertions(+)
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
diff --git a/tools/testing/selftests/ftrace/config b/tools/testing/selftests/ftrace/config
index 544de0db5f58..d2f503722020 100644
--- a/tools/testing/selftests/ftrace/config
+++ b/tools/testing/selftests/ftrace/config
@@ -27,3 +27,4 @@ CONFIG_STACK_TRACER=y
CONFIG_TRACER_SNAPSHOT=y
CONFIG_UPROBES=y
CONFIG_UPROBE_EVENTS=y
+CONFIG_WPROBE_EVENTS=y
diff --git a/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc b/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
new file mode 100644
index 000000000000..647c37d5e4c8
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
@@ -0,0 +1,63 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Generic dynamic event - add/remove wprobe events
+# requires: dynamic_events "w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]":README
+
+echo 0 > events/enable
+echo > dynamic_events
+
+# Use jiffies as a variable that is frequently written to.
+TARGET=jiffies
+
+echo "w:my_wprobe w@$TARGET" >> dynamic_events
+
+if ! grep -q my_wprobe dynamic_events; then
+ echo "Failed to create wprobe event"
+ exit_fail
+fi
+
+if [ ! -d events/wprobes/my_wprobe ]; then
+ echo "Failed to create wprobe event directory"
+ exit_fail
+fi
+
+echo 1 > events/wprobes/my_wprobe/enable
+
+# Check if the event is enabled
+if ! grep -q 1 events/wprobes/my_wprobe/enable; then
+ echo "Failed to enable wprobe event"
+ exit_fail
+fi
+
+# Let some time pass to trigger the breakpoint
+sleep 1
+
+# Check if we got any trace output
+if ! grep -q my_wprobe trace; then
+ echo "wprobe event was not triggered"
+ exit_fail
+fi
+
+echo 0 > events/wprobes/my_wprobe/enable
+
+# Check if the event is disabled
+if ! grep -q 0 events/wprobes/my_wprobe/enable; then
+ echo "Failed to disable wprobe event"
+ exit_fail
+fi
+
+echo "-:my_wprobe" >> dynamic_events
+
+if grep -q my_wprobe dynamic_events; then
+ echo "Failed to remove wprobe event"
+ exit_fail
+fi
+
+if [ -d events/wprobes/my_wprobe ]; then
+ echo "Failed to remove wprobe event directory"
+ exit_fail
+fi
+
+clear_trace
+
+exit 0
^ permalink raw reply related
* [PATCH v10 02/11] x86: hw_breakpoint: Add a kconfig to clarify when a breakpoint fires
From: Masami Hiramatsu (Google) @ 2026-07-22 23:02 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add CONFIG_HAVE_POST_BREAKPOINT_HOOK which indicates the hw_breakpoint
on that architecture fires after the target memory has been modified.
This is currently x86 only behavior.
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
arch/Kconfig | 10 ++++++++++
arch/x86/Kconfig | 1 +
kernel/trace/Kconfig | 1 +
3 files changed, 12 insertions(+)
diff --git a/arch/Kconfig b/arch/Kconfig
index fa7507ac8e13..959aee9568ff 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -457,6 +457,16 @@ config HAVE_MIXED_BREAKPOINTS_REGS
Select this option if your arch implements breakpoints under the
latter fashion.
+config HAVE_POST_BREAKPOINT_HOOK
+ bool
+ depends on HAVE_HW_BREAKPOINT
+ help
+ Depending on the arch implementation of hardware breakpoints,
+ some of them provide breakpoint hook after the target memory
+ is modified.
+ Select this option if your arch implements breakpoints overflow
+ handler hooks after the target memory is modified.
+
config HAVE_USER_RETURN_NOTIFIER
bool
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index bdad90f210e4..6b7e14ef8cfb 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -246,6 +246,7 @@ config X86
select HAVE_FUNCTION_TRACER
select HAVE_GCC_PLUGINS
select HAVE_HW_BREAKPOINT
+ select HAVE_POST_BREAKPOINT_HOOK
select HAVE_IOREMAP_PROT
select HAVE_IRQ_EXIT_ON_IRQ_STACK if X86_64
select HAVE_IRQ_TIME_ACCOUNTING
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index b58c2565024f..d9b6fa5c35d9 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -866,6 +866,7 @@ config WPROBE_EVENTS
bool "Enable wprobe-based dynamic events"
depends on TRACING
depends on HAVE_HW_BREAKPOINT
+ depends on HAVE_POST_BREAKPOINT_HOOK
select PROBE_EVENTS
select DYNAMIC_EVENTS
help
^ permalink raw reply related
* [PATCH v10 01/11] tracing: wprobe: Add watchpoint probe event based on hardware breakpoint
From: Masami Hiramatsu (Google) @ 2026-07-22 23:02 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
In-Reply-To: <178476134787.26117.10094977293012760490.stgit@devnote2>
From: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Add a new probe event for the hardware breakpoint called wprobe-event.
This wprobe allows user to trace (watch) the memory access at the
specified memory address.
The new syntax is;
w[:[GROUP/]EVENT] [r|w|rw]@[ADDR|SYM][:SIZE] [FETCH_ARGs]
User also can use $addr to fetch the accessed address and $value to fetch
the accessed memory value (shorthand for '+0($addr)'). No other variables
are supported.
For example, tracing updates of the jiffies;
/sys/kernel/tracing # echo 'w:my_jiffies w@jiffies' >> dynamic_events
/sys/kernel/tracing # cat dynamic_events
w:wprobes/my_jiffies w@jiffies:4
/sys/kernel/tracing # echo 1 > events/wprobes/my_jiffies/enable
/sys/kernel/tracing # head -n 20 trace | tail -n 5
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
<idle>-0 [000] d.Z1. 206.547317: my_jiffies: (tick_do_update_jiffies64+0xbe/0x130)
<idle>-0 [000] d.Z1. 206.548341: my_jiffies: (tick_do_update_jiffies64+0xbe/0x130)
<idle>-0 [000] d.Z1. 206.549346: my_jiffies: (tick_do_update_jiffies64+0xbe/0x130)
Assisted-by: Antigravity:gemini-3.5-flash
Signed-off-by: Masami Hiramatsu (Google) <mhiramat@kernel.org>
---
Changes in v10:
- Use zalloc_flex() in alloc_trace_wprobe().
- Support matching command arguments in trace_wprobe_match().
Changes in v9:
- Rebased on probes/for-next branch.
- Add WPROBE_NO_SIBLING error log to explicitly reject sibling probes
since event triggers identify the target wprobe by event name.
- Use traceprobe_parse_event_name() to properly validate group/event
names instead of using the raw command string directly.
- Generate unique event name (w_0x<addr>) for anonymous address-based
watchpoints to avoid naming collisions.
- Call traceprobe_update_arg() in __register_trace_wprobe() to resolve
@symbol fetch arguments, consistent with kprobe and fprobe.
Changes in v8:
- Include required header files.
- Use READ_ONCE(tw->addr) in trace handler to safely check dynamically
updated addresses.
- Prohibit unsafe perf support by returning -EOPNOTSUPP in
wprobe_register().
- Add rollback logic to unregister already-enabled sibling probes if
registration fails mid-loop.
- Resolve symbol offsets dynamically in trace_wprobe_show() using
kallsyms_lookup_name().
- Fix memory leak of parse_address_spec()'s symbol output in
__trace_wprobe_create().
- Print "rw" instead of "x" for read-write type breakpoints in
trace_wprobe_show().
- Document the $value fetcharg in wprobetrace.rst.
Changes in v7:
- Include IS_ERR_PCPU fix.
- use seq_print_ip_sym_offset().
- fix checkpatch warning on DEFINE_FREE()
- Use bp->attr.bp_addr instead of tw->addr because it can be updated from another CPU.
---
Documentation/trace/index.rst | 1
Documentation/trace/wprobetrace.rst | 70 +++
include/linux/trace_events.h | 2
kernel/trace/Kconfig | 13 +
kernel/trace/Makefile | 1
kernel/trace/trace.c | 9
kernel/trace/trace.h | 5
kernel/trace/trace_probe.c | 21 +
kernel/trace/trace_probe.h | 9
kernel/trace/trace_wprobe.c | 746 +++++++++++++++++++++++++++++++++++
10 files changed, 873 insertions(+), 4 deletions(-)
create mode 100644 Documentation/trace/wprobetrace.rst
create mode 100644 kernel/trace/trace_wprobe.c
diff --git a/Documentation/trace/index.rst b/Documentation/trace/index.rst
index 5d9bf4694d5d..2f04f32001ed 100644
--- a/Documentation/trace/index.rst
+++ b/Documentation/trace/index.rst
@@ -36,6 +36,7 @@ the Linux kernel.
kprobes
kprobetrace
fprobetrace
+ wprobetrace
eprobetrace
fprobe
ring-buffer-design
diff --git a/Documentation/trace/wprobetrace.rst b/Documentation/trace/wprobetrace.rst
new file mode 100644
index 000000000000..eb4f10607530
--- /dev/null
+++ b/Documentation/trace/wprobetrace.rst
@@ -0,0 +1,70 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=======================================
+Watchpoint probe (wprobe) Event Tracing
+=======================================
+
+.. Author: Masami Hiramatsu <mhiramat@kernel.org>
+
+Overview
+--------
+
+Wprobe event is a dynamic event based on the hardware breakpoint, which is
+similar to other probe events, but it is for watching data access. It allows
+you to trace which code accesses a specified data.
+
+As same as other dynamic events, wprobe events are defined via
+`dynamic_events` interface file on tracefs.
+
+Synopsis of wprobe-events
+-------------------------
+::
+
+ w:[GRP/][EVENT] SPEC [FETCHARGS] : Probe on data access
+
+ GRP : Group name for wprobe. If omitted, use "wprobes" for it.
+ EVENT : Event name for wprobe. If omitted, an event name is
+ generated based on the address or symbol.
+ SPEC : Breakpoint specification.
+ [r|w|rw]@<ADDRESS|SYMBOL[+|-OFFS]>[:LENGTH]
+
+ r|w|rw : Access type, r for read, w for write, and rw for both.
+ Default is rw if omitted.
+ ADDRESS : Address to trace (hexadecimal).
+ SYMBOL : Symbol name to trace.
+ LENGTH : Length of the data to trace in bytes. (1, 2, 4, or 8)
+
+ FETCHARGS : Arguments. Each probe can have up to 128 args.
+ $addr : Fetch the accessing address.
+ $value : Fetch the memory value at the accessing address (same as +0($addr)).
+ @ADDR : Fetch memory at ADDR (ADDR should be in kernel)
+ @SYM[+|-offs] : Fetch memory at SYM +|- offs (SYM should be a data symbol)
+ +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*1)(\*2)
+ \IMM : Store an immediate value to the argument.
+ NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
+ FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
+ (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
+ (x8/x16/x32/x64), "char", "string", "ustring", "symbol", "symstr"
+ and bitfield are supported.
+
+ (\*1) this is useful for fetching a field of data structures.
+ (\*2) "u" means user-space dereference.
+
+For the details of TYPE, see :ref:`kprobetrace documentation <kprobetrace_types>`.
+
+Usage examples
+--------------
+Here is an example to add a wprobe event on a variable `jiffies`.
+::
+
+ # echo 'w:my_jiffies w@jiffies' >> dynamic_events
+ # cat dynamic_events
+ w:wprobes/my_jiffies w@jiffies
+ # echo 1 > events/wprobes/enable
+ # cat trace | head
+ # TASK-PID CPU# ||||| TIMESTAMP FUNCTION
+ # | | | ||||| | |
+ <idle>-0 [000] d.Z1. 717.026259: my_jiffies: (tick_do_update_jiffies64+0xbe/0x130)
+ <idle>-0 [000] d.Z1. 717.026373: my_jiffies: (tick_do_update_jiffies64+0xbe/0x130)
+
+You can see the code which writes to `jiffies` is `tick_do_update_jiffies64()`.
diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
index 308c76b57d13..d1e5ab71d928 100644
--- a/include/linux/trace_events.h
+++ b/include/linux/trace_events.h
@@ -328,6 +328,7 @@ enum {
TRACE_EVENT_FL_UPROBE_BIT,
TRACE_EVENT_FL_EPROBE_BIT,
TRACE_EVENT_FL_FPROBE_BIT,
+ TRACE_EVENT_FL_WPROBE_BIT,
TRACE_EVENT_FL_CUSTOM_BIT,
TRACE_EVENT_FL_TEST_STR_BIT,
};
@@ -358,6 +359,7 @@ enum {
TRACE_EVENT_FL_UPROBE = (1 << TRACE_EVENT_FL_UPROBE_BIT),
TRACE_EVENT_FL_EPROBE = (1 << TRACE_EVENT_FL_EPROBE_BIT),
TRACE_EVENT_FL_FPROBE = (1 << TRACE_EVENT_FL_FPROBE_BIT),
+ TRACE_EVENT_FL_WPROBE = (1 << TRACE_EVENT_FL_WPROBE_BIT),
TRACE_EVENT_FL_CUSTOM = (1 << TRACE_EVENT_FL_CUSTOM_BIT),
TRACE_EVENT_FL_TEST_STR = (1 << TRACE_EVENT_FL_TEST_STR_BIT),
};
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 0ab5916575a9..b58c2565024f 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -862,6 +862,19 @@ config EPROBE_EVENTS
convert the type of an event field. For example, turn an
address into a string.
+config WPROBE_EVENTS
+ bool "Enable wprobe-based dynamic events"
+ depends on TRACING
+ depends on HAVE_HW_BREAKPOINT
+ select PROBE_EVENTS
+ select DYNAMIC_EVENTS
+ help
+ This allows the user to add watchpoint tracing events based on
+ hardware breakpoints on the fly via the ftrace interface.
+
+ Those events can be inserted wherever hardware breakpoints can be
+ set, and record accessed memory address and values.
+
config BPF_EVENTS
depends on BPF_SYSCALL
depends on (KPROBE_EVENTS || UPROBE_EVENTS) && PERF_EVENTS
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index f934ff586bd4..141c8323de20 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -126,6 +126,7 @@ obj-$(CONFIG_FTRACE_RECORD_RECURSION) += trace_recursion_record.o
obj-$(CONFIG_FPROBE) += fprobe.o
obj-$(CONFIG_RETHOOK) += rethook.o
obj-$(CONFIG_FPROBE_EVENTS) += trace_fprobe.o
+obj-$(CONFIG_WPROBE_EVENTS) += trace_wprobe.o
obj-$(CONFIG_TRACEPOINT_BENCHMARK) += trace_benchmark.o
obj-$(CONFIG_RV) += rv/
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index c9e182d40059..1bc27c0ad029 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -4294,8 +4294,12 @@ static const char readme_msg[] =
" uprobe_events\t\t- Create/append/remove/show the userspace dynamic events\n"
"\t\t\t Write into this file to define/undefine new trace events.\n"
#endif
+#ifdef CONFIG_WPROBE_EVENTS
+ " wprobe_events\t\t- Create/append/remove/show the hardware breakpoint dynamic events\n"
+ "\t\t\t Write into this file to define/undefine new trace events.\n"
+#endif
#if defined(CONFIG_KPROBE_EVENTS) || defined(CONFIG_UPROBE_EVENTS) || \
- defined(CONFIG_FPROBE_EVENTS)
+ defined(CONFIG_FPROBE_EVENTS) || defined(CONFIG_WPROBE_EVENTS)
"\t accepts: event-definitions (one definition per line)\n"
#if defined(CONFIG_KPROBE_EVENTS) || defined(CONFIG_UPROBE_EVENTS)
"\t Format: p[:[<group>/][<event>]] <place> [<args>]\n"
@@ -4305,6 +4309,9 @@ static const char readme_msg[] =
"\t f[:[<group>/][<event>]] <func-name>[%return] [<args>]\n"
"\t t[:[<group>/][<event>]] <tracepoint> [<args>]\n"
#endif
+#ifdef CONFIG_WPROBE_EVENTS
+ "\t w[:[<group>/][<event>]] [r|w|rw]@<addr>[:<len>]\n"
+#endif
#ifdef CONFIG_HIST_TRIGGERS
"\t s:[synthetic/]<event> <field> [<field>]\n"
#endif
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 80fe152af1dd..2f07c5c4ffc8 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -179,6 +179,11 @@ struct fexit_trace_entry_head {
unsigned long ret_ip;
};
+struct wprobe_trace_entry_head {
+ struct trace_entry ent;
+ unsigned long ip;
+};
+
#define TRACE_BUF_SIZE 1024
struct trace_array;
diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c
index 7568f5e68de7..5d5e9b477b86 100644
--- a/kernel/trace/trace_probe.c
+++ b/kernel/trace/trace_probe.c
@@ -1405,6 +1405,23 @@ static int parse_probe_vars(char *orig_arg, const struct fetch_type *t,
return 0;
}
+ /* wprobe only support "$addr" and "$value" variable */
+ if (ctx->flags & TPARG_FL_WPROBE) {
+ if (!strcmp(arg, "addr")) {
+ code->op = FETCH_OP_BADDR;
+ return 0;
+ }
+ if (!strcmp(arg, "value")) {
+ code->op = FETCH_OP_BADDR;
+ code++;
+ code->op = FETCH_OP_DEREF;
+ code->offset = 0;
+ *pcode = code;
+ return 0;
+ }
+ goto inval;
+ }
+
if (strcmp(arg, "comm") == 0 || strcmp(arg, "COMM") == 0) {
code->op = FETCH_OP_COMM;
return 0;
@@ -1464,8 +1481,8 @@ static int parse_probe_arg_register(char *arg, struct fetch_insn *code,
{
int ret;
- if (ctx->flags & (TPARG_FL_TEVENT | TPARG_FL_FPROBE)) {
- /* eprobe and fprobe do not handle registers */
+ if (ctx->flags & (TPARG_FL_TEVENT | TPARG_FL_FPROBE | TPARG_FL_WPROBE)) {
+ /* eprobe, fprobe and wprobe do not handle registers */
trace_probe_log_err(ctx->offset, BAD_VAR);
return -EINVAL;
}
diff --git a/kernel/trace/trace_probe.h b/kernel/trace/trace_probe.h
index ebdc706e7cb6..7380502a85af 100644
--- a/kernel/trace/trace_probe.h
+++ b/kernel/trace/trace_probe.h
@@ -91,6 +91,7 @@ typedef int (*print_type_func_t)(struct trace_seq *, void *, void *);
FETCH_OP(STACK, param), /* Stack: .param = index */ \
FETCH_OP(STACKP, none), /* Stack pointer */ \
FETCH_OP(RETVAL, none), /* Return value */ \
+ FETCH_OP(BADDR, none), /* Break address */ \
FETCH_OP(IMM, imm), /* Immediate: .immediate */ \
FETCH_OP(COMM, none), /* Current comm */ \
FETCH_OP(CURRENT, none), /* Current task_struct address */\
@@ -420,6 +421,7 @@ static inline int traceprobe_get_entry_data_size(struct trace_probe *tp)
#define TPARG_FL_USER BIT(4)
#define TPARG_FL_FPROBE BIT(5)
#define TPARG_FL_TPOINT BIT(6)
+#define TPARG_FL_WPROBE BIT(7)
#define TPARG_FL_LOC_MASK GENMASK(4, 0)
static inline bool tparg_is_function_entry(unsigned int flags)
@@ -546,6 +548,10 @@ extern int traceprobe_define_arg_fields(struct trace_event_call *event_call,
C(ARG_TOO_LONG, "Argument expression is too long"), \
C(ARRAY_NO_CLOSE, "Array is not closed"), \
C(ARRAY_TOO_BIG, "Array number is too big"), \
+ C(BAD_ACCESS_ADDR, "Invalid access memory address"), \
+ C(BAD_ACCESS_FMT, "Access memory address requires @"), \
+ C(BAD_ACCESS_LEN, "This memory access length is not supported"), \
+ C(BAD_ACCESS_TYPE, "Bad memory access type"), \
C(BAD_ADDR_SUFFIX, "Invalid probed address suffix"), \
C(BAD_ARG_NAME, "Argument name must follow the same rules as C identifiers"), \
C(BAD_ARG_NUM, "Invalid argument number"), \
@@ -628,7 +634,8 @@ extern int traceprobe_define_arg_fields(struct trace_event_call *event_call,
C(TYPECAST_NOT_EVENT, "Typecasts are only for eprobe fields"), \
C(TYPECAST_REQ_FIELD, "Typecast requires a field access"), \
C(TYPECAST_SYM_OFFSET, "@SYM+/-OFFSET with typecast needs parentheses"), \
- C(USED_ARG_NAME, "This argument name is already used"),
+ C(USED_ARG_NAME, "This argument name is already used"), \
+ C(WPROBE_NO_SIBLING, "Watchpoint probe does not support sibling probes"),
#undef C
#define C(a, b) TP_ERR_##a
diff --git a/kernel/trace/trace_wprobe.c b/kernel/trace/trace_wprobe.c
new file mode 100644
index 000000000000..4c23c310a383
--- /dev/null
+++ b/kernel/trace/trace_wprobe.c
@@ -0,0 +1,746 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Hardware-breakpoint-based tracing events
+ *
+ * Copyright (C) 2023, Masami Hiramatsu <mhiramat@kernel.org>
+ */
+#define pr_fmt(fmt) "trace_wprobe: " fmt
+
+#include <linux/compiler.h>
+#include <linux/hw_breakpoint.h>
+#include <linux/kallsyms.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/perf_event.h>
+#include <linux/rculist.h>
+#include <linux/security.h>
+#include <linux/tracepoint.h>
+#include <linux/uaccess.h>
+
+#include <asm/ptrace.h>
+
+#include "trace_dynevent.h"
+#include "trace_probe.h"
+#include "trace_probe_kernel.h"
+#include "trace_probe_tmpl.h"
+#include "trace_output.h"
+
+#define WPROBE_EVENT_SYSTEM "wprobes"
+
+static int trace_wprobe_create(const char *raw_command);
+static int trace_wprobe_show(struct seq_file *m, struct dyn_event *ev);
+static int trace_wprobe_release(struct dyn_event *ev);
+static bool trace_wprobe_is_busy(struct dyn_event *ev);
+static bool trace_wprobe_match(const char *system, const char *event,
+ int argc, const char **argv, struct dyn_event *ev);
+
+static struct dyn_event_operations trace_wprobe_ops = {
+ .create = trace_wprobe_create,
+ .show = trace_wprobe_show,
+ .is_busy = trace_wprobe_is_busy,
+ .free = trace_wprobe_release,
+ .match = trace_wprobe_match,
+};
+
+struct trace_wprobe {
+ struct dyn_event devent;
+ struct perf_event * __percpu *bp_event;
+ unsigned long addr;
+ int len;
+ int type;
+ const char *symbol;
+ struct trace_probe tp;
+};
+
+static bool is_trace_wprobe(struct dyn_event *ev)
+{
+ return ev->ops == &trace_wprobe_ops;
+}
+
+static struct trace_wprobe *to_trace_wprobe(struct dyn_event *ev)
+{
+ return container_of(ev, struct trace_wprobe, devent);
+}
+
+#define for_each_trace_wprobe(pos, dpos) \
+ for_each_dyn_event(dpos) \
+ if (is_trace_wprobe(dpos) && (pos = to_trace_wprobe(dpos)))
+
+static bool trace_wprobe_is_busy(struct dyn_event *ev)
+{
+ struct trace_wprobe *tw = to_trace_wprobe(ev);
+
+ return trace_probe_is_enabled(&tw->tp);
+}
+
+static bool trace_wprobe_match(const char *system, const char *event,
+ int argc, const char **argv, struct dyn_event *ev)
+{
+ struct trace_wprobe *tw = to_trace_wprobe(ev);
+
+ if (event[0] != '\0' && strcmp(trace_probe_name(&tw->tp), event))
+ return false;
+
+ if (system && strcmp(trace_probe_group_name(&tw->tp), system))
+ return false;
+
+ return trace_probe_match_command_args(&tw->tp, argc, argv);
+}
+
+/*
+ * Note that we don't verify the fetch_insn code, since it does not come
+ * from user space.
+ */
+static int
+process_fetch_insn(struct fetch_insn *code, void *rec, void *edata,
+ void *dest, void *base)
+{
+ void *baddr = rec;
+ unsigned long val;
+ int ret;
+
+retry:
+ /* 1st stage: get value from context */
+ switch (code->op) {
+ case FETCH_OP_BADDR:
+ val = (unsigned long)baddr;
+ break;
+ case FETCH_NOP_SYMBOL: /* Ignore a place holder */
+ code++;
+ goto retry;
+ default:
+ ret = process_common_fetch_insn(code, &val);
+ if (ret < 0)
+ return ret;
+ }
+ code++;
+
+ return process_fetch_insn_bottom(code, val, dest, base);
+}
+NOKPROBE_SYMBOL(process_fetch_insn)
+
+static void wprobe_trace_handler(struct trace_wprobe *tw,
+ unsigned long addr,
+ struct pt_regs *regs,
+ struct trace_event_file *trace_file)
+{
+ struct wprobe_trace_entry_head *entry;
+ struct trace_event_call *call = trace_probe_event_call(&tw->tp);
+ struct trace_event_buffer fbuffer;
+ int dsize;
+
+ if (WARN_ON_ONCE(call != trace_file->event_call))
+ return;
+
+ if (trace_trigger_soft_disabled(trace_file))
+ return;
+
+ if (READ_ONCE(tw->addr) != addr)
+ return;
+
+ dsize = __get_data_size(&tw->tp, (void *)addr, NULL);
+
+ entry = trace_event_buffer_reserve(&fbuffer, trace_file,
+ sizeof(*entry) + tw->tp.size + dsize);
+ if (!entry)
+ return;
+
+ entry->ip = instruction_pointer(regs);
+ store_trace_args(&entry[1], &tw->tp, (void *)addr, NULL, sizeof(*entry), dsize);
+
+ fbuffer.regs = regs;
+ trace_event_buffer_commit(&fbuffer);
+}
+
+static void wprobe_perf_handler(struct perf_event *bp,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ struct trace_wprobe *tw = bp->overflow_handler_context;
+ struct event_file_link *link;
+ unsigned long addr = bp->attr.bp_addr;
+
+ trace_probe_for_each_link_rcu(link, &tw->tp)
+ wprobe_trace_handler(tw, addr, regs, link->file);
+}
+
+static int __register_trace_wprobe(struct trace_wprobe *tw)
+{
+ struct perf_event_attr attr;
+ int i, ret;
+
+ if (tw->bp_event)
+ return -EINVAL;
+
+ for (i = 0; i < tw->tp.nr_args; i++) {
+ ret = traceprobe_update_arg(&tw->tp.args[i]);
+ if (ret)
+ return ret;
+ }
+
+ hw_breakpoint_init(&attr);
+ attr.bp_addr = tw->addr;
+ attr.bp_len = tw->len;
+ attr.bp_type = tw->type;
+
+ tw->bp_event = register_wide_hw_breakpoint(&attr, wprobe_perf_handler, tw);
+ if (IS_ERR_PCPU(tw->bp_event)) {
+ int ret = PTR_ERR_PCPU(tw->bp_event);
+
+ tw->bp_event = NULL;
+ return ret;
+ }
+
+ return 0;
+}
+
+static void __unregister_trace_wprobe(struct trace_wprobe *tw)
+{
+ if (tw->bp_event) {
+ unregister_wide_hw_breakpoint(tw->bp_event);
+ tw->bp_event = NULL;
+ }
+}
+
+static void free_trace_wprobe(struct trace_wprobe *tw)
+{
+ if (tw) {
+ trace_probe_cleanup(&tw->tp);
+ kfree(tw->symbol);
+ kfree(tw);
+ }
+}
+DEFINE_FREE(free_trace_wprobe, struct trace_wprobe *,
+ if (!IS_ERR_OR_NULL(_T))
+ free_trace_wprobe(_T))
+
+
+static struct trace_wprobe *alloc_trace_wprobe(const char *group,
+ const char *event,
+ const char *symbol,
+ unsigned long addr,
+ int len, int type, int nargs)
+{
+ struct trace_wprobe *tw __free(free_trace_wprobe) = NULL;
+ int ret;
+
+ tw = kzalloc_flex(*tw, tp.args, nargs);
+ if (!tw)
+ return ERR_PTR(-ENOMEM);
+
+ if (symbol) {
+ tw->symbol = kstrdup(symbol, GFP_KERNEL);
+ if (!tw->symbol)
+ return ERR_PTR(-ENOMEM);
+ }
+ tw->addr = addr;
+ tw->len = len;
+ tw->type = type;
+
+ ret = trace_probe_init(&tw->tp, event, group, false, nargs);
+ if (ret < 0)
+ return ERR_PTR(ret);
+
+ dyn_event_init(&tw->devent, &trace_wprobe_ops);
+ return_ptr(tw);
+}
+
+static struct trace_wprobe *find_trace_wprobe(const char *event,
+ const char *group)
+{
+ struct dyn_event *pos;
+ struct trace_wprobe *tw;
+
+ for_each_trace_wprobe(tw, pos)
+ if (strcmp(trace_probe_name(&tw->tp), event) == 0 &&
+ strcmp(trace_probe_group_name(&tw->tp), group) == 0)
+ return tw;
+ return NULL;
+}
+
+static enum print_line_t
+print_wprobe_event(struct trace_iterator *iter, int flags,
+ struct trace_event *event)
+{
+ struct wprobe_trace_entry_head *field;
+ struct trace_seq *s = &iter->seq;
+ struct trace_probe *tp;
+
+ field = (struct wprobe_trace_entry_head *)iter->ent;
+ tp = trace_probe_primary_from_call(
+ container_of(event, struct trace_event_call, event));
+ if (WARN_ON_ONCE(!tp))
+ goto out;
+
+ trace_seq_printf(s, "%s: (", trace_probe_name(tp));
+
+ if (!seq_print_ip_sym_offset(s, field->ip, flags))
+ goto out;
+
+ trace_seq_putc(s, ')');
+
+ if (trace_probe_print_args(s, tp->args, tp->nr_args,
+ (u8 *)&field[1], field) < 0)
+ goto out;
+
+ trace_seq_putc(s, '\n');
+out:
+ return trace_handle_return(s);
+}
+
+static int wprobe_event_define_fields(struct trace_event_call *event_call)
+{
+ int ret;
+ struct wprobe_trace_entry_head field;
+ struct trace_probe *tp;
+
+ tp = trace_probe_primary_from_call(event_call);
+ if (WARN_ON_ONCE(!tp))
+ return -ENOENT;
+
+ DEFINE_FIELD(unsigned long, ip, FIELD_STRING_IP, 0);
+
+ return traceprobe_define_arg_fields(event_call, sizeof(field), tp);
+}
+
+static struct trace_event_functions wprobe_funcs = {
+ .trace = print_wprobe_event
+};
+
+static struct trace_event_fields wprobe_fields_array[] = {
+ { .type = TRACE_FUNCTION_TYPE,
+ .define_fields = wprobe_event_define_fields },
+ {}
+};
+
+static int wprobe_register(struct trace_event_call *event,
+ enum trace_reg type, void *data);
+
+static inline void init_trace_event_call(struct trace_wprobe *tw)
+{
+ struct trace_event_call *call = trace_probe_event_call(&tw->tp);
+
+ call->event.funcs = &wprobe_funcs;
+ call->class->fields_array = wprobe_fields_array;
+ call->flags = TRACE_EVENT_FL_WPROBE;
+ call->class->reg = wprobe_register;
+}
+
+static int register_wprobe_event(struct trace_wprobe *tw)
+{
+ init_trace_event_call(tw);
+ return trace_probe_register_event_call(&tw->tp);
+}
+
+static int register_trace_wprobe_event(struct trace_wprobe *tw)
+{
+ struct trace_wprobe *old_tw;
+ int ret;
+
+ guard(mutex)(&event_mutex);
+
+ old_tw = find_trace_wprobe(trace_probe_name(&tw->tp),
+ trace_probe_group_name(&tw->tp));
+ if (old_tw) {
+ /*
+ * Wprobe does not support sibling probes because the event
+ * trigger (set_wprobe/clear_wprobe) identifies the target
+ * wprobe by its event name. Having multiple wprobes sharing
+ * the same event name would make the target ambiguous.
+ */
+ trace_probe_log_set_index(0);
+ trace_probe_log_err(0, WPROBE_NO_SIBLING);
+ return -EBUSY;
+ }
+
+ ret = register_wprobe_event(tw);
+ if (ret)
+ return ret;
+
+ dyn_event_add(&tw->devent, trace_probe_event_call(&tw->tp));
+ return 0;
+}
+static int unregister_wprobe_event(struct trace_wprobe *tw)
+{
+ return trace_probe_unregister_event_call(&tw->tp);
+}
+
+static int unregister_trace_wprobe(struct trace_wprobe *tw)
+{
+ if (trace_probe_has_sibling(&tw->tp))
+ goto unreg;
+
+ if (trace_probe_is_enabled(&tw->tp))
+ return -EBUSY;
+
+ if (trace_event_dyn_busy(trace_probe_event_call(&tw->tp)))
+ return -EBUSY;
+
+ if (unregister_wprobe_event(tw))
+ return -EBUSY;
+
+unreg:
+ __unregister_trace_wprobe(tw);
+ dyn_event_remove(&tw->devent);
+ trace_probe_unlink(&tw->tp);
+
+ return 0;
+}
+
+static int enable_trace_wprobe(struct trace_event_call *call,
+ struct trace_event_file *file)
+{
+ struct trace_probe *tp;
+ struct trace_wprobe *tw;
+ bool enabled;
+ int ret = 0;
+
+ tp = trace_probe_primary_from_call(call);
+ if (WARN_ON_ONCE(!tp))
+ return -ENODEV;
+ enabled = trace_probe_is_enabled(tp);
+
+ if (file) {
+ ret = trace_probe_add_file(tp, file);
+ if (ret)
+ return ret;
+ } else {
+ trace_probe_set_flag(tp, TP_FLAG_PROFILE);
+ }
+
+ if (!enabled) {
+ list_for_each_entry(tw, trace_probe_probe_list(tp), tp.list) {
+ ret = __register_trace_wprobe(tw);
+ if (ret < 0) {
+ struct trace_wprobe *tmp;
+
+ list_for_each_entry(tmp, trace_probe_probe_list(tp), tp.list) {
+ if (tmp == tw)
+ break;
+ __unregister_trace_wprobe(tmp);
+ }
+ if (file)
+ trace_probe_remove_file(tp, file);
+ else
+ trace_probe_clear_flag(tp, TP_FLAG_PROFILE);
+ return ret;
+ }
+ }
+ }
+
+ return 0;
+}
+
+static int disable_trace_wprobe(struct trace_event_call *call,
+ struct trace_event_file *file)
+{
+ struct trace_wprobe *tw;
+ struct trace_probe *tp;
+
+ tp = trace_probe_primary_from_call(call);
+ if (WARN_ON_ONCE(!tp))
+ return -ENODEV;
+
+ if (file) {
+ if (!trace_probe_get_file_link(tp, file))
+ return -ENOENT;
+ if (!trace_probe_has_single_file(tp))
+ goto out;
+ trace_probe_clear_flag(tp, TP_FLAG_TRACE);
+ } else {
+ trace_probe_clear_flag(tp, TP_FLAG_PROFILE);
+ }
+
+ if (!trace_probe_is_enabled(tp)) {
+ list_for_each_entry(tw, trace_probe_probe_list(tp), tp.list) {
+ __unregister_trace_wprobe(tw);
+ }
+ }
+
+out:
+ if (file)
+ trace_probe_remove_file(tp, file);
+
+ return 0;
+}
+
+static int wprobe_register(struct trace_event_call *event,
+ enum trace_reg type, void *data)
+{
+ struct trace_event_file *file = data;
+
+ switch (type) {
+ case TRACE_REG_REGISTER:
+ return enable_trace_wprobe(event, file);
+ case TRACE_REG_UNREGISTER:
+ return disable_trace_wprobe(event, file);
+
+#ifdef CONFIG_PERF_EVENTS
+ case TRACE_REG_PERF_REGISTER:
+ case TRACE_REG_PERF_UNREGISTER:
+ case TRACE_REG_PERF_OPEN:
+ case TRACE_REG_PERF_CLOSE:
+ case TRACE_REG_PERF_ADD:
+ case TRACE_REG_PERF_DEL:
+ return -EOPNOTSUPP;
+#endif
+ }
+ return 0;
+}
+
+static int parse_address_spec(const char *spec, unsigned long *addr, int *type,
+ int *len, char **symbol)
+{
+ char *_spec __free(kfree) = NULL;
+ int _len = HW_BREAKPOINT_LEN_4;
+ int _type = HW_BREAKPOINT_RW;
+ unsigned long _addr = 0;
+ char *at, *col;
+
+ _spec = kstrdup(spec, GFP_KERNEL);
+ if (!_spec)
+ return -ENOMEM;
+
+ at = strchr(_spec, '@');
+ col = strchr(_spec, ':');
+
+ if (!at) {
+ trace_probe_log_err(0, BAD_ACCESS_FMT);
+ return -EINVAL;
+ }
+
+ if (at != _spec) {
+ *at = '\0';
+
+ if (strcmp(_spec, "r") == 0)
+ _type = HW_BREAKPOINT_R;
+ else if (strcmp(_spec, "w") == 0)
+ _type = HW_BREAKPOINT_W;
+ else if (strcmp(_spec, "rw") == 0)
+ _type = HW_BREAKPOINT_RW;
+ else {
+ trace_probe_log_err(0, BAD_ACCESS_TYPE);
+ return -EINVAL;
+ }
+ }
+
+ if (col) {
+ *col = '\0';
+ if (kstrtoint(col + 1, 0, &_len)) {
+ trace_probe_log_err(col + 1 - _spec, BAD_ACCESS_LEN);
+ return -EINVAL;
+ }
+
+ switch (_len) {
+ case 1:
+ _len = HW_BREAKPOINT_LEN_1;
+ break;
+ case 2:
+ _len = HW_BREAKPOINT_LEN_2;
+ break;
+ case 4:
+ _len = HW_BREAKPOINT_LEN_4;
+ break;
+ case 8:
+ _len = HW_BREAKPOINT_LEN_8;
+ break;
+ default:
+ trace_probe_log_err(col + 1 - _spec, BAD_ACCESS_LEN);
+ return -EINVAL;
+ }
+ }
+
+ if (kstrtoul(at + 1, 0, &_addr) != 0) {
+ char *off_str = strpbrk(at + 1, "+-");
+ int offset = 0;
+
+ if (off_str) {
+ if (kstrtoint(off_str, 0, &offset) != 0) {
+ trace_probe_log_err(off_str - _spec, BAD_PROBE_ADDR);
+ return -EINVAL;
+ }
+ *off_str = '\0';
+ }
+ _addr = kallsyms_lookup_name(at + 1);
+ if (!_addr) {
+ trace_probe_log_err(at + 1 - _spec, BAD_ACCESS_ADDR);
+ return -ENOENT;
+ }
+ _addr += offset;
+ *symbol = kstrdup(at + 1, GFP_KERNEL);
+ if (!*symbol)
+ return -ENOMEM;
+ }
+
+ *addr = _addr;
+ *type = _type;
+ *len = _len;
+ return 0;
+}
+
+static int __trace_wprobe_create(int argc, const char *argv[])
+{
+ /*
+ * Argument syntax:
+ * b[:[GRP/][EVENT]] SPEC
+ *
+ * SPEC:
+ * [r|w|rw]@[ADDR|SYMBOL[+OFFS]][:LEN]
+ */
+ struct traceprobe_parse_context *ctx __free(traceprobe_parse_context) = NULL;
+ struct trace_wprobe *tw __free(free_trace_wprobe) = NULL;
+ const char *event = NULL, *group = WPROBE_EVENT_SYSTEM;
+ const char *tplog __free(trace_probe_log_clear) = NULL;
+ char *symbol __free(kfree) = NULL;
+ char *gbuf __free(kfree) = NULL;
+ char *ebuf __free(kfree) = NULL;
+ unsigned long addr;
+ int len, type, i;
+ int ret = 0;
+
+ if (argv[0][0] != 'w')
+ return -ECANCELED;
+
+ if (argc < 2)
+ return -EINVAL;
+
+ tplog = trace_probe_log_init("wprobe", argc, argv);
+
+ if (argv[0][1] != '\0') {
+ if (argv[0][1] != ':') {
+ trace_probe_log_set_index(0);
+ trace_probe_log_err(1, BAD_MAXACT_TYPE);
+ return -EINVAL;
+ }
+ event = &argv[0][2];
+ }
+
+ trace_probe_log_set_index(1);
+ ret = parse_address_spec(argv[1], &addr, &type, &len, &symbol);
+ if (ret < 0)
+ return ret;
+
+ trace_probe_log_set_index(0);
+ if (event) {
+ gbuf = kmalloc(MAX_EVENT_NAME_LEN, GFP_KERNEL);
+ if (!gbuf)
+ return -ENOMEM;
+ ret = traceprobe_parse_event_name(&event, &group, gbuf,
+ event - argv[0]);
+ if (ret)
+ return ret;
+ }
+
+ if (!event) {
+ /* Make a new event name */
+ ebuf = kmalloc(MAX_EVENT_NAME_LEN, GFP_KERNEL);
+ if (!ebuf)
+ return -ENOMEM;
+ if (symbol)
+ snprintf(ebuf, MAX_EVENT_NAME_LEN, "%s", symbol);
+ else
+ snprintf(ebuf, MAX_EVENT_NAME_LEN, "w_0x%lx", addr);
+ sanitize_event_name(ebuf);
+ event = ebuf;
+ }
+
+ argc -= 2; argv += 2;
+ tw = alloc_trace_wprobe(group, event, symbol, addr, len, type, argc);
+ if (IS_ERR(tw))
+ return PTR_ERR(tw);
+
+ ctx = kzalloc_obj(*ctx);
+ if (!ctx)
+ return -ENOMEM;
+
+ ctx->flags = TPARG_FL_KERNEL | TPARG_FL_WPROBE;
+
+ /* parse arguments */
+ for (i = 0; i < argc; i++) {
+ trace_probe_log_set_index(i + 2);
+ ctx->offset = 0;
+ ret = traceprobe_parse_probe_arg(&tw->tp, i, argv[i], ctx);
+ if (ret)
+ return ret; /* This can be -ENOMEM */
+ }
+
+ ret = traceprobe_set_print_fmt(&tw->tp, PROBE_PRINT_NORMAL);
+ if (ret < 0)
+ return ret;
+
+ ret = register_trace_wprobe_event(tw);
+ if (!ret)
+ tw = NULL; /* To avoid free */
+
+ return ret;
+}
+
+static int trace_wprobe_create(const char *raw_command)
+{
+ return trace_probe_create(raw_command, __trace_wprobe_create);
+}
+
+static int trace_wprobe_release(struct dyn_event *ev)
+{
+ struct trace_wprobe *tw = to_trace_wprobe(ev);
+ int ret = unregister_trace_wprobe(tw);
+
+ if (!ret)
+ free_trace_wprobe(tw);
+ return ret;
+}
+
+static int trace_wprobe_show(struct seq_file *m, struct dyn_event *ev)
+{
+ struct trace_wprobe *tw = to_trace_wprobe(ev);
+ int i;
+
+ seq_printf(m, "w:%s/%s", trace_probe_group_name(&tw->tp),
+ trace_probe_name(&tw->tp));
+
+ const char *type_str;
+
+ if (tw->type == HW_BREAKPOINT_R)
+ type_str = "r";
+ else if (tw->type == HW_BREAKPOINT_W)
+ type_str = "w";
+ else
+ type_str = "rw";
+
+ int len;
+
+ if (tw->len == HW_BREAKPOINT_LEN_1)
+ len = 1;
+ else if (tw->len == HW_BREAKPOINT_LEN_2)
+ len = 2;
+ else if (tw->len == HW_BREAKPOINT_LEN_4)
+ len = 4;
+ else
+ len = 8;
+
+ if (tw->symbol) {
+ unsigned long sym_addr = kallsyms_lookup_name(tw->symbol);
+ long offset = sym_addr ? (long)(tw->addr - sym_addr) : 0;
+
+ if (offset)
+ seq_printf(m, " %s@%s%+ld:%d", type_str, tw->symbol, offset, len);
+ else
+ seq_printf(m, " %s@%s:%d", type_str, tw->symbol, len);
+ } else {
+ seq_printf(m, " %s@0x%lx:%d", type_str, tw->addr, len);
+ }
+
+ for (i = 0; i < tw->tp.nr_args; i++)
+ seq_printf(m, " %s=%s", tw->tp.args[i].name, tw->tp.args[i].comm);
+ seq_putc(m, '\n');
+
+ return 0;
+}
+
+static __init int init_wprobe_trace(void)
+{
+ return dyn_event_register(&trace_wprobe_ops);
+}
+fs_initcall(init_wprobe_trace);
+
^ permalink raw reply related
* [PATCH v10 00/11] tracing: wprobe: x86: Add wprobe for watchpoint
From: Masami Hiramatsu (Google) @ 2026-07-22 23:02 UTC (permalink / raw)
To: Steven Rostedt, Peter Zijlstra, Ingo Molnar, x86
Cc: Jinchao Wang, Mathieu Desnoyers, Masami Hiramatsu,
Thomas Gleixner, Borislav Petkov, Dave Hansen, H . Peter Anvin,
Alexander Shishkin, Ian Rogers, linux-kernel, linux-trace-kernel,
linux-doc, linux-perf-users
Hi,
Here is the 10th version of the series for adding new wprobe (watch probe)
which provides memory access tracing event. Moreover, this can be used
via event trigger. Thus it can trace memory access on a dynamically
allocated objects too.
The previous version is here:
https://lore.kernel.org/all/178429796992.157981.3393977217853767915.stgit@devnote2/
This version add BTF support in wprobe trigger command[11/11]. With this
feature, user do not need to decode the offset of structure fields.
(Since the previous version lacked base-commit tag, Sashiko failed to
apply it for review. Thus this is just add a BTF support.)
To support kprobe events, we need to check the previous NMI context
or need to use an atomic refcount etc. But that should be another
patch for review.
To support arm64, we need to avoid major pagefault on single-stepping
issue. I think we can solve it with checking whether the address is the
kernel (which must not cause a major page fault), or not.
Public branch
-------------
I will push this branch as topic/wprobe-v2 to my tree so that it
can be easily tested.
https://git.kernel.org/pub/scm/linux/kernel/git/mhiramat/linux.git/log/?h=topic/wprobe-v2
This is based on linux-trace tree's probes/for-next.
Usage
-----
The basic usage of this wprobe is similar to other probes;
w:[GRP/][EVENT] [r|w|rw]@<ADDRESS|SYMBOL[+OFFS]> [FETCHARGS]
This defines a new wprobe event. For example, to trace jiffies update,
you can do;
echo 'w:my_jiffies w@jiffies:8 value=+0($addr)' >> dynamic_events
echo 1 > events/wprobes/my_jiffies/enable
Moreover, this can be combined with event trigger to trace the memory
accecss on slab objects. The trigger syntax is;
set_wprobe:WPROBE_EVENT:FIELD[+OFFSET] [if FILTER]
clear_wprobe:WPROBE_EVENT[:FIELD[+OFFSET]] [if FILTER]
set_wprobe sets WPROBE_EVENT's watch address on FIELD[+OFFSET].
clear_wprobe clears WPROBE_EVENT's watch address if it is set to
FIELD[+OFFSET]. If FIELD is omitted, forcibly clear the watch address
when trigger event is hit.
For example, trace the first 8 byte of the dentry data structure passed
to do_truncate() until it is deleted by dentry_kill().
(Note: all tracefs setup uses '>>' so that it does not kick do_truncate())
# echo 'w:watch rw@0:8 address=$addr value=+0($addr)' > dynamic_events
# echo 'f:truncate do_truncate dentry=$arg2' >> dynamic_events
# echo 'set_wprobe:watch:dentry' >> events/fprobes/truncate/trigger
# echo 'f:dentry_kill dentry_kill dentry=$arg1' >> dynamic_events
# echo 'clear_wprobe:watch:dentry' >> events/fprobes/dentry_kill/trigger
# echo 1 >> events/fprobes/truncate/enable
# echo 1 >> events/fprobes/dentry_kill/enable
# echo aaa > /tmp/hoge
# echo bbb > /tmp/hoge
# echo ccc > /tmp/hoge
# rm /tmp/hoge
Then, the trace data will show;
# tracer: nop
#
# entries-in-buffer/entries-written: 32/32 #P:8
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
sh-107 [004] ...1. 9.990418: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618
sh-107 [004] ...1. 9.990914: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b3de78
sh-107 [004] ...1. 9.993175: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049ddd40
sh-107 [004] ..... 9.995198: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880048083a8
sh-107 [004] ...1. 9.995389: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db998
sh-107 [004] ..Zff 9.997503: watch: (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080
sh-107 [004] ..Zff 9.997509: watch: (path_openat+0x211/0xda0) address=0xffff8880048083a8 value=0x8200080
sh-107 [004] ..Zff 9.997514: watch: (path_openat+0xa56/0xda0) address=0xffff8880048083a8 value=0x8200080
sh-107 [004] ..Zff 9.997518: watch: (path_openat+0xae2/0xda0) address=0xffff8880048083a8 value=0x8200080
sh-107 [004] ..... 9.997521: truncate: (do_truncate+0x4/0x120) dentry=0xffff8880048083a8
sh-107 [004] ...1. 9.997582: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004808270
sh-107 [004] ...1. 9.999365: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db728
sh-107 [004] ...1. 9.999388: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b1c000
rm-113 [005] ..Zff 10.000965: watch: (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.000971: watch: (path_lookupat+0x97/0x1e0) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.000984: watch: (lookup_fast+0xaa/0x150) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.000988: watch: (path_lookupat+0x97/0x1e0) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.001010: watch: (lookup_one_qstr_excl+0x28/0x140) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.001014: watch: (lookup_one_qstr_excl+0xd1/0x140) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.001018: watch: (may_delete_dentry+0x1c/0x200) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.001021: watch: (may_delete_dentry+0x195/0x200) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] ..Zff 10.001031: watch: (vfs_unlink+0x5e/0x260) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] d.Z.. 10.001067: watch: (d_make_discardable+0x1b/0x40) address=0xffff8880048083a8 value=0x8200080
rm-113 [005] d.Z.. 10.001071: watch: (d_make_discardable+0x29/0x40) address=0xffff8880048083a8 value=0x200080
rm-113 [005] ...1. 10.001072: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880048083a8
rm-113 [005] ...1. 10.001218: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880048083a8
sh-107 [004] ...1. 10.001416: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db110
sh-107 [004] ...1. 10.001444: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff8880049db248
sh-107 [004] ...1. 10.001500: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618
sh-107 [004] ...1. 10.002067: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b41e78
sh-107 [004] ...1. 10.904920: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004b41e78
sh-107 [004] ...1. 10.905129: dentry_kill: (dentry_kill+0x0/0x2c0) dentry=0xffff888004ad6618
Thank you,
---
base-commit: 1a416ae446afa42d2d8500ce25bd61c564508721
Jinchao Wang (2):
x86/hw_breakpoint: Unify breakpoint install/uninstall
x86/hw_breakpoint: Add arch_reinstall_hw_breakpoint
Masami Hiramatsu (Google) (9):
tracing: wprobe: Add watchpoint probe event based on hardware breakpoint
x86: hw_breakpoint: Add a kconfig to clarify when a breakpoint fires
selftests: tracing: Add a basic testcase for wprobe
selftests: tracing: Add syntax testcase for wprobe
HWBP: Add modify_wide_hw_breakpoint_local() API
tracing: wprobe: Add wprobe event trigger
selftests: ftrace: Add wprobe trigger testcase
tracing/wprobe: Support BTF typecast in fetchargs
tracing/wprobe: Support BTF typecast in wprobe trigger command
Documentation/trace/index.rst | 1
Documentation/trace/wprobetrace.rst | 163 +++
arch/Kconfig | 20
arch/x86/Kconfig | 2
arch/x86/include/asm/hw_breakpoint.h | 8
arch/x86/kernel/hw_breakpoint.c | 164 ++-
include/linux/hw_breakpoint.h | 6
include/linux/trace_events.h | 3
kernel/events/hw_breakpoint.c | 61 +
kernel/trace/Kconfig | 24
kernel/trace/Makefile | 1
kernel/trace/trace.c | 9
kernel/trace/trace.h | 7
kernel/trace/trace_events_trigger.c | 12
kernel/trace/trace_probe.c | 24
kernel/trace/trace_probe.h | 14
kernel/trace/trace_wprobe.c | 1300 ++++++++++++++++++++
tools/testing/selftests/ftrace/config | 2
.../ftrace/test.d/dynevent/add_remove_wprobe.tc | 63 +
.../test.d/dynevent/wprobes_syntax_errors.tc | 20
.../test.d/trigger/trigger-wprobe-btf-offset.tc | 73 +
.../test.d/trigger/trigger-wprobe-btf-typecast.tc | 80 +
.../ftrace/test.d/trigger/trigger-wprobe.tc | 70 +
23 files changed, 2056 insertions(+), 71 deletions(-)
create mode 100644 Documentation/trace/wprobetrace.rst
create mode 100644 kernel/trace/trace_wprobe.c
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/add_remove_wprobe.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/dynevent/wprobes_syntax_errors.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-offset.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe-btf-typecast.tc
create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-wprobe.tc
--
Masami Hiramatsu (Google) <mhiramat@kernel.org>
^ permalink raw reply
* Re: [PATCH v10 0/8] mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings
From: Andrew Morton @ 2026-07-22 22:48 UTC (permalink / raw)
To: Stanislav Kinsburskii
Cc: Jason Gunthorpe, Leon Romanovsky, David Hildenbrand,
Lorenzo Stoakes, Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, Michal Hocko, Jonathan Corbet, Shuah Khan,
Shuah Khan, K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui,
Long Li, Lyude Paul, Danilo Krummrich, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, David Airlie, Simona Vetter,
Min Ma, Lizhi Hou, Oded Gabbay, linux-mm, linux-kernel, linux-doc,
linux-kselftest, linux-hyperv, dri-devel, nouveau, linux-rdma
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
On Wed, 22 Jul 2026 14:44:22 -0700 Stanislav Kinsburskii <skinsburskii@gmail.com> wrote:
> This series extends the HMM framework to support userfaultfd-backed memory
> by allowing the mmap read lock to be dropped during hmm_range_fault().
Thanks, I've updated mm.git's mm-unstable branch to this version.
Sashiko pointed at a few things, some pre-existing. The jiffies race
in [8/8] looks legit.
https://sashiko.dev/#/patchset/20260722-hmm-v10-v1-0-606464dd601a@gmail.com
> Changes in v10:
> - Included contended mmap_lock acquisition in the
> hmm_range_fault_unlocked_timeout() retry budget.
> - Dropped the redundant top-level fatal_signal_pending() check in the HMM
> unlocked retry loop; mmap_read_lock_killable() now covers that path.
> - Restored the absolute outer timeout in drm_gpusvm_get_pages(), since it can
> run from GPU page-fault workers and must not rely on the worker task’s fatal
> signal state to stop invalidation retries.
Here's how v10 altered mm.git:
drivers/gpu/drm/drm_gpusvm.c | 11 +++++++++--
mm/hmm.c | 29 +++++++++++++----------------
2 files changed, 22 insertions(+), 18 deletions(-)
--- a/drivers/gpu/drm/drm_gpusvm.c~b
+++ a/drivers/gpu/drm/drm_gpusvm.c
@@ -1405,7 +1405,9 @@ int drm_gpusvm_get_pages(struct drm_gpus
.dev_private_owner = ctx->device_private_page_owner,
};
void *zdd;
- unsigned long timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ unsigned long timeout =
+ jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ unsigned long remaining;
unsigned long i, j;
unsigned long npages = npages_in_range(pages_start, pages_end);
unsigned long num_dma_mapped;
@@ -1420,6 +1422,11 @@ int drm_gpusvm_get_pages(struct drm_gpus
struct dma_iova_state *state = &svm_pages->state;
retry:
+ if (time_after_eq(jiffies, timeout))
+ return -EBUSY;
+
+ remaining = timeout - jiffies;
+
hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
if (drm_gpusvm_pages_valid_unlocked(gpusvm, svm_pages))
goto set_seqno;
@@ -1434,7 +1441,7 @@ retry:
}
hmm_range.hmm_pfns = pfns;
- err = hmm_range_fault_unlocked_timeout(&hmm_range, timeout);
+ err = hmm_range_fault_unlocked_timeout(&hmm_range, remaining);
mmput(mm);
if (err)
goto err_free;
--- a/mm/hmm.c~b
+++ a/mm/hmm.c
@@ -790,22 +790,14 @@ int hmm_range_fault_unlocked_timeout(str
int ret;
do {
- if (fatal_signal_pending(current))
- return -EINTR;
-
- if (timeout) {
- /*
- * If the previous fault dropped mmap_lock, then the fault
- * handler made progress. Restart the retry timeout in that
- * case, but keep the existing deadline for ordinary -EBUSY
- * retries.
- */
- if (!locked)
- deadline = jiffies + timeout;
-
- if (time_after(jiffies, deadline))
- return -EBUSY;
- }
+ /*
+ * If the previous fault dropped mmap_lock, then the fault
+ * handler made progress. Restart the retry timeout in that
+ * case, but keep the existing deadline for ordinary -EBUSY
+ * retries.
+ */
+ if (timeout && !locked)
+ deadline = jiffies + timeout;
range->notifier_seq =
mmu_interval_read_begin(range->notifier);
@@ -814,6 +806,11 @@ int hmm_range_fault_unlocked_timeout(str
if (ret)
return ret;
+ if (timeout && time_after(jiffies, deadline)) {
+ mmap_read_unlock(mm);
+ return -EBUSY;
+ }
+
locked = true;
ret = hmm_range_fault_locked(range, &locked);
if (locked)
_
^ permalink raw reply
* [PATCH v10 8/8] drm/gpusvm: Use hmm_range_fault_unlocked_timeout() for range faults
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
Several GPU SVM paths take mmap_read_lock() only to call hmm_range_fault()
and open-code mmu interval sequence setup before each HMM walk. They also
retry -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires.
Use hmm_range_fault_unlocked_timeout() for those faults. The HMM helper now
owns mmap_lock acquisition and refreshes range->notifier_seq for its
internal retries, while GPU SVM keeps its existing driver-lock validation
with mmu_interval_read_retry() after a successful fault.
drm_gpusvm_scan_mm() and drm_gpusvm_range_evict() pass
HMM_RANGE_DEFAULT_TIMEOUT as the helper retry budget for each HMM fault
attempt. drm_gpusvm_get_pages() keeps its existing absolute outer deadline
because it can be reached from GPU page-fault workers, where fatal signals
from the faulting process cannot stop an endless invalidation retry loop.
It passes the remaining time from that deadline to HMM for each fault
attempt.
Leave drm_gpusvm_check_pages() on hmm_range_fault() because that path is
called with the mmap lock already held by its caller.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/gpu/drm/drm_gpusvm.c | 60 ++++++++------------------------------------
1 file changed, 10 insertions(+), 50 deletions(-)
diff --git a/drivers/gpu/drm/drm_gpusvm.c b/drivers/gpu/drm/drm_gpusvm.c
index 958cb605aedd..b946f920b7a0 100644
--- a/drivers/gpu/drm/drm_gpusvm.c
+++ b/drivers/gpu/drm/drm_gpusvm.c
@@ -773,8 +773,7 @@ enum drm_gpusvm_scan_result drm_gpusvm_scan_mm(struct drm_gpusvm_range *range,
.end = end,
.dev_private_owner = dev_private_owner,
};
- unsigned long timeout =
- jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ unsigned long timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
enum drm_gpusvm_scan_result state = DRM_GPUSVM_SCAN_UNPOPULATED, new_state;
unsigned long *pfns;
unsigned long npages = npages_in_range(start, end);
@@ -788,22 +787,7 @@ enum drm_gpusvm_scan_result drm_gpusvm_scan_mm(struct drm_gpusvm_range *range,
hmm_range.hmm_pfns = pfns;
retry:
- hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
- mmap_read_lock(range->gpusvm->mm);
-
- while (true) {
- err = hmm_range_fault(&hmm_range);
- if (err == -EBUSY) {
- if (time_after(jiffies, timeout))
- break;
-
- hmm_range.notifier_seq =
- mmu_interval_read_begin(notifier);
- continue;
- }
- break;
- }
- mmap_read_unlock(range->gpusvm->mm);
+ err = hmm_range_fault_unlocked_timeout(&hmm_range, timeout);
if (err)
goto err_free;
@@ -1408,6 +1392,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
void *zdd;
unsigned long timeout =
jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ unsigned long remaining;
unsigned long i, j;
unsigned long npages = npages_in_range(pages_start, pages_end);
unsigned long num_dma_mapped;
@@ -1422,9 +1407,11 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
struct dma_iova_state *state = &svm_pages->state;
retry:
- if (time_after(jiffies, timeout))
+ if (time_after_eq(jiffies, timeout))
return -EBUSY;
+ remaining = timeout - jiffies;
+
hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
if (drm_gpusvm_pages_valid_unlocked(gpusvm, svm_pages))
goto set_seqno;
@@ -1439,21 +1426,7 @@ int drm_gpusvm_get_pages(struct drm_gpusvm *gpusvm,
}
hmm_range.hmm_pfns = pfns;
- while (true) {
- mmap_read_lock(mm);
- err = hmm_range_fault(&hmm_range);
- mmap_read_unlock(mm);
-
- if (err == -EBUSY) {
- if (time_after(jiffies, timeout))
- break;
-
- hmm_range.notifier_seq =
- mmu_interval_read_begin(notifier);
- continue;
- }
- break;
- }
+ err = hmm_range_fault_unlocked_timeout(&hmm_range, remaining);
mmput(mm);
if (err)
goto err_free;
@@ -1720,8 +1693,7 @@ int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
.end = drm_gpusvm_range_end(range),
.dev_private_owner = NULL,
};
- unsigned long timeout =
- jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ unsigned long timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
unsigned long *pfns;
unsigned long npages = npages_in_range(drm_gpusvm_range_start(range),
drm_gpusvm_range_end(range));
@@ -1736,24 +1708,12 @@ int drm_gpusvm_range_evict(struct drm_gpusvm *gpusvm,
return -ENOMEM;
hmm_range.hmm_pfns = pfns;
- while (!time_after(jiffies, timeout)) {
- hmm_range.notifier_seq = mmu_interval_read_begin(notifier);
- if (time_after(jiffies, timeout)) {
- err = -ETIME;
- break;
- }
-
- mmap_read_lock(mm);
- err = hmm_range_fault(&hmm_range);
- mmap_read_unlock(mm);
- if (err != -EBUSY)
- break;
- }
+ err = hmm_range_fault_unlocked_timeout(&hmm_range, timeout);
kvfree(pfns);
mmput(mm);
- return err;
+ return err == -EBUSY ? -ETIME : err;
}
EXPORT_SYMBOL_GPL(drm_gpusvm_range_evict);
--
2.43.0
^ permalink raw reply related
* [PATCH v10 7/8] accel/amdxdna: Use hmm_range_fault_unlocked_timeout() for range population
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
aie2_populate_range() takes mmap_read_lock() only around hmm_range_fault().
It also open-codes the mmu interval sequence setup before each HMM walk and
retries -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT expires.
Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
the mmap lock and refreshes mapp->range.notifier_seq for its internal
retries, so the driver only needs to call the helper and then validate the
sequence before marking the mapping populated.
Pass HMM_RANGE_DEFAULT_TIMEOUT as the helper retry budget for each HMM
population attempt. This scopes the timeout to repeated HMM notifier
retries while preserving the existing outer loop that moves between invalid
mappings and restarts when the interval is invalidated before the driver
updates its mapping state.
Keep returning -ETIME when the HMM retry budget expires, matching the
driver's existing timeout error convention.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/accel/amdxdna/aie2_ctx.c | 23 ++++-------------------
1 file changed, 4 insertions(+), 19 deletions(-)
diff --git a/drivers/accel/amdxdna/aie2_ctx.c b/drivers/accel/amdxdna/aie2_ctx.c
index 54486960cbf5..b5b4ca263002 100644
--- a/drivers/accel/amdxdna/aie2_ctx.c
+++ b/drivers/accel/amdxdna/aie2_ctx.c
@@ -1034,7 +1034,7 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
bool found;
int ret;
- timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
again:
found = false;
down_write(&xdna->notifier_lock);
@@ -1061,24 +1061,9 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
return -EFAULT;
}
- mapp->range.notifier_seq = mmu_interval_read_begin(&mapp->notifier);
- mmap_read_lock(mm);
- ret = hmm_range_fault(&mapp->range);
- mmap_read_unlock(mm);
- if (ret) {
- if (time_after(jiffies, timeout)) {
- ret = -ETIME;
- goto put_mm;
- }
-
- if (ret == -EBUSY) {
- amdxdna_umap_put(mapp);
- mmput(mm);
- goto again;
- }
-
+ ret = hmm_range_fault_unlocked_timeout(&mapp->range, timeout);
+ if (ret)
goto put_mm;
- }
down_write(&xdna->notifier_lock);
if (mmu_interval_read_retry(&mapp->notifier, mapp->range.notifier_seq)) {
@@ -1096,7 +1081,7 @@ static int aie2_populate_range(struct amdxdna_gem_obj *abo)
put_mm:
amdxdna_umap_put(mapp);
mmput(mm);
- return ret;
+ return ret == -EBUSY ? -ETIME : ret;
}
int aie2_cmd_submit(struct amdxdna_hwctx *hwctx, struct amdxdna_sched_job *job, u64 *seq)
--
2.43.0
^ permalink raw reply related
* [PATCH v10 6/8] RDMA/umem: Use hmm_range_fault_unlocked_timeout() for ODP faults
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
ib_umem_odp_map_dma_and_lock() takes mmap_read_lock() only around
hmm_range_fault(), then retries -EBUSY until HMM_RANGE_DEFAULT_TIMEOUT
expires.
Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
the mmap lock and refreshes range->notifier_seq for its internal retries.
ODP keeps using HMM_RANGE_DEFAULT_TIMEOUT for each HMM fault attempt,
while interval invalidation retries continue to be handled by the existing
outer loop.
ODP still validates the interval notifier sequence while holding umem_mutex
before DMA mapping pages.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/infiniband/core/umem_odp.c | 18 +++++-------------
1 file changed, 5 insertions(+), 13 deletions(-)
diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c
index 404fa1cc3254..9cc21cd762d9 100644
--- a/drivers/infiniband/core/umem_odp.c
+++ b/drivers/infiniband/core/umem_odp.c
@@ -329,7 +329,7 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
struct mm_struct *owning_mm = umem_odp->umem.owning_mm;
int pfn_index, dma_index, ret = 0, start_idx;
unsigned int page_shift, hmm_order, pfn_start_idx;
- unsigned long num_pfns, current_seq;
+ unsigned long num_pfns;
struct hmm_range range = {};
unsigned long timeout;
@@ -363,26 +363,18 @@ int ib_umem_odp_map_dma_and_lock(struct ib_umem_odp *umem_odp, u64 user_virt,
}
range.hmm_pfns = &(umem_odp->map.pfn_list[pfn_start_idx]);
- timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
+ timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
retry:
- current_seq = range.notifier_seq =
- mmu_interval_read_begin(&umem_odp->notifier);
-
- mmap_read_lock(owning_mm);
- ret = hmm_range_fault(&range);
- mmap_read_unlock(owning_mm);
- if (unlikely(ret)) {
- if (ret == -EBUSY && !time_after(jiffies, timeout))
- goto retry;
+ ret = hmm_range_fault_unlocked_timeout(&range, timeout);
+ if (unlikely(ret))
goto out_put_mm;
- }
start_idx = (range.start - ib_umem_start(umem_odp)) >> page_shift;
dma_index = start_idx;
mutex_lock(&umem_odp->umem_mutex);
- if (mmu_interval_read_retry(&umem_odp->notifier, current_seq)) {
+ if (mmu_interval_read_retry(&umem_odp->notifier, range.notifier_seq)) {
mutex_unlock(&umem_odp->umem_mutex);
goto retry;
}
--
2.43.0
^ permalink raw reply related
* [PATCH v10 5/8] drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
nouveau_range_fault() takes mmap_read_lock() only to call
hmm_range_fault(). It also keeps a single HMM_RANGE_DEFAULT_TIMEOUT
deadline across both HMM -EBUSY retries and post-fault
mmu_interval_read_retry() retries.
Use hmm_range_fault_unlocked_timeout() instead. The HMM helper now owns
the mmap lock and refreshes range->notifier_seq for its internal retries.
Nouveau keeps its existing absolute deadline in the outer loop and passes
the remaining jiffies to the helper for each fault attempt, so retries
caused by mmu_interval_read_retry() do not reset the overall retry budget.
Nouveau still validates the interval notifier sequence while holding
svmm->mutex before programming the GPU mapping.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/gpu/drm/nouveau/nouveau_svm.c | 20 +++++++++++---------
1 file changed, 11 insertions(+), 9 deletions(-)
diff --git a/drivers/gpu/drm/nouveau/nouveau_svm.c b/drivers/gpu/drm/nouveau/nouveau_svm.c
index dcc92131488e..58735446d783 100644
--- a/drivers/gpu/drm/nouveau/nouveau_svm.c
+++ b/drivers/gpu/drm/nouveau/nouveau_svm.c
@@ -678,20 +678,22 @@ static int nouveau_range_fault(struct nouveau_svmm *svmm,
range.end = notifier->notifier.interval_tree.last + 1;
while (true) {
- if (time_after(jiffies, timeout)) {
+ long remaining = timeout - jiffies;
+
+ /*
+ * The HMM timeout only bounds retries while HMM is walking and
+ * faulting the range. This fault is handled by a kernel worker,
+ * so fatal signals from the faulting process cannot stop an
+ * endless stream of invalidations here.
+ */
+ if (time_after_eq(jiffies, timeout)) {
ret = -EBUSY;
goto out;
}
- range.notifier_seq = mmu_interval_read_begin(range.notifier);
- mmap_read_lock(mm);
- ret = hmm_range_fault(&range);
- mmap_read_unlock(mm);
- if (ret) {
- if (ret == -EBUSY)
- continue;
+ ret = hmm_range_fault_unlocked_timeout(&range, remaining);
+ if (ret)
goto out;
- }
mutex_lock(&svmm->mutex);
if (mmu_interval_read_retry(range.notifier,
--
2.43.0
^ permalink raw reply related
* [PATCH v10 4/8] mshv: Use hmm_range_fault_unlocked_timeout() for region faults
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
MSHV currently faults movable memory regions by taking mmap_read_lock()
around hmm_range_fault(). That prevents the fault path from handling VMAs
whose fault handlers need to drop mmap_lock, such as userfaultfd-backed
mappings.
Use hmm_range_fault_unlocked_timeout() instead. Passing a timeout of 0
preserves MSHV's existing unbounded retry behavior while letting the HMM
helper own mmap_lock acquisition and refresh range->notifier_seq internally
before walking the range. After the fault succeeds, MSHV still takes
mreg_mutex and checks mmu_interval_read_retry() before installing the pages
into the region, so the existing invalidation synchronization is preserved.
Fold the small fault-and-lock helper into mshv_region_range_fault(), since
the remaining retry path is just the standard "fault, take the driver lock,
check the interval notifier sequence" pattern.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
drivers/hv/mshv_regions.c | 54 +++++++++--------------------------------------
1 file changed, 10 insertions(+), 44 deletions(-)
diff --git a/drivers/hv/mshv_regions.c b/drivers/hv/mshv_regions.c
index 6d65e5b42152..dddaade31b5d 100644
--- a/drivers/hv/mshv_regions.c
+++ b/drivers/hv/mshv_regions.c
@@ -381,46 +381,6 @@ int mshv_region_get(struct mshv_mem_region *region)
return kref_get_unless_zero(®ion->mreg_refcount);
}
-/**
- * mshv_region_hmm_fault_and_lock - Handle HMM faults and lock the memory region
- * @region: Pointer to the memory region structure
- * @range: Pointer to the HMM range structure
- *
- * This function performs the following steps:
- * 1. Reads the notifier sequence for the HMM range.
- * 2. Acquires a read lock on the memory map.
- * 3. Handles HMM faults for the specified range.
- * 4. Releases the read lock on the memory map.
- * 5. If successful, locks the memory region mutex.
- * 6. Verifies if the notifier sequence has changed during the operation.
- * If it has, releases the mutex and returns -EBUSY to match with
- * hmm_range_fault() return code for repeating.
- *
- * Return: 0 on success, a negative error code otherwise.
- */
-static int mshv_region_hmm_fault_and_lock(struct mshv_mem_region *region,
- struct hmm_range *range)
-{
- int ret;
-
- range->notifier_seq = mmu_interval_read_begin(range->notifier);
- mmap_read_lock(region->mreg_mni.mm);
- ret = hmm_range_fault(range);
- mmap_read_unlock(region->mreg_mni.mm);
- if (ret)
- return ret;
-
- mutex_lock(®ion->mreg_mutex);
-
- if (mmu_interval_read_retry(range->notifier, range->notifier_seq)) {
- mutex_unlock(®ion->mreg_mutex);
- cond_resched();
- return -EBUSY;
- }
-
- return 0;
-}
-
/**
* mshv_region_range_fault - Handle memory range faults for a given region.
* @region: Pointer to the memory region structure.
@@ -452,13 +412,19 @@ static int mshv_region_range_fault(struct mshv_mem_region *region,
range.start = region->start_uaddr + page_offset * HV_HYP_PAGE_SIZE;
range.end = range.start + page_count * HV_HYP_PAGE_SIZE;
- do {
- ret = mshv_region_hmm_fault_and_lock(region, &range);
- } while (ret == -EBUSY);
-
+again:
+ ret = hmm_range_fault_unlocked_timeout(&range, 0);
if (ret)
goto out;
+ mutex_lock(®ion->mreg_mutex);
+
+ if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
+ mutex_unlock(®ion->mreg_mutex);
+ cond_resched();
+ goto again;
+ }
+
for (i = 0; i < page_count; i++)
region->mreg_pages[page_offset + i] = hmm_pfn_to_page(pfns[i]);
--
2.43.0
^ permalink raw reply related
* [PATCH v10 3/8] selftests/mm: add HMM test for mmap lock-dropping faults
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
Add test_hmm coverage for the HMM lock-dropping fault path. The test module
gets a new HMM_DMIRROR_READ_UNLOCKED ioctl that calls
hmm_range_fault_unlocked_timeout() with a timeout of 0, exercising the
unbounded retry mode while allowing the mmap lock to be dropped during
fault handling.
Add a userfaultfd_read selftest that registers an anonymous mapping with
UFFDIO_REGISTER_MODE_MISSING, services the faults from a handler thread
with UFFDIO_COPY, and verifies that HMM can read back the data supplied by
the handler. This exercises the path where handle_mm_fault() drops
mmap_lock and hmm_range_fault_unlocked_timeout() restarts the walk
internally.
Assisted-by: GitHub-Copilot:claude-opus-4.6
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
lib/test_hmm.c | 107 ++++++++++++++++++++++-
lib/test_hmm_uapi.h | 1 +
tools/testing/selftests/mm/hmm-tests.c | 150 +++++++++++++++++++++++++++++++++
3 files changed, 257 insertions(+), 1 deletion(-)
diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index 45c0cb992218..6205fb313bd0 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -389,6 +389,67 @@ static int dmirror_range_fault(struct dmirror *dmirror,
return ret;
}
+static int dmirror_range_fault_unlocked(struct dmirror *dmirror,
+ struct hmm_range *range,
+ unsigned long timeout)
+{
+ int ret;
+
+ while (true) {
+ ret = hmm_range_fault_unlocked_timeout(range, timeout);
+ if (ret)
+ goto out;
+
+ mutex_lock(&dmirror->mutex);
+ if (mmu_interval_read_retry(range->notifier,
+ range->notifier_seq)) {
+ mutex_unlock(&dmirror->mutex);
+ continue;
+ }
+ break;
+ }
+
+ ret = dmirror_do_fault(dmirror, range);
+
+ mutex_unlock(&dmirror->mutex);
+out:
+ return ret;
+}
+
+static int dmirror_fault_unlocked(struct dmirror *dmirror,
+ unsigned long start,
+ unsigned long end, bool write,
+ unsigned long timeout)
+{
+ struct mm_struct *mm = dmirror->notifier.mm;
+ unsigned long addr;
+ unsigned long pfns[32];
+ struct hmm_range range = {
+ .notifier = &dmirror->notifier,
+ .hmm_pfns = pfns,
+ .pfn_flags_mask = 0,
+ .default_flags =
+ HMM_PFN_REQ_FAULT | (write ? HMM_PFN_REQ_WRITE : 0),
+ .dev_private_owner = dmirror->mdevice,
+ };
+ int ret = 0;
+
+ if (!mmget_not_zero(mm))
+ return -EFAULT;
+
+ for (addr = start; addr < end; addr = range.end) {
+ range.start = addr;
+ range.end = min(addr + (ARRAY_SIZE(pfns) << PAGE_SHIFT), end);
+
+ ret = dmirror_range_fault_unlocked(dmirror, &range, timeout);
+ if (ret)
+ break;
+ }
+
+ mmput(mm);
+ return ret;
+}
+
static int dmirror_fault(struct dmirror *dmirror, unsigned long start,
unsigned long end, bool write)
{
@@ -488,6 +549,48 @@ static int dmirror_read(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd)
return ret;
}
+static int dmirror_read_unlocked(struct dmirror *dmirror,
+ struct hmm_dmirror_cmd *cmd,
+ unsigned long timeout)
+{
+ struct dmirror_bounce bounce;
+ unsigned long start, end;
+ unsigned long size = cmd->npages << PAGE_SHIFT;
+ int ret;
+
+ start = cmd->addr;
+ end = start + size;
+ if (end < start)
+ return -EINVAL;
+
+ ret = dmirror_bounce_init(&bounce, start, size);
+ if (ret)
+ return ret;
+
+ while (1) {
+ mutex_lock(&dmirror->mutex);
+ ret = dmirror_do_read(dmirror, start, end, &bounce);
+ mutex_unlock(&dmirror->mutex);
+ if (ret != -ENOENT)
+ break;
+
+ start = cmd->addr + (bounce.cpages << PAGE_SHIFT);
+ ret = dmirror_fault_unlocked(dmirror, start, end, false, timeout);
+ if (ret)
+ break;
+ cmd->faults++;
+ }
+
+ if (ret == 0) {
+ if (copy_to_user(u64_to_user_ptr(cmd->ptr), bounce.ptr,
+ bounce.size))
+ ret = -EFAULT;
+ }
+ cmd->cpages = bounce.cpages;
+ dmirror_bounce_fini(&bounce);
+ return ret;
+}
+
static int dmirror_do_write(struct dmirror *dmirror, unsigned long start,
unsigned long end, struct dmirror_bounce *bounce)
{
@@ -1572,7 +1675,9 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp,
dmirror->flags = cmd.npages;
ret = 0;
break;
-
+ case HMM_DMIRROR_READ_UNLOCKED:
+ ret = dmirror_read_unlocked(dmirror, &cmd, 0);
+ break;
default:
return -EINVAL;
}
diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h
index f94c6d457338..ea9b0ec404fb 100644
--- a/lib/test_hmm_uapi.h
+++ b/lib/test_hmm_uapi.h
@@ -38,6 +38,7 @@ struct hmm_dmirror_cmd {
#define HMM_DMIRROR_CHECK_EXCLUSIVE _IOWR('H', 0x06, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_RELEASE _IOWR('H', 0x07, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAGS _IOWR('H', 0x08, struct hmm_dmirror_cmd)
+#define HMM_DMIRROR_READ_UNLOCKED _IOWR('H', 0x09, struct hmm_dmirror_cmd)
#define HMM_DMIRROR_FLAG_FAIL_ALLOC (1ULL << 0)
diff --git a/tools/testing/selftests/mm/hmm-tests.c b/tools/testing/selftests/mm/hmm-tests.c
index 6fccbdab02ee..5acb728666f8 100644
--- a/tools/testing/selftests/mm/hmm-tests.c
+++ b/tools/testing/selftests/mm/hmm-tests.c
@@ -29,6 +29,10 @@
#include <sys/mman.h>
#include <sys/ioctl.h>
#include <sys/time.h>
+#include <sys/syscall.h>
+#include <sys/eventfd.h>
+#include <linux/userfaultfd.h>
+#include <poll.h>
/*
* This is a private UAPI to the kernel test module so it isn't exported
@@ -2952,4 +2956,150 @@ TEST_F_TIMEOUT(hmm, benchmark_thp_migration, 120)
&thp_results, ®ular_results);
}
}
+/*
+ * Test that HMM can fault in pages backed by userfaultfd using the
+ * hmm_range_fault_unlocked_timeout() path with no timeout. This exercises
+ * the lock-drop retry logic in the HMM framework.
+ */
+struct uffd_thread_args {
+ int uffd;
+ int stop_fd;
+ void *page_buffer;
+ unsigned long page_size;
+};
+
+static void *uffd_handler_thread(void *arg)
+{
+ struct uffd_thread_args *args = arg;
+ struct uffd_msg msg;
+ struct uffdio_copy copy;
+ struct pollfd pollfd[2];
+ int ret;
+
+ pollfd[0].fd = args->uffd;
+ pollfd[0].events = POLLIN;
+ pollfd[1].fd = args->stop_fd;
+ pollfd[1].events = POLLIN;
+
+ while (1) {
+ ret = poll(pollfd, 2, -1);
+ if (ret <= 0)
+ break;
+ if (pollfd[1].revents)
+ break;
+ if (!(pollfd[0].revents & POLLIN))
+ break;
+
+ ret = read(args->uffd, &msg, sizeof(msg));
+ if (ret != sizeof(msg))
+ break;
+
+ if (msg.event != UFFD_EVENT_PAGEFAULT)
+ break;
+
+ /* Fill the page with a known pattern */
+ memset(args->page_buffer, 0xAB, args->page_size);
+
+ copy.dst = msg.arg.pagefault.address & ~(args->page_size - 1);
+ copy.src = (unsigned long)args->page_buffer;
+ copy.len = args->page_size;
+ copy.mode = 0;
+ copy.copy = 0;
+
+ ret = ioctl(args->uffd, UFFDIO_COPY, ©);
+ if (ret < 0)
+ break;
+ }
+
+ return NULL;
+}
+
+TEST_F(hmm, userfaultfd_read)
+{
+ struct hmm_buffer *buffer;
+ struct uffd_thread_args uffd_args;
+ unsigned long npages;
+ unsigned long size;
+ unsigned long i;
+ unsigned char *ptr;
+ pthread_t thread;
+ int uffd;
+ int stop_fd;
+ int ret;
+ struct uffdio_api api;
+ struct uffdio_register reg;
+ uint64_t stop = 1;
+ ssize_t nwrite;
+
+ npages = 4;
+ size = npages << self->page_shift;
+
+ /* Create userfaultfd */
+ uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK);
+ if (uffd < 0)
+ SKIP(return, "userfaultfd not available");
+
+ api.api = UFFD_API;
+ api.features = 0;
+ ret = ioctl(uffd, UFFDIO_API, &api);
+ ASSERT_EQ(ret, 0);
+
+ buffer = malloc(sizeof(*buffer));
+ ASSERT_NE(buffer, NULL);
+
+ buffer->fd = -1;
+ buffer->size = size;
+ buffer->mirror = malloc(size);
+ ASSERT_NE(buffer->mirror, NULL);
+
+ /* Create anonymous mapping */
+ buffer->ptr = mmap(NULL, size,
+ PROT_READ | PROT_WRITE,
+ MAP_PRIVATE | MAP_ANONYMOUS,
+ -1, 0);
+ ASSERT_NE(buffer->ptr, MAP_FAILED);
+
+ /* Register the region with userfaultfd */
+ reg.range.start = (unsigned long)buffer->ptr;
+ reg.range.len = size;
+ reg.mode = UFFDIO_REGISTER_MODE_MISSING;
+ ret = ioctl(uffd, UFFDIO_REGISTER, ®);
+ ASSERT_EQ(ret, 0);
+
+ /* Set up the handler thread */
+ uffd_args.uffd = uffd;
+ stop_fd = eventfd(0, EFD_CLOEXEC);
+ ASSERT_GE(stop_fd, 0);
+ uffd_args.stop_fd = stop_fd;
+ uffd_args.page_buffer = malloc(self->page_size);
+ ASSERT_NE(uffd_args.page_buffer, NULL);
+ uffd_args.page_size = self->page_size;
+
+ ret = pthread_create(&thread, NULL, uffd_handler_thread, &uffd_args);
+ ASSERT_EQ(ret, 0);
+
+ /*
+ * Use the unlocked read path which allows the mmap lock to be
+ * dropped during the fault, enabling userfaultfd resolution.
+ */
+ ret = hmm_dmirror_cmd(self->fd, HMM_DMIRROR_READ_UNLOCKED,
+ buffer, npages);
+ ASSERT_EQ(ret, 0);
+ ASSERT_EQ(buffer->cpages, npages);
+
+ /* Verify the device read the data filled by the uffd handler */
+ ptr = buffer->mirror;
+ for (i = 0; i < size; ++i)
+ ASSERT_EQ(ptr[i], (unsigned char)0xAB);
+
+ nwrite = write(stop_fd, &stop, sizeof(stop));
+ ASSERT_EQ(nwrite, sizeof(stop));
+ pthread_join(thread, NULL);
+ close(stop_fd);
+ free(uffd_args.page_buffer);
+ close(uffd);
+ hmm_buffer_free(buffer);
+}
+
+
TEST_HARNESS_MAIN
--
2.43.0
^ permalink raw reply related
* [PATCH v10 2/8] mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call. This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock. Drivers that need to populate device page tables
for such mappings have no way to do so today.
Add hmm_range_fault_unlocked_timeout() for callers that do not need to hold
mmap_lock across any work outside the HMM fault itself. The helper takes
mmap_read_lock_killable() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still held.
The timeout is specified in jiffies; passing 0 retries indefinitely, while
a non-zero timeout makes the helper return -EBUSY when the retry budget
expires. The retry deadline is set before refreshing the notifier sequence
and acquiring mmap_lock, so contended mmap_lock acquisition is included in
the retry budget.
When handle_mm_fault() drops mmap_lock, or when the range is invalidated,
hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and
retries the walk internally. If the lock was dropped, the retry deadline is
also restarted because a lock-dropping fault handler made progress.
Ordinary -EBUSY retries keep the existing deadline, preserving the caller's
timeout policy for repeated mmu-notifier invalidations.
The caller only needs to perform the usual post-success
mmu_interval_read_retry() check while holding its update lock before
consuming the pfns. If mmap_lock acquisition is interrupted or a fatal
signal is pending during retry handling, -EINTR is returned instead.
The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout(). The
existing hmm_range_fault() path still passes no locked state, does not
allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper
preserving the existing API contract for current callers.
The previous refactor that moved page fault handling out of the page-table
walk callbacks is what makes this change small. Faults now run after
walk_page_range() has unwound, with only mmap_lock held, so dropping it
does not interact with the walker's pte spinlock or hugetlb_vma_lock.
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.
Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
Documentation/mm/hmm.rst | 79 ++++++++++++++++-------
include/linux/hmm.h | 2 +
mm/hmm.c | 162 ++++++++++++++++++++++++++++++++++++++---------
3 files changed, 191 insertions(+), 52 deletions(-)
diff --git a/Documentation/mm/hmm.rst b/Documentation/mm/hmm.rst
index 7d61b7a8b65b..e021218ada58 100644
--- a/Documentation/mm/hmm.rst
+++ b/Documentation/mm/hmm.rst
@@ -156,42 +156,57 @@ During the ops->invalidate() callback the device driver must perform the
update action to the range (mark range read only, or fully unmap, etc.). The
device must complete the update before the driver callback returns.
-When the device driver wants to populate a range of virtual addresses, it can
-use::
+When the device driver wants to populate a range of virtual addresses, the
+normal interface is::
- int hmm_range_fault(struct hmm_range *range);
+ int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout);
It will trigger a page fault on missing or read-only entries if write access is
requested (see below). Page faults use the generic mm page fault code path just
-like a CPU page fault. The usage pattern is::
+like a CPU page fault.
+
+The caller must not hold ``mmap_read_lock`` before the call.
+``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and
+allows ``handle_mm_fault()`` to drop it during fault handling. This is required
+for VMAs whose fault handlers may release the mmap lock, for example regions
+managed by ``userfaultfd``.
+
+If the mmap lock is dropped or the range is invalidated, the function refreshes
+``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is returned
+if mmap lock acquisition is interrupted or a fatal signal is pending during
+retry handling.
+
+The timeout is specified in jiffies; passing ``0`` means retry indefinitely. The
+timeout exists to preserve caller policy for repeated mmu-notifier invalidation
+and is checked between retry attempts. HMM does not interrupt page fault
+handling when the timeout expires, but returns ``-EBUSY`` if the retry budget is
+exhausted before a stable range is obtained.
+
+The usage pattern is::
int driver_populate_range(...)
{
struct hmm_range range;
+ unsigned long timeout;
...
+ timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
range.notifier = &interval_sub;
range.start = ...;
range.end = ...;
range.hmm_pfns = ...;
- if (!mmget_not_zero(interval_sub->notifier.mm))
+ if (!mmget_not_zero(interval_sub.mm))
return -EFAULT;
again:
- range.notifier_seq = mmu_interval_read_begin(&interval_sub);
- mmap_read_lock(mm);
- ret = hmm_range_fault(&range);
- if (ret) {
- mmap_read_unlock(mm);
- if (ret == -EBUSY)
- goto again;
- return ret;
- }
- mmap_read_unlock(mm);
+ ret = hmm_range_fault_unlocked_timeout(&range, timeout);
+ if (ret)
+ goto out_put;
take_lock(driver->update);
- if (mmu_interval_read_retry(&ni, range.notifier_seq) {
+ if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
release_lock(driver->update);
goto again;
}
@@ -200,13 +215,31 @@ like a CPU page fault. The usage pattern is::
* under the update lock */
release_lock(driver->update);
- return 0;
+ ret = 0;
+
+ out_put:
+ mmput(interval_sub.mm);
+ return ret;
}
The driver->update lock is the same lock that the driver takes inside its
invalidate() callback. That lock must be held before calling
mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
-update.
+update. The retry check must use the same notifier and sequence number stored
+in ``range`` by ``hmm_range_fault_unlocked_timeout()``.
+
+Holding the mmap lock across HMM faults
+=======================================
+
+Most callers should use ``hmm_range_fault_unlocked_timeout()``. If a driver
+really needs to hold the mmap lock across work outside HMM, it can use::
+
+ int hmm_range_fault(struct hmm_range *range);
+
+The mmap lock must be held by the caller and will remain held on return. This
+interface cannot support VMAs whose fault handlers need to drop the mmap lock.
+New callers should prefer ``hmm_range_fault_unlocked_timeout()`` unless they
+have a specific requirement to keep the mmap lock held across the call.
Leverage default_flags and pfn_flags_mask
=========================================
@@ -221,8 +254,8 @@ permission, it sets::
range->default_flags = HMM_PFN_REQ_FAULT;
range->pfn_flags_mask = 0;
-and calls hmm_range_fault() as described above. This will fill fault all pages
-in the range with at least read permission.
+and calls the HMM range fault helper as described above. This will fault
+all pages in the range with at least read permission.
Now let's say the driver wants to do the same except for one page in the range for
which it wants to have write permission. Now driver set::
@@ -236,9 +269,9 @@ address == range->start + (index_of_write << PAGE_SHIFT) it will fault with
write permission i.e., if the CPU pte does not have write permission set then HMM
will call handle_mm_fault().
-After hmm_range_fault completes the flag bits are set to the current state of
-the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is
-writable.
+After the HMM range fault helper completes the flag bits are set to the
+current state of the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be
+set if the page is writable.
Represent and manage device memory from core kernel point of view
diff --git a/include/linux/hmm.h b/include/linux/hmm.h
index db75ffc949a7..6f04e3932f5b 100644
--- a/include/linux/hmm.h
+++ b/include/linux/hmm.h
@@ -123,6 +123,8 @@ struct hmm_range {
* Please see Documentation/mm/hmm.rst for how to use the range API.
*/
int hmm_range_fault(struct hmm_range *range);
+int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout);
/*
* HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range
diff --git a/mm/hmm.c b/mm/hmm.c
index bc9361a715fa..87952c1259c5 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -32,6 +32,7 @@
struct hmm_vma_walk {
struct hmm_range *range;
+ bool *locked;
unsigned long last;
unsigned long end;
unsigned int required_fault;
@@ -44,6 +45,14 @@ struct hmm_vma_walk {
*/
#define HMM_FAULT_PENDING -EAGAIN
+/*
+ * Internal sentinel returned by hmm_do_fault() when handle_mm_fault()
+ * completes a page fault with the mmap lock dropped. hmm_do_fault() sets
+ * *locked = false; the outer loop consumes the sentinel and never propagates
+ * it to the caller.
+ */
+#define HMM_FAULT_UNLOCKED -ENOLCK
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -73,9 +82,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
*
* Called by the walk callbacks when they discover that part of the range
* needs a page fault. The callback records what to fault and returns
- * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
- * walk_page_range() and invokes handle_mm_fault() from a context where no
- * page-table or hugetlb_vma_lock is held.
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops
+ * back out of walk_page_range() and invokes handle_mm_fault() from a context
+ * where no page-table or hugetlb_vma_lock is held.
*/
static int hmm_record_fault(unsigned long addr, unsigned long end,
unsigned int required_fault,
@@ -624,7 +633,7 @@ static const struct mm_walk_ops hmm_walk_ops = {
/*
* hmm_do_fault - fault in a range recorded by a walk callback
*
- * Called from the outer loop in hmm_range_fault() after a callback
+ * Called from the outer loop in hmm_range_fault_locked() after a callback
* returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
* the page-table spinlock and any hugetlb_vma_lock acquired by the walk
* framework have already been released by the unwind.
@@ -641,6 +650,9 @@ static int hmm_do_fault(struct mm_struct *mm,
unsigned int fault_flags = FAULT_FLAG_REMOTE;
struct vm_area_struct *vma;
+ if (hmm_vma_walk->locked)
+ fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+
vma = vma_lookup(mm, addr);
if (!vma)
return -EFAULT;
@@ -651,37 +663,34 @@ static int hmm_do_fault(struct mm_struct *mm,
fault_flags |= FAULT_FLAG_WRITE;
}
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
+ for (; addr < end; addr += PAGE_SIZE) {
+ vm_fault_t ret;
+
+ ret = handle_mm_fault(vma, addr, fault_flags, NULL);
+
+ if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
+ *hmm_vma_walk->locked = false;
+ return HMM_FAULT_UNLOCKED;
+ }
+
+ if (ret & VM_FAULT_ERROR) {
+ int err = vm_fault_to_errno(ret, 0);
+
+ if (WARN_ON(!err))
+ err = -EINVAL;
+
+ return err;
+ }
+ }
return -EBUSY;
}
-/**
- * hmm_range_fault - try to fault some address in a virtual address range
- * @range: argument structure
- *
- * Returns 0 on success or one of the following error codes:
- *
- * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
- * (e.g., device file vma).
- * -ENOMEM: Out of memory.
- * -EPERM: Invalid permission (e.g., asking for write and range is read
- * only).
- * -EBUSY: The range has been invalidated and the caller needs to wait for
- * the invalidation to finish.
- * -EFAULT: A page was requested to be valid and could not be made valid
- * ie it has no backing VMA or it is illegal to access
- *
- * This is similar to get_user_pages(), except that it can read the page tables
- * without mutating them (ie causing faults).
- */
-int hmm_range_fault(struct hmm_range *range)
+static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
{
struct hmm_vma_walk hmm_vma_walk = {
.range = range,
+ .locked = locked,
.last = range->start,
};
struct mm_struct *mm = range->notifier->mm;
@@ -704,8 +713,14 @@ int hmm_range_fault(struct hmm_range *range)
* returns -EBUSY so the loop re-walks and picks up the
* now-present entries.
*/
- if (ret == HMM_FAULT_PENDING)
+ if (ret == HMM_FAULT_PENDING) {
ret = hmm_do_fault(mm, &hmm_vma_walk);
+ if (ret == HMM_FAULT_UNLOCKED) {
+ if (fatal_signal_pending(current))
+ return -EINTR;
+ return -EBUSY;
+ }
+ }
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
@@ -715,8 +730,97 @@ int hmm_range_fault(struct hmm_range *range)
} while (ret == -EBUSY);
return ret;
}
+
+/**
+ * hmm_range_fault - try to fault some address in a virtual address range
+ * @range: argument structure
+ *
+ * Returns 0 on success or one of the following error codes:
+ *
+ * -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
+ * (e.g., device file vma).
+ * -ENOMEM: Out of memory.
+ * -EPERM: Invalid permission (e.g., asking for write and range is read
+ * only).
+ * -EBUSY: The range has been invalidated and the caller needs to wait for
+ * the invalidation to finish.
+ * -EFAULT: A page was requested to be valid and could not be made valid
+ * ie it has no backing VMA or it is illegal to access
+ *
+ * This is similar to get_user_pages(), except that it can read the page tables
+ * without mutating them (ie causing faults).
+ *
+ * The mmap lock must be held by the caller and will remain held on return.
+ * New users should prefer hmm_range_fault_unlocked_timeout() unless they
+ * specifically need to keep the mmap lock held across the call. This helper
+ * cannot support VMAs whose fault handlers need to drop the mmap lock.
+ */
+int hmm_range_fault(struct hmm_range *range)
+{
+ return hmm_range_fault_locked(range, NULL);
+}
EXPORT_SYMBOL(hmm_range_fault);
+/**
+ * hmm_range_fault_unlocked_timeout - fault in a range with a retry timeout
+ * @range: argument structure
+ * @timeout: timeout in jiffies for internal -EBUSY retries, or 0 to retry
+ * indefinitely
+ *
+ * The caller must not hold the mmap lock. The function takes the mmap read
+ * lock internally and allows handle_mm_fault() to drop it during faults. If
+ * the mmap lock is dropped or the range is invalidated, the function refreshes
+ * range->notifier_seq and restarts the walk internally.
+ *
+ * Passing 0 for @timeout retries indefinitely. A non-zero @timeout is a caller
+ * policy limit for repeated mmu-notifier invalidation retries. HMM does not
+ * interrupt page fault handling when the timeout expires, but returns -EBUSY
+ * if the retry budget is exhausted before a stable range is obtained.
+ *
+ * Returns 0 on success or one of the error codes documented for
+ * hmm_range_fault(). -EINTR is returned if mmap_lock acquisition is
+ * interrupted or a fatal signal is pending during retry handling.
+ */
+int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
+ unsigned long timeout)
+{
+ struct mm_struct *mm = range->notifier->mm;
+ unsigned long deadline = 0;
+ bool locked = false;
+ int ret;
+
+ do {
+ /*
+ * If the previous fault dropped mmap_lock, then the fault
+ * handler made progress. Restart the retry timeout in that
+ * case, but keep the existing deadline for ordinary -EBUSY
+ * retries.
+ */
+ if (timeout && !locked)
+ deadline = jiffies + timeout;
+
+ range->notifier_seq =
+ mmu_interval_read_begin(range->notifier);
+
+ ret = mmap_read_lock_killable(mm);
+ if (ret)
+ return ret;
+
+ if (timeout && time_after(jiffies, deadline)) {
+ mmap_read_unlock(mm);
+ return -EBUSY;
+ }
+
+ locked = true;
+ ret = hmm_range_fault_locked(range, &locked);
+ if (locked)
+ mmap_read_unlock(mm);
+ } while (ret == -EBUSY);
+
+ return ret;
+}
+EXPORT_SYMBOL(hmm_range_fault_unlocked_timeout);
+
/**
* hmm_dma_map_alloc - Allocate HMM map structure
* @dev: device to allocate structure for
--
2.43.0
^ permalink raw reply related
* [PATCH v10 1/8] mm/hmm: move page fault handling out of walk callbacks
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
In-Reply-To: <20260722-hmm-v10-v1-0-606464dd601a@gmail.com>
hmm_range_fault() currently triggers page faults from inside the page-table
walk callbacks: hmm_vma_walk_pmd(), hmm_vma_walk_pud(),
hmm_vma_walk_hugetlb_entry() and the pte-level helper all call
hmm_vma_fault(), which in turn calls handle_mm_fault() while the walker
still holds nested locks. The pte spinlock is dropped explicitly by each
caller, and the hugetlb path manually drops and retakes
hugetlb_vma_lock_read around the fault to dodge a deadlock against the walk
framework's unconditional unlock.
This layering does not extend cleanly to fault handlers that may release
mmap_lock (VM_FAULT_RETRY, VM_FAULT_COMPLETED). If the lock is dropped
while walk_page_range() is mid-traversal, the VMA can be freed before the
walk framework's matching hugetlb_vma_unlock_read(), turning that unlock
into a use-after-free.
Split the responsibilities the way get_user_pages() does. Walk callbacks
become inspect-only: when they detect a range that needs to be faulted in,
they record it in struct hmm_vma_walk and return a private sentinel
(HMM_FAULT_PENDING). The outer loop in hmm_range_fault() then drops out of
walk_page_range(), invokes a new helper hmm_do_fault() that calls
handle_mm_fault() with only mmap_lock held, and restarts the walk so the
now-present entries are collected into hmm_pfns.
No functional change for existing callers. As a side effect the hugetlb
callback no longer needs the hugetlb_vma_{un}lock_read dance, and every
fault-path exit from the callbacks now releases the pte spinlock on a
single, common path. This refactor is also a precursor for adding an
unlockable variant of hmm_range_fault() in a follow-up patch.
Reviewed-by: Jason Gunthorpe <jgg@nvidia.com>
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
---
mm/hmm.c | 118 ++++++++++++++++++++++++++++++++++++++++-----------------------
1 file changed, 75 insertions(+), 43 deletions(-)
diff --git a/mm/hmm.c b/mm/hmm.c
index e5c1f4deed24..bc9361a715fa 100644
--- a/mm/hmm.c
+++ b/mm/hmm.c
@@ -33,8 +33,17 @@
struct hmm_vma_walk {
struct hmm_range *range;
unsigned long last;
+ unsigned long end;
+ unsigned int required_fault;
};
+/*
+ * Internal sentinel returned by walk callbacks when they need a page fault.
+ * The callback stores end/required_fault in hmm_vma_walk; the outer loop
+ * consumes the sentinel and never propagates it to the caller.
+ */
+#define HMM_FAULT_PENDING -EAGAIN
+
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -60,37 +69,25 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
}
/*
- * hmm_vma_fault() - fault in a range lacking valid pmd or pte(s)
- * @addr: range virtual start address (inclusive)
- * @end: range virtual end address (exclusive)
- * @required_fault: HMM_NEED_* flags
- * @walk: mm_walk structure
- * Return: -EBUSY after page fault, or page fault error
+ * hmm_record_fault() - record a range that needs to be faulted in
*
- * This function will be called whenever pmd_none() or pte_none() returns true,
- * or whenever there is no page directory covering the virtual address range.
+ * Called by the walk callbacks when they discover that part of the range
+ * needs a page fault. The callback records what to fault and returns
+ * HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
+ * walk_page_range() and invokes handle_mm_fault() from a context where no
+ * page-table or hugetlb_vma_lock is held.
*/
-static int hmm_vma_fault(unsigned long addr, unsigned long end,
- unsigned int required_fault, struct mm_walk *walk)
+static int hmm_record_fault(unsigned long addr, unsigned long end,
+ unsigned int required_fault,
+ struct mm_walk *walk)
{
struct hmm_vma_walk *hmm_vma_walk = walk->private;
- struct vm_area_struct *vma = walk->vma;
- unsigned int fault_flags = FAULT_FLAG_REMOTE;
WARN_ON_ONCE(!required_fault);
hmm_vma_walk->last = addr;
-
- if (required_fault & HMM_NEED_WRITE_FAULT) {
- if (!(vma->vm_flags & VM_WRITE))
- return -EPERM;
- fault_flags |= FAULT_FLAG_WRITE;
- }
-
- for (; addr < end; addr += PAGE_SIZE)
- if (handle_mm_fault(vma, addr, fault_flags, NULL) &
- VM_FAULT_ERROR)
- return -EFAULT;
- return -EBUSY;
+ hmm_vma_walk->end = end;
+ hmm_vma_walk->required_fault = required_fault;
+ return HMM_FAULT_PENDING;
}
static unsigned int hmm_pte_need_fault(const struct hmm_vma_walk *hmm_vma_walk,
@@ -174,7 +171,7 @@ static int hmm_vma_walk_hole(unsigned long addr, unsigned long end,
return hmm_pfns_fill(addr, end, range, HMM_PFN_ERROR);
}
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
return hmm_pfns_fill(addr, end, range, 0);
}
@@ -209,7 +206,7 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr,
required_fault =
hmm_range_need_fault(hmm_vma_walk, hmm_pfns, npages, cpu_flags);
if (required_fault)
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT);
for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) {
@@ -328,7 +325,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr,
fault:
pte_unmap(ptep);
/* Fault any virtual address we were asked to fault */
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
#ifdef CONFIG_ARCH_HAS_PMD_SOFTLEAVES
@@ -371,7 +368,7 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start,
npages, 0);
if (required_fault) {
if (softleaf_is_device_private(entry))
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
else
return -EFAULT;
}
@@ -517,7 +514,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,
npages, cpu_flags);
if (required_fault) {
spin_unlock(ptl);
- return hmm_vma_fault(addr, end, required_fault, walk);
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT);
@@ -564,21 +561,8 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask,
required_fault =
hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags);
if (required_fault) {
- int ret;
-
spin_unlock(ptl);
- hugetlb_vma_unlock_read(vma);
- /*
- * Avoid deadlock: drop the vma lock before calling
- * hmm_vma_fault(), which will itself potentially take and
- * drop the vma lock. This is also correct from a
- * protection point of view, because there is no further
- * use here of either pte or ptl after dropping the vma
- * lock.
- */
- ret = hmm_vma_fault(addr, end, required_fault, walk);
- hugetlb_vma_lock_read(vma);
- return ret;
+ return hmm_record_fault(addr, end, required_fault, walk);
}
pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT);
@@ -637,6 +621,44 @@ static const struct mm_walk_ops hmm_walk_ops = {
.walk_lock = PGWALK_RDLOCK,
};
+/*
+ * hmm_do_fault - fault in a range recorded by a walk callback
+ *
+ * Called from the outer loop in hmm_range_fault() after a callback
+ * returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
+ * the page-table spinlock and any hugetlb_vma_lock acquired by the walk
+ * framework have already been released by the unwind.
+ *
+ * Returns -EBUSY on success (all pages faulted, caller should re-walk).
+ * Returns a negative errno on failure.
+ */
+static int hmm_do_fault(struct mm_struct *mm,
+ struct hmm_vma_walk *hmm_vma_walk)
+{
+ unsigned long addr = hmm_vma_walk->last;
+ unsigned long end = hmm_vma_walk->end;
+ unsigned int required_fault = hmm_vma_walk->required_fault;
+ unsigned int fault_flags = FAULT_FLAG_REMOTE;
+ struct vm_area_struct *vma;
+
+ vma = vma_lookup(mm, addr);
+ if (!vma)
+ return -EFAULT;
+
+ if (required_fault & HMM_NEED_WRITE_FAULT) {
+ if (!(vma->vm_flags & VM_WRITE))
+ return -EPERM;
+ fault_flags |= FAULT_FLAG_WRITE;
+ }
+
+ for (; addr < end; addr += PAGE_SIZE)
+ if (handle_mm_fault(vma, addr, fault_flags, NULL) &
+ VM_FAULT_ERROR)
+ return -EFAULT;
+
+ return -EBUSY;
+}
+
/**
* hmm_range_fault - try to fault some address in a virtual address range
* @range: argument structure
@@ -674,6 +696,16 @@ int hmm_range_fault(struct hmm_range *range)
return -EBUSY;
ret = walk_page_range(mm, hmm_vma_walk.last, range->end,
&hmm_walk_ops, &hmm_vma_walk);
+ /*
+ * When HMM_FAULT_PENDING is returned a walk callback
+ * recorded a range that needs handle_mm_fault();
+ * hmm_do_fault() runs the fault outside walk_page_range()
+ * (so no page-table or hugetlb_vma_lock is held) and
+ * returns -EBUSY so the loop re-walks and picks up the
+ * now-present entries.
+ */
+ if (ret == HMM_FAULT_PENDING)
+ ret = hmm_do_fault(mm, &hmm_vma_walk);
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
--
2.43.0
^ permalink raw reply related
* [PATCH v10 0/8] mm/hmm: Add mmap lock-drop support for userfaultfd-backed mappings
From: Stanislav Kinsburskii @ 2026-07-22 21:44 UTC (permalink / raw)
To: Jason Gunthorpe, Leon Romanovsky, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R. Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, Michal Hocko,
Jonathan Corbet, Shuah Khan, Shuah Khan, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li, Lyude Paul,
Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Min Ma, Lizhi Hou,
Oded Gabbay, skinsburskii
Cc: linux-mm, linux-kernel, linux-doc, linux-kselftest, linux-hyperv,
dri-devel, nouveau, linux-rdma, Jason Gunthorpe
This series extends the HMM framework to support userfaultfd-backed memory
by allowing the mmap read lock to be dropped during hmm_range_fault().
Some page fault handlers — most notably userfaultfd — require the mmap lock
to be released so that userspace can resolve the fault. The current HMM
interface never sets FAULT_FLAG_ALLOW_RETRY, making it impossible to fault
in pages from userfaultfd-registered regions.
This series follows the established int *locked pattern from
get_user_pages_remote() in mm/gup.c. A new helper function,
hmm_range_fault_locked(), accepts an int *locked parameter. When the
mmap lock is dropped during fault resolution (VM_FAULT_RETRY or
VM_FAULT_COMPLETED), the function returns 0 with *locked = 0, signalling
the caller to restart its walk. The existing hmm_range_fault() is
refactored into a thin wrapper that passes NULL, preserving current
behavior for all existing callers.
Possible approaches to lift this limitation are documented in
Documentation/mm/hmm.rst.
Changes in v10:
- Included contended mmap_lock acquisition in the
hmm_range_fault_unlocked_timeout() retry budget.
- Dropped the redundant top-level fatal_signal_pending() check in the HMM
unlocked retry loop; mmap_read_lock_killable() now covers that path.
- Restored the absolute outer timeout in drm_gpusvm_get_pages(), since it can
run from GPU page-fault workers and must not rely on the worker task’s fatal
signal state to stop invalidation retries.
Changes in v9:
- Folded the fixups into the full 8-patch series instead of sending a
separate fixup series.
- Clarified that the HMM timeout bounds repeated HMM/mmu-notifier retry
attempts, with the helper refreshing range->notifier_seq internally.
- Kept nouveau’s explicit outer deadline because its retry loop runs in a GPU
fault worker and cannot rely on fatal signals from the faulting process.
- Converted amdxdna, and GPU SVM callers to pass the shole timeout budget to
hmm_range_fault_unlocked_timeout().
Changes in v8:
- Make hmm_range_fault_unlocked_timeout() the primary documented HMM
range-fault API, and move hmm_range_fault() into the “use only if the
caller really must hold mmap_lock” category.
- Clarify that the timeout is a retry budget for repeated mmu-notifier
invalidation retries. HMM does not interrupt an in-progress page fault
when the timeout expires.
- Restart the retry timeout only when handle_mm_fault() dropped
mmap_lock, because that indicates a lock-dropping fault handler such as
userfaultfd made progress. Ordinary -EBUSY retries keep the existing
deadline.
- Remove the attempted timeout selftest. The remaining selftest covers the
intended userfaultfd path by resolving missing-page faults through
HMM_DMIRROR_READ_UNLOCKED and hmm_range_fault_unlocked_timeout(..., 0).
Changes in v7:
- Replaced the unlocked HMM API with
hmm_range_fault_unlocked_timeout(). The helper now takes a timeout in
jiffies, with 0 meaning retry indefinitely.
- Moved -EBUSY retry handling into the HMM helper for the unlocked path.
The helper refreshes range->notifier_seq internally before each retry.
- Switched the unlocked path to mmap_read_lock_killable() and return
-EINTR if mmap lock acquisition is interrupted or a fatal signal is
pending during retry handling.
- Removed the redundant non-timeout hmm_range_fault_unlocked() interface.
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the timeout API
and the intended caller pattern.
- Updated the HMM selftests to use hmm_range_fault_unlocked_timeout()
only, including coverage for the finite-timeout path.
- Added in-tree users of the new helper:
- mshv
- nouveau
- RDMA/umem
- amdxdna
- drm/gpusvm
- Preserved each converted driver’s existing timeout convention:
- unbounded retry where the old code retried indefinitely,
- HMM_RANGE_DEFAULT_TIMEOUT where the old code used that budget,
- existing driver-specific timeout return values such as -ETIME.
- Used max_t(long, timeout - jiffies, 1) when passing remaining time from
absolute jiffies deadlines to avoid unsigned underflow while keeping a
minimum one-jiffy retry window.
- Left callers on hmm_range_fault() when they already need to hold
mmap_lock across surrounding work, such as drm_gpusvm_check_pages().
Changes in v6:
- Reworked the new API from the external int *locked pattern to
hmm_range_fault_unlocked(), which owns mmap_read_lock() internally.
- Changed the dropped-lock contract: hmm_range_fault_unlocked() now returns
-EBUSY when the mmap lock is dropped, and callers restart with a fresh
mmu_interval_read_begin() sequence.
- Kept hmm_range_fault() as the locked variant for existing users, preserving
its caller-held mmap lock contract.
- Added an in-tree user by converting the MSHV region fault path to
hmm_range_fault_unlocked().
- Updated Documentation/mm/hmm.rst and kernel-doc to describe the unlocked
helper and retry pattern.
- Updated commit messages to match the new API and return semantics.
- Kept the userfaultfd HMM selftest using the test_hmm unlocked read ioctl
path.
Changes in v5:
- Rework hmm_range_fault_unlockable() retry handling to retry
VM_FAULT_RETRY internally with FAULT_FLAG_TRIED set, matching the
fixup_user_fault() pattern and avoiding repeated first-retry lock drops.
- Distinguish VM_FAULT_RETRY from VM_FAULT_COMPLETED: retry faults now
reacquire the mmap lock internally, while completed faults return to the
caller with *locked = 0 so the caller can restart with a fresh notifier
sequence.
- Document the two *locked return states, including the -EINTR case when a
fatal signal is pending after the mmap lock has already been dropped.
- Update comments around HMM_FAULT_UNLOCKED and the HMM fault loop to match
the current hmm_range_fault_unlockable() implementation.
Changes in v4:
- Rebased on 7.2-rc1
Changes in v3:
- Return -EFAULT from dmirror_fault_unlockable() when the mirrored mm can
no longer be pinned.
- Add an eventfd stop signal for the userfaultfd handler thread to avoid
waiting for the poll timeout on successful test completion.
Changes in v2:
- Split into a preparatory refactor (new patch 1) that moves
handle_mm_fault() out of the walk callbacks, plus a smaller feature
patch on top. Suggested by David Hildenbrand.
- Hugetlb regions are now supported on the unlockable path; the v1
-EFAULT short-circuit and the hugetlb_vma_lock_read drop/retake
dance are gone.
- Distinct internal sentinels for "needs fault" (HMM_FAULT_PENDING)
and "lock dropped" (HMM_FAULT_UNLOCKED).
- Outer loop now re-walks after a successful internal fault so the
faulted pfns end up in range->hmm_pfns.
- Kernel-doc on hmm_range_fault_unlockable() and the
Documentation/mm/hmm.rst example match the implementation.
- Dropped the mshv driver conversion (v1 patch 2); will post
separately.
- Selftest converted to drive the path through test_hmm with a
userfaultfd handler (new HMM_DMIRROR_READ_UNLOCKABLE ioctl).
---
Stanislav Kinsburskii (8):
mm/hmm: move page fault handling out of walk callbacks
mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support
selftests/mm: add HMM test for mmap lock-dropping faults
mshv: Use hmm_range_fault_unlocked_timeout() for region faults
drm/nouveau: Use hmm_range_fault_unlocked_timeout() for SVM faults
RDMA/umem: Use hmm_range_fault_unlocked_timeout() for ODP faults
accel/amdxdna: Use hmm_range_fault_unlocked_timeout() for range population
drm/gpusvm: Use hmm_range_fault_unlocked_timeout() for range faults
Documentation/mm/hmm.rst | 79 +++++++---
drivers/accel/amdxdna/aie2_ctx.c | 23 +--
drivers/gpu/drm/drm_gpusvm.c | 60 ++------
drivers/gpu/drm/nouveau/nouveau_svm.c | 20 +--
drivers/hv/mshv_regions.c | 54 ++-----
drivers/infiniband/core/umem_odp.c | 18 +--
include/linux/hmm.h | 2 +
lib/test_hmm.c | 107 +++++++++++++-
lib/test_hmm_uapi.h | 1 +
mm/hmm.c | 256 +++++++++++++++++++++++++--------
tools/testing/selftests/mm/hmm-tests.c | 150 +++++++++++++++++++
11 files changed, 551 insertions(+), 219 deletions(-)
---
base-commit: a4271215228d2bd2a50eea03ab0a948b36036948
change-id: 20260722-hmm-v10-727db2bb9d34
Best regards,
--
Stanislav Kinsburskii <skinsburskii@gmail.com>
^ permalink raw reply
* Re: [PATCH v2] arch: arm64: add early_param idle=<wfi|yield|nop>
From: Will Deacon @ 2026-07-22 21:43 UTC (permalink / raw)
To: Yureka Lilian
Cc: Jonathan Corbet, Shuah Khan, Catalin Marinas, Anshuman Khandual,
linux-doc, linux-kernel, linux-arm-kernel
In-Reply-To: <20260711-arm64-idle-param-v2-1-0ab67652a435@cyberchaos.dev>
On Sat, Jul 11, 2026 at 09:35:25AM +0200, Yureka Lilian wrote:
> diff --git a/arch/arm64/lib/delay.c b/arch/arm64/lib/delay.c
> index e278e060e78a..2452990ed37a 100644
> --- a/arch/arm64/lib/delay.c
> +++ b/arch/arm64/lib/delay.c
> @@ -15,6 +15,8 @@
>
> #include <clocksource/arm_arch_timer.h>
>
> +#include "../kernel/idle.h"
> +
> #define USECS_TO_CYCLES(time_usecs) \
> xloops_to_cycles((time_usecs) * 0x10C7UL)
>
> @@ -49,7 +51,8 @@ void __delay(unsigned long cycles)
> * Start with WFIT. If an interrupt makes us resume
> * early, use a WFET loop to complete the delay.
> */
> - wfit(end);
> + if (likely(idle == ARM64_IDLE_WFI))
> + wfit(end);
Rather than scatter the idle implementation check across all users of
WFI*, why not move this into the macro itself? That way, the callers can
all stay like they are but the macro behaves as specified.
Will
^ permalink raw reply
* Re: [PATCH v5] arm64: errata: work around NVIDIA Olympus device store/load ordering
From: Will Deacon @ 2026-07-22 21:39 UTC (permalink / raw)
To: Catalin Marinas, Vladimir Murzin, Shanker Donthineni
Cc: kernel-team, Will Deacon, linux-arm-kernel, Mark Rutland,
linux-kernel, linux-doc, Vikram Sethi, Jason Sequeira,
Jason Gunthorpe
In-Reply-To: <20260715204856.3707449-1-sdonthineni@nvidia.com>
On Wed, 15 Jul 2026 15:48:56 -0500, Shanker Donthineni wrote:
> On systems with NVIDIA Olympus cores, a Device-nGnR* load can be
> observed by a peripheral before an older, non-overlapping Device-nGnR*
> store to the same peripheral. This breaks the program-order guarantee
> that software expects for Device-nGnR* accesses and can leave a
> peripheral in an incorrect state.
>
> The erratum can occur only when all of the following apply:
>
> [...]
Applied to arm64 (for-next/errata), thanks!
[1/1] arm64: errata: work around NVIDIA Olympus device store/load ordering
https://git.kernel.org/arm64/c/12aab25ca56e
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
^ permalink raw reply
* Re: [PATCH v5 00/36] Private Memory NUMA Nodes
From: Gregory Price @ 2026-07-22 20:56 UTC (permalink / raw)
To: Richard Cheng
Cc: linux-mm, Zhigang.Luo, arun.george, balbirs, brendan.jackman,
yuzenghui, apopple, alucerop, matthew.brost, akpm, david, ljs,
liam, vbabka, rppt, surenb, mhocko, corbet, skhan, gregkh, rafael,
dakr, djbw, vishal.l.verma, dave.jiang, alison.schofield, osandov,
jannh, pfalcato, jackmanb, hannes, ziy, pbonzini, osalvador,
joshua.hahnjy, rakie.kim, byungchul, ying.huang, kasong, qi.zheng,
shakeel.butt, baohua, axelrasmussen, yuanchu, weixugc, yury.norov,
linux, longman, ridong.chen, tj, mkoutny, sj, jgg, jhubbard,
peterx, baolin.wang, npache, ryan.roberts, dev.jain, lance.yang,
usama.arif, xu.xin16, chengming.zhou, roman.gushchin, muchun.song,
linux-kernel, linux-doc, driver-core, nvdimm, linux-cxl,
linux-debuggers, linux-fsdevel, kvm, cgroups, damon,
linux-kselftest, kernel-team
In-Reply-To: <amDVQopyvolxfX8K@gourry-fedora-PF4VCD3F>
On Wed, Jul 22, 2026 at 11:40:06AM -0400, Gregory Price wrote:
> On Wed, Jul 22, 2026 at 10:20:00PM +0800, Richard Cheng wrote:
> >
> > I saw the reply in patch 5, so in fact there's not only one
> > dedicate zonelist, but numerous ?
> > I raise the question because zonelist was supposed to be a
> > global, unbypasssable thing in MM design, but now what you
> > are trying to do is to seperate the whole global list into several
> > parts ?
> >
> > Can you explain why in current design you don't consider to support
> > something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination
> > of what a global zonelist should look like, no matter private or non.
> >
>
> First let me say that there's nothing that prevents us from doing this,
> and we *could* make this the default case - but this decision was
> intentional by me.
>
You caught me on something here - I am wrong about my own code.
I described to you how I originally implemented zonelist isolation,
which I walked back when I was testing mempolicy and demotion.
The current code and the comments in the code are correct, we get
the following zonelists.
N_MEMORY : 0,1
N_MEMORY_PRIVATE: 2,3
ZONELIST_PRIVATE[0] = {0,1,2,3}
ZONELIST_PRIVATE[1] = {1,0,2,3}
ZONELIST_PRIVATE[2] = {2,0,1,3}
ZONELIST_PRIVATE[3] = {3,0,1,2}
Meaning everything except for MPOL_F_RELATIVE_NODES works (because
of cpuset isolation, but more on this in a moment).
So what I shipped here is what you are describing.
Looking back at my notes - I realized a few things that lead me to
walk back that isolation:
1) mempolicy makes more sense this way.
2) reclaim/demotion is easier to implement without isolation.
3) The individual service checks and nodelists handle the rest.
4) It overally just makes more sense. If something breaks isolation
it's because there's a bug, not because of a structural issue.
For example in demotion we do this:
demote_folio_list():
/* Get demotion targets, find the best one, and demote */
node_get_allowed_targets(pgdat, &allowed_mask);
target_nid = next_demotion_node(pgdat->node_id, &allowed_mask);
migrate_pages(demote_folios, alloc_demote_folio, ...);
Where the first step already filters on CAP_DEMOTION.
The result is still clean - no special iterators needed.
What I realized from your questions here is that since i walked back the
zonelist isolation, I think i can actually include any CAP_USER_NUMA
node in cpuset.mems and allow parititioning fairly trivially.
I think this might be the last majority complexity falling out.
Thank you for the questions - this has helped.
~Gregory
^ permalink raw reply
* Re: [PATCH 2/4] sched/cache: Add PR_SCHED_CACHE prctl for per-mm control
From: Tim Chen @ 2026-07-22 20:56 UTC (permalink / raw)
To: Chen, Yu C, Yangyu Chen
Cc: K Prateek Nayak, Dietmar Eggemann, Valentin Schneider,
Jonathan Corbet, Shuah Khan, Yangyu Chen, linux-kernel,
linux-kselftest, linux-doc, Peter Zijlstra, Ingo Molnar,
Juri Lelli, Vincent Guittot, chen.yu@linux.dev
In-Reply-To: <656bbcec-fc0a-4c60-b4ec-c060f800c1cc@intel.com>
On Wed, 2026-07-22 at 18:13 +0800, Chen, Yu C wrote:
> Hi Yangyu,
>
> On 7/22/2026 5:10 PM, Yangyu Chen wrote:
> > Cache aware scheduling is currently controlled only through global
> > debugfs knobs, but the right aggressiveness is workload and platform
> > specific. A multi-threaded Verilator run is one example: its RSS is
> > large while only a small part of it is hot, so an RSS-based footprint
> > estimate should not decide whether it is aggregated; and packing its
> > threads onto the SMT siblings of one LLC beats spreading them across
> > LLCs on some platforms (e.g. AMD EPYC Turin) but not on others (e.g.
> > EPYC Milan). Such choices cannot be made globally for the whole
> > machine. Add a prctl interface to override the knobs per process
> > (per mm_struct):
> >
> > prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SET, attr, value, 0);
> > prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, attr, &value, 0);
> >
> > A single prctl command implements both directions, like
> > PR_RSEQ_SLICE_EXTENSION. The attributes are a per-process enable
> > (effective only while the feature is globally active), the two
> > aggregation tolerances, the overaggr percentage (applied where a
> > task's own migration is admitted; group level statistics span many
> > processes and keep using the global value), and an inherit mask
> > selecting which attributes an mm created by execve() keeps. fork()
> > always inherits everything, and the mask itself lives on the
> > task_struct so it survives both, which lets a numactl-like launcher
> > configure a workload and exec it.
> >
> > The overrides live in mm->sc_stat with -1 meaning "follow the global
> > default"; GET stores the raw value through an int pointer so this
> > sentinel round-trips without being mistaken for an errno.
> > mm_init_sched() gains the creating task to tell fork (p != current)
> > from exec (p == current) apart. A disabled mm has its preferred LLC
> > invalidated at the existing invalidation points, so all group-level
> > statistics self-neutralize.
> >
> > Also sync the tools/perf/trace/beauty copy of prctl.h.
> >
> > Assisted-by: Claude:claude-fable-5
> > Signed-off-by: Yangyu Chen <cyy@cyyself.name>
>
> [ ... ]
>
> > +static int sched_cache_set_attr(unsigned long attr, unsigned long val)
> > +{
> > + struct mm_struct *mm = current->mm;
>
> As preparation work, should we first decouple sc_stat from
> mm_struct and tie this stat to per-task task_struct? In this
> way, we could have per-task cache preference control and extend
> it to tasks/threads/process/cgroup if needed, which looks more
> flexible IMO. We have a proposal here:
> https://github.com/chen-yu-surf/linux/commit/bd43a0b6dd189d5091fb88630208cb7bf67b3165.patch
>
> which introduces a pointer in task_struct:
> struct sched_cache_group __rcu *sched_cache_grp;
>
> > + bool def = sched_cache_val_default(val);
> > + int ival = def ? -1 : (int)val;
> > +
> > + switch (attr) {
> > + case PR_SCHED_CACHE_ENABLE:
> > + if (!def && val > 1)
> > + return -EINVAL;
> > + WRITE_ONCE(mm->sc_stat.user_enabled, ival);
> > + /*
> > + * Drop the preferred LLC hint on any change: a process
> > + * that became disabled must stop being honored right
> > + * away, and one that became enabled re-establishes the
> > + * hint within an epoch anyway. This is best effort: an
> > + * in-flight task_cache_work() scan re-checks the enable
> > + * before publishing a new preference, and a lost race
> > + * is corrected at the next tick.
> > + */
> > + WRITE_ONCE(mm->sc_stat.cpu, -1);
> > + break;
>
> After we switching from per mm_struct to per task control, we could provide
> fine-gain control at task/process/process group granularity(similar to
> core-scheduling)
We are planning to introduce the concept of a sched_group. And tasks in a sched
group can be grouped by mm, or using prctl to explicitly group them together.
We could enhance prctl to introduce per sched_group parameters like aggr_tolerance*
if it makes sense.
Tim
>
> int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid,
> unsigned long cookie, unsigned long type);
> pid argument: the PID of the target task. 0 means "the calling task."
> pid_type : PIDTYPE_PID targets the single thread,
> PIDTYPE_TGID the whole thread group and PIDTYPE_PGID the process
> group of the target task.
>
> And the proposal is here:
> https://github.com/chen-yu-surf/linux/commit/17718b7cef1d03948e9fd3bcd0b5a49aba7aae2d.patch
>
> thanks,
> Chenyu
^ permalink raw reply
* Re: [PATCH bpf-next v5 8/8] selftests: net: add test for XDP_PASS skb checksum invalidation
From: Stanislav Fomichev @ 2026-07-22 20:40 UTC (permalink / raw)
To: Jakub Kicinski
Cc: Lorenzo Bianconi, Donald Hunter, David S. Miller, Eric Dumazet,
Paolo Abeni, Simon Horman, Alexei Starovoitov, Daniel Borkmann,
Jesper Dangaard Brouer, John Fastabend, Stanislav Fomichev,
Andrew Lunn, Tony Nguyen, Przemek Kitszel, Alexander Lobakin,
Andrii Nakryiko, Martin KaFai Lau, Eduard Zingerman, Song Liu,
Yonghong Song, KP Singh, Hao Luo, Jiri Olsa, Shuah Khan,
Maciej Fijalkowski, Jonathan Corbet, Shuah Khan,
Kumar Kartikeya Dwivedi, Emil Tsalapatis, Vladimir Vdovin,
Jakub Sitnicki, netdev, bpf, intel-wired-lan, linux-kselftest,
linux-doc
In-Reply-To: <20260722113525.34092660@kernel.org>
On 07/22, Jakub Kicinski wrote:
> On Wed, 22 Jul 2026 11:00:07 -0700 Stanislav Fomichev wrote:
> > > > This is about current drivers that only report UNNECESSARY with xdp: the xdp
> > > > prog has to maintain in-packet checksum if it changes the payload. I think
> > > > it's a fair assumption?
> > >
> > > What about decap? If we decap the header that device validated
> > > as UNNECESSARY there's no possibility of maintaining the checksum
> > > (by which IIUC you mean correcting it in along the payload changes).
> > >
> > > I think we'd need some API to "decrement the csum level" ?
> > > Or some heuristic in the drivers to decrement if the head moved
> > > by at least len(IP+UDP) into the packet ?
> >
> > True :-/ I guess one more case to document?
>
> What would we be documenting? That it's a known bug and we have no idea
> how to address it? Or am I missing something?
Yeah, that UNECESSARY + decap in xdp has this corner case of inner
csum being not validated (and currently no way to flip it to NONE from
bpf).
(idk whether we really have nics that do csum_level>=2, so I'd just add
bpf_invalidate_csum or something, but it's a separate conversation)
> Again:
> .. the ask was to sketch out the plan of explicitly
> updating/invalidating the checksum even if we don't
> implement it today
SGTM, I'll leave it up to Lorenzo. I had this TODAY vs TOMORROW summary,
we can expand on that with UNNECESSARY+csum_level plus maybe more
details on the shape of kfuncs for TOMORROW section..
^ permalink raw reply
* Re: [PATCH] docs: link AI coding assistants and tool-generated content pages
From: Jonathan Corbet @ 2026-07-22 20:38 UTC (permalink / raw)
To: Lorenzo Stoakes
Cc: Linus Torvalds, Justin Stitt, Laurent Pinchart, Carlos Maiolino,
Jakub Kicinski, Jori Koolstra, Krzysztof Kozlowski, Brian Foster,
Christoph Hellwig, David Disseldorp, Mark Brown, Jani Nikula,
Jens Axboe, David Hildenbrand, Vlastimil Babka, Christian Brauner,
Jeff Layton, workflows, linux-doc, linux-kernel, linux-fsdevel
In-Reply-To: <20260702164901.121128-1-ljs@kernel.org>
Lorenzo Stoakes <ljs@kernel.org> writes:
> Both of these guidelines are complimentary to one another - one specifying
> guidelines on AI coding assistants, and the other specifying those for
> tool-generated content.
>
> Since they are complimentary to one another, provide a link to each in the
> other.
>
> Signed-off-by: Lorenzo Stoakes <ljs@kernel.org>
> ---
> Documentation/process/coding-assistants.rst | 4 ++++
> Documentation/process/generated-content.rst | 7 +++++++
> 2 files changed, 11 insertions(+)
Applied (finally), thanks.
jon
^ permalink raw reply
* [PATCH v3 4/4] KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature
From: Congkai Tan @ 2026-07-22 20:27 UTC (permalink / raw)
To: Oliver Upton, kvmarm, linux-arm-kernel
Cc: Congkai Tan, Marc Zyngier, Joey Gouly, Suzuki K Poulose,
Zenghui Yu, Catalin Marinas, Will Deacon, Paolo Bonzini,
Jonathan Corbet, Shuah Khan, Haris Okanovic, Geoff Blake,
Stanislav Spassov, kvm, linux-doc, linux-kselftest, linux-kernel
In-Reply-To: <20260722202702.4165917-1-congkai@amazon.com>
Introduce a new vCPU feature KVM_ARM_VCPU_PMU_V3_STRICT. When set, KVM
does not create a default PMU when initializing the vCPU, and userspace
must select one explicitly via KVM_ARM_VCPU_PMU_V3_SET_PMU before the
first KVM_RUN.
The flag forces the VMM to be aware of the PMU implementation of the
guest to be created, so that certain information about the PMU becomes
deterministic (if on a heterogeneous system) and becomes safe to be
exposed to the guest. It can be used as an umbrella flag to gate future
PMUv3 UAPI changes.
When no default PMU is created, kvm->arch.arm_pmu stays NULL until SET_PMU
runs, so kvm_arm_pmu_v3_init() now refuses to run if kvm->arch.arm_pmu
is NULL.
Signed-off-by: Congkai Tan <congkai@amazon.com>
Reviewed-by: Geoff Blake <blakgeof@amazon.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Reviewed-by: Stanislav Spassov <stanspas@amazon.de>
Co-developed-by: Oliver Upton <oupton@kernel.org>
Signed-off-by: Oliver Upton <oupton@kernel.org>
---
Documentation/virt/kvm/api.rst | 11 +++++++++++
Documentation/virt/kvm/devices/vcpu.rst | 11 +++++++++--
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/kvm/arm.c | 19 +++++++++++++++----
arch/arm64/kvm/pmu-emul.c | 18 +++++++++++++++++-
include/uapi/linux/kvm.h | 1 +
6 files changed, 54 insertions(+), 8 deletions(-)
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index a5f9ee92f43e..4988c32df4bf 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -3515,6 +3515,17 @@ Possible features:
Depends on KVM_CAP_ARM_PSCI_0_2.
- KVM_ARM_VCPU_PMU_V3: Emulate PMUv3 for the CPU.
Depends on KVM_CAP_ARM_PMU_V3.
+ - KVM_ARM_VCPU_PMU_V3_STRICT: Enable strict PMUv3 UAPI.
+ Requires KVM_ARM_VCPU_PMU_V3. Depends on KVM_CAP_ARM_PMU_V3_STRICT.
+ When enabled:
+
+ * Userspace must explicitly select a PMU implementation before
+ initializing the PMU or configuring a PMU event filter
+
+ * If the PMU implements FEAT_PMUv3p4, PMMIR_EL1.SLOTS provides the
+ hardware value of the underlying implementation
+
+ * Writes to PMCR_EL0.N via KVM_SET_ONE_REG are ignored
- KVM_ARM_VCPU_PTRAUTH_ADDRESS: Enables Address Pointer authentication
for arm64 only.
diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst
index 66e714f2fcfa..deb5c51bc00c 100644
--- a/Documentation/virt/kvm/devices/vcpu.rst
+++ b/Documentation/virt/kvm/devices/vcpu.rst
@@ -53,8 +53,9 @@ Returns:
======= ======================================================
-EEXIST Interrupt number already used
-ENODEV PMUv3 not supported or GIC not initialized
- -ENXIO PMUv3 not supported, missing VCPU feature or interrupt
- number not set (non-GICv5 guests, only)
+ -ENXIO PMUv3 not supported, missing VCPU feature, missing
+ hardware PMU, or interrupt number not set (non-GICv5
+ guests, only)
-EBUSY PMUv3 already initialized
======= ======================================================
@@ -62,6 +63,9 @@ Request the initialization of the PMUv3. If using the PMUv3 with an in-kernel
virtual GIC implementation, this must be done after initializing the in-kernel
irqchip.
+When the KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature is enabled this must be done
+after selecting a hardware PMU.
+
1.3 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_FILTER
-----------------------------------------
@@ -108,6 +112,9 @@ hardware event. Filtering event 0x1E (CHAIN) has no effect either, as it
isn't strictly speaking an event. Filtering the cycle counter is possible
using event 0x11 (CPU_CYCLES).
+When the KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature is enabled this must be done
+after selecting a hardware PMU.
+
1.4 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_SET_PMU
------------------------------------------
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 9e035d587970..39f7fc740d07 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -39,7 +39,7 @@
#define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS
-#define KVM_VCPU_MAX_FEATURES 9
+#define KVM_VCPU_MAX_FEATURES 10
#define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1)
#define KVM_REQ_SLEEP \
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 50adfff75be8..7dbefdd846aa 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -452,6 +452,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
r = get_num_wrps();
break;
case KVM_CAP_ARM_PMU_V3:
+ case KVM_CAP_ARM_PMU_V3_STRICT:
r = kvm_supports_guest_pmuv3();
break;
case KVM_CAP_ARM_INJECT_SERROR_ESR:
@@ -1563,8 +1564,10 @@ static unsigned long system_supported_vcpu_features(void)
if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1))
clear_bit(KVM_ARM_VCPU_EL1_32BIT, &features);
- if (!kvm_supports_guest_pmuv3())
+ if (!kvm_supports_guest_pmuv3()) {
clear_bit(KVM_ARM_VCPU_PMU_V3, &features);
+ clear_bit(KVM_ARM_VCPU_PMU_V3_STRICT, &features);
+ }
if (!system_supports_sve())
clear_bit(KVM_ARM_VCPU_SVE, &features);
@@ -1605,6 +1608,11 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu,
test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features))
return -EINVAL;
+ /* Strict PMUv3 UAPI requires PMUv3. */
+ if (test_bit(KVM_ARM_VCPU_PMU_V3_STRICT, &features) &&
+ !test_bit(KVM_ARM_VCPU_PMU_V3, &features))
+ return -EINVAL;
+
if (!test_bit(KVM_ARM_VCPU_EL1_32BIT, &features))
return 0;
@@ -1634,10 +1642,13 @@ static int kvm_setup_vcpu(struct kvm_vcpu *vcpu)
int ret = 0;
/*
- * When the vCPU has a PMU, but no PMU is set for the guest
- * yet, set the default one.
+ * When the vCPU has a PMU, but no PMU is set for the guest yet, set
+ * the default one. If KVM_ARM_VCPU_PMU_V3_STRICT is set, no default
+ * PMU is created, and userspace must select a PMU via
+ * KVM_ARM_VCPU_PMU_V3_SET_PMU.
*/
- if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu)
+ if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu &&
+ !kvm_vcpu_has_pmuv3_strict(vcpu))
ret = kvm_arm_set_default_pmu(kvm);
/* Prepare for nested if required */
diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c
index bcfd0a91e114..5b1af7e2176f 100644
--- a/arch/arm64/kvm/pmu-emul.c
+++ b/arch/arm64/kvm/pmu-emul.c
@@ -939,6 +939,10 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu)
static int kvm_arm_pmu_v3_init(struct kvm_vcpu *vcpu)
{
+ /* Only possible when using KVM_ARM_VCPU_PMU_V3_STRICT */
+ if (!vcpu->kvm->arch.arm_pmu)
+ return -ENXIO;
+
if (irqchip_in_kernel(vcpu->kvm)) {
int ret;
@@ -1009,6 +1013,14 @@ u8 kvm_arm_pmu_get_max_counters(struct kvm *kvm)
{
struct arm_pmu *arm_pmu = kvm->arch.arm_pmu;
+ /*
+ * Under KVM_ARM_VCPU_PMU_V3_STRICT no PMU exists until userspace sets
+ * one, so this can be reached before arm_pmu is set. Report no
+ * counters in that case.
+ */
+ if (!arm_pmu)
+ return 0;
+
/*
* PMUv3 requires that all event counters are capable of counting any
* event, though the same may not be true of non-PMUv3 hardware.
@@ -1050,7 +1062,8 @@ static void kvm_arm_set_pmu(struct kvm *kvm, struct arm_pmu *arm_pmu)
}
/**
- * kvm_arm_set_default_pmu - No PMU set, get the default one.
+ * kvm_arm_set_default_pmu - No PMU set and KVM_ARM_VCPU_PMU_V3_STRICT not
+ * set, get the default one.
* @kvm: The kvm pointer
*
* The observant among you will notice that the supported_cpus
@@ -1190,6 +1203,9 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr)
if (kvm_vm_has_ran_once(kvm))
return -EBUSY;
+ if (!kvm->arch.arm_pmu)
+ return -ENXIO;
+
if (!kvm->arch.pmu_filter) {
kvm->arch.pmu_filter = bitmap_alloc(nr_events, GFP_KERNEL_ACCOUNT);
if (!kvm->arch.pmu_filter)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 419011097fa8..9fc8dfdfd65f 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -997,6 +997,7 @@ struct kvm_enable_cap {
#define KVM_CAP_S390_KEYOP 247
#define KVM_CAP_S390_VSIE_ESAMODE 248
#define KVM_CAP_S390_HPAGE_2G 249
+#define KVM_CAP_ARM_PMU_V3_STRICT 250
struct kvm_irq_routing_irqchip {
__u32 irqchip;
--
2.50.1
^ permalink raw reply related
* [PATCH v3 3/4] KVM: arm64: Ignore writes to PMCR_EL0.N when using strict UAPI
From: Congkai Tan @ 2026-07-22 20:27 UTC (permalink / raw)
To: Oliver Upton, kvmarm, linux-arm-kernel
Cc: Congkai Tan, Marc Zyngier, Joey Gouly, Suzuki K Poulose,
Zenghui Yu, Catalin Marinas, Will Deacon, Paolo Bonzini,
Jonathan Corbet, Shuah Khan, Haris Okanovic, Geoff Blake,
Stanislav Spassov, kvm, linux-doc, linux-kselftest, linux-kernel
In-Reply-To: <20260722202702.4165917-1-congkai@amazon.com>
From: Oliver Upton <oupton@kernel.org>
KVM allows userspace to configure the number of event counters by
writing to PMCR_EL0.N. While this makes a bit of sense prior to NV, the
value of the field is context-dependent whether or not the vCPU is at
EL2 or EL1. As such, the vCPU attribute was added afterwards and made
mandatory for NV.
As we're in the process of papering over old mistakes, force userspace
to use the vCPU attribute if KVM_ARM_VCPU_PMU_V3_STRICT is set.
Signed-off-by: Oliver Upton <oupton@kernel.org>
Reviewed-by: Geoff Blake <blakgeof@amazon.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Signed-off-by: Congkai Tan <congkai@amazon.com>
---
arch/arm64/kvm/sys_regs.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 4c6f608c4fe5..e93f11b90813 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1502,6 +1502,7 @@ static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
*/
if (!kvm_vm_has_ran_once(kvm) &&
!vcpu_has_nv(vcpu) &&
+ !kvm_vcpu_has_pmuv3_strict(vcpu) &&
new_n <= kvm_arm_pmu_get_max_counters(kvm))
kvm->arch.nr_pmu_counters = new_n;
--
2.50.1
^ permalink raw reply related
* [PATCH v3 1/4] KVM: arm64: Expose PMMIR_EL1.SLOTS under strict PMUv3 UAPI
From: Congkai Tan @ 2026-07-22 20:26 UTC (permalink / raw)
To: Oliver Upton, kvmarm, linux-arm-kernel
Cc: Congkai Tan, Marc Zyngier, Joey Gouly, Suzuki K Poulose,
Zenghui Yu, Catalin Marinas, Will Deacon, Paolo Bonzini,
Jonathan Corbet, Shuah Khan, Haris Okanovic, Geoff Blake,
Stanislav Spassov, kvm, linux-doc, linux-kselftest, linux-kernel
In-Reply-To: <20260722202702.4165917-1-congkai@amazon.com>
Introduce a new field pmmir_slots in struct kvm_arch to store
PMMIR_EL1.SLOTS. It only saves the actual hardware PMU value when
the VMM explicitly selects a PMU under KVM_ARM_VCPU_PMU_V3_STRICT.
Otherwise, it stays 0 after allocation.
Use this field to implement guest access, userspace get, and userspace
set for PMMIR_EL1:
- access_pmmir(): uses the value in kvm->arch.pmmir_slots directly. If
the VMM selected a PMU and KVM_ARM_VCPU_PMU_V3_STRICT is set, the guest
can correctly read the underlying core's SLOTS. Otherwise, it continues
to read 0 since the true SLOTS value can be nondeterministic.
- get_pmmir(): same as access_pmmir().
- set_pmmir(): only the SLOTS field is writable; a value setting any
other bit is rejected with -EINVAL, since get_pmmir() returns SLOTS
zero-extended. A value of 0 resets kvm->arch.pmmir_slots to 0 for
backward compatibility, as the register is RAZ in older KVM, a value
matching the current SLOTS is accepted as a no-op, and anything else is
rejected with -EINVAL. Once the VM has run PMMIR_EL1 is immutable, so a
mismatching write then returns -EBUSY.
The register is now exposed via KVM_GET_REG_LIST for PMUv3 vCPUs, so add
it to the get-reg-list selftest's PMU register list.
Signed-off-by: Congkai Tan <congkai@amazon.com>
Reviewed-by: Geoff Blake <blakgeof@amazon.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Reviewed-by: Stanislav Spassov <stanspas@amazon.de>
Co-developed-by: Oliver Upton <oupton@kernel.org>
Signed-off-by: Oliver Upton <oupton@kernel.org>
---
arch/arm64/include/asm/kvm_host.h | 3 +
arch/arm64/include/uapi/asm/kvm.h | 1 +
arch/arm64/kvm/pmu-emul.c | 11 ++++
arch/arm64/kvm/sys_regs.c | 63 ++++++++++++++++++-
include/kvm/arm_pmu.h | 4 ++
.../selftests/kvm/arm64/get-reg-list.c | 1 +
6 files changed, 81 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index bae2c4f92ef5..9e035d587970 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -387,6 +387,9 @@ struct kvm_arch {
/* Maximum number of counters for the guest */
u8 nr_pmu_counters;
+ /* PMMIR_EL1.SLOTS value exposed to the guest. */
+ u8 pmmir_slots;
+
/* Hypercall features firmware registers' descriptor */
struct kvm_smccc_features smccc_feat;
struct maple_tree smccc_filter;
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 1c13bfa2d38a..019e5e3d892e 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -106,6 +106,7 @@ struct kvm_regs {
#define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */
#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */
#define KVM_ARM_VCPU_HAS_EL2_E2H0 8 /* Limit NV support to E2H RES0 */
+#define KVM_ARM_VCPU_PMU_V3_STRICT 9 /* No default PMU creation */
struct kvm_vcpu_init {
__u32 target;
diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c
index 98305bbfc095..7ac00423fa8d 100644
--- a/arch/arm64/kvm/pmu-emul.c
+++ b/arch/arm64/kvm/pmu-emul.c
@@ -1092,6 +1092,17 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id)
kvm_arm_set_pmu(kvm, arm_pmu);
cpumask_copy(kvm->arch.supported_cpus, &arm_pmu->supported_cpus);
+
+ /*
+ * Since a specific PMU is explicitly selected,
+ * PMMIR_EL1.SLOTS is deterministic to the guest.
+ * If KVM_ARM_VCPU_PMU_V3_STRICT is set, snapshot
+ * the value to allow the guest to read it.
+ */
+ if (kvm_vcpu_has_pmuv3_strict(vcpu))
+ kvm->arch.pmmir_slots =
+ FIELD_GET(ARMV8_PMU_SLOTS,
+ arm_pmu->reg_pmmir);
ret = 0;
break;
}
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 5d5c579d4579..4c6f608c4fe5 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -1367,6 +1367,64 @@ static bool access_pminten(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
return true;
}
+static bool access_pmmir(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
+ const struct sys_reg_desc *r)
+{
+ if (p->is_write)
+ return write_to_read_only(vcpu, p, r);
+
+ /*
+ * If KVM_ARM_VCPU_PMU_V3_STRICT is set and PMU was explicitly
+ * selected, the underlying hardware SLOTS value was read into this
+ * field. Otherwise, it stays 0. All other PMMIR_EL1 fields are RAZ.
+ */
+ p->regval = FIELD_PREP(ARMV8_PMU_SLOTS, vcpu->kvm->arch.pmmir_slots);
+ return true;
+}
+
+static int get_pmmir(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
+ u64 *val)
+{
+ *val = FIELD_PREP(ARMV8_PMU_SLOTS, vcpu->kvm->arch.pmmir_slots);
+ return 0;
+}
+
+static int set_pmmir(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r,
+ u64 val)
+{
+ struct kvm *kvm = vcpu->kvm;
+ u8 slots = FIELD_GET(ARMV8_PMU_SLOTS, val);
+
+ /*
+ * Only the SLOTS field is exposed (get_pmmir returns just that field),
+ * so reject a write that sets any other bit rather than silently
+ * masking it.
+ */
+ if (val & ~(u64)ARMV8_PMU_SLOTS)
+ return -EINVAL;
+
+ guard(mutex)(&kvm->arch.config_lock);
+
+ /*
+ * Once the VM has started PMMIR_EL1 is immutable. Reject any write
+ * that does not match the current value.
+ */
+ if (kvm_vm_has_ran_once(kvm))
+ return slots == kvm->arch.pmmir_slots ? 0 : -EBUSY;
+
+ /*
+ * Only SLOTS = 0 is honored for backwards compatibility with the
+ * old RAZ behavior. Reject any non-zero write that does not match
+ * the current value.
+ */
+ if (!slots)
+ kvm->arch.pmmir_slots = 0;
+ else if (slots != kvm->arch.pmmir_slots)
+ return -EINVAL;
+
+ return 0;
+}
+
static bool access_pmovs(struct kvm_vcpu *vcpu, struct sys_reg_params *p,
const struct sys_reg_desc *r)
{
@@ -3448,7 +3506,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
{ PMU_SYS_REG(PMINTENCLR_EL1),
.access = access_pminten, .reg = PMINTENSET_EL1,
.get_user = get_pmreg, .set_user = set_pmreg },
- { SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi },
+ { PMU_SYS_REG(PMMIR_EL1), .access = access_pmmir, .reset = NULL,
+ .get_user = get_pmmir, .set_user = set_pmmir },
{ SYS_DESC(SYS_MAIR_EL1), access_vm_reg, reset_unknown, MAIR_EL1 },
{ SYS_DESC(SYS_PIRE0_EL1), NULL, reset_unknown, PIRE0_EL1,
@@ -4593,7 +4652,7 @@ static const struct sys_reg_desc cp15_regs[] = {
{ CP15_PMU_SYS_REG(HI, 0, 9, 14, 4), .access = access_pmceid },
{ CP15_PMU_SYS_REG(HI, 0, 9, 14, 5), .access = access_pmceid },
/* PMMIR */
- { CP15_PMU_SYS_REG(DIRECT, 0, 9, 14, 6), .access = trap_raz_wi },
+ { CP15_PMU_SYS_REG(DIRECT, 0, 9, 14, 6), .access = access_pmmir },
/* PRRR/MAIR0 */
{ AA32(LO), Op1( 0), CRn(10), CRm( 2), Op2( 0), access_vm_reg, NULL, MAIR_EL1 },
diff --git a/include/kvm/arm_pmu.h b/include/kvm/arm_pmu.h
index b5e5942204fc..6b4a118d17ca 100644
--- a/include/kvm/arm_pmu.h
+++ b/include/kvm/arm_pmu.h
@@ -75,6 +75,9 @@ void kvm_vcpu_pmu_resync_el0(void);
#define kvm_vcpu_has_pmu(vcpu) \
(vcpu_has_feature(vcpu, KVM_ARM_VCPU_PMU_V3))
+#define kvm_vcpu_has_pmuv3_strict(vcpu) \
+ (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PMU_V3_STRICT))
+
/*
* Updates the vcpu's view of the pmu events for this cpu.
* Must be called before every vcpu run after disabling interrupts, to ensure
@@ -160,6 +163,7 @@ static inline u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1)
}
#define kvm_vcpu_has_pmu(vcpu) ({ false; })
+#define kvm_vcpu_has_pmuv3_strict(vcpu) ({ false; })
static inline void kvm_pmu_update_vcpu_events(struct kvm_vcpu *vcpu) {}
static inline void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu) {}
static inline void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu) {}
diff --git a/tools/testing/selftests/kvm/arm64/get-reg-list.c b/tools/testing/selftests/kvm/arm64/get-reg-list.c
index 0a3a94c4cca1..cfa99979d57c 100644
--- a/tools/testing/selftests/kvm/arm64/get-reg-list.c
+++ b/tools/testing/selftests/kvm/arm64/get-reg-list.c
@@ -532,6 +532,7 @@ static __u64 base_regs[] = {
static __u64 pmu_regs[] = {
ARM64_SYS_REG(3, 0, 9, 14, 1), /* PMINTENSET_EL1 */
ARM64_SYS_REG(3, 0, 9, 14, 2), /* PMINTENCLR_EL1 */
+ ARM64_SYS_REG(3, 0, 9, 14, 6), /* PMMIR_EL1 */
ARM64_SYS_REG(3, 3, 9, 12, 0), /* PMCR_EL0 */
ARM64_SYS_REG(3, 3, 9, 12, 1), /* PMCNTENSET_EL0 */
ARM64_SYS_REG(3, 3, 9, 12, 2), /* PMCNTENCLR_EL0 */
--
2.50.1
^ permalink raw reply related
* [PATCH v3 2/4] KVM: arm64: Advertise STALL_SLOT* in PMCEID1 under strict PMUv3 UAPI
From: Congkai Tan @ 2026-07-22 20:27 UTC (permalink / raw)
To: Oliver Upton, kvmarm, linux-arm-kernel
Cc: Congkai Tan, Marc Zyngier, Joey Gouly, Suzuki K Poulose,
Zenghui Yu, Catalin Marinas, Will Deacon, Paolo Bonzini,
Jonathan Corbet, Shuah Khan, Haris Okanovic, Geoff Blake,
Stanislav Spassov, kvm, linux-doc, linux-kselftest, linux-kernel
In-Reply-To: <20260722202702.4165917-1-congkai@amazon.com>
Skip masking STALL_SLOT, STALL_SLOT_FRONTEND and STALL_SLOT_BACKEND out
of PMCEID1 when KVM_ARM_VCPU_PMU_V3_STRICT is set, because this is when
PMMIR_EL1.SLOTS is exposed to guests, making these events meaningful for
collection.
Change the parameter of compute_pmceid1() from arm_pmu to kvm_vcpu, to
check if KVM_ARM_VCPU_PMU_V3_STRICT is set. Also updated the signature of
compute_pmceid0() for consistency.
Signed-off-by: Congkai Tan <congkai@amazon.com>
Reviewed-by: Geoff Blake <blakgeof@amazon.com>
Reviewed-by: Haris Okanovic <harisokn@amazon.com>
Reviewed-by: Stanislav Spassov <stanspas@amazon.de>
---
arch/arm64/kvm/pmu-emul.c | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)
diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c
index 7ac00423fa8d..bcfd0a91e114 100644
--- a/arch/arm64/kvm/pmu-emul.c
+++ b/arch/arm64/kvm/pmu-emul.c
@@ -838,9 +838,9 @@ static u64 __compute_pmceid(struct arm_pmu *pmu, bool pmceid1)
return ((u64)hi[pmceid1] << 32) | lo[pmceid1];
}
-static u64 compute_pmceid0(struct arm_pmu *pmu)
+static u64 compute_pmceid0(struct kvm_vcpu *vcpu)
{
- u64 val = __compute_pmceid(pmu, 0);
+ u64 val = __compute_pmceid(vcpu->kvm->arch.arm_pmu, 0);
/* always support SW_INCR */
val |= BIT(ARMV8_PMUV3_PERFCTR_SW_INCR);
@@ -849,32 +849,33 @@ static u64 compute_pmceid0(struct arm_pmu *pmu)
return val;
}
-static u64 compute_pmceid1(struct arm_pmu *pmu)
+static u64 compute_pmceid1(struct kvm_vcpu *vcpu)
{
- u64 val = __compute_pmceid(pmu, 1);
+ u64 val = __compute_pmceid(vcpu->kvm->arch.arm_pmu, 1);
/*
- * Don't advertise STALL_SLOT*, as PMMIR_EL0 is handled
- * as RAZ
+ * If KVM_ARM_VCPU_PMU_V3_STRICT is not set, PMMIR_EL1 is
+ * unconditionally RAZ, so don't advertise STALL_SLOT* events.
*/
- val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) |
- BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) |
- BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32));
+ if (!kvm_vcpu_has_pmuv3_strict(vcpu))
+ val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) |
+ BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) |
+ BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32));
+
return val;
}
u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1)
{
- struct arm_pmu *cpu_pmu = vcpu->kvm->arch.arm_pmu;
unsigned long *bmap = vcpu->kvm->arch.pmu_filter;
u64 val, mask = 0;
int base, i, nr_events;
if (!pmceid1) {
- val = compute_pmceid0(cpu_pmu);
+ val = compute_pmceid0(vcpu);
base = 0;
} else {
- val = compute_pmceid1(cpu_pmu);
+ val = compute_pmceid1(vcpu);
base = 32;
}
--
2.50.1
^ permalink raw reply related
* [PATCH v3 0/4] KVM: arm64: Expose PMMIR_EL1.SLOTS to guests
From: Congkai Tan @ 2026-07-22 20:26 UTC (permalink / raw)
To: Oliver Upton, kvmarm, linux-arm-kernel
Cc: Congkai Tan, Marc Zyngier, Joey Gouly, Suzuki K Poulose,
Zenghui Yu, Catalin Marinas, Will Deacon, Paolo Bonzini,
Jonathan Corbet, Shuah Khan, Haris Okanovic, Geoff Blake,
Stanislav Spassov, kvm, linux-doc, linux-kselftest, linux-kernel
Today when the perf tool runs in a guest on cores with PMUv3p4, it fails
to parse the default metrics with "Failure to read '#slots'", since perf
can only read 0 from sysfs caps/slots, which is backed by PMMIR_EL1.SLOTS
that KVM traps as RAZ/WI.
Taking into account backward compatibility and heterogeneous systems, the
exposure of PMMIR_EL1.SLOTS is gated behind a new vCPU feature flag:
- Patch 1 exposes PMMIR_EL1.SLOTS of the selected PMU under the (as yet
unsettable) KVM_ARM_VCPU_PMU_V3_STRICT flag, and adds userspace get/set
for PMMIR_EL1 so that SLOTS can be reset to 0 for backward
compatibility.
- Patch 2 stops masking STALL_SLOT* in PMCEID1 under the flag.
- Patch 3 makes strict-mode userspace configure the number of event
counters via the KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS attribute rather
than PMCR_EL0.N, whose meaning is context-dependent under NV.
- Patch 4 adds the KVM_ARM_VCPU_PMU_V3_STRICT flag itself: when set, KVM
does not create a default PMU during vCPU init, and the VMM must select
one explicitly via KVM_ARM_VCPU_PMU_V3_SET_PMU before the first
KVM_RUN. This flips the gate on and can serve as an umbrella flag for
future PMUv3 UAPI changes.
When the flag is not set, behaviors are unchanged.
v2: https://lore.kernel.org/r/20260702190421.420992-1-congkai@amazon.com
v1: https://lore.kernel.org/r/20260601193954.2103455-1-congkai@amazon.com
v2 -> v3 changes:
- Reordered so the KVM_ARM_VCPU_PMU_V3_STRICT flag / KVM_CAP exposure
comes last (now patch 4), after the flag-guarded behavior patches.
- Reworked on top of Oliver Upton's refinements (thanks!):
* Move the NULL-arm_pmu check into kvm_arm_pmu_v3_init() (returns
-ENXIO) instead of failing at the first KVM_RUN, and report zero
counters from kvm_arm_pmu_get_max_counters() before a PMU is set.
* Documentation and -ENXIO ordering notes for the PMU device
attributes (SET_PMU must precede INIT/FILTER).
- New patch (Oliver): ignore userspace writes to PMCR_EL0.N under the
strict flag and require the KVM_ARM_VCPU_PMU_V3_SET_NR_COUNTERS
attribute, since PMCR_EL0.N is context-dependent under NV.
- Advertise the KVM_CAP_ARM_PMU_V3_STRICT capability.
- Documentation fixes in api.rst (the flag Requires KVM_ARM_VCPU_PMU_V3).
Congkai Tan (3):
KVM: arm64: Expose PMMIR_EL1.SLOTS under strict PMUv3 UAPI
KVM: arm64: Advertise STALL_SLOT* in PMCEID1 under strict PMUv3 UAPI
KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature
Oliver Upton (1):
KVM: arm64: Ignore writes to PMCR_EL0.N when using strict UAPI
Documentation/virt/kvm/api.rst | 11 ++++
Documentation/virt/kvm/devices/vcpu.rst | 11 +++-
arch/arm64/include/asm/kvm_host.h | 5 +-
arch/arm64/include/uapi/asm/kvm.h | 1 +
arch/arm64/kvm/arm.c | 19 ++++--
arch/arm64/kvm/pmu-emul.c | 54 ++++++++++++----
arch/arm64/kvm/sys_regs.c | 64 ++++++++++++++++++-
include/kvm/arm_pmu.h | 4 ++
include/uapi/linux/kvm.h | 1 +
.../selftests/kvm/arm64/get-reg-list.c | 1 +
10 files changed, 149 insertions(+), 22 deletions(-)
base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda
--
2.50.1
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox