* [PATCH bpf-next v3 0/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
0 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
bpf_perf_event_output(), bpf_get_stackid() and bpf_get_stack() of raw
tracepoint and tracing programs share three per-CPU pt_regs slots,
picked by a nest counter. These programs can be preempted, and tasks
then leave the slots out of order, so a sample or stack trace of one
task gets the registers of another. Patch 3 fills in a pt_regs on the
stack of each helper instead and removes the slots.
The slots also limited how deep these helpers nest. Two things need a
limit of their own without them, and both can already be hit today
without raw tracepoint programs:
- bpf_perf_event_output() WARNs when entered a fourth time on a CPU.
- Build ID collection in bpf_get_stack()/bpf_get_stackid() and
bpf_find_vma() take mmap_read_trylock(), which fires the mmap_lock
tracepoints. Programs on them that call these helpers nest into each
other as deep as there are programs. Patch 2 limits it to three
levels.
Results on bpf-next 0fb18ce8a2e9, x86-64, debug config, CONFIG_PREEMPT=y,
preempt=full. "base" is that tree, "v2" adds only patch 3, "v3" all
three.
The slot race, 5 runs of 10s each. Two raw tracepoint programs on
sys_enter call the helper, two threads make syscalls on CPU 0 and a
SCHED_FIFO thread preempts them every 20us. For bpf_perf_event_output()
the reproducer reads the return address of each program's helper call
from its JIT image and checks the IP of every sample; bpf_get_stack()
and bpf_get_stackid() get an empty kernel stack instead, because the
unwinder looks for the other task's sp on the current stack:
base v3
bpf_perf_event_output, wrong IP 7-18 0 per program
bpf_get_stack, empty stacks 362-535 0
bpf_get_stackid, empty stacks 168-364 0
The bpf_get_stackid() case with bpftrace, whose kstack calls it; @a[]
and @b[] count the empty stacks:
bpftrace -e '
rawtracepoint:sys_enter /comm == "dd"/ { @a[kstack(1)] = count(); }
rawtracepoint:sys_enter /comm == "dd"/ { @b[kstack(1)] = count(); }' &
taskset -c 0 dd if=/dev/zero of=/dev/null bs=1 &
taskset -c 0 dd if=/dev/zero of=/dev/null bs=1 &
taskset -c 0 chrt -f 1 perl -e 'select(undef, undef, undef, 2e-5) while 1' &
Nesting. For the WARN, a raw tracepoint or kprobe program calls
bpf_perf_event_output(), and three fentry programs on
perf_event_output() call it too. For mmap_read_trylock(), six programs
on mmap_lock_acquire_returned call bpf_get_stack() with build IDs, or
bpf_find_vma(), while the process does 1000 mmap()/munmap():
base v2 v3
WARN, started from raw tracepoint no yes no
WARN, started from kprobe yes - no
bpf_get_stack, build IDs: depth 4 6 4
calls 1.5M 5.9M 1.5M
bpf_find_vma: depth 6 6 4
calls 5.9M 5.9M 1.5M
Depth counts the programs on the stack; at 4, the last one gets -EBUSY
or the fallback to IPs. The kprobe case wasn't run on v2, which doesn't
change that path.
---
Changes in v3:
- New patch 1: don't WARN when bpf_perf_event_output() nests too deep
(reported by the BPF CI bot).
- New patch 2: limit nesting of mmap_read_trylock() in BPF helpers
(per Alexei's suggestion).
- Patch 3: correct the commit message, bpf_perf_event_output() keeps
its own nesting limit.
- Link to v2: https://lore.kernel.org/r/20261119-bpf-raw-tp-regs-preempt-v2-1-e1e1b90857d6@gmail.com
Changes in v2:
- Use a pt_regs on the stack of each helper and remove the per-CPU
slots, instead of disabling preemption while a slot is in use
(per Andrii's suggestion).
- Link to v1: https://lore.kernel.org/r/20261116-bpf-raw-tp-regs-preempt-v1-1-d01b6dd8573c@gmail.com
---
Mike Bazov (3):
bpf: Don't WARN when bpf_perf_event_output() nests too deep
bpf: Limit nesting of mmap_read_trylock() in BPF helpers
bpf: Use an on-stack pt_regs in the raw tracepoint helpers
kernel/bpf/mmap_unlock_work.h | 16 +++++++--
kernel/trace/bpf_trace.c | 80 +++++++------------------------------------
2 files changed, 26 insertions(+), 70 deletions(-)
---
base-commit: 8835113107b2cf75df33a71e265cdeec5d5d13e0
change-id: 20261116-bpf-raw-tp-regs-preempt-1f7019989ab9
Best regards,
--
Mike Bazov <mike.bazov@gmail.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 0/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
0 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
bpf_perf_event_output(), bpf_get_stackid() and bpf_get_stack() of raw
tracepoint and tracing programs share three per-CPU pt_regs slots,
picked by a nest counter. These programs can be preempted, and tasks
then leave the slots out of order, so a sample or stack trace of one
task gets the registers of another. Patch 3 fills in a pt_regs on the
stack of each helper instead and removes the slots.
The slots also limited how deep these helpers nest. Two things need a
limit of their own without them, and both can already be hit today
without raw tracepoint programs:
- bpf_perf_event_output() WARNs when entered a fourth time on a CPU.
- Build ID collection in bpf_get_stack()/bpf_get_stackid() and
bpf_find_vma() take mmap_read_trylock(), which fires the mmap_lock
tracepoints. Programs on them that call these helpers nest into each
other as deep as there are programs. Patch 2 limits it to three
levels.
Results on bpf-next 0fb18ce8a2e9, x86-64, debug config, CONFIG_PREEMPT=y,
preempt=full. "base" is that tree, "v2" adds only patch 3, "v3" all
three.
The slot race, 5 runs of 10s each. Two raw tracepoint programs on
sys_enter call the helper, two threads make syscalls on CPU 0 and a
SCHED_FIFO thread preempts them every 20us. For bpf_perf_event_output()
the reproducer reads the return address of each program's helper call
from its JIT image and checks the IP of every sample; bpf_get_stack()
and bpf_get_stackid() get an empty kernel stack instead, because the
unwinder looks for the other task's sp on the current stack:
base v3
bpf_perf_event_output, wrong IP 7-18 0 per program
bpf_get_stack, empty stacks 362-535 0
bpf_get_stackid, empty stacks 168-364 0
The bpf_get_stackid() case with bpftrace, whose kstack calls it; @a[]
and @b[] count the empty stacks:
bpftrace -e '
rawtracepoint:sys_enter /comm == "dd"/ { @a[kstack(1)] = count(); }
rawtracepoint:sys_enter /comm == "dd"/ { @b[kstack(1)] = count(); }' &
taskset -c 0 dd if=/dev/zero of=/dev/null bs=1 &
taskset -c 0 dd if=/dev/zero of=/dev/null bs=1 &
taskset -c 0 chrt -f 1 perl -e 'select(undef, undef, undef, 2e-5) while 1' &
Nesting. For the WARN, a raw tracepoint or kprobe program calls
bpf_perf_event_output(), and three fentry programs on
perf_event_output() call it too. For mmap_read_trylock(), six programs
on mmap_lock_acquire_returned call bpf_get_stack() with build IDs, or
bpf_find_vma(), while the process does 1000 mmap()/munmap():
base v2 v3
WARN, started from raw tracepoint no yes no
WARN, started from kprobe yes - no
bpf_get_stack, build IDs: depth 4 6 4
calls 1.5M 5.9M 1.5M
bpf_find_vma: depth 6 6 4
calls 5.9M 5.9M 1.5M
Depth counts the programs on the stack; at 4, the last one gets -EBUSY
or the fallback to IPs. The kprobe case wasn't run on v2, which doesn't
change that path.
---
Changes in v3:
- New patch 1: don't WARN when bpf_perf_event_output() nests too deep
(reported by the BPF CI bot).
- New patch 2: limit nesting of mmap_read_trylock() in BPF helpers
(per Alexei's suggestion).
- Patch 3: correct the commit message, bpf_perf_event_output() keeps
its own nesting limit.
- Link to v2: https://lore.kernel.org/r/20261119-bpf-raw-tp-regs-preempt-v2-1-e1e1b90857d6@gmail.com
Changes in v2:
- Use a pt_regs on the stack of each helper and remove the per-CPU
slots, instead of disabling preemption while a slot is in use
(per Andrii's suggestion).
- Link to v1: https://lore.kernel.org/r/20261116-bpf-raw-tp-regs-preempt-v1-1-d01b6dd8573c@gmail.com
---
Mike Bazov (3):
bpf: Don't WARN when bpf_perf_event_output() nests too deep
bpf: Limit nesting of mmap_read_trylock() in BPF helpers
bpf: Use an on-stack pt_regs in the raw tracepoint helpers
kernel/bpf/mmap_unlock_work.h | 16 +++++++--
kernel/trace/bpf_trace.c | 80 +++++++------------------------------------
2 files changed, 26 insertions(+), 70 deletions(-)
---
base-commit: 8835113107b2cf75df33a71e265cdeec5d5d13e0
change-id: 20261116-bpf-raw-tp-regs-preempt-1f7019989ab9
Best regards,
--
Mike Bazov <mike.bazov@gmail.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep
2026-11-20 10:59 ` Mike Bazov via B4 Relay
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
-1 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
bpf_perf_event_output() takes one of three per-CPU perf_sample_data
slots per nesting level, and fails with -EBUSY and a WARN_ON_ONCE() when
it is entered a fourth time on a CPU. Programs get there without any
bug: a kprobe program calls it, and fentry programs on
perf_event_output() call it again from inside. Each program's own
recursion check lets it in once, so three of them reach the fourth
level. With panic_on_warn, that takes the machine down.
Keep the -EBUSY and drop the warning, as commit 3880cdbed1c4 ("bpf: Fix
WARN() in get_bpf_raw_tp_regs") did for the raw tracepoint regs slots.
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/trace/bpf_trace.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index b4ff9d4e8..8910d5d05 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -702,7 +702,7 @@ BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map,
sds = this_cpu_ptr(&bpf_trace_sds);
nest_level = this_cpu_inc_return(bpf_trace_nest_level);
- if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
+ if (nest_level > ARRAY_SIZE(sds->sds)) {
err = -EBUSY;
goto out;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
0 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
From: Mike Bazov <mike.bazov@gmail.com>
bpf_perf_event_output() takes one of three per-CPU perf_sample_data
slots per nesting level, and fails with -EBUSY and a WARN_ON_ONCE() when
it is entered a fourth time on a CPU. Programs get there without any
bug: a kprobe program calls it, and fentry programs on
perf_event_output() call it again from inside. Each program's own
recursion check lets it in once, so three of them reach the fourth
level. With panic_on_warn, that takes the machine down.
Keep the -EBUSY and drop the warning, as commit 3880cdbed1c4 ("bpf: Fix
WARN() in get_bpf_raw_tp_regs") did for the raw tracepoint regs slots.
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/trace/bpf_trace.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index b4ff9d4e8..8910d5d05 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -702,7 +702,7 @@ BPF_CALL_5(bpf_perf_event_output, struct pt_regs *, regs, struct bpf_map *, map,
sds = this_cpu_ptr(&bpf_trace_sds);
nest_level = this_cpu_inc_return(bpf_trace_nest_level);
- if (WARN_ON_ONCE(nest_level > ARRAY_SIZE(sds->sds))) {
+ if (nest_level > ARRAY_SIZE(sds->sds)) {
err = -EBUSY;
goto out;
}
--
2.43.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 2/3] bpf: Limit nesting of mmap_read_trylock() in BPF helpers
2026-11-20 10:59 ` Mike Bazov via B4 Relay
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
-1 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
bpf_get_stack() and bpf_get_stackid() with BPF_F_USER_BUILD_ID, and
bpf_find_vma(), take mmap_read_trylock() on the current mm. That fires
the mmap_lock tracepoints, so a program on one of them that calls these
helpers runs the tracepoint again from inside itself:
prog 1 on mmap_lock_acquire_returned
bpf_get_stack(BPF_F_USER_STACK | BPF_F_USER_BUILD_ID)
mmap_read_trylock()
prog 2 on mmap_lock_acquire_returned
bpf_get_stack(...)
mmap_read_trylock()
...
The recursion check of each program only stops it from running inside
itself, so this nests as deep as there are programs attached, and each
level runs all programs that aren't on the stack yet: the work per
tracepoint hit grows with the factorial of the number of programs. With
irqs disabled, the per-CPU irq_work already allows only one level; with
irqs enabled nothing limits it.
Count the levels per CPU with irqs enabled too and fail with -EBUSY past
three, the same depth as the other per-CPU nesting limits of BPF
tracing. The helpers then fall back as they do when the trylock fails:
bpf_get_stack() and bpf_get_stackid() return IPs instead of build IDs,
and bpf_find_vma() returns -EBUSY.
The counter is per CPU, and programs with irqs enabled can be
preempted, so a preempted holder counts against the other tasks on its
CPU. They can hit the limit early and get the fallback; they never get
wrong data.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/bpf/mmap_unlock_work.h | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h
index 1834db20b..02af564e6 100644
--- a/kernel/bpf/mmap_unlock_work.h
+++ b/kernel/bpf/mmap_unlock_work.h
@@ -13,6 +13,7 @@ struct mmap_unlock_irq_work {
struct irq_work irq_work;
struct mm_struct *mm;
atomic_t active;
+ int nest_level;
};
DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
@@ -28,8 +29,16 @@ static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void)
{
struct mmap_unlock_irq_work *work;
- if (!irqs_disabled())
- return NULL;
+ if (!irqs_disabled()) {
+ /*
+ * mmap_read_trylock() fires the mmap_lock tracepoints, and
+ * programs on them can get here again.
+ */
+ if (this_cpu_inc_return(mmap_unlock_work.nest_level) <= 3)
+ return NULL;
+ this_cpu_dec(mmap_unlock_work.nest_level);
+ return ERR_PTR(-EBUSY);
+ }
/*
* PREEMPT_RT does not allow to trylock mmap sem in interrupt
@@ -51,12 +60,15 @@ bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work)
{
if (work)
atomic_set_release(&work->active, 0);
+ else
+ this_cpu_dec(mmap_unlock_work.nest_level);
}
static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm)
{
if (!work) {
mmap_read_unlock(mm);
+ bpf_mmap_unlock_guard_put(NULL);
} else {
work->mm = mm;
--
2.43.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 2/3] bpf: Limit nesting of mmap_read_trylock() in BPF helpers
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
0 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
From: Mike Bazov <mike.bazov@gmail.com>
bpf_get_stack() and bpf_get_stackid() with BPF_F_USER_BUILD_ID, and
bpf_find_vma(), take mmap_read_trylock() on the current mm. That fires
the mmap_lock tracepoints, so a program on one of them that calls these
helpers runs the tracepoint again from inside itself:
prog 1 on mmap_lock_acquire_returned
bpf_get_stack(BPF_F_USER_STACK | BPF_F_USER_BUILD_ID)
mmap_read_trylock()
prog 2 on mmap_lock_acquire_returned
bpf_get_stack(...)
mmap_read_trylock()
...
The recursion check of each program only stops it from running inside
itself, so this nests as deep as there are programs attached, and each
level runs all programs that aren't on the stack yet: the work per
tracepoint hit grows with the factorial of the number of programs. With
irqs disabled, the per-CPU irq_work already allows only one level; with
irqs enabled nothing limits it.
Count the levels per CPU with irqs enabled too and fail with -EBUSY past
three, the same depth as the other per-CPU nesting limits of BPF
tracing. The helpers then fall back as they do when the trylock fails:
bpf_get_stack() and bpf_get_stackid() return IPs instead of build IDs,
and bpf_find_vma() returns -EBUSY.
The counter is per CPU, and programs with irqs enabled can be
preempted, so a preempted holder counts against the other tasks on its
CPU. They can hit the limit early and get the fallback; they never get
wrong data.
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/bpf/mmap_unlock_work.h | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h
index 1834db20b..02af564e6 100644
--- a/kernel/bpf/mmap_unlock_work.h
+++ b/kernel/bpf/mmap_unlock_work.h
@@ -13,6 +13,7 @@ struct mmap_unlock_irq_work {
struct irq_work irq_work;
struct mm_struct *mm;
atomic_t active;
+ int nest_level;
};
DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
@@ -28,8 +29,16 @@ static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void)
{
struct mmap_unlock_irq_work *work;
- if (!irqs_disabled())
- return NULL;
+ if (!irqs_disabled()) {
+ /*
+ * mmap_read_trylock() fires the mmap_lock tracepoints, and
+ * programs on them can get here again.
+ */
+ if (this_cpu_inc_return(mmap_unlock_work.nest_level) <= 3)
+ return NULL;
+ this_cpu_dec(mmap_unlock_work.nest_level);
+ return ERR_PTR(-EBUSY);
+ }
/*
* PREEMPT_RT does not allow to trylock mmap sem in interrupt
@@ -51,12 +60,15 @@ bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work)
{
if (work)
atomic_set_release(&work->active, 0);
+ else
+ this_cpu_dec(mmap_unlock_work.nest_level);
}
static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm)
{
if (!work) {
mmap_read_unlock(mm);
+ bpf_mmap_unlock_guard_put(NULL);
} else {
work->mm = mm;
--
2.43.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 3/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
2026-11-20 10:59 ` Mike Bazov via B4 Relay
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
-1 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
bpf_perf_event_output(), bpf_get_stackid() and bpf_get_stack() of raw
tracepoint and tracing programs have no pt_regs, so they fill in one of
three per-CPU slots, picked by a per-CPU nest counter:
get_bpf_raw_tp_regs() level++, use regs[level - 1]
put_bpf_raw_tp_regs() level--
That only works if the users of a CPU leave in the reverse order they
came, as a task, an interrupt and an NMI do. These programs run with
migration disabled but preemption enabled, so tasks can leave in any
order:
task A task B task C
get: regs[0]
<preempted>
get: regs[1]
<preempted>
put: level 1
get: regs[1]
fills in its regs
uses regs[1]
B sends a sample or a stack trace with the registers of C. Profilers and
tracers then attribute it to the wrong task, and nothing reports an
error. The per-CPU recursion counter of a program doesn't prevent it:
the slots are shared by all programs, so it takes two of them.
Fill in a pt_regs on the stack of each helper instead, as
bpf_get_stack_raw_tp_sleepable() already does, and remove the slots.
Each caller then has its own registers, however it nests or gets
preempted. This also removes the three-level nesting limit the slots put
on the three helpers. bpf_perf_event_output() keeps its own limit, and
the mmap_read_trylock() for build IDs has a separate one.
The cost is sizeof(struct pt_regs) of stack per call, 168 bytes on
x86-64, in helpers that can also run in IRQ and NMI context, and
zeroing it on every call. It has to be zeroed: perf_fetch_caller_regs()
sets only a few registers, and a sample with PERF_SAMPLE_REGS_INTR
copies all of them to user space.
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/trace/bpf_trace.c | 78 +++++++-----------------------------------------
1 file changed, 11 insertions(+), 67 deletions(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 8910d5d05..5c49c139d 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1624,51 +1624,13 @@ pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
}
}
-/*
- * bpf_raw_tp_regs are separate from bpf_pt_regs used from skb/xdp
- * to avoid potential recursive reuse issue when/if tracepoints are added
- * inside bpf_*_event_output, bpf_get_stackid and/or bpf_get_stack.
- *
- * Since raw tracepoints run despite bpf_prog_active, support concurrent usage
- * in normal, irq, and nmi context.
- */
-struct bpf_raw_tp_regs {
- struct pt_regs regs[3];
-};
-static DEFINE_PER_CPU(struct bpf_raw_tp_regs, bpf_raw_tp_regs);
-static DEFINE_PER_CPU(int, bpf_raw_tp_nest_level);
-static struct pt_regs *get_bpf_raw_tp_regs(void)
-{
- struct bpf_raw_tp_regs *tp_regs = this_cpu_ptr(&bpf_raw_tp_regs);
- int nest_level = this_cpu_inc_return(bpf_raw_tp_nest_level);
-
- if (nest_level > ARRAY_SIZE(tp_regs->regs)) {
- this_cpu_dec(bpf_raw_tp_nest_level);
- return ERR_PTR(-EBUSY);
- }
-
- return &tp_regs->regs[nest_level - 1];
-}
-
-static void put_bpf_raw_tp_regs(void)
-{
- this_cpu_dec(bpf_raw_tp_nest_level);
-}
-
BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags, void *, data, u64, size)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
-
- perf_fetch_caller_regs(regs);
- ret = ____bpf_perf_event_output(regs, map, flags, data, size);
+ struct pt_regs regs = {};
- put_bpf_raw_tp_regs();
- return ret;
+ perf_fetch_caller_regs(®s);
+ return ____bpf_perf_event_output(®s, map, flags, data, size);
}
static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
@@ -1689,18 +1651,12 @@ extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto;
BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
+ struct pt_regs regs = {};
- perf_fetch_caller_regs(regs);
+ perf_fetch_caller_regs(®s);
/* similar to bpf_perf_event_output_tp, but pt_regs fetched differently */
- ret = bpf_get_stackid((unsigned long) regs, (unsigned long) map,
- flags, 0, 0);
- put_bpf_raw_tp_regs();
- return ret;
+ return bpf_get_stackid((unsigned long) ®s, (unsigned long) map,
+ flags, 0, 0);
}
static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
@@ -1715,17 +1671,11 @@ static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
+ struct pt_regs regs = {};
- perf_fetch_caller_regs(regs);
- ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
- (unsigned long) size, flags, 0);
- put_bpf_raw_tp_regs();
- return ret;
+ perf_fetch_caller_regs(®s);
+ return bpf_get_stack((unsigned long) ®s, (unsigned long) buf,
+ (unsigned long) size, flags, 0);
}
static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
@@ -1741,12 +1691,6 @@ static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
BPF_CALL_4(bpf_get_stack_raw_tp_sleepable, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
- /*
- * Don't take a per-CPU bpf_raw_tp_regs slot here: those slots are a
- * LIFO nest counter and are vulnerable to slot overwrite and
- * corruption across sleep. Sleepable programs never run in
- * IRQ/NMI context, so a local pt_regs is sufficient.
- */
struct pt_regs regs = {};
perf_fetch_caller_regs(®s);
--
2.43.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH bpf-next v3 3/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers
@ 2026-11-20 10:59 ` Mike Bazov via B4 Relay
0 siblings, 0 replies; 9+ messages in thread
From: Mike Bazov via B4 Relay @ 2026-11-20 10:59 UTC (permalink / raw)
To: Song Liu, Jiri Olsa, Ihor Solodrai, Alexei Starovoitov,
Daniel Borkmann, Andrii Nakryiko, Eduard Zingerman,
Kumar Kartikeya Dwivedi, Martin KaFai Lau, Yonghong Song,
Emil Tsalapatis, KP Singh, Matt Bobrowski, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers
Cc: bpf, linux-kernel, linux-trace-kernel, Mike Bazov
From: Mike Bazov <mike.bazov@gmail.com>
bpf_perf_event_output(), bpf_get_stackid() and bpf_get_stack() of raw
tracepoint and tracing programs have no pt_regs, so they fill in one of
three per-CPU slots, picked by a per-CPU nest counter:
get_bpf_raw_tp_regs() level++, use regs[level - 1]
put_bpf_raw_tp_regs() level--
That only works if the users of a CPU leave in the reverse order they
came, as a task, an interrupt and an NMI do. These programs run with
migration disabled but preemption enabled, so tasks can leave in any
order:
task A task B task C
get: regs[0]
<preempted>
get: regs[1]
<preempted>
put: level 1
get: regs[1]
fills in its regs
uses regs[1]
B sends a sample or a stack trace with the registers of C. Profilers and
tracers then attribute it to the wrong task, and nothing reports an
error. The per-CPU recursion counter of a program doesn't prevent it:
the slots are shared by all programs, so it takes two of them.
Fill in a pt_regs on the stack of each helper instead, as
bpf_get_stack_raw_tp_sleepable() already does, and remove the slots.
Each caller then has its own registers, however it nests or gets
preempted. This also removes the three-level nesting limit the slots put
on the three helpers. bpf_perf_event_output() keeps its own limit, and
the mmap_read_trylock() for build IDs has a separate one.
The cost is sizeof(struct pt_regs) of stack per call, 168 bytes on
x86-64, in helpers that can also run in IRQ and NMI context, and
zeroing it on every call. It has to be zeroed: perf_fetch_caller_regs()
sets only a few registers, and a sample with PERF_SAMPLE_REGS_INTR
copies all of them to user space.
Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
---
kernel/trace/bpf_trace.c | 78 +++++++-----------------------------------------
1 file changed, 11 insertions(+), 67 deletions(-)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 8910d5d05..5c49c139d 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1624,51 +1624,13 @@ pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
}
}
-/*
- * bpf_raw_tp_regs are separate from bpf_pt_regs used from skb/xdp
- * to avoid potential recursive reuse issue when/if tracepoints are added
- * inside bpf_*_event_output, bpf_get_stackid and/or bpf_get_stack.
- *
- * Since raw tracepoints run despite bpf_prog_active, support concurrent usage
- * in normal, irq, and nmi context.
- */
-struct bpf_raw_tp_regs {
- struct pt_regs regs[3];
-};
-static DEFINE_PER_CPU(struct bpf_raw_tp_regs, bpf_raw_tp_regs);
-static DEFINE_PER_CPU(int, bpf_raw_tp_nest_level);
-static struct pt_regs *get_bpf_raw_tp_regs(void)
-{
- struct bpf_raw_tp_regs *tp_regs = this_cpu_ptr(&bpf_raw_tp_regs);
- int nest_level = this_cpu_inc_return(bpf_raw_tp_nest_level);
-
- if (nest_level > ARRAY_SIZE(tp_regs->regs)) {
- this_cpu_dec(bpf_raw_tp_nest_level);
- return ERR_PTR(-EBUSY);
- }
-
- return &tp_regs->regs[nest_level - 1];
-}
-
-static void put_bpf_raw_tp_regs(void)
-{
- this_cpu_dec(bpf_raw_tp_nest_level);
-}
-
BPF_CALL_5(bpf_perf_event_output_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags, void *, data, u64, size)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
-
- perf_fetch_caller_regs(regs);
- ret = ____bpf_perf_event_output(regs, map, flags, data, size);
+ struct pt_regs regs = {};
- put_bpf_raw_tp_regs();
- return ret;
+ perf_fetch_caller_regs(®s);
+ return ____bpf_perf_event_output(®s, map, flags, data, size);
}
static const struct bpf_func_proto bpf_perf_event_output_proto_raw_tp = {
@@ -1689,18 +1651,12 @@ extern const struct bpf_func_proto bpf_xdp_get_buff_len_trace_proto;
BPF_CALL_3(bpf_get_stackid_raw_tp, struct bpf_raw_tracepoint_args *, args,
struct bpf_map *, map, u64, flags)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
+ struct pt_regs regs = {};
- perf_fetch_caller_regs(regs);
+ perf_fetch_caller_regs(®s);
/* similar to bpf_perf_event_output_tp, but pt_regs fetched differently */
- ret = bpf_get_stackid((unsigned long) regs, (unsigned long) map,
- flags, 0, 0);
- put_bpf_raw_tp_regs();
- return ret;
+ return bpf_get_stackid((unsigned long) ®s, (unsigned long) map,
+ flags, 0, 0);
}
static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
@@ -1715,17 +1671,11 @@ static const struct bpf_func_proto bpf_get_stackid_proto_raw_tp = {
BPF_CALL_4(bpf_get_stack_raw_tp, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
- struct pt_regs *regs = get_bpf_raw_tp_regs();
- int ret;
-
- if (IS_ERR(regs))
- return PTR_ERR(regs);
+ struct pt_regs regs = {};
- perf_fetch_caller_regs(regs);
- ret = bpf_get_stack((unsigned long) regs, (unsigned long) buf,
- (unsigned long) size, flags, 0);
- put_bpf_raw_tp_regs();
- return ret;
+ perf_fetch_caller_regs(®s);
+ return bpf_get_stack((unsigned long) ®s, (unsigned long) buf,
+ (unsigned long) size, flags, 0);
}
static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
@@ -1741,12 +1691,6 @@ static const struct bpf_func_proto bpf_get_stack_proto_raw_tp = {
BPF_CALL_4(bpf_get_stack_raw_tp_sleepable, struct bpf_raw_tracepoint_args *, args,
void *, buf, u32, size, u64, flags)
{
- /*
- * Don't take a per-CPU bpf_raw_tp_regs slot here: those slots are a
- * LIFO nest counter and are vulnerable to slot overwrite and
- * corruption across sleep. Sleepable programs never run in
- * IRQ/NMI context, so a local pt_regs is sufficient.
- */
struct pt_regs regs = {};
perf_fetch_caller_regs(®s);
--
2.43.0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep
2026-11-20 10:59 ` Mike Bazov via B4 Relay
(?)
@ 2026-11-20 11:03 ` sashiko-bot
-1 siblings, 0 replies; 9+ messages in thread
From: sashiko-bot @ 2026-11-20 11:03 UTC (permalink / raw)
To: Mike Bazov; +Cc: bpf, linux-trace-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] bpf: missing Fixes: trailer for bpf_perf_event_output()
WARN_ON_ONCE() fix
--
commit ae922aea5dc145502d3e9cea52835a863d8d8bb7
Author: Mike Bazov <mike.bazov@gmail.com>
> bpf: Don't WARN when bpf_perf_event_output() nests too deep
>
> bpf_perf_event_output() takes one of three per-CPU perf_sample_data
> slots per nesting level, and fails with -EBUSY and a WARN_ON_ONCE() when
> it is entered a fourth time on a CPU. Programs get there without any
> bug: a kprobe program calls it, and fentry programs on
> perf_event_output() call it again from inside. Each program's own
> recursion check lets it in once, so three of them reach the fourth
> level. With panic_on_warn, that takes the machine down.
>
> Keep the -EBUSY and drop the warning, as commit 3880cdbed1c4 ("bpf: Fix
> WARN() in get_bpf_raw_tp_regs") did for the raw tracepoint regs slots.
>
> Signed-off-by: Mike Bazov <mike.bazov@gmail.com>
[Severity: Low]
As with commit 3880cdbed1c4 ("bpf: Fix WARN() in get_bpf_raw_tp_regs"),
should this include a Fixes: tag for the commit that added the
WARN_ON_ONCE()?
Fixes: 9594dc3c7e71 ("bpf: fix nested bpf tracepoints with per-cpu data")
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261120-bpf-raw-tp-regs-preempt-v3-0-ec109009a5d2@gmail.com?part=1
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-11-20 11:03 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20 10:59 [PATCH bpf-next v3 0/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov
2026-11-20 10:59 ` Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 1/3] bpf: Don't WARN when bpf_perf_event_output() nests too deep Mike Bazov
2026-11-20 10:59 ` Mike Bazov via B4 Relay
2026-11-20 11:03 ` sashiko-bot
2026-11-20 10:59 ` [PATCH bpf-next v3 2/3] bpf: Limit nesting of mmap_read_trylock() in BPF helpers Mike Bazov
2026-11-20 10:59 ` Mike Bazov via B4 Relay
2026-11-20 10:59 ` [PATCH bpf-next v3 3/3] bpf: Use an on-stack pt_regs in the raw tracepoint helpers Mike Bazov
2026-11-20 10:59 ` Mike Bazov via B4 Relay
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.