* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 21:02 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Alexei Starovoitov, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner, Peter Zijlstra, Linus Torvalds
In-Reply-To: <3c0df437-f6e5-47c6-aed5-f4cc26fe627a@efficios.com>
On Fri, 9 Jan 2026 15:21:19 -0500
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> * preempt disable/enable pair: 1.1 ns
> * srcu-fast lock/unlock: 1.5 ns
>
> CONFIG_RCU_REF_SCALE_TEST=y
> * migrate disable/enable pair: 3.0 ns
> * calls to migrate disable/enable pair within noinline functions: 17.0 ns
>
> CONFIG_RCU_REF_SCALE_TEST=m
> * migrate disable/enable pair: 22.0 ns
OUCH! So migrate disable/enable has a much larger overhead when executed in
a module than in the kernel? This means all spin_locks() in modules
converted to mutexes in PREEMPT_RT are taking this hit!
It looks like it has to allow access to the rq->nr_pinned. There's a hack to
expose this part of the rq struct for in-kernel by the following:
kernel/sched/rq-offsets.c: DEFINE(RQ_nr_pinned, offsetof(struct rq, nr_pinned));
Then for the in-kernel code we have:
#define this_rq_raw() arch_raw_cpu_ptr(&runqueues)
#else
#define this_rq_raw() PERCPU_PTR(&runqueues)
#endif
#define this_rq_pinned() (*(unsigned int *)((void *)this_rq_raw() + RQ_nr_pinned))
Looking at the scheduler code, the rq->nr_pinned is referenced by a static
function with:
static inline bool rq_has_pinned_tasks(struct rq *rq)
{
return rq->nr_pinned;
}
Which is only referenced in hotplug code and a balance_push() path in load
balancing. Does this variable really need to be in the runqueue struct?
Why not just make it a per-cpu variable. Maybe call it cpu_nr_pinned_tasks,
and export that for all to use?
It will not only fix the discrepancy between the overhead of
migrate_disable/enable in modules vs in-kernel. But it also removes the
hack to expose a portion of the runqueue.
-- Steve
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Mathieu Desnoyers @ 2026-01-09 20:21 UTC (permalink / raw)
To: Steven Rostedt, Alexei Starovoitov
Cc: LKML, Linux trace kernel, bpf, Masami Hiramatsu, Paul E. McKenney,
Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <20260109141930.6deb2a0a@gandalf.local.home>
On 2026-01-09 14:19, Steven Rostedt wrote:
> On Fri, 9 Jan 2026 11:10:16 -0800
> Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
>
> \> >
>>> We also have to consider that migrate disable is *not* cheap at all
>>> compared to preempt disable.
>>
>> Looks like your complaint comes from lack of engagement in kernel
>> development.
>
> No need to make comments like that. The Linux kernel is an ocean of code.
> It's very hard to keep up on everything that is happening. I knew of work
> being done on migrate_disable but I didn't know what the impacts of that
> work was. Mathieu is still very much involved and engaged in kernel
> development.
Thanks Steven. I guess Alexei missed my recent involvement in other
areas of the kernel.
As Steven pointed out, the kernel is vast, so I cannot keep up with
the progress on every single topic. That being said, I very recently
(about 1 month ago) tried using migrate disable for the RSS tracking
improvements (hierarchical percpu counters), and found that the overhead
of migrate disable was large compared to preempt disable. The generated
assembler is also orders of magnitude larger (on x86-64).
Creating small placeholder functions which just call preempt/migrate
disable and enable for a preempt RT build:
0000000000002a20 <test_preempt_disable>:
2a20: f3 0f 1e fa endbr64
2a24: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 2a2b <test_preempt_disable+0xb>
2a2b: e9 00 00 00 00 jmp 2a30 <test_preempt_disable+0x10>
0000000000002a40 <test_preempt_enable>:
2a40: f3 0f 1e fa endbr64
2a44: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 2a4b <test_preempt_enable+0xb>
2a4b: 74 05 je 2a52 <test_preempt_enable+0x12>
2a4d: e9 00 00 00 00 jmp 2a52 <test_preempt_enable+0x12>
2a52: e8 00 00 00 00 call 2a57 <test_preempt_enable+0x17>
2a57: e9 00 00 00 00 jmp 2a5c <test_preempt_enable+0x1c>
0000000000002920 <test_migrate_disable>:
2920: f3 0f 1e fa endbr64
2924: 65 48 8b 15 00 00 00 mov %gs:0x0(%rip),%rdx # 292c <test_migrate_disable+0xc>
292b: 00
292c: 0f b7 82 38 07 00 00 movzwl 0x738(%rdx),%eax
2933: 66 85 c0 test %ax,%ax
2936: 74 0f je 2947 <test_migrate_disable+0x27>
2938: 83 c0 01 add $0x1,%eax
293b: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
2942: e9 00 00 00 00 jmp 2947 <test_migrate_disable+0x27>
2947: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 294e <test_migrate_disable+0x2e>
294e: 65 48 8b 05 00 00 00 mov %gs:0x0(%rip),%rax # 2956 <test_migrate_disable+0x36>
2955: 00
2956: 83 80 00 00 00 00 01 addl $0x1,0x0(%rax)
295d: b8 01 00 00 00 mov $0x1,%eax
2962: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
2969: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 2970 <test_migrate_disable+0x50>
2970: 74 05 je 2977 <test_migrate_disable+0x57>
2972: e9 00 00 00 00 jmp 2977 <test_migrate_disable+0x57>
2977: e8 00 00 00 00 call 297c <test_migrate_disable+0x5c>
297c: e9 00 00 00 00 jmp 2981 <test_migrate_disable+0x61>
00000000000029a0 <test_migrate_enable>:
29a0: f3 0f 1e fa endbr64
29a4: 65 48 8b 15 00 00 00 mov %gs:0x0(%rip),%rdx # 29ac <test_migrate_enable+0xc>
29ab: 00
29ac: 0f b7 82 38 07 00 00 movzwl 0x738(%rdx),%eax
29b3: 66 85 c0 test %ax,%ax
29b6: 74 0f je 29c7 <test_migrate_enable+0x27>
29b8: 83 c0 01 add $0x1,%eax
29bb: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
29c2: e9 00 00 00 00 jmp 29c7 <test_migrate_enable+0x27>
29c7: 65 ff 05 00 00 00 00 incl %gs:0x0(%rip) # 29ce <test_migrate_enable+0x2e>
29ce: 65 48 8b 05 00 00 00 mov %gs:0x0(%rip),%rax # 29d6 <test_migrate_enable+0x36>
29d5: 00
29d6: 83 80 00 00 00 00 01 addl $0x1,0x0(%rax)
29dd: b8 01 00 00 00 mov $0x1,%eax
29e2: 66 89 82 38 07 00 00 mov %ax,0x738(%rdx)
29e9: 65 ff 0d 00 00 00 00 decl %gs:0x0(%rip) # 29f0 <test_migrate_enable+0x50>
29f0: 74 05 je 29f7 <test_migrate_enable+0x57>
29f2: e9 00 00 00 00 jmp 29f7 <test_migrate_enable+0x57>
29f7: e8 00 00 00 00 call 29fc <test_migrate_enable+0x5c>
29fc: e9 00 00 00 00 jmp 2a01 <test_migrate_enable+0x61>
>
>> migrate_disable _was_ not cheap.
>> Try to benchmark it now.
>> It's inlined. It's a fraction of extra overhead on top of preempt_disable.
>
> It would be good to have a benchmark of the two. What about fast_srcu? Is
> that fast enough to replace the preempt_disable()? If so, then could we
> just make this the same for both RT and !RT?
I've modified kernel/rcu/refscale.c to compare those:
AMD EPYC 9654 96-Core Processor, kernel baseline: v6.18.1
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_LAZY is not set
# CONFIG_PREEMPT_RT is not set
* preempt disable/enable pair: 1.1 ns
* srcu-fast lock/unlock: 1.5 ns
CONFIG_RCU_REF_SCALE_TEST=y
* migrate disable/enable pair: 3.0 ns
* calls to migrate disable/enable pair within noinline functions: 17.0 ns
CONFIG_RCU_REF_SCALE_TEST=m
* migrate disable/enable pair: 22.0 ns
When I attempted using migrate disable, I configured refscale as
a module, which gave me the appalling 22 ns overhead. It looks like
the implementation of migrate disable/enable now differs depending on
whether it's used from the core kernel or from a module. That's rather
unexpected.
It seems to be done on purpose though (INSTANTIATE_EXPORTED_MIGRATE_DISABLE)
to work around the fact that it is not possible to export the runqueues
variable.
That's the kind of compilation context dependent overhead variability I'd
rather avoid in the implementation of the tracepoint instrumentation API.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
^ permalink raw reply
* Re: [PATCH v2 2/3] Documentation: tracing: Add PCI controller event documentation
From: Bjorn Helgaas @ 2026-01-09 20:18 UTC (permalink / raw)
To: Shawn Lin
Cc: Manivannan Sadhasivam, Bjorn Helgaas, linux-rockchip, linux-pci,
linux-trace-kernel, linux-doc, Steven Rostedt, Masami Hiramatsu
In-Reply-To: <1767929389-143957-3-git-send-email-shawn.lin@rock-chips.com>
On Fri, Jan 09, 2026 at 11:29:48AM +0800, Shawn Lin wrote:
> The available tracepoint, pcie_ltssm_state_transition, monitors the LTSSM state
> transistion for debugging purpose. Add description about it.
s/transistion/transition/
> Documentation/trace/events-pci-conotroller.rst | 41 ++++++++++++++++++++++++++
s/events-pci-conotroller.rst/events-pci-controller.rst/
> +The PCI controller tracing system provides tracepoints to monitor controller level
> +information for debugging purpose. The events normally show up here:
> +up here:
Wrap text to fit in 80 columns (unless it's a command line or similar
that shouldn't be split).
^ permalink raw reply
* Re: [PATCH v2 2/3] Documentation: tracing: Add PCI controller event documentation
From: kernel test robot @ 2026-01-09 20:08 UTC (permalink / raw)
To: Shawn Lin, Manivannan Sadhasivam, Bjorn Helgaas
Cc: oe-kbuild-all, linux-rockchip, linux-pci, linux-trace-kernel,
linux-doc, Steven Rostedt, Masami Hiramatsu, Shawn Lin
In-Reply-To: <1767929389-143957-3-git-send-email-shawn.lin@rock-chips.com>
Hi Shawn,
kernel test robot noticed the following build warnings:
[auto build test WARNING on pci/next]
[also build test WARNING on next-20260109]
[cannot apply to pci/for-linus mani-mhi/mhi-next trace/for-next linus/master v6.19-rc4]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Shawn-Lin/PCI-trace-Add-PCI-controller-LTSSM-transition-tracepoint/20260109-153843
base: https://git.kernel.org/pub/scm/linux/kernel/git/pci/pci.git next
patch link: https://lore.kernel.org/r/1767929389-143957-3-git-send-email-shawn.lin%40rock-chips.com
patch subject: [PATCH v2 2/3] Documentation: tracing: Add PCI controller event documentation
reproduce: (https://download.01.org/0day-ci/archive/20260109/202601092104.0IlUz26P-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202601092104.0IlUz26P-lkp@intel.com/
All warnings (new ones prefixed by >>):
WARNING: No kernel-doc for file ./include/linux/delay.h
ERROR: Cannot find file ./include/linux/delay.h
WARNING: No kernel-doc for file ./include/linux/delay.h
ERROR: Cannot find file ./include/linux/delay.h
WARNING: No kernel-doc for file ./include/linux/delay.h
>> Documentation/trace/events-pci-conotroller.rst:21: WARNING: Title underline too short.
--
ERROR: Cannot find file ./include/linux/mutex.h
ERROR: Cannot find file ./include/linux/mutex.h
WARNING: No kernel-doc for file ./include/linux/mutex.h
ERROR: Cannot find file ./include/linux/fwctl.h
WARNING: No kernel-doc for file ./include/linux/fwctl.h
>> Documentation/trace/events-pci-conotroller.rst: WARNING: document isn't included in any toctree [toc.not_included]
vim +21 Documentation/trace/events-pci-conotroller.rst
19
20 pcie_ltssm_state_transition
> 21 -----------------------
22
23 Monitors PCIe LTSSM state transition including state and rate information
24 ::
25
26 pcie_ltssm_state_transition "dev: %s state: %s rate: %s\n"
27
28 **Parameters**:
29
30 * ``dev`` - PCIe root port name
31 * ``state`` - PCIe LTSSM state
32 * ``rate`` - PCIe bus speed
33
34 **Example Usage**:
35
36 # Enable the tracepoint
37 echo 1 > /sys/kernel/debug/tracing/events/pci/pcie_ltssm_state_transition/enable
38
39 # Monitor events (the following output is generated when a device is linking)
40 cat /sys/kernel/debug/tracing/trace_pipe
> 41 kworker/0:0-9 [000] ..... 5.600221: pcie_ltssm_state_transition: dev: a40000000.pcie state: RCVRY_EQ2 rate: 8.0 GT/s
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH] arm64/mm: Fix annotated branch unbootable kernel
From: Steven Rostedt @ 2026-01-09 19:50 UTC (permalink / raw)
To: Will Deacon
Cc: Breno Leitao, Catalin Marinas, Mark Rutland, Laura Abbott,
linux-arm-kernel, linux-kernel, linux-trace-kernel,
Masami Hiramatsu, puranjay, usamaarif642, kernel-team, stable
In-Reply-To: <aVwp_BJx84gXHPlD@willie-the-truck>
[ Resending with my kernel.org email, as I received a bunch of messages from gmail saying it's blocking me :-p ]
On Mon, 5 Jan 2026 21:15:40 +0000
Will Deacon <will@kernel.org> wrote:
> > Another approach is to disable profiling on all arch/arm64 code, similarly to
> > x86, where DISABLE_BRANCH_PROFILING is called for all arch/x86 code. See
> > commit 2cbb20b008dba ("tracing: Disable branch profiling in noinstr
> > code").
>
> Yes, let's start with arch/arm64/. We know that's safe and then if
> somebody wants to make it finer-grained, it's on them to figure out a
> way to do it without playing whack-a-mole.
OK, so by adding -DDISABLE_BRANCH_PROFILING to the Makefile configs and for
the files that were audited, could be opt-in?
CFLAGS_REMOVE_<autdit_file>.o = -DDISABLE_BRANCH_PROFILING
And add that for each file that has been fully audited?
-- Steve
^ permalink raw reply
* Re: [PATCH] arm64/mm: Fix annotated branch unbootable kernel
From: Steven Rostedt @ 2026-01-09 19:47 UTC (permalink / raw)
To: Will Deacon
Cc: Breno Leitao, Catalin Marinas, Mark Rutland, Laura Abbott,
linux-arm-kernel, linux-kernel, linux-trace-kernel,
Masami Hiramatsu, puranjay, usamaarif642, kernel-team, stable
In-Reply-To: <aVwp_BJx84gXHPlD@willie-the-truck>
On Mon, 5 Jan 2026 21:15:40 +0000
Will Deacon <will@kernel.org> wrote:
> > Another approach is to disable profiling on all arch/arm64 code, similarly to
> > x86, where DISABLE_BRANCH_PROFILING is called for all arch/x86 code. See
> > commit 2cbb20b008dba ("tracing: Disable branch profiling in noinstr
> > code").
>
> Yes, let's start with arch/arm64/. We know that's safe and then if
> somebody wants to make it finer-grained, it's on them to figure out a
> way to do it without playing whack-a-mole.
OK, so by adding -DDISABLE_BRANCH_PROFILING to the Makefile configs and for
the files that were audited, could be opt-in?
CFLAGS_REMOVE_<autdit_file>.o = -DDISABLE_BRANCH_PROFILING
And add that for each file that has been fully audited?
-- Steve
^ permalink raw reply
* Re: [PATCH v2] tracing: Replace use of system_wq with system_dfl_wq
From: Steven Rostedt @ 2026-01-09 19:27 UTC (permalink / raw)
To: Marco Crivellari
Cc: linux-kernel, linux-trace-kernel, Tejun Heo, Lai Jiangshan,
Frederic Weisbecker, Sebastian Andrzej Siewior, Michal Hocko,
Masami Hiramatsu, Mathieu Desnoyers
In-Reply-To: <20251230094236.358ae2d8@gandalf.local.home>
On Tue, 30 Dec 2025 09:42:36 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
> > Suggested-by: Tejun Heo <tj@kernel.org>
> > Signed-off-by: Marco Crivellari <marco.crivellari@suse.com>
> > ---
> > Changes in v2:
> > - use system_dfl_wq instead of system_percpu_wq because this workload has no
> > benefits being per-cpu.
>
> Acked-by: Steven Rostedt (Google) <rostedt@goodmis.org>
I guess I'll take this through my tree, as I'm pulling in patches now to
base off of -rc5.
-- Steve
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Yonghong Song @ 2026-01-09 19:19 UTC (permalink / raw)
To: Alexei Starovoitov, Mathieu Desnoyers
Cc: Steven Rostedt, LKML, Linux trace kernel, bpf, Masami Hiramatsu,
Paul E. McKenney, Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <CAADnVQJMa+p_BcYxKUgve2=sqRBwSs3wLGAGhbA0r6hwFpJ+6Q@mail.gmail.com>
On 1/9/26 11:10 AM, Alexei Starovoitov wrote:
> On Fri, Jan 9, 2026 at 6:45 AM Mathieu Desnoyers
> <mathieu.desnoyers@efficios.com> wrote:
>> On 2026-01-08 22:05, Steven Rostedt wrote:
>>> From: "Paul E. McKenney" <paulmck@kernel.org>
>> [...]
>>
>> I disagree with many elements of the proposed approach.
>>
>> On one end we have BPF wanting to hook on arbitrary tracepoints without
>> adding significant latency to PREEMPT RT kernels.
>>
>> One the other hand, we have high-speed tracers which execute very short
>> critical sections to serialize trace data into ring buffers.
>>
>> All of those users register to the tracepoint API.
>>
>> We also have to consider that migrate disable is *not* cheap at all
>> compared to preempt disable.
> Looks like your complaint comes from lack of engagement in kernel
> development.
> migrate_disable _was_ not cheap.
> Try to benchmark it now.
> It's inlined. It's a fraction of extra overhead on top of preempt_disable.
>
The following are related patches to inline migrate_disable():
35561bab768977c9e05f1f1a9bc00134c85f3e28 arch: Add the macro COMPILE_OFFSETS to all the asm-offsets.c
88a90315a99a9120cd471bf681515cc77cd7cdb8 rcu: Replace preempt.h with sched.h in include/linux/rcupdate.h
378b7708194fff77c9020392067329931c3fcc04 sched: Make migrate_{en,dis}able() inline
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 19:19 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Mathieu Desnoyers, LKML, Linux trace kernel, bpf,
Masami Hiramatsu, Paul E. McKenney, Sebastian Andrzej Siewior,
Thomas Gleixner
In-Reply-To: <CAADnVQJMa+p_BcYxKUgve2=sqRBwSs3wLGAGhbA0r6hwFpJ+6Q@mail.gmail.com>
On Fri, 9 Jan 2026 11:10:16 -0800
Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote:
\> >
> > We also have to consider that migrate disable is *not* cheap at all
> > compared to preempt disable.
>
> Looks like your complaint comes from lack of engagement in kernel
> development.
No need to make comments like that. The Linux kernel is an ocean of code.
It's very hard to keep up on everything that is happening. I knew of work
being done on migrate_disable but I didn't know what the impacts of that
work was. Mathieu is still very much involved and engaged in kernel
development.
> migrate_disable _was_ not cheap.
> Try to benchmark it now.
> It's inlined. It's a fraction of extra overhead on top of preempt_disable.
It would be good to have a benchmark of the two. What about fast_srcu? Is
that fast enough to replace the preempt_disable()? If so, then could we
just make this the same for both RT and !RT?
-- Steve
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Alexei Starovoitov @ 2026-01-09 19:10 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: Steven Rostedt, LKML, Linux trace kernel, bpf, Masami Hiramatsu,
Paul E. McKenney, Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <da261242-482f-4b47-81c6-b065c5a95c4b@efficios.com>
On Fri, Jan 9, 2026 at 6:45 AM Mathieu Desnoyers
<mathieu.desnoyers@efficios.com> wrote:
>
> On 2026-01-08 22:05, Steven Rostedt wrote:
> > From: "Paul E. McKenney" <paulmck@kernel.org>
> [...]
>
> I disagree with many elements of the proposed approach.
>
> On one end we have BPF wanting to hook on arbitrary tracepoints without
> adding significant latency to PREEMPT RT kernels.
>
> One the other hand, we have high-speed tracers which execute very short
> critical sections to serialize trace data into ring buffers.
>
> All of those users register to the tracepoint API.
>
> We also have to consider that migrate disable is *not* cheap at all
> compared to preempt disable.
Looks like your complaint comes from lack of engagement in kernel
development.
migrate_disable _was_ not cheap.
Try to benchmark it now.
It's inlined. It's a fraction of extra overhead on top of preempt_disable.
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Mathieu Desnoyers @ 2026-01-09 18:58 UTC (permalink / raw)
To: Steven Rostedt
Cc: LKML, Linux trace kernel, bpf, Masami Hiramatsu, Paul E. McKenney,
Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <20260109122142.108982d9@gandalf.local.home>
On 2026-01-09 12:21, Steven Rostedt wrote:
> On Fri, 9 Jan 2026 09:40:17 -0500
> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
>
>> On 2026-01-08 22:05, Steven Rostedt wrote:
>>> From: "Paul E. McKenney" <paulmck@kernel.org>
>> [...]
>>
>> I disagree with many elements of the proposed approach.
>>
>> On one end we have BPF wanting to hook on arbitrary tracepoints without
>> adding significant latency to PREEMPT RT kernels.
>>
>> One the other hand, we have high-speed tracers which execute very short
>> critical sections to serialize trace data into ring buffers.
>>
>> All of those users register to the tracepoint API.
>>
>> We also have to consider that migrate disable is *not* cheap at all
>> compared to preempt disable.
>
> To be fair, every spin_lock() converted into a mutex in PREEMPT_RT now
> calls migrate_disable() instead of preempt_disable(). I'm just saying the
> overhead of migrate_disable() in PREEMPT_RT is not limited to tracepoints.
That's a fair point. That being said, the kernel code which relies on
spin locks for fast-paths tends to use raw spinlocks (e.g. scheduler),
which AFAIU does not need migrate disable because those still disable
preemption even on preempt RT.
>
>>
>> So I'm wondering why all tracepoint users need to pay the migrate
>> disable runtime overhead on preempt RT kernels for the sake of BPF ?
>
> We could argue that it is to keep the same paradigm as non RT. Where
> the code expects to stay on the same CPU. This is why we needed to add it
> to spin_lock() code. Only a few places in the kernel expect spin_lock() to
> pin the current task on the CPU, but because of those few cases, we needed
> to make all callers of spin_lock() call migrate disable :-/
For tracepoints we have a handful of tracer users, so I don't consider
the scope of the audit to be at the same scale as spinlocks. Changing
each tracer user to either disable preemption or migration if they need
it on preempt-RT is fairly straightforward. And if tracers don't need
to stay on a single CPU, that's even better.
I'm also inclined to consider moving !RT kernels to srcu-fast rather
than RCU to make RT and !RT more alike. The performance of
srcu-fast read-side is excellent, at least on my EPYC test machine.
This would open the door to relax the tracer callback requirements on
preempt-off/migrate-off if their callback implementation can deal with
it.
>
>>
>> Using SRCU-fast to protect tracepoint callback iteration makes sense
>> for preempt-rt, but I'd recommend moving the migrate disable guard
>> within the bpf callback code rather than slowing down other tracers
>> which execute within a short amount of time. Other tracers can then
>> choose to disable preemption rather than migration if that's a better
>> fit for their needs.
>
> This is a discussion with the BPF folks.
FWIW, the approach I'm proposing would be similar to what I've done for
faultable syscall tracepoints.
[...]
>>> +
>>> + trace_ctx = tracing_gen_ctx_dec();
>>> + /* The migration counter starts at bit 4 */
>>> + return trace_ctx - (1 << 4);
>>
>> We should turn this hardcoded "4" value into an enum label or a
>> define. That define should be exposed by tracepoint.h. We should
>> not hardcode expectations about the implementation of distinct APIs
>> across the tracing subsystem.
>
> This is exposed to user space already, so that 4 will never change. And
> this is specifically for "trace events" which are what are attached to
> tracepoints. No other tracepoint caller needs to know about this "4". This
> value goes into the common_preempt_count of the event. libtraceevent
> already parses this.
>
> I have no problem making this an enum (or define) and using it here and
> where it is set in trace.c:tracing_gen_ctx_irq_test(). But it belongs in
> trace_event.h not tracepoint.h.
>
> I can call it TRACE_MIGRATION_SHIFT
>
> #define TRACE_MIGRATION_SHIFT 4
I'm OK with that.
[...]
>>> }
>>>
>>> /*
>>> diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h
>>> index 4f22136fd465..6fb58387e9f1 100644
>>> --- a/include/trace/trace_events.h
>>> +++ b/include/trace/trace_events.h
>>> @@ -429,6 +429,22 @@ do_trace_event_raw_event_##call(void *__data, proto) \
>>> trace_event_buffer_commit(&fbuffer); \
>>> }
>>>
>>> +/*
>>> + * When PREEMPT_RT is enabled, the tracepoint does not disable preemption
>>> + * but instead disables migration. The callbacks for the trace events
>>> + * need to have a consistent state so that it can reflect the proper
>>> + * preempt_disabled counter.
>>
>> Having those defines within trace_events.h is poking holes within any
>> hope of abstraction we can have from the tracepoint.h API. This adds
>> strong coupling between tracepoint and trace_event.h.
>
> OK, I see you are worried about the coupling between the behavior of
> tracepoint.h and trace_event.h.
>
>>
>> Rather than hardcoding preempt counter expectations across tracepoint
>> and trace-events, we should expose a #define in tracepoint.h which
>> will make the preempt counter nesting level available to other
>> parts of the kernel such as trace_events.h. This way we keep everything
>> in one place and we don't add cross-references about subtle preempt
>> counter nesting level details.
>
> OK, so how do we pass this information from tracepoint.h to the users? I
> hate to add another field to task_struct for this.
This could be as simple as adding something akin to
/*
* Level of preempt disable nesting between tracepoint caller code and
* tracer callback.
*/
#ifdef CONFIG_PREEMPT_RT
# define TRACEPOINT_PREEMPT_DISABLE_NESTING 0
#else
# define TRACEPOINT_PREEMPT_DISABLE_NESTING 1
#endif
to tracepoint.h and then use that from trace_event.h.
>
>>
>> [...]
>>
>>> + *
>>> + * Returns a pointer to the data on the ring buffer or NULL if the
>>> + * event was not reserved (event was filtered, too big, or the buffer
>>> + * simply was disabled for write).
>>
>> odd spaces here.
>
> You mean the indentation? I could add it more and also a colon:
>
> * Returns: A pointer to the data on the ring buffer or NULL if the
> * event was not reserved (event was filtered, too big, or the
> * buffer simply was disabled for write).
>
> Would that work better?
Yes.
[...]
>>
>> So I don't understand this comment, and also I don't understand why we
>> need to chain the callbacks rather than just call the appropriate
>> call_rcu based on the tracepoint "is_faultable" state.
>>
>> What am I missing ?
>
> Ah, you are right. I think this was the result of trying different ways of
> synchronization. The non-faultable version should be wrapped to either call
> normal RCU synchronization or SRCU synchronization.
Yes. Or we switch even !RT kernels non-faultable tracepoints to
srcu-fast, considering its low overhead. This could come as a future
step though.
> I can send a new version, or do we want to wait if the BPF folks have a
> better idea about the "migrate disable" issue?
I'd recommend going ahead and move the migrate disable into the BPF
callback similarly to what did for the faultable syscall tracepoint, and
let the bpf folks comment on the resulting patch. It's not that much
code, and showing the change will ground the discussion into something
more tangible.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
^ permalink raw reply
* Re: [PATCH v5] tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast
From: Steven Rostedt @ 2026-01-09 17:21 UTC (permalink / raw)
To: Mathieu Desnoyers
Cc: LKML, Linux trace kernel, bpf, Masami Hiramatsu, Paul E. McKenney,
Sebastian Andrzej Siewior, Thomas Gleixner
In-Reply-To: <da261242-482f-4b47-81c6-b065c5a95c4b@efficios.com>
On Fri, 9 Jan 2026 09:40:17 -0500
Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:
> On 2026-01-08 22:05, Steven Rostedt wrote:
> > From: "Paul E. McKenney" <paulmck@kernel.org>
> [...]
>
> I disagree with many elements of the proposed approach.
>
> On one end we have BPF wanting to hook on arbitrary tracepoints without
> adding significant latency to PREEMPT RT kernels.
>
> One the other hand, we have high-speed tracers which execute very short
> critical sections to serialize trace data into ring buffers.
>
> All of those users register to the tracepoint API.
>
> We also have to consider that migrate disable is *not* cheap at all
> compared to preempt disable.
To be fair, every spin_lock() converted into a mutex in PREEMPT_RT now
calls migrate_disable() instead of preempt_disable(). I'm just saying the
overhead of migrate_disable() in PREEMPT_RT is not limited to tracepoints.
>
> So I'm wondering why all tracepoint users need to pay the migrate
> disable runtime overhead on preempt RT kernels for the sake of BPF ?
We could argue that it is to keep the same paradigm as non RT. Where
the code expects to stay on the same CPU. This is why we needed to add it
to spin_lock() code. Only a few places in the kernel expect spin_lock() to
pin the current task on the CPU, but because of those few cases, we needed
to make all callers of spin_lock() call migrate disable :-/
>
> Using SRCU-fast to protect tracepoint callback iteration makes sense
> for preempt-rt, but I'd recommend moving the migrate disable guard
> within the bpf callback code rather than slowing down other tracers
> which execute within a short amount of time. Other tracers can then
> choose to disable preemption rather than migration if that's a better
> fit for their needs.
This is a discussion with the BPF folks.
>
> > diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h
> > index 3690221ba3d8..a2704c35eda8 100644
> > --- a/include/linux/trace_events.h
> > +++ b/include/linux/trace_events.h
> > @@ -222,6 +222,26 @@ static inline unsigned int tracing_gen_ctx_dec(void)
> > return trace_ctx;
> > }
> >
> > +/*
> > + * When PREEMPT_RT is enabled, trace events are called with disabled
> > + * migration. The trace events need to know if the tracepoint disabled
> > + * migration or not so that what is recorded to the ring buffer shows
> > + * the state of when the trace event triggered, and not the state caused
> > + * by the trace event.
> > + */
> > +#ifdef CONFIG_PREEMPT_RT
> > +static inline unsigned int tracing_gen_ctx_dec_cond(void)
> > +{
> > + unsigned int trace_ctx;
> > +
> > + trace_ctx = tracing_gen_ctx_dec();
> > + /* The migration counter starts at bit 4 */
> > + return trace_ctx - (1 << 4);
>
> We should turn this hardcoded "4" value into an enum label or a
> define. That define should be exposed by tracepoint.h. We should
> not hardcode expectations about the implementation of distinct APIs
> across the tracing subsystem.
This is exposed to user space already, so that 4 will never change. And
this is specifically for "trace events" which are what are attached to
tracepoints. No other tracepoint caller needs to know about this "4". This
value goes into the common_preempt_count of the event. libtraceevent
already parses this.
I have no problem making this an enum (or define) and using it here and
where it is set in trace.c:tracing_gen_ctx_irq_test(). But it belongs in
trace_event.h not tracepoint.h.
I can call it TRACE_MIGRATION_SHIFT
#define TRACE_MIGRATION_SHIFT 4
>
> [...]
>
> > --- a/include/linux/tracepoint.h
> > +++ b/include/linux/tracepoint.h
> > @@ -100,6 +100,25 @@ void for_each_tracepoint_in_module(struct module *mod,
> > }
> > #endif /* CONFIG_MODULES */
> >
> > +/*
> > + * BPF programs can attach to the tracepoint callbacks. But if the
> > + * callbacks are called with preemption disabled, the BPF programs
> > + * can cause quite a bit of latency. When PREEMPT_RT is enabled,
> > + * instead of disabling preemption, use srcu_fast_notrace() for
> > + * synchronization. As BPF programs that are attached to tracepoints
> > + * expect to stay on the same CPU, also disable migration.
> > + */
> > +#ifdef CONFIG_PREEMPT_RT
> > +extern struct srcu_struct tracepoint_srcu;
> > +# define tracepoint_sync() synchronize_srcu(&tracepoint_srcu);
> > +# define tracepoint_guard() \
> > + guard(srcu_fast_notrace)(&tracepoint_srcu); \
> > + guard(migrate)()
> > +#else
> > +# define tracepoint_sync() synchronize_rcu();
> > +# define tracepoint_guard() guard(preempt_notrace)()
> > +#endif
>
> Doing migrate disable on PREEMPT RT for BPF vs preempt disable in other
> tracers should come in a separate preparation patch. It belongs to the
> tracers, not to tracepoints.
That's fair.
>
> [...]
> \
> > diff --git a/include/trace/perf.h b/include/trace/perf.h
> > index a1754b73a8f5..348ad1d9b556 100644
> > --- a/include/trace/perf.h
> > +++ b/include/trace/perf.h
> > @@ -71,6 +71,7 @@ perf_trace_##call(void *__data, proto) \
> > u64 __count __attribute__((unused)); \
> > struct task_struct *__task __attribute__((unused)); \
> > \
> > + guard(preempt_notrace)(); \
> > do_perf_trace_##call(__data, args); \
> > }
> >
> > @@ -85,9 +86,8 @@ perf_trace_##call(void *__data, proto) \
> > struct task_struct *__task __attribute__((unused)); \
> > \
> > might_fault(); \
> > - preempt_disable_notrace(); \
> > + guard(preempt_notrace)(); \
> > do_perf_trace_##call(__data, args); \
> > - preempt_enable_notrace();
>
> Move this to a perf-specific preparation patch. \
Sure.
> > }
> >
> > /*
> > diff --git a/include/trace/trace_events.h b/include/trace/trace_events.h
> > index 4f22136fd465..6fb58387e9f1 100644
> > --- a/include/trace/trace_events.h
> > +++ b/include/trace/trace_events.h
> > @@ -429,6 +429,22 @@ do_trace_event_raw_event_##call(void *__data, proto) \
> > trace_event_buffer_commit(&fbuffer); \
> > }
> >
> > +/*
> > + * When PREEMPT_RT is enabled, the tracepoint does not disable preemption
> > + * but instead disables migration. The callbacks for the trace events
> > + * need to have a consistent state so that it can reflect the proper
> > + * preempt_disabled counter.
>
> Having those defines within trace_events.h is poking holes within any
> hope of abstraction we can have from the tracepoint.h API. This adds
> strong coupling between tracepoint and trace_event.h.
OK, I see you are worried about the coupling between the behavior of
tracepoint.h and trace_event.h.
>
> Rather than hardcoding preempt counter expectations across tracepoint
> and trace-events, we should expose a #define in tracepoint.h which
> will make the preempt counter nesting level available to other
> parts of the kernel such as trace_events.h. This way we keep everything
> in one place and we don't add cross-references about subtle preempt
> counter nesting level details.
OK, so how do we pass this information from tracepoint.h to the users? I
hate to add another field to task_struct for this.
>
> > + */
> > +#ifdef CONFIG_PREEMPT_RT
> > +/* disable preemption for RT so that the counters still match */
> > +# define trace_event_guard() guard(preempt_notrace)()
> > +/* Have syscalls up the migrate disable counter to emulate non-syscalls */
> > +# define trace_syscall_event_guard() guard(migrate)()
> > +#else
> > +# define trace_event_guard()
> > +# define trace_syscall_event_guard()
> > +#endif
> This should be moved to separate tracer-specific prep patches.
>
> [...]
>
> > + * The @trace_file is the desrciptor with information about the status
>
> descriptor
Oops.
>
> [...]
>
> > + *
> > + * Returns a pointer to the data on the ring buffer or NULL if the
> > + * event was not reserved (event was filtered, too big, or the buffer
> > + * simply was disabled for write).
>
> odd spaces here.
You mean the indentation? I could add it more and also a colon:
* Returns: A pointer to the data on the ring buffer or NULL if the
* event was not reserved (event was filtered, too big, or the
* buffer simply was disabled for write).
Would that work better?
>
> [...]
>
> >
> > +#ifdef CONFIG_PREEMPT_RT
> > +static void srcu_free_old_probes(struct rcu_head *head)
> > +{
> > + kfree(container_of(head, struct tp_probes, rcu));
> > +}
> > +
> > +static void rcu_free_old_probes(struct rcu_head *head)
> > +{
> > + call_srcu(&tracepoint_srcu, head, srcu_free_old_probes);
> > +}
> > +#else
> > static void rcu_free_old_probes(struct rcu_head *head)
> > {
> > kfree(container_of(head, struct tp_probes, rcu));
> > }
> > +#endif
> >
> > static inline void release_probes(struct tracepoint *tp, struct tracepoint_func *old)
> > {
> > @@ -112,6 +149,13 @@ static inline void release_probes(struct tracepoint *tp, struct tracepoint_func
> > struct tp_probes *tp_probes = container_of(old,
> > struct tp_probes, probes[0]);
> >
> > + /*
> > + * Tracepoint probes are protected by either RCU or
> > + * Tasks Trace RCU and also by SRCU. By calling the SRCU
>
> I'm confused.
>
> My understanding is that in !RT we have:
>
> - RCU (!tracepoint_is_faultable(tp))
> - RCU tasks trace (tracepoint_is_faultable(tp))
>
> And for RT:
>
> - SRCU-fast (!tracepoint_is_faultable(tp))
> - RCU tasks trace (tracepoint_is_faultable(tp))
>
> So I don't understand this comment, and also I don't understand why we
> need to chain the callbacks rather than just call the appropriate
> call_rcu based on the tracepoint "is_faultable" state.
>
> What am I missing ?
Ah, you are right. I think this was the result of trying different ways of
synchronization. The non-faultable version should be wrapped to either call
normal RCU synchronization or SRCU synchronization.
I can send a new version, or do we want to wait if the BPF folks have a
better idea about the "migrate disable" issue?
Thanks for the review.
-- Steve
>
> > + * callback in the [Tasks Trace] RCU callback we cover
> > + * both cases. So let us chain the SRCU and [Tasks Trace]
> > + * RCU callbacks to wait for both grace periods.
> > + */
> > if (tracepoint_is_faultable(tp))
> > call_rcu_tasks_trace(&tp_probes->rcu, rcu_free_old_probes);
> > else
>
> Thanks,
>
> Mathieu
>
^ permalink raw reply
* Re: [PATCH] ring-buffer: Use a housekeeping CPU to wake up waiters
From: Petr Tesarik @ 2026-01-09 16:54 UTC (permalink / raw)
To: Steven Rostedt
Cc: Masami Hiramatsu, Mathieu Desnoyers, Sebastian Andrzej Siewior,
Clark Williams, linux-kernel, linux-trace-kernel, linux-rt-devel
In-Reply-To: <20260109111506.66ad545a@gandalf.local.home>
On Fri, 9 Jan 2026 11:15:06 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:
> On Fri, 9 Jan 2026 09:57:56 +0100
> Petr Tesarik <ptesarik@suse.com> wrote:
>
> > I have removed both "inline" and "noinline" in v2, leaving it at the
> > discretion of the compiler. If you believe it deserves a "noinline",
> > feel free to add it. FWIW on x86-64, I didn't observe any measurable
> > diference either in latency or instruction cache footprint.
>
> Please add the noinline. I went through and added strategic "noinline" and
> "__always_inline", as well as placing "likely()" and "unlikely()" and
> dropped the cost of adding an event from just under 300ns down to less than
> 150ns.
>
> This code is called during function tracing (hit at every function call),
> and yes, every little bit helps!
I'm going to wait a few days for further comments, then add it in v3.
Thanks for your time!
Petr T
^ permalink raw reply
* Re: [PATCH bpf-next 1/3] bpf, x64: Call perf_snapshot_branch_stack in trampoline
From: Leon Hwang @ 2026-01-09 16:31 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David S . Miller, David Ahern, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, X86 ML, H . Peter Anvin,
Matt Bobrowski, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Shuah Khan, Network Development, LKML,
linux-trace-kernel, open list:KERNEL SELFTEST FRAMEWORK,
kernel-patches-bot
In-Reply-To: <CAADnVQK4O-igzuSvfgjG1ZqdUBXrjNL=4tJZuS1uy36GCD2mVg@mail.gmail.com>
On 2026/1/10 00:24, Alexei Starovoitov wrote:
> On Fri, Jan 9, 2026 at 7:37 AM Leon Hwang <leon.hwang@linux.dev> wrote:
>>
[...]
>> @@ -3366,6 +3416,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
>>
>> save_args(m, &prog, regs_off, false, flags);
>>
>> + if (bpf_prog_copy_branch_snapshot(fentry)) {
>> + /* Get branch snapshot asap. */
>> + if (invoke_branch_snapshot(&prog, image, rw_image)) {
>> + ret = -EINVAL;
>> + goto cleanup;
>> + }
>> + }
>
> Andrii already tried to do it.
> I hated it back then and still hate the idea.
> We're not going to add custom logic for one specific use case
> no matter how appealing it sounds to save very limited LBR entries.
> The HW will get better, but we will be stuck with this optimization forever.
>
Understood, thanks for the explanation.
I won’t pursue this approach further.
Thanks,
Leon
^ permalink raw reply
* Re: [PATCH bpf-next 1/3] bpf, x64: Call perf_snapshot_branch_stack in trampoline
From: Alexei Starovoitov @ 2026-01-09 16:24 UTC (permalink / raw)
To: Leon Hwang
Cc: bpf, Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David S . Miller, David Ahern, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, X86 ML, H . Peter Anvin,
Matt Bobrowski, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Shuah Khan, Network Development, LKML,
linux-trace-kernel, open list:KERNEL SELFTEST FRAMEWORK,
kernel-patches-bot
In-Reply-To: <20260109153420.32181-2-leon.hwang@linux.dev>
On Fri, Jan 9, 2026 at 7:37 AM Leon Hwang <leon.hwang@linux.dev> wrote:
>
> When the PMU LBR is running in branch-sensitive mode,
> 'perf_snapshot_branch_stack()' may capture branch entries from the
> trampoline entry up to the call site inside a BPF program. These branch
> entries are not useful for analyzing the control flow of the tracee.
>
> To eliminate such noise for tracing programs, the branch snapshot should
> be taken as early as possible:
>
> * Call 'perf_snapshot_branch_stack()' at the very beginning of the
> trampoline for fentry programs.
> * Call 'perf_snapshot_branch_stack()' immediately after invoking the
> tracee for fexit programs.
>
> With this change, LBR snapshots remain meaningful even when multiple BPF
> programs execute before the one requesting LBR data.
>
> In addition, more relevant branch entries can be captured on AMD CPUs,
> which provide a 16-entry-deep LBR stack.
>
> Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
> ---
> arch/x86/net/bpf_jit_comp.c | 66 +++++++++++++++++++++++++++++++++++++
> include/linux/bpf.h | 16 ++++++++-
> 2 files changed, 81 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
> index e3b1c4b1d550..a71a6c675392 100644
> --- a/arch/x86/net/bpf_jit_comp.c
> +++ b/arch/x86/net/bpf_jit_comp.c
> @@ -12,6 +12,7 @@
> #include <linux/bpf.h>
> #include <linux/memory.h>
> #include <linux/sort.h>
> +#include <linux/perf_event.h>
> #include <asm/extable.h>
> #include <asm/ftrace.h>
> #include <asm/set_memory.h>
> @@ -19,6 +20,7 @@
> #include <asm/text-patching.h>
> #include <asm/unwind.h>
> #include <asm/cfi.h>
> +#include "../events/perf_event.h"
>
> static bool all_callee_regs_used[4] = {true, true, true, true};
>
> @@ -3137,6 +3139,54 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
> return 0;
> }
>
> +DEFINE_PER_CPU(struct bpf_tramp_branch_entries, bpf_branch_snapshot);
> +
> +static int invoke_branch_snapshot(u8 **pprog, void *image, void *rw_image)
> +{
> + struct bpf_tramp_branch_entries __percpu *pptr = &bpf_branch_snapshot;
> + u8 *prog = *pprog;
> +
> + /*
> + * Emit:
> + *
> + * struct bpf_tramp_branch_entries *br = this_cpu_ptr(&bpf_branch_snapshot);
> + * br->cnt = static_call(perf_snapshot_branch_stack)(br->entries, x86_pmu.lbr_nr);
> + */
> +
> + /* mov rbx, &bpf_branch_snapshot */
> + emit_mov_imm64(&prog, BPF_REG_6, (long) pptr >> 32, (u32)(long) pptr);
> +#ifdef CONFIG_SMP
> + /* add rbx, gs:[<off>] */
> + EMIT2(0x65, 0x48);
> + EMIT3(0x03, 0x1C, 0x25);
> + EMIT((u32)(unsigned long)&this_cpu_off, 4);
> +#endif
> + /* mov esi, x86_pmu.lbr_nr */
> + EMIT1_off32(0xBE, x86_pmu.lbr_nr);
> + /* lea rdi, [rbx + offsetof(struct bpf_tramp_branch_entries, entries)] */
> + EMIT4(0x48, 0x8D, 0x7B, offsetof(struct bpf_tramp_branch_entries, entries));
> + /* call static_call_query(perf_snapshot_branch_stack) */
> + if (emit_rsb_call(&prog, static_call_query(perf_snapshot_branch_stack),
> + image + (prog - (u8 *)rw_image)))
> + return -EINVAL;
> + /* mov dword ptr [rbx], eax */
> + EMIT2(0x89, 0x03);
> +
> + *pprog = prog;
> + return 0;
> +}
> +
> +static bool bpf_prog_copy_branch_snapshot(struct bpf_tramp_links *tl)
> +{
> + bool copy = false;
> + int i;
> +
> + for (i = 0; i < tl->nr_links; i++)
> + copy = copy || tl->links[i]->link.prog->copy_branch_snapshot;
> +
> + return copy;
> +}
> +
> /* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */
> #define LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack) \
> __LOAD_TCC_PTR(-round_up(stack, 8) - 8)
> @@ -3366,6 +3416,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
>
> save_args(m, &prog, regs_off, false, flags);
>
> + if (bpf_prog_copy_branch_snapshot(fentry)) {
> + /* Get branch snapshot asap. */
> + if (invoke_branch_snapshot(&prog, image, rw_image)) {
> + ret = -EINVAL;
> + goto cleanup;
> + }
> + }
Andrii already tried to do it.
I hated it back then and still hate the idea.
We're not going to add custom logic for one specific use case
no matter how appealing it sounds to save very limited LBR entries.
The HW will get better, but we will be stuck with this optimization forever.
pw-bot: cr
^ permalink raw reply
* Re: [PATCHv2 bpf-next 1/2] arm64/ftrace,bpf: Fix partial regs after bpf_prog_run
From: Steven Rostedt @ 2026-01-09 16:19 UTC (permalink / raw)
To: Will Deacon
Cc: Jiri Olsa, Masami Hiramatsu, Mahe Tardy, Peter Zijlstra, bpf,
linux-trace-kernel, linux-arm-kernel, x86, Yonghong Song,
Song Liu, Andrii Nakryiko, Mark Rutland
In-Reply-To: <aWEG685zlaV0o7M7@willie-the-truck>
On Fri, 9 Jan 2026 13:47:23 +0000
Will Deacon <will@kernel.org> wrote:
> I think the AI thingy is right about dropping the const qualifier here
> but overall I prefer this to the previous revisions. Thanks for sticking
> with it!
Yep, I missed that too :-p
I do think AI is better at reviewing than writing code ;-)
-- Steve
^ permalink raw reply
* Re: [PATCHv2 bpf-next 1/2] arm64/ftrace,bpf: Fix partial regs after bpf_prog_run
From: Steven Rostedt @ 2026-01-09 16:18 UTC (permalink / raw)
To: Jiri Olsa
Cc: Masami Hiramatsu, Will Deacon, Mahe Tardy, Peter Zijlstra, bpf,
linux-trace-kernel, linux-arm-kernel, x86, Yonghong Song,
Song Liu, Andrii Nakryiko, Mark Rutland
In-Reply-To: <20260109093454.389295-1-jolsa@kernel.org>
On Fri, 9 Jan 2026 10:34:53 +0100
Jiri Olsa <jolsa@kernel.org> wrote:
> Mahe reported issue with bpf_override_return helper not working when
> executed from kprobe.multi bpf program on arm.
>
> The problem is that on arm we use alternate storage for pt_regs object
> that is passed to bpf_prog_run and if any register is changed (which
> is the case of bpf_override_return) it's not propagated back to actual
> pt_regs object.
>
> Fixing this by introducing and calling ftrace_partial_regs_update function
> to propagate the values of changed registers (ip and stack).
>
> Reported-by: Mahe Tardy <mahe.tardy@gmail.com>
> Signed-off-by: Jiri Olsa <jolsa@kernel.org>
> ---
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
-- Steve
^ permalink raw reply
* Re: [PATCH] ring-buffer: Use a housekeeping CPU to wake up waiters
From: Steven Rostedt @ 2026-01-09 16:15 UTC (permalink / raw)
To: Petr Tesarik
Cc: Masami Hiramatsu, Mathieu Desnoyers, Sebastian Andrzej Siewior,
Clark Williams, linux-kernel, linux-trace-kernel, linux-rt-devel
In-Reply-To: <20260109095756.13deb429@mordecai>
On Fri, 9 Jan 2026 09:57:56 +0100
Petr Tesarik <ptesarik@suse.com> wrote:
> I have removed both "inline" and "noinline" in v2, leaving it at the
> discretion of the compiler. If you believe it deserves a "noinline",
> feel free to add it. FWIW on x86-64, I didn't observe any measurable
> diference either in latency or instruction cache footprint.
Please add the noinline. I went through and added strategic "noinline" and
"__always_inline", as well as placing "likely()" and "unlikely()" and
dropped the cost of adding an event from just under 300ns down to less than
150ns.
This code is called during function tracing (hit at every function call),
and yes, every little bit helps!
-- Steve
^ permalink raw reply
* [PATCH bpf-next 3/3] selftests/bpf: Add BPF_BRANCH_SNAPSHOT_F_COPY test
From: Leon Hwang @ 2026-01-09 15:34 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David S . Miller, David Ahern, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H . Peter Anvin,
Matt Bobrowski, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Shuah Khan, Leon Hwang, netdev, linux-kernel,
linux-trace-kernel, linux-kselftest, kernel-patches-bot
In-Reply-To: <20260109153420.32181-1-leon.hwang@linux.dev>
Add test for BPF_BRANCH_SNAPSHOT_F_COPY flag by adding flag to the
callsite of bpf_get_branch_snapshot helper.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
---
.../bpf/prog_tests/get_branch_snapshot.c | 26 ++++++++++++++++---
.../selftests/bpf/progs/get_branch_snapshot.c | 3 ++-
2 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
index 0394a1156d99..6b8ab1655ab0 100644
--- a/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
+++ b/tools/testing/selftests/bpf/prog_tests/get_branch_snapshot.c
@@ -73,7 +73,7 @@ static void close_perf_events(void)
free(pfd_array);
}
-void serial_test_get_branch_snapshot(void)
+static void test_branch_snapshot(int flags)
{
struct get_branch_snapshot *skel = NULL;
int err;
@@ -89,8 +89,14 @@ void serial_test_get_branch_snapshot(void)
goto cleanup;
}
- skel = get_branch_snapshot__open_and_load();
- if (!ASSERT_OK_PTR(skel, "get_branch_snapshot__open_and_load"))
+ skel = get_branch_snapshot__open();
+ if (!ASSERT_OK_PTR(skel, "get_branch_snapshot__open"))
+ goto cleanup;
+
+ skel->rodata->flags = flags;
+
+ err = get_branch_snapshot__load(skel);
+ if (!ASSERT_OK(err, "get_branch_snapshot__load"))
goto cleanup;
err = kallsyms_find("bpf_testmod_loop_test", &skel->bss->address_low);
@@ -128,3 +134,17 @@ void serial_test_get_branch_snapshot(void)
get_branch_snapshot__destroy(skel);
close_perf_events();
}
+
+void serial_test_get_branch_snapshot(void)
+{
+ test_branch_snapshot(0);
+}
+
+enum {
+ BPF_BRANCH_SNAPSHOT_F_COPY = 1, /* Copy branch snapshot from bpf_branch_snapshot. */
+};
+
+void serial_test_copy_branch_snapshot(void)
+{
+ test_branch_snapshot(BPF_BRANCH_SNAPSHOT_F_COPY);
+}
diff --git a/tools/testing/selftests/bpf/progs/get_branch_snapshot.c b/tools/testing/selftests/bpf/progs/get_branch_snapshot.c
index 511ac634eef0..47a1984bdf46 100644
--- a/tools/testing/selftests/bpf/progs/get_branch_snapshot.c
+++ b/tools/testing/selftests/bpf/progs/get_branch_snapshot.c
@@ -6,6 +6,7 @@
char _license[] SEC("license") = "GPL";
+volatile const int flags = 0;
__u64 test1_hits = 0;
__u64 address_low = 0;
__u64 address_high = 0;
@@ -25,7 +26,7 @@ int BPF_PROG(test1, int n, int ret)
{
long i;
- total_entries = bpf_get_branch_snapshot(entries, sizeof(entries), 0);
+ total_entries = bpf_get_branch_snapshot(entries, sizeof(entries), flags);
total_entries /= sizeof(struct perf_branch_entry);
for (i = 0; i < ENTRY_CNT; i++) {
--
2.52.0
^ permalink raw reply related
* [PATCH bpf-next 2/3] bpf: Introduce BPF_BRANCH_SNAPSHOT_F_COPY flag for bpf_get_branch_snapshot helper
From: Leon Hwang @ 2026-01-09 15:34 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David S . Miller, David Ahern, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H . Peter Anvin,
Matt Bobrowski, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Shuah Khan, Leon Hwang, netdev, linux-kernel,
linux-trace-kernel, linux-kselftest, kernel-patches-bot
In-Reply-To: <20260109153420.32181-1-leon.hwang@linux.dev>
Introduce BPF_BRANCH_SNAPSHOT_F_COPY flag for tracing programs to copy
branch entries from *bpf_branch_snapshot*.
Instead of introducing a new kfunc, extend bpf_get_branch_snapshot
helper to add the BPF_BRANCH_SNAPSHOT_F_COPY flag support.
Therefore, when BPF_BRANCH_SNAPSHOT_F_COPY is specified:
* Check the *flags* value in verifier's 'check_helper_call()'.
* Skip inlining 'bpf_get_branch_snapshot()' helper in verifier's
'do_misc_fixups()'.
* 'memcpy()' branch entries in the 'bpf_get_branch_snapshot()' helper.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
---
include/linux/bpf.h | 4 ++++
include/linux/bpf_verifier.h | 1 +
kernel/bpf/verifier.c | 30 ++++++++++++++++++++++++++++++
kernel/trace/bpf_trace.c | 17 ++++++++++++++---
4 files changed, 49 insertions(+), 3 deletions(-)
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 16dc21836a06..71ce225e5160 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -1249,6 +1249,10 @@ struct bpf_tramp_branch_entries {
DECLARE_PER_CPU(struct bpf_tramp_branch_entries, bpf_branch_snapshot);
#endif
+enum {
+ BPF_BRANCH_SNAPSHOT_F_COPY = 1, /* Copy branch snapshot from bpf_branch_snapshot. */
+};
+
/* Different use cases for BPF trampoline:
* 1. replace nop at the function entry (kprobe equivalent)
* flags = BPF_TRAMP_F_RESTORE_REGS
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 130bcbd66f60..c60a145e0466 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -561,6 +561,7 @@ struct bpf_insn_aux_data {
bool non_sleepable; /* helper/kfunc may be called from non-sleepable context */
bool is_iter_next; /* bpf_iter_<type>_next() kfunc call */
bool call_with_percpu_alloc_ptr; /* {this,per}_cpu_ptr() with prog percpu alloc */
+ bool copy_branch_snapshot; /* BPF_BRANCH_SNAPSHOT_F_COPY for bpf_get_branch_snapshot helper */
u8 alu_state; /* used in combination with alu_limit */
/* true if STX or LDX instruction is a part of a spill/fill
* pattern for a bpf_fastcall call.
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 53635ea2e41b..0a537f9c2f8c 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -11772,6 +11772,33 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn
err = push_callback_call(env, insn, insn_idx, meta.subprogno,
set_user_ringbuf_callback_state);
break;
+ case BPF_FUNC_get_branch_snapshot:
+ {
+ u64 flags;
+
+ if (!is_reg_const(®s[BPF_REG_3], false)) {
+ verbose(env, "Flags in bpf_get_branch_snapshot helper must be const.\n");
+ return -EINVAL;
+ }
+ flags = reg_const_value(®s[BPF_REG_3], false);
+ if (flags & ~BPF_BRANCH_SNAPSHOT_F_COPY) {
+ verbose(env, "Invalid flags in bpf_get_branch_snapshot helper.\n");
+ return -EINVAL;
+ }
+
+ if (flags & BPF_BRANCH_SNAPSHOT_F_COPY) {
+ if (env->prog->type != BPF_PROG_TYPE_TRACING ||
+ (env->prog->expected_attach_type != BPF_TRACE_FENTRY &&
+ env->prog->expected_attach_type != BPF_TRACE_FEXIT)) {
+ verbose(env, "Only fentry and fexit programs support BPF_BRANCH_SNAPSHOT_F_COPY.\n");
+ return -EINVAL;
+ }
+
+ env->insn_aux_data[insn_idx].copy_branch_snapshot = true;
+ env->prog->copy_branch_snapshot = true;
+ }
+ break;
+ }
}
if (err)
@@ -23370,6 +23397,9 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
*/
BUILD_BUG_ON(br_entry_size != 24);
+ if (env->insn_aux_data[i + delta].copy_branch_snapshot)
+ goto patch_call_imm;
+
/* if (unlikely(flags)) return -EINVAL */
insn_buf[0] = BPF_JMP_IMM(BPF_JNE, BPF_REG_3, 0, 7);
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 6e076485bf70..e9e1698cf608 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -1172,10 +1172,20 @@ BPF_CALL_3(bpf_get_branch_snapshot, void *, buf, u32, size, u64, flags)
static const u32 br_entry_size = sizeof(struct perf_branch_entry);
u32 entry_cnt = size / br_entry_size;
- entry_cnt = static_call(perf_snapshot_branch_stack)(buf, entry_cnt);
-
- if (unlikely(flags))
+ if (likely(!flags)) {
+ entry_cnt = static_call(perf_snapshot_branch_stack)(buf, entry_cnt);
+#ifdef CONFIG_X86_64
+ } else if (flags & BPF_BRANCH_SNAPSHOT_F_COPY) {
+ struct bpf_tramp_branch_entries *br;
+
+ br = this_cpu_ptr(&bpf_branch_snapshot);
+ entry_cnt = min_t(u32, entry_cnt, br->cnt);
+ if (entry_cnt)
+ memcpy(buf, (void *) br->entries, entry_cnt * br_entry_size);
+#endif
+ } else {
return -EINVAL;
+ }
if (!entry_cnt)
return -ENOENT;
@@ -1189,6 +1199,7 @@ const struct bpf_func_proto bpf_get_branch_snapshot_proto = {
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_UNINIT_MEM,
.arg2_type = ARG_CONST_SIZE_OR_ZERO,
+ .arg3_type = ARG_ANYTHING,
};
BPF_CALL_3(get_func_arg, void *, ctx, u32, n, u64 *, value)
--
2.52.0
^ permalink raw reply related
* [PATCH bpf-next 1/3] bpf, x64: Call perf_snapshot_branch_stack in trampoline
From: Leon Hwang @ 2026-01-09 15:34 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David S . Miller, David Ahern, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H . Peter Anvin,
Matt Bobrowski, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Shuah Khan, Leon Hwang, netdev, linux-kernel,
linux-trace-kernel, linux-kselftest, kernel-patches-bot
In-Reply-To: <20260109153420.32181-1-leon.hwang@linux.dev>
When the PMU LBR is running in branch-sensitive mode,
'perf_snapshot_branch_stack()' may capture branch entries from the
trampoline entry up to the call site inside a BPF program. These branch
entries are not useful for analyzing the control flow of the tracee.
To eliminate such noise for tracing programs, the branch snapshot should
be taken as early as possible:
* Call 'perf_snapshot_branch_stack()' at the very beginning of the
trampoline for fentry programs.
* Call 'perf_snapshot_branch_stack()' immediately after invoking the
tracee for fexit programs.
With this change, LBR snapshots remain meaningful even when multiple BPF
programs execute before the one requesting LBR data.
In addition, more relevant branch entries can be captured on AMD CPUs,
which provide a 16-entry-deep LBR stack.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
---
arch/x86/net/bpf_jit_comp.c | 66 +++++++++++++++++++++++++++++++++++++
include/linux/bpf.h | 16 ++++++++-
2 files changed, 81 insertions(+), 1 deletion(-)
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index e3b1c4b1d550..a71a6c675392 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -12,6 +12,7 @@
#include <linux/bpf.h>
#include <linux/memory.h>
#include <linux/sort.h>
+#include <linux/perf_event.h>
#include <asm/extable.h>
#include <asm/ftrace.h>
#include <asm/set_memory.h>
@@ -19,6 +20,7 @@
#include <asm/text-patching.h>
#include <asm/unwind.h>
#include <asm/cfi.h>
+#include "../events/perf_event.h"
static bool all_callee_regs_used[4] = {true, true, true, true};
@@ -3137,6 +3139,54 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog,
return 0;
}
+DEFINE_PER_CPU(struct bpf_tramp_branch_entries, bpf_branch_snapshot);
+
+static int invoke_branch_snapshot(u8 **pprog, void *image, void *rw_image)
+{
+ struct bpf_tramp_branch_entries __percpu *pptr = &bpf_branch_snapshot;
+ u8 *prog = *pprog;
+
+ /*
+ * Emit:
+ *
+ * struct bpf_tramp_branch_entries *br = this_cpu_ptr(&bpf_branch_snapshot);
+ * br->cnt = static_call(perf_snapshot_branch_stack)(br->entries, x86_pmu.lbr_nr);
+ */
+
+ /* mov rbx, &bpf_branch_snapshot */
+ emit_mov_imm64(&prog, BPF_REG_6, (long) pptr >> 32, (u32)(long) pptr);
+#ifdef CONFIG_SMP
+ /* add rbx, gs:[<off>] */
+ EMIT2(0x65, 0x48);
+ EMIT3(0x03, 0x1C, 0x25);
+ EMIT((u32)(unsigned long)&this_cpu_off, 4);
+#endif
+ /* mov esi, x86_pmu.lbr_nr */
+ EMIT1_off32(0xBE, x86_pmu.lbr_nr);
+ /* lea rdi, [rbx + offsetof(struct bpf_tramp_branch_entries, entries)] */
+ EMIT4(0x48, 0x8D, 0x7B, offsetof(struct bpf_tramp_branch_entries, entries));
+ /* call static_call_query(perf_snapshot_branch_stack) */
+ if (emit_rsb_call(&prog, static_call_query(perf_snapshot_branch_stack),
+ image + (prog - (u8 *)rw_image)))
+ return -EINVAL;
+ /* mov dword ptr [rbx], eax */
+ EMIT2(0x89, 0x03);
+
+ *pprog = prog;
+ return 0;
+}
+
+static bool bpf_prog_copy_branch_snapshot(struct bpf_tramp_links *tl)
+{
+ bool copy = false;
+ int i;
+
+ for (i = 0; i < tl->nr_links; i++)
+ copy = copy || tl->links[i]->link.prog->copy_branch_snapshot;
+
+ return copy;
+}
+
/* mov rax, qword ptr [rbp - rounded_stack_depth - 8] */
#define LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack) \
__LOAD_TCC_PTR(-round_up(stack, 8) - 8)
@@ -3366,6 +3416,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
save_args(m, &prog, regs_off, false, flags);
+ if (bpf_prog_copy_branch_snapshot(fentry)) {
+ /* Get branch snapshot asap. */
+ if (invoke_branch_snapshot(&prog, image, rw_image)) {
+ ret = -EINVAL;
+ goto cleanup;
+ }
+ }
+
if (flags & BPF_TRAMP_F_CALL_ORIG) {
/* arg1: mov rdi, im */
emit_mov_imm64(&prog, BPF_REG_1, (long) im >> 32, (u32) (long) im);
@@ -3422,6 +3480,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im
emit_nops(&prog, X86_PATCH_SIZE);
}
+ if (bpf_prog_copy_branch_snapshot(fexit)) {
+ /* Get branch snapshot asap. */
+ if (invoke_branch_snapshot(&prog, image, rw_image)) {
+ ret = -EINVAL;
+ goto cleanup;
+ }
+ }
+
if (fmod_ret->nr_links) {
/* From Intel 64 and IA-32 Architectures Optimization
* Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 5936f8e2996f..16dc21836a06 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -6,6 +6,7 @@
#include <uapi/linux/bpf.h>
#include <uapi/linux/filter.h>
+#include <uapi/linux/perf_event.h>
#include <crypto/sha2.h>
#include <linux/workqueue.h>
@@ -1236,6 +1237,18 @@ struct bpf_tramp_links {
struct bpf_tramp_run_ctx;
+#ifdef CONFIG_X86_64
+/* Same as MAX_LBR_ENTRIES in arch/x86/events/perf_event.h */
+#define MAX_BRANCH_ENTRIES 32
+
+struct bpf_tramp_branch_entries {
+ int cnt;
+ struct perf_branch_entry entries[MAX_BRANCH_ENTRIES];
+};
+
+DECLARE_PER_CPU(struct bpf_tramp_branch_entries, bpf_branch_snapshot);
+#endif
+
/* Different use cases for BPF trampoline:
* 1. replace nop at the function entry (kprobe equivalent)
* flags = BPF_TRAMP_F_RESTORE_REGS
@@ -1780,7 +1793,8 @@ struct bpf_prog {
call_get_stack:1, /* Do we call bpf_get_stack() or bpf_get_stackid() */
call_get_func_ip:1, /* Do we call get_func_ip() */
tstamp_type_access:1, /* Accessed __sk_buff->tstamp_type */
- sleepable:1; /* BPF program is sleepable */
+ sleepable:1, /* BPF program is sleepable */
+ copy_branch_snapshot:1; /* Copy branch snapshot from prefetched buffer */
enum bpf_prog_type type; /* Type of BPF program */
enum bpf_attach_type expected_attach_type; /* For some prog types */
u32 len; /* Number of filter blocks */
--
2.52.0
^ permalink raw reply related
* [PATCH bpf-next 0/3] bpf: Introduce BPF_BRANCH_SNAPSHOT_F_COPY flag for bpf_get_branch_snapshot helper
From: Leon Hwang @ 2026-01-09 15:34 UTC (permalink / raw)
To: bpf
Cc: Alexei Starovoitov, Daniel Borkmann, Andrii Nakryiko,
Martin KaFai Lau, Eduard Zingerman, Song Liu, Yonghong Song,
John Fastabend, KP Singh, Stanislav Fomichev, Hao Luo, Jiri Olsa,
David S . Miller, David Ahern, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H . Peter Anvin,
Matt Bobrowski, Steven Rostedt, Masami Hiramatsu,
Mathieu Desnoyers, Shuah Khan, Leon Hwang, netdev, linux-kernel,
linux-trace-kernel, linux-kselftest, kernel-patches-bot
When the PMU LBR is running in branch-sensitive mode,
'perf_snapshot_branch_stack()' may capture branch entries from the
trampoline entry up to the call site inside a BPF program. These branch
entries are not useful for analyzing the control flow of the tracee.
To eliminate such noise for tracing programs, the branch snapshot should
be taken as early as possible:
* Call 'perf_snapshot_branch_stack()' at the very beginning of the
trampoline for fentry programs.
* Call 'perf_snapshot_branch_stack()' immediately after invoking the
tracee for fexit programs.
With this change, LBR snapshots remain meaningful even when multiple BPF
programs execute before the one requesting LBR data.
In addition, more relevant branch entries can be captured on AMD CPUs,
which provide a 16-entry-deep LBR stack.
Testing
The series was tested in a VM configured with LBR enabled:
vmtest --kvm-cpu-args 'host,pmu=on,lbr-fmt=0x5' -k $(make -s image_name) -
Branch records were verified using bpfsnoop [1]:
/path/to/bpfsnoop -k '(l)icmp_rcv' -E 1 -v \
--kernel-vmlinux /path/to/kernel/vmlinux
For comparison, the following command was used without
BPF_BRANCH_SNAPSHOT_F_COPY:
/path/to/bpfsnoop -k '(l)icmp_rcv' -E 1 -v \
--force-get-branch-snapshot --kernel-vmlinux /path/to/kernel/vmlinux
Without BPF_BRANCH_SNAPSHOT_F_COPY, no branch records related to the
tracee are captured. With it enabled, 17 branch records from the tracee
are observed.
Detailed verification results are available in the gist [2].
With this series applied, retsnoop [3] can benefit from improved LBR
support when using the '--lbr --fentries' options.
Links:
[1] https://github.com/bpfsnoop/bpfsnoop
[2] https://gist.github.com/Asphaltt/cffdeb4b2f2db4c3c42f91a59109f9e7
[3] https://github.com/anakryiko/retsnoop
Leon Hwang (3):
bpf, x64: Call perf_snapshot_branch_stack in trampoline
bpf: Introduce BPF_BRANCH_SNAPSHOT_F_COPY flag for
bpf_get_branch_snapshot helper
selftests/bpf: Add BPF_BRANCH_SNAPSHOT_F_COPY test
arch/x86/net/bpf_jit_comp.c | 66 +++++++++++++++++++
include/linux/bpf.h | 18 ++++-
include/linux/bpf_verifier.h | 1 +
kernel/bpf/verifier.c | 30 +++++++++
kernel/trace/bpf_trace.c | 17 ++++-
.../bpf/prog_tests/get_branch_snapshot.c | 26 +++++++-
.../selftests/bpf/progs/get_branch_snapshot.c | 3 +-
7 files changed, 153 insertions(+), 8 deletions(-)
--
2.52.0
^ permalink raw reply
* [PATCH 2/2] Documentation/trace: Fix links to other documents
From: Petr Vorel @ 2026-01-09 15:23 UTC (permalink / raw)
To: linux-doc; +Cc: Petr Vorel, linux-trace-kernel, Jonathan Corbet
In-Reply-To: <20260109152336.84674-1-pvorel@suse.cz>
Link to another document does not require 'file:'. Removing it fixes
links in generated html docs.
Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Documentation/trace/fprobe.rst | 2 +-
Documentation/trace/ftrace-uses.rst | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Documentation/trace/fprobe.rst b/Documentation/trace/fprobe.rst
index 06b0edad01796..1d9e0b1693494 100644
--- a/Documentation/trace/fprobe.rst
+++ b/Documentation/trace/fprobe.rst
@@ -79,7 +79,7 @@ The above is defined by including the header::
Same as ftrace, the registered callbacks will start being called some time
after the register_fprobe() is called and before it returns. See
-:file:`Documentation/trace/ftrace.rst`.
+`Documentation/trace/ftrace.rst`.
Also, the unregister_fprobe() will guarantee that both enter and exit
handlers are no longer being called by functions after unregister_fprobe()
diff --git a/Documentation/trace/ftrace-uses.rst b/Documentation/trace/ftrace-uses.rst
index e225cc46b71eb..e7cea7b1a74d9 100644
--- a/Documentation/trace/ftrace-uses.rst
+++ b/Documentation/trace/ftrace-uses.rst
@@ -253,7 +253,7 @@ If @buf is NULL and reset is set, all functions will be enabled for tracing.
The @buf can also be a glob expression to enable all functions that
match a specific pattern.
-See Filter Commands in :file:`Documentation/trace/ftrace.rst`.
+See Filter Commands in `Documentation/trace/ftrace.rst`.
To just trace the schedule function:
--
2.51.0
^ permalink raw reply related
* Re: [PATCH 3/5] tracing: Have tracer option be instance specific
From: Steven Rostedt @ 2026-01-09 15:23 UTC (permalink / raw)
To: Dan Carpenter
Cc: oe-kbuild, linux-kernel, linux-trace-kernel, lkp, oe-kbuild-all,
Masami Hiramatsu, Mark Rutland, Mathieu Desnoyers, Andrew Morton,
Linux Memory Management List
In-Reply-To: <202511071533.domGENBS-lkp@intel.com>
On Mon, 5 Jan 2026 18:23:31 +0300
Dan Carpenter <dan.carpenter@linaro.org> wrote:
> Hi Steven,
Hi Dan,
>
> kernel test robot noticed the following build warnings:
>
> https://git-scm.com/docs/git-format-patch#_base_tree_information]
>
> url: https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/tracing-Remove-dummy-options-and-flags/20251106-010511
> base: https://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace for-next
> patch link: https://lore.kernel.org/r/20251105161935.545400234%40kernel.org
This is an old patch (v1) and v3 fixed this issue:
https://lore.kernel.org/all/20251110234341.767135255@kernel.org/
and v4 was applied and is now in mainline.
The patch you are reporting on was sent Nov 5th, the fix was sent Nov 10th.
Why is this sending reports about an old patch that is obsolete? Is there
something wrong with your setup?
-- Steve
> patch subject: [PATCH 3/5] tracing: Have tracer option be instance specific
> config: i386-randconfig-r072-20251107 (https://download.01.org/0day-ci/archive/20251107/202511071533.domGENBS-lkp@intel.com/config)
> compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Reported-by: Dan Carpenter <dan.carpenter@linaro.org>
> | Closes: https://lore.kernel.org/r/202511071533.domGENBS-lkp@intel.com/
>
> smatch warnings:
> kernel/trace/trace.c:6313 tracing_set_tracer() warn: iterator used outside loop: 't'
>
> vim +/t +6313 kernel/trace/trace.c
>
> 9c5b9d3d65e4858 Masami Hiramatsu 2020-01-11 6285 int tracing_set_tracer(struct trace_array *tr, const char *buf)
> 09d23a1d8a82e81 Steven Rostedt (Red Hat 2015-02-03 6286) {
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 6287 struct tracer *trace;
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 6288 struct tracers *t;
> 12883efb670c28d Steven Rostedt (Red Hat 2013-03-05 6289) #ifdef CONFIG_TRACER_MAX_TRACE
> 34600f0e9c33c9c Steven Rostedt 2013-01-22 6290 bool had_max_tr;
> 12883efb670c28d Steven Rostedt (Red Hat 2013-03-05 6291) #endif
> d33b10c0c73adca Steven Rostedt 2024-12-24 6292 int ret;
> bc0c38d139ec7fc Steven Rostedt 2008-05-12 6293
> d33b10c0c73adca Steven Rostedt 2024-12-24 6294 guard(mutex)(&trace_types_lock);
> 1027fcb206a0fb8 Steven Rostedt 2009-03-12 6295
> 7a1d1e4b9639ff0 Steven Rostedt (Google 2024-06-12 6296) update_last_data(tr);
> 7a1d1e4b9639ff0 Steven Rostedt (Google 2024-06-12 6297)
> a1f157c7a3bb342 Zheng Yejian 2023-09-06 6298 if (!tr->ring_buffer_expanded) {
> 2b6080f28c7cc3e Steven Rostedt 2012-05-11 6299 ret = __tracing_resize_ring_buffer(tr, trace_buf_size,
> 438ced1720b5840 Vaibhav Nagarnaik 2012-02-02 6300 RING_BUFFER_ALL_CPUS);
> 73c5162aa362a54 Steven Rostedt 2009-03-11 6301 if (ret < 0)
> d33b10c0c73adca Steven Rostedt 2024-12-24 6302 return ret;
> 73c5162aa362a54 Steven Rostedt 2009-03-11 6303 ret = 0;
> 73c5162aa362a54 Steven Rostedt 2009-03-11 6304 }
> 73c5162aa362a54 Steven Rostedt 2009-03-11 6305
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 6306 list_for_each_entry(t, &tr->tracers, list) {
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 6307 if (strcmp(t->tracer->name, buf) == 0)
> bc0c38d139ec7fc Steven Rostedt 2008-05-12 6308 break;
> bc0c38d139ec7fc Steven Rostedt 2008-05-12 6309 }
> d33b10c0c73adca Steven Rostedt 2024-12-24 6310 if (!t)
>
> t can't be NULL here. It needs to be if (list_entry_is_head()) return;
>
> d33b10c0c73adca Steven Rostedt 2024-12-24 6311 return -EINVAL;
> d33b10c0c73adca Steven Rostedt 2024-12-24 6312
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 @6313 if (t->tracer == tr->current_trace)
> d33b10c0c73adca Steven Rostedt 2024-12-24 6314 return 0;
> bc0c38d139ec7fc Steven Rostedt 2008-05-12 6315
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 6316 trace = t->tracer;
> 4699bbb369ba1d3 Steven Rostedt 2025-11-05 6317
>
^ permalink raw reply
* Re: [PATCH V4 2/2] mm/khugepaged: retry with sync writeback for MADV_COLLAPSE
From: David Hildenbrand (Red Hat) @ 2026-01-09 14:46 UTC (permalink / raw)
To: Shivank Garg, Andrew Morton, Lorenzo Stoakes
Cc: Zi Yan, Baolin Wang, Liam R . Howlett, Nico Pache, Ryan Roberts,
Dev Jain, Barry Song, Lance Yang, Steven Rostedt,
Masami Hiramatsu, Mathieu Desnoyers, Zach O'Keefe, linux-mm,
linux-kernel, linux-trace-kernel, Branden Moore
In-Reply-To: <20251215084615.5283-7-shivankg@amd.com>
On 12/15/25 09:46, Shivank Garg wrote:
> When MADV_COLLAPSE is called on file-backed mappings (e.g., executable
> text sections), the pages may still be dirty from recent writes.
> collapse_file() will trigger async writeback and fail with
> SCAN_PAGE_DIRTY_OR_WRITEBACK (-EAGAIN).
>
> MADV_COLLAPSE is a synchronous operation where userspace expects
> immediate results. If the collapse fails due to dirty pages, perform
> synchronous writeback on the specific range and retry once.
>
> This avoids spurious failures for freshly written executables while
> avoiding unnecessary synchronous I/O for mappings that are already clean.
>
> Reported-by: Branden Moore <Branden.Moore@amd.com>
> Closes: https://lore.kernel.org/all/4e26fe5e-7374-467c-a333-9dd48f85d7cc@amd.com
> Fixes: 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE")
> Suggested-by: David Hildenbrand <david@kernel.org>
> Tested-by: Lance Yang <lance.yang@linux.dev>
> Signed-off-by: Shivank Garg <shivankg@amd.com>
> ---
> mm/khugepaged.c | 40 ++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 40 insertions(+)
>
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 219dfa2e523c..6c8c35d3e0c9 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -22,6 +22,7 @@
> #include <linux/dax.h>
> #include <linux/ksm.h>
> #include <linux/pgalloc.h>
> +#include <linux/backing-dev.h>
>
> #include <asm/tlb.h>
> #include "internal.h"
> @@ -2787,9 +2788,11 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
> hend = end & HPAGE_PMD_MASK;
>
> for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
> + bool retried = false;
> int result = SCAN_FAIL;
>
> if (!mmap_locked) {
> +retry:
Jumping into an if block is nasty :)
> cond_resched();
> mmap_read_lock(mm);
> mmap_locked = true;
> @@ -2819,6 +2822,43 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
> if (!mmap_locked)
> *lock_dropped = true;
>
> + /*
> + * If the file-backed VMA has dirty pages, the scan triggers
> + * async writeback and returns SCAN_PAGE_DIRTY_OR_WRITEBACK.
> + * Since MADV_COLLAPSE is sync, we force sync writeback and
> + * retry once.
> + */
> + if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !retried) {
> + /*
> + * File scan drops the lock. We must re-acquire it to
> + * safely inspect the VMA and hold the file reference.
> + */
> + if (!mmap_locked) {
> + cond_resched();
> + mmap_read_lock(mm);
> + mmap_locked = true;
> + result = hugepage_vma_revalidate(mm, addr, false, &vma, cc);
> + if (result != SCAN_SUCCEED)
> + goto handle_result;
> + }
> +
> + if (!vma_is_anonymous(vma) && vma->vm_file &&
> + mapping_can_writeback(vma->vm_file->f_mapping)) {
> + struct file *file = get_file(vma->vm_file);
> + pgoff_t pgoff = linear_page_index(vma, addr);
> + loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
> + loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
> +
> + mmap_read_unlock(mm);
> + mmap_locked = false;
> + *lock_dropped = true;
> + filemap_write_and_wait_range(file->f_mapping, lstart, lend);
> + fput(file);
> + retried = true;
> + goto retry;
> + }
> + }
> +
This looks a bit complicated. Can't we move that handing up, where we have most of that
information already? Or am I missing something important?
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 97d1b2824386f..c7271877c5220 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -22,6 +22,7 @@
#include <linux/dax.h>
#include <linux/ksm.h>
#include <linux/pgalloc.h>
+#include <linux/backing-dev.h>
#include <asm/tlb.h>
#include "internal.h"
@@ -2786,7 +2787,9 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
int result = SCAN_FAIL;
+ bool triggered_wb = false;
+retry:
if (!mmap_locked) {
cond_resched();
mmap_read_lock(mm);
@@ -2809,6 +2812,16 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
mmap_locked = false;
result = hpage_collapse_scan_file(mm, addr, file, pgoff,
cc);
+
+ if (result == SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb &&
+ mapping_can_writeback(file->f_mapping)) {
+ loff_t lstart = (loff_t)pgoff << PAGE_SHIFT;
+ loff_t lend = lstart + HPAGE_PMD_SIZE - 1;
+
+ filemap_write_and_wait_range(file->f_mapping, lstart, lend);
+ triggered_wb = true;
+ goto retry;
+ }
fput(file);
} else {
result = hpage_collapse_scan_pmd(mm, vma, addr,
--
Cheers
David
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox