[PATCH 0/2] arm64: ftrace: enable single ftrace_ops for direct calls

[PATCH 0/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

Beyond the DIRECT_CALLS support on arm64 [1], select
HAVE_SINGLE_FTRACE_DIRECT_OPS for arm64, which enables the BPF
tracing_multi link on arm64 [2].

All tracing_multi link selftests are passed on arm64.

 cd tools/testing/selftests/bpf
 ./test_progs -a 'tracing_multi_*'
 #545/1   tracing_multi_attach_rollback/put:OK
 #545/2   tracing_multi_attach_rollback/unlink:OK
 #545     tracing_multi_attach_rollback:OK
 #546     tracing_multi_bench_attach:OK
 #547/1   tracing_multi_test/skel_api:OK
 #547/2   tracing_multi_test/link_api_pattern:OK
 #547/3   tracing_multi_test/link_api_ids:OK
 #547/4   tracing_multi_test/module_skel_api:OK
 #547/5   tracing_multi_test/module_link_api_pattern:OK
 #547/6   tracing_multi_test/module_link_api_ids:OK
 #547/7   tracing_multi_test/intersect:OK
 #547/8   tracing_multi_test/cookies:OK
 #547/9   tracing_multi_test/session:OK
 #547/10  tracing_multi_test/attach_api_fails:OK
 #547/11  tracing_multi_test/fentry_direct_access:OK
 #547/12  tracing_multi_test/fexit_direct_access:OK
 #547/13  tracing_multi_test/fsession_direct_access:OK
 #547/14  tracing_multi_test/fentry_after_multi:OK
 #547     tracing_multi_test:OK
 Summary: 3/16 PASSED, 0 SKIPPED, 0 FAILED

 ./test_progs -a 'fill_link_info/tracing_multi_*'
 #132/11  fill_link_info/tracing_multi_link_info:OK
 #132/12  fill_link_info/tracing_multi_invalid_ubuff:OK
 #132     fill_link_info:OK
 Summary: 1/2 PASSED, 0 SKIPPED, 0 FAILED

Links:
[1] https://lore.kernel.org/all/20260609-arm64-ftrace-direct-calls-v1-0-4a46f266697f@linux.dev/
[2] https://github.com/kernel-patches/bpf/pull/12986

Leon Hwang (2):
  arm64: ftrace: enable single ftrace_ops for direct calls
  selftests/bpf: Enable tracing_multi tests on arm64

 arch/arm64/Kconfig                                     | 2 ++
 arch/arm64/kernel/ftrace.c                             | 3 +--
 tools/testing/selftests/bpf/prog_tests/tracing_multi.c | 9 +++++++--
 tools/testing/selftests/bpf/test_progs.h               | 6 ++++++
 4 files changed, 16 insertions(+), 4 deletions(-)

-- 
2.55.0

[PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. Its implementation is therefore gated by
HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
DYNAMIC_FTRACE_WITH_DIRECT_CALLS.

Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
direct calls. This enables BPF tracing multi links on arm64. Also
generalize the unreachable-trampoline comment because the single-ops path
does not use ops->direct_call.

Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
---
 arch/arm64/Kconfig         | 2 ++
 arch/arm64/kernel/ftrace.c | 3 +--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 0de419ed780f..c98dca76859b 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -188,6 +188,8 @@ config ARM64
 		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
 	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
 		if DYNAMIC_FTRACE_WITH_ARGS
+	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
+		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS
 	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
 		if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \
 		    (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
index e1a3c0b3a051..56ba72a87dfa 100644
--- a/arch/arm64/kernel/ftrace.c
+++ b/arch/arm64/kernel/ftrace.c
@@ -301,8 +301,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec,
 
 	/*
 	 * If a custom trampoline is unreachable, rely on the ftrace_caller
-	 * trampoline which knows how to indirectly reach that trampoline
-	 * through ops->direct_call.
+	 * trampoline which knows how to indirectly reach that trampoline.
 	 */
 	if (*addr != FTRACE_ADDR && !reachable_by_bl(*addr, pc))
 		*addr = FTRACE_ADDR;
-- 
2.55.0

[PATCH 2/2] selftests/bpf: Enable tracing_multi tests on arm64

[Leon Hwang]

arm64 now supports dynamic ftrace direct calls backed by a single
ftrace_ops, which enables BPF tracing multi links.

Define ARCH_SUPPORTS_TRACING_MULTI_LINK for x86-64 and arm64, and use the
shared capability to gate the tracing_multi functional, rollback,
and attach benchmark tests.

This allows the tests to run on arm64 while continuing to skip
architectures without tracing multi link support. It also prevents the
rollback test from treating an unsupported-link error as its expected
attachment failure.

Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
---
 tools/testing/selftests/bpf/prog_tests/tracing_multi.c | 9 +++++++--
 tools/testing/selftests/bpf/test_progs.h               | 6 ++++++
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/bpf/prog_tests/tracing_multi.c b/tools/testing/selftests/bpf/prog_tests/tracing_multi.c
index f02ffc7f41d7..5aa48e090340 100644
--- a/tools/testing/selftests/bpf/prog_tests/tracing_multi.c
+++ b/tools/testing/selftests/bpf/prog_tests/tracing_multi.c
@@ -612,7 +612,7 @@ void serial_test_tracing_multi_bench_attach(void)
 	struct btf *btf;
 	int err;
 
-#ifndef __x86_64__
+#if !ARCH_SUPPORTS_TRACING_MULTI_LINK
 	test__skip();
 	return;
 #endif
@@ -923,6 +923,11 @@ static void test_rollback_unlink(void)
 
 void serial_test_tracing_multi_attach_rollback(void)
 {
+#if !ARCH_SUPPORTS_TRACING_MULTI_LINK
+	test__skip();
+	return;
+#endif
+
 	if (test__start_subtest("put"))
 		test_rollback_put();
 	if (test__start_subtest("unlink"))
@@ -931,7 +936,7 @@ void serial_test_tracing_multi_attach_rollback(void)
 
 void test_tracing_multi_test(void)
 {
-#ifndef __x86_64__
+#if !ARCH_SUPPORTS_TRACING_MULTI_LINK
 	test__skip();
 	return;
 #endif
diff --git a/tools/testing/selftests/bpf/test_progs.h b/tools/testing/selftests/bpf/test_progs.h
index 2cf950afcd85..b1498e84773c 100644
--- a/tools/testing/selftests/bpf/test_progs.h
+++ b/tools/testing/selftests/bpf/test_progs.h
@@ -567,4 +567,10 @@ void validate_msgs(const char *log_buf, struct expected_msgs *msgs,
 void free_msgs(struct expected_msgs *msgs);
 void verify_test_stderr(struct bpf_object *obj, struct bpf_program *prog);
 
+#if defined(__x86_64__) || defined(__aarch64__)
+#define ARCH_SUPPORTS_TRACING_MULTI_LINK 1
+#else
+#define ARCH_SUPPORTS_TRACING_MULTI_LINK 0
+#endif
+
 #endif /* __TEST_PROGS_H */
-- 
2.55.0

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Missing RCU read barrier (`rcu_dereference()`) when accessing `direct_functions`.
--

--- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls ---
commit 738274783fa707248af104af5e45da296f7e4d84
Author: Leon Hwang <leon.hwang@linux.dev>

arm64: ftrace: enable single ftrace_ops for direct calls

The BPF tracing multi link updates several direct-call sites through one
ftrace_ops. This commit selects HAVE_SINGLE_FTRACE_DIRECT_OPS on arm64 when
DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled. It also generalizes the
unreachable-trampoline comment.

> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 39eba438dad43..94fa6e72c2867 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -188,6 +188,8 @@ config ARM64
>  		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
>  	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
>  		if DYNAMIC_FTRACE_WITH_ARGS
> +	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
> +		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS

[Severity: High]
This is a pre-existing issue, but by enabling HAVE_SINGLE_FTRACE_DIRECT_OPS
on arm64, does this expose the architecture to a missing RCU read barrier
in the ftrace code?

When HAVE_SINGLE_FTRACE_DIRECT_OPS is selected, call_direct_funcs() switches
from using READ_ONCE(ops->direct_call) to using ftrace_find_rec_direct():

kernel/trace/ftrace.c:call_direct_funcs() {
...
#ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
	addr = ftrace_find_rec_direct(ip);
#else
...
}

Looking at ftrace_find_rec_direct(), it accesses the __rcu pointer
direct_functions directly without an rcu_dereference():

kernel/trace/ftrace.c:ftrace_find_rec_direct() {
	struct ftrace_func_entry *entry;

	entry = __ftrace_lookup_ip(direct_functions, ip);
...
}

Without the READ_ONCE() provided by rcu_dereference(), the compiler might
reload the pointer between computing the hash key and accessing the bucket
array inside __ftrace_lookup_ip().

If a concurrent writer substitutes a new hash table with a different size
between these reloads, could the computed key exceed the new bucket array
bounds and lead to an out-of-bounds dereference?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260727142844.21212-1-leon.hwang@linux.dev?part=1

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

On 27/7/26 22:52, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> 
> Pre-existing issues:
> - [High] Missing RCU read barrier (`rcu_dereference()`) when accessing `direct_functions`.
> --
> 
> --- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls ---
> commit 738274783fa707248af104af5e45da296f7e4d84
> Author: Leon Hwang <leon.hwang@linux.dev>
> 
> arm64: ftrace: enable single ftrace_ops for direct calls
> 
> The BPF tracing multi link updates several direct-call sites through one
> ftrace_ops. This commit selects HAVE_SINGLE_FTRACE_DIRECT_OPS on arm64 when
> DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled. It also generalizes the
> unreachable-trampoline comment.
> 
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 39eba438dad43..94fa6e72c2867 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -188,6 +188,8 @@ config ARM64
>>  		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
>>  	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
>>  		if DYNAMIC_FTRACE_WITH_ARGS
>> +	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
>> +		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS
> 
> [Severity: High]
> This is a pre-existing issue, but by enabling HAVE_SINGLE_FTRACE_DIRECT_OPS
> on arm64, does this expose the architecture to a missing RCU read barrier
> in the ftrace code?
> 
> When HAVE_SINGLE_FTRACE_DIRECT_OPS is selected, call_direct_funcs() switches
> from using READ_ONCE(ops->direct_call) to using ftrace_find_rec_direct():
> 
> kernel/trace/ftrace.c:call_direct_funcs() {
> ...
> #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
> 	addr = ftrace_find_rec_direct(ip);
> #else
> ...
> }
> 
> Looking at ftrace_find_rec_direct(), it accesses the __rcu pointer
> direct_functions directly without an rcu_dereference():
> 
> kernel/trace/ftrace.c:ftrace_find_rec_direct() {
> 	struct ftrace_func_entry *entry;
> 
> 	entry = __ftrace_lookup_ip(direct_functions, ip);
> ...
> }
> 
> Without the READ_ONCE() provided by rcu_dereference(), the compiler might
> reload the pointer between computing the hash key and accessing the bucket
> array inside __ftrace_lookup_ip().
> 
> If a concurrent writer substitutes a new hash table with a different size
> between these reloads, could the computed key exceed the new bucket array
> bounds and lead to an out-of-bounds dereference?
> 


Hi Steve, Masami,

Could you take a look at this issue? The issue looks legitimate.

Thanks,
Leon

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Steven Rostedt]

On Tue, 28 Jul 2026 10:19:22 +0800
Leon Hwang <leon.hwang@linux.dev> wrote:

> On 27/7/26 22:52, sashiko-bot@kernel.org wrote:
> > Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> > 
> > Pre-existing issues:
> > - [High] Missing RCU read barrier (`rcu_dereference()`) when accessing `direct_functions`.
> > --
> > 
> > --- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls ---
> > commit 738274783fa707248af104af5e45da296f7e4d84
> > Author: Leon Hwang <leon.hwang@linux.dev>
> > 
> > arm64: ftrace: enable single ftrace_ops for direct calls
> > 
> > The BPF tracing multi link updates several direct-call sites through one
> > ftrace_ops. This commit selects HAVE_SINGLE_FTRACE_DIRECT_OPS on arm64 when
> > DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled. It also generalizes the
> > unreachable-trampoline comment.
> >   
> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> >> index 39eba438dad43..94fa6e72c2867 100644
> >> --- a/arch/arm64/Kconfig
> >> +++ b/arch/arm64/Kconfig
> >> @@ -188,6 +188,8 @@ config ARM64
> >>  		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
> >>  	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> >>  		if DYNAMIC_FTRACE_WITH_ARGS
> >> +	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
> >> +		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS  
> > 
> > [Severity: High]
> > This is a pre-existing issue, but by enabling HAVE_SINGLE_FTRACE_DIRECT_OPS
> > on arm64, does this expose the architecture to a missing RCU read barrier
> > in the ftrace code?
> > 
> > When HAVE_SINGLE_FTRACE_DIRECT_OPS is selected, call_direct_funcs() switches
> > from using READ_ONCE(ops->direct_call) to using ftrace_find_rec_direct():
> > 
> > kernel/trace/ftrace.c:call_direct_funcs() {
> > ...
> > #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
> > 	addr = ftrace_find_rec_direct(ip);
> > #else
> > ...
> > }
> > 
> > Looking at ftrace_find_rec_direct(), it accesses the __rcu pointer
> > direct_functions directly without an rcu_dereference():
> > 
> > kernel/trace/ftrace.c:ftrace_find_rec_direct() {
> > 	struct ftrace_func_entry *entry;
> > 
> > 	entry = __ftrace_lookup_ip(direct_functions, ip);
> > ...
> > }
> > 
> > Without the READ_ONCE() provided by rcu_dereference(), the compiler might
> > reload the pointer between computing the hash key and accessing the bucket
> > array inside __ftrace_lookup_ip().
> > 
> > If a concurrent writer substitutes a new hash table with a different size
> > between these reloads, could the computed key exceed the new bucket array
> > bounds and lead to an out-of-bounds dereference?
> >   
> 
> 
> Hi Steve, Masami,
> 
> Could you take a look at this issue? The issue looks legitimate.

Yeah, it's possible (but unlikely) for the compiler to do that. But there's
nothing harming the code to do:

( Untested and not even compiled )

diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index f93e34dd2328..5ec494790312 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -2639,7 +2639,8 @@ unsigned long ftrace_find_rec_direct(unsigned long ip)
 {
 	struct ftrace_func_entry *entry;
 
-	entry = __ftrace_lookup_ip(direct_functions, ip);
+	guard(preempt_notrace)();
+	entry = __ftrace_lookup_ip(rcu_dereference_sched(direct_functions), ip);
 	if (!entry)
 		return 0;
 

-- Steve

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

On 28/7/26 23:46, Steven Rostedt wrote:
> On Tue, 28 Jul 2026 10:19:22 +0800
> Leon Hwang <leon.hwang@linux.dev> wrote:
> 
>> On 27/7/26 22:52, sashiko-bot@kernel.org wrote:
>>> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
>>>
>>> Pre-existing issues:
>>> - [High] Missing RCU read barrier (`rcu_dereference()`) when accessing `direct_functions`.
>>> --
>>>
>>> --- Patch [1]: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls ---
>>> commit 738274783fa707248af104af5e45da296f7e4d84
>>> Author: Leon Hwang <leon.hwang@linux.dev>
>>>
>>> arm64: ftrace: enable single ftrace_ops for direct calls
>>>
>>> The BPF tracing multi link updates several direct-call sites through one
>>> ftrace_ops. This commit selects HAVE_SINGLE_FTRACE_DIRECT_OPS on arm64 when
>>> DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled. It also generalizes the
>>> unreachable-trampoline comment.
>>>   
>>>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>>>> index 39eba438dad43..94fa6e72c2867 100644
>>>> --- a/arch/arm64/Kconfig
>>>> +++ b/arch/arm64/Kconfig
>>>> @@ -188,6 +188,8 @@ config ARM64
>>>>  		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
>>>>  	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
>>>>  		if DYNAMIC_FTRACE_WITH_ARGS
>>>> +	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
>>>> +		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS  
>>>
>>> [Severity: High]
>>> This is a pre-existing issue, but by enabling HAVE_SINGLE_FTRACE_DIRECT_OPS
>>> on arm64, does this expose the architecture to a missing RCU read barrier
>>> in the ftrace code?
>>>
>>> When HAVE_SINGLE_FTRACE_DIRECT_OPS is selected, call_direct_funcs() switches
>>> from using READ_ONCE(ops->direct_call) to using ftrace_find_rec_direct():
>>>
>>> kernel/trace/ftrace.c:call_direct_funcs() {
>>> ...
>>> #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS
>>> 	addr = ftrace_find_rec_direct(ip);
>>> #else
>>> ...
>>> }
>>>
>>> Looking at ftrace_find_rec_direct(), it accesses the __rcu pointer
>>> direct_functions directly without an rcu_dereference():
>>>
>>> kernel/trace/ftrace.c:ftrace_find_rec_direct() {
>>> 	struct ftrace_func_entry *entry;
>>>
>>> 	entry = __ftrace_lookup_ip(direct_functions, ip);
>>> ...
>>> }
>>>
>>> Without the READ_ONCE() provided by rcu_dereference(), the compiler might
>>> reload the pointer between computing the hash key and accessing the bucket
>>> array inside __ftrace_lookup_ip().
>>>
>>> If a concurrent writer substitutes a new hash table with a different size
>>> between these reloads, could the computed key exceed the new bucket array
>>> bounds and lead to an out-of-bounds dereference?
>>>   
>>
>>
>> Hi Steve, Masami,
>>
>> Could you take a look at this issue? The issue looks legitimate.
> 
> Yeah, it's possible (but unlikely) for the compiler to do that. But there's
> nothing harming the code to do:


Good to know it's unlikely in practice.

If you don't mind, I'll send a separate patch for the fix based on the
diff below. It compiles and works fine in my testing.

Thanks,
Leon

> 
> ( Untested and not even compiled )
> 
> diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
> index f93e34dd2328..5ec494790312 100644
> --- a/kernel/trace/ftrace.c
> +++ b/kernel/trace/ftrace.c
> @@ -2639,7 +2639,8 @@ unsigned long ftrace_find_rec_direct(unsigned long ip)
>  {
>  	struct ftrace_func_entry *entry;
>  
> -	entry = __ftrace_lookup_ip(direct_functions, ip);
> +	guard(preempt_notrace)();
> +	entry = __ftrace_lookup_ip(rcu_dereference_sched(direct_functions), ip);
>  	if (!entry)
>  		return 0;
>  
> 
> -- Steve
> 

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Jiri Olsa]

On Mon, Jul 27, 2026 at 10:28:43PM +0800, Leon Hwang wrote:
> The BPF tracing multi link updates several direct-call sites through one
> ftrace_ops. Its implementation is therefore gated by
> HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
> DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
> 
> Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
> direct calls. This enables BPF tracing multi links on arm64. Also
> generalize the unreachable-trampoline comment because the single-ops path
> does not use ops->direct_call.

hi,
iirc arm needs ops->direct_call for trampolines to work properly,
that's the reason we enabled single ftrace_ops on x86 only:

  424f6a361096 bpf,x86: Use single ftrace_ops for direct calls

I don't understand the arm usage of op->direct_call, but it looks like
(codex thinks) that the fast path won't always work without op->direct_call
being set

from arch/arm64/kernel/entry-ftrace.S:

	#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
		/*
		 * If the op has a direct call, handle it immediately without
		 * saving/restoring registers.
		 */
		ldr     x17, [x11, #FTRACE_OPS_DIRECT_CALL]             // op->direct_call
		cbnz    x17, ftrace_caller_direct
	#endif
	#endif

jirka

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

On 29/7/26 19:56, Jiri Olsa wrote:
> On Mon, Jul 27, 2026 at 10:28:43PM +0800, Leon Hwang wrote:
>> The BPF tracing multi link updates several direct-call sites through one
>> ftrace_ops. Its implementation is therefore gated by
>> HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
>> DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
>>
>> Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
>> direct calls. This enables BPF tracing multi links on arm64. Also
>> generalize the unreachable-trampoline comment because the single-ops path
>> does not use ops->direct_call.
> 
> hi,
> iirc arm needs ops->direct_call for trampolines to work properly,
> that's the reason we enabled single ftrace_ops on x86 only:
> 
>   424f6a361096 bpf,x86: Use single ftrace_ops for direct calls
> 
> I don't understand the arm usage of op->direct_call, but it looks like
> (codex thinks) that the fast path won't always work without op->direct_call
> being set


In theory, op->direct_call is the fallback for the case:
HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF
direct ops. This is what Codex thinks about.

However, after searching the call sites of
register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would
set ops->direct_call, the actual callers are in bpf/trampoline.c, which
have been gated with HAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for
normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y, op->direct_call is unused.

If someone uses register_ftrace_direct() or
modify_ftrace_direct/_nolock() outside BPF in the future,
op->direct_call will be used.

So, I'd like to drop the comment change in the next revision.

Thanks,
Leon

> 
> from arch/arm64/kernel/entry-ftrace.S:
> 
> 	#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
> 		/*
> 		 * If the op has a direct call, handle it immediately without
> 		 * saving/restoring registers.
> 		 */
> 		ldr     x17, [x11, #FTRACE_OPS_DIRECT_CALL]             // op->direct_call
> 		cbnz    x17, ftrace_caller_direct
> 	#endif
> 	#endif
> 
> jirka

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Jiri Olsa]

On Thu, Jul 30, 2026 at 11:05:42AM +0800, Leon Hwang wrote:
> On 29/7/26 19:56, Jiri Olsa wrote:
> > On Mon, Jul 27, 2026 at 10:28:43PM +0800, Leon Hwang wrote:
> >> The BPF tracing multi link updates several direct-call sites through one
> >> ftrace_ops. Its implementation is therefore gated by
> >> HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
> >> DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
> >>
> >> Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
> >> direct calls. This enables BPF tracing multi links on arm64. Also
> >> generalize the unreachable-trampoline comment because the single-ops path
> >> does not use ops->direct_call.
> > 
> > hi,
> > iirc arm needs ops->direct_call for trampolines to work properly,
> > that's the reason we enabled single ftrace_ops on x86 only:
> > 
> >   424f6a361096 bpf,x86: Use single ftrace_ops for direct calls
> > 
> > I don't understand the arm usage of op->direct_call, but it looks like
> > (codex thinks) that the fast path won't always work without op->direct_call
> > being set
> 
> 
> In theory, op->direct_call is the fallback for the case:
> HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF
> direct ops. This is what Codex thinks about.
> 
> However, after searching the call sites of
> register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would
> set ops->direct_call, the actual callers are in bpf/trampoline.c, which
> have been gated with HAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for
> normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y, op->direct_call is unused.
> 
> If someone uses register_ftrace_direct() or
> modify_ftrace_direct/_nolock() outside BPF in the future,
> op->direct_call will be used.

so any 2nd user on the same function triggers the slow path,
and that's fine, the single_ops handles slow path correctly

but IIUC the out-of-range target will now always go through slow path,
which in current code it doesn't, right?

jirka

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Ihor Solodrai]

On 7/27/26 7:28 AM, Leon Hwang wrote:
> The BPF tracing multi link updates several direct-call sites through one
> ftrace_ops. Its implementation is therefore gated by
> HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
> DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
> 
> Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
> direct calls. This enables BPF tracing multi links on arm64. Also
> generalize the unreachable-trampoline comment because the single-ops path
> does not use ops->direct_call.

Hi Leon,

I don't think this change can land as is yet. The series doesn't even
apply cleanly to bpf-next, but that's minor.

More importantly, it depends on Jose's series [1], which is not in the
mainline yet. And there Mark has raised performance concerns [2] and
the discussion still seems to be open.

[1] https://lore.kernel.org/all/20260609-arm64-ftrace-direct-calls-v1-0-4a46f266697f@linux.dev/
[2] https://lore.kernel.org/all/amjnf5gz0xP5PTSB@J2N7QTR9R3/

> 
> Assisted-by: Codex:gpt-5.6-sol
> Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
> ---
>  arch/arm64/Kconfig         | 2 ++
>  arch/arm64/kernel/ftrace.c | 3 +--
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 0de419ed780f..c98dca76859b 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -188,6 +188,8 @@ config ARM64
>  		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
>  	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
>  		if DYNAMIC_FTRACE_WITH_ARGS
> +	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
> +		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS\

The select is only conditional on DYNAMIC_FTRACE_WITH_DIRECT_CALLS, so
it can be set along with HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS. And AFAIU
this would make the fast path effectively dead: every BPF direct call
routed through ftrace_caller now goes through the slow path.

As Mark noted in the other thread, on arm64 trampolines come from
EXECMEM_BPF, so they always land out of BL range (chance of landing in
range is 256M/terabytes).

I vibe-slop-coded a benchmark and ran it on a Neoverse V2 machine, and
toggling your config change seems to be causing a 1.3x regression in
the tracing overhead:

  do-nothing fentry (r0=0; exit) on __arm64_sys_getpid, 20 M calls, min-of-N, several boots. Results:

              untraced (base)   traced        overhead
  Kernel A    ~128 ns/call      ~145.5 ns     ~17.6 ns   (fast path: br x17)
  Kernel B    ~127 ns/call      ~150.4 ns     ~23.4 ns   (slow path: save regs + call_direct_funcs + hash)

This confirms Jiri's suspicion.

However my understanding is the regression should mostly disappear in
case some version of in-range trampoline allocation on arm64 lands.

So, I think the landing sequence should be something like follows:
  * in-range BPF-trampoline allocation that Jose proposed [3]
  * then HAVE_SINGLE_FTRACE_DIRECT_OPS selection

After all of that reaches mainline, then a selftest patch can go
through the bpf-next.

[3] https://lore.kernel.org/all/amn7cBQ-cqs2XlKV@linux.dev/

>  	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
>  		if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \
>  		    (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
> diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
> index e1a3c0b3a051..56ba72a87dfa 100644
> --- a/arch/arm64/kernel/ftrace.c
> +++ b/arch/arm64/kernel/ftrace.c
> @@ -301,8 +301,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec,
>  
>  	/*
>  	 * If a custom trampoline is unreachable, rely on the ftrace_caller
> -	 * trampoline which knows how to indirectly reach that trampoline
> -	 * through ops->direct_call.
> +	 * trampoline which knows how to indirectly reach that trampoline.
>  	 */
>  	if (*addr != FTRACE_ADDR && !reachable_by_bl(*addr, pc))
>  		*addr = FTRACE_ADDR;

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

On 31/7/26 07:03, Ihor Solodrai wrote:
> On 7/27/26 7:28 AM, Leon Hwang wrote:
>> The BPF tracing multi link updates several direct-call sites through one
>> ftrace_ops. Its implementation is therefore gated by
>> HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
>> DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
>>
>> Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
>> direct calls. This enables BPF tracing multi links on arm64. Also
>> generalize the unreachable-trampoline comment because the single-ops path
>> does not use ops->direct_call.
> 
> Hi Leon,
> 
> I don't think this change can land as is yet. The series doesn't even
> apply cleanly to bpf-next, but that's minor.
> 
> More importantly, it depends on Jose's series [1], which is not in the
> mainline yet. And there Mark has raised performance concerns [2] and
> the discussion still seems to be open.
> 
> [1] https://lore.kernel.org/all/20260609-arm64-ftrace-direct-calls-v1-0-4a46f266697f@linux.dev/
> [2] https://lore.kernel.org/all/amjnf5gz0xP5PTSB@J2N7QTR9R3/
> 
>>
>> Assisted-by: Codex:gpt-5.6-sol
>> Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
>> ---
>>  arch/arm64/Kconfig         | 2 ++
>>  arch/arm64/kernel/ftrace.c | 3 +--
>>  2 files changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 0de419ed780f..c98dca76859b 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -188,6 +188,8 @@ config ARM64
>>  		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
>>  	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
>>  		if DYNAMIC_FTRACE_WITH_ARGS
>> +	select HAVE_SINGLE_FTRACE_DIRECT_OPS \
>> +		if DYNAMIC_FTRACE_WITH_DIRECT_CALLS\
> 
> The select is only conditional on DYNAMIC_FTRACE_WITH_DIRECT_CALLS, so
> it can be set along with HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS. And AFAIU
> this would make the fast path effectively dead: every BPF direct call
> routed through ftrace_caller now goes through the slow path.
> 
> As Mark noted in the other thread, on arm64 trampolines come from
> EXECMEM_BPF, so they always land out of BL range (chance of landing in
> range is 256M/terabytes).
> 
> I vibe-slop-coded a benchmark and ran it on a Neoverse V2 machine, and
> toggling your config change seems to be causing a 1.3x regression in
> the tracing overhead:
> 
>   do-nothing fentry (r0=0; exit) on __arm64_sys_getpid, 20 M calls, min-of-N, several boots. Results:
> 
>               untraced (base)   traced        overhead
>   Kernel A    ~128 ns/call      ~145.5 ns     ~17.6 ns   (fast path: br x17)
>   Kernel B    ~127 ns/call      ~150.4 ns     ~23.4 ns   (slow path: save regs + call_direct_funcs + hash)


Thanks for your testing.

> 
> This confirms Jiri's suspicion.

True.

> 
> However my understanding is the regression should mostly disappear in
> case some version of in-range trampoline allocation on arm64 lands.
> 
> So, I think the landing sequence should be something like follows:
>   * in-range BPF-trampoline allocation that Jose proposed [3]
>   * then HAVE_SINGLE_FTRACE_DIRECT_OPS selection
> 
> After all of that reaches mainline, then a selftest patch can go
> through the bpf-next.

Sounds reasonable.

This series is based on Jose's series and is intended for the arm64
tree, rather than bpf-next. Like Jose's series, this series aims to
enable BPF tracing_multi link on arm64. And yes, with in-range
BPF-trampoline allocation, the regression should mostly disappear.

I'll follow the sequence and repost the patches afterward.

Thanks,
Leon

> 
> [3] https://lore.kernel.org/all/amn7cBQ-cqs2XlKV@linux.dev/
> 
>>  	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
>>  		if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \
>>  		    (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
>> diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
>> index e1a3c0b3a051..56ba72a87dfa 100644
>> --- a/arch/arm64/kernel/ftrace.c
>> +++ b/arch/arm64/kernel/ftrace.c
>> @@ -301,8 +301,7 @@ static bool ftrace_find_callable_addr(struct dyn_ftrace *rec,
>>  
>>  	/*
>>  	 * If a custom trampoline is unreachable, rely on the ftrace_caller
>> -	 * trampoline which knows how to indirectly reach that trampoline
>> -	 * through ops->direct_call.
>> +	 * trampoline which knows how to indirectly reach that trampoline.
>>  	 */
>>  	if (*addr != FTRACE_ADDR && !reachable_by_bl(*addr, pc))
>>  		*addr = FTRACE_ADDR;
> 

Re: [PATCH 1/2] arm64: ftrace: enable single ftrace_ops for direct calls

[Leon Hwang]

On 30/7/26 18:58, Jiri Olsa wrote:
> On Thu, Jul 30, 2026 at 11:05:42AM +0800, Leon Hwang wrote:
>> On 29/7/26 19:56, Jiri Olsa wrote:
>>> On Mon, Jul 27, 2026 at 10:28:43PM +0800, Leon Hwang wrote:
>>>> The BPF tracing multi link updates several direct-call sites through one
>>>> ftrace_ops. Its implementation is therefore gated by
>>>> HAVE_SINGLE_FTRACE_DIRECT_OPS in addition to
>>>> DYNAMIC_FTRACE_WITH_DIRECT_CALLS.
>>>>
>>>> Select HAVE_SINGLE_FTRACE_DIRECT_OPS whenever arm64 enables dynamic ftrace
>>>> direct calls. This enables BPF tracing multi links on arm64. Also
>>>> generalize the unreachable-trampoline comment because the single-ops path
>>>> does not use ops->direct_call.
>>>
>>> hi,
>>> iirc arm needs ops->direct_call for trampolines to work properly,
>>> that's the reason we enabled single ftrace_ops on x86 only:
>>>
>>>   424f6a361096 bpf,x86: Use single ftrace_ops for direct calls
>>>
>>> I don't understand the arm usage of op->direct_call, but it looks like
>>> (codex thinks) that the fast path won't always work without op->direct_call
>>> being set
>>
>>
>> In theory, op->direct_call is the fallback for the case:
>> HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS=y, out-of-range target, and non-BPF
>> direct ops. This is what Codex thinks about.
>>
>> However, after searching the call sites of
>> register_ftrace_direct()/__modify_ftrace_direct() in kernel, which would
>> set ops->direct_call, the actual callers are in bpf/trampoline.c, which
>> have been gated with HAVE_SINGLE_FTRACE_DIRECT_OPS. That said, for
>> normal usage of HAVE_SINGLE_FTRACE_DIRECT_OPS=y, op->direct_call is unused.
>>
>> If someone uses register_ftrace_direct() or
>> modify_ftrace_direct/_nolock() outside BPF in the future,
>> op->direct_call will be used.
> 
> so any 2nd user on the same function triggers the slow path,
> and that's fine, the single_ops handles slow path correctly
> 
> but IIUC the out-of-range target will now always go through slow path,
> which in current code it doesn't, right?
> 

Right.

Pls see the sibling thread with Ihor. With in-range BPF trampoline
allocation, the slow-path regression should mostly disappear.

Let's continue the discussion in that thread.

Thanks,
Leon