[PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Jose Fernandez (Anthropic)]

On arm64, HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is currently selected
only when DYNAMIC_FTRACE_WITH_CALL_OPS is available. CALL_OPS, in
turn, is mutually exclusive with kCFI: the pre-function NOPs it needs
would change the offset of the pre-function type hash (see
baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")),
and the compiler support needed to reconcile the two does not exist
yet.

The result is that a CONFIG_CFI=y arm64 kernel has no
ftrace direct calls at all, so register_fentry() fails with -ENOTSUPP
and no BPF trampoline can attach: fentry/fexit, fmod_ret and BPF LSM
programs are all unavailable. Deployments that want both kCFI
hardening and BPF-based security monitoring currently have to give
one of them up. systemd's bpf-restrict-fs feature hits this today:
https://lore.kernel.org/all/20250610232418.GA3544567@ax162/

CALL_OPS is an optimization for direct calls, not a dependency.
In-BL-range trampolines are reached by a direct branch without
consulting the ops pointer, and out-of-range trampolines already
fall back to ftrace_caller, where the DIRECT_CALLS machinery
(call_direct_funcs() storing the trampoline in ftrace_regs, the
ftrace_caller tail-call) is gated on DIRECT_CALLS alone. s390 and
loongarch ship HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS this way,
without having CALL_OPS at all.

Patch 1 prepares ftrace_modify_call() to build without CALL_OPS by
widening its #ifdef and using the existing ftrace_rec_update_ops()
wrapper (no functional change for current configurations). Patch 2
drops the CALL_OPS requirement from the DIRECT_CALLS select.

Configurations that keep CALL_OPS (clang !CFI, and GCC without
CC_OPTIMIZE_FOR_SIZE) are unchanged. We verified this: in an arm64
clang build, every object file is byte-identical before and after
the series except ftrace.o itself, and its disassembly is identical.
CFI builds (and GCC -Os builds) gain working direct calls, with
out-of-range attachments taking the ftrace_caller dispatch path
instead of the per-callsite fast path.

We tested on a 6.18.y-based kernel and on this base with clang
kCFI builds (CONFIG_CFI=y, enforcing) under qemu (TCG, and KVM on an
arm64 host) and on GB200-based arm64 hardware: fentry/fexit, fmod_ret
and BPF LSM programs load, attach and execute; the ftrace-direct
sample modules (including both modify samples, exercising
ftrace_modify_call()) run cleanly; no CFI violations observed. The
fentry_test, fexit_test, fentry_fexit, fexit_sleep, fexit_stress,
modify_return, tracing_struct, lsm and trampoline_count selftests and
the ftrace direct-call selftests (test.d/direct) pass on the new
configuration with results identical to a CALL_OPS kernel built from
the same tree, and a broader test_progs sweep showed no differences
attributable to this series. Without the series, all of the above
fail at attach time with -ENOTSUPP.

riscv has the same gap (its DIRECT_CALLS select also requires
CALL_OPS, and its CALL_OPS is likewise !CFI); if this approach is
acceptable for arm64 we can follow up there.

---
Jose Fernandez (Anthropic) (2):
      arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
      arm64: ftrace: allow DIRECT_CALLS without CALL_OPS

 arch/arm64/Kconfig         | 2 +-
 arch/arm64/kernel/ftrace.c | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)
---
base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731
change-id: 20260607-arm64-ftrace-direct-calls-152230ef7077

Best regards,
--  
Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>

[PATCH 1/2] arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS

[Jose Fernandez (Anthropic)]

ftrace_modify_call() is guarded by CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
and calls ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec)) directly,
which only exists when CALL_OPS is enabled.

Generic ftrace also needs ftrace_modify_call() when
CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled, to retarget a
callsite between two non-FTRACE_ADDR destinations, as happens when a
direct trampoline is modified. The next patch allows DIRECT_CALLS without
CALL_OPS, so widen the guard to cover both configurations and switch
the body to the ftrace_rec_update_ops() wrapper, which already has a
stub for the !CALL_OPS case. ftrace_make_call() already uses the same
wrapper today.

No functional change: with CALL_OPS enabled, ftrace_rec_update_ops()
expands to the exact call this replaces.

Assisted-by: Claude:unspecified
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
 arch/arm64/kernel/ftrace.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
index 5a1554a441628..e1a3c0b3a0514 100644
--- a/arch/arm64/kernel/ftrace.c
+++ b/arch/arm64/kernel/ftrace.c
@@ -409,7 +409,8 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
 	return ftrace_modify_code(pc, old, new, true);
 }
 
-#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
+#if defined(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS) || \
+	defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS)
 int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,
 		       unsigned long addr)
 {
@@ -417,7 +418,7 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,
 	u32 old, new;
 	int ret;
 
-	ret = ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec));
+	ret = ftrace_rec_update_ops(rec);
 	if (ret)
 		return ret;
 

-- 
2.52.0

[PATCH 2/2] arm64: ftrace: allow DIRECT_CALLS without CALL_OPS

[Jose Fernandez (Anthropic)]

arm64 gained ftrace direct calls in commit 2aa6ac03516d ("arm64:
ftrace: Add direct call support") on top of
DYNAMIC_FTRACE_WITH_CALL_OPS, using the per-callsite ops pointer as a
fast path to reach the direct trampoline. Since commit baaf553d3bc3
("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS"), CALL_OPS is
mutually exclusive with CFI: the pre-function NOPs would change the
offset of the pre-function kCFI type hash, and the compiler support
needed to keep that offset consistent does not exist yet.

The result is that a CONFIG_CFI=y kernel loses CALL_OPS, and with it
DIRECT_CALLS, and with it every BPF trampoline attachment to kernel
functions: register_fentry() returns -ENOTSUPP, so fentry/fexit,
fmod_ret and BPF LSM programs cannot attach at all. This is a real
problem for hardened arm64 deployments that rely on BPF LSM for
security monitoring while keeping kCFI enabled.

CALL_OPS is an optimization for direct calls, not a dependency. When
the direct trampoline is within BL range, the callsite branches
straight to it and ftrace_caller is not involved. When it is out of
range, ftrace_find_callable_addr() already falls back to
ftrace_caller, and the DIRECT_CALLS machinery there
(FREGS_DIRECT_TRAMP, ftrace_caller_direct_late) is gated on
DIRECT_CALLS alone, not CALL_OPS: the ops dispatch invokes
call_direct_funcs(), which stores the trampoline address in
ftrace_regs, and ftrace_caller tail-calls it. s390 and loongarch use
this same mechanism for HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS without
having CALL_OPS at all, and DYNAMIC_FTRACE_WITH_ARGS without CALL_OPS
is already a supported arm64 configuration (GCC builds with
CC_OPTIMIZE_FOR_SIZE do not satisfy the CALL_OPS select condition).

Drop the CALL_OPS requirement from the
HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select. Configurations that
keep CALL_OPS (!CFI clang builds, and GCC builds without
CC_OPTIMIZE_FOR_SIZE) are unchanged. CALL_OPS-less configurations
take the ftrace_caller ops-dispatch path for out-of-range direct
calls, trading the per-callsite fast path for working BPF
trampolines; in-range attachments still branch directly with no
overhead. GCC -Os builds also gain DIRECT_CALLS as a side effect.
That is intended: s390 and loongarch already ship DIRECT_CALLS
without any per-callsite fast path.

Assisted-by: Claude:unspecified
Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
---
 arch/arm64/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943b..2cd7d536671c9 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -188,7 +188,7 @@ config ARM64
 		if (GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS || \
 		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
 	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
-		if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
+		if DYNAMIC_FTRACE_WITH_ARGS
 	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
 		if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \
 		    (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))

-- 
2.52.0

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Puranjay Mohan]

On Tue, Jun 9, 2026 at 6:19 AM Jose Fernandez (Anthropic)
<jose.fernandez@linux.dev> wrote:
>
> On arm64, HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is currently selected
> only when DYNAMIC_FTRACE_WITH_CALL_OPS is available. CALL_OPS, in
> turn, is mutually exclusive with kCFI: the pre-function NOPs it needs
> would change the offset of the pre-function type hash (see
> baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")),
> and the compiler support needed to reconcile the two does not exist
> yet.
>
> The result is that a CONFIG_CFI=y arm64 kernel has no
> ftrace direct calls at all, so register_fentry() fails with -ENOTSUPP
> and no BPF trampoline can attach: fentry/fexit, fmod_ret and BPF LSM
> programs are all unavailable. Deployments that want both kCFI
> hardening and BPF-based security monitoring currently have to give
> one of them up. systemd's bpf-restrict-fs feature hits this today:
> https://lore.kernel.org/all/20250610232418.GA3544567@ax162/
>
> CALL_OPS is an optimization for direct calls, not a dependency.
> In-BL-range trampolines are reached by a direct branch without
> consulting the ops pointer, and out-of-range trampolines already
> fall back to ftrace_caller, where the DIRECT_CALLS machinery
> (call_direct_funcs() storing the trampoline in ftrace_regs, the
> ftrace_caller tail-call) is gated on DIRECT_CALLS alone. s390 and
> loongarch ship HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS this way,
> without having CALL_OPS at all.
>
> Patch 1 prepares ftrace_modify_call() to build without CALL_OPS by
> widening its #ifdef and using the existing ftrace_rec_update_ops()
> wrapper (no functional change for current configurations). Patch 2
> drops the CALL_OPS requirement from the DIRECT_CALLS select.
>
> Configurations that keep CALL_OPS (clang !CFI, and GCC without
> CC_OPTIMIZE_FOR_SIZE) are unchanged. We verified this: in an arm64
> clang build, every object file is byte-identical before and after
> the series except ftrace.o itself, and its disassembly is identical.
> CFI builds (and GCC -Os builds) gain working direct calls, with
> out-of-range attachments taking the ftrace_caller dispatch path
> instead of the per-callsite fast path.
>
> We tested on a 6.18.y-based kernel and on this base with clang
> kCFI builds (CONFIG_CFI=y, enforcing) under qemu (TCG, and KVM on an
> arm64 host) and on GB200-based arm64 hardware: fentry/fexit, fmod_ret
> and BPF LSM programs load, attach and execute; the ftrace-direct
> sample modules (including both modify samples, exercising
> ftrace_modify_call()) run cleanly; no CFI violations observed. The
> fentry_test, fexit_test, fentry_fexit, fexit_sleep, fexit_stress,
> modify_return, tracing_struct, lsm and trampoline_count selftests and
> the ftrace direct-call selftests (test.d/direct) pass on the new
> configuration with results identical to a CALL_OPS kernel built from
> the same tree, and a broader test_progs sweep showed no differences
> attributable to this series. Without the series, all of the above
> fail at attach time with -ENOTSUPP.
>
> riscv has the same gap (its DIRECT_CALLS select also requires
> CALL_OPS, and its CALL_OPS is likewise !CFI); if this approach is
> acceptable for arm64 we can follow up there.
>

It looks correct to me and should work.

Reviewed-by: Puranjay Mohan <puranjay@kernel.org>

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Clayton Craft]

On Mon Jun 8, 2026 at 10:19 PM PDT, Jose Fernandez (Anthropic) wrote:
> On arm64, HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is currently selected
> only when DYNAMIC_FTRACE_WITH_CALL_OPS is available. CALL_OPS, in
> turn, is mutually exclusive with kCFI: the pre-function NOPs it needs
> would change the offset of the pre-function type hash (see
> baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")),
> and the compiler support needed to reconcile the two does not exist
> yet.
>
> The result is that a CONFIG_CFI=y arm64 kernel has no
> ftrace direct calls at all, so register_fentry() fails with -ENOTSUPP
> and no BPF trampoline can attach: fentry/fexit, fmod_ret and BPF LSM
> programs are all unavailable. Deployments that want both kCFI
> hardening and BPF-based security monitoring currently have to give
> one of them up. systemd's bpf-restrict-fs feature hits this today:
> https://lore.kernel.org/all/20250610232418.GA3544567@ax162/
>
> CALL_OPS is an optimization for direct calls, not a dependency.
> In-BL-range trampolines are reached by a direct branch without
> consulting the ops pointer, and out-of-range trampolines already
> fall back to ftrace_caller, where the DIRECT_CALLS machinery
> (call_direct_funcs() storing the trampoline in ftrace_regs, the
> ftrace_caller tail-call) is gated on DIRECT_CALLS alone. s390 and
> loongarch ship HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS this way,
> without having CALL_OPS at all.
>
> Patch 1 prepares ftrace_modify_call() to build without CALL_OPS by
> widening its #ifdef and using the existing ftrace_rec_update_ops()
> wrapper (no functional change for current configurations). Patch 2
> drops the CALL_OPS requirement from the DIRECT_CALLS select.
>
> Configurations that keep CALL_OPS (clang !CFI, and GCC without
> CC_OPTIMIZE_FOR_SIZE) are unchanged. We verified this: in an arm64
> clang build, every object file is byte-identical before and after
> the series except ftrace.o itself, and its disassembly is identical.
> CFI builds (and GCC -Os builds) gain working direct calls, with
> out-of-range attachments taking the ftrace_caller dispatch path
> instead of the per-callsite fast path.
>
> We tested on a 6.18.y-based kernel and on this base with clang
> kCFI builds (CONFIG_CFI=y, enforcing) under qemu (TCG, and KVM on an
> arm64 host) and on GB200-based arm64 hardware: fentry/fexit, fmod_ret
> and BPF LSM programs load, attach and execute; the ftrace-direct
> sample modules (including both modify samples, exercising
> ftrace_modify_call()) run cleanly; no CFI violations observed. The
> fentry_test, fexit_test, fentry_fexit, fexit_sleep, fexit_stress,
> modify_return, tracing_struct, lsm and trampoline_count selftests and
> the ftrace direct-call selftests (test.d/direct) pass on the new
> configuration with results identical to a CALL_OPS kernel built from
> the same tree, and a broader test_progs sweep showed no differences
> attributable to this series. Without the series, all of the above
> fail at attach time with -ENOTSUPP.
>
> riscv has the same gap (its DIRECT_CALLS select also requires
> CALL_OPS, and its CALL_OPS is likewise !CFI); if this approach is
> acceptable for arm64 we can follow up there.
>
> ---
> Jose Fernandez (Anthropic) (2):
>       arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
>       arm64: ftrace: allow DIRECT_CALLS without CALL_OPS
>
>  arch/arm64/Kconfig         | 2 +-
>  arch/arm64/kernel/ftrace.c | 5 +++--
>  2 files changed, 4 insertions(+), 3 deletions(-)
> ---
> base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731
> change-id: 20260607-arm64-ftrace-direct-calls-152230ef7077
>
> Best regards,
> --
> Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>

Thanks for fixing this! I gave it a try on an aarch64 laptop (macbook m2)
using kernel 7.0.11 with CFI=y && DEBUG_INFO_BTF=y, built with clang. AFAIK it
seems to work great, I am able to finally run systemd with BPF support (e.g.,
unprivileged nspawn works now)

Tested-by: Clayton Craft <craftyguy@postmarketos.org>

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Nathan Chancellor]

Hi Jose,

On Tue, Jun 09, 2026 at 05:19:25AM +0000, Jose Fernandez (Anthropic) wrote:
> Jose Fernandez (Anthropic) (2):
>       arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
>       arm64: ftrace: allow DIRECT_CALLS without CALL_OPS

Thanks, I applied these two changes on -next and it looks like it
resolves the issue I originally noticed with systemd's restrict-fs
program not working on both of my arm64 machines.

Tested-by: Nathan Chancellor <nathan@kernel.org>

-- 
Cheers,
Nathan

Re: [PATCH 1/2] arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS

[Xu Kuohai]

On 6/9/2026 1:19 PM, Jose Fernandez (Anthropic) wrote:
> ftrace_modify_call() is guarded by CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
> and calls ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec)) directly,
> which only exists when CALL_OPS is enabled.
> 
> Generic ftrace also needs ftrace_modify_call() when
> CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is enabled, to retarget a
> callsite between two non-FTRACE_ADDR destinations, as happens when a
> direct trampoline is modified. The next patch allows DIRECT_CALLS without
> CALL_OPS, so widen the guard to cover both configurations and switch
> the body to the ftrace_rec_update_ops() wrapper, which already has a
> stub for the !CALL_OPS case. ftrace_make_call() already uses the same
> wrapper today.
> 
> No functional change: with CALL_OPS enabled, ftrace_rec_update_ops()
> expands to the exact call this replaces.
> 
> Assisted-by: Claude:unspecified
> Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
> ---
>   arch/arm64/kernel/ftrace.c | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c
> index 5a1554a441628..e1a3c0b3a0514 100644
> --- a/arch/arm64/kernel/ftrace.c
> +++ b/arch/arm64/kernel/ftrace.c
> @@ -409,7 +409,8 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
>   	return ftrace_modify_code(pc, old, new, true);
>   }
>   
> -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS
> +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_CALL_OPS) || \
> +	defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS)
>   int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,
>   		       unsigned long addr)
>   {
> @@ -417,7 +418,7 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr,
>   	u32 old, new;
>   	int ret;
>   
> -	ret = ftrace_rec_set_ops(rec, arm64_rec_get_ops(rec));
> +	ret = ftrace_rec_update_ops(rec);
>   	if (ret)
>   		return ret;
>   
>
Acked-by: Xu Kuohai <xukuohai@huawei.com>

Re: [PATCH 2/2] arm64: ftrace: allow DIRECT_CALLS without CALL_OPS

[Xu Kuohai]

On 6/9/2026 1:19 PM, Jose Fernandez (Anthropic) wrote:
> arm64 gained ftrace direct calls in commit 2aa6ac03516d ("arm64:
> ftrace: Add direct call support") on top of
> DYNAMIC_FTRACE_WITH_CALL_OPS, using the per-callsite ops pointer as a
> fast path to reach the direct trampoline. Since commit baaf553d3bc3
> ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS"), CALL_OPS is
> mutually exclusive with CFI: the pre-function NOPs would change the
> offset of the pre-function kCFI type hash, and the compiler support
> needed to keep that offset consistent does not exist yet.
> 
> The result is that a CONFIG_CFI=y kernel loses CALL_OPS, and with it
> DIRECT_CALLS, and with it every BPF trampoline attachment to kernel
> functions: register_fentry() returns -ENOTSUPP, so fentry/fexit,
> fmod_ret and BPF LSM programs cannot attach at all. This is a real
> problem for hardened arm64 deployments that rely on BPF LSM for
> security monitoring while keeping kCFI enabled.
> 
> CALL_OPS is an optimization for direct calls, not a dependency. When
> the direct trampoline is within BL range, the callsite branches
> straight to it and ftrace_caller is not involved. When it is out of
> range, ftrace_find_callable_addr() already falls back to
> ftrace_caller, and the DIRECT_CALLS machinery there
> (FREGS_DIRECT_TRAMP, ftrace_caller_direct_late) is gated on
> DIRECT_CALLS alone, not CALL_OPS: the ops dispatch invokes
> call_direct_funcs(), which stores the trampoline address in
> ftrace_regs, and ftrace_caller tail-calls it. s390 and loongarch use
> this same mechanism for HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS without
> having CALL_OPS at all, and DYNAMIC_FTRACE_WITH_ARGS without CALL_OPS
> is already a supported arm64 configuration (GCC builds with
> CC_OPTIMIZE_FOR_SIZE do not satisfy the CALL_OPS select condition).
> 
> Drop the CALL_OPS requirement from the
> HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS select. Configurations that
> keep CALL_OPS (!CFI clang builds, and GCC builds without
> CC_OPTIMIZE_FOR_SIZE) are unchanged. CALL_OPS-less configurations
> take the ftrace_caller ops-dispatch path for out-of-range direct
> calls, trading the per-callsite fast path for working BPF
> trampolines; in-range attachments still branch directly with no
> overhead. GCC -Os builds also gain DIRECT_CALLS as a side effect.
> That is intended: s390 and loongarch already ship DIRECT_CALLS
> without any per-callsite fast path.
> 
> Assisted-by: Claude:unspecified
> Signed-off-by: Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
> ---
>   arch/arm64/Kconfig | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index fe60738e5943b..2cd7d536671c9 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -188,7 +188,7 @@ config ARM64
>   		if (GCC_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS || \
>   		    CLANG_SUPPORTS_DYNAMIC_FTRACE_WITH_ARGS)
>   	select HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS \
> -		if DYNAMIC_FTRACE_WITH_ARGS && DYNAMIC_FTRACE_WITH_CALL_OPS
> +		if DYNAMIC_FTRACE_WITH_ARGS
>   	select HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS \
>   		if (DYNAMIC_FTRACE_WITH_ARGS && !CFI && \
>   		    (CC_IS_CLANG || !CC_OPTIMIZE_FOR_SIZE))
> 
Acked-by: Xu Kuohai <xukuohai@huawei.com>

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Jose Fernandez (Anthropic)]

Hi all,

Just a gentle ping on this series. It has Reviewed-by from Puranjay
Mohan, Acked-by from Xu Kuohai on both patches, and Tested-by from
Clayton Craft and Nathan Chancellor, with no outstanding comments since
Jun 11.

Could you please consider picking this up for the next cycle?

Thanks,
Jose

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Steven Rostedt]

On Fri, 10 Jul 2026 14:07:01 +0000
"Jose Fernandez (Anthropic)" <jose.fernandez@linux.dev> wrote:

> Hi all,
> 
> Just a gentle ping on this series. It has Reviewed-by from Puranjay
> Mohan, Acked-by from Xu Kuohai on both patches, and Tested-by from
> Clayton Craft and Nathan Chancellor, with no outstanding comments since
> Jun 11.
> 
> Could you please consider picking this up for the next cycle?
> 

As this is arm64 specific, the decision lies with the arm64 maintainers.

-- Steve

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Mark Rutland]

On Fri, Jul 10, 2026 at 02:07:01PM +0000, Jose Fernandez (Anthropic) wrote:
> Hi all,
> 
> Just a gentle ping on this series. It has Reviewed-by from Puranjay
> Mohan, Acked-by from Xu Kuohai on both patches, and Tested-by from
> Clayton Craft and Nathan Chancellor, with no outstanding comments since
> Jun 11.
> 
> Could you please consider picking this up for the next cycle?

Sorry, this is on my queue to look at, but I have been busy elsewhere
over the last few weeks.

I'll get to it shortly.

Mark.

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Will Deacon]

On Tue, 09 Jun 2026 05:19:25 +0000, Jose Fernandez (Anthropic) wrote:
> On arm64, HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is currently selected
> only when DYNAMIC_FTRACE_WITH_CALL_OPS is available. CALL_OPS, in
> turn, is mutually exclusive with kCFI: the pre-function NOPs it needs
> would change the offset of the pre-function type hash (see
> baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")),
> and the compiler support needed to reconcile the two does not exist
> yet.
> 
> [...]

Applied to arm64 (for-next/misc), thanks!

[1/2] arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
      https://git.kernel.org/arm64/c/123b4fc0f857
[2/2] arm64: ftrace: allow DIRECT_CALLS without CALL_OPS
      https://git.kernel.org/arm64/c/9315e22b0c0a

Cheers,
-- 
Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Mark Rutland]

On Tue, Jun 09, 2026 at 05:19:25AM +0000, Jose Fernandez (Anthropic) wrote:
> On arm64, HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS is currently selected
> only when DYNAMIC_FTRACE_WITH_CALL_OPS is available. CALL_OPS, in
> turn, is mutually exclusive with kCFI: the pre-function NOPs it needs
> would change the offset of the pre-function type hash (see
> baaf553d3bc3 ("arm64: Implement HAVE_DYNAMIC_FTRACE_WITH_CALL_OPS")),
> and the compiler support needed to reconcile the two does not exist
> yet.
> 
> The result is that a CONFIG_CFI=y arm64 kernel has no
> ftrace direct calls at all, so register_fentry() fails with -ENOTSUPP
> and no BPF trampoline can attach: fentry/fexit, fmod_ret and BPF LSM
> programs are all unavailable. Deployments that want both kCFI
> hardening and BPF-based security monitoring currently have to give
> one of them up. systemd's bpf-restrict-fs feature hits this today:
> https://lore.kernel.org/all/20250610232418.GA3544567@ax162/
> 
> CALL_OPS is an optimization for direct calls, not a dependency.
> In-BL-range trampolines are reached by a direct branch without
> consulting the ops pointer, and out-of-range trampolines already
> fall back to ftrace_caller, where the DIRECT_CALLS machinery
> (call_direct_funcs() storing the trampoline in ftrace_regs, the
> ftrace_caller tail-call) is gated on DIRECT_CALLS alone. 

It's true that this is not a strict functional dependency, but there is
a substantial performance cliff here.

With CALL_OPS we only fall back to the ftrace_ops_list_func() when there
are multiple ops associated with a given callsite. Where we have a
single ops for a callsite, if the direct call is out of range, the
ftrace caller will call trace_ops::direct_call.

Without CALL_OPS, if there are unrelated ops registered (attached to
difrerent callsites), the ftrace_caller trampoline will call
ftrace_ops_list_func(), and we'll iterate over all of the registered ops
to find the relevant ops and trampoline, every time an out-of-range
direct call is used.

So the big questions are:

(1) How often is a direct call being made?
(2) Where does BPF allocate its direct call trampolines?

From a skim of arch/arm64/mm/init.c, EXECMEM_BPF covers
VMALLOC_START..VMALLOC_END, the vast majority of which is out-of-range
for a BL from the core kernel or a module. So if EXECMEM_BPF is used,
that's likely to be problematic.  Hopefully I'm missing something such
that BPF trampolines are almost always allocated in-range of a BL?

When we originally implemented CALL_OPS and DIRECT_CALLS for arm64, the
understanding was that we'd have many ops with direct calls. We believed
that the ftrace_ops_list_func() penalty was so substantial that it
wasn't worthwhile supporting direct calls in a configuration where
bouncing through that would be common.

If we need to support this configuration, then I strongly suspect we
need to make other changes too in order for it to be worthwhile.

> s390 and loongarch ship HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS this
> way, without having CALL_OPS at all.

Following from the comment regarding EXECMEM_BPF above, it looks like
both s390 and loongarch don't define EXECMEM_BPF, and only define
EXECMEM_DEFAULT with a subsstantially limited range, which I suspect
keeps their direct calls within range.

> Patch 1 prepares ftrace_modify_call() to build without CALL_OPS by
> widening its #ifdef and using the existing ftrace_rec_update_ops()
> wrapper (no functional change for current configurations). Patch 2
> drops the CALL_OPS requirement from the DIRECT_CALLS select.
> 
> Configurations that keep CALL_OPS (clang !CFI, and GCC without
> CC_OPTIMIZE_FOR_SIZE) are unchanged. We verified this: in an arm64
> clang build, every object file is byte-identical before and after
> the series except ftrace.o itself, and its disassembly is identical.
> CFI builds (and GCC -Os builds) gain working direct calls, with
> out-of-range attachments taking the ftrace_caller dispatch path
> instead of the per-callsite fast path.

Can you comment on how you've tested in-range vs out-of-range
attachements here?

> We tested on a 6.18.y-based kernel and on this base with clang
> kCFI builds (CONFIG_CFI=y, enforcing) under qemu (TCG, and KVM on an
> arm64 host) and on GB200-based arm64 hardware: fentry/fexit, fmod_ret
> and BPF LSM programs load, attach and execute; the ftrace-direct
> sample modules (including both modify samples, exercising
> ftrace_modify_call()) run cleanly; no CFI violations observed. The
> fentry_test, fexit_test, fentry_fexit, fexit_sleep, fexit_stress,
> modify_return, tracing_struct, lsm and trampoline_count selftests and
> the ftrace direct-call selftests (test.d/direct) pass on the new
> configuration with results identical to a CALL_OPS kernel built from
> the same tree, and a broader test_progs sweep showed no differences
> attributable to this series. Without the series, all of the above
> fail at attach time with -ENOTSUPP.

Have you made any performance measurements, or was this just functional
enablement?

Mark.

> riscv has the same gap (its DIRECT_CALLS select also requires
> CALL_OPS, and its CALL_OPS is likewise !CFI); if this approach is
> acceptable for arm64 we can follow up there.
> 
> ---
> Jose Fernandez (Anthropic) (2):
>       arm64: ftrace: prepare ftrace_modify_call() for use without CALL_OPS
>       arm64: ftrace: allow DIRECT_CALLS without CALL_OPS
> 
>  arch/arm64/Kconfig         | 2 +-
>  arch/arm64/kernel/ftrace.c | 5 +++--
>  2 files changed, 4 insertions(+), 3 deletions(-)
> ---
> base-commit: 254f49634ee16a731174d2ae34bc50bd5f45e731
> change-id: 20260607-arm64-ftrace-direct-calls-152230ef7077
> 
> Best regards,
> --  
> Jose Fernandez (Anthropic) <jose.fernandez@linux.dev>
> 

Re: [PATCH 0/2] arm64: ftrace: support DIRECT_CALLS without CALL_OPS

[Jose Fernandez (Anthropic)]

Hi Mark,

Thanks for the review, answers inline.

On Tue, Jul 28, 2026 at 06:31:43PM +0100, Mark Rutland wrote:
> So the big questions are:
>
> (1) How often is a direct call being made?

The workloads that want this are always-on. In our case a BPF LSM
security monitor covers file, exec and socket hooks, so it sits on
syscall-rate paths on busy machines, and fentry-based observability
behaves the same way. That is why we measured per-call dispatch cost
below.

> (2) Where does BPF allocate its direct call trampolines?
>
> From a skim of arch/arm64/mm/init.c, EXECMEM_BPF covers
> VMALLOC_START..VMALLOC_END, the vast majority of which is out-of-range
> for a BL from the core kernel or a module. So if EXECMEM_BPF is used,
> that's likely to be problematic.  Hopefully I'm missing something such
> that BPF trampolines are almost always allocated in-range of a BL?

You're not missing anything. BPF trampolines come from bpf_prog_pack,
which allocates via EXECMEM_BPF, and we confirmed the consequence at
runtime. Across every boot and attach in the measurements below, the
trampoline landed in the same execmem slot while the kernel text moved
with KASLR, 50-90 TB out of BL range. Only a handful of boots, but
EXECMEM_BPF spans all of vmalloc and has no reason to land near
kernel text, so I would expect out-of-range to be the common case
here rather than the exception.

> When we originally implemented CALL_OPS and DIRECT_CALLS for arm64, the
> understanding was that we'd have many ops with direct calls. We believed
> that the ftrace_ops_list_func() penalty was so substantial that it
> wasn't worthwhile supporting direct calls in a configuration where
> bouncing through that would be common.
>
> If we need to support this configuration, then I strongly suspect we
> need to make other changes too in order for it to be worthwhile.

Agreed. One idea I would like your opinion on is an in-range
allocation preference for BPF trampolines, since execmem seems to
have the shape for it on arm64 already. EXECMEM_DEFAULT prefers the
module_direct_base window ("prefer to allocate within direct branch
range of the kernel") and falls back when that neighborhood is full,
while EXECMEM_BPF spans all of vmalloc with no preference. Giving
trampolines the same preference would need no new dispatch machinery,
and the measurements below suggest it recovers 93% of the gap to the
CALL_OPS baseline. Does that direction sound right to you, or is
there something in the CALL_OPS design that argues against it? If it
holds up I am happy to write it as a follow-up to this series.

> Following from the comment regarding EXECMEM_BPF above, it looks like
> both s390 and loongarch don't define EXECMEM_BPF, and only define
> EXECMEM_DEFAULT with a subsstantially limited range, which I suspect
> keeps their direct calls within range.

That matches our reading. Their limited EXECMEM_DEFAULT ranges keep
direct calls in range by construction, same idea as the above.

> Can you comment on how you've tested in-range vs out-of-range
> attachements here?

Reachability is decided per attach at patch time.
ftrace_find_callable_addr() picks the direct BL when the target is in
range and ftrace_caller when it is not. In our runs we read the live
trampoline and patch-site addresses from /proc/kallsyms and computed
the BL-range verdict for every attach, and every attach came out
out-of-range. So all of the original series validation was
out-of-range too, per the allocation behavior above.

To get numbers for the in-range side we forced it on a test rig
instead of writing the real change. A small bench-only patch (six
lines) makes arch_alloc_bpf_trampoline() allocate from
execmem_alloc(EXECMEM_MODULE_TEXT, ...) instead of bpf_prog_pack, and
the guest boots with nokaslr so the module window is always within
reach. From there the existing reach check does the rest and patches
the direct BL by itself. So the rig only tells us what in-range is
worth, the allocator preference above is what I would actually
implement.

> Have you made any performance measurements, or was this just functional
> enablement?

The original posting was functional enablement, so after your review
we measured the per-call cost. The rig is a KVM guest with 8 vcpus on
an arm64 Graviton metal host, 6.18.36 with this series, fentry on
__arm64_sys_getpid, 20M calls per run, per-call overhead taken as
traced minus untraced. Six passes across three fresh boots per
configuration, ranges below are min to max over those six passes.

  CALL_OPS baseline (CFI off):              17.8-18.3 ns
  series path, out-of-range, no other ops:  34.5-35.6 ns  (~2.0x)
  same, plus N unrelated ftrace_ops
  (function tracer on vfs_read):
    N=1                                     41 ns
    N=4                                     53 ns
    N=8                                     68 ns
    N=16                                    98 ns
  in-range trampoline (rig above):          18.7-19.4 ns  (~1.2 ns more)

The extra-ops rows are the ftrace_ops_list_func() cost you were
worried about, a straight 3.9 ns per registered op, and paid
everywhere since the traced callsite never matches their filter. The
in-range trampoline closes 93% of the gap to baseline, though the
passes still sit a bit above, so not quite identical.

Thanks,
Jose