* [PATCH v8] arm64: Use static call trampolines when kCFI is enabled
@ 2026-03-31 11:04 Ard Biesheuvel
2026-03-31 12:06 ` Ard Biesheuvel
2026-04-01 10:03 ` Will Deacon
0 siblings, 2 replies; 5+ messages in thread
From: Ard Biesheuvel @ 2026-03-31 11:04 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-hardening, will, mark.rutland, Ard Biesheuvel,
Carlos Llamas, Sami Tolvanen, Sean Christopherson, Kees Cook,
Peter Zijlstra, Will McVicker
From: Ard Biesheuvel <ardb@kernel.org>
Implement arm64 support for the 'unoptimized' static call variety, which
routes all calls through a trampoline that performs a tail call to the
chosen function, and wire it up for use when kCFI is enabled. This works
around an issue with kCFI and generic static calls, where the prototypes
of default handlers such as __static_call_nop() and __static_call_ret0()
don't match the expected prototype of the call site, resulting in kCFI
false positives [0].
Since static call targets may be located in modules loaded out of direct
branching range, this needs a ADRP/ADD pair to load the branch target
into R16 and a branch-to-register (BR) instruction to perform an
indirect call. This is the exact code sequence that is used by modules
when the call target is out of direct branching range.
Unlike on x86, there is no pressing need on arm64 to avoid indirect
calls at all cost, but hiding it from the compiler as is done here does
have some benefits:
- the literal is located in .rodata, which gives us the same robustness
advantage that code patching does;
- no D-cache pollution from fetching hash values from .text sections.
From an execution speed PoV, this is unlikely to make any difference at
all.
[0] https://lore.kernel.org/all/20260311225822.1565895-1-cmllamas@google.com/
Cc: Carlos Llamas <cmllamas@google.com>
Cc: Sami Tolvanen <samitolvanen@google.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will McVicker <willmcvicker@google.com>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
v8: Simplify the trampoline by combining the NULL and RET0 cases, and
dropping the conditional branch and return
v7: https://lore.kernel.org/all/20260313061852.4025964-1-cmllamas@google.com/
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/static_call.h | 31 ++++++++++++++++++++
arch/arm64/kernel/Makefile | 1 +
arch/arm64/kernel/static_call.c | 23 +++++++++++++++
arch/arm64/kernel/vmlinux.lds.S | 1 +
5 files changed, 57 insertions(+)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 38dba5f7e4d2..9ea19b74b6c3 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -252,6 +252,7 @@ config ARM64
select HAVE_RSEQ
select HAVE_RUST if RUSTC_SUPPORTS_ARM64
select HAVE_STACKPROTECTOR
+ select HAVE_STATIC_CALL if CFI
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_KPROBES
select HAVE_KRETPROBES
diff --git a/arch/arm64/include/asm/static_call.h b/arch/arm64/include/asm/static_call.h
new file mode 100644
index 000000000000..b73960c949e4
--- /dev/null
+++ b/arch/arm64/include/asm/static_call.h
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_STATIC_CALL_H
+#define _ASM_STATIC_CALL_H
+
+#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, target) \
+ asm(" .pushsection .static_call.text, \"ax\" \n" \
+ " .align 4 \n" \
+ " .globl " name " \n" \
+ name ": \n" \
+ " hint 34 /* BTI C */ \n" \
+ " adrp x16, 1f \n" \
+ " ldr x16, [x16, :lo12:1f] \n" \
+ " br x16 \n" \
+ " .type " name ", %function \n" \
+ " .size " name ", . - " name " \n" \
+ " .popsection \n" \
+ " .pushsection .rodata, \"a\" \n" \
+ " .align 3 \n" \
+ "1: .quad " #target " \n" \
+ " .popsection \n")
+
+#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \
+ __ARCH_DEFINE_STATIC_CALL_TRAMP(STATIC_CALL_TRAMP_STR(name), #func)
+
+#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \
+ ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0)
+
+#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) \
+ ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0)
+
+#endif /* _ASM_STATIC_CALL_H */
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 76f32e424065..fe627100d199 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -46,6 +46,7 @@ obj-$(CONFIG_MODULES) += module.o module-plts.o
obj-$(CONFIG_PERF_EVENTS) += perf_regs.o perf_callchain.o
obj-$(CONFIG_HARDLOCKUP_DETECTOR_PERF) += watchdog_hld.o
obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
+obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o
obj-$(CONFIG_CPU_PM) += sleep.o suspend.o
obj-$(CONFIG_KGDB) += kgdb.o
obj-$(CONFIG_EFI) += efi.o efi-rt-wrapper.o
diff --git a/arch/arm64/kernel/static_call.c b/arch/arm64/kernel/static_call.c
new file mode 100644
index 000000000000..8b3a19e10871
--- /dev/null
+++ b/arch/arm64/kernel/static_call.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/static_call.h>
+#include <linux/memory.h>
+#include <asm/text-patching.h>
+
+void arch_static_call_transform(void *site, void *tramp, void *func, bool tail)
+{
+ u64 literal;
+ int ret;
+
+ if (!func)
+ func = __static_call_return0;
+
+ /* decode the instructions to discover the literal address */
+ literal = ALIGN_DOWN((u64)tramp + 4, SZ_4K) +
+ aarch64_insn_adrp_get_offset(le32_to_cpup(tramp + 4)) +
+ 8 * aarch64_insn_decode_immediate(AARCH64_INSN_IMM_12,
+ le32_to_cpup(tramp + 8));
+
+ ret = aarch64_insn_write_literal_u64((void *)literal, (u64)func);
+ WARN_ON_ONCE(ret);
+}
+EXPORT_SYMBOL_GPL(arch_static_call_transform);
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 2964aad0362e..2d1e75263f03 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -191,6 +191,7 @@ SECTIONS
LOCK_TEXT
KPROBES_TEXT
HYPERVISOR_TEXT
+ STATIC_CALL_TEXT
*(.gnu.warning)
}
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v8] arm64: Use static call trampolines when kCFI is enabled
2026-03-31 11:04 [PATCH v8] arm64: Use static call trampolines when kCFI is enabled Ard Biesheuvel
@ 2026-03-31 12:06 ` Ard Biesheuvel
2026-04-01 10:03 ` Will Deacon
1 sibling, 0 replies; 5+ messages in thread
From: Ard Biesheuvel @ 2026-03-31 12:06 UTC (permalink / raw)
To: Ard Biesheuvel, linux-arm-kernel
Cc: linux-hardening, Will Deacon, Mark Rutland, Carlos Llamas,
Sami Tolvanen, Sean Christopherson, Kees Cook, Peter Zijlstra,
Will McVicker
On Tue, 31 Mar 2026, at 13:04, Ard Biesheuvel wrote:
> From: Ard Biesheuvel <ardb@kernel.org>
>
> Implement arm64 support for the 'unoptimized' static call variety, which
> routes all calls through a trampoline that performs a tail call to the
> chosen function, and wire it up for use when kCFI is enabled. This works
> around an issue with kCFI and generic static calls, where the prototypes
> of default handlers such as __static_call_nop() and __static_call_ret0()
> don't match the expected prototype of the call site, resulting in kCFI
> false positives [0].
>
> Since static call targets may be located in modules loaded out of direct
> branching range, this needs a ADRP/ADD pair to load the branch target
Sashiko correctly points out that this should say ADRP/LDR rather than
ADRP/ADD, and this means that the sequence is in fact different from the
one used by modules.
> into R16 and a branch-to-register (BR) instruction to perform an
> indirect call. This is the exact code sequence that is used by modules
> when the call target is out of direct branching range.
>
... so please drop this last sentence when applying.
> Unlike on x86, there is no pressing need on arm64 to avoid indirect
> calls at all cost, but hiding it from the compiler as is done here does
> have some benefits:
> - the literal is located in .rodata, which gives us the same robustness
> advantage that code patching does;
> - no D-cache pollution from fetching hash values from .text sections.
>
> From an execution speed PoV, this is unlikely to make any difference at
> all.
>
> [0] https://lore.kernel.org/all/20260311225822.1565895-1-cmllamas@google.com/
>
> Cc: Carlos Llamas <cmllamas@google.com>
> Cc: Sami Tolvanen <samitolvanen@google.com>
> Cc: Sean Christopherson <seanjc@google.com>
> Cc: Kees Cook <kees@kernel.org>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Will McVicker <willmcvicker@google.com>
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
> v8: Simplify the trampoline by combining the NULL and RET0 cases, and
> dropping the conditional branch and return
> v7: https://lore.kernel.org/all/20260313061852.4025964-1-cmllamas@google.com/
>
> arch/arm64/Kconfig | 1 +
> arch/arm64/include/asm/static_call.h | 31 ++++++++++++++++++++
> arch/arm64/kernel/Makefile | 1 +
> arch/arm64/kernel/static_call.c | 23 +++++++++++++++
> arch/arm64/kernel/vmlinux.lds.S | 1 +
> 5 files changed, 57 insertions(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 38dba5f7e4d2..9ea19b74b6c3 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -252,6 +252,7 @@ config ARM64
> select HAVE_RSEQ
> select HAVE_RUST if RUSTC_SUPPORTS_ARM64
> select HAVE_STACKPROTECTOR
> + select HAVE_STATIC_CALL if CFI
> select HAVE_SYSCALL_TRACEPOINTS
> select HAVE_KPROBES
> select HAVE_KRETPROBES
> diff --git a/arch/arm64/include/asm/static_call.h
> b/arch/arm64/include/asm/static_call.h
> new file mode 100644
> index 000000000000..b73960c949e4
> --- /dev/null
> +++ b/arch/arm64/include/asm/static_call.h
> @@ -0,0 +1,31 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _ASM_STATIC_CALL_H
> +#define _ASM_STATIC_CALL_H
> +
> +#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, target) \
> + asm(" .pushsection .static_call.text, \"ax\" \n" \
> + " .align 4 \n" \
> + " .globl " name " \n" \
> + name ": \n" \
> + " hint 34 /* BTI C */ \n" \
> + " adrp x16, 1f \n" \
> + " ldr x16, [x16, :lo12:1f] \n" \
> + " br x16 \n" \
> + " .type " name ", %function \n" \
> + " .size " name ", . - " name " \n" \
> + " .popsection \n" \
> + " .pushsection .rodata, \"a\" \n" \
> + " .align 3 \n" \
> + "1: .quad " #target " \n" \
> + " .popsection \n")
> +
> +#define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) \
> + __ARCH_DEFINE_STATIC_CALL_TRAMP(STATIC_CALL_TRAMP_STR(name), #func)
> +
> +#define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) \
> + ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0)
> +
> +#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) \
> + ARCH_DEFINE_STATIC_CALL_TRAMP(name, __static_call_return0)
> +
> +#endif /* _ASM_STATIC_CALL_H */
> diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
> index 76f32e424065..fe627100d199 100644
> --- a/arch/arm64/kernel/Makefile
> +++ b/arch/arm64/kernel/Makefile
> @@ -46,6 +46,7 @@ obj-$(CONFIG_MODULES) += module.o module-plts.o
> obj-$(CONFIG_PERF_EVENTS) += perf_regs.o perf_callchain.o
> obj-$(CONFIG_HARDLOCKUP_DETECTOR_PERF) += watchdog_hld.o
> obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
> +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o
> obj-$(CONFIG_CPU_PM) += sleep.o suspend.o
> obj-$(CONFIG_KGDB) += kgdb.o
> obj-$(CONFIG_EFI) += efi.o efi-rt-wrapper.o
> diff --git a/arch/arm64/kernel/static_call.c
> b/arch/arm64/kernel/static_call.c
> new file mode 100644
> index 000000000000..8b3a19e10871
> --- /dev/null
> +++ b/arch/arm64/kernel/static_call.c
> @@ -0,0 +1,23 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include <linux/static_call.h>
> +#include <linux/memory.h>
> +#include <asm/text-patching.h>
> +
> +void arch_static_call_transform(void *site, void *tramp, void *func,
> bool tail)
> +{
> + u64 literal;
> + int ret;
> +
> + if (!func)
> + func = __static_call_return0;
> +
> + /* decode the instructions to discover the literal address */
> + literal = ALIGN_DOWN((u64)tramp + 4, SZ_4K) +
> + aarch64_insn_adrp_get_offset(le32_to_cpup(tramp + 4)) +
> + 8 * aarch64_insn_decode_immediate(AARCH64_INSN_IMM_12,
> + le32_to_cpup(tramp + 8));
> +
> + ret = aarch64_insn_write_literal_u64((void *)literal, (u64)func);
> + WARN_ON_ONCE(ret);
> +}
> +EXPORT_SYMBOL_GPL(arch_static_call_transform);
> diff --git a/arch/arm64/kernel/vmlinux.lds.S
> b/arch/arm64/kernel/vmlinux.lds.S
> index 2964aad0362e..2d1e75263f03 100644
> --- a/arch/arm64/kernel/vmlinux.lds.S
> +++ b/arch/arm64/kernel/vmlinux.lds.S
> @@ -191,6 +191,7 @@ SECTIONS
> LOCK_TEXT
> KPROBES_TEXT
> HYPERVISOR_TEXT
> + STATIC_CALL_TEXT
> *(.gnu.warning)
> }
>
> --
> 2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v8] arm64: Use static call trampolines when kCFI is enabled
2026-03-31 11:04 [PATCH v8] arm64: Use static call trampolines when kCFI is enabled Ard Biesheuvel
2026-03-31 12:06 ` Ard Biesheuvel
@ 2026-04-01 10:03 ` Will Deacon
2026-04-01 12:03 ` Ard Biesheuvel
1 sibling, 1 reply; 5+ messages in thread
From: Will Deacon @ 2026-04-01 10:03 UTC (permalink / raw)
To: linux-arm-kernel, Ard Biesheuvel
Cc: catalin.marinas, kernel-team, Will Deacon, linux-hardening,
mark.rutland, Ard Biesheuvel, Carlos Llamas, Sami Tolvanen,
Sean Christopherson, Kees Cook, Peter Zijlstra, Will McVicker
On Tue, 31 Mar 2026 13:04:23 +0200, Ard Biesheuvel wrote:
> Implement arm64 support for the 'unoptimized' static call variety, which
> routes all calls through a trampoline that performs a tail call to the
> chosen function, and wire it up for use when kCFI is enabled. This works
> around an issue with kCFI and generic static calls, where the prototypes
> of default handlers such as __static_call_nop() and __static_call_ret0()
> don't match the expected prototype of the call site, resulting in kCFI
> false positives [0].
>
> [...]
Applied to arm64 (for-next/fixes), thanks!
[1/1] arm64: Use static call trampolines when kCFI is enabled
https://git.kernel.org/arm64/c/e70c2335f889
Cheers,
--
Will
https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v8] arm64: Use static call trampolines when kCFI is enabled
2026-04-01 10:03 ` Will Deacon
@ 2026-04-01 12:03 ` Ard Biesheuvel
2026-04-01 14:31 ` Catalin Marinas
0 siblings, 1 reply; 5+ messages in thread
From: Ard Biesheuvel @ 2026-04-01 12:03 UTC (permalink / raw)
To: Will Deacon, linux-arm-kernel, Ard Biesheuvel, Catalin Marinas
Cc: kernel-team, linux-hardening, Mark Rutland, Carlos Llamas,
Sami Tolvanen, Sean Christopherson, Kees Cook, Peter Zijlstra,
Will McVicker
On Wed, 1 Apr 2026, at 12:03, Will Deacon wrote:
> On Tue, 31 Mar 2026 13:04:23 +0200, Ard Biesheuvel wrote:
>> Implement arm64 support for the 'unoptimized' static call variety, which
>> routes all calls through a trampoline that performs a tail call to the
>> chosen function, and wire it up for use when kCFI is enabled. This works
>> around an issue with kCFI and generic static calls, where the prototypes
>> of default handlers such as __static_call_nop() and __static_call_ret0()
>> don't match the expected prototype of the call site, resulting in kCFI
>> false positives [0].
>>
>> [...]
>
> Applied to arm64 (for-next/fixes), thanks!
>
> [1/1] arm64: Use static call trampolines when kCFI is enabled
> https://git.kernel.org/arm64/c/e70c2335f889
>
Thanks,
I just spotted that the function name gets stringified twice inadvertently.
E.g., the assembler may see
.quad "__static_call_return0"
rather than the intended
.quad __static_call_return0
The assembler does not seem to care, and still emits an ABS64 relocation against the correct symbol, but it is definitely unintentional.
I can send a follow-up fix if you prefer, or you could just tweak the patch in place:
--- a/arch/arm64/include/asm/static_call.h
+++ b/arch/arm64/include/asm/static_call.h
@@ -16,7 +16,7 @@
" .popsection \n" \
" .pushsection .rodata, \"a\" \n" \
" .align 3 \n" \
- "1: .quad " #target " \n" \
+ "1: .quad " target " \n" \
" .popsection \n")
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH v8] arm64: Use static call trampolines when kCFI is enabled
2026-04-01 12:03 ` Ard Biesheuvel
@ 2026-04-01 14:31 ` Catalin Marinas
0 siblings, 0 replies; 5+ messages in thread
From: Catalin Marinas @ 2026-04-01 14:31 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Will Deacon, linux-arm-kernel, Ard Biesheuvel, kernel-team,
linux-hardening, Mark Rutland, Carlos Llamas, Sami Tolvanen,
Sean Christopherson, Kees Cook, Peter Zijlstra, Will McVicker
On Wed, Apr 01, 2026 at 02:03:22PM +0200, Ard Biesheuvel wrote:
>
> On Wed, 1 Apr 2026, at 12:03, Will Deacon wrote:
> > On Tue, 31 Mar 2026 13:04:23 +0200, Ard Biesheuvel wrote:
> >> Implement arm64 support for the 'unoptimized' static call variety, which
> >> routes all calls through a trampoline that performs a tail call to the
> >> chosen function, and wire it up for use when kCFI is enabled. This works
> >> around an issue with kCFI and generic static calls, where the prototypes
> >> of default handlers such as __static_call_nop() and __static_call_ret0()
> >> don't match the expected prototype of the call site, resulting in kCFI
> >> false positives [0].
> >>
> >> [...]
> >
> > Applied to arm64 (for-next/fixes), thanks!
> >
> > [1/1] arm64: Use static call trampolines when kCFI is enabled
> > https://git.kernel.org/arm64/c/e70c2335f889
> >
>
> Thanks,
>
> I just spotted that the function name gets stringified twice inadvertently.
>
> E.g., the assembler may see
>
> .quad "__static_call_return0"
>
> rather than the intended
>
> .quad __static_call_return0
>
> The assembler does not seem to care, and still emits an ABS64 relocation against the correct symbol, but it is definitely unintentional.
>
> I can send a follow-up fix if you prefer, or you could just tweak the patch in place:
>
> --- a/arch/arm64/include/asm/static_call.h
> +++ b/arch/arm64/include/asm/static_call.h
> @@ -16,7 +16,7 @@
> " .popsection \n" \
> " .pushsection .rodata, \"a\" \n" \
> " .align 3 \n" \
> - "1: .quad " #target " \n" \
> + "1: .quad " target " \n" \
> " .popsection \n")
Not sure Will is going to look at this until next week. I folded it into
the arm64 for-next/fixes branch and pushed out.
--
Catalin
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-04-01 14:32 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-31 11:04 [PATCH v8] arm64: Use static call trampolines when kCFI is enabled Ard Biesheuvel
2026-03-31 12:06 ` Ard Biesheuvel
2026-04-01 10:03 ` Will Deacon
2026-04-01 12:03 ` Ard Biesheuvel
2026-04-01 14:31 ` Catalin Marinas
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox