* Re: [PATCH v6 8/9] sframe: Initialize debug info for kernel sections
From: Jens Remus @ 2026-05-19 11:20 UTC (permalink / raw)
To: Dylan Hatch
Cc: Prasanna Kumar T S M, Puranjay Mohan, Song Liu, joe.lawrence,
linux-toolchains, linux-kernel, live-patching, linux-arm-kernel,
Randy Dunlap, Mostafa Saleh, Herbert Xu, David S. Miller,
Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland
In-Reply-To: <20260519064950.493949-9-dylanbhatch@google.com>
On 5/19/2026 8:49 AM, Dylan Hatch wrote:
> Setup the optional unwinder debug information for kernel .sframe
> sections. Modules are indicated by the format "(<module-name>)".
>
> Suggested-by: Jens Remus <jremus@linux.ibm.com>
> Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
> ---
> kernel/unwind/sframe.c | 4 ++++
> kernel/unwind/sframe_debug.h | 13 +++++++++++++
> 2 files changed, 17 insertions(+)
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
jremus@de.ibm.com / jremus@linux.ibm.com
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Ehningen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [PATCH v5 0/8] unwind, arm64: add sframe unwinder for kernel
From: Jens Remus @ 2026-05-19 11:05 UTC (permalink / raw)
To: Dylan Hatch, Mostafa Saleh
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <CADBMgpx38SUUuYYCm612STqh01jqv817WnJeeXYTD7Uc1r-fug@mail.gmail.com>
Hi Dylan and Mostafa!
On 5/18/2026 7:55 PM, Dylan Hatch wrote:
> On Fri, May 15, 2026 at 4:32 AM Mostafa Saleh <smostafa@google.com> wrote:
>> On Tue, Apr 28, 2026 at 06:36:35PM +0000, Dylan Hatch wrote:
>>> Implement a generic kernel sframe-based [1] unwinder. The main goal is
>>> to improve reliable stacktrace on arm64 by unwinding across exception
>>> boundaries.
>>>
>>> On x86, the ORC unwinder provides reliable stacktrace through similar
>>> methodology, but arm64 lacks the necessary support from objtool to
>>> create ORC unwind tables.
>>>
>>> Currently, there's already a sframe unwinder proposed for userspace: [2].
>>> To maintain common definitions and algorithms for sframe lookup, a
>>> substantial portion of this patch series aims to refactor the sframe
>>> lookup code to support both kernel and userspace sframe sections.
>>>
>>> Currently, only GNU Binutils support sframe. This series relies on the
>>> Sframe V3 format, which is supported in binutils 2.46.
>>>
>>> These patches are based on Steven Rostedt's sframe/core branch [3],
>>> which is and aggregation of existing work done for x86 sframe userspace
>>> unwind, and contains [2]. This branch is, in turn, based on Linux
>>> v7.0-rc3. This full series (applied to the sframe/core branch) is
>>> available on github: [4].
>>>
>>
>> Not sure if related, but after updating my toolchain
>> (aarch64-linux-gnu-gcc (Debian 15.2.0-4) 15.2.0), I hit link errors:
>> ld.lld: error: arch/arm64/kernel/vdso/vgettimeofday.o:(.sframe) is being placed in '.sframe'
>> ld.lld: error: arch/arm64/kernel/vdso/vgetrandom.o:(.sframe) is being placed in '.sframe`
>
> Previously when developing against the SFrame V2 format, I had fixed
> these warnings with the VDSO Makefile change currently in this series:
>
> diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
> index 7dec05dd33b7..c60ef921956f 100644
> --- a/arch/arm64/kernel/vdso/Makefile
> +++ b/arch/arm64/kernel/vdso/Makefile
> @@ -38,7 +38,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO
> CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) \
> $(RANDSTRUCT_CFLAGS) $(KSTACK_ERASE_CFLAGS) \
> $(GCC_PLUGINS_CFLAGS) \
> - $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) \
> + $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CC_FLAGS_SFRAME) \
> -Wmissing-prototypes -Wmissing-declarations
>
> CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
>
> But the warnings seem to have returned after upgrading my toolchain,
> possibly due to SFrame V3 or some confounding change in GCC. The
> --gsframe in the assembler should be set to 'no' by default, so
> perhaps GCC is providing an override --gsframe internally?
Could it be that your build of binutils was configured with
--enable-default-sframe, so that the GNU assembler defaults to generate
.sframe? AFAIK this configure option was meant for distributors and
package maintainers.
You can check as follows whether --gsframe defaults to "no" or "yes":
$ as --help | grep -A1 gsframe
--gsframe[={no|yes}] whether to generate SFrame stack trace information
(default: no)
...
>
>>
>> I applied this series hoping that fix it, but it doesn't, so far I
>> have this hack :
>> diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
>> index 52314be29191..53bdf757ee44 100644
>> --- a/arch/arm64/kernel/vdso/vdso.lds.S
>> +++ b/arch/arm64/kernel/vdso/vdso.lds.S
>> @@ -77,7 +77,7 @@ SECTIONS
>> /DISCARD/ : {
>> *(.data .data.* .gnu.linkonce.d.* .sdata*)
>> *(.bss .sbss .dynbss .dynsbss)
>> - *(.eh_frame .eh_frame_hdr)
>> + *(.eh_frame .eh_frame_hdr .sframe)
>> }
>> }
>>
>> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
>> index 60c8c22fd3e4..759903acd6fc 100644
>> --- a/include/asm-generic/vmlinux.lds.h
>> +++ b/include/asm-generic/vmlinux.lds.h
>> @@ -1064,6 +1064,7 @@
>> /* ld.bfd warns about .gnu.version* even when not emitted */ \
>> *(.gnu.version*) \
>> *(__tracepoint_check) \
>> + *(.sframe) \
>>
>> #define DISCARDS \
>> /DISCARD/ : { \
>
> Since this series only handles kernel stacktrace, I believe it's
> better to omit the .sframe section entirely in the case where only
> ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME is enabled. I think this hack may
> work better for this purpose:
>
> diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
> index c60ef921956f..29f802bfedb1 100644
> --- a/arch/arm64/kernel/vdso/Makefile
> +++ b/arch/arm64/kernel/vdso/Makefile
> @@ -41,7 +41,7 @@ CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os
> $(CC_FLAGS_SCS) \
> $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CC_FLAGS_SFRAME) \
> -Wmissing-prototypes -Wmissing-declarations
>
> -CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
> +CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
> -Wa,--gsframe=no
>
> CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_REMOVE_VDSO)
> CFLAGS_REMOVE_vgetrandom.o = $(CC_FLAGS_REMOVE_VDSO)
>
> Though, I don't understand why it is necessary to provide --gsframe=no
> explicitly. If this approach seems ok to other folks/maintainers, I
> can fold this into my series.
Maybe build the VDSO separately with V=1 to see what assembler/compiler
options are effectively used (e.g. for vgettimeofday.o and vgetrandom.o
mentioned in the linker error message above)?
$ make mrproper
$ make defconfig
$ ./scripts/config --enable HAVE_UNWIND_KERNEL_SFRAME # enable kernel sframe unwinder
$ make arch/arm64/kernel/vdso/ V=1
>
> On the topic of SFrame for VDSO, Jens has a patch adding support for
> this as part of a series to support userspace SFrame unwinding for
> arm64:
>
> https://lore.kernel.org/lkml/20260417150827.1183376-4-jremus@linux.ibm.com/
Any feedback is very welcome. :-)
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
jremus@de.ibm.com / jremus@linux.ibm.com
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Ehningen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* [PATCH v6 9/9] unwind: arm64: Use sframe to unwind interrupt frames
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
Add unwind_next_frame_sframe() function to unwind by sframe info if
present. Use this method at exception boundaries, falling back to
frame-pointer unwind only on failure. In such failure cases, the
stacktrace is considered unreliable.
During normal unwind, prefer frame pointer unwind (for better
performance) with sframe as a backup.
This change restores the LR behavior originally introduced in commit
c2c6b27b5aa14fa2 ("arm64: stacktrace: unwind exception boundaries"),
But later removed in commit 32ed1205682e ("arm64: stacktrace: Skip
reporting LR at exception boundaries")
This can be done because the sframe data can be used to determine
whether the LR is current for the PC value recovered from pt_regs at the
exception boundary.
Signed-off-by: Weinan Liu <wnliu@google.com>
Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
arch/arm64/kernel/stacktrace.c | 222 ++++++++++++++++++++++++++++++---
1 file changed, 202 insertions(+), 20 deletions(-)
diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index 3ebcf8c53fb0..cee860ca8ce5 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -13,6 +13,7 @@
#include <linux/sched.h>
#include <linux/sched/debug.h>
#include <linux/sched/task_stack.h>
+#include <linux/sframe.h>
#include <linux/stacktrace.h>
#include <asm/efi.h>
@@ -26,6 +27,7 @@ enum kunwind_source {
KUNWIND_SOURCE_CALLER,
KUNWIND_SOURCE_TASK,
KUNWIND_SOURCE_REGS_PC,
+ KUNWIND_SOURCE_REGS_LR,
};
union unwind_flags {
@@ -45,6 +47,7 @@ union unwind_flags {
* @kr_cur: When KRETPROBES is selected, holds the kretprobe instance
* associated with the most recently encountered replacement lr
* value.
+ * @unreliable: Stacktrace is unreliable.
*/
struct kunwind_state {
struct unwind_state common;
@@ -56,6 +59,7 @@ struct kunwind_state {
enum kunwind_source source;
union unwind_flags flags;
struct pt_regs *regs;
+ bool unreliable;
};
static __always_inline void
@@ -181,7 +185,6 @@ int kunwind_next_regs_pc(struct kunwind_state *state)
state->regs = regs;
state->common.pc = regs->pc;
state->common.fp = regs->regs[29];
- state->regs = NULL;
state->source = KUNWIND_SOURCE_REGS_PC;
return 0;
}
@@ -244,6 +247,168 @@ kunwind_next_frame_record(struct kunwind_state *state)
return 0;
}
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+
+static __always_inline struct stack_info *
+get_word(struct unwind_state *state, unsigned long *word)
+{
+ unsigned long addr = *word;
+ struct stack_info *info;
+
+ info = unwind_find_stack(state, addr, sizeof(addr));
+ if (!info)
+ return info;
+
+ *word = READ_ONCE(*(unsigned long *)addr);
+
+ return info;
+}
+
+static __always_inline int
+get_consume_word(struct unwind_state *state, unsigned long *word)
+{
+ struct stack_info *info;
+ unsigned long addr = *word;
+
+ info = get_word(state, word);
+ if (!info)
+ return -EINVAL;
+
+ unwind_consume_stack(state, info, addr, sizeof(addr));
+ return 0;
+}
+
+/*
+ * Unwind from a pt_regs according to sframe.
+ */
+static __always_inline int
+kunwind_next_regs_sframe(struct kunwind_state *state)
+{
+ struct unwind_frame frame;
+ unsigned long cfa, fp, ra;
+ enum kunwind_source source = KUNWIND_SOURCE_FRAME;
+ struct pt_regs *regs = state->regs;
+
+ int err;
+
+ if (WARN_ON_ONCE(state->source != KUNWIND_SOURCE_REGS_PC))
+ return -EINVAL;
+ if (WARN_ON_ONCE(!state->regs))
+ return -EINVAL;
+
+ /* FP/SP alignment 8 bytes */
+ if (state->common.fp & 0x7)
+ return -EINVAL;
+
+ err = sframe_find_kernel(state->common.pc, &frame);
+ if (err)
+ return -EINVAL;
+
+ /*
+ * A kernel unwind should always end at a FRAME_META_TYPE_FINAL
+ * frame. There should be no outermost frames within the kernel.
+ */
+ if (frame.outermost)
+ return -EINVAL;
+
+ /* Get the Canonical Frame Address (CFA) */
+ switch (frame.cfa.rule) {
+ case UNWIND_CFA_RULE_SP_OFFSET:
+ cfa = state->regs->sp;
+ break;
+ case UNWIND_CFA_RULE_FP_OFFSET:
+ if (state->common.fp < state->regs->sp)
+ return -EINVAL;
+ cfa = state->common.fp;
+ break;
+ /*
+ * UNWIND_CFA_RULE_REG_OFFSET and UNWIND_CFA_RULE_REG_OFFSET_DEREF not
+ * implemented -- flexible FDEs are not currently generated by assembler
+ * for arm64.
+ */
+ default:
+ WARN_ON_ONCE(1);
+ return -EINVAL;
+ }
+ cfa += frame.cfa.offset;
+
+ /* CFA alignment 16 bytes */
+ if (cfa & 0x15)
+ return -EINVAL;
+
+ /* Get the Return Address (RA) */
+ switch (frame.ra.rule) {
+ case UNWIND_RULE_RETAIN:
+ ra = regs->regs[30];
+ source = KUNWIND_SOURCE_REGS_LR;
+ break;
+
+ /*
+ * UNWIND_RULE_CFA_OFFSET doesn't make sense for RA.
+ * The return address cannot legitimately be a stack address.
+ */
+ case UNWIND_RULE_CFA_OFFSET_DEREF:
+ ra = cfa + frame.ra.offset;
+ break;
+ /*
+ * UNWIND_RULE_REG_OFFSET and UNWIND_RULE_REG_OFFSET_DEREF not
+ * implemented -- flexible FDEs are not currently generated by assembler
+ * for arm64.
+ */
+ default:
+ WARN_ON_ONCE(1);
+ return -EINVAL;
+ }
+
+ /* Get the Frame Pointer (FP) */
+ switch (frame.fp.rule) {
+ case UNWIND_RULE_RETAIN:
+ fp = state->common.fp;
+ break;
+ /*
+ * UNWIND_RULE_CFA_OFFSET is currently not used for FP
+ * (e.g. SFrame cannot represent this rule).
+ */
+ case UNWIND_RULE_CFA_OFFSET_DEREF:
+ fp = cfa + frame.fp.offset;
+ break;
+ /*
+ * UNWIND_RULE_REG_OFFSET and UNWIND_RULE_REG_OFFSET_DEREF not
+ * implemented -- flexible FDEs are not currently generated by assembler
+ * for arm64.
+ */
+ default:
+ WARN_ON_ONCE(1);
+ return -EINVAL;
+ }
+
+ /*
+ * Consume RA and FP from the stack. The frame record puts FP at a lower
+ * address than RA, so we always read FP first.
+ */
+ if (frame.fp.rule & UNWIND_RULE_DEREF &&
+ !get_word(&state->common, &fp))
+ return -EINVAL;
+
+ if (frame.ra.rule & UNWIND_RULE_DEREF &&
+ get_consume_word(&state->common, &ra))
+ return -EINVAL;
+
+ state->common.pc = ra;
+ state->common.fp = fp;
+
+ state->source = source;
+
+ return 0;
+}
+
+#else /* !CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
+
+static __always_inline int
+unwind_next_frame_sframe(struct kunwind_state *state) { return -EINVAL; }
+
+#endif /* !CONFIG_HAVE_UNWIND_KERNEL_SFRAME*/
+
/*
* Unwind from one frame record (A) to the next frame record (B).
*
@@ -259,10 +424,20 @@ kunwind_next(struct kunwind_state *state)
state->flags.all = 0;
switch (state->source) {
+ case KUNWIND_SOURCE_REGS_PC:
+ err = kunwind_next_regs_sframe(state);
+
+ if (err && err != -ENOENT) {
+ /* Fallback to FP based unwinder */
+ err = kunwind_next_frame_record(state);
+ state->unreliable = true;
+ }
+ state->regs = NULL;
+ break;
case KUNWIND_SOURCE_FRAME:
case KUNWIND_SOURCE_CALLER:
case KUNWIND_SOURCE_TASK:
- case KUNWIND_SOURCE_REGS_PC:
+ case KUNWIND_SOURCE_REGS_LR:
err = kunwind_next_frame_record(state);
break;
default:
@@ -390,34 +565,40 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry,
kunwind_stack_walk(arch_kunwind_consume_entry, &data, task, regs);
}
+struct kunwind_reliable_consume_entry_data {
+ stack_trace_consume_fn consume_entry;
+ void *cookie;
+ bool unreliable;
+};
+
static __always_inline bool
-arch_reliable_kunwind_consume_entry(const struct kunwind_state *state, void *cookie)
+arch_kunwind_reliable_consume_entry(const struct kunwind_state *state, void *cookie)
{
- /*
- * At an exception boundary we can reliably consume the saved PC. We do
- * not know whether the LR was live when the exception was taken, and
- * so we cannot perform the next unwind step reliably.
- *
- * All that matters is whether the *entire* unwind is reliable, so give
- * up as soon as we hit an exception boundary.
- */
- if (state->source == KUNWIND_SOURCE_REGS_PC)
- return false;
+ struct kunwind_reliable_consume_entry_data *data = cookie;
- return arch_kunwind_consume_entry(state, cookie);
+ if (state->unreliable) {
+ data->unreliable = true;
+ return false;
+ }
+ return data->consume_entry(data->cookie, state->common.pc);
}
-noinline noinstr int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry,
- void *cookie,
- struct task_struct *task)
+noinline notrace int arch_stack_walk_reliable(
+ stack_trace_consume_fn consume_entry,
+ void *cookie, struct task_struct *task)
{
- struct kunwind_consume_entry_data data = {
+ struct kunwind_reliable_consume_entry_data data = {
.consume_entry = consume_entry,
.cookie = cookie,
+ .unreliable = false,
};
- return kunwind_stack_walk(arch_reliable_kunwind_consume_entry, &data,
- task, NULL);
+ kunwind_stack_walk(arch_kunwind_reliable_consume_entry, &data, task, NULL);
+
+ if (data.unreliable)
+ return -EINVAL;
+
+ return 0;
}
struct bpf_unwind_consume_entry_data {
@@ -452,6 +633,7 @@ static const char *state_source_string(const struct kunwind_state *state)
case KUNWIND_SOURCE_CALLER: return "C";
case KUNWIND_SOURCE_TASK: return "T";
case KUNWIND_SOURCE_REGS_PC: return "P";
+ case KUNWIND_SOURCE_REGS_LR: return "L";
default: return "U";
}
}
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 8/9] sframe: Initialize debug info for kernel sections
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
Setup the optional unwinder debug information for kernel .sframe
sections. Modules are indicated by the format "(<module-name>)".
Suggested-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
kernel/unwind/sframe.c | 4 ++++
kernel/unwind/sframe_debug.h | 13 +++++++++++++
2 files changed, 17 insertions(+)
diff --git a/kernel/unwind/sframe.c b/kernel/unwind/sframe.c
index e8ede0343cb2..d256e72620fe 100644
--- a/kernel/unwind/sframe.c
+++ b/kernel/unwind/sframe.c
@@ -1036,6 +1036,8 @@ void __init init_sframe_table(void)
kernel_sfsec.text_start = (unsigned long)_stext;
kernel_sfsec.text_end = (unsigned long)_etext;
+ dbg_init(&kernel_sfsec);
+
if (WARN_ON(sframe_read_header(&kernel_sfsec)))
return;
if (WARN_ON(sframe_validate_section(&kernel_sfsec)))
@@ -1099,6 +1101,8 @@ void sframe_module_init(struct module *mod, void *sframe, size_t sframe_size,
sec->text_start = (unsigned long)text;
sec->text_end = (unsigned long)text + text_size;
+ dbg_init(sec);
+
if (WARN_ON(sframe_read_header(sec)))
return;
if (WARN_ON(sframe_sort_fdes(sec)))
diff --git a/kernel/unwind/sframe_debug.h b/kernel/unwind/sframe_debug.h
index e568be4172b1..6c7ab3aa7c9e 100644
--- a/kernel/unwind/sframe_debug.h
+++ b/kernel/unwind/sframe_debug.h
@@ -32,6 +32,19 @@ static inline void dbg_init(struct sframe_section *sec)
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma;
+ if (sec->sec_type == SFRAME_KERNEL) {
+ if (sec == &kernel_sfsec) {
+ sec->filename = kstrdup("(vmlinux)", GFP_KERNEL);
+ } else {
+ struct module *mod = container_of(sec, struct module,
+ arch.sframe_sec);
+ sec->filename = kasprintf(GFP_KERNEL, "(%s)",
+ mod->name);
+ }
+
+ return;
+ }
+
guard(mmap_read_lock)(mm);
vma = vma_lookup(mm, sec->sframe_start);
if (!vma)
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 7/9] sframe: Introduce in-kernel SFRAME_VALIDATION
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
Generalize the __safe* helpers to support a non-user-access code path.
This requires arch-specific function address validation. This is because
arm64 vmlinux keeps .exit.text (normally discarded), and .rodata.text
sections both of which lie outside the bounds of the normal .text.
.rodata.text contains code that is never executed by the kernel mapping,
but for which the toolchain nonetheless generates sframe data, and needs
to be considered valid for a PC lookup.
Additionally .init.text lies outside .text for all arches and must be
accounted for as well.
Suggested-by: Jens Remus <jremus@linux.ibm.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
arch/Kconfig | 2 +-
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/include/asm/unwind_sframe.h | 46 ++++++++++++++++++++++++++
arch/arm64/kernel/vmlinux.lds.S | 2 ++
include/linux/sframe.h | 2 ++
kernel/unwind/sframe.c | 25 ++++++++++++--
6 files changed, 75 insertions(+), 3 deletions(-)
diff --git a/arch/Kconfig b/arch/Kconfig
index f931b5848593..fa1f43f47a53 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -503,7 +503,7 @@ config HAVE_UNWIND_USER_SFRAME
config SFRAME_VALIDATION
bool "Enable .sframe section debugging"
- depends on HAVE_UNWIND_USER_SFRAME
+ depends on UNWIND_SFRAME_LOOKUP
depends on DYNAMIC_DEBUG
help
When adding an .sframe section for a task, validate the entire
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 51b0d594239e..5edb4304f661 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -23,6 +23,7 @@ extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[];
+extern char _srodatatext[], _erodatatext[];
static inline size_t entry_tramp_text_size(void)
{
diff --git a/arch/arm64/include/asm/unwind_sframe.h b/arch/arm64/include/asm/unwind_sframe.h
index 876412881196..eb269a54b9ef 100644
--- a/arch/arm64/include/asm/unwind_sframe.h
+++ b/arch/arm64/include/asm/unwind_sframe.h
@@ -2,7 +2,53 @@
#ifndef _ASM_ARM64_UNWIND_SFRAME_H
#define _ASM_ARM64_UNWIND_SFRAME_H
+#include <linux/module.h>
+#include <linux/sframe.h>
+#include <asm/sections.h>
+
#define SFRAME_REG_SP 31
#define SFRAME_REG_FP 29
+static inline bool sframe_func_start_addr_valid(struct sframe_section *sec,
+ unsigned long func_addr)
+{
+ /* Common case for unwinding */
+ if (sec->text_start <= func_addr && func_addr < sec->text_end)
+ return true;
+
+ if (sec->sec_type != SFRAME_KERNEL)
+ return false;
+
+ /*
+ * Account for vmlinux and module code outside the normal .text section.
+ * The toolchain still generates sframe data for these functions, so
+ * sframe lookups on them should be allowed.
+ */
+ if (sec == &kernel_sfsec) {
+ if (is_kernel_inittext(func_addr))
+ return true;
+
+ /* .exit.text is retained in vmlinux on arm64. */
+ if (func_addr >= (unsigned long)__exittext_begin &&
+ func_addr < (unsigned long)__exittext_end)
+ return true;
+
+ /*
+ * .rodata.text is never executed from the kernel mapping, but
+ * still has sframe data
+ */
+ if (func_addr >= (unsigned long)_srodatatext &&
+ func_addr < (unsigned long)_erodatatext)
+ return true;
+ } else {
+ struct module *mod = container_of(sec, struct module,
+ arch.sframe_sec);
+ if (within_module_mem_type(func_addr, mod, MOD_INIT_TEXT))
+ return true;
+ }
+
+ return false;
+}
+#define sframe_func_start_addr_valid sframe_func_start_addr_valid
+
#endif /* _ASM_ARM64_UNWIND_SFRAME_H */
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index e1ac876200a3..68700b4d5070 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -225,12 +225,14 @@ SECTIONS
/* code sections that are never executed via the kernel mapping */
.rodata.text : {
+ _srodatatext = .;
TRAMP_TEXT
HIBERNATE_TEXT
KEXEC_TEXT
IDMAP_TEXT
. = ALIGN(PAGE_SIZE);
}
+ _erodatatext = .;
idmap_pg_dir = .;
. += PAGE_SIZE;
diff --git a/include/linux/sframe.h b/include/linux/sframe.h
index 27f5a66190af..ac3aa9db7d91 100644
--- a/include/linux/sframe.h
+++ b/include/linux/sframe.h
@@ -34,6 +34,8 @@ struct sframe_section {
signed char fp_off;
};
+extern struct sframe_section kernel_sfsec __ro_after_init;
+
#endif /* CONFIG_UNWIND_SFRAME_LOOKUP */
#ifdef CONFIG_HAVE_UNWIND_USER_SFRAME
diff --git a/kernel/unwind/sframe.c b/kernel/unwind/sframe.c
index dfa013450705..e8ede0343cb2 100644
--- a/kernel/unwind/sframe.c
+++ b/kernel/unwind/sframe.c
@@ -24,10 +24,18 @@
#include "sframe.h"
#include "sframe_debug.h"
+#ifndef sframe_func_start_addr_valid
+static inline bool sframe_func_start_addr_valid(struct sframe_section *sec,
+ unsigned long func_addr)
+{
+ return (sec->text_start <= func_addr && func_addr < sec->text_end);
+}
+#endif
+
#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
static bool sframe_init __ro_after_init;
-static struct sframe_section kernel_sfsec __ro_after_init;
+struct sframe_section kernel_sfsec __ro_after_init;
#endif /* CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
@@ -155,7 +163,7 @@ static __always_inline int __read_fde(struct sframe_section *sec,
sizeof(struct sframe_fde_v3), Efault);
func_addr = fde_addr + _fde.func_start_off;
- if (func_addr < sec->text_start || func_addr >= sec->text_end)
+ if (!sframe_func_start_addr_valid(sec, func_addr))
return -EINVAL;
fda_addr = sec->fres_start + _fde.fres_off;
@@ -607,6 +615,9 @@ static int safe_read_fde(struct sframe_section *sec,
{
int ret;
+ if (sec->sec_type == SFRAME_KERNEL)
+ return __read_fde(sec, fde_num, fde);
+
if (!user_read_access_begin((void __user *)sec->sframe_start,
sec->sframe_end - sec->sframe_start))
return -EFAULT;
@@ -622,6 +633,9 @@ static int safe_read_fre(struct sframe_section *sec,
{
int ret;
+ if (sec->sec_type == SFRAME_KERNEL)
+ return __read_fre(sec, fde, fre_addr, fre);
+
if (!user_read_access_begin((void __user *)sec->sframe_start,
sec->sframe_end - sec->sframe_start))
return -EFAULT;
@@ -636,6 +650,9 @@ static int safe_read_fre_datawords(struct sframe_section *sec,
{
int ret;
+ if (sec->sec_type == SFRAME_KERNEL)
+ return __read_fre_datawords(sec, fde, fre);
+
if (!user_read_access_begin((void __user *)sec->sframe_start,
sec->sframe_end - sec->sframe_start))
return -EFAULT;
@@ -1021,6 +1038,8 @@ void __init init_sframe_table(void)
if (WARN_ON(sframe_read_header(&kernel_sfsec)))
return;
+ if (WARN_ON(sframe_validate_section(&kernel_sfsec)))
+ return;
sframe_init = true;
}
@@ -1084,6 +1103,8 @@ void sframe_module_init(struct module *mod, void *sframe, size_t sframe_size,
return;
if (WARN_ON(sframe_sort_fdes(sec)))
return;
+ if (WARN_ON(sframe_validate_section(sec)))
+ return;
mod->arch.sframe_init = true;
}
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 6/9] arm64/module, sframe: Add sframe support for modules
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
Add sframe table to mod_arch_specific and support sframe PC lookups when
an .sframe section can be found on incoming modules. SFRAME_F_FDE_SORTED
is not set for module .sframe, so FDES are sorted right after the sframe
header is read.
Signed-off-by: Weinan Liu <wnliu@google.com>
Suggested-by: Jens Remus <jremus@linux.ibm.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
arch/arm64/include/asm/module.h | 6 +++
arch/arm64/kernel/module.c | 8 +++
include/linux/sframe.h | 3 ++
kernel/unwind/sframe.c | 90 +++++++++++++++++++++++++++++++--
4 files changed, 104 insertions(+), 3 deletions(-)
diff --git a/arch/arm64/include/asm/module.h b/arch/arm64/include/asm/module.h
index fb9b88eebeb1..07f309c51eee 100644
--- a/arch/arm64/include/asm/module.h
+++ b/arch/arm64/include/asm/module.h
@@ -6,6 +6,7 @@
#define __ASM_MODULE_H
#include <asm-generic/module.h>
+#include <linux/sframe.h>
struct mod_plt_sec {
int plt_shndx;
@@ -17,6 +18,11 @@ struct mod_arch_specific {
struct mod_plt_sec core;
struct mod_plt_sec init;
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+ struct sframe_section sframe_sec;
+ bool sframe_init;
+#endif
+
/* for CONFIG_DYNAMIC_FTRACE */
struct plt_entry *ftrace_trampolines;
struct plt_entry *init_ftrace_trampolines;
diff --git a/arch/arm64/kernel/module.c b/arch/arm64/kernel/module.c
index 24adb581af0e..427f187e9531 100644
--- a/arch/arm64/kernel/module.c
+++ b/arch/arm64/kernel/module.c
@@ -18,6 +18,7 @@
#include <linux/moduleloader.h>
#include <linux/random.h>
#include <linux/scs.h>
+#include <linux/sframe.h>
#include <asm/alternative.h>
#include <asm/insn.h>
@@ -515,5 +516,12 @@ int module_finalize(const Elf_Ehdr *hdr,
}
}
+ s = find_section(hdr, sechdrs, ".sframe");
+ if (s) {
+ struct module_memory *t = &me->mem[MOD_TEXT];
+
+ sframe_module_init(me, (void *)s->sh_addr, s->sh_size,
+ t->base, t->size);
+ }
return module_init_ftrace_plt(hdr, sechdrs, me);
}
diff --git a/include/linux/sframe.h b/include/linux/sframe.h
index 5b7341b61a7c..27f5a66190af 100644
--- a/include/linux/sframe.h
+++ b/include/linux/sframe.h
@@ -28,6 +28,7 @@ struct sframe_section {
unsigned long fres_start;
unsigned long fres_end;
unsigned int num_fdes;
+ bool fdes_sorted;
signed char ra_off;
signed char fp_off;
@@ -80,6 +81,8 @@ extern int sframe_find_kernel(unsigned long ip, struct unwind_frame *frame);
#else
static inline void __init init_sframe_table(void) {}
+static inline void sframe_module_init(struct module *mod, void *sframe, size_t sframe_size,
+ void *text, size_t text_size) {}
#endif /* CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
diff --git a/kernel/unwind/sframe.c b/kernel/unwind/sframe.c
index c8ec1e9989fc..dfa013450705 100644
--- a/kernel/unwind/sframe.c
+++ b/kernel/unwind/sframe.c
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/string_helpers.h>
#include <linux/sframe.h>
+#include <linux/sort.h>
#include <linux/syscalls.h>
#include <linux/unwind_types.h>
#include <asm/unwind_sframe.h>
@@ -186,6 +187,9 @@ static __always_inline int __find_fde(struct sframe_section *sec,
struct sframe_fde_v3 *first, *low, *high, *found = NULL;
int ret;
+ if (!sec->fdes_sorted)
+ return -EINVAL;
+
first = (void *)sec->fdes_start;
low = first;
high = first + sec->num_fdes - 1;
@@ -740,7 +744,6 @@ static int sframe_read_header(struct sframe_section *sec)
if (shdr.preamble.magic != SFRAME_MAGIC ||
shdr.preamble.version != SFRAME_VERSION_3 ||
- !(shdr.preamble.flags & SFRAME_F_FDE_SORTED) ||
!(shdr.preamble.flags & SFRAME_F_FDE_FUNC_START_PCREL) ||
shdr.auxhdr_len) {
dbg_sec("bad/unsupported sframe header\n");
@@ -770,6 +773,7 @@ static int sframe_read_header(struct sframe_section *sec)
return -EINVAL;
}
+ sec->fdes_sorted = shdr.preamble.flags & SFRAME_F_FDE_SORTED;
sec->num_fdes = num_fdes;
sec->fdes_start = fdes_start;
sec->fres_start = fres_start;
@@ -984,10 +988,27 @@ SYSCALL_DEFINE5(stacktrace_setup, int, op, unsigned long, addr_start,
int sframe_find_kernel(unsigned long ip, struct unwind_frame *frame)
{
- if (!frame || !sframe_init)
+ struct sframe_section *sec;
+
+ if (!frame)
return -EINVAL;
- return __sframe_find(&kernel_sfsec, ip, frame);
+ if (is_ksym_addr(ip)) {
+ if (!sframe_init)
+ return -EINVAL;
+
+ sec = &kernel_sfsec;
+ } else {
+ struct module *mod;
+
+ mod = __module_address(ip);
+ if (!mod || !mod->arch.sframe_init)
+ return -EINVAL;
+
+ sec = &mod->arch.sframe_sec;
+ }
+
+ return __sframe_find(sec, ip, frame);
}
void __init init_sframe_table(void)
@@ -1004,4 +1025,67 @@ void __init init_sframe_table(void)
sframe_init = true;
}
+static int sframe_sort_cmp_fde(const void *a, const void *b)
+{
+ const struct sframe_fde_v3 *fde_a = a, *fde_b = b;
+ unsigned long func_start_a, func_start_b;
+
+ func_start_a = (unsigned long)fde_a + fde_a->func_start_off;
+ func_start_b = (unsigned long)fde_b + fde_b->func_start_off;
+
+ return cmp_int(func_start_a, func_start_b);
+}
+
+static void sframe_sort_swap_fde(void *a, void *b, int size)
+{
+ struct sframe_fde_v3 *fde_a = a, *fde_b = b;
+ struct sframe_fde_v3 temp;
+ long delta;
+
+ /* Swap potentially unaligned FDE */
+ memcpy(&temp, fde_a, sizeof(struct sframe_fde_v3));
+ memcpy(fde_a, fde_b, sizeof(struct sframe_fde_v3));
+ memcpy(fde_b, &temp, sizeof(struct sframe_fde_v3));
+
+ /* Adjust FDE function start offset from FDE */
+ delta = (long)((unsigned long)fde_b - (unsigned long)fde_a);
+ fde_a->func_start_off += delta;
+ fde_b->func_start_off -= delta;
+}
+
+static int sframe_sort_fdes(struct sframe_section *sec)
+{
+ void *fdes = (void *)sec->fdes_start;
+ size_t num_fdes = sec->num_fdes;
+
+ if (sec->sec_type != SFRAME_KERNEL)
+ return -EINVAL;
+ if (sec->fdes_sorted)
+ return 0;
+
+ sort(fdes, num_fdes, sizeof(struct sframe_fde_v3),
+ sframe_sort_cmp_fde, sframe_sort_swap_fde);
+ sec->fdes_sorted = true;
+ return 0;
+}
+
+void sframe_module_init(struct module *mod, void *sframe, size_t sframe_size,
+ void *text, size_t text_size)
+{
+ struct sframe_section *sec = &mod->arch.sframe_sec;
+
+ sec->sec_type = SFRAME_KERNEL;
+ sec->sframe_start = (unsigned long)sframe;
+ sec->sframe_end = (unsigned long)sframe + sframe_size;
+ sec->text_start = (unsigned long)text;
+ sec->text_end = (unsigned long)text + text_size;
+
+ if (WARN_ON(sframe_read_header(sec)))
+ return;
+ if (WARN_ON(sframe_sort_fdes(sec)))
+ return;
+
+ mod->arch.sframe_init = true;
+}
+
#endif /* CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 5/9] sframe: Provide PC lookup for vmlinux .sframe section
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
With HAVE_UNWIND_KERNEL_SFRAME, read in the .sframe section at boot.
This provides unwind data as an alternative/supplement to frame pointer
based unwinding.
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
arch/arm64/kernel/setup.c | 2 ++
include/linux/sframe.h | 14 ++++++++++++++
kernel/unwind/sframe.c | 36 ++++++++++++++++++++++++++++++++++++
3 files changed, 52 insertions(+)
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 23c05dc7a8f2..4a633bc7aefb 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -32,6 +32,7 @@
#include <linux/sched/task.h>
#include <linux/scs.h>
#include <linux/mm.h>
+#include <linux/sframe.h>
#include <asm/acpi.h>
#include <asm/fixmap.h>
@@ -375,6 +376,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
"This indicates a broken bootloader or old kernel\n",
boot_args[1], boot_args[2], boot_args[3]);
}
+ init_sframe_table();
}
static inline bool cpu_can_disable(unsigned int cpu)
diff --git a/include/linux/sframe.h b/include/linux/sframe.h
index 0cb2924367bc..5b7341b61a7c 100644
--- a/include/linux/sframe.h
+++ b/include/linux/sframe.h
@@ -69,4 +69,18 @@ static inline int sframe_find_user(unsigned long ip, struct unwind_frame *frame)
#endif /* CONFIG_HAVE_UNWIND_USER_SFRAME */
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+
+void __init init_sframe_table(void);
+void sframe_module_init(struct module *mod, void *sframe, size_t sframe_size,
+ void *text, size_t text_size);
+
+extern int sframe_find_kernel(unsigned long ip, struct unwind_frame *frame);
+
+#else
+
+static inline void __init init_sframe_table(void) {}
+
+#endif /* CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
+
#endif /* _LINUX_SFRAME_H */
diff --git a/kernel/unwind/sframe.c b/kernel/unwind/sframe.c
index a2ab9a3e07b4..c8ec1e9989fc 100644
--- a/kernel/unwind/sframe.c
+++ b/kernel/unwind/sframe.c
@@ -16,10 +16,20 @@
#include <linux/unwind_types.h>
#include <asm/unwind_sframe.h>
#include <uapi/linux/stacktrace.h>
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+#include <linux/kallsyms.h>
+#endif
#include "sframe.h"
#include "sframe_debug.h"
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+
+static bool sframe_init __ro_after_init;
+static struct sframe_section kernel_sfsec __ro_after_init;
+
+#endif /* CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
+
struct sframe_fde_internal {
unsigned long func_addr;
u32 func_size;
@@ -969,3 +979,29 @@ SYSCALL_DEFINE5(stacktrace_setup, int, op, unsigned long, addr_start,
}
#endif /* CONFIG_HAVE_UNWIND_USER_SFRAME */
+
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+
+int sframe_find_kernel(unsigned long ip, struct unwind_frame *frame)
+{
+ if (!frame || !sframe_init)
+ return -EINVAL;
+
+ return __sframe_find(&kernel_sfsec, ip, frame);
+}
+
+void __init init_sframe_table(void)
+{
+ kernel_sfsec.sec_type = SFRAME_KERNEL;
+ kernel_sfsec.sframe_start = (unsigned long)__start_sframe;
+ kernel_sfsec.sframe_end = (unsigned long)__end_sframe;
+ kernel_sfsec.text_start = (unsigned long)_stext;
+ kernel_sfsec.text_end = (unsigned long)_etext;
+
+ if (WARN_ON(sframe_read_header(&kernel_sfsec)))
+ return;
+
+ sframe_init = true;
+}
+
+#endif /* CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 4/9] arm64, crypto/lib: Annotate leaf functions with CFI info.
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
DWARF CFI (Call Frame Information) specifies how to recover return
address and callee-saved registers for annotated functions. These
annotations are generated by the compiler, but for assembly, they must
be annotated by hand.
Add simple CFI annotations to assembly leaf functions so that the LR can
be recovered by the unwinder when an exception is taken from one of them.
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
arch/arm64/crypto/aes-ce-ccm-core.S | 12 +++----
arch/arm64/crypto/aes-neonbs-core.S | 40 +++++++++++------------
arch/arm64/crypto/ghash-ce-core.S | 20 ++++++------
arch/arm64/crypto/sm4-ce-ccm-core.S | 16 +++++-----
arch/arm64/crypto/sm4-ce-cipher-core.S | 4 +--
arch/arm64/crypto/sm4-ce-core.S | 44 +++++++++++++-------------
arch/arm64/crypto/sm4-ce-gcm-core.S | 16 +++++-----
arch/arm64/crypto/sm4-neon-core.S | 12 +++----
arch/arm64/include/asm/linkage.h | 30 ++++++++++++++++++
arch/arm64/lib/clear_page.S | 4 +--
arch/arm64/lib/clear_user.S | 4 +--
arch/arm64/lib/copy_from_user.S | 4 +--
arch/arm64/lib/copy_page.S | 4 +--
arch/arm64/lib/copy_to_user.S | 4 +--
arch/arm64/lib/memchr.S | 4 +--
arch/arm64/lib/memcmp.S | 4 +--
arch/arm64/lib/memcpy.S | 8 ++---
arch/arm64/lib/memset.S | 8 ++---
arch/arm64/lib/mte.S | 28 ++++++++--------
arch/arm64/lib/strchr.S | 4 +--
arch/arm64/lib/strcmp.S | 4 +--
arch/arm64/lib/strlen.S | 4 +--
arch/arm64/lib/strncmp.S | 4 +--
arch/arm64/lib/strnlen.S | 4 +--
arch/arm64/lib/tishift.S | 12 +++----
25 files changed, 164 insertions(+), 134 deletions(-)
diff --git a/arch/arm64/crypto/aes-ce-ccm-core.S b/arch/arm64/crypto/aes-ce-ccm-core.S
index f2624238fd95..519309c886b9 100644
--- a/arch/arm64/crypto/aes-ce-ccm-core.S
+++ b/arch/arm64/crypto/aes-ce-ccm-core.S
@@ -80,7 +80,7 @@ CPU_LE( rev x8, x8 )
ret
.endm
-SYM_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail)
+SYM_LEAF_FUNC_START_LOCAL(ce_aes_ccm_crypt_tail)
eor v0.16b, v0.16b, v5.16b /* final round mac */
eor v1.16b, v1.16b, v5.16b /* final round enc */
@@ -113,7 +113,7 @@ SYM_INNER_LABEL(ce_aes_ccm_final, SYM_L_LOCAL)
eor v0.16b, v0.16b, v1.16b /* en-/decrypt the mac */
0: st1 {v0.16b}, [x5] /* store result */
ret
-SYM_FUNC_END(ce_aes_ccm_crypt_tail)
+SYM_LEAF_FUNC_END(ce_aes_ccm_crypt_tail)
/*
* void ce_aes_ccm_encrypt(u8 out[], u8 const in[], u32 cbytes,
@@ -123,15 +123,15 @@ SYM_FUNC_END(ce_aes_ccm_crypt_tail)
* u8 const rk[], u32 rounds, u8 mac[],
* u8 ctr[], u8 const final_iv[]);
*/
-SYM_FUNC_START(ce_aes_ccm_encrypt)
+SYM_LEAF_FUNC_START(ce_aes_ccm_encrypt)
movi v22.16b, #255
aes_ccm_do_crypt 1
-SYM_FUNC_END(ce_aes_ccm_encrypt)
+SYM_LEAF_FUNC_END(ce_aes_ccm_encrypt)
-SYM_FUNC_START(ce_aes_ccm_decrypt)
+SYM_LEAF_FUNC_START(ce_aes_ccm_decrypt)
movi v22.16b, #0
aes_ccm_do_crypt 0
-SYM_FUNC_END(ce_aes_ccm_decrypt)
+SYM_LEAF_FUNC_END(ce_aes_ccm_decrypt)
.section ".rodata", "a"
.align 6
diff --git a/arch/arm64/crypto/aes-neonbs-core.S b/arch/arm64/crypto/aes-neonbs-core.S
index baf450717b24..34b5c3c63c22 100644
--- a/arch/arm64/crypto/aes-neonbs-core.S
+++ b/arch/arm64/crypto/aes-neonbs-core.S
@@ -381,7 +381,7 @@ ISRM0: .octa 0x0306090c00070a0d01040b0e0205080f
/*
* void aesbs_convert_key(u8 out[], u32 const rk[], int rounds)
*/
-SYM_FUNC_START(aesbs_convert_key)
+SYM_LEAF_FUNC_START(aesbs_convert_key)
ld1 {v7.4s}, [x1], #16 // load round 0 key
ld1 {v17.4s}, [x1], #16 // load round 1 key
@@ -426,10 +426,10 @@ SYM_FUNC_START(aesbs_convert_key)
eor v17.16b, v17.16b, v7.16b
str q17, [x0]
ret
-SYM_FUNC_END(aesbs_convert_key)
+SYM_LEAF_FUNC_END(aesbs_convert_key)
.align 4
-SYM_FUNC_START_LOCAL(aesbs_encrypt8)
+SYM_LEAF_FUNC_START_LOCAL(aesbs_encrypt8)
ldr q9, [bskey], #16 // round 0 key
ldr q8, M0SR
ldr q24, SR
@@ -489,10 +489,10 @@ SYM_FUNC_START_LOCAL(aesbs_encrypt8)
eor v2.16b, v2.16b, v12.16b
eor v5.16b, v5.16b, v12.16b
ret
-SYM_FUNC_END(aesbs_encrypt8)
+SYM_LEAF_FUNC_END(aesbs_encrypt8)
.align 4
-SYM_FUNC_START_LOCAL(aesbs_decrypt8)
+SYM_LEAF_FUNC_START_LOCAL(aesbs_decrypt8)
lsl x9, rounds, #7
add bskey, bskey, x9
@@ -554,7 +554,7 @@ SYM_FUNC_START_LOCAL(aesbs_decrypt8)
eor v3.16b, v3.16b, v12.16b
eor v5.16b, v5.16b, v12.16b
ret
-SYM_FUNC_END(aesbs_decrypt8)
+SYM_LEAF_FUNC_END(aesbs_decrypt8)
/*
* aesbs_ecb_encrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
@@ -621,21 +621,21 @@ SYM_FUNC_END(aesbs_decrypt8)
.endm
.align 4
-SYM_TYPED_FUNC_START(aesbs_ecb_encrypt)
+SYM_TYPED_LEAF_FUNC_START(aesbs_ecb_encrypt)
__ecb_crypt aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5
-SYM_FUNC_END(aesbs_ecb_encrypt)
+SYM_LEAF_FUNC_END(aesbs_ecb_encrypt)
.align 4
-SYM_TYPED_FUNC_START(aesbs_ecb_decrypt)
+SYM_TYPED_LEAF_FUNC_START(aesbs_ecb_decrypt)
__ecb_crypt aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5
-SYM_FUNC_END(aesbs_ecb_decrypt)
+SYM_LEAF_FUNC_END(aesbs_ecb_decrypt)
/*
* aesbs_cbc_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
* int blocks, u8 iv[])
*/
.align 4
-SYM_FUNC_START(aesbs_cbc_decrypt)
+SYM_LEAF_FUNC_START(aesbs_cbc_decrypt)
frame_push 6
mov x19, x0
@@ -719,7 +719,7 @@ SYM_FUNC_START(aesbs_cbc_decrypt)
2: frame_pop
ret
-SYM_FUNC_END(aesbs_cbc_decrypt)
+SYM_LEAF_FUNC_END(aesbs_cbc_decrypt)
.macro next_tweak, out, in, const, tmp
sshr \tmp\().2d, \in\().2d, #63
@@ -735,7 +735,7 @@ SYM_FUNC_END(aesbs_cbc_decrypt)
* aesbs_xts_decrypt(u8 out[], u8 const in[], u8 const rk[], int rounds,
* int blocks, u8 iv[])
*/
-SYM_FUNC_START_LOCAL(__xts_crypt8)
+SYM_LEAF_FUNC_START_LOCAL(__xts_crypt8)
movi v18.2s, #0x1
movi v19.2s, #0x87
uzp1 v18.4s, v18.4s, v19.4s
@@ -766,7 +766,7 @@ SYM_FUNC_START_LOCAL(__xts_crypt8)
mov bskey, x2
mov rounds, x3
br x16
-SYM_FUNC_END(__xts_crypt8)
+SYM_LEAF_FUNC_END(__xts_crypt8)
.macro __xts_crypt, do8, o0, o1, o2, o3, o4, o5, o6, o7
frame_push 0, 32
@@ -800,13 +800,13 @@ SYM_FUNC_END(__xts_crypt8)
ret
.endm
-SYM_TYPED_FUNC_START(aesbs_xts_encrypt)
+SYM_TYPED_LEAF_FUNC_START(aesbs_xts_encrypt)
__xts_crypt aesbs_encrypt8, v0, v1, v4, v6, v3, v7, v2, v5
-SYM_FUNC_END(aesbs_xts_encrypt)
+SYM_LEAF_FUNC_END(aesbs_xts_encrypt)
-SYM_TYPED_FUNC_START(aesbs_xts_decrypt)
+SYM_TYPED_LEAF_FUNC_START(aesbs_xts_decrypt)
__xts_crypt aesbs_decrypt8, v0, v1, v6, v4, v2, v7, v3, v5
-SYM_FUNC_END(aesbs_xts_decrypt)
+SYM_LEAF_FUNC_END(aesbs_xts_decrypt)
.macro next_ctr, v
mov \v\().d[1], x8
@@ -820,7 +820,7 @@ SYM_FUNC_END(aesbs_xts_decrypt)
* aesbs_ctr_encrypt(u8 out[], u8 const in[], u8 const rk[],
* int rounds, int blocks, u8 iv[])
*/
-SYM_FUNC_START(aesbs_ctr_encrypt)
+SYM_LEAF_FUNC_START(aesbs_ctr_encrypt)
frame_push 0
ldp x7, x8, [x5]
ld1 {v0.16b}, [x5]
@@ -863,4 +863,4 @@ CPU_LE( rev x8, x8 )
st1 {v0.16b}, [x5]
frame_pop
ret
-SYM_FUNC_END(aesbs_ctr_encrypt)
+SYM_LEAF_FUNC_END(aesbs_ctr_encrypt)
diff --git a/arch/arm64/crypto/ghash-ce-core.S b/arch/arm64/crypto/ghash-ce-core.S
index 33772d8fe6b5..3471430c2a53 100644
--- a/arch/arm64/crypto/ghash-ce-core.S
+++ b/arch/arm64/crypto/ghash-ce-core.S
@@ -66,7 +66,7 @@
* void pmull_ghash_update_p64(int blocks, u64 dg[], const char *src,
* u64 const h[4][2], const char *head)
*/
-SYM_FUNC_START(pmull_ghash_update_p64)
+SYM_LEAF_FUNC_START(pmull_ghash_update_p64)
ld1 {SHASH.2d}, [x3]
ld1 {XL.2d}, [x1]
@@ -173,7 +173,7 @@ CPU_LE( rev64 T1.16b, T1.16b )
5: st1 {XL.2d}, [x1]
ret
-SYM_FUNC_END(pmull_ghash_update_p64)
+SYM_LEAF_FUNC_END(pmull_ghash_update_p64)
KS0 .req v8
KS1 .req v9
@@ -417,9 +417,9 @@ CPU_LE( rev w8, w8 )
* u64 const h[4][2], u64 dg[], u8 ctr[],
* u32 const rk[], int rounds, u8 tag[])
*/
-SYM_FUNC_START(pmull_gcm_encrypt)
+SYM_LEAF_FUNC_START(pmull_gcm_encrypt)
pmull_gcm_do_crypt 1
-SYM_FUNC_END(pmull_gcm_encrypt)
+SYM_LEAF_FUNC_END(pmull_gcm_encrypt)
/*
* int pmull_gcm_decrypt(int bytes, u8 dst[], const u8 src[],
@@ -427,11 +427,11 @@ SYM_FUNC_END(pmull_gcm_encrypt)
* u32 const rk[], int rounds, const u8 l[],
* const u8 tag[], u64 authsize)
*/
-SYM_FUNC_START(pmull_gcm_decrypt)
+SYM_LEAF_FUNC_START(pmull_gcm_decrypt)
pmull_gcm_do_crypt 0
-SYM_FUNC_END(pmull_gcm_decrypt)
+SYM_LEAF_FUNC_END(pmull_gcm_decrypt)
-SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
+SYM_LEAF_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
movi MASK.16b, #0xe1
shl MASK.2d, MASK.2d, #57
@@ -512,9 +512,9 @@ SYM_FUNC_START_LOCAL(pmull_gcm_ghash_4x)
eor XL.16b, XL.16b, T2.16b
ret
-SYM_FUNC_END(pmull_gcm_ghash_4x)
+SYM_LEAF_FUNC_END(pmull_gcm_ghash_4x)
-SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
+SYM_LEAF_FUNC_START_LOCAL(pmull_gcm_enc_4x)
ld1 {KS0.16b}, [x5] // load upper counter
sub w10, w8, #4
sub w11, w8, #3
@@ -577,7 +577,7 @@ SYM_FUNC_START_LOCAL(pmull_gcm_enc_4x)
eor INP3.16b, INP3.16b, KS3.16b
ret
-SYM_FUNC_END(pmull_gcm_enc_4x)
+SYM_LEAF_FUNC_END(pmull_gcm_enc_4x)
.section ".rodata", "a"
.align 6
diff --git a/arch/arm64/crypto/sm4-ce-ccm-core.S b/arch/arm64/crypto/sm4-ce-ccm-core.S
index fa85856f33ce..20a8853609e0 100644
--- a/arch/arm64/crypto/sm4-ce-ccm-core.S
+++ b/arch/arm64/crypto/sm4-ce-ccm-core.S
@@ -37,7 +37,7 @@
.align 3
-SYM_FUNC_START(sm4_ce_cbcmac_update)
+SYM_LEAF_FUNC_START(sm4_ce_cbcmac_update)
/* input:
* x0: round key array, CTX
* x1: mac
@@ -81,10 +81,10 @@ SYM_FUNC_START(sm4_ce_cbcmac_update)
.Lcbcmac_end:
st1 {RMAC.16b}, [x1]
ret
-SYM_FUNC_END(sm4_ce_cbcmac_update)
+SYM_LEAF_FUNC_END(sm4_ce_cbcmac_update)
.align 3
-SYM_FUNC_START(sm4_ce_ccm_final)
+SYM_LEAF_FUNC_START(sm4_ce_ccm_final)
/* input:
* x0: round key array, CTX
* x1: ctr0 (big endian, 128 bit)
@@ -102,10 +102,10 @@ SYM_FUNC_START(sm4_ce_ccm_final)
st1 {RMAC.16b}, [x2]
ret
-SYM_FUNC_END(sm4_ce_ccm_final)
+SYM_LEAF_FUNC_END(sm4_ce_ccm_final)
.align 3
-SYM_TYPED_FUNC_START(sm4_ce_ccm_enc)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_ccm_enc)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -214,10 +214,10 @@ SYM_TYPED_FUNC_START(sm4_ce_ccm_enc)
.Lccm_enc_ret:
ret
-SYM_FUNC_END(sm4_ce_ccm_enc)
+SYM_LEAF_FUNC_END(sm4_ce_ccm_enc)
.align 3
-SYM_TYPED_FUNC_START(sm4_ce_ccm_dec)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_ccm_dec)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -326,4 +326,4 @@ SYM_TYPED_FUNC_START(sm4_ce_ccm_dec)
.Lccm_dec_ret:
ret
-SYM_FUNC_END(sm4_ce_ccm_dec)
+SYM_LEAF_FUNC_END(sm4_ce_ccm_dec)
diff --git a/arch/arm64/crypto/sm4-ce-cipher-core.S b/arch/arm64/crypto/sm4-ce-cipher-core.S
index 4ac6cfbc5797..7aea346cb14c 100644
--- a/arch/arm64/crypto/sm4-ce-cipher-core.S
+++ b/arch/arm64/crypto/sm4-ce-cipher-core.S
@@ -15,7 +15,7 @@
* void sm4_ce_do_crypt(const u32 *rk, u32 *out, const u32 *in);
*/
.text
-SYM_FUNC_START(sm4_ce_do_crypt)
+SYM_LEAF_FUNC_START(sm4_ce_do_crypt)
ld1 {v8.4s}, [x2]
ld1 {v0.4s-v3.4s}, [x0], #64
CPU_LE( rev32 v8.16b, v8.16b )
@@ -33,4 +33,4 @@ CPU_LE( rev32 v8.16b, v8.16b )
CPU_LE( rev32 v8.16b, v8.16b )
st1 {v8.4s}, [x1]
ret
-SYM_FUNC_END(sm4_ce_do_crypt)
+SYM_LEAF_FUNC_END(sm4_ce_do_crypt)
diff --git a/arch/arm64/crypto/sm4-ce-core.S b/arch/arm64/crypto/sm4-ce-core.S
index 1f3625c2c67e..6af5b10859b8 100644
--- a/arch/arm64/crypto/sm4-ce-core.S
+++ b/arch/arm64/crypto/sm4-ce-core.S
@@ -40,7 +40,7 @@
.align 3
-SYM_FUNC_START(sm4_ce_expand_key)
+SYM_LEAF_FUNC_START(sm4_ce_expand_key)
/* input:
* x0: 128-bit key
* x1: rkey_enc
@@ -86,10 +86,10 @@ SYM_FUNC_START(sm4_ce_expand_key)
st1 {v20.16b-v23.16b}, [x2]
ret;
-SYM_FUNC_END(sm4_ce_expand_key)
+SYM_LEAF_FUNC_END(sm4_ce_expand_key)
.align 3
-SYM_FUNC_START(sm4_ce_crypt_block)
+SYM_LEAF_FUNC_START(sm4_ce_crypt_block)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -102,10 +102,10 @@ SYM_FUNC_START(sm4_ce_crypt_block)
st1 {v0.16b}, [x1];
ret;
-SYM_FUNC_END(sm4_ce_crypt_block)
+SYM_LEAF_FUNC_END(sm4_ce_crypt_block)
.align 3
-SYM_FUNC_START(sm4_ce_crypt)
+SYM_LEAF_FUNC_START(sm4_ce_crypt)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -153,10 +153,10 @@ SYM_FUNC_START(sm4_ce_crypt)
.Lcrypt_end:
ret;
-SYM_FUNC_END(sm4_ce_crypt)
+SYM_LEAF_FUNC_END(sm4_ce_crypt)
.align 3
-SYM_FUNC_START(sm4_ce_cbc_enc)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_enc)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -208,10 +208,10 @@ SYM_FUNC_START(sm4_ce_cbc_enc)
st1 {RIV.16b}, [x3]
ret
-SYM_FUNC_END(sm4_ce_cbc_enc)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_enc)
.align 3
-SYM_FUNC_START(sm4_ce_cbc_dec)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_dec)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -306,10 +306,10 @@ SYM_FUNC_START(sm4_ce_cbc_dec)
st1 {RIV.16b}, [x3]
ret
-SYM_FUNC_END(sm4_ce_cbc_dec)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_dec)
.align 3
-SYM_FUNC_START(sm4_ce_cbc_cts_enc)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_cts_enc)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -354,10 +354,10 @@ SYM_FUNC_START(sm4_ce_cbc_cts_enc)
st1 {v1.16b}, [x1]
ret
-SYM_FUNC_END(sm4_ce_cbc_cts_enc)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_cts_enc)
.align 3
-SYM_FUNC_START(sm4_ce_cbc_cts_dec)
+SYM_LEAF_FUNC_START(sm4_ce_cbc_cts_dec)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -400,10 +400,10 @@ SYM_FUNC_START(sm4_ce_cbc_cts_dec)
st1 {v0.16b}, [x1]
ret
-SYM_FUNC_END(sm4_ce_cbc_cts_dec)
+SYM_LEAF_FUNC_END(sm4_ce_cbc_cts_dec)
.align 3
-SYM_FUNC_START(sm4_ce_ctr_enc)
+SYM_LEAF_FUNC_START(sm4_ce_ctr_enc)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -506,7 +506,7 @@ SYM_FUNC_START(sm4_ce_ctr_enc)
stp x7, x8, [x3]
ret
-SYM_FUNC_END(sm4_ce_ctr_enc)
+SYM_LEAF_FUNC_END(sm4_ce_ctr_enc)
#define tweak_next(vt, vin, RTMP) \
@@ -517,7 +517,7 @@ SYM_FUNC_END(sm4_ce_ctr_enc)
eor vt.16b, vt.16b, RTMP.16b;
.align 3
-SYM_FUNC_START(sm4_ce_xts_enc)
+SYM_LEAF_FUNC_START(sm4_ce_xts_enc)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -681,10 +681,10 @@ SYM_FUNC_START(sm4_ce_xts_enc)
.Lxts_enc_ret:
ret
-SYM_FUNC_END(sm4_ce_xts_enc)
+SYM_LEAF_FUNC_END(sm4_ce_xts_enc)
.align 3
-SYM_FUNC_START(sm4_ce_xts_dec)
+SYM_LEAF_FUNC_START(sm4_ce_xts_dec)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -848,10 +848,10 @@ SYM_FUNC_START(sm4_ce_xts_dec)
.Lxts_dec_ret:
ret
-SYM_FUNC_END(sm4_ce_xts_dec)
+SYM_LEAF_FUNC_END(sm4_ce_xts_dec)
.align 3
-SYM_FUNC_START(sm4_ce_mac_update)
+SYM_LEAF_FUNC_START(sm4_ce_mac_update)
/* input:
* x0: round key array, CTX
* x1: digest
@@ -917,7 +917,7 @@ SYM_FUNC_START(sm4_ce_mac_update)
.Lmac_ret:
st1 {RMAC.16b}, [x1]
ret
-SYM_FUNC_END(sm4_ce_mac_update)
+SYM_LEAF_FUNC_END(sm4_ce_mac_update)
.section ".rodata", "a"
diff --git a/arch/arm64/crypto/sm4-ce-gcm-core.S b/arch/arm64/crypto/sm4-ce-gcm-core.S
index 347f25d75727..dac6db8160f2 100644
--- a/arch/arm64/crypto/sm4-ce-gcm-core.S
+++ b/arch/arm64/crypto/sm4-ce-gcm-core.S
@@ -259,7 +259,7 @@
#define RH4 v19
.align 3
-SYM_FUNC_START(sm4_ce_pmull_ghash_setup)
+SYM_LEAF_FUNC_START(sm4_ce_pmull_ghash_setup)
/* input:
* x0: round key array, CTX
* x1: ghash table
@@ -293,10 +293,10 @@ SYM_FUNC_START(sm4_ce_pmull_ghash_setup)
st1 {RH1.16b-RH4.16b}, [x1]
ret
-SYM_FUNC_END(sm4_ce_pmull_ghash_setup)
+SYM_LEAF_FUNC_END(sm4_ce_pmull_ghash_setup)
.align 3
-SYM_FUNC_START(pmull_ghash_update)
+SYM_LEAF_FUNC_START(pmull_ghash_update)
/* input:
* x0: ghash table
* x1: ghash result
@@ -368,10 +368,10 @@ SYM_FUNC_START(pmull_ghash_update)
st1 {RHASH.2d}, [x1]
ret
-SYM_FUNC_END(pmull_ghash_update)
+SYM_LEAF_FUNC_END(pmull_ghash_update)
.align 3
-SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_enc)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_pmull_gcm_enc)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -534,7 +534,7 @@ SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_enc)
st1 {RHASH.2d}, [x5]
ret
-SYM_FUNC_END(sm4_ce_pmull_gcm_enc)
+SYM_LEAF_FUNC_END(sm4_ce_pmull_gcm_enc)
#undef RR1
#undef RR3
@@ -582,7 +582,7 @@ SYM_FUNC_END(sm4_ce_pmull_gcm_enc)
#define RH3 v20
.align 3
-SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_dec)
+SYM_TYPED_LEAF_FUNC_START(sm4_ce_pmull_gcm_dec)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -726,7 +726,7 @@ SYM_TYPED_FUNC_START(sm4_ce_pmull_gcm_dec)
st1 {RHASH.2d}, [x5]
ret
-SYM_FUNC_END(sm4_ce_pmull_gcm_dec)
+SYM_LEAF_FUNC_END(sm4_ce_pmull_gcm_dec)
.section ".rodata", "a"
.align 4
diff --git a/arch/arm64/crypto/sm4-neon-core.S b/arch/arm64/crypto/sm4-neon-core.S
index 734dc7193610..d1fe37fce13a 100644
--- a/arch/arm64/crypto/sm4-neon-core.S
+++ b/arch/arm64/crypto/sm4-neon-core.S
@@ -257,7 +257,7 @@
.align 3
-SYM_FUNC_START(sm4_neon_crypt)
+SYM_LEAF_FUNC_START(sm4_neon_crypt)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -318,10 +318,10 @@ SYM_FUNC_START(sm4_neon_crypt)
.Lcrypt_end:
ret
-SYM_FUNC_END(sm4_neon_crypt)
+SYM_LEAF_FUNC_END(sm4_neon_crypt)
.align 3
-SYM_FUNC_START(sm4_neon_cbc_dec)
+SYM_LEAF_FUNC_START(sm4_neon_cbc_dec)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -435,10 +435,10 @@ SYM_FUNC_START(sm4_neon_cbc_dec)
st1 {RIV.16b}, [x3]
ret
-SYM_FUNC_END(sm4_neon_cbc_dec)
+SYM_LEAF_FUNC_END(sm4_neon_cbc_dec)
.align 3
-SYM_FUNC_START(sm4_neon_ctr_crypt)
+SYM_LEAF_FUNC_START(sm4_neon_ctr_crypt)
/* input:
* x0: round key array, CTX
* x1: dst
@@ -563,4 +563,4 @@ SYM_FUNC_START(sm4_neon_ctr_crypt)
stp x7, x8, [x3]
ret
-SYM_FUNC_END(sm4_neon_ctr_crypt)
+SYM_LEAF_FUNC_END(sm4_neon_ctr_crypt)
diff --git a/arch/arm64/include/asm/linkage.h b/arch/arm64/include/asm/linkage.h
index 40bd17add539..6b8cb36a3343 100644
--- a/arch/arm64/include/asm/linkage.h
+++ b/arch/arm64/include/asm/linkage.h
@@ -3,6 +3,12 @@
#ifdef __ASSEMBLER__
#include <asm/assembler.h>
+
+/*
+ * Do not generate .eh_frame. Only generate .debug_frame and optionally
+ * .sframe (via assembler option --gsframe[-N]).
+ */
+ .cfi_sections .debug_frame
#endif
#define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT
@@ -43,4 +49,28 @@
SYM_TYPED_START(name, SYM_L_GLOBAL, SYM_A_ALIGN) \
bti c ;
+
+/*
+ * SYM_[TYPED_]LEAF_FUNC_[START|END] macros add CFI minimal CFI directives
+ * allowing .sframe data to be generated for functions which do not modify the
+ * LR (x30). Unwind data will not be correct if these macros are used on
+ * non-leaf functions, as additional CFI directives would be necessary in such
+ * cases.
+ */
+#define SYM_LEAF_FUNC_START(name) \
+ .cfi_startproc ; \
+ SYM_FUNC_START(name)
+
+#define SYM_LEAF_FUNC_END(name) \
+ .cfi_endproc ; \
+ SYM_FUNC_END(name)
+
+#define SYM_LEAF_FUNC_START_LOCAL(name) \
+ .cfi_startproc ; \
+ SYM_FUNC_START_LOCAL(name)
+
+#define SYM_TYPED_LEAF_FUNC_START(name) \
+ .cfi_startproc ; \
+ SYM_TYPED_FUNC_START(name)
+
#endif
diff --git a/arch/arm64/lib/clear_page.S b/arch/arm64/lib/clear_page.S
index bd6f7d5eb6eb..fceb875c3570 100644
--- a/arch/arm64/lib/clear_page.S
+++ b/arch/arm64/lib/clear_page.S
@@ -14,7 +14,7 @@
* Parameters:
* x0 - dest
*/
-SYM_FUNC_START(__pi_clear_page)
+SYM_LEAF_FUNC_START(__pi_clear_page)
#ifdef CONFIG_AS_HAS_MOPS
.arch_extension mops
alternative_if_not ARM64_HAS_MOPS
@@ -48,6 +48,6 @@ alternative_else_nop_endif
tst x0, #(PAGE_SIZE - 1)
b.ne 2b
ret
-SYM_FUNC_END(__pi_clear_page)
+SYM_LEAF_FUNC_END(__pi_clear_page)
SYM_FUNC_ALIAS(clear_page, __pi_clear_page)
EXPORT_SYMBOL(clear_page)
diff --git a/arch/arm64/lib/clear_user.S b/arch/arm64/lib/clear_user.S
index de9a303b6ad0..cf07c010ca92 100644
--- a/arch/arm64/lib/clear_user.S
+++ b/arch/arm64/lib/clear_user.S
@@ -17,7 +17,7 @@
* Alignment fixed up by hardware.
*/
-SYM_FUNC_START(__arch_clear_user)
+SYM_LEAF_FUNC_START(__arch_clear_user)
add x2, x0, x1
#ifdef CONFIG_AS_HAS_MOPS
@@ -68,5 +68,5 @@ USER(7f, sttrb wzr, [x2, #-1])
8: add x0, x0, #4 // ...or the second word of the 4-7 byte case
9: sub x0, x2, x0
ret
-SYM_FUNC_END(__arch_clear_user)
+SYM_LEAF_FUNC_END(__arch_clear_user)
EXPORT_SYMBOL(__arch_clear_user)
diff --git a/arch/arm64/lib/copy_from_user.S b/arch/arm64/lib/copy_from_user.S
index 400057d607ec..e9bb9c2dd8e1 100644
--- a/arch/arm64/lib/copy_from_user.S
+++ b/arch/arm64/lib/copy_from_user.S
@@ -61,7 +61,7 @@
end .req x5
srcin .req x15
-SYM_FUNC_START(__arch_copy_from_user)
+SYM_LEAF_FUNC_START(__arch_copy_from_user)
add end, x0, x2
mov srcin, x1
#include "copy_template.S"
@@ -79,5 +79,5 @@ USER(9998f, ldtrb tmp1w, [srcin])
strb tmp1w, [dst], #1
9998: sub x0, end, dst // bytes not copied
ret
-SYM_FUNC_END(__arch_copy_from_user)
+SYM_LEAF_FUNC_END(__arch_copy_from_user)
EXPORT_SYMBOL(__arch_copy_from_user)
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index e6374e7e5511..b6048d648306 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -17,7 +17,7 @@
* x0 - dest
* x1 - src
*/
-SYM_FUNC_START(__pi_copy_page)
+SYM_LEAF_FUNC_START(__pi_copy_page)
#ifdef CONFIG_AS_HAS_MOPS
.arch_extension mops
alternative_if_not ARM64_HAS_MOPS
@@ -77,6 +77,6 @@ alternative_else_nop_endif
stnp x16, x17, [x0, #112 - 256]
ret
-SYM_FUNC_END(__pi_copy_page)
+SYM_LEAF_FUNC_END(__pi_copy_page)
SYM_FUNC_ALIAS(copy_page, __pi_copy_page)
EXPORT_SYMBOL(copy_page)
diff --git a/arch/arm64/lib/copy_to_user.S b/arch/arm64/lib/copy_to_user.S
index 819f2e3fc7a9..aea10a5b3cba 100644
--- a/arch/arm64/lib/copy_to_user.S
+++ b/arch/arm64/lib/copy_to_user.S
@@ -60,7 +60,7 @@
end .req x5
srcin .req x15
-SYM_FUNC_START(__arch_copy_to_user)
+SYM_LEAF_FUNC_START(__arch_copy_to_user)
add end, x0, x2
mov srcin, x1
#include "copy_template.S"
@@ -79,5 +79,5 @@ USER(9998f, sttrb tmp1w, [dst])
add dst, dst, #1
9998: sub x0, end, dst // bytes not copied
ret
-SYM_FUNC_END(__arch_copy_to_user)
+SYM_LEAF_FUNC_END(__arch_copy_to_user)
EXPORT_SYMBOL(__arch_copy_to_user)
diff --git a/arch/arm64/lib/memchr.S b/arch/arm64/lib/memchr.S
index 37a9f2a4f7f4..909599bba5bf 100644
--- a/arch/arm64/lib/memchr.S
+++ b/arch/arm64/lib/memchr.S
@@ -38,7 +38,7 @@
.p2align 4
nop
-SYM_FUNC_START(__pi_memchr)
+SYM_LEAF_FUNC_START(__pi_memchr)
and chrin, chrin, #0xff
lsr wordcnt, cntin, #3
cbz wordcnt, L(byte_loop)
@@ -71,6 +71,6 @@ CPU_LE( rev tmp, tmp)
L(not_found):
mov result, #0
ret
-SYM_FUNC_END(__pi_memchr)
+SYM_LEAF_FUNC_END(__pi_memchr)
SYM_FUNC_ALIAS_WEAK(memchr, __pi_memchr)
EXPORT_SYMBOL_NOKASAN(memchr)
diff --git a/arch/arm64/lib/memcmp.S b/arch/arm64/lib/memcmp.S
index a5ccf2c55f91..91ee3a00e664 100644
--- a/arch/arm64/lib/memcmp.S
+++ b/arch/arm64/lib/memcmp.S
@@ -32,7 +32,7 @@
#define tmp1 x7
#define tmp2 x8
-SYM_FUNC_START(__pi_memcmp)
+SYM_LEAF_FUNC_START(__pi_memcmp)
subs limit, limit, 8
b.lo L(less8)
@@ -134,6 +134,6 @@ L(byte_loop):
b.eq L(byte_loop)
sub result, data1w, data2w
ret
-SYM_FUNC_END(__pi_memcmp)
+SYM_LEAF_FUNC_END(__pi_memcmp)
SYM_FUNC_ALIAS_WEAK(memcmp, __pi_memcmp)
EXPORT_SYMBOL_NOKASAN(memcmp)
diff --git a/arch/arm64/lib/memcpy.S b/arch/arm64/lib/memcpy.S
index 9b99106fb95f..90caf402ea7d 100644
--- a/arch/arm64/lib/memcpy.S
+++ b/arch/arm64/lib/memcpy.S
@@ -57,7 +57,7 @@
The loop tail is handled by always copying 64 bytes from the end.
*/
-SYM_FUNC_START_LOCAL(__pi_memcpy_generic)
+SYM_LEAF_FUNC_START_LOCAL(__pi_memcpy_generic)
add srcend, src, count
add dstend, dstin, count
cmp count, 128
@@ -238,11 +238,11 @@ L(copy64_from_start):
stp B_l, B_h, [dstin, 16]
stp C_l, C_h, [dstin]
ret
-SYM_FUNC_END(__pi_memcpy_generic)
+SYM_LEAF_FUNC_END(__pi_memcpy_generic)
#ifdef CONFIG_AS_HAS_MOPS
.arch_extension mops
-SYM_FUNC_START(__pi_memcpy)
+SYM_LEAF_FUNC_START(__pi_memcpy)
alternative_if_not ARM64_HAS_MOPS
b __pi_memcpy_generic
alternative_else_nop_endif
@@ -252,7 +252,7 @@ alternative_else_nop_endif
cpym [dst]!, [src]!, count!
cpye [dst]!, [src]!, count!
ret
-SYM_FUNC_END(__pi_memcpy)
+SYM_LEAF_FUNC_END(__pi_memcpy)
#else
SYM_FUNC_ALIAS(__pi_memcpy, __pi_memcpy_generic)
#endif
diff --git a/arch/arm64/lib/memset.S b/arch/arm64/lib/memset.S
index 97157da65ec6..8ee307f5891b 100644
--- a/arch/arm64/lib/memset.S
+++ b/arch/arm64/lib/memset.S
@@ -43,7 +43,7 @@ dst .req x8
tmp3w .req w9
tmp3 .req x9
-SYM_FUNC_START_LOCAL(__pi_memset_generic)
+SYM_LEAF_FUNC_START_LOCAL(__pi_memset_generic)
mov dst, dstin /* Preserve return value. */
and A_lw, val, #255
orr A_lw, A_lw, A_lw, lsl #8
@@ -202,11 +202,11 @@ SYM_FUNC_START_LOCAL(__pi_memset_generic)
ands count, count, zva_bits_x
b.ne .Ltail_maybe_long
ret
-SYM_FUNC_END(__pi_memset_generic)
+SYM_LEAF_FUNC_END(__pi_memset_generic)
#ifdef CONFIG_AS_HAS_MOPS
.arch_extension mops
-SYM_FUNC_START(__pi_memset)
+SYM_LEAF_FUNC_START(__pi_memset)
alternative_if_not ARM64_HAS_MOPS
b __pi_memset_generic
alternative_else_nop_endif
@@ -216,7 +216,7 @@ alternative_else_nop_endif
setm [dst]!, count!, val_x
sete [dst]!, count!, val_x
ret
-SYM_FUNC_END(__pi_memset)
+SYM_LEAF_FUNC_END(__pi_memset)
#else
SYM_FUNC_ALIAS(__pi_memset, __pi_memset_generic)
#endif
diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S
index 5018ac03b6bf..442202cd02a3 100644
--- a/arch/arm64/lib/mte.S
+++ b/arch/arm64/lib/mte.S
@@ -27,14 +27,14 @@
* Clear the tags in a page
* x0 - address of the page to be cleared
*/
-SYM_FUNC_START(mte_clear_page_tags)
+SYM_LEAF_FUNC_START(mte_clear_page_tags)
multitag_transfer_size x1, x2
1: stgm xzr, [x0]
add x0, x0, x1
tst x0, #(PAGE_SIZE - 1)
b.ne 1b
ret
-SYM_FUNC_END(mte_clear_page_tags)
+SYM_LEAF_FUNC_END(mte_clear_page_tags)
/*
* Zero the page and tags at the same time
@@ -42,7 +42,7 @@ SYM_FUNC_END(mte_clear_page_tags)
* Parameters:
* x0 - address to the beginning of the page
*/
-SYM_FUNC_START(mte_zero_clear_page_tags)
+SYM_LEAF_FUNC_START(mte_zero_clear_page_tags)
and x0, x0, #(1 << MTE_TAG_SHIFT) - 1 // clear the tag
mrs x1, dczid_el0
tbnz x1, #4, 2f // Branch if DC GZVA is prohibited
@@ -60,14 +60,14 @@ SYM_FUNC_START(mte_zero_clear_page_tags)
tst x0, #(PAGE_SIZE - 1)
b.ne 2b
ret
-SYM_FUNC_END(mte_zero_clear_page_tags)
+SYM_LEAF_FUNC_END(mte_zero_clear_page_tags)
/*
* Copy the tags from the source page to the destination one
* x0 - address of the destination page
* x1 - address of the source page
*/
-SYM_FUNC_START(mte_copy_page_tags)
+SYM_LEAF_FUNC_START(mte_copy_page_tags)
mov x2, x0
mov x3, x1
multitag_transfer_size x5, x6
@@ -78,7 +78,7 @@ SYM_FUNC_START(mte_copy_page_tags)
tst x2, #(PAGE_SIZE - 1)
b.ne 1b
ret
-SYM_FUNC_END(mte_copy_page_tags)
+SYM_LEAF_FUNC_END(mte_copy_page_tags)
/*
* Read tags from a user buffer (one tag per byte) and set the corresponding
@@ -89,7 +89,7 @@ SYM_FUNC_END(mte_copy_page_tags)
* Returns:
* x0 - number of tags read/set
*/
-SYM_FUNC_START(mte_copy_tags_from_user)
+SYM_LEAF_FUNC_START(mte_copy_tags_from_user)
mov x3, x1
cbz x2, 2f
1:
@@ -103,7 +103,7 @@ USER(2f, ldtrb w4, [x1])
// exception handling and function return
2: sub x0, x1, x3 // update the number of tags set
ret
-SYM_FUNC_END(mte_copy_tags_from_user)
+SYM_LEAF_FUNC_END(mte_copy_tags_from_user)
/*
* Get the tags from a kernel address range and write the tag values to the
@@ -114,7 +114,7 @@ SYM_FUNC_END(mte_copy_tags_from_user)
* Returns:
* x0 - number of tags read/set
*/
-SYM_FUNC_START(mte_copy_tags_to_user)
+SYM_LEAF_FUNC_START(mte_copy_tags_to_user)
mov x3, x0
cbz x2, 2f
1:
@@ -129,14 +129,14 @@ USER(2f, sttrb w4, [x0])
// exception handling and function return
2: sub x0, x0, x3 // update the number of tags copied
ret
-SYM_FUNC_END(mte_copy_tags_to_user)
+SYM_LEAF_FUNC_END(mte_copy_tags_to_user)
/*
* Save the tags in a page
* x0 - page address
* x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes
*/
-SYM_FUNC_START(mte_save_page_tags)
+SYM_LEAF_FUNC_START(mte_save_page_tags)
multitag_transfer_size x7, x5
1:
mov x2, #0
@@ -153,14 +153,14 @@ SYM_FUNC_START(mte_save_page_tags)
b.ne 1b
ret
-SYM_FUNC_END(mte_save_page_tags)
+SYM_LEAF_FUNC_END(mte_save_page_tags)
/*
* Restore the tags in a page
* x0 - page address
* x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes
*/
-SYM_FUNC_START(mte_restore_page_tags)
+SYM_LEAF_FUNC_START(mte_restore_page_tags)
multitag_transfer_size x7, x5
1:
ldr x2, [x1], #8
@@ -174,4 +174,4 @@ SYM_FUNC_START(mte_restore_page_tags)
b.ne 1b
ret
-SYM_FUNC_END(mte_restore_page_tags)
+SYM_LEAF_FUNC_END(mte_restore_page_tags)
diff --git a/arch/arm64/lib/strchr.S b/arch/arm64/lib/strchr.S
index 94ee67a6b212..455582efd07a 100644
--- a/arch/arm64/lib/strchr.S
+++ b/arch/arm64/lib/strchr.S
@@ -18,7 +18,7 @@
* Returns:
* x0 - address of first occurrence of 'c' or 0
*/
-SYM_FUNC_START(__pi_strchr)
+SYM_LEAF_FUNC_START(__pi_strchr)
and w1, w1, #0xff
1: ldrb w2, [x0], #1
cmp w2, w1
@@ -28,7 +28,7 @@ SYM_FUNC_START(__pi_strchr)
cmp w2, w1
csel x0, x0, xzr, eq
ret
-SYM_FUNC_END(__pi_strchr)
+SYM_LEAF_FUNC_END(__pi_strchr)
SYM_FUNC_ALIAS_WEAK(strchr, __pi_strchr)
EXPORT_SYMBOL_NOKASAN(strchr)
diff --git a/arch/arm64/lib/strcmp.S b/arch/arm64/lib/strcmp.S
index 9b89b4533607..d0ce2040a32b 100644
--- a/arch/arm64/lib/strcmp.S
+++ b/arch/arm64/lib/strcmp.S
@@ -53,7 +53,7 @@
NUL too in big-endian, byte-reverse the data before the NUL check. */
-SYM_FUNC_START(__pi_strcmp)
+SYM_LEAF_FUNC_START(__pi_strcmp)
sub off2, src2, src1
mov zeroones, REP8_01
and tmp, src1, 7
@@ -185,6 +185,6 @@ L(tail):
L(done):
sub result, data1, data2
ret
-SYM_FUNC_END(__pi_strcmp)
+SYM_LEAF_FUNC_END(__pi_strcmp)
SYM_FUNC_ALIAS_WEAK(strcmp, __pi_strcmp)
EXPORT_SYMBOL_NOKASAN(strcmp)
diff --git a/arch/arm64/lib/strlen.S b/arch/arm64/lib/strlen.S
index 4919fe81ae54..a5d4151548b5 100644
--- a/arch/arm64/lib/strlen.S
+++ b/arch/arm64/lib/strlen.S
@@ -79,7 +79,7 @@
whether the first fetch, which may be misaligned, crosses a page
boundary. */
-SYM_FUNC_START(__pi_strlen)
+SYM_LEAF_FUNC_START(__pi_strlen)
and tmp1, srcin, MIN_PAGE_SIZE - 1
mov zeroones, REP8_01
cmp tmp1, MIN_PAGE_SIZE - 16
@@ -208,6 +208,6 @@ L(page_cross):
csel data1, data1, tmp4, eq
csel data2, data2, tmp2, eq
b L(page_cross_entry)
-SYM_FUNC_END(__pi_strlen)
+SYM_LEAF_FUNC_END(__pi_strlen)
SYM_FUNC_ALIAS_WEAK(strlen, __pi_strlen)
EXPORT_SYMBOL_NOKASAN(strlen)
diff --git a/arch/arm64/lib/strncmp.S b/arch/arm64/lib/strncmp.S
index fe7bbc0b42a7..8fd5c5d7dc2a 100644
--- a/arch/arm64/lib/strncmp.S
+++ b/arch/arm64/lib/strncmp.S
@@ -58,7 +58,7 @@
#define LS_BK lsl
#endif
-SYM_FUNC_START(__pi_strncmp)
+SYM_LEAF_FUNC_START(__pi_strncmp)
cbz limit, L(ret0)
eor tmp1, src1, src2
mov zeroones, #REP8_01
@@ -305,6 +305,6 @@ L(syndrome_check):
L(ret0):
mov result, #0
ret
-SYM_FUNC_END(__pi_strncmp)
+SYM_LEAF_FUNC_END(__pi_strncmp)
SYM_FUNC_ALIAS_WEAK(strncmp, __pi_strncmp)
EXPORT_SYMBOL_NOKASAN(strncmp)
diff --git a/arch/arm64/lib/strnlen.S b/arch/arm64/lib/strnlen.S
index d5ac0e10a01d..9f3f02e3f7e2 100644
--- a/arch/arm64/lib/strnlen.S
+++ b/arch/arm64/lib/strnlen.S
@@ -47,7 +47,7 @@ limit_wd .req x14
#define REP8_7f 0x7f7f7f7f7f7f7f7f
#define REP8_80 0x8080808080808080
-SYM_FUNC_START(__pi_strnlen)
+SYM_LEAF_FUNC_START(__pi_strnlen)
cbz limit, .Lhit_limit
mov zeroones, #REP8_01
bic src, srcin, #15
@@ -156,7 +156,7 @@ CPU_LE( lsr tmp2, tmp2, tmp4 ) /* Shift (tmp1 & 63). */
.Lhit_limit:
mov len, limit
ret
-SYM_FUNC_END(__pi_strnlen)
+SYM_LEAF_FUNC_END(__pi_strnlen)
SYM_FUNC_ALIAS_WEAK(strnlen, __pi_strnlen)
EXPORT_SYMBOL_NOKASAN(strnlen)
diff --git a/arch/arm64/lib/tishift.S b/arch/arm64/lib/tishift.S
index a88613834fb0..b12d7f6a6003 100644
--- a/arch/arm64/lib/tishift.S
+++ b/arch/arm64/lib/tishift.S
@@ -7,7 +7,7 @@
#include <asm/assembler.h>
-SYM_FUNC_START(__ashlti3)
+SYM_LEAF_FUNC_START(__ashlti3)
cbz x2, 1f
mov x3, #64
sub x3, x3, x2
@@ -26,10 +26,10 @@ SYM_FUNC_START(__ashlti3)
lsl x1, x0, x1
mov x0, x2
ret
-SYM_FUNC_END(__ashlti3)
+SYM_LEAF_FUNC_END(__ashlti3)
EXPORT_SYMBOL(__ashlti3)
-SYM_FUNC_START(__ashrti3)
+SYM_LEAF_FUNC_START(__ashrti3)
cbz x2, 1f
mov x3, #64
sub x3, x3, x2
@@ -48,10 +48,10 @@ SYM_FUNC_START(__ashrti3)
asr x0, x1, x0
mov x1, x2
ret
-SYM_FUNC_END(__ashrti3)
+SYM_LEAF_FUNC_END(__ashrti3)
EXPORT_SYMBOL(__ashrti3)
-SYM_FUNC_START(__lshrti3)
+SYM_LEAF_FUNC_START(__lshrti3)
cbz x2, 1f
mov x3, #64
sub x3, x3, x2
@@ -70,5 +70,5 @@ SYM_FUNC_START(__lshrti3)
lsr x0, x1, x0
mov x1, x2
ret
-SYM_FUNC_END(__lshrti3)
+SYM_LEAF_FUNC_END(__lshrti3)
EXPORT_SYMBOL(__lshrti3)
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 3/9] arm64: entry: add unwind info for call_on_irq_stack()
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
From: Weinan Liu <wnliu@google.com>
DWARF CFI (Call Frame Information) specifies how to recover the return
address and callee-saved registers at each PC in a given function.
Compilers are able to generate the CFI annotations when they compile
the code to assembly language. For handcrafted assembly, we need to
annotate them by hand.
Frame pointers alone are usually sufficient to recover stack frames
(without CFI), except at the exception boundary, where more information
is needed to determine if the LR is live.
Since an exception can be taken from call_on_irq_stack(), annotate it
with CFI. The actual entry assembly functions are left untouched, since
they are not expected to take exceptions themselves.
Signed-off-by: Weinan Liu <wnliu@google.com>
Suggested-by: Jens Remus <jremus@linux.ibm.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
arch/arm64/kernel/entry.S | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index e0db14e9c843..5f4172ba4274 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -30,6 +30,12 @@
#include <asm/asm-uaccess.h>
#include <asm/unistd.h>
+/*
+ * Do not generate .eh_frame. Only generate .debug_frame and optionally
+ * .sframe (via assembler option --gsframe[-N]).
+ */
+ .cfi_sections .debug_frame
+
.macro clear_gp_regs
.irp n,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29
mov x\n, xzr
@@ -870,6 +876,7 @@ NOKPROBE(ret_from_fork)
* Calls func(regs) using this CPU's irq stack and shadow irq stack.
*/
SYM_FUNC_START(call_on_irq_stack)
+ .cfi_startproc
save_and_disable_daif x9
#ifdef CONFIG_SHADOW_CALL_STACK
get_current_task x16
@@ -880,6 +887,9 @@ SYM_FUNC_START(call_on_irq_stack)
/* Create a frame record to save our LR and SP (implicit in FP) */
stp x29, x30, [sp, #-16]!
mov x29, sp
+ .cfi_def_cfa 29, 16
+ .cfi_offset 29, -16
+ .cfi_offset 30, -8
ldr_this_cpu x16, irq_stack_ptr, x17
@@ -895,9 +905,13 @@ SYM_FUNC_START(call_on_irq_stack)
*/
mov sp, x29
ldp x29, x30, [sp], #16
+ .cfi_restore 29
+ .cfi_restore 30
+ .cfi_def_cfa 31, 0
scs_load_current
restore_irq x9
ret
+ .cfi_endproc
SYM_FUNC_END(call_on_irq_stack)
NOKPROBE(call_on_irq_stack)
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 2/9] arm64, unwind: build kernel with sframe V3 info
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
Build with -Wa,--gsframe-3 flags to generate a .sframe section. This
will be used for in-kernel reliable stacktrace in cases where the frame
pointer alone is insufficient.
Currently, the sframe format only supports arm64, x86_64 and s390x
architectures.
Signed-off-by: Weinan Liu <wnliu@google.com>
Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
MAINTAINERS | 2 +-
Makefile | 8 ++++++++
arch/Kconfig | 21 +++++++++++++++++++++
arch/arm64/Kconfig | 1 +
arch/arm64/include/asm/unwind_sframe.h | 8 ++++++++
arch/arm64/kernel/vdso/Makefile | 2 +-
include/asm-generic/sections.h | 4 ++++
include/asm-generic/vmlinux.lds.h | 15 +++++++++++++++
8 files changed, 59 insertions(+), 2 deletions(-)
create mode 100644 arch/arm64/include/asm/unwind_sframe.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 54613c683fdb..046d06dcdb86 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -27862,8 +27862,8 @@ STACK UNWINDING
M: Josh Poimboeuf <jpoimboe@kernel.org>
M: Steven Rostedt <rostedt@goodmis.org>
S: Maintained
+F: arch/*/include/asm/unwind_sframe.h
F: arch/*/include/asm/unwind_user.h
-F: arch/*/include/asm/unwind_user_sframe.h
F: include/asm-generic/unwind_user.h
F: include/linux/sframe.h
F: include/linux/unwind*.h
diff --git a/Makefile b/Makefile
index 9f88dcaae382..227fda16deb1 100644
--- a/Makefile
+++ b/Makefile
@@ -1147,6 +1147,14 @@ endif
# Ensure compilers do not transform certain loops into calls to wcslen()
KBUILD_CFLAGS += -fno-builtin-wcslen
+# build with sframe table
+ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+CC_FLAGS_SFRAME := -Wa,--gsframe-3
+KBUILD_CFLAGS += $(CC_FLAGS_SFRAME)
+KBUILD_AFLAGS += $(CC_FLAGS_SFRAME)
+export CC_FLAGS_SFRAME
+endif
+
# change __FILE__ to the relative path to the source directory
ifdef building_out_of_srctree
KBUILD_CPPFLAGS += -fmacro-prefix-map=$(srcroot)/=
diff --git a/arch/Kconfig b/arch/Kconfig
index 6eeafd86347b..f931b5848593 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -520,6 +520,27 @@ config SFRAME_VALIDATION
If unsure, say N.
+config ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME
+ bool
+ help
+ An architecture can select this if it enables the SFrame (Simple
+ Frame) unwinder for unwinding kernel stack traces. It uses an unwind
+ table that is directly generated by the toolchain based on DWARF CFI
+ information.
+
+config HAVE_UNWIND_KERNEL_SFRAME
+ bool "Sframe unwinder"
+ depends on AS_SFRAME3
+ depends on 64BIT
+ depends on ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME
+ select UNWIND_SFRAME_LOOKUP
+ help
+ This option enables the SFrame (Simple Frame) unwinder for unwinding
+ kernel stack traces. It uses unwind an table that is directly
+ generated by the toolchain based on DWARF CFI information. In
+ practice, this can provide more reliable stacktrace results than
+ unwinding with frame pointers alone.
+
config HAVE_PERF_REGS
bool
help
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index fe60738e5943..c3ef478469e5 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -86,6 +86,7 @@ config ARM64
select ARCH_SUPPORTS_SCHED_SMT
select ARCH_SUPPORTS_SCHED_CLUSTER
select ARCH_SUPPORTS_SCHED_MC
+ select ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME
select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT
select ARCH_WANT_DEFAULT_BPF_JIT
diff --git a/arch/arm64/include/asm/unwind_sframe.h b/arch/arm64/include/asm/unwind_sframe.h
new file mode 100644
index 000000000000..876412881196
--- /dev/null
+++ b/arch/arm64/include/asm/unwind_sframe.h
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_ARM64_UNWIND_SFRAME_H
+#define _ASM_ARM64_UNWIND_SFRAME_H
+
+#define SFRAME_REG_SP 31
+#define SFRAME_REG_FP 29
+
+#endif /* _ASM_ARM64_UNWIND_SFRAME_H */
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index 7dec05dd33b7..c60ef921956f 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -38,7 +38,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO
CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) \
$(RANDSTRUCT_CFLAGS) $(KSTACK_ERASE_CFLAGS) \
$(GCC_PLUGINS_CFLAGS) \
- $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) \
+ $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CC_FLAGS_SFRAME) \
-Wmissing-prototypes -Wmissing-declarations
CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
diff --git a/include/asm-generic/sections.h b/include/asm-generic/sections.h
index 0755bc39b0d8..336d27011a58 100644
--- a/include/asm-generic/sections.h
+++ b/include/asm-generic/sections.h
@@ -31,6 +31,7 @@
* __irqentry_text_start, __irqentry_text_end
* __softirqentry_text_start, __softirqentry_text_end
* __start_opd, __end_opd
+ * __start_sframe, __end_sframe
*/
extern char _text[], _stext[], _etext[];
extern char _data[], _sdata[], _edata[];
@@ -53,6 +54,9 @@ extern char __ctors_start[], __ctors_end[];
/* Start and end of .opd section - used for function descriptors. */
extern char __start_opd[], __end_opd[];
+/* Start and end of .sframe section - used for stack unwinding. */
+extern char __start_sframe[], __end_sframe[];
+
/* Start and end of instrumentation protected text section */
extern char __noinstr_text_start[], __noinstr_text_end[];
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 60c8c22fd3e4..6aeed39097dd 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -491,6 +491,8 @@
*(.rodata1) \
} \
\
+ SFRAME \
+ \
/* PCI quirks */ \
.pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \
BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early, _pci_fixups_early, __start, __end) \
@@ -904,6 +906,19 @@
#define TRACEDATA
#endif
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+#define SFRAME \
+ /* sframe */ \
+ .sframe : AT(ADDR(.sframe) - LOAD_OFFSET) { \
+ __start_sframe = .; \
+ KEEP(*(.sframe)) \
+ KEEP(*(.init.sframe)) \
+ __end_sframe = .; \
+ }
+#else
+#define SFRAME
+#endif
+
#ifdef CONFIG_PRINTK_INDEX
#define PRINTK_INDEX \
.printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) { \
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 1/9] sframe: Allow kernelspace sframe sections
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
In-Reply-To: <20260519064950.493949-1-dylanbhatch@google.com>
Generalize the sframe lookup code to support kernelspace sections. This
is done by defining a SFRAME_LOOKUP option that can be activated
separate from HAVE_UNWIND_USER_SFRAME, as there will be other client to
this library than just userspace unwind.
Sframe section location is now tracked in a separate sec_type field to
determine whether user-access functions are necessary to read the sframe
data. Relevant type delarations are moved and renamed to reflect the
non-user sframe support.
Reviewed-by: Jens Remus <jremus@linux.ibm.com>
Signed-off-by: Dylan Hatch <dylanbhatch@google.com>
---
MAINTAINERS | 2 +-
arch/Kconfig | 4 +
.../{unwind_user_sframe.h => unwind_sframe.h} | 6 +-
arch/x86/include/asm/unwind_user.h | 12 +-
include/linux/sframe.h | 48 ++--
include/linux/unwind_types.h | 46 +++
include/linux/unwind_user_types.h | 41 ---
kernel/unwind/Makefile | 2 +-
kernel/unwind/sframe.c | 270 ++++++++++++------
kernel/unwind/user.c | 45 +--
10 files changed, 295 insertions(+), 181 deletions(-)
rename arch/x86/include/asm/{unwind_user_sframe.h => unwind_sframe.h} (50%)
create mode 100644 include/linux/unwind_types.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 6812f581d44b..54613c683fdb 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -27858,7 +27858,7 @@ F: Documentation/driver-api/uio-howto.rst
F: drivers/uio/
F: include/linux/uio_driver.h
-USERSPACE STACK UNWINDING
+STACK UNWINDING
M: Josh Poimboeuf <jpoimboe@kernel.org>
M: Steven Rostedt <rostedt@goodmis.org>
S: Maintained
diff --git a/arch/Kconfig b/arch/Kconfig
index 78dad97bf2a4..6eeafd86347b 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -486,6 +486,9 @@ config AS_SFRAME3
def_bool $(as-instr,.cfi_startproc\n.cfi_endproc,-Wa$(comma)--gsframe-3)
select AS_SFRAME
+config UNWIND_SFRAME_LOOKUP
+ bool
+
config UNWIND_USER
bool
@@ -496,6 +499,7 @@ config HAVE_UNWIND_USER_FP
config HAVE_UNWIND_USER_SFRAME
bool
select UNWIND_USER
+ select UNWIND_SFRAME_LOOKUP
config SFRAME_VALIDATION
bool "Enable .sframe section debugging"
diff --git a/arch/x86/include/asm/unwind_user_sframe.h b/arch/x86/include/asm/unwind_sframe.h
similarity index 50%
rename from arch/x86/include/asm/unwind_user_sframe.h
rename to arch/x86/include/asm/unwind_sframe.h
index d828ae1a4aac..44d42e6ffde4 100644
--- a/arch/x86/include/asm/unwind_user_sframe.h
+++ b/arch/x86/include/asm/unwind_sframe.h
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_X86_UNWIND_USER_SFRAME_H
-#define _ASM_X86_UNWIND_USER_SFRAME_H
+#ifndef _ASM_X86_UNWIND_SFRAME_H
+#define _ASM_X86_UNWIND_SFRAME_H
#ifdef CONFIG_X86_64
@@ -9,4 +9,4 @@
#endif
-#endif /* _ASM_X86_UNWIND_USER_SFRAME_H */
+#endif /* _ASM_X86_UNWIND_SFRAME_H */
diff --git a/arch/x86/include/asm/unwind_user.h b/arch/x86/include/asm/unwind_user.h
index b80f0ec0f7a7..1c7e31ca5d8e 100644
--- a/arch/x86/include/asm/unwind_user.h
+++ b/arch/x86/include/asm/unwind_user.h
@@ -54,30 +54,30 @@ static inline int unwind_user_get_reg(unsigned long *val, unsigned int regnum)
#define ARCH_INIT_USER_FP_FRAME(ws) \
.cfa = { \
- .rule = UNWIND_USER_CFA_RULE_FP_OFFSET,\
+ .rule = UNWIND_CFA_RULE_FP_OFFSET,\
.offset = 2*(ws), \
}, \
.ra = { \
- .rule = UNWIND_USER_RULE_CFA_OFFSET_DEREF,\
+ .rule = UNWIND_RULE_CFA_OFFSET_DEREF,\
.offset = -1*(ws), \
}, \
.fp = { \
- .rule = UNWIND_USER_RULE_CFA_OFFSET_DEREF,\
+ .rule = UNWIND_RULE_CFA_OFFSET_DEREF,\
.offset = -2*(ws), \
}, \
.outermost = false,
#define ARCH_INIT_USER_FP_ENTRY_FRAME(ws) \
.cfa = { \
- .rule = UNWIND_USER_CFA_RULE_SP_OFFSET,\
+ .rule = UNWIND_CFA_RULE_SP_OFFSET,\
.offset = 1*(ws), \
}, \
.ra = { \
- .rule = UNWIND_USER_RULE_CFA_OFFSET_DEREF,\
+ .rule = UNWIND_RULE_CFA_OFFSET_DEREF,\
.offset = -1*(ws), \
}, \
.fp = { \
- .rule = UNWIND_USER_RULE_RETAIN,\
+ .rule = UNWIND_RULE_RETAIN,\
}, \
.outermost = false,
diff --git a/include/linux/sframe.h b/include/linux/sframe.h
index b79c5ec09229..0cb2924367bc 100644
--- a/include/linux/sframe.h
+++ b/include/linux/sframe.h
@@ -3,37 +3,46 @@
#define _LINUX_SFRAME_H
#include <linux/mm_types.h>
+#include <linux/unwind_types.h>
#include <linux/srcu.h>
-#include <linux/unwind_user_types.h>
-#ifdef CONFIG_HAVE_UNWIND_USER_SFRAME
+#ifdef CONFIG_UNWIND_SFRAME_LOOKUP
+
+enum sframe_sec_type {
+ SFRAME_KERNEL,
+ SFRAME_USER,
+};
struct sframe_section {
- struct rcu_head rcu;
+ struct rcu_head rcu;
#ifdef CONFIG_DYNAMIC_DEBUG
- const char *filename;
+ const char *filename;
#endif
- unsigned long sframe_start;
- unsigned long sframe_end;
- unsigned long text_start;
- unsigned long text_end;
-
- unsigned long fdes_start;
- unsigned long fres_start;
- unsigned long fres_end;
- unsigned int num_fdes;
-
- signed char ra_off;
- signed char fp_off;
+ enum sframe_sec_type sec_type;
+ unsigned long sframe_start;
+ unsigned long sframe_end;
+ unsigned long text_start;
+ unsigned long text_end;
+
+ unsigned long fdes_start;
+ unsigned long fres_start;
+ unsigned long fres_end;
+ unsigned int num_fdes;
+
+ signed char ra_off;
+ signed char fp_off;
};
+#endif /* CONFIG_UNWIND_SFRAME_LOOKUP */
+
+#ifdef CONFIG_HAVE_UNWIND_USER_SFRAME
+
#define INIT_MM_SFRAME .sframe_mt = MTREE_INIT(sframe_mt, 0),
extern void sframe_free_mm(struct mm_struct *mm);
extern int sframe_add_section(unsigned long sframe_start, unsigned long sframe_end,
unsigned long text_start, unsigned long text_end);
extern int sframe_remove_section(unsigned long sframe_addr);
-extern int sframe_find(unsigned long ip, struct unwind_user_frame *frame);
static inline bool current_has_sframe(void)
{
@@ -42,6 +51,8 @@ static inline bool current_has_sframe(void)
return mm && !mtree_empty(&mm->sframe_mt);
}
+extern int sframe_find_user(unsigned long ip, struct unwind_frame *frame);
+
#else /* !CONFIG_HAVE_UNWIND_USER_SFRAME */
#define INIT_MM_SFRAME
@@ -52,9 +63,10 @@ static inline int sframe_add_section(unsigned long sframe_start, unsigned long s
return -ENOSYS;
}
static inline int sframe_remove_section(unsigned long sframe_addr) { return -ENOSYS; }
-static inline int sframe_find(unsigned long ip, struct unwind_user_frame *frame) { return -ENOSYS; }
static inline bool current_has_sframe(void) { return false; }
+static inline int sframe_find_user(unsigned long ip, struct unwind_frame *frame) { return -ENOSYS; }
+
#endif /* CONFIG_HAVE_UNWIND_USER_SFRAME */
#endif /* _LINUX_SFRAME_H */
diff --git a/include/linux/unwind_types.h b/include/linux/unwind_types.h
new file mode 100644
index 000000000000..08bcb0aa04aa
--- /dev/null
+++ b/include/linux/unwind_types.h
@@ -0,0 +1,46 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_UNWIND_TYPES_H
+#define _LINUX_UNWIND_TYPES_H
+
+#define UNWIND_RULE_DEREF BIT(31)
+
+enum unwind_cfa_rule {
+ UNWIND_CFA_RULE_SP_OFFSET, /* CFA = SP + offset */
+ UNWIND_CFA_RULE_FP_OFFSET, /* CFA = FP + offset */
+ UNWIND_CFA_RULE_REG_OFFSET, /* CFA = reg + offset */
+ /* DEREF variants */
+ UNWIND_CFA_RULE_REG_OFFSET_DEREF = /* CFA = *(reg + offset) */
+ UNWIND_CFA_RULE_REG_OFFSET | UNWIND_RULE_DEREF,
+};
+
+struct unwind_cfa_rule_data {
+ enum unwind_cfa_rule rule;
+ s32 offset;
+ unsigned int regnum;
+};
+
+enum unwind_rule {
+ UNWIND_RULE_RETAIN, /* entity = entity */
+ UNWIND_RULE_CFA_OFFSET, /* entity = CFA + offset */
+ UNWIND_RULE_REG_OFFSET, /* entity = register + offset */
+ /* DEREF variants */
+ UNWIND_RULE_CFA_OFFSET_DEREF = /* entity = *(CFA + offset) */
+ UNWIND_RULE_CFA_OFFSET | UNWIND_RULE_DEREF,
+ UNWIND_RULE_REG_OFFSET_DEREF = /* entity = *(register + offset) */
+ UNWIND_RULE_REG_OFFSET | UNWIND_RULE_DEREF,
+};
+
+struct unwind_rule_data {
+ enum unwind_rule rule;
+ s32 offset;
+ unsigned int regnum;
+};
+
+struct unwind_frame {
+ struct unwind_cfa_rule_data cfa;
+ struct unwind_rule_data ra;
+ struct unwind_rule_data fp;
+ bool outermost;
+};
+
+#endif /* _LINUX_UNWIND_TYPES_H */
diff --git a/include/linux/unwind_user_types.h b/include/linux/unwind_user_types.h
index 059e5c76f2f3..646e5fb774db 100644
--- a/include/linux/unwind_user_types.h
+++ b/include/linux/unwind_user_types.h
@@ -27,47 +27,6 @@ struct unwind_stacktrace {
unsigned long *entries;
};
-#define UNWIND_USER_RULE_DEREF BIT(31)
-
-enum unwind_user_cfa_rule {
- UNWIND_USER_CFA_RULE_SP_OFFSET, /* CFA = SP + offset */
- UNWIND_USER_CFA_RULE_FP_OFFSET, /* CFA = FP + offset */
- UNWIND_USER_CFA_RULE_REG_OFFSET, /* CFA = reg + offset */
- /* DEREF variants */
- UNWIND_USER_CFA_RULE_REG_OFFSET_DEREF = /* CFA = *(reg + offset) */
- UNWIND_USER_CFA_RULE_REG_OFFSET | UNWIND_USER_RULE_DEREF,
-};
-
-struct unwind_user_cfa_rule_data {
- enum unwind_user_cfa_rule rule;
- s32 offset;
- unsigned int regnum;
-};
-
-enum unwind_user_rule {
- UNWIND_USER_RULE_RETAIN, /* entity = entity */
- UNWIND_USER_RULE_CFA_OFFSET, /* entity = CFA + offset */
- UNWIND_USER_RULE_REG_OFFSET, /* entity = register + offset */
- /* DEREF variants */
- UNWIND_USER_RULE_CFA_OFFSET_DEREF = /* entity = *(CFA + offset) */
- UNWIND_USER_RULE_CFA_OFFSET | UNWIND_USER_RULE_DEREF,
- UNWIND_USER_RULE_REG_OFFSET_DEREF = /* entity = *(register + offset) */
- UNWIND_USER_RULE_REG_OFFSET | UNWIND_USER_RULE_DEREF,
-};
-
-struct unwind_user_rule_data {
- enum unwind_user_rule rule;
- s32 offset;
- unsigned int regnum;
-};
-
-struct unwind_user_frame {
- struct unwind_user_cfa_rule_data cfa;
- struct unwind_user_rule_data ra;
- struct unwind_user_rule_data fp;
- bool outermost;
-};
-
struct unwind_user_state {
unsigned long ip;
unsigned long sp;
diff --git a/kernel/unwind/Makefile b/kernel/unwind/Makefile
index 146038165865..c5f9f8124564 100644
--- a/kernel/unwind/Makefile
+++ b/kernel/unwind/Makefile
@@ -1,2 +1,2 @@
obj-$(CONFIG_UNWIND_USER) += user.o deferred.o
- obj-$(CONFIG_HAVE_UNWIND_USER_SFRAME) += sframe.o
+ obj-$(CONFIG_UNWIND_SFRAME_LOOKUP) += sframe.o
diff --git a/kernel/unwind/sframe.c b/kernel/unwind/sframe.c
index 5400f481b05d..a2ab9a3e07b4 100644
--- a/kernel/unwind/sframe.c
+++ b/kernel/unwind/sframe.c
@@ -13,8 +13,8 @@
#include <linux/string_helpers.h>
#include <linux/sframe.h>
#include <linux/syscalls.h>
-#include <asm/unwind_user_sframe.h>
-#include <linux/unwind_user_types.h>
+#include <linux/unwind_types.h>
+#include <asm/unwind_sframe.h>
#include <uapi/linux/stacktrace.h>
#include "sframe.h"
@@ -46,8 +46,6 @@ struct sframe_fre_internal {
unsigned char dw_size;
};
-DEFINE_STATIC_SRCU(sframe_srcu);
-
static __always_inline unsigned char fre_type_to_size(unsigned char fre_type)
{
if (fre_type > 2)
@@ -62,6 +60,77 @@ static __always_inline unsigned char dataword_size_enum_to_size(unsigned char da
return 1 << dataword_size;
}
+#ifdef CONFIG_HAVE_UNWIND_USER_SFRAME
+
+DEFINE_STATIC_SRCU(sframe_srcu);
+
+#define UNSAFE_USER_COPY(to, from, size, label) \
+ unsafe_copy_from_user(to, (void __user *)from, size, label)
+
+#define UNSAFE_USER_GET(to, from, type, label) \
+ unsafe_get_user(to, (type __user *)from, label)
+
+#else /* !CONFIG_HAVE_UNWIND_USER_SFRAME */
+
+#define UNSAFE_USER_COPY(to, from, size, label) do { \
+ (void)to; (void)from; (void)size; \
+ goto label; \
+} while (0)
+
+#define UNSAFE_USER_GET(to, from, type, label) do { \
+ (void)to; (void)from; \
+ goto label; \
+} while (0)
+
+#endif /* !CONFIG_HAVE_UNWIND_USER_SFRAME */
+
+#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
+
+#define KERNEL_COPY(to, from, size, label) memcpy(to, (void *)from, size)
+#define KERNEL_GET(to, from, type, label) ({ (to) = *(type *)(from); })
+
+#else /* !CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
+
+#define KERNEL_COPY(to, from, size, label) do { \
+ (void)(to); (void)(from); (void)size; \
+ goto label; \
+} while (0)
+
+#define KERNEL_GET(to, from, type, label) do { \
+ (void)(to); (void)(from); \
+ goto label; \
+} while (0)
+
+#endif /* !CONFIG_HAVE_UNWIND_KERNEL_SFRAME */
+
+#define DATA_COPY(sec, to, from, size, label) \
+({ \
+ switch (sec->sec_type) { \
+ case SFRAME_KERNEL: \
+ KERNEL_COPY(to, from, size, label); \
+ break; \
+ case SFRAME_USER: \
+ UNSAFE_USER_COPY(to, from, size, label); \
+ break; \
+ default: \
+ goto label; \
+ } \
+})
+
+#define DATA_GET(sec, to, from, type, label) \
+({ \
+ switch (sec->sec_type) { \
+ case SFRAME_KERNEL: \
+ KERNEL_GET(to, from, type, label); \
+ break; \
+ case SFRAME_USER: \
+ UNSAFE_USER_GET(to, from, type, label); \
+ break; \
+ default: \
+ goto label; \
+ } \
+})
+
static __always_inline int __read_fde(struct sframe_section *sec,
unsigned int fde_num,
struct sframe_fde_internal *fde)
@@ -71,8 +140,8 @@ static __always_inline int __read_fde(struct sframe_section *sec,
struct sframe_fda_v3 _fda;
fde_addr = sec->fdes_start + (fde_num * sizeof(struct sframe_fde_v3));
- unsafe_copy_from_user(&_fde, (void __user *)fde_addr,
- sizeof(struct sframe_fde_v3), Efault);
+ DATA_COPY(sec, &_fde, fde_addr,
+ sizeof(struct sframe_fde_v3), Efault);
func_addr = fde_addr + _fde.func_start_off;
if (func_addr < sec->text_start || func_addr >= sec->text_end)
@@ -81,8 +150,8 @@ static __always_inline int __read_fde(struct sframe_section *sec,
fda_addr = sec->fres_start + _fde.fres_off;
if (fda_addr + sizeof(struct sframe_fda_v3) > sec->fres_end)
return -EINVAL;
- unsafe_copy_from_user(&_fda, (void __user *)fda_addr,
- sizeof(struct sframe_fda_v3), Efault);
+ DATA_COPY(sec, &_fda, fda_addr,
+ sizeof(struct sframe_fda_v3), Efault);
fde->func_addr = func_addr;
fde->func_size = _fde.func_size;
@@ -104,21 +173,21 @@ static __always_inline int __find_fde(struct sframe_section *sec,
struct sframe_fde_internal *fde)
{
unsigned long func_addr_low = 0, func_addr_high = ULONG_MAX;
- struct sframe_fde_v3 __user *first, *low, *high, *found = NULL;
+ struct sframe_fde_v3 *first, *low, *high, *found = NULL;
int ret;
- first = (void __user *)sec->fdes_start;
+ first = (void *)sec->fdes_start;
low = first;
high = first + sec->num_fdes - 1;
while (low <= high) {
- struct sframe_fde_v3 __user *mid;
+ struct sframe_fde_v3 *mid;
s64 func_off;
unsigned long func_addr;
mid = low + ((high - low) / 2);
- unsafe_get_user(func_off, (s64 __user *)mid, Efault);
+ DATA_GET(sec, func_off, mid, s64, Efault);
func_addr = (unsigned long)mid + func_off;
if (ip >= func_addr) {
@@ -156,47 +225,47 @@ static __always_inline int __find_fde(struct sframe_section *sec,
return -EFAULT;
}
-#define ____UNSAFE_GET_USER_INC(to, from, type, label) \
+#define ____GET_INC(sec, to, from, type, label) \
({ \
type __to; \
- unsafe_get_user(__to, (type __user *)from, label); \
+ DATA_GET(sec, __to, from, type, label); \
from += sizeof(__to); \
to = __to; \
})
-#define __UNSAFE_GET_USER_INC(to, from, size, label, u_or_s) \
+#define __GET_INC(sec, to, from, size, label, u_or_s) \
({ \
switch (size) { \
case 1: \
- ____UNSAFE_GET_USER_INC(to, from, u_or_s##8, label); \
+ ____GET_INC(sec, to, from, u_or_s##8, label); \
break; \
case 2: \
- ____UNSAFE_GET_USER_INC(to, from, u_or_s##16, label); \
+ ____GET_INC(sec, to, from, u_or_s##16, label); \
break; \
case 4: \
- ____UNSAFE_GET_USER_INC(to, from, u_or_s##32, label); \
+ ____GET_INC(sec, to, from, u_or_s##32, label); \
break; \
default: \
return -EFAULT; \
} \
})
-#define UNSAFE_GET_USER_UNSIGNED_INC(to, from, size, label) \
- __UNSAFE_GET_USER_INC(to, from, size, label, u)
+#define GET_UNSIGNED_INC(sec, to, from, size, label) \
+ __GET_INC(sec, to, from, size, label, u)
-#define UNSAFE_GET_USER_SIGNED_INC(to, from, size, label) \
- __UNSAFE_GET_USER_INC(to, from, size, label, s)
+#define GET_SIGNED_INC(sec, to, from, size, label) \
+ __GET_INC(sec, to, from, size, label, s)
-#define UNSAFE_GET_USER_INC(to, from, size, label) \
- _Generic(to, \
- u8 : UNSAFE_GET_USER_UNSIGNED_INC(to, from, size, label), \
- u16 : UNSAFE_GET_USER_UNSIGNED_INC(to, from, size, label), \
- u32 : UNSAFE_GET_USER_UNSIGNED_INC(to, from, size, label), \
- u64 : UNSAFE_GET_USER_UNSIGNED_INC(to, from, size, label), \
- s8 : UNSAFE_GET_USER_SIGNED_INC(to, from, size, label), \
- s16 : UNSAFE_GET_USER_SIGNED_INC(to, from, size, label), \
- s32 : UNSAFE_GET_USER_SIGNED_INC(to, from, size, label), \
- s64 : UNSAFE_GET_USER_SIGNED_INC(to, from, size, label))
+#define GET_INC(sec, to, from, size, label) \
+ _Generic(to, \
+ u8 : GET_UNSIGNED_INC(sec, to, from, size, label), \
+ u16 : GET_UNSIGNED_INC(sec, to, from, size, label), \
+ u32 : GET_UNSIGNED_INC(sec, to, from, size, label), \
+ u64 : GET_UNSIGNED_INC(sec, to, from, size, label), \
+ s8 : GET_SIGNED_INC(sec, to, from, size, label), \
+ s16 : GET_SIGNED_INC(sec, to, from, size, label), \
+ s32 : GET_SIGNED_INC(sec, to, from, size, label), \
+ s64 : GET_SIGNED_INC(sec, to, from, size, label))
static __always_inline int
__read_default_fre_datawords(struct sframe_section *sec,
@@ -209,19 +278,19 @@ __read_default_fre_datawords(struct sframe_section *sec,
s32 cfa_off, ra_off, fp_off;
unsigned int cfa_regnum;
- UNSAFE_GET_USER_INC(cfa_off, cur, dataword_size, Efault);
+ GET_INC(sec, cfa_off, cur, dataword_size, Efault);
dataword_count--;
ra_off = sec->ra_off;
if (!ra_off && dataword_count) {
dataword_count--;
- UNSAFE_GET_USER_INC(ra_off, cur, dataword_size, Efault);
+ GET_INC(sec, ra_off, cur, dataword_size, Efault);
}
fp_off = sec->fp_off;
if (!fp_off && dataword_count) {
dataword_count--;
- UNSAFE_GET_USER_INC(fp_off, cur, dataword_size, Efault);
+ GET_INC(sec, fp_off, cur, dataword_size, Efault);
}
if (dataword_count)
@@ -257,17 +326,17 @@ __read_flex_fde_fre_datawords(struct sframe_section *sec,
if (dataword_count < 2)
return -EFAULT;
- UNSAFE_GET_USER_INC(cfa_ctl, cur, dataword_size, Efault);
- UNSAFE_GET_USER_INC(cfa_off, cur, dataword_size, Efault);
+ GET_INC(sec, cfa_ctl, cur, dataword_size, Efault);
+ GET_INC(sec, cfa_off, cur, dataword_size, Efault);
dataword_count -= 2;
ra_off = sec->ra_off;
ra_ctl = ra_off ? 2 : 0; /* regnum=0, deref_p=(ra_off != 0), reg_p=0 */
if (dataword_count >= 2) {
- UNSAFE_GET_USER_INC(ra_ctl, cur, dataword_size, Efault);
+ GET_INC(sec, ra_ctl, cur, dataword_size, Efault);
dataword_count--;
if (ra_ctl) {
- UNSAFE_GET_USER_INC(ra_off, cur, dataword_size, Efault);
+ GET_INC(sec, ra_off, cur, dataword_size, Efault);
dataword_count--;
} else {
/* Padding RA location info */
@@ -278,10 +347,10 @@ __read_flex_fde_fre_datawords(struct sframe_section *sec,
fp_off = sec->fp_off;
fp_ctl = fp_off ? 2 : 0; /* regnum=0, deref_p=(fp_off != 0), reg_p=0 */
if (dataword_count >= 2) {
- UNSAFE_GET_USER_INC(fp_ctl, cur, dataword_size, Efault);
+ GET_INC(sec, fp_ctl, cur, dataword_size, Efault);
dataword_count--;
if (fp_ctl) {
- UNSAFE_GET_USER_INC(fp_off, cur, dataword_size, Efault);
+ GET_INC(sec, fp_off, cur, dataword_size, Efault);
dataword_count--;
} else {
/* Padding FP location info */
@@ -355,11 +424,11 @@ static __always_inline int __read_fre(struct sframe_section *sec,
if (fre_addr + addr_size + 1 > sec->fres_end)
return -EFAULT;
- UNSAFE_GET_USER_INC(ip_off, cur, addr_size, Efault);
+ GET_INC(sec, ip_off, cur, addr_size, Efault);
if (fde_pctype == SFRAME_FDE_PCTYPE_INC && ip_off > fde->func_size)
return -EFAULT;
- UNSAFE_GET_USER_INC(info, cur, 1, Efault);
+ GET_INC(sec, info, cur, 1, Efault);
dataword_count = SFRAME_V3_FRE_DATAWORD_COUNT(info);
dataword_size = dataword_size_enum_to_size(SFRAME_V3_FRE_DATAWORD_SIZE(info));
if (!dataword_size)
@@ -382,7 +451,7 @@ static __always_inline int __read_fre(struct sframe_section *sec,
}
static __always_inline int
-sframe_init_cfa_rule_data(struct unwind_user_cfa_rule_data *cfa_rule_data,
+sframe_init_cfa_rule_data(struct unwind_cfa_rule_data *cfa_rule_data,
u32 ctlword, s32 offset)
{
bool deref_p = SFRAME_V3_FLEX_FDE_CTRLWORD_DEREF_P(ctlword);
@@ -393,13 +462,13 @@ sframe_init_cfa_rule_data(struct unwind_user_cfa_rule_data *cfa_rule_data,
switch (regnum) {
case SFRAME_REG_SP:
- cfa_rule_data->rule = UNWIND_USER_CFA_RULE_SP_OFFSET;
+ cfa_rule_data->rule = UNWIND_CFA_RULE_SP_OFFSET;
break;
case SFRAME_REG_FP:
- cfa_rule_data->rule = UNWIND_USER_CFA_RULE_FP_OFFSET;
+ cfa_rule_data->rule = UNWIND_CFA_RULE_FP_OFFSET;
break;
default:
- cfa_rule_data->rule = UNWIND_USER_CFA_RULE_REG_OFFSET;
+ cfa_rule_data->rule = UNWIND_CFA_RULE_REG_OFFSET;
cfa_rule_data->regnum = regnum;
}
} else {
@@ -407,7 +476,7 @@ sframe_init_cfa_rule_data(struct unwind_user_cfa_rule_data *cfa_rule_data,
}
if (deref_p)
- cfa_rule_data->rule |= UNWIND_USER_RULE_DEREF;
+ cfa_rule_data->rule |= UNWIND_RULE_DEREF;
cfa_rule_data->offset = offset;
@@ -415,27 +484,27 @@ sframe_init_cfa_rule_data(struct unwind_user_cfa_rule_data *cfa_rule_data,
}
static __always_inline void
-sframe_init_rule_data(struct unwind_user_rule_data *rule_data,
+sframe_init_rule_data(struct unwind_rule_data *rule_data,
u32 ctlword, s32 offset)
{
bool deref_p = SFRAME_V3_FLEX_FDE_CTRLWORD_DEREF_P(ctlword);
bool reg_p = SFRAME_V3_FLEX_FDE_CTRLWORD_REG_P(ctlword);
if (!ctlword && !offset) {
- rule_data->rule = UNWIND_USER_RULE_RETAIN;
+ rule_data->rule = UNWIND_RULE_RETAIN;
return;
}
if (reg_p) {
unsigned int regnum = SFRAME_V3_FLEX_FDE_CTRLWORD_REGNUM(ctlword);
- rule_data->rule = UNWIND_USER_RULE_REG_OFFSET;
+ rule_data->rule = UNWIND_RULE_REG_OFFSET;
rule_data->regnum = regnum;
} else {
- rule_data->rule = UNWIND_USER_RULE_CFA_OFFSET;
+ rule_data->rule = UNWIND_RULE_CFA_OFFSET;
}
if (deref_p)
- rule_data->rule |= UNWIND_USER_RULE_DEREF;
+ rule_data->rule |= UNWIND_RULE_DEREF;
rule_data->offset = offset;
}
@@ -443,7 +512,7 @@ sframe_init_rule_data(struct unwind_user_rule_data *rule_data,
static __always_inline int __find_fre(struct sframe_section *sec,
struct sframe_fde_internal *fde,
unsigned long ip,
- struct unwind_user_frame *frame)
+ struct unwind_frame *frame)
{
unsigned char fde_pctype = SFRAME_V3_FDE_PCTYPE(fde->info);
struct sframe_fre_internal *fre, *prev_fre = NULL;
@@ -503,40 +572,18 @@ static __always_inline int __find_fre(struct sframe_section *sec,
return 0;
}
-int sframe_find(unsigned long ip, struct unwind_user_frame *frame)
+static __always_inline int __sframe_find(struct sframe_section *sec,
+ unsigned long ip,
+ struct unwind_frame *frame)
{
- struct mm_struct *mm = current->mm;
- struct sframe_section *sec;
struct sframe_fde_internal fde;
int ret;
- if (!mm)
- return -EINVAL;
-
- guard(srcu)(&sframe_srcu);
-
- sec = mtree_load(&mm->sframe_mt, ip);
- if (!sec)
- return -EINVAL;
-
- if (!user_read_access_begin((void __user *)sec->sframe_start,
- sec->sframe_end - sec->sframe_start))
- return -EFAULT;
-
ret = __find_fde(sec, ip, &fde);
if (ret)
- goto end;
-
- ret = __find_fre(sec, &fde, ip, frame);
-end:
- user_read_access_end();
-
- if (ret == -EFAULT) {
- dbg_sec("removing bad .sframe section\n");
- WARN_ON_ONCE(sframe_remove_section(sec->sframe_start));
- }
+ return ret;
- return ret;
+ return __find_fre(sec, &fde, ip, frame);
}
#ifdef CONFIG_SFRAME_VALIDATION
@@ -661,20 +708,23 @@ static int sframe_validate_section(struct sframe_section *sec) { return 0; }
#endif /* !CONFIG_SFRAME_VALIDATION */
-static void free_section(struct sframe_section *sec)
-{
- dbg_free(sec);
- kfree(sec);
-}
-
static int sframe_read_header(struct sframe_section *sec)
{
unsigned long header_end, fdes_start, fdes_end, fres_start, fres_end;
struct sframe_header shdr;
unsigned int num_fdes;
- if (copy_from_user(&shdr, (void __user *)sec->sframe_start, sizeof(shdr))) {
- dbg_sec("header usercopy failed\n");
+ switch (sec->sec_type) {
+ case SFRAME_USER:
+ if (copy_from_user(&shdr, (void __user *)sec->sframe_start, sizeof(shdr))) {
+ dbg_sec("header usercopy failed\n");
+ return -EFAULT;
+ }
+ break;
+ case SFRAME_KERNEL:
+ shdr = *(struct sframe_header *)sec->sframe_start;
+ break;
+ default:
return -EFAULT;
}
@@ -721,6 +771,45 @@ static int sframe_read_header(struct sframe_section *sec)
return 0;
}
+#ifdef CONFIG_HAVE_UNWIND_USER_SFRAME
+
+int sframe_find_user(unsigned long ip, struct unwind_frame *frame)
+{
+ struct mm_struct *mm = current->mm;
+ struct sframe_section *sec;
+ int ret;
+
+ if (!mm)
+ return -EINVAL;
+
+ guard(srcu)(&sframe_srcu);
+
+ sec = mtree_load(&mm->sframe_mt, ip);
+ if (!sec)
+ return -EINVAL;
+
+ if (!user_read_access_begin((void __user *)sec->sframe_start,
+ sec->sframe_end - sec->sframe_start))
+ return -EFAULT;
+
+ ret = __sframe_find(sec, ip, frame);
+
+ user_read_access_end();
+
+ if (ret == -EFAULT) {
+ dbg_sec("removing bad .sframe section\n");
+ WARN_ON_ONCE(sframe_remove_section(sec->sframe_start));
+ }
+
+ return ret;
+}
+
+static void free_section(struct sframe_section *sec)
+{
+ dbg_free(sec);
+ kfree(sec);
+}
+
int sframe_add_section(unsigned long sframe_start, unsigned long sframe_end,
unsigned long text_start, unsigned long text_end)
{
@@ -757,6 +846,7 @@ int sframe_add_section(unsigned long sframe_start, unsigned long sframe_end,
if (!sec)
return -ENOMEM;
+ sec->sec_type = SFRAME_USER;
sec->sframe_start = sframe_start;
sec->sframe_end = sframe_end;
sec->text_start = text_start;
@@ -877,3 +967,5 @@ SYSCALL_DEFINE5(stacktrace_setup, int, op, unsigned long, addr_start,
}
return -EINVAL;
}
+
+#endif /* CONFIG_HAVE_UNWIND_USER_SFRAME */
diff --git a/kernel/unwind/user.c b/kernel/unwind/user.c
index 3d596da588d0..5670579e3990 100644
--- a/kernel/unwind/user.c
+++ b/kernel/unwind/user.c
@@ -8,6 +8,7 @@
#include <linux/unwind_user.h>
#include <linux/uaccess.h>
#include <linux/sframe.h>
+#include <linux/unwind_types.h>
#define for_each_user_frame(state) \
for (unwind_user_start(state); !(state)->done; unwind_user_next(state))
@@ -28,7 +29,7 @@ get_user_word(unsigned long *word, unsigned long base, int off, unsigned int ws)
}
static int unwind_user_next_common(struct unwind_user_state *state,
- const struct unwind_user_frame *frame)
+ const struct unwind_frame *frame)
{
unsigned long cfa, fp, ra;
@@ -40,16 +41,16 @@ static int unwind_user_next_common(struct unwind_user_state *state,
/* Get the Canonical Frame Address (CFA) */
switch (frame->cfa.rule) {
- case UNWIND_USER_CFA_RULE_SP_OFFSET:
+ case UNWIND_CFA_RULE_SP_OFFSET:
cfa = state->sp;
break;
- case UNWIND_USER_CFA_RULE_FP_OFFSET:
+ case UNWIND_CFA_RULE_FP_OFFSET:
if (state->fp < state->sp)
return -EINVAL;
cfa = state->fp;
break;
- case UNWIND_USER_CFA_RULE_REG_OFFSET:
- case UNWIND_USER_CFA_RULE_REG_OFFSET_DEREF:
+ case UNWIND_CFA_RULE_REG_OFFSET:
+ case UNWIND_CFA_RULE_REG_OFFSET_DEREF:
if (!state->topmost || unwind_user_get_reg(&cfa, frame->cfa.regnum))
return -EINVAL;
break;
@@ -58,7 +59,7 @@ static int unwind_user_next_common(struct unwind_user_state *state,
return -EINVAL;
}
cfa += frame->cfa.offset;
- if (frame->cfa.rule & UNWIND_USER_RULE_DEREF &&
+ if (frame->cfa.rule & UNWIND_RULE_DEREF &&
get_user_word(&cfa, cfa, 0, state->ws))
return -EINVAL;
@@ -76,19 +77,19 @@ static int unwind_user_next_common(struct unwind_user_state *state,
/* Get the Return Address (RA) */
switch (frame->ra.rule) {
- case UNWIND_USER_RULE_RETAIN:
+ case UNWIND_RULE_RETAIN:
if (!state->topmost || unwind_user_get_ra_reg(&ra))
return -EINVAL;
break;
/*
- * UNWIND_USER_RULE_CFA_OFFSET doesn't make sense for RA.
+ * UNWIND_RULE_CFA_OFFSET doesn't make sense for RA.
* A return address cannot legitimately be a stack address.
*/
- case UNWIND_USER_RULE_CFA_OFFSET_DEREF:
+ case UNWIND_RULE_CFA_OFFSET_DEREF:
ra = cfa + frame->ra.offset;
break;
- case UNWIND_USER_RULE_REG_OFFSET:
- case UNWIND_USER_RULE_REG_OFFSET_DEREF:
+ case UNWIND_RULE_REG_OFFSET:
+ case UNWIND_RULE_REG_OFFSET_DEREF:
if (!state->topmost || unwind_user_get_reg(&ra, frame->ra.regnum))
return -EINVAL;
ra += frame->ra.offset;
@@ -97,24 +98,24 @@ static int unwind_user_next_common(struct unwind_user_state *state,
WARN_ON_ONCE(1);
return -EINVAL;
}
- if (frame->ra.rule & UNWIND_USER_RULE_DEREF &&
+ if (frame->ra.rule & UNWIND_RULE_DEREF &&
get_user_word(&ra, ra, 0, state->ws))
return -EINVAL;
/* Get the Frame Pointer (FP) */
switch (frame->fp.rule) {
- case UNWIND_USER_RULE_RETAIN:
+ case UNWIND_RULE_RETAIN:
fp = state->fp;
break;
/*
- * UNWIND_USER_RULE_CFA_OFFSET is currently not used for FP
+ * UNWIND_RULE_CFA_OFFSET is currently not used for FP
* (e.g. SFrame cannot represent this rule).
*/
- case UNWIND_USER_RULE_CFA_OFFSET_DEREF:
+ case UNWIND_RULE_CFA_OFFSET_DEREF:
fp = cfa + frame->fp.offset;
break;
- case UNWIND_USER_RULE_REG_OFFSET:
- case UNWIND_USER_RULE_REG_OFFSET_DEREF:
+ case UNWIND_RULE_REG_OFFSET:
+ case UNWIND_RULE_REG_OFFSET_DEREF:
if (!state->topmost || unwind_user_get_reg(&fp, frame->fp.regnum))
return -EINVAL;
fp += frame->fp.offset;
@@ -123,7 +124,7 @@ static int unwind_user_next_common(struct unwind_user_state *state,
WARN_ON_ONCE(1);
return -EINVAL;
}
- if (frame->fp.rule & UNWIND_USER_RULE_DEREF &&
+ if (frame->fp.rule & UNWIND_RULE_DEREF &&
get_user_word(&fp, fp, 0, state->ws))
return -EINVAL;
@@ -139,13 +140,13 @@ static int unwind_user_next_fp(struct unwind_user_state *state)
struct pt_regs *regs = task_pt_regs(current);
if (state->topmost && unwind_user_at_function_start(regs)) {
- const struct unwind_user_frame fp_entry_frame = {
+ const struct unwind_frame fp_entry_frame = {
ARCH_INIT_USER_FP_ENTRY_FRAME(state->ws)
};
return unwind_user_next_common(state, &fp_entry_frame);
}
- const struct unwind_user_frame fp_frame = {
+ const struct unwind_frame fp_frame = {
ARCH_INIT_USER_FP_FRAME(state->ws)
};
return unwind_user_next_common(state, &fp_frame);
@@ -153,10 +154,10 @@ static int unwind_user_next_fp(struct unwind_user_state *state)
static int unwind_user_next_sframe(struct unwind_user_state *state)
{
- struct unwind_user_frame frame;
+ struct unwind_frame frame;
/* sframe expects the frame to be local storage */
- if (sframe_find(state->ip, &frame))
+ if (sframe_find_user(state->ip, &frame))
return -ENOENT;
return unwind_user_next_common(state, &frame);
}
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply related
* [PATCH v6 0/9] unwind, arm64: add sframe unwinder for kernel
From: Dylan Hatch @ 2026-05-19 6:49 UTC (permalink / raw)
To: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Jens Remus
Cc: Dylan Hatch, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Mostafa Saleh, Herbert Xu,
David S. Miller
Implement a generic kernel sframe-based [1] unwinder. The main goal is
to improve reliable stacktrace on arm64 by unwinding across exception
boundaries.
On x86, the ORC unwinder provides reliable stacktrace through similar
methodology, but arm64 lacks the necessary support from objtool to
create ORC unwind tables.
Currently, there's already a sframe unwinder proposed for userspace: [2].
To maintain common definitions and algorithms for sframe lookup, a
substantial portion of this patch series aims to refactor the sframe
lookup code to support both kernel and userspace sframe sections.
Currently, only GNU Binutils support sframe. This series relies on the
Sframe V3 format, which is supported in binutils 2.46.
These patches are based on Steven Rostedt's sframe/core branch [3],
which is and aggregation of existing work done for x86 sframe userspace
unwind, and contains [2]. This branch is, in turn, based on Linux
v7.1-rc2. This full series (applied to the sframe/core branch) is
available on github: [4].
Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/all/20260505121718.3572346-1-jremus@linux.ibm.com/
[3]: https://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git/log/?h=sframe/core
[4]: https://github.com/dylanbhatch/linux/tree/sframe-v6
Changes since v5:
- Rebase on latest sframe/core branch [3] (based on v7.1-rc2).
- (Mark) Drop CFI annotations from el1*_64_* entry functions.
- (Mark) Add CFI annotations for leaf functions in lib/ and crypto/.
- (Jens) Sort module FDEs at load-time, drop linear search method.
- (Jens) Fix mistake in module SFrame validation where temp copy is not
yet embedded within a struct module.
- (Jens) Initialize debug info for kernel .sframe sections.
- (Mark) Move kernel-specific unwind fields to struct kunwind_state.
- (Mark) Drop SP from unwind state.
- (Mark) Rename unwind_next_frame_sframe -> kunwind_next_regs_sframe,
add checks to assert a correct KUNWIND_SOURCE_REGS_PC state.
- (Mark) Drop unused flexible FDE handling.
- (Mark) Check CFA alignment to 16 bytes instead of 8 bytes.
- (Mark) For non-KUNWIND_SOURCE_REGS_PC state, drop the fallback to
SFrame unwind if FP unwind fails in kunwind_next().
Dylan Hatch (8):
sframe: Allow kernelspace sframe sections
arm64, unwind: build kernel with sframe V3 info
arm64, crypto/lib: Annotate leaf functions with CFI info.
sframe: Provide PC lookup for vmlinux .sframe section
arm64/module, sframe: Add sframe support for modules
sframe: Introduce in-kernel SFRAME_VALIDATION
sframe: Initialize debug info for kernel sections
unwind: arm64: Use sframe to unwind interrupt frames
Weinan Liu (1):
arm64: entry: add unwind info for call_on_irq_stack()
MAINTAINERS | 4 +-
Makefile | 8 +
arch/Kconfig | 27 +-
arch/arm64/Kconfig | 1 +
arch/arm64/crypto/aes-ce-ccm-core.S | 12 +-
arch/arm64/crypto/aes-neonbs-core.S | 40 +-
arch/arm64/crypto/ghash-ce-core.S | 20 +-
arch/arm64/crypto/sm4-ce-ccm-core.S | 16 +-
arch/arm64/crypto/sm4-ce-cipher-core.S | 4 +-
arch/arm64/crypto/sm4-ce-core.S | 44 +-
arch/arm64/crypto/sm4-ce-gcm-core.S | 16 +-
arch/arm64/crypto/sm4-neon-core.S | 12 +-
arch/arm64/include/asm/linkage.h | 30 ++
arch/arm64/include/asm/module.h | 6 +
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/include/asm/unwind_sframe.h | 54 +++
arch/arm64/kernel/entry.S | 14 +
arch/arm64/kernel/module.c | 8 +
arch/arm64/kernel/setup.c | 2 +
arch/arm64/kernel/stacktrace.c | 222 +++++++++-
arch/arm64/kernel/vdso/Makefile | 2 +-
arch/arm64/kernel/vmlinux.lds.S | 2 +
arch/arm64/lib/clear_page.S | 4 +-
arch/arm64/lib/clear_user.S | 4 +-
arch/arm64/lib/copy_from_user.S | 4 +-
arch/arm64/lib/copy_page.S | 4 +-
arch/arm64/lib/copy_to_user.S | 4 +-
arch/arm64/lib/memchr.S | 4 +-
arch/arm64/lib/memcmp.S | 4 +-
arch/arm64/lib/memcpy.S | 8 +-
arch/arm64/lib/memset.S | 8 +-
arch/arm64/lib/mte.S | 28 +-
arch/arm64/lib/strchr.S | 4 +-
arch/arm64/lib/strcmp.S | 4 +-
arch/arm64/lib/strlen.S | 4 +-
arch/arm64/lib/strncmp.S | 4 +-
arch/arm64/lib/strnlen.S | 4 +-
arch/arm64/lib/tishift.S | 12 +-
.../{unwind_user_sframe.h => unwind_sframe.h} | 6 +-
arch/x86/include/asm/unwind_user.h | 12 +-
include/asm-generic/sections.h | 4 +
include/asm-generic/vmlinux.lds.h | 15 +
include/linux/sframe.h | 67 ++-
include/linux/unwind_types.h | 46 ++
include/linux/unwind_user_types.h | 41 --
kernel/unwind/Makefile | 2 +-
kernel/unwind/sframe.c | 419 ++++++++++++++----
kernel/unwind/sframe_debug.h | 13 +
kernel/unwind/user.c | 45 +-
49 files changed, 979 insertions(+), 340 deletions(-)
create mode 100644 arch/arm64/include/asm/unwind_sframe.h
rename arch/x86/include/asm/{unwind_user_sframe.h => unwind_sframe.h} (50%)
create mode 100644 include/linux/unwind_types.h
--
2.54.0.563.g4f69b47b94-goog
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Dylan Hatch @ 2026-05-19 6:29 UTC (permalink / raw)
To: Mark Rutland
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <afTYzAF_x41pyilu@J2N7QTR9R3>
Hi Mark,
I'm sending a v6 shortly that should address all/most of your
feedback, but I wanted to circle back on a question you had:
On Fri, May 1, 2026 at 9:46 AM Mark Rutland <mark.rutland@arm.com> wrote:
> > + /*
> > + * Consume RA and FP from the stack. The frame record puts FP at a lower
> > + * address than RA, so we always read FP first.
> > + */
> > + if (frame.fp.rule & UNWIND_RULE_DEREF &&
> > + !get_word(&state->common, &fp))
> > + return -EINVAL;
>
> Why is this get_word() rather than get_consume_word()?
I use get_word() here because get_consume_word(), in calling
unwind_consume_stack() under the hood, consumes the stack up to the
given address+size such that another unwind step cannot consume it
again. If the subsequent call to get_consume_word() fails, the stack
needs to be in a state such that we can fall back on a frame pointer
unwind. But if we were to use get_consume_word() here, the fallback
call to kunwind_next_frame_record() would not be able to consume the
FP from the stack because it would already have been consumed by the
failed call to unwind_next_frame_sframe().
By only calling get_consume_word() on the RA at the end, we defer
making any changes to the underlying unwind state stack until we are
sure the SFrame unwind step will succeed.
>
> > +
> > + if (frame.ra.rule & UNWIND_RULE_DEREF &&
> > + get_consume_word(&state->common, &ra))
> > + return -EINVAL;
> > +
> > + state->common.pc = ra;
> > + state->common.sp = cfa;
Please let me know if this reasoning seems sound.
Thanks,
Dylan
^ permalink raw reply
* Re: [PATCH v5 3/8] arm64: entry: add unwind info for various kernel entries
From: Dylan Hatch @ 2026-05-18 22:41 UTC (permalink / raw)
To: Mark Rutland
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <agbgMb6jrgiFFHRX@J2N7QTR9R3>
On Fri, May 15, 2026 at 1:58 AM Mark Rutland <mark.rutland@arm.com> wrote:
>
> On Thu, May 14, 2026 at 08:30:43PM -0700, Dylan Hatch wrote:
> > On Wed, Apr 29, 2026 at 8:26 AM Mark Rutland <mark.rutland@arm.com> wrote:
> > > On Tue, Apr 28, 2026 at 06:36:38PM +0000, Dylan Hatch wrote:
> > > > From: Weinan Liu <wnliu@google.com>
> > > >
> > > > DWARF CFI (Call Frame Information) specifies how to recover the return
> > > > address and callee-saved registers at each PC in a given function.
> > > > Compilers are able to generate the CFI annotations when they compile
> > > > the code to assembly language. For handcrafted assembly, we need to
> > > > annotate them by hand.
> > > >
> > > > Annotate minimal CFI to enable stacktracing using SFrame for kernel
> > > > exception entries through el1*_64_*() paths
> > >
> > > I thought we were only consuming SFrame when unwinding an exeption
> > > boundary?
> > >
> > > We shouldn't be taking exceptions _from_ the entry assembly functions
> > > unless something has gone horribly wrong, and so I don't see why we'd
> > > need CFI entries for the entry assembly functions.
> > >
> > > Am I missing some reason we need CFI entries for the entry assembly
> > > functions? I strongly suspect it is not necessary to add these, and I'd
> > > prefer to omit them.
> >
> > I believe the el1 entry functions are called in an exception, and are
> > called before call_on_irq_stack.
>
> Yes, but I don't think that matters. See below for more details.
>
> > Example stacktrace segment:
> >
> > [ 262.119564] handle_percpu_devid_irq+0xb4/0x348
> > [ 262.119913] handle_irq_desc+0x3c/0x68
> > [ 262.120196] generic_handle_domain_irq+0x20/0x40
> > [ 262.120678] gic_handle_irq+0x48/0xe0
> > [ 262.121005] call_on_irq_stack+0x30/0x48
> > [ 262.121412] do_interrupt_handler+0x88/0xa0
> > [ 262.121779] el1_interrupt+0x38/0x58
> > [ 262.122089] el1h_64_irq_handler+0x18/0x30
> > [ 262.122617] el1h_64_irq+0x6c/0x70
>
> The segment immediately above can be unwound using FP, as frame records
> were created consistently, and there are no exception boundaries. No CFI
> needed.
Ah right -- with the logic in stacktrace.c changed to use SFrame only
when recovering the next frame from a pt_regs, the FP alone is
sufficient if we know these entry functions won't take an exception.
This patch was originally implemented with an SFrame-only unwinder in
mind, so my mental model still hadn't back-propagated the new logic to
this patch :)
>
> It's legitimate to take an exception from parts of call_on_irq_stack(),
> so it makes sense for that to have CFI, but for the specific unwind
> segment above, CFI isn't necessary.
>
> Everything in the stacktrace segment above was executed *after* HW took
> the exception.
>
> << EXCEPTION BOUNDARY HERE >>
>
> Everything in the stacktrace segment(s) below was executed *before* HW
> took the exception.
>
> The unwinder knows that it has crossed this exception boundary by virtue
> of finding a FRAME_META_TYPE_PT_REGS frame record.
>
> > [ 262.123159] _raw_spin_unlock_irq+0x10/0x60 (P)
>
> The unwinder knows that the value of pt_regs::pc was *definitely* the PC
> at the time the exception was taken, so that entry is reliable. No CFI
> needed.
>
> > [ 262.123720] __filemap_add_folio+0x200/0x580 (L)
>
> The unwinder doesn't know whether the LR should be used, and needs CFI
> to determine that.
>
> After this point, an FP unwind can be used until we encounter the next
> exception boundary.
Right, and this is what is implemented in the final patch of this series.
>
> > [ 262.124145] filemap_add_folio+0xec/0x300
> > [ 262.124674] page_cache_ra_unbounded+0x128/0x368
> > [ 262.125338] do_page_cache_ra+0x70/0x98
> > [ 262.125875] page_cache_ra_order+0x460/0x4e0
>
> The segment immediately above can be unwound using FP. No CFI needed.
>
> > Here, el1h_64_irq is the last function that appears in the exception
> > stack before _raw_spin_unlock_irq and __filemap_add_folio are
> > recovered from the saved PC and LR, respectively. So we therefore need
> > the CFI annotations in order to unwind through the full exception
> > boundary.
> >
> > Is my interpretation here correct?
>
> Given you say "full exception boundary" here, I think we might be using
> the term "exception boundary" to mean different things.
>
> As per the example above, I'm using "exception boundary" to mean the a
> point between two entries in the stacktrace where HW took an exception.
>
> Did my comments on the example help?
I admit I may have been using the term "exception boundary" with a
vague definition, which was partly the source of my confusion. Thanks
for the example, it did help.
>
> > > > and irq entries through call_on_irq_stack()
> > >
> > > Needing some sort of unwind annotations for call_on_irq_stack() makes
> > > sense to me, but don't we need something for other assembly functions
> > > too?
> > >
> > > We can interrupt things like memset(); I assume we'll treat those as
> > > unreliable until annotated?
> >
> > While looking into adding these annotations, I noticed a pattern where
> > a sibling call is made to a local function:
> >
> > SYM_FUNC_START(__pi_memset)
> > alternative_if_not ARM64_HAS_MOPS
> > b __pi_memset_generic
> > alternative_else_nop_endif
> >
> > mov dst, dstin
> > setp [dst]!, count!, val_x
> > setm [dst]!, count!, val_x
> > sete [dst]!, count!, val_x
> > ret
> > SYM_FUNC_END(__pi_memset)
> >
> > In this case, do we consider the stacktrace unreliable since
> > __pi_memset may not appear in the trace?
>
> This is a tail-call, and __pi_memset_generic() will not return to
> __pi_memset(). Once the branch to __pi_memset_generic() has been
> executed, it's fine for __pi_memset() to not show up in the trace.
>
> The key thing is that no more instructions from __pi_memset() itself
> will be executed unless it was called again (from its entry point).
>
> > Or is this not important because assembly functions cannot be directly
> > livepatched anyway?
>
> To the best of my knowledge, reliable stacktrace is only used to
> determine whether any thread is still within an old version of a
> patchable function (including where it's within a callee thereof).
>
> I am not aware of a case where we'd need to detect whether a thread is
> still within a non-patchable function, but I can't rule out that as a
> possibility.
>
> That's more of a question for the livepatching maintainers.
>
> Thanks,
> Mark.
Thanks,
Dylan
^ permalink raw reply
* Re: [PATCH v5 0/8] unwind, arm64: add sframe unwinder for kernel
From: Dylan Hatch @ 2026-05-18 17:55 UTC (permalink / raw)
To: Mostafa Saleh
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Mark Rutland, Prasanna Kumar T S M,
Puranjay Mohan, Song Liu, joe.lawrence, linux-toolchains,
linux-kernel, live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <agcEMEl-QR0g6DgF@google.com>
Hi Mostafa,
On Fri, May 15, 2026 at 4:32 AM Mostafa Saleh <smostafa@google.com> wrote:
>
> On Tue, Apr 28, 2026 at 06:36:35PM +0000, Dylan Hatch wrote:
> > Implement a generic kernel sframe-based [1] unwinder. The main goal is
> > to improve reliable stacktrace on arm64 by unwinding across exception
> > boundaries.
> >
> > On x86, the ORC unwinder provides reliable stacktrace through similar
> > methodology, but arm64 lacks the necessary support from objtool to
> > create ORC unwind tables.
> >
> > Currently, there's already a sframe unwinder proposed for userspace: [2].
> > To maintain common definitions and algorithms for sframe lookup, a
> > substantial portion of this patch series aims to refactor the sframe
> > lookup code to support both kernel and userspace sframe sections.
> >
> > Currently, only GNU Binutils support sframe. This series relies on the
> > Sframe V3 format, which is supported in binutils 2.46.
> >
> > These patches are based on Steven Rostedt's sframe/core branch [3],
> > which is and aggregation of existing work done for x86 sframe userspace
> > unwind, and contains [2]. This branch is, in turn, based on Linux
> > v7.0-rc3. This full series (applied to the sframe/core branch) is
> > available on github: [4].
> >
>
> Not sure if related, but after updating my toolchain
> (aarch64-linux-gnu-gcc (Debian 15.2.0-4) 15.2.0), I hit link errors:
> ld.lld: error: arch/arm64/kernel/vdso/vgettimeofday.o:(.sframe) is being placed in '.sframe'
> ld.lld: error: arch/arm64/kernel/vdso/vgetrandom.o:(.sframe) is being placed in '.sframe`
Previously when developing against the SFrame V2 format, I had fixed
these warnings with the VDSO Makefile change currently in this series:
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index 7dec05dd33b7..c60ef921956f 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -38,7 +38,7 @@ ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO
CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) \
$(RANDSTRUCT_CFLAGS) $(KSTACK_ERASE_CFLAGS) \
$(GCC_PLUGINS_CFLAGS) \
- $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) \
+ $(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CC_FLAGS_SFRAME) \
-Wmissing-prototypes -Wmissing-declarations
CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
But the warnings seem to have returned after upgrading my toolchain,
possibly due to SFrame V3 or some confounding change in GCC. The
--gsframe in the assembler should be set to 'no' by default, so
perhaps GCC is providing an override --gsframe internally?
>
> I applied this series hoping that fix it, but it doesn't, so far I
> have this hack :
> diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
> index 52314be29191..53bdf757ee44 100644
> --- a/arch/arm64/kernel/vdso/vdso.lds.S
> +++ b/arch/arm64/kernel/vdso/vdso.lds.S
> @@ -77,7 +77,7 @@ SECTIONS
> /DISCARD/ : {
> *(.data .data.* .gnu.linkonce.d.* .sdata*)
> *(.bss .sbss .dynbss .dynsbss)
> - *(.eh_frame .eh_frame_hdr)
> + *(.eh_frame .eh_frame_hdr .sframe)
> }
> }
>
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 60c8c22fd3e4..759903acd6fc 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -1064,6 +1064,7 @@
> /* ld.bfd warns about .gnu.version* even when not emitted */ \
> *(.gnu.version*) \
> *(__tracepoint_check) \
> + *(.sframe) \
>
> #define DISCARDS \
> /DISCARD/ : { \
Since this series only handles kernel stacktrace, I believe it's
better to omit the .sframe section entirely in the case where only
ARCH_SUPPORTS_UNWIND_KERNEL_SFRAME is enabled. I think this hack may
work better for this purpose:
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index c60ef921956f..29f802bfedb1 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -41,7 +41,7 @@ CC_FLAGS_REMOVE_VDSO := $(CC_FLAGS_FTRACE) -Os
$(CC_FLAGS_SCS) \
$(CC_FLAGS_LTO) $(CC_FLAGS_CFI) $(CC_FLAGS_SFRAME) \
-Wmissing-prototypes -Wmissing-declarations
-CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
+CC_FLAGS_ADD_VDSO := -O2 -mcmodel=tiny -fasynchronous-unwind-tables
-Wa,--gsframe=no
CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_REMOVE_VDSO)
CFLAGS_REMOVE_vgetrandom.o = $(CC_FLAGS_REMOVE_VDSO)
Though, I don't understand why it is necessary to provide --gsframe=no
explicitly. If this approach seems ok to other folks/maintainers, I
can fold this into my series.
On the topic of SFrame for VDSO, Jens has a patch adding support for
this as part of a series to support userspace SFrame unwinding for
arm64:
https://lore.kernel.org/lkml/20260417150827.1183376-4-jremus@linux.ibm.com/
>
>
> Thanks,
> Mostafa
>
Thanks,
Dylan
^ permalink raw reply related
* Re: [PATCH v5 0/8] unwind, arm64: add sframe unwinder for kernel
From: Mostafa Saleh @ 2026-05-15 11:32 UTC (permalink / raw)
To: Dylan Hatch
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Mark Rutland, Prasanna Kumar T S M,
Puranjay Mohan, Song Liu, joe.lawrence, linux-toolchains,
linux-kernel, live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <20260428183643.3796063-1-dylanbhatch@google.com>
On Tue, Apr 28, 2026 at 06:36:35PM +0000, Dylan Hatch wrote:
> Implement a generic kernel sframe-based [1] unwinder. The main goal is
> to improve reliable stacktrace on arm64 by unwinding across exception
> boundaries.
>
> On x86, the ORC unwinder provides reliable stacktrace through similar
> methodology, but arm64 lacks the necessary support from objtool to
> create ORC unwind tables.
>
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> To maintain common definitions and algorithms for sframe lookup, a
> substantial portion of this patch series aims to refactor the sframe
> lookup code to support both kernel and userspace sframe sections.
>
> Currently, only GNU Binutils support sframe. This series relies on the
> Sframe V3 format, which is supported in binutils 2.46.
>
> These patches are based on Steven Rostedt's sframe/core branch [3],
> which is and aggregation of existing work done for x86 sframe userspace
> unwind, and contains [2]. This branch is, in turn, based on Linux
> v7.0-rc3. This full series (applied to the sframe/core branch) is
> available on github: [4].
>
Not sure if related, but after updating my toolchain
(aarch64-linux-gnu-gcc (Debian 15.2.0-4) 15.2.0), I hit link errors:
ld.lld: error: arch/arm64/kernel/vdso/vgettimeofday.o:(.sframe) is being placed in '.sframe'
ld.lld: error: arch/arm64/kernel/vdso/vgetrandom.o:(.sframe) is being placed in '.sframe`
I applied this series hoping that fix it, but it doesn't, so far I
have this hack :
diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
index 52314be29191..53bdf757ee44 100644
--- a/arch/arm64/kernel/vdso/vdso.lds.S
+++ b/arch/arm64/kernel/vdso/vdso.lds.S
@@ -77,7 +77,7 @@ SECTIONS
/DISCARD/ : {
*(.data .data.* .gnu.linkonce.d.* .sdata*)
*(.bss .sbss .dynbss .dynsbss)
- *(.eh_frame .eh_frame_hdr)
+ *(.eh_frame .eh_frame_hdr .sframe)
}
}
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 60c8c22fd3e4..759903acd6fc 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -1064,6 +1064,7 @@
/* ld.bfd warns about .gnu.version* even when not emitted */ \
*(.gnu.version*) \
*(__tracepoint_check) \
+ *(.sframe) \
#define DISCARDS \
/DISCARD/ : { \
Thanks,
Mostafa
^ permalink raw reply related
* Re: [PATCH v5 3/8] arm64: entry: add unwind info for various kernel entries
From: Mark Rutland @ 2026-05-15 8:58 UTC (permalink / raw)
To: Dylan Hatch
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <CADBMgpxBeYUdA5X8BPgkgz=KQyN=NQ760bXygwXfvVRScNzgbA@mail.gmail.com>
On Thu, May 14, 2026 at 08:30:43PM -0700, Dylan Hatch wrote:
> On Wed, Apr 29, 2026 at 8:26 AM Mark Rutland <mark.rutland@arm.com> wrote:
> > On Tue, Apr 28, 2026 at 06:36:38PM +0000, Dylan Hatch wrote:
> > > From: Weinan Liu <wnliu@google.com>
> > >
> > > DWARF CFI (Call Frame Information) specifies how to recover the return
> > > address and callee-saved registers at each PC in a given function.
> > > Compilers are able to generate the CFI annotations when they compile
> > > the code to assembly language. For handcrafted assembly, we need to
> > > annotate them by hand.
> > >
> > > Annotate minimal CFI to enable stacktracing using SFrame for kernel
> > > exception entries through el1*_64_*() paths
> >
> > I thought we were only consuming SFrame when unwinding an exeption
> > boundary?
> >
> > We shouldn't be taking exceptions _from_ the entry assembly functions
> > unless something has gone horribly wrong, and so I don't see why we'd
> > need CFI entries for the entry assembly functions.
> >
> > Am I missing some reason we need CFI entries for the entry assembly
> > functions? I strongly suspect it is not necessary to add these, and I'd
> > prefer to omit them.
>
> I believe the el1 entry functions are called in an exception, and are
> called before call_on_irq_stack.
Yes, but I don't think that matters. See below for more details.
> Example stacktrace segment:
>
> [ 262.119564] handle_percpu_devid_irq+0xb4/0x348
> [ 262.119913] handle_irq_desc+0x3c/0x68
> [ 262.120196] generic_handle_domain_irq+0x20/0x40
> [ 262.120678] gic_handle_irq+0x48/0xe0
> [ 262.121005] call_on_irq_stack+0x30/0x48
> [ 262.121412] do_interrupt_handler+0x88/0xa0
> [ 262.121779] el1_interrupt+0x38/0x58
> [ 262.122089] el1h_64_irq_handler+0x18/0x30
> [ 262.122617] el1h_64_irq+0x6c/0x70
The segment immediately above can be unwound using FP, as frame records
were created consistently, and there are no exception boundaries. No CFI
needed.
It's legitimate to take an exception from parts of call_on_irq_stack(),
so it makes sense for that to have CFI, but for the specific unwind
segment above, CFI isn't necessary.
Everything in the stacktrace segment above was executed *after* HW took
the exception.
<< EXCEPTION BOUNDARY HERE >>
Everything in the stacktrace segment(s) below was executed *before* HW
took the exception.
The unwinder knows that it has crossed this exception boundary by virtue
of finding a FRAME_META_TYPE_PT_REGS frame record.
> [ 262.123159] _raw_spin_unlock_irq+0x10/0x60 (P)
The unwinder knows that the value of pt_regs::pc was *definitely* the PC
at the time the exception was taken, so that entry is reliable. No CFI
needed.
> [ 262.123720] __filemap_add_folio+0x200/0x580 (L)
The unwinder doesn't know whether the LR should be used, and needs CFI
to determine that.
After this point, an FP unwind can be used until we encounter the next
exception boundary.
> [ 262.124145] filemap_add_folio+0xec/0x300
> [ 262.124674] page_cache_ra_unbounded+0x128/0x368
> [ 262.125338] do_page_cache_ra+0x70/0x98
> [ 262.125875] page_cache_ra_order+0x460/0x4e0
The segment immediately above can be unwound using FP. No CFI needed.
> Here, el1h_64_irq is the last function that appears in the exception
> stack before _raw_spin_unlock_irq and __filemap_add_folio are
> recovered from the saved PC and LR, respectively. So we therefore need
> the CFI annotations in order to unwind through the full exception
> boundary.
>
> Is my interpretation here correct?
Given you say "full exception boundary" here, I think we might be using
the term "exception boundary" to mean different things.
As per the example above, I'm using "exception boundary" to mean the a
point between two entries in the stacktrace where HW took an exception.
Did my comments on the example help?
> > > and irq entries through call_on_irq_stack()
> >
> > Needing some sort of unwind annotations for call_on_irq_stack() makes
> > sense to me, but don't we need something for other assembly functions
> > too?
> >
> > We can interrupt things like memset(); I assume we'll treat those as
> > unreliable until annotated?
>
> While looking into adding these annotations, I noticed a pattern where
> a sibling call is made to a local function:
>
> SYM_FUNC_START(__pi_memset)
> alternative_if_not ARM64_HAS_MOPS
> b __pi_memset_generic
> alternative_else_nop_endif
>
> mov dst, dstin
> setp [dst]!, count!, val_x
> setm [dst]!, count!, val_x
> sete [dst]!, count!, val_x
> ret
> SYM_FUNC_END(__pi_memset)
>
> In this case, do we consider the stacktrace unreliable since
> __pi_memset may not appear in the trace?
This is a tail-call, and __pi_memset_generic() will not return to
__pi_memset(). Once the branch to __pi_memset_generic() has been
executed, it's fine for __pi_memset() to not show up in the trace.
The key thing is that no more instructions from __pi_memset() itself
will be executed unless it was called again (from its entry point).
> Or is this not important because assembly functions cannot be directly
> livepatched anyway?
To the best of my knowledge, reliable stacktrace is only used to
determine whether any thread is still within an old version of a
patchable function (including where it's within a callee thereof).
I am not aware of a case where we'd need to detect whether a thread is
still within a non-patchable function, but I can't rule out that as a
possibility.
That's more of a question for the livepatching maintainers.
Thanks,
Mark.
^ permalink raw reply
* Re: [PATCH v5 3/8] arm64: entry: add unwind info for various kernel entries
From: Dylan Hatch @ 2026-05-15 3:30 UTC (permalink / raw)
To: Mark Rutland
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <afIjFLbUrdxWA6eR@J2N7QTR9R3.cambridge.arm.com>
On Wed, Apr 29, 2026 at 8:26 AM Mark Rutland <mark.rutland@arm.com> wrote:
>
> Hi Dylan,
>
> On Tue, Apr 28, 2026 at 06:36:38PM +0000, Dylan Hatch wrote:
> > From: Weinan Liu <wnliu@google.com>
> >
> > DWARF CFI (Call Frame Information) specifies how to recover the return
> > address and callee-saved registers at each PC in a given function.
> > Compilers are able to generate the CFI annotations when they compile
> > the code to assembly language. For handcrafted assembly, we need to
> > annotate them by hand.
> >
> > Annotate minimal CFI to enable stacktracing using SFrame for kernel
> > exception entries through el1*_64_*() paths
>
> I thought we were only consuming SFrame when unwinding an exeption
> boundary?
>
> We shouldn't be taking exceptions _from_ the entry assembly functions
> unless something has gone horribly wrong, and so I don't see why we'd
> need CFI entries for the entry assembly functions.
>
> Am I missing some reason we need CFI entries for the entry assembly
> functions? I strongly suspect it is not necessary to add these, and I'd
> prefer to omit them.
I believe the el1 entry functions are called in an exception, and are
called before call_on_irq_stack. Example stacktrace segment:
[ 262.119564] handle_percpu_devid_irq+0xb4/0x348
[ 262.119913] handle_irq_desc+0x3c/0x68
[ 262.120196] generic_handle_domain_irq+0x20/0x40
[ 262.120678] gic_handle_irq+0x48/0xe0
[ 262.121005] call_on_irq_stack+0x30/0x48
[ 262.121412] do_interrupt_handler+0x88/0xa0
[ 262.121779] el1_interrupt+0x38/0x58
[ 262.122089] el1h_64_irq_handler+0x18/0x30
[ 262.122617] el1h_64_irq+0x6c/0x70
[ 262.123159] _raw_spin_unlock_irq+0x10/0x60 (P)
[ 262.123720] __filemap_add_folio+0x200/0x580 (L)
[ 262.124145] filemap_add_folio+0xec/0x300
[ 262.124674] page_cache_ra_unbounded+0x128/0x368
[ 262.125338] do_page_cache_ra+0x70/0x98
[ 262.125875] page_cache_ra_order+0x460/0x4e0
Here, el1h_64_irq is the last function that appears in the exception
stack before _raw_spin_unlock_irq and __filemap_add_folio are
recovered from the saved PC and LR, respectively. So we therefore need
the CFI annotations in order to unwind through the full exception
boundary.
Is my interpretation here correct?
>
> > and irq entries through call_on_irq_stack()
>
> Needing some sort of unwind annotations for call_on_irq_stack() makes
> sense to me, but don't we need something for other assembly functions
> too?
>
> We can interrupt things like memset(); I assume we'll treat those as
> unreliable until annotated?
While looking into adding these annotations, I noticed a pattern where
a sibling call is made to a local function:
SYM_FUNC_START(__pi_memset)
alternative_if_not ARM64_HAS_MOPS
b __pi_memset_generic
alternative_else_nop_endif
mov dst, dstin
setp [dst]!, count!, val_x
setm [dst]!, count!, val_x
sete [dst]!, count!, val_x
ret
SYM_FUNC_END(__pi_memset)
In this case, do we consider the stacktrace unreliable since
__pi_memset may not appear in the trace? Or is this not important
because assembly functions cannot be directly livepatched anyway?
Thanks,
Dylan
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Mark Rutland @ 2026-05-12 10:18 UTC (permalink / raw)
To: Jens Remus
Cc: Dylan Hatch, Roman Gushchin, Weinan Liu, Will Deacon,
Josh Poimboeuf, Indu Bhagat, Peter Zijlstra, Steven Rostedt,
Catalin Marinas, Jiri Kosina, Prasanna Kumar T S M,
Puranjay Mohan, Song Liu, joe.lawrence, linux-toolchains,
linux-kernel, live-patching, linux-arm-kernel, Randy Dunlap,
Heiko Carstens
In-Reply-To: <0542f042-14fb-4588-bc3a-5031249d9834@linux.ibm.com>
On Tue, May 12, 2026 at 10:55:28AM +0200, Jens Remus wrote:
> On 5/12/2026 5:00 AM, Dylan Hatch wrote:
> > On Fri, May 1, 2026 at 9:46 AM Mark Rutland <mark.rutland@arm.com>
> > wrote:
>
> >> (1) For correctness, we'll need to address a latent issue with
> >> unwinding across an fgraph return trampoline, where the return
> >> address is transiently unrecoverable.
> >> I think we can solve that with some restructuring of that code,
> >> restoring the original address *before* removing that from the
> >> fgraph return stack, and ensuring that the unwinder can find it.
> >
> > If my understanding is correct, the issue arrises in
> > return_to_handler as the return address is recovered:
> >
> > mov x0, sp bl ftrace_return_to_handler // addr =
> > ftrace_return_to_hander(fregs); mov x30, x0 // restore the original
> > return address
> >
> > Because ftrace_return_to_handler pops the return address from the
> > return stack before it can be restored into the LR, it cannot be
> > recovered.
>
> Based on reliable-stacktrace.rst section "4.4 Rewriting of return
> addresses" I wonder whether the following might work:
>
> - If an unwound RA points at return_to_handler the actual RA needs to
> be obtained using ftrace_graph_ret_addr(). This might already be
> taken into account if ftrace_graph_ret_addr() is used unconditionally.
>
> - If an unwound RA points into return_to_handler() mark the stack trace
> as unreliable. This could be accomplished by marking LR in
> return_to_handler() as undefined (i.e. .cfi_undefined 30) to use
> SFrame's outermost frame indication to stop and mark the stack trace
> as unreliable:
We don't currently have any CFI annotations for return_to_handler(), so
if we interrupt that, any unwind will naturally be marked as unreliable.
The problem is that we can try an unwind from an interrupted *callee* of
return_to_handler(). In that case, we'll unwind through
return_to_handler() using the frame pointer, without consulting SFrame.
In that case, the PC will be part-way through return_to_handler(), but
we only call ftrace_graph_ret_addr() when the PC is the start of
return_to_handler, and so we don't even try to recover the return
address.
We can handle that better by checking whether the PC is *within*
return_to_handler(), and aborting when the original return address
cannot be recoverted. I'm happy to go put that together, nad longer term
I would like to do the better reovery I described above such that we can
*always* recover the return address.
Mark.
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Mark Rutland @ 2026-05-12 10:07 UTC (permalink / raw)
To: Dylan Hatch
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <CADBMgpx9YxNUO6wLP7mYKxWW8L78Hk9gPwHrMjXUwPyUmGEu9w@mail.gmail.com>
On Mon, May 11, 2026 at 08:00:21PM -0700, Dylan Hatch wrote:
> Hi Mark,
>
> Thanks for all the feedback and help on this. I'm planning on getting
> your comments addressed in the coming days, but I have some initial
> clarifying questions.
>
> On Fri, May 1, 2026 at 9:46 AM Mark Rutland <mark.rutland@arm.com> wrote:
> >
> > Hi Dylan,
> >
> > Thanks for putting this together. I think this is looking pretty good.
> > However, there are some things that aren't quite right and need some
> > work, which I've commented on below.
> >
> > More generally, there are a few things that aren't addressed by this
> > series that we will also need to address. Importantly:
> >
> > (1) For correctness, we'll need to address a latent issue with unwinding
> > across an fgraph return trampoline, where the return address is
> > transiently unrecoverable.
> >
> > Before this series, that doesn't matter for livepatching because the
> > livepatching code isn't called synchronously within the fgraph
> > handler, and unwinds which cross an exception boundary are marked as
> > unreliable.
> >
> > After this series, that does matter as we can unwind across an
> > exception boundary, and might happen to interrupt that transient
> > window.
> >
> > I think we can solve that with some restructuring of that code,
> > restoring the original address *before* removing that from the
> > fgraph return stack, and ensuring that the unwinder can find it.
>
> If my understanding is correct, the issue arrises in return_to_handler
> as the return address is recovered:
>
> mov x0, sp
> bl ftrace_return_to_handler // addr = ftrace_return_to_hander(fregs);
> mov x30, x0 // restore the original return address
>
> Because ftrace_return_to_handler pops the return address from the
> return stack before it can be restored into the LR, it cannot be
> recovered.
Yes.
To be clear, please don't worry about solving that for the next version
of this series; let's get the SFrame bits into shape first. I just
wanted to highlight that there's some more general work that we'll need
to do.
I think we can *detect* this case (and mark the unwind as unreliable)
with some tiny changes to the arm64 code. I'm happy to put that
together.
> Based on this, I believe you are suggesting to restructure this code
> path such that the return address is removed from the return stack
> only after it has been restored to LR. But since kernel/trace/fgraph.c
> is core kernel code, will this end up requiring either (1) a similar
> restructuring of other arches supporting ftrace, or (2) an
> arm64-specific implementation of this recovery logic?
Yes, I am say that to *recover* the address we'd need to make changes to
core code.
In the mean time we can *detect* this case with some minimal changes to
arm64 code, and abort. As above, I'm happy to go put that together.
> It looks to me like there is essentially the same recovery pattern on
> other arches; is there a reason this transient unrecoverability isn't
> an issue for reliable unwind on other platforms?
Yep; on all architectures there's a transient period where the address
cannot be recovered. It's not a correctness issue so long as the
architecture detects this case and marks the unwind as unreliable.
IIUC x86 will mark the unwind as unreliable in this case.
I don't know whether other architectures detect this reliably. That's a
question for loonarch, parisc, powerpc, and s390 folk.
> > I'm not immediately sure whether kretprobes has a similar issue.
> >
> > (2) To make unwinding generally possible, we'll need to annotate some
> > assembly functions as unwindable. We'll need to do that for string
> > routines under lib/, and probably some crypto code, but we don't
> > need to do that for most code in head.S, entry.S, etc.
> >
> > The vast majority of relevant assembly functions are leaf functions
> > (where the return address is never moved out of the LR), so we can
> > probably get away with a simple annotation for those that avoids the
> > need for open-coded CFI directives everywhere.
>
> Are you suggesting something like a SYM_LEAF_FUNC_(START|END), that
> wraps CFI directives for leaf functions?
Yep; that's exactly the sort of thing I was thinking of.
That or have a seaprate annotation we can add, e.g.
SYM_FUNC_START(foo)
SYM_FUN_END(foo)
SYM_FUNC_IS_LEAF_AND_DOES_NOT_TOUCH_LR(foo)
> > I've pushed some reliable stacktrace tests to:
> >
> > git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git stacktrace/tests
> >
> > That finds the fgraph issue (regardless of this series). When merged
> > with this series triggers a warning in kunwind_next_frame_record_meta(),
> > where unwind_next_frame_sframe() calls that erroneously as a fallback.
>
> Thanks for the pointer on these tests, they're super useful! I've been
> able to reproduce the fgraph failure you mentioned.
Great!
Mark.
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Jens Remus @ 2026-05-12 8:55 UTC (permalink / raw)
To: Dylan Hatch, Mark Rutland
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Heiko Carstens
In-Reply-To: <CADBMgpx9YxNUO6wLP7mYKxWW8L78Hk9gPwHrMjXUwPyUmGEu9w@mail.gmail.com>
Hello Dylan and Mark!
On 5/12/2026 5:00 AM, Dylan Hatch wrote:
> On Fri, May 1, 2026 at 9:46 AM Mark Rutland <mark.rutland@arm.com>
> wrote:
>> More generally, there are a few things that aren't addressed by
>> this series that we will also need to address. Importantly:
>>
>> (1) For correctness, we'll need to address a latent issue with
>> unwinding across an fgraph return trampoline, where the return
>> address is transiently unrecoverable.
>>
>> Before this series, that doesn't matter for livepatching because
>> the livepatching code isn't called synchronously within the fgraph
>> handler, and unwinds which cross an exception boundary are marked
>> as unreliable.
>>
>> After this series, that does matter as we can unwind across an
>> exception boundary, and might happen to interrupt that transient
>> window.
>>
>> I think we can solve that with some restructuring of that code,
>> restoring the original address *before* removing that from the
>> fgraph return stack, and ensuring that the unwinder can find it.
>
> If my understanding is correct, the issue arrises in
> return_to_handler as the return address is recovered:
>
> mov x0, sp bl ftrace_return_to_handler // addr =
> ftrace_return_to_hander(fregs); mov x30, x0 // restore the original
> return address
>
> Because ftrace_return_to_handler pops the return address from the
> return stack before it can be restored into the LR, it cannot be
> recovered.
Based on reliable-stacktrace.rst section "4.4 Rewriting of return
addresses" I wonder whether the following might work:
- If an unwound RA points at return_to_handler the actual RA needs to
be obtained using ftrace_graph_ret_addr(). This might already be
taken into account if ftrace_graph_ret_addr() is used unconditionally.
- If an unwound RA points into return_to_handler() mark the stack trace
as unreliable. This could be accomplished by marking LR in
return_to_handler() as undefined (i.e. .cfi_undefined 30) to use
SFrame's outermost frame indication to stop and mark the stack trace
as unreliable:
diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S
@@ -329,8 +329,12 @@ SYM_FUNC_END(ftrace_stub_graph)
* @fp is checked against the value passed by ftrace_graph_caller().
*/
SYM_CODE_START(return_to_handler)
+ .cfi_startproc
+ /* Mark unwinding of LR as unreliable */
+ .cfi_undefined 30
/* Make room for ftrace_regs */
sub sp, sp, #FREGS_SIZE
+ .cfi_adjust_cfa_offset -FREGS_SIZE
/* Save return value regs */
stp x0, x1, [sp, #FREGS_X0]
@@ -344,6 +348,8 @@ SYM_CODE_START(return_to_handler)
mov x0, sp
bl ftrace_return_to_handler // addr = ftrace_return_to_hander(fregs);
mov x30, x0 // restore the original return address
+ /* Mark unwinding of LR as reliable */
+ .cfi_restore 30
/* Restore return value regs */
ldp x0, x1, [sp, #FREGS_X0]
@@ -351,7 +357,9 @@ SYM_CODE_START(return_to_handler)
ldp x4, x5, [sp, #FREGS_X4]
ldp x6, x7, [sp, #FREGS_X6]
add sp, sp, #FREGS_SIZE
+ .cfi_adjust_cfa_offset FREGS_SIZE
ret
+ .cfi_endproc
SYM_CODE_END(return_to_handler)
#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
Notes regarding above:
- return_to_handler() saves the caller's FP in ftrace_regs but never
restores it. I suppose this is because ftrace_regs is an input to
ftrace_return_to_handler(). The DWARF CFI above assumes SP and FP
can be restored at all times as SP=CFA and FP=FP.
- One might be tempted to add a .cfi_register 30, 0 after the call to
ftrace_return_to_handler. This would be wrong, because if unwinding
comes from ftrace_return_to_handler() the unwound RA will point there
and the unwinding logic would erroneously assume x0 to contain the RA.
- The DWARF CFI could be simplified as follows to just convey that
unwinding through return_to_handler is impossible at all times:
diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S
@@ -329,6 +329,9 @@ SYM_FUNC_END(ftrace_stub_graph)
* @fp is checked against the value passed by ftrace_graph_caller().
*/
SYM_CODE_START(return_to_handler)
+ .cfi_startproc simple
+ /* Mark unwinding of LR as unreliable */
+ .cfi_undefined 30
/* Make room for ftrace_regs */
sub sp, sp, #FREGS_SIZE
@@ -353,5 +356,6 @@ SYM_CODE_START(return_to_handler)
add sp, sp, #FREGS_SIZE
ret
+ .cfi_endproc
SYM_CODE_END(return_to_handler)
#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
>
> Based on this, I believe you are suggesting to restructure this code
> path such that the return address is removed from the return stack
> only after it has been restored to LR. But since kernel/trace/
> fgraph.c is core kernel code, will this end up requiring either (1)
> a similar restructuring of other arches supporting ftrace, or (2) an
> arm64-specific implementation of this recovery logic?
>
> It looks to me like there is essentially the same recovery pattern
> on other arches; is there a reason this transient unrecoverability
> isn't an issue for reliable unwind on other platforms?
>
>>
>> I'm not immediately sure whether kretprobes has a similar issue.
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
jremus@de.ibm.com / jremus@linux.ibm.com
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Ehningen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Dylan Hatch @ 2026-05-12 3:00 UTC (permalink / raw)
To: Mark Rutland
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Jens Remus, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap
In-Reply-To: <afTYzAF_x41pyilu@J2N7QTR9R3>
Hi Mark,
Thanks for all the feedback and help on this. I'm planning on getting
your comments addressed in the coming days, but I have some initial
clarifying questions.
On Fri, May 1, 2026 at 9:46 AM Mark Rutland <mark.rutland@arm.com> wrote:
>
> Hi Dylan,
>
> Thanks for putting this together. I think this is looking pretty good.
> However, there are some things that aren't quite right and need some
> work, which I've commented on below.
>
> More generally, there are a few things that aren't addressed by this
> series that we will also need to address. Importantly:
>
> (1) For correctness, we'll need to address a latent issue with unwinding
> across an fgraph return trampoline, where the return address is
> transiently unrecoverable.
>
> Before this series, that doesn't matter for livepatching because the
> livepatching code isn't called synchronously within the fgraph
> handler, and unwinds which cross an exception boundary are marked as
> unreliable.
>
> After this series, that does matter as we can unwind across an
> exception boundary, and might happen to interrupt that transient
> window.
>
> I think we can solve that with some restructuring of that code,
> restoring the original address *before* removing that from the
> fgraph return stack, and ensuring that the unwinder can find it.
If my understanding is correct, the issue arrises in return_to_handler
as the return address is recovered:
mov x0, sp
bl ftrace_return_to_handler // addr = ftrace_return_to_hander(fregs);
mov x30, x0 // restore the original return address
Because ftrace_return_to_handler pops the return address from the
return stack before it can be restored into the LR, it cannot be
recovered.
Based on this, I believe you are suggesting to restructure this code
path such that the return address is removed from the return stack
only after it has been restored to LR. But since kernel/trace/fgraph.c
is core kernel code, will this end up requiring either (1) a similar
restructuring of other arches supporting ftrace, or (2) an
arm64-specific implementation of this recovery logic?
It looks to me like there is essentially the same recovery pattern on
other arches; is there a reason this transient unrecoverability isn't
an issue for reliable unwind on other platforms?
>
> I'm not immediately sure whether kretprobes has a similar issue.
>
> (2) To make unwinding generally possible, we'll need to annotate some
> assembly functions as unwindable. We'll need to do that for string
> routines under lib/, and probably some crypto code, but we don't
> need to do that for most code in head.S, entry.S, etc.
>
> The vast majority of relevant assembly functions are leaf functions
> (where the return address is never moved out of the LR), so we can
> probably get away with a simple annotation for those that avoids the
> need for open-coded CFI directives everywhere.
Are you suggesting something like a SYM_LEAF_FUNC_(START|END), that
wraps CFI directives for leaf functions?
>
> I've pushed some reliable stacktrace tests to:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git stacktrace/tests
>
> That finds the fgraph issue (regardless of this series). When merged
> with this series triggers a warning in kunwind_next_frame_record_meta(),
> where unwind_next_frame_sframe() calls that erroneously as a fallback.
Thanks for the pointer on these tests, they're super useful! I've been
able to reproduce the fgraph failure you mentioned.
Thanks,
Dylan
^ permalink raw reply
* Re: [PATCH v5 0/8] unwind, arm64: add sframe unwinder for kernel
From: Dylan Hatch @ 2026-05-12 1:10 UTC (permalink / raw)
To: Jens Remus
Cc: Roman Gushchin, Weinan Liu, Will Deacon, Josh Poimboeuf,
Indu Bhagat, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Mark Rutland, Prasanna Kumar T S M, Puranjay Mohan,
Song Liu, joe.lawrence, linux-toolchains, linux-kernel,
live-patching, linux-arm-kernel, Randy Dunlap, Heiko Carstens
In-Reply-To: <549d10b6-ba2b-4ae9-86ef-6157e13b6ee3@linux.ibm.com>
On Thu, Apr 30, 2026 at 3:11 AM Jens Remus <jremus@linux.ibm.com> wrote:
>
> On 4/28/2026 8:36 PM, Dylan Hatch wrote:
> > Implement a generic kernel sframe-based [1] unwinder. The main goal is
> > to improve reliable stacktrace on arm64 by unwinding across exception
> > boundaries.
>
> Please add support to initialize the optional sframe unwinder debug
> information. Either in the appropriate patches in this series or as a
> separate patch.
Sounds good, I'll add this in as a separate patch in the next version.
>
> Note that for the module case I wonder whether it would be preferable
> to somehow indicate that it is a module name in the string, e.g.
> "(<module-name>)" or "<module-name> (module)"?
I don't have a strong preference, though I agree it makes sense to
indicate that the section is from a module. For now I'll add the
parentheses "(<module-name>)".
>
> diff --git a/kernel/unwind/sframe.c b/kernel/unwind/sframe.c
> --- a/kernel/unwind/sframe.c
> +++ b/kernel/unwind/sframe.c
> @@ -1028,6 +1028,8 @@ void __init init_sframe_table(void)
> kernel_sfsec.text_start = (unsigned long)_stext;
> kernel_sfsec.text_end = (unsigned long)_etext;
>
> + dbg_init(&kernel_sfsec);
> +
> if (WARN_ON(sframe_read_header(&kernel_sfsec)))
> return;
> if (WARN_ON(sframe_validate_section(&kernel_sfsec)))
> @@ -1047,6 +1049,8 @@ void sframe_module_init(struct module *mod, void *sframe, size_t sframe_size,
> sec->text_start = (unsigned long)text;
> sec->text_end = (unsigned long)text + text_size;
>
> + dbg_init(sec);
> +
> if (WARN_ON(sframe_read_header(sec)))
> return;
> if (WARN_ON(sframe_validate_section(sec)))
> diff --git a/kernel/unwind/sframe_debug.h b/kernel/unwind/sframe_debug.h
> --- a/kernel/unwind/sframe_debug.h
> +++ b/kernel/unwind/sframe_debug.h
> @@ -32,6 +32,18 @@ static inline void dbg_init(struct sframe_section *sec)
> struct mm_struct *mm = current->mm;
> struct vm_area_struct *vma;
>
> + if (sec->sec_type == SFRAME_KERNEL) {
> + if (sec == &kernel_sfsec) {
> + sec->filename = kstrdup("(vmlinux)", GFP_KERNEL);
> + } else {
> + struct module *mod = container_of(sec, struct module,
> + arch.sframe_sec);
> + sec->filename = kstrdup(mod->name, GFP_KERNEL);
> + }
> +
> + return;
> + }
> +
> guard(mmap_read_lock)(mm);
> vma = vma_lookup(mm, sec->sframe_start);
> if (!vma)
>
> Regards,
> Jens
> --
> Jens Remus
> Linux on Z Development (D3303)
> jremus@de.ibm.com / jremus@linux.ibm.com
>
> IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Ehningen; Registergericht: Amtsgericht Stuttgart, HRB 243294
> IBM Data Privacy Statement: https://www.ibm.com/privacy/
>
Thanks for the suggestion,
Dylan
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Jens Remus @ 2026-05-05 15:52 UTC (permalink / raw)
To: Mark Rutland, Indu Bhagat
Cc: Dylan Hatch, Roman Gushchin, Weinan Liu, Will Deacon,
Josh Poimboeuf, Peter Zijlstra, Steven Rostedt, Catalin Marinas,
Jiri Kosina, Prasanna Kumar T S M, Puranjay Mohan, Song Liu,
joe.lawrence, linux-toolchains, linux-kernel, live-patching,
linux-arm-kernel, Randy Dunlap, Heiko Carstens
In-Reply-To: <afnGhsCYEUG0LXR5@J2N7QTR9R3>
On 5/5/2026 12:29 PM, Mark Rutland wrote:
> On Mon, May 04, 2026 at 10:47:26AM +0200, Jens Remus wrote:
>> On 5/1/2026 6:46 PM, Mark Rutland wrote:
>>> Thanks for putting this together. I think this is looking pretty good.
>>> However, there are some things that aren't quite right and need some
>>> work, which I've commented on below.
>>
>>> (2) To make unwinding generally possible, we'll need to annotate some
>>> assembly functions as unwindable. We'll need to do that for string
>>> routines under lib/, and probably some crypto code, but we don't
>>> need to do that for most code in head.S, entry.S, etc.
>>>
>>> The vast majority of relevant assembly functions are leaf functions
>>> (where the return address is never moved out of the LR), so we can
>>> probably get away with a simple annotation for those that avoids the
>>> need for open-coded CFI directives everywhere.
>>
>> Wrapping them in .cfi_startproc ... .cfi_endproc should do. For instance
>> by extending SYM_FUNC_START() and SYM_FUNC_END() or introducing flavors
>> that do. Or where you thinking of something else?
>
> I was expecting we'd do something like that, either with distinct
> versions, or some entirely separate annotation.
>
> We can't override SYM_FUNC_START() or SYM_FUNC_END() since those are
> also used for non-leaf functions. The bulk of the work is going to be
> making sure we only annotate leaf functions specifically, which will
> require some human analysis.
Makes sense.
>>> On Tue, Apr 28, 2026 at 06:36:43PM +0000, Dylan Hatch wrote:
>>>> diff --git a/arch/arm64/include/asm/stacktrace/common.h b/arch/arm64/include/asm/stacktrace/common.h
>>
>>>> @@ -21,6 +21,8 @@ struct stack_info {
>>>> *
>>>> * @fp: The fp value in the frame record (or the real fp)
>>>> * @pc: The lr value in the frame record (or the real lr)
>>>> + * @sp: The sp value at the call site of the current function.
>>>> + * @unreliable: Stacktrace is unreliable.
>>>> *
>>>> * @stack: The stack currently being unwound.
>>>> * @stacks: An array of stacks which can be unwound.
>>>> @@ -29,7 +31,11 @@ struct stack_info {
>>>> struct unwind_state {
>>>> unsigned long fp;
>>>> unsigned long pc;
>>>> +#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
>>>> + unsigned long sp;
>>>> +#endif
>>>
>>> As this is only used by the kernel unwinder (and not the hyp unwinder),
>>> this should live in struct kunwind_state in stacktrace.c.
>>>
>>> That said, for unwinding across exception boundaries we should not need
>>> this, as the SP value will be in the pt_regs. If we only use SFrame for
>>> the exception boundary case, we can remove this entirely. I would
>>> strongly prefer that we do that.
>
>>>> + /* Get the Canonical Frame Address (CFA) */
>>>> + switch (frame.cfa.rule) {
>>>> + case UNWIND_CFA_RULE_SP_OFFSET:
>>>> + cfa = state->common.sp;
>>
>> IIUC you suggest this to be changed as follows?
>>
>> cfa = regs->regs[31];
>
> I was suggesting:
>
> cfa = regs->sp;
>
> Note: arm64's struct pt_regs has:
>
> union {
> struct user_pt_regs user_regs;
> struct {
> u64 regs[31];
> u64 sp;
> u64 pc;
> u64 pstate;
> };
> };
>
> ... so regs->regs[31] would be an out-of-bounds array access.
Aww, my bad! Of course!
>
> [...]
>
>>>> + case UNWIND_CFA_RULE_REG_OFFSET:
>>>> + case UNWIND_CFA_RULE_REG_OFFSET_DEREF:
>>>> + /* regs only available in topmost/interrupt frame */
>>>> + if (!regs || frame.cfa.regnum > 30)
>>>> + return -EINVAL;
>>>> + cfa = regs->regs[frame.cfa.regnum];
>>>> + break;
>>>
>>> Do we ever expect to see UNWIND_CFA_RULE_REG_OFFSET or
>>> UNWIND_CFA_RULE_REG_OFFSET_DEREF in practice for kernel code?
>>
>> No. Those can only occur with SFrame V3 flexible FDE, which are
>> currently not generated by GNU assembler for arm64/aarch64, and thus
>> could be omitted in the arm64-specific kernel sframe unwinder:
>>
>> https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gas/config/tc-aarch64.h;hb=binutils-2_46#l342
>
> Ok.
>
> Do we know whether there are currently cases on aarch64 that cannot be
> encoded in SFrame (without flexible FDE), or whether SFrame without
> flexible FDE is sufficient for arm64 as-is? ... or do we have
> counter-examples today?
Not that I am aware of. IIUC this is why Indu, the SFrame maintainer,
did not enable SFrame V3 flexible FDE for arm64/aarch64 in the GNU
assembler.
> Looking at:
>
> https://sourceware.org/binutils/docs/sframe-spec.html#Flexible-FDE-Type-Interpretation-1
>
> For arm64 I'm not sure whether we'd encounter the DRAP or stack
> realignment cases within the kernel (perhaps with SVE?), nor whether the
> Register-based RA/FP Locations cases would apply if we assume that we
> continue to use frame records.
@Indu: Can you provide more insight?
> [... ]
>
>> I must admit that while reviewing I thought it would be future-proof to
>> have support for rules that can only be represented with SFrame V3
>> flexible FDE, even if they are currently not used on arm64. Ideally
>> kunwind_next_sframe() could be made common, if another architecture
>> would implement kernel unwinding using sframe.
>
> While I understand that principle, I think that for now it would be
> better to keep this arch-specific and minimal:
>
> * We have arch-specific concerns (e.g. the FRAME_META_TYPE_FINAL
> frames), and factoring that into generic code is going to be painful
> to adapt (which we're likely to need to do in the near future), and to
> maintain going forwards. Keeping that arch-specific for now will make
> it easier/quicker to get to a stable state.
>
> * Code which isn't used is liable to be wrong or made wrong by accident.
> For example, with all the SP cases I mentioned in my initial reply.
>
> We can certainly look at making that more generic in future, but for now
> I'd prefer to omit the code that cannot be used (and have some sort of
> build or boot/module-load time check that SFrame only has elements that
> we expect), and make sure that we thoroughly test the cases that do
> exist in practice.
Makes sense.
> Do we expect SFrame V3 flexible FDE to be generated by toolchains in the
> near future?
There is work in progress to implement SFrame V3 generation in LLVM.
Their implementation could be less restrictive regarding to which
SFrame V3 feature to enable on arm64/aarch64. But even then for
SFrame V3 flexible FDE to be generated there must be DWARF CFI patterns
that can only be represented using those. So if those do not exist
(see your previous question above), then they should not be generated.
@Indu: What are your thoughts on this?
> [...]
>
>>>> + /* CFA alignment 8 bytes */
>>>> + if (cfa & 0x7)
>>>> + return -EINVAL;
>>>
>>> If the CFA is the SP upon entry to the function, then per AAPCS64 rules
>>> it should be aligned to 16 bytes. Otherwise, where has this 8 byte
>>> alignment requirement come from? Does SFrame mandate that?
>>
>> That originates from the common unwind user logic (see
>> kernel/unwind/user.c, unwind_user_next_common()), which currently
>> assumes 8-byte/4-byte SP alignment for all 64-bit/32-bit architectures.
>>
>> So checking for 16-byte alignment here would make sense.
>
> Just to confirm, am I correct to understand that the SFrame definition
> of CFA is intended to be the same as the DWARF definition of CFA, and so
Correct.
> for arm64 the CFA is the SP when the function is called?
Correct.
>
> That's the case for DWARF on arm64:
>
> https://github.com/ARM-software/abi-aa/releases/download/2025Q4/aadwarf64.pdf
> https://github.com/ARM-software/abi-aa/blob/daa7a94ca55973736c0e434a67a6e4bbcd35d7fa/aadwarf64/aadwarf64.rst
>
> | The CFA is the value of the stack pointer (sp) at the call site in the
> | previous frame.
>
> I couldn't find an explciit statement to that effect in:
>
> https://sourceware.org/binutils/docs/sframe-spec.html
>
> ... but I guess that is implied, given the other bits inherited from
> DWARF.
I assume so.
>
> I see that the documented behaviour for CFA on AMD64 and s390x are
> consistent with their DWARF behaviour.
Yes.
>
>>>> +
>>>> + /* Get the Return Address (RA) */
>>>> + switch (frame.ra.rule) {
>>>> + case UNWIND_RULE_RETAIN:
>>>> + /* regs only available in topmost/interrupt frame */
>>>> + if (!regs)
>>>> + return -EINVAL;
>>>> + ra = regs->regs[30];
>>>> + source = KUNWIND_SOURCE_REGS_LR;
>>>> + break;
>>>> + /* UNWIND_USER_RULE_CFA_OFFSET not implemented on purpose */
>>
>> Nit: s/UNWIND_USER_RULE_CFA_OFFSET/UNWIND_RULE_CFA_OFFSET/
>>
>>>
>>> It would be better for the comment to say *why* that's not implemented.
>>>
>>> I assume that's because UNWIND_USER_RULE_CFA_OFFSET would mean that the return
>>> address is a stack address, and that's obviously not legitimate.
>>
>> That and SFrame V3 currently cannot represent FP/RA as CFA + offset
>> (i.e. UNWIND_RULE_CFA_OFFSET; .cfi_val_offset FP/RA).
>>
>> The comment originates from the common unwind user logic (see
>> kernel/unwind/user.c). I am open to improve that. What about:
>>
>> /*
>> * UNWIND_RULE_CFA_OFFSET not implemented on purpose, as a stack
>> * address cannot be a legitimate return address value. It is
>> * also not used (e.g. not represented in sframe).
>> */
>
> I'd go with something simpler, e.g.
>
> /*
> * UNWIND_RULE_CFA_OFFSET doesn't make sense for RA.
> * The return address cannot legitimately be a stack addres.
> */
Thanks! I have updated the comments accordingly in my latest unwind
user sframe patch series sent today:
[PATCH v14 14/19] unwind_user: Flexible FP/RA recovery rules
https://lore.kernel.org/all/20260505121718.3572346-15-jremus@linux.ibm.com/
[...]
>>>> + default:
>>>> + WARN_ON_ONCE(1);
>>>> + return -EINVAL;
>>>> + }
>>>> +
>>>> + /* Get the Frame Pointer (FP) */
>>>> + switch (frame.fp.rule) {
>>>> + case UNWIND_RULE_RETAIN:
>>>> + fp = state->common.fp;
>>>> + break;
>>>> + /* UNWIND_USER_RULE_CFA_OFFSET not implemented on purpose */
>>>
>>> As for RA, the comment should explain why that's not implemented.
>>
>> I am open to improve the comment in the the common unwind user logic.
>> What about:
>>
>> /*
>> * UNWIND_RULE_CFA_OFFSET not implemented on purpose, as it is
>> * not used (e.g. not represented in sframe).
>> */
>
> For me, this wording raises more questions, e.g.
>
> * Does 'not used' mean that toolchains don't use that, or that the spec
> doesn't permit that?
unwind user currently only supports frame pointer, with SFrame to be
hopefully added soon. Out of these only SFrame requires ("uses") these
elaborated rules.
>
> * Does 'not represented' mean that this is not represntable, or that
> toolchains currently don't generate SFrame with the appropriate
> elements.
In DWARF CFI it is representable using .cfi_val_offset <FP>, <offset>.
But SFrame V3 currently cannot represent this:
"Note that, using a value of 0 as padding data word, does mean that
currently, e.g., for RA [JR: FP likewise], the rule RA = CFA + 0 cannot
be encoded. NB: RA = CFA + 0 is distinct from RA = *(CFA + 0)."
https://sourceware.org/binutils/docs/sframe-spec.html#Flexible-FDE-Type-Interpretation-1
>
> IIUC you're saying that this *is* representable with flexible FDE, but
> current toolchains don't generate that.
Thanks for the feedback! I changed it as follows to clarify:
/*
* UNWIND_USER_RULE_CFA_OFFSET is currently not used for FP
* (e.g. SFrame cannot represent this rule).
*/
Regards,
Jens
--
Jens Remus
Linux on Z Development (D3303)
jremus@de.ibm.com / jremus@linux.ibm.com
IBM Deutschland Research & Development GmbH; Vorsitzender des Aufsichtsrats: Wolfgang Wendt; Geschäftsführung: David Faller; Sitz der Gesellschaft: Ehningen; Registergericht: Amtsgericht Stuttgart, HRB 243294
IBM Data Privacy Statement: https://www.ibm.com/privacy/
^ permalink raw reply
* Re: [PATCH v5 8/8] unwind: arm64: Use sframe to unwind interrupt frames
From: Mark Rutland @ 2026-05-05 10:29 UTC (permalink / raw)
To: Jens Remus
Cc: Dylan Hatch, Roman Gushchin, Weinan Liu, Will Deacon,
Josh Poimboeuf, Indu Bhagat, Peter Zijlstra, Steven Rostedt,
Catalin Marinas, Jiri Kosina, Prasanna Kumar T S M,
Puranjay Mohan, Song Liu, joe.lawrence, linux-toolchains,
linux-kernel, live-patching, linux-arm-kernel, Randy Dunlap,
Heiko Carstens
In-Reply-To: <bc3fb59b-9d80-4957-af51-20db38e3487e@linux.ibm.com>
On Mon, May 04, 2026 at 10:47:26AM +0200, Jens Remus wrote:
> Hello Mark,
>
> I mostly have comments regarding your the SFrame related remarks.
Thanks for this; I have a few more questions and comments below.
> On 5/1/2026 6:46 PM, Mark Rutland wrote:
> > Thanks for putting this together. I think this is looking pretty good.
> > However, there are some things that aren't quite right and need some
> > work, which I've commented on below.
>
> > (2) To make unwinding generally possible, we'll need to annotate some
> > assembly functions as unwindable. We'll need to do that for string
> > routines under lib/, and probably some crypto code, but we don't
> > need to do that for most code in head.S, entry.S, etc.
> >
> > The vast majority of relevant assembly functions are leaf functions
> > (where the return address is never moved out of the LR), so we can
> > probably get away with a simple annotation for those that avoids the
> > need for open-coded CFI directives everywhere.
>
> Wrapping them in .cfi_startproc ... .cfi_endproc should do. For instance
> by extending SYM_FUNC_START() and SYM_FUNC_END() or introducing flavors
> that do. Or where you thinking of something else?
I was expecting we'd do something like that, either with distinct
versions, or some entirely separate annotation.
We can't override SYM_FUNC_START() or SYM_FUNC_END() since those are
also used for non-leaf functions. The bulk of the work is going to be
making sure we only annotate leaf functions specifically, which will
require some human analysis.
> > On Tue, Apr 28, 2026 at 06:36:43PM +0000, Dylan Hatch wrote:
> >> diff --git a/arch/arm64/include/asm/stacktrace/common.h b/arch/arm64/include/asm/stacktrace/common.h
>
> >> @@ -21,6 +21,8 @@ struct stack_info {
> >> *
> >> * @fp: The fp value in the frame record (or the real fp)
> >> * @pc: The lr value in the frame record (or the real lr)
> >> + * @sp: The sp value at the call site of the current function.
> >> + * @unreliable: Stacktrace is unreliable.
> >> *
> >> * @stack: The stack currently being unwound.
> >> * @stacks: An array of stacks which can be unwound.
> >> @@ -29,7 +31,11 @@ struct stack_info {
> >> struct unwind_state {
> >> unsigned long fp;
> >> unsigned long pc;
> >> +#ifdef CONFIG_HAVE_UNWIND_KERNEL_SFRAME
> >> + unsigned long sp;
> >> +#endif
> >
> > As this is only used by the kernel unwinder (and not the hyp unwinder),
> > this should live in struct kunwind_state in stacktrace.c.
> >
> > That said, for unwinding across exception boundaries we should not need
> > this, as the SP value will be in the pt_regs. If we only use SFrame for
> > the exception boundary case, we can remove this entirely. I would
> > strongly prefer that we do that.
> >> + /* Get the Canonical Frame Address (CFA) */
> >> + switch (frame.cfa.rule) {
> >> + case UNWIND_CFA_RULE_SP_OFFSET:
> >> + cfa = state->common.sp;
>
> IIUC you suggest this to be changed as follows?
>
> cfa = regs->regs[31];
I was suggesting:
cfa = regs->sp;
Note: arm64's struct pt_regs has:
union {
struct user_pt_regs user_regs;
struct {
u64 regs[31];
u64 sp;
u64 pc;
u64 pstate;
};
};
... so regs->regs[31] would be an out-of-bounds array access.
[...]
> >> + case UNWIND_CFA_RULE_REG_OFFSET:
> >> + case UNWIND_CFA_RULE_REG_OFFSET_DEREF:
> >> + /* regs only available in topmost/interrupt frame */
> >> + if (!regs || frame.cfa.regnum > 30)
> >> + return -EINVAL;
> >> + cfa = regs->regs[frame.cfa.regnum];
> >> + break;
> >
> > Do we ever expect to see UNWIND_CFA_RULE_REG_OFFSET or
> > UNWIND_CFA_RULE_REG_OFFSET_DEREF in practice for kernel code?
>
> No. Those can only occur with SFrame V3 flexible FDE, which are
> currently not generated by GNU assembler for arm64/aarch64, and thus
> could be omitted in the arm64-specific kernel sframe unwinder:
>
> https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gas/config/tc-aarch64.h;hb=binutils-2_46#l342
Ok.
Do we know whether there are currently cases on aarch64 that cannot be
encoded in SFrame (without flexible FDE), or whether SFrame without
flexible FDE is sufficient for arm64 as-is? ... or do we have
counter-examples today?
Looking at:
https://sourceware.org/binutils/docs/sframe-spec.html#Flexible-FDE-Type-Interpretation-1
For arm64 I'm not sure whether we'd encounter the DRAP or stack
realignment cases within the kernel (perhaps with SVE?), nor whether the
Register-based RA/FP Locations cases would apply if we assume that we
continue to use frame records.
[... ]
> I must admit that while reviewing I thought it would be future-proof to
> have support for rules that can only be represented with SFrame V3
> flexible FDE, even if they are currently not used on arm64. Ideally
> kunwind_next_sframe() could be made common, if another architecture
> would implement kernel unwinding using sframe.
While I understand that principle, I think that for now it would be
better to keep this arch-specific and minimal:
* We have arch-specific concerns (e.g. the FRAME_META_TYPE_FINAL
frames), and factoring that into generic code is going to be painful
to adapt (which we're likely to need to do in the near future), and to
maintain going forwards. Keeping that arch-specific for now will make
it easier/quicker to get to a stable state.
* Code which isn't used is liable to be wrong or made wrong by accident.
For example, with all the SP cases I mentioned in my initial reply.
We can certainly look at making that more generic in future, but for now
I'd prefer to omit the code that cannot be used (and have some sort of
build or boot/module-load time check that SFrame only has elements that
we expect), and make sure that we thoroughly test the cases that do
exist in practice.
Do we expect SFrame V3 flexible FDE to be generated by toolchains in the
near future?
[...]
> >> + /* CFA alignment 8 bytes */
> >> + if (cfa & 0x7)
> >> + return -EINVAL;
> >
> > If the CFA is the SP upon entry to the function, then per AAPCS64 rules
> > it should be aligned to 16 bytes. Otherwise, where has this 8 byte
> > alignment requirement come from? Does SFrame mandate that?
>
> That originates from the common unwind user logic (see
> kernel/unwind/user.c, unwind_user_next_common()), which currently
> assumes 8-byte/4-byte SP alignment for all 64-bit/32-bit architectures.
>
> So checking for 16-byte alignment here would make sense.
Just to confirm, am I correct to understand that the SFrame definition
of CFA is intended to be the same as the DWARF definition of CFA, and so
for arm64 the CFA is the SP when the function is called?
That's the case for DWARF on arm64:
https://github.com/ARM-software/abi-aa/releases/download/2025Q4/aadwarf64.pdf
https://github.com/ARM-software/abi-aa/blob/daa7a94ca55973736c0e434a67a6e4bbcd35d7fa/aadwarf64/aadwarf64.rst
| The CFA is the value of the stack pointer (sp) at the call site in the
| previous frame.
I couldn't find an explciit statement to that effect in:
https://sourceware.org/binutils/docs/sframe-spec.html
... but I guess that is implied, given the other bits inherited from
DWARF.
I see that the documented behaviour for CFA on AMD64 and s390x are
consistent with their DWARF behaviour.
> >> +
> >> + /* Get the Return Address (RA) */
> >> + switch (frame.ra.rule) {
> >> + case UNWIND_RULE_RETAIN:
> >> + /* regs only available in topmost/interrupt frame */
> >> + if (!regs)
> >> + return -EINVAL;
> >> + ra = regs->regs[30];
> >> + source = KUNWIND_SOURCE_REGS_LR;
> >> + break;
> >> + /* UNWIND_USER_RULE_CFA_OFFSET not implemented on purpose */
>
> Nit: s/UNWIND_USER_RULE_CFA_OFFSET/UNWIND_RULE_CFA_OFFSET/
>
> >
> > It would be better for the comment to say *why* that's not implemented.
> >
> > I assume that's because UNWIND_USER_RULE_CFA_OFFSET would mean that the return
> > address is a stack address, and that's obviously not legitimate.
>
> That and SFrame V3 currently cannot represent FP/RA as CFA + offset
> (i.e. UNWIND_RULE_CFA_OFFSET; .cfi_val_offset FP/RA).
>
> The comment originates from the common unwind user logic (see
> kernel/unwind/user.c). I am open to improve that. What about:
>
> /*
> * UNWIND_RULE_CFA_OFFSET not implemented on purpose, as a stack
> * address cannot be a legitimate return address value. It is
> * also not used (e.g. not represented in sframe).
> */
I'd go with something simpler, e.g.
/*
* UNWIND_RULE_CFA_OFFSET doesn't make sense for RA.
* The return address cannot legitimately be a stack addres.
*/
[...]
> >
> > I don't think we expect UNWIND_RULE_REG_OFFSET unless that's sometimes used
> > instead of UNWIND_RULE_RETAIN to express that the return address is in x30
> > (with zero offset).
>
> No. Unless there would be nonsense .cfi_register 30, 30, which would
> require SFrame V3 flexible FDE to be represented.
Ok.
> @Indu: We may consider to treat .cfi_register <reg>, <reg> (for FP/RA)
> like .cfi_restore <reg> in the GNU assembler?
>
> > Similarly, if the address is on the stack it should be in a frame
> > record. Would we ever expect UNWIND_RULE_REG_OFFSET_DEREF rather than
> > UNWIND_RULE_CFA_OFFSET_DEREF?
>
> No. See above (SFrame V3 flexible FDE).
Ok.
> >> + default:
> >> + WARN_ON_ONCE(1);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + /* Get the Frame Pointer (FP) */
> >> + switch (frame.fp.rule) {
> >> + case UNWIND_RULE_RETAIN:
> >> + fp = state->common.fp;
> >> + break;
> >> + /* UNWIND_USER_RULE_CFA_OFFSET not implemented on purpose */
> >
> > As for RA, the comment should explain why that's not implemented.
>
> I am open to improve the comment in the the common unwind user logic.
> What about:
>
> /*
> * UNWIND_RULE_CFA_OFFSET not implemented on purpose, as it is
> * not used (e.g. not represented in sframe).
> */
For me, this wording raises more questions, e.g.
* Does 'not used' mean that toolchains don't use that, or that the spec
doesn't permit that?
* Does 'not represented' mean that this is not represntable, or that
toolchains currently don't generate SFrame with the appropriate
elements.
IIUC you're saying that this *is* representable with flexible FDE, but
current toolchains don't generate that.
Mark.
^ permalink raw reply
page: next (older)
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox