[PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

This patchset implements a generic kernel sframe-based [1] unwinder.
The main goal is to support reliable stacktraces on arm64.

On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
required support from objtool: it cannot generate orc unwind tables for
arm64.

Currently, there's already a sframe unwinder proposed for userspace: [2].
Since the sframe unwind table algorithm is similar, these two proposal
could integrate common functionality in the future.

There are some incomplete features or challenges:
  - The unwinder doesn't yet work with kernel modules. The `start_addr` of
    FRE from kernel modules doesn't appear correct, preventing us from
    unwinding functions from kernel modules.
  - Currently, only GCC supports sframe.

Ref:
[1]: https://sourceware.org/binutils/docs/sframe-spec.html
[2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/

Madhavan T. Venkataraman (1):
  arm64: Define TIF_PATCH_PENDING for livepatch

Weinan Liu (7):
  unwind: build kernel with sframe info
  arm64: entry: add unwind info for various kernel entries
  unwind: add sframe v2 header
  unwind: Implement generic sframe unwinder library
  unwind: arm64: Add sframe unwinder on arm64
  unwind: arm64: add reliable stacktrace support for arm64
  arm64: Enable livepatch for ARM64

 Makefile                                   |   6 +
 arch/Kconfig                               |   8 +
 arch/arm64/Kconfig                         |   3 +
 arch/arm64/Kconfig.debug                   |  10 +
 arch/arm64/include/asm/stacktrace/common.h |   6 +
 arch/arm64/include/asm/thread_info.h       |   4 +-
 arch/arm64/kernel/entry-common.c           |   4 +
 arch/arm64/kernel/entry.S                  |  10 +
 arch/arm64/kernel/setup.c                  |   2 +
 arch/arm64/kernel/stacktrace.c             | 102 ++++++++++
 include/asm-generic/vmlinux.lds.h          |  12 ++
 include/linux/sframe_lookup.h              |  43 +++++
 kernel/Makefile                            |   1 +
 kernel/sframe.h                            | 215 +++++++++++++++++++++
 kernel/sframe_lookup.c                     | 196 +++++++++++++++++++
 15 files changed, 621 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/sframe_lookup.h
 create mode 100644 kernel/sframe.h
 create mode 100644 kernel/sframe_lookup.c

-- 
2.48.1.262.g85cc9f2d1e-goog

[PATCH 1/8] unwind: build kernel with sframe info

[Weinan Liu]

Use the -Wa,--gsframe flags to build the code, so GAS will generate
a new .sframe section for the stack trace information.
Currently, the sframe format only supports arm64 and x86_64
architectures. Add this configuration on arm64 to enable sframe
unwinder in the future.

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 Makefile                          |  6 ++++++
 arch/Kconfig                      |  8 ++++++++
 arch/arm64/Kconfig.debug          | 10 ++++++++++
 include/asm-generic/vmlinux.lds.h | 12 ++++++++++++
 4 files changed, 36 insertions(+)

diff --git a/Makefile b/Makefile
index b9464c88ac72..35200c39b98d 100644
--- a/Makefile
+++ b/Makefile
@@ -1064,6 +1064,12 @@ ifdef CONFIG_CC_IS_GCC
 KBUILD_CFLAGS   += -fconserve-stack
 endif
 
+# build with sframe table
+ifdef CONFIG_SFRAME_UNWIND_TABLE
+KBUILD_CFLAGS	+= -Wa,--gsframe
+KBUILD_AFLAGS	+= -Wa,--gsframe
+endif
+
 # change __FILE__ to the relative path to the source directory
 ifdef building_out_of_srctree
 KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srcroot)/=)
diff --git a/arch/Kconfig b/arch/Kconfig
index 6682b2a53e34..ae70f7dbe326 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX
 	  An architecture can select this if it provides arch/<arch>/tools/Makefile
 	  with .arch.vmlinux.o target to be linked into vmlinux.
 
+config AS_HAS_SFRAME_SUPPORT
+	# Detect availability of the AS option -Wa,--gsframe for generating
+	# sframe unwind table.
+	def_bool $(cc-option,-Wa$(comma)--gsframe)
+
+config SFRAME_UNWIND_TABLE
+	bool
+
 endmenu
diff --git a/arch/arm64/Kconfig.debug b/arch/arm64/Kconfig.debug
index 265c4461031f..ed619fcb18b3 100644
--- a/arch/arm64/Kconfig.debug
+++ b/arch/arm64/Kconfig.debug
@@ -20,4 +20,14 @@ config ARM64_RELOC_TEST
 	depends on m
 	tristate "Relocation testing module"
 
+config SFRAME_UNWINDER
+	bool "Sframe unwinder"
+	depends on AS_HAS_SFRAME_SUPPORT
+	depends on 64BIT
+	select SFRAME_UNWIND_TABLE
+	help
+	  This option enables the sframe (Simple Frame) unwinder for unwinding
+	  kernel stack traces. It uses unwind table that is direclty generated
+	  by toolchain based on DWARF CFI information
+
 source "drivers/hwtracing/coresight/Kconfig"
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 54504013c749..6a437bd084c7 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -469,6 +469,8 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 		*(.rodata1)						\
 	}								\
 									\
+	SFRAME								\
+									\
 	/* PCI quirks */						\
 	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
 		BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early,  _pci_fixups_early,  __start, __end) \
@@ -886,6 +888,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 #define TRACEDATA
 #endif
 
+#ifdef CONFIG_SFRAME_UNWIND_TABLE
+#define SFRAME							\
+	/* sframe */						\
+	.sframe        : AT(ADDR(.sframe) - LOAD_OFFSET) {	\
+		BOUNDED_SECTION_BY(.sframe, _sframe_header)	\
+	}
+#else
+#define SFRAME
+#endif
+
 #ifdef CONFIG_PRINTK_INDEX
 #define PRINTK_INDEX							\
 	.printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) {		\
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 1/8] unwind: build kernel with sframe info

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> Use the -Wa,--gsframe flags to build the code, so GAS will generate
> a new .sframe section for the stack trace information.
> Currently, the sframe format only supports arm64 and x86_64
> architectures. Add this configuration on arm64 to enable sframe
> unwinder in the future.
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   Makefile                          |  6 ++++++
>   arch/Kconfig                      |  8 ++++++++
>   arch/arm64/Kconfig.debug          | 10 ++++++++++
>   include/asm-generic/vmlinux.lds.h | 12 ++++++++++++
>   4 files changed, 36 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index b9464c88ac72..35200c39b98d 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1064,6 +1064,12 @@ ifdef CONFIG_CC_IS_GCC
>   KBUILD_CFLAGS   += -fconserve-stack
>   endif
>   
> +# build with sframe table
> +ifdef CONFIG_SFRAME_UNWIND_TABLE
> +KBUILD_CFLAGS	+= -Wa,--gsframe
> +KBUILD_AFLAGS	+= -Wa,--gsframe
> +endif
> +
>   # change __FILE__ to the relative path to the source directory
>   ifdef building_out_of_srctree
>   KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srcroot)/=)
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 6682b2a53e34..ae70f7dbe326 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX
>   	  An architecture can select this if it provides arch/<arch>/tools/Makefile
>   	  with .arch.vmlinux.o target to be linked into vmlinux.
>   
> +config AS_HAS_SFRAME_SUPPORT
> +	# Detect availability of the AS option -Wa,--gsframe for generating
> +	# sframe unwind table.
> +	def_bool $(cc-option,-Wa$(comma)--gsframe)
> +
> +config SFRAME_UNWIND_TABLE
> +	bool
> +
>   endmenu
> diff --git a/arch/arm64/Kconfig.debug b/arch/arm64/Kconfig.debug
> index 265c4461031f..ed619fcb18b3 100644
> --- a/arch/arm64/Kconfig.debug
> +++ b/arch/arm64/Kconfig.debug
> @@ -20,4 +20,14 @@ config ARM64_RELOC_TEST
>   	depends on m
>   	tristate "Relocation testing module"
>   
> +config SFRAME_UNWINDER
> +	bool "Sframe unwinder"
> +	depends on AS_HAS_SFRAME_SUPPORT
> +	depends on 64BIT
> +	select SFRAME_UNWIND_TABLE
> +	help
> +	  This option enables the sframe (Simple Frame) unwinder for unwinding
> +	  kernel stack traces. It uses unwind table that is direclty generated
> +	  by toolchain based on DWARF CFI information
> +
>   source "drivers/hwtracing/coresight/Kconfig"
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 54504013c749..6a437bd084c7 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -469,6 +469,8 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
>   		*(.rodata1)						\
>   	}								\
>   									\
> +	SFRAME								\
> +									\
>   	/* PCI quirks */						\
>   	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
>   		BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early,  _pci_fixups_early,  __start, __end) \
> @@ -886,6 +888,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
>   #define TRACEDATA
>   #endif
>   
> +#ifdef CONFIG_SFRAME_UNWIND_TABLE
> +#define SFRAME							\
> +	/* sframe */						\
> +	.sframe        : AT(ADDR(.sframe) - LOAD_OFFSET) {	\
> +		BOUNDED_SECTION_BY(.sframe, _sframe_header)	\
> +	}
> +#else
> +#define SFRAME
> +#endif
> +
>   #ifdef CONFIG_PRINTK_INDEX
>   #define PRINTK_INDEX							\
>   	.printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) {		\

Looks good to me.

Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

Re: [PATCH 1/8] unwind: build kernel with sframe info

[Indu Bhagat]

On 1/27/25 1:33 PM, Weinan Liu wrote:
> Use the -Wa,--gsframe flags to build the code, so GAS will generate
> a new .sframe section for the stack trace information.
> Currently, the sframe format only supports arm64 and x86_64
> architectures. Add this configuration on arm64 to enable sframe
> unwinder in the future.
> 
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   Makefile                          |  6 ++++++
>   arch/Kconfig                      |  8 ++++++++
>   arch/arm64/Kconfig.debug          | 10 ++++++++++
>   include/asm-generic/vmlinux.lds.h | 12 ++++++++++++
>   4 files changed, 36 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index b9464c88ac72..35200c39b98d 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1064,6 +1064,12 @@ ifdef CONFIG_CC_IS_GCC
>   KBUILD_CFLAGS   += -fconserve-stack
>   endif
>   
> +# build with sframe table
> +ifdef CONFIG_SFRAME_UNWIND_TABLE
> +KBUILD_CFLAGS	+= -Wa,--gsframe
> +KBUILD_AFLAGS	+= -Wa,--gsframe
> +endif
> +
>   # change __FILE__ to the relative path to the source directory
>   ifdef building_out_of_srctree
>   KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srcroot)/=)
> diff --git a/arch/Kconfig b/arch/Kconfig
> index 6682b2a53e34..ae70f7dbe326 100644
> --- a/arch/Kconfig
> +++ b/arch/Kconfig
> @@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX
>   	  An architecture can select this if it provides arch/<arch>/tools/Makefile
>   	  with .arch.vmlinux.o target to be linked into vmlinux.
>   
> +config AS_HAS_SFRAME_SUPPORT
> +	# Detect availability of the AS option -Wa,--gsframe for generating
> +	# sframe unwind table.
> +	def_bool $(cc-option,-Wa$(comma)--gsframe)
> +

Since the version of an admissible SFrame section needs to be atleast 
SFRAME_VERSION_2, it will make sense to include SFrame version check 
when detecting compatible toolchain.

> +config SFRAME_UNWIND_TABLE
> +	bool
> +
>   endmenu
> diff --git a/arch/arm64/Kconfig.debug b/arch/arm64/Kconfig.debug
> index 265c4461031f..ed619fcb18b3 100644
> --- a/arch/arm64/Kconfig.debug
> +++ b/arch/arm64/Kconfig.debug
> @@ -20,4 +20,14 @@ config ARM64_RELOC_TEST
>   	depends on m
>   	tristate "Relocation testing module"
>   
> +config SFRAME_UNWINDER
> +	bool "Sframe unwinder"
> +	depends on AS_HAS_SFRAME_SUPPORT
> +	depends on 64BIT
> +	select SFRAME_UNWIND_TABLE
> +	help
> +	  This option enables the sframe (Simple Frame) unwinder for unwinding
> +	  kernel stack traces. It uses unwind table that is direclty generated
> +	  by toolchain based on DWARF CFI information
> +
>   source "drivers/hwtracing/coresight/Kconfig"
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 54504013c749..6a437bd084c7 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -469,6 +469,8 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
>   		*(.rodata1)						\
>   	}								\
>   									\
> +	SFRAME								\
> +									\
>   	/* PCI quirks */						\
>   	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
>   		BOUNDED_SECTION_PRE_LABEL(.pci_fixup_early,  _pci_fixups_early,  __start, __end) \
> @@ -886,6 +888,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
>   #define TRACEDATA
>   #endif
>   
> +#ifdef CONFIG_SFRAME_UNWIND_TABLE
> +#define SFRAME							\
> +	/* sframe */						\
> +	.sframe        : AT(ADDR(.sframe) - LOAD_OFFSET) {	\
> +		BOUNDED_SECTION_BY(.sframe, _sframe_header)	\
> +	}
> +#else
> +#define SFRAME
> +#endif
> +
>   #ifdef CONFIG_PRINTK_INDEX
>   #define PRINTK_INDEX							\
>   	.printk_index : AT(ADDR(.printk_index) - LOAD_OFFSET) {		\

Re: [PATCH 1/8] unwind: build kernel with sframe info

[Josh Poimboeuf]

On Tue, Feb 04, 2025 at 04:22:27PM -0800, Indu Bhagat wrote:
> > +++ b/arch/Kconfig
> > @@ -1736,4 +1736,12 @@ config ARCH_WANTS_PRE_LINK_VMLINUX
> >   	  An architecture can select this if it provides arch/<arch>/tools/Makefile
> >   	  with .arch.vmlinux.o target to be linked into vmlinux.
> > +config AS_HAS_SFRAME_SUPPORT
> > +	# Detect availability of the AS option -Wa,--gsframe for generating
> > +	# sframe unwind table.
> > +	def_bool $(cc-option,-Wa$(comma)--gsframe)
> > +
> 
> Since the version of an admissible SFrame section needs to be atleast
> SFRAME_VERSION_2, it will make sense to include SFrame version check when
> detecting compatible toolchain.

Also, Jens pointed out that checking for '-Wa,--gsframe' isn't
sufficient, as it still succeeds for architectures that aren't yet
supported:

  https://lore.kernel.org/b89bcb68-d010-4041-aacf-15b934675705@linux.ibm.com

so in my patches I have:

config AS_SFRAME
	def_bool $(as-instr,.cfi_sections .sframe\n.cfi_startproc\n.cfi_endproc)

Though I'm not sure how you would add an SFrame version check to that.

-- 
Josh

[PATCH 2/8] arm64: entry: add unwind info for various kernel entries

[Weinan Liu]

DWARF CFI (Call Frame Information) specifies how to recover the return
address and callee-saved registers at each PC in a given function.
Compilers are able to generate the CFI annotations when they compile
the code to assembly language. For handcrafted assembly, we need to
annotate them by hand.

Annotate CFI unwind info for assembly for interrupt and exception
handlers.

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 arch/arm64/kernel/entry.S | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 5ae2a34b50bd..fe3e3e29ee5d 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -579,7 +579,12 @@ SYM_CODE_START_LOCAL(el\el\ht\()_\regsize\()_\label)
 	.if \el == 0
 	b	ret_to_user
 	.else
+	.cfi_startproc
+	.cfi_def_cfa_offset PT_REGS_SIZE
+	.cfi_offset 29, S_FP - PT_REGS_SIZE
+	.cfi_offset 30, S_LR - PT_REGS_SIZE
 	b	ret_to_kernel
+	.cfi_endproc
 	.endif
 SYM_CODE_END(el\el\ht\()_\regsize\()_\label)
 	.endm
@@ -889,6 +894,10 @@ SYM_FUNC_START(call_on_irq_stack)
 	/* Move to the new stack and call the function there */
 	add	sp, x16, #IRQ_STACK_SIZE
 	blr	x1
+	.cfi_startproc
+	.cfi_def_cfa 29, 16
+	.cfi_offset 29, -16
+	.cfi_offset 30, -8
 
 	/*
 	 * Restore the SP from the FP, and restore the FP and LR from the frame
@@ -898,6 +907,7 @@ SYM_FUNC_START(call_on_irq_stack)
 	ldp	x29, x30, [sp], #16
 	scs_load_current
 	ret
+	.cfi_endproc
 SYM_FUNC_END(call_on_irq_stack)
 NOKPROBE(call_on_irq_stack)
 
-- 
2.48.1.262.g85cc9f2d1e-goog

[PATCH 3/8] unwind: add sframe v2 header

[Weinan Liu]

Add sframe header so that we know how to access the sframe section
generated by compilers.

This is the sframe header file borrowed from the patchset [1]
Josh Poimboeuf according to sframe v2 spec [2].

[1]: https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2]: https://sourceware.org/binutils/docs/sframe-spec.html

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 kernel/sframe.h | 215 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 215 insertions(+)
 create mode 100644 kernel/sframe.h

diff --git a/kernel/sframe.h b/kernel/sframe.h
new file mode 100644
index 000000000000..11b9be7ad82e
--- /dev/null
+++ b/kernel/sframe.h
@@ -0,0 +1,215 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2023, Oracle and/or its affiliates.
+ *
+ * This file contains definitions for the SFrame stack tracing format, which is
+ * documented at https://sourceware.org/binutils/docs
+ */
+#ifndef _SFRAME_H
+#define _SFRAME_H
+
+#include <linux/types.h>
+
+#define SFRAME_VERSION_1	1
+#define SFRAME_VERSION_2	2
+#define SFRAME_MAGIC		0xdee2
+
+/* Function Descriptor Entries are sorted on PC. */
+#define SFRAME_F_FDE_SORTED	0x1
+/* Frame-pointer based stack tracing. Defined, but not set. */
+#define SFRAME_F_FRAME_POINTER	0x2
+
+#define SFRAME_CFA_FIXED_FP_INVALID 0
+#define SFRAME_CFA_FIXED_RA_INVALID 0
+
+/* Supported ABIs/Arch. */
+#define SFRAME_ABI_AARCH64_ENDIAN_BIG	    1 /* AARCH64 big endian. */
+#define SFRAME_ABI_AARCH64_ENDIAN_LITTLE    2 /* AARCH64 little endian. */
+#define SFRAME_ABI_AMD64_ENDIAN_LITTLE	    3 /* AMD64 little endian. */
+
+/* SFrame FRE types. */
+#define SFRAME_FRE_TYPE_ADDR1	0
+#define SFRAME_FRE_TYPE_ADDR2	1
+#define SFRAME_FRE_TYPE_ADDR4	2
+
+/*
+ * SFrame Function Descriptor Entry types.
+ *
+ * The SFrame format has two possible representations for functions. The
+ * choice of which type to use is made according to the instruction patterns
+ * in the relevant program stub.
+ */
+
+/* Unwinders perform a (PC >= FRE_START_ADDR) to look up a matching FRE. */
+#define SFRAME_FDE_TYPE_PCINC	0
+/*
+ * Unwinders perform a (PC & FRE_START_ADDR_AS_MASK >= FRE_START_ADDR_AS_MASK)
+ * to look up a matching FRE. Typical usecases are pltN entries, trampolines
+ * etc.
+ */
+#define SFRAME_FDE_TYPE_PCMASK	1
+
+/**
+ * struct sframe_preamble - SFrame Preamble.
+ * @magic: Magic number (SFRAME_MAGIC).
+ * @version: Format version number (SFRAME_VERSION).
+ * @flags: Various flags.
+ */
+struct sframe_preamble {
+	u16 magic;
+	u8  version;
+	u8  flags;
+} __packed;
+
+/**
+ * struct sframe_header - SFrame Header.
+ * @preamble: SFrame preamble.
+ * @abi_arch: Identify the arch (including endianness) and ABI.
+ * @cfa_fixed_fp_offset: Offset for the Frame Pointer (FP) from CFA may be
+ *	  fixed for some ABIs ((e.g, in AMD64 when -fno-omit-frame-pointer is
+ *	  used). When fixed, this field specifies the fixed stack frame offset
+ *	  and the individual FREs do not need to track it. When not fixed, it
+ *	  is set to SFRAME_CFA_FIXED_FP_INVALID, and the individual FREs may
+ *	  provide the applicable stack frame offset, if any.
+ * @cfa_fixed_ra_offset: Offset for the Return Address from CFA is fixed for
+ *	  some ABIs. When fixed, this field specifies the fixed stack frame
+ *	  offset and the individual FREs do not need to track it. When not
+ *	  fixed, it is set to SFRAME_CFA_FIXED_FP_INVALID.
+ * @auxhdr_len: Number of bytes making up the auxiliary header, if any.
+ *	  Some ABI/arch, in the future, may use this space for extending the
+ *	  information in SFrame header. Auxiliary header is contained in bytes
+ *	  sequentially following the sframe_header.
+ * @num_fdes: Number of SFrame FDEs in this SFrame section.
+ * @num_fres: Number of SFrame Frame Row Entries.
+ * @fre_len:  Number of bytes in the SFrame Frame Row Entry section.
+ * @fdes_off: Offset of SFrame Function Descriptor Entry section.
+ * @fres_off: Offset of SFrame Frame Row Entry section.
+ */
+struct sframe_header {
+	struct sframe_preamble preamble;
+	u8  abi_arch;
+	s8  cfa_fixed_fp_offset;
+	s8  cfa_fixed_ra_offset;
+	u8  auxhdr_len;
+	u32 num_fdes;
+	u32 num_fres;
+	u32 fre_len;
+	u32 fdes_off;
+	u32 fres_off;
+} __packed;
+
+#define SFRAME_HDR_SIZE(sframe_hdr)	\
+	((sizeof(struct sframe_header) + (sframe_hdr).auxhdr_len))
+
+/* Two possible keys for executable (instruction) pointers signing. */
+#define SFRAME_AARCH64_PAUTH_KEY_A    0 /* Key A. */
+#define SFRAME_AARCH64_PAUTH_KEY_B    1 /* Key B. */
+
+/**
+ * struct sframe_fde - SFrame Function Descriptor Entry.
+ * @start_addr: Function start address. Encoded as a signed offset,
+ *	  relative to the current FDE.
+ * @size: Size of the function in bytes.
+ * @fres_off: Offset of the first SFrame Frame Row Entry of the function,
+ *	  relative to the beginning of the SFrame Frame Row Entry sub-section.
+ * @fres_num: Number of frame row entries for the function.
+ * @info: Additional information for deciphering the stack trace
+ *	  information for the function. Contains information about SFrame FRE
+ *	  type, SFrame FDE type, PAC authorization A/B key, etc.
+ * @rep_size: Block size for SFRAME_FDE_TYPE_PCMASK
+ * @padding: Unused
+ */
+struct sframe_fde {
+	s32 start_addr;
+	u32 size;
+	u32 fres_off;
+	u32 fres_num;
+	u8  info;
+	u8  rep_size;
+	u16 padding;
+} __packed;
+
+/*
+ * 'func_info' in SFrame FDE contains additional information for deciphering
+ * the stack trace information for the function. In V1, the information is
+ * organized as follows:
+ *   - 4-bits: Identify the FRE type used for the function.
+ *   - 1-bit: Identify the FDE type of the function - mask or inc.
+ *   - 1-bit: PAC authorization A/B key (aarch64).
+ *   - 2-bits: Unused.
+ * ---------------------------------------------------------------------
+ * |  Unused  |  PAC auth A/B key (aarch64) |  FDE type |   FRE type   |
+ * |          |        Unused (amd64)       |           |              |
+ * ---------------------------------------------------------------------
+ * 8          6                             5           4              0
+ */
+
+/* Note: Set PAC auth key to SFRAME_AARCH64_PAUTH_KEY_A by default.  */
+#define SFRAME_FUNC_INFO(fde_type, fre_enc_type) \
+	(((SFRAME_AARCH64_PAUTH_KEY_A & 0x1) << 5) | \
+	 (((fde_type) & 0x1) << 4) | ((fre_enc_type) & 0xf))
+
+#define SFRAME_FUNC_FRE_TYPE(data)	  ((data) & 0xf)
+#define SFRAME_FUNC_FDE_TYPE(data)	  (((data) >> 4) & 0x1)
+#define SFRAME_FUNC_PAUTH_KEY(data)	  (((data) >> 5) & 0x1)
+
+/*
+ * Size of stack frame offsets in an SFrame Frame Row Entry. A single
+ * SFrame FRE has all offsets of the same size. Offset size may vary
+ * across frame row entries.
+ */
+#define SFRAME_FRE_OFFSET_1B	  0
+#define SFRAME_FRE_OFFSET_2B	  1
+#define SFRAME_FRE_OFFSET_4B	  2
+
+/* An SFrame Frame Row Entry can be SP or FP based.  */
+#define SFRAME_BASE_REG_FP	0
+#define SFRAME_BASE_REG_SP	1
+
+/*
+ * The index at which a specific offset is presented in the variable length
+ * bytes of an FRE.
+ */
+#define SFRAME_FRE_CFA_OFFSET_IDX   0
+/*
+ * The RA stack offset, if present, will always be at index 1 in the variable
+ * length bytes of the FRE.
+ */
+#define SFRAME_FRE_RA_OFFSET_IDX    1
+/*
+ * The FP stack offset may appear at offset 1 or 2, depending on the ABI as RA
+ * may or may not be tracked.
+ */
+#define SFRAME_FRE_FP_OFFSET_IDX    2
+
+/*
+ * 'fre_info' in SFrame FRE contains information about:
+ *   - 1 bit: base reg for CFA
+ *   - 4 bits: Number of offsets (N). A value of up to 3 is allowed to track
+ *   all three of CFA, FP and RA (fixed implicit order).
+ *   - 2 bits: information about size of the offsets (S) in bytes.
+ *     Valid values are SFRAME_FRE_OFFSET_1B, SFRAME_FRE_OFFSET_2B,
+ *     SFRAME_FRE_OFFSET_4B
+ *   - 1 bit: Mangled RA state bit (aarch64 only).
+ * ---------------------------------------------------------------
+ * | Mangled-RA (aarch64) |  Size of   |   Number of  | base_reg |
+ * |  Unused (amd64)      |  offsets   |    offsets   |          |
+ * ---------------------------------------------------------------
+ * 8                      7             5             1          0
+ */
+
+/* Note: Set mangled_ra_p to zero by default. */
+#define SFRAME_FRE_INFO(base_reg_id, offset_num, offset_size) \
+	(((0 & 0x1) << 7) | (((offset_size) & 0x3) << 5) | \
+	 (((offset_num) & 0xf) << 1) | ((base_reg_id) & 0x1))
+
+/* Set the mangled_ra_p bit as indicated. */
+#define SFRAME_FRE_INFO_UPDATE_MANGLED_RA_P(mangled_ra_p, fre_info) \
+	((((mangled_ra_p) & 0x1) << 7) | ((fre_info) & 0x7f))
+
+#define SFRAME_FRE_CFA_BASE_REG_ID(data)	  ((data) & 0x1)
+#define SFRAME_FRE_OFFSET_COUNT(data)		  (((data) >> 1) & 0xf)
+#define SFRAME_FRE_OFFSET_SIZE(data)		  (((data) >> 5) & 0x3)
+#define SFRAME_FRE_MANGLED_RA_P(data)		  (((data) >> 7) & 0x1)
+
+#endif /* _SFRAME_H */
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 3/8] unwind: add sframe v2 header

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> Add sframe header so that we know how to access the sframe section
> generated by compilers.
>
> This is the sframe header file borrowed from the patchset [1]
> Josh Poimboeuf according to sframe v2 spec [2].
>
> [1]: https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
> [2]: https://sourceware.org/binutils/docs/sframe-spec.html
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   kernel/sframe.h | 215 ++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 215 insertions(+)
>   create mode 100644 kernel/sframe.h
>
> diff --git a/kernel/sframe.h b/kernel/sframe.h
> new file mode 100644
> index 000000000000..11b9be7ad82e
> --- /dev/null
> +++ b/kernel/sframe.h
> @@ -0,0 +1,215 @@
> +/* SPDX-License-Identifier: GPL-2.0-or-later */
> +/*
> + * Copyright (C) 2023, Oracle and/or its affiliates.
> + *
> + * This file contains definitions for the SFrame stack tracing format, which is
> + * documented at https://sourceware.org/binutils/docs
> + */
> +#ifndef _SFRAME_H
> +#define _SFRAME_H
> +
> +#include <linux/types.h>
> +
> +#define SFRAME_VERSION_1	1
> +#define SFRAME_VERSION_2	2
> +#define SFRAME_MAGIC		0xdee2
> +
> +/* Function Descriptor Entries are sorted on PC. */
> +#define SFRAME_F_FDE_SORTED	0x1
> +/* Frame-pointer based stack tracing. Defined, but not set. */
> +#define SFRAME_F_FRAME_POINTER	0x2
> +
> +#define SFRAME_CFA_FIXED_FP_INVALID 0
> +#define SFRAME_CFA_FIXED_RA_INVALID 0
> +
> +/* Supported ABIs/Arch. */
> +#define SFRAME_ABI_AARCH64_ENDIAN_BIG	    1 /* AARCH64 big endian. */
> +#define SFRAME_ABI_AARCH64_ENDIAN_LITTLE    2 /* AARCH64 little endian. */
> +#define SFRAME_ABI_AMD64_ENDIAN_LITTLE	    3 /* AMD64 little endian. */
> +
> +/* SFrame FRE types. */
> +#define SFRAME_FRE_TYPE_ADDR1	0
> +#define SFRAME_FRE_TYPE_ADDR2	1
> +#define SFRAME_FRE_TYPE_ADDR4	2
> +
> +/*
> + * SFrame Function Descriptor Entry types.
> + *
> + * The SFrame format has two possible representations for functions. The
> + * choice of which type to use is made according to the instruction patterns
> + * in the relevant program stub.
> + */
> +
> +/* Unwinders perform a (PC >= FRE_START_ADDR) to look up a matching FRE. */
> +#define SFRAME_FDE_TYPE_PCINC	0
> +/*
> + * Unwinders perform a (PC & FRE_START_ADDR_AS_MASK >= FRE_START_ADDR_AS_MASK)
> + * to look up a matching FRE. Typical usecases are pltN entries, trampolines
> + * etc.
> + */
> +#define SFRAME_FDE_TYPE_PCMASK	1
> +
> +/**
> + * struct sframe_preamble - SFrame Preamble.
> + * @magic: Magic number (SFRAME_MAGIC).
> + * @version: Format version number (SFRAME_VERSION).
> + * @flags: Various flags.
> + */
> +struct sframe_preamble {
> +	u16 magic;
> +	u8  version;
> +	u8  flags;
> +} __packed;
> +
> +/**
> + * struct sframe_header - SFrame Header.
> + * @preamble: SFrame preamble.
> + * @abi_arch: Identify the arch (including endianness) and ABI.
> + * @cfa_fixed_fp_offset: Offset for the Frame Pointer (FP) from CFA may be
> + *	  fixed for some ABIs ((e.g, in AMD64 when -fno-omit-frame-pointer is
> + *	  used). When fixed, this field specifies the fixed stack frame offset
> + *	  and the individual FREs do not need to track it. When not fixed, it
> + *	  is set to SFRAME_CFA_FIXED_FP_INVALID, and the individual FREs may
> + *	  provide the applicable stack frame offset, if any.
> + * @cfa_fixed_ra_offset: Offset for the Return Address from CFA is fixed for
> + *	  some ABIs. When fixed, this field specifies the fixed stack frame
> + *	  offset and the individual FREs do not need to track it. When not
> + *	  fixed, it is set to SFRAME_CFA_FIXED_FP_INVALID.
> + * @auxhdr_len: Number of bytes making up the auxiliary header, if any.
> + *	  Some ABI/arch, in the future, may use this space for extending the
> + *	  information in SFrame header. Auxiliary header is contained in bytes
> + *	  sequentially following the sframe_header.
> + * @num_fdes: Number of SFrame FDEs in this SFrame section.
> + * @num_fres: Number of SFrame Frame Row Entries.
> + * @fre_len:  Number of bytes in the SFrame Frame Row Entry section.
> + * @fdes_off: Offset of SFrame Function Descriptor Entry section.
> + * @fres_off: Offset of SFrame Frame Row Entry section.
> + */
> +struct sframe_header {
> +	struct sframe_preamble preamble;
> +	u8  abi_arch;
> +	s8  cfa_fixed_fp_offset;
> +	s8  cfa_fixed_ra_offset;
> +	u8  auxhdr_len;
> +	u32 num_fdes;
> +	u32 num_fres;
> +	u32 fre_len;
> +	u32 fdes_off;
> +	u32 fres_off;
> +} __packed;
> +
> +#define SFRAME_HDR_SIZE(sframe_hdr)	\
> +	((sizeof(struct sframe_header) + (sframe_hdr).auxhdr_len))
> +
> +/* Two possible keys for executable (instruction) pointers signing. */
> +#define SFRAME_AARCH64_PAUTH_KEY_A    0 /* Key A. */
> +#define SFRAME_AARCH64_PAUTH_KEY_B    1 /* Key B. */
> +
> +/**
> + * struct sframe_fde - SFrame Function Descriptor Entry.
> + * @start_addr: Function start address. Encoded as a signed offset,
> + *	  relative to the current FDE.
> + * @size: Size of the function in bytes.
> + * @fres_off: Offset of the first SFrame Frame Row Entry of the function,
> + *	  relative to the beginning of the SFrame Frame Row Entry sub-section.
> + * @fres_num: Number of frame row entries for the function.
> + * @info: Additional information for deciphering the stack trace
> + *	  information for the function. Contains information about SFrame FRE
> + *	  type, SFrame FDE type, PAC authorization A/B key, etc.
> + * @rep_size: Block size for SFRAME_FDE_TYPE_PCMASK
> + * @padding: Unused
> + */
> +struct sframe_fde {
> +	s32 start_addr;
> +	u32 size;
> +	u32 fres_off;
> +	u32 fres_num;
> +	u8  info;
> +	u8  rep_size;
> +	u16 padding;
> +} __packed;
> +
> +/*
> + * 'func_info' in SFrame FDE contains additional information for deciphering
> + * the stack trace information for the function. In V1, the information is
> + * organized as follows:
> + *   - 4-bits: Identify the FRE type used for the function.
> + *   - 1-bit: Identify the FDE type of the function - mask or inc.
> + *   - 1-bit: PAC authorization A/B key (aarch64).
> + *   - 2-bits: Unused.
> + * ---------------------------------------------------------------------
> + * |  Unused  |  PAC auth A/B key (aarch64) |  FDE type |   FRE type   |
> + * |          |        Unused (amd64)       |           |              |
> + * ---------------------------------------------------------------------
> + * 8          6                             5           4              0
> + */
> +
> +/* Note: Set PAC auth key to SFRAME_AARCH64_PAUTH_KEY_A by default.  */
> +#define SFRAME_FUNC_INFO(fde_type, fre_enc_type) \
> +	(((SFRAME_AARCH64_PAUTH_KEY_A & 0x1) << 5) | \
> +	 (((fde_type) & 0x1) << 4) | ((fre_enc_type) & 0xf))
> +
> +#define SFRAME_FUNC_FRE_TYPE(data)	  ((data) & 0xf)
> +#define SFRAME_FUNC_FDE_TYPE(data)	  (((data) >> 4) & 0x1)
> +#define SFRAME_FUNC_PAUTH_KEY(data)	  (((data) >> 5) & 0x1)
> +
> +/*
> + * Size of stack frame offsets in an SFrame Frame Row Entry. A single
> + * SFrame FRE has all offsets of the same size. Offset size may vary
> + * across frame row entries.
> + */
> +#define SFRAME_FRE_OFFSET_1B	  0
> +#define SFRAME_FRE_OFFSET_2B	  1
> +#define SFRAME_FRE_OFFSET_4B	  2
> +
> +/* An SFrame Frame Row Entry can be SP or FP based.  */
> +#define SFRAME_BASE_REG_FP	0
> +#define SFRAME_BASE_REG_SP	1
> +
> +/*
> + * The index at which a specific offset is presented in the variable length
> + * bytes of an FRE.
> + */
> +#define SFRAME_FRE_CFA_OFFSET_IDX   0
> +/*
> + * The RA stack offset, if present, will always be at index 1 in the variable
> + * length bytes of the FRE.
> + */
> +#define SFRAME_FRE_RA_OFFSET_IDX    1
> +/*
> + * The FP stack offset may appear at offset 1 or 2, depending on the ABI as RA
> + * may or may not be tracked.
> + */
> +#define SFRAME_FRE_FP_OFFSET_IDX    2
> +
> +/*
> + * 'fre_info' in SFrame FRE contains information about:
> + *   - 1 bit: base reg for CFA
> + *   - 4 bits: Number of offsets (N). A value of up to 3 is allowed to track
> + *   all three of CFA, FP and RA (fixed implicit order).
> + *   - 2 bits: information about size of the offsets (S) in bytes.
> + *     Valid values are SFRAME_FRE_OFFSET_1B, SFRAME_FRE_OFFSET_2B,
> + *     SFRAME_FRE_OFFSET_4B
> + *   - 1 bit: Mangled RA state bit (aarch64 only).
> + * ---------------------------------------------------------------
> + * | Mangled-RA (aarch64) |  Size of   |   Number of  | base_reg |
> + * |  Unused (amd64)      |  offsets   |    offsets   |          |
> + * ---------------------------------------------------------------
> + * 8                      7             5             1          0
> + */
> +
> +/* Note: Set mangled_ra_p to zero by default. */
> +#define SFRAME_FRE_INFO(base_reg_id, offset_num, offset_size) \
> +	(((0 & 0x1) << 7) | (((offset_size) & 0x3) << 5) | \
> +	 (((offset_num) & 0xf) << 1) | ((base_reg_id) & 0x1))
> +
> +/* Set the mangled_ra_p bit as indicated. */
> +#define SFRAME_FRE_INFO_UPDATE_MANGLED_RA_P(mangled_ra_p, fre_info) \
> +	((((mangled_ra_p) & 0x1) << 7) | ((fre_info) & 0x7f))
> +
> +#define SFRAME_FRE_CFA_BASE_REG_ID(data)	  ((data) & 0x1)
> +#define SFRAME_FRE_OFFSET_COUNT(data)		  (((data) >> 1) & 0xf)
> +#define SFRAME_FRE_OFFSET_SIZE(data)		  (((data) >> 5) & 0x3)
> +#define SFRAME_FRE_MANGLED_RA_P(data)		  (((data) >> 7) & 0x1)
> +
> +#endif /* _SFRAME_H */

Looks good to me.

Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

Re: [PATCH 3/8] unwind: add sframe v2 header

[Josh Poimboeuf]

On Mon, Jan 27, 2025 at 09:33:05PM +0000, Weinan Liu wrote:
> Add sframe header so that we know how to access the sframe section
> generated by compilers.
> 
> This is the sframe header file borrowed from the patchset [1]
> Josh Poimboeuf according to sframe v2 spec [2].
> 
> [1]: https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
> [2]: https://sourceware.org/binutils/docs/sframe-spec.html

FWIW, my v4 patchset has a more streamlined version of this file.

  https://lore.kernel.org/f27e8463783febfa0dabb0432a3dd6be8ad98412.1737511963.git.jpoimboe@kernel.org

-- 
Josh

[PATCH 4/8] unwind: Implement generic sframe unwinder library

[Weinan Liu]

This change introduces a kernel space unwinder using sframe table for
architectures without ORC unwinder support.

The implementation is adapted from Josh's userspace sframe unwinder
proposal[1] according to the sframe v2 spec[2].

[1] https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
[2] https://sourceware.org/binutils/docs/sframe-spec.html

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 include/linux/sframe_lookup.h |  43 ++++++++
 kernel/Makefile               |   1 +
 kernel/sframe_lookup.c        | 196 ++++++++++++++++++++++++++++++++++
 3 files changed, 240 insertions(+)
 create mode 100644 include/linux/sframe_lookup.h
 create mode 100644 kernel/sframe_lookup.c

diff --git a/include/linux/sframe_lookup.h b/include/linux/sframe_lookup.h
new file mode 100644
index 000000000000..1c26cf1f38d4
--- /dev/null
+++ b/include/linux/sframe_lookup.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_SFRAME_LOOKUP_H
+#define _LINUX_SFRAME_LOOKUP_H
+
+/**
+ * struct sframe_ip_entry - sframe unwind info for given ip
+ * @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame
+ *              Pointer(FP) or Stack Pointer(SP)
+ * @ra_offset: Offset for the Return Address from CFA.
+ * @fp_offset: Offset for the Frame Pointer (FP) from CFA.
+ * @use_fp: Use FP to get next CFA or not
+ */
+struct sframe_ip_entry {
+	int32_t cfa_offset;
+	int32_t ra_offset;
+	int32_t fp_offset;
+	bool use_fp;
+};
+
+/**
+ * struct sframe_table - sframe struct of a table
+ * @sfhdr_p: Pointer to sframe header
+ * @fde_p: Pointer to the first of sframe frame description entry(FDE).
+ * @fre_p: Pointer to the first of sframe frame row entry(FRE).
+ */
+struct sframe_table {
+	struct sframe_header *sfhdr_p;
+	struct sframe_fde *fde_p;
+	char *fre_p;
+};
+
+#ifdef CONFIG_SFRAME_UNWINDER
+void init_sframe_table(void);
+int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry);
+#else
+static inline void init_sframe_table(void) {}
+static inline int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry)
+{
+	return -EINVAL;
+}
+#endif
+
+#endif /* _LINUX_SFRAME_LOOKUP_H */
diff --git a/kernel/Makefile b/kernel/Makefile
index 87866b037fbe..705c9277e5cd 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -131,6 +131,7 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue.o
 
 obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o
 obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o
+obj-$(CONFIG_SFRAME_UNWINDER) += sframe_lookup.o
 
 CFLAGS_stackleak.o += $(DISABLE_STACKLEAK_PLUGIN)
 obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
new file mode 100644
index 000000000000..846f1da95d89
--- /dev/null
+++ b/kernel/sframe_lookup.c
@@ -0,0 +1,196 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/module.h>
+#include <linux/sort.h>
+#include <linux/sframe_lookup.h>
+#include <linux/kallsyms.h>
+#include "sframe.h"
+
+#define pr_fmt(fmt)	"sframe: " fmt
+
+extern char __start_sframe_header[];
+extern char __stop_sframe_header[];
+
+static bool sframe_init __ro_after_init;
+static struct sframe_table sftbl;
+
+#define SFRAME_READ_TYPE(in, out, type)					\
+({									\
+	type __tmp;							\
+	memcpy(&__tmp, in, sizeof(__tmp));				\
+	in += sizeof(__tmp);						\
+	out = __tmp;							\
+})
+
+#define SFRAME_READ_ROW_ADDR(in_addr, out_addr, type)			\
+({									\
+	switch (type) {							\
+	case SFRAME_FRE_TYPE_ADDR1:					\
+		SFRAME_READ_TYPE(in_addr, out_addr, u8);		\
+		break;							\
+	case SFRAME_FRE_TYPE_ADDR2:					\
+		SFRAME_READ_TYPE(in_addr, out_addr, u16);		\
+		break;							\
+	case SFRAME_FRE_TYPE_ADDR4:					\
+		SFRAME_READ_TYPE(in_addr, out_addr, u32);		\
+		break;							\
+	default:							\
+		break;							\
+	}								\
+})
+
+#define SFRAME_READ_ROW_OFFSETS(in_addr, out_addr, size)		\
+({									\
+	switch (size) {							\
+	case 1:								\
+		SFRAME_READ_TYPE(in_addr, out_addr, s8);		\
+		break;							\
+	case 2:								\
+		SFRAME_READ_TYPE(in_addr, out_addr, s16);		\
+		break;							\
+	case 4:								\
+		SFRAME_READ_TYPE(in_addr, out_addr, s32);		\
+		break;							\
+	default:							\
+		break;							\
+	}								\
+})
+
+static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned long pc)
+{
+	int l, r, m, f;
+	int32_t ip;
+	struct sframe_fde *fdep;
+
+	if (!tbl || !tbl->sfhdr_p || !tbl->fde_p)
+		return NULL;
+
+	ip = (pc - (unsigned long)tbl->sfhdr_p);
+
+	/* Do a binary range search to find the rightmost FDE start_addr < ip */
+	l = m = f = 0;
+	r = tbl->sfhdr_p->num_fdes;
+	while (l < r) {
+		m = l + ((r - l) / 2);
+		fdep = tbl->fde_p + m;
+		if (fdep->start_addr > ip)
+			r = m;
+		else
+			l = m + 1;
+	}
+	/* use l - 1 because l will be the first item fdep->start_addr > ip */
+	f = l - 1;
+	if (f >= tbl->sfhdr_p->num_fdes || f < 0)
+		return NULL;
+	fdep = tbl->fde_p + f;
+	if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
+		return NULL;
+
+	return fdep;
+}
+
+static int find_fre(const struct sframe_table *tbl, unsigned long pc,
+		const struct sframe_fde *fdep, struct sframe_ip_entry *entry)
+{
+	int i, offset_size, offset_count;
+	char *fres, *offsets_loc;
+	int32_t ip_off;
+	uint32_t next_row_ip_off;
+	uint8_t fre_info, fde_type = SFRAME_FUNC_FDE_TYPE(fdep->info),
+			fre_type = SFRAME_FUNC_FRE_TYPE(fdep->info);
+
+	fres = tbl->fre_p + fdep->fres_off;
+
+	/*  Whether PCs in the FREs should be treated as masks or not */
+	if (fde_type == SFRAME_FDE_TYPE_PCMASK)
+		ip_off = pc % fdep->rep_size;
+	else
+		ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - fdep->start_addr;
+
+	if (ip_off < 0 || ip_off >= fdep->size)
+		return -EINVAL;
+
+	/*
+	 * FRE structure starts by address of the entry with variants length. Use
+	 * two pointers to track current head(fres) and the address of last
+	 * offset(offsets_loc)
+	 */
+	for (i = 0; i < fdep->fres_num; i++) {
+		SFRAME_READ_ROW_ADDR(fres, next_row_ip_off, fre_type);
+		if (ip_off < next_row_ip_off)
+			break;
+		SFRAME_READ_TYPE(fres, fre_info, u8);
+		offsets_loc = fres;
+		/*
+		 * jump to the start of next fre
+		 * fres += fre_offets_cnt*offset_size
+		 */
+		fres += SFRAME_FRE_OFFSET_COUNT(fre_info) << SFRAME_FRE_OFFSET_SIZE(fre_info);
+	}
+
+	offset_size = 1 << SFRAME_FRE_OFFSET_SIZE(fre_info);
+	offset_count = SFRAME_FRE_OFFSET_COUNT(fre_info);
+
+	if (offset_count > 0) {
+		SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->cfa_offset, offset_size);
+		offset_count--;
+	}
+	if (offset_count > 0 && !entry->ra_offset) {
+		SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->ra_offset, offset_size);
+		offset_count--;
+	}
+	if (offset_count > 0 && !entry->fp_offset) {
+		SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->fp_offset, offset_size);
+		offset_count--;
+	}
+	if (offset_count)
+		return -EINVAL;
+
+	entry->use_fp = SFRAME_FRE_CFA_BASE_REG_ID(fre_info) == SFRAME_BASE_REG_FP;
+
+	return 0;
+}
+
+int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry)
+{
+	struct sframe_fde *fdep;
+	struct sframe_table *sftbl_p = &sftbl;
+	int err;
+
+	if (!sframe_init)
+		return -EINVAL;
+
+	memset(entry, 0, sizeof(*entry));
+	entry->ra_offset = sftbl_p->sfhdr_p->cfa_fixed_ra_offset;
+	entry->fp_offset = sftbl_p->sfhdr_p->cfa_fixed_fp_offset;
+
+	fdep = find_fde(sftbl_p, pc);
+	if (!fdep)
+		return -EINVAL;
+	err = find_fre(sftbl_p, pc, fdep, entry);
+	if (err)
+		return err;
+
+	return 0;
+}
+
+void __init init_sframe_table(void)
+{
+	size_t sframe_size = (void *)__stop_sframe_header - (void *)__start_sframe_header;
+	void *sframe_buf = __start_sframe_header;
+
+	if (sframe_size <= 0)
+		return;
+	sftbl.sfhdr_p = sframe_buf;
+	if (!sftbl.sfhdr_p || sftbl.sfhdr_p->preamble.magic != SFRAME_MAGIC ||
+	    sftbl.sfhdr_p->preamble.version != SFRAME_VERSION_2 ||
+	    !(sftbl.sfhdr_p->preamble.flags & SFRAME_F_FDE_SORTED)) {
+		pr_warn("WARNING: Unable to read sframe header.  Disabling unwinder.\n");
+		return;
+	}
+
+	sftbl.fde_p = (struct sframe_fde *)(__start_sframe_header + SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
+						+ sftbl.sfhdr_p->fdes_off);
+	sftbl.fre_p = __start_sframe_header + SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
+		+ sftbl.sfhdr_p->fres_off;
+	sframe_init = true;
+}
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> This change introduces a kernel space unwinder using sframe table for
> architectures without ORC unwinder support.
>
> The implementation is adapted from Josh's userspace sframe unwinder
> proposal[1] according to the sframe v2 spec[2].
>
> [1] https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
> [2] https://sourceware.org/binutils/docs/sframe-spec.html
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   include/linux/sframe_lookup.h |  43 ++++++++
>   kernel/Makefile               |   1 +
>   kernel/sframe_lookup.c        | 196 ++++++++++++++++++++++++++++++++++
>   3 files changed, 240 insertions(+)
>   create mode 100644 include/linux/sframe_lookup.h
>   create mode 100644 kernel/sframe_lookup.c
>
> diff --git a/include/linux/sframe_lookup.h b/include/linux/sframe_lookup.h
> new file mode 100644
> index 000000000000..1c26cf1f38d4
> --- /dev/null
> +++ b/include/linux/sframe_lookup.h
> @@ -0,0 +1,43 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef _LINUX_SFRAME_LOOKUP_H
> +#define _LINUX_SFRAME_LOOKUP_H
> +
> +/**
> + * struct sframe_ip_entry - sframe unwind info for given ip
> + * @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame
> + *              Pointer(FP) or Stack Pointer(SP)
> + * @ra_offset: Offset for the Return Address from CFA.
> + * @fp_offset: Offset for the Frame Pointer (FP) from CFA.
> + * @use_fp: Use FP to get next CFA or not
> + */
> +struct sframe_ip_entry {
> +	int32_t cfa_offset;
> +	int32_t ra_offset;

The ra_offset is not present for x86_64 in SFrame FRE as per the spec. I 
am wondering whether this struct should change based on the architecture 
or just set ra_offset calculated from cfa_offset for x86_64.

> +	int32_t fp_offset;
> +	bool use_fp;
> +};
> +
> +/**
> + * struct sframe_table - sframe struct of a table
> + * @sfhdr_p: Pointer to sframe header
> + * @fde_p: Pointer to the first of sframe frame description entry(FDE).
> + * @fre_p: Pointer to the first of sframe frame row entry(FRE).
> + */
> +struct sframe_table {
> +	struct sframe_header *sfhdr_p;
> +	struct sframe_fde *fde_p;
> +	char *fre_p;
> +};
> +
> +#ifdef CONFIG_SFRAME_UNWINDER
> +void init_sframe_table(void);
> +int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry);
> +#else
> +static inline void init_sframe_table(void) {}
> +static inline int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry)
> +{
> +	return -EINVAL;
> +}
> +#endif
> +
> +#endif /* _LINUX_SFRAME_LOOKUP_H */
> diff --git a/kernel/Makefile b/kernel/Makefile
> index 87866b037fbe..705c9277e5cd 100644
> --- a/kernel/Makefile
> +++ b/kernel/Makefile
> @@ -131,6 +131,7 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue.o
>   
>   obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o
>   obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o
> +obj-$(CONFIG_SFRAME_UNWINDER) += sframe_lookup.o
>   
>   CFLAGS_stackleak.o += $(DISABLE_STACKLEAK_PLUGIN)
>   obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
> diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
> new file mode 100644
> index 000000000000..846f1da95d89
> --- /dev/null
> +++ b/kernel/sframe_lookup.c
> @@ -0,0 +1,196 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +#include <linux/module.h>
> +#include <linux/sort.h>
> +#include <linux/sframe_lookup.h>
> +#include <linux/kallsyms.h>
> +#include "sframe.h"
> +
> +#define pr_fmt(fmt)	"sframe: " fmt
> +
> +extern char __start_sframe_header[];
> +extern char __stop_sframe_header[];
> +
> +static bool sframe_init __ro_after_init;
> +static struct sframe_table sftbl;
> +
> +#define SFRAME_READ_TYPE(in, out, type)					\
> +({									\
> +	type __tmp;							\
> +	memcpy(&__tmp, in, sizeof(__tmp));				\
> +	in += sizeof(__tmp);						\
> +	out = __tmp;							\
> +})
> +
> +#define SFRAME_READ_ROW_ADDR(in_addr, out_addr, type)			\
> +({									\
> +	switch (type) {							\
> +	case SFRAME_FRE_TYPE_ADDR1:					\
> +		SFRAME_READ_TYPE(in_addr, out_addr, u8);		\
> +		break;							\
> +	case SFRAME_FRE_TYPE_ADDR2:					\
> +		SFRAME_READ_TYPE(in_addr, out_addr, u16);		\
> +		break;							\
> +	case SFRAME_FRE_TYPE_ADDR4:					\
> +		SFRAME_READ_TYPE(in_addr, out_addr, u32);		\
> +		break;							\
> +	default:							\
> +		break;							\
> +	}								\
> +})
> +
> +#define SFRAME_READ_ROW_OFFSETS(in_addr, out_addr, size)		\
> +({									\
> +	switch (size) {							\
> +	case 1:								\
> +		SFRAME_READ_TYPE(in_addr, out_addr, s8);		\
> +		break;							\
> +	case 2:								\
> +		SFRAME_READ_TYPE(in_addr, out_addr, s16);		\
> +		break;							\
> +	case 4:								\
> +		SFRAME_READ_TYPE(in_addr, out_addr, s32);		\
> +		break;							\
> +	default:							\
> +		break;							\
> +	}								\
> +})
> +
> +static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned long pc)
> +{
> +	int l, r, m, f;
> +	int32_t ip;
> +	struct sframe_fde *fdep;
> +
> +	if (!tbl || !tbl->sfhdr_p || !tbl->fde_p)
> +		return NULL;
> +
> +	ip = (pc - (unsigned long)tbl->sfhdr_p);
> +
> +	/* Do a binary range search to find the rightmost FDE start_addr < ip */
> +	l = m = f = 0;
> +	r = tbl->sfhdr_p->num_fdes;
> +	while (l < r) {
> +		m = l + ((r - l) / 2);
> +		fdep = tbl->fde_p + m;
> +		if (fdep->start_addr > ip)
> +			r = m;
> +		else
> +			l = m + 1;
> +	}
> +	/* use l - 1 because l will be the first item fdep->start_addr > ip */
> +	f = l - 1;
> +	if (f >= tbl->sfhdr_p->num_fdes || f < 0)
> +		return NULL;
> +	fdep = tbl->fde_p + f;
> +	if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
> +		return NULL;
> +
> +	return fdep;
> +}
> +
> +static int find_fre(const struct sframe_table *tbl, unsigned long pc,
> +		const struct sframe_fde *fdep, struct sframe_ip_entry *entry)
> +{
> +	int i, offset_size, offset_count;
> +	char *fres, *offsets_loc;
> +	int32_t ip_off;
> +	uint32_t next_row_ip_off;
> +	uint8_t fre_info, fde_type = SFRAME_FUNC_FDE_TYPE(fdep->info),
> +			fre_type = SFRAME_FUNC_FRE_TYPE(fdep->info);
> +
> +	fres = tbl->fre_p + fdep->fres_off;
> +
> +	/*  Whether PCs in the FREs should be treated as masks or not */
> +	if (fde_type == SFRAME_FDE_TYPE_PCMASK)
> +		ip_off = pc % fdep->rep_size;
> +	else
> +		ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - fdep->start_addr;
> +
> +	if (ip_off < 0 || ip_off >= fdep->size)
> +		return -EINVAL;
> +
> +	/*
> +	 * FRE structure starts by address of the entry with variants length. Use
> +	 * two pointers to track current head(fres) and the address of last
> +	 * offset(offsets_loc)
> +	 */
> +	for (i = 0; i < fdep->fres_num; i++) {
> +		SFRAME_READ_ROW_ADDR(fres, next_row_ip_off, fre_type);
> +		if (ip_off < next_row_ip_off)
> +			break;
> +		SFRAME_READ_TYPE(fres, fre_info, u8);
> +		offsets_loc = fres;
> +		/*
> +		 * jump to the start of next fre
> +		 * fres += fre_offets_cnt*offset_size
> +		 */
> +		fres += SFRAME_FRE_OFFSET_COUNT(fre_info) << SFRAME_FRE_OFFSET_SIZE(fre_info);
> +	}
> +
> +	offset_size = 1 << SFRAME_FRE_OFFSET_SIZE(fre_info);
> +	offset_count = SFRAME_FRE_OFFSET_COUNT(fre_info);
> +
> +	if (offset_count > 0) {
> +		SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->cfa_offset, offset_size);
> +		offset_count--;
> +	}
> +	if (offset_count > 0 && !entry->ra_offset) {
> +		SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->ra_offset, offset_size);
> +		offset_count--;
> +	}
> +	if (offset_count > 0 && !entry->fp_offset) {
> +		SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->fp_offset, offset_size);
> +		offset_count--;
> +	}
> +	if (offset_count)
> +		return -EINVAL;
> +
> +	entry->use_fp = SFRAME_FRE_CFA_BASE_REG_ID(fre_info) == SFRAME_BASE_REG_FP;
> +
> +	return 0;
> +}
> +
> +int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry)
> +{
> +	struct sframe_fde *fdep;
> +	struct sframe_table *sftbl_p = &sftbl;
> +	int err;
> +
> +	if (!sframe_init)
> +		return -EINVAL;
> +
> +	memset(entry, 0, sizeof(*entry));
> +	entry->ra_offset = sftbl_p->sfhdr_p->cfa_fixed_ra_offset;
> +	entry->fp_offset = sftbl_p->sfhdr_p->cfa_fixed_fp_offset;
> +
> +	fdep = find_fde(sftbl_p, pc);
> +	if (!fdep)
> +		return -EINVAL;
> +	err = find_fre(sftbl_p, pc, fdep, entry);
> +	if (err)
> +		return err;
> +
> +	return 0;
> +}
> +
> +void __init init_sframe_table(void)
> +{
> +	size_t sframe_size = (void *)__stop_sframe_header - (void *)__start_sframe_header;
> +	void *sframe_buf = __start_sframe_header;
> +
> +	if (sframe_size <= 0)
> +		return;
> +	sftbl.sfhdr_p = sframe_buf;
> +	if (!sftbl.sfhdr_p || sftbl.sfhdr_p->preamble.magic != SFRAME_MAGIC ||
> +	    sftbl.sfhdr_p->preamble.version != SFRAME_VERSION_2 ||
> +	    !(sftbl.sfhdr_p->preamble.flags & SFRAME_F_FDE_SORTED)) {
> +		pr_warn("WARNING: Unable to read sframe header.  Disabling unwinder.\n");
> +		return;
> +	}
> +
> +	sftbl.fde_p = (struct sframe_fde *)(__start_sframe_header + SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
> +						+ sftbl.sfhdr_p->fdes_off);
> +	sftbl.fre_p = __start_sframe_header + SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
> +		+ sftbl.sfhdr_p->fres_off;
> +	sframe_init = true;
> +}
Other than the minor suggestion, the code looks good to me.

Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library

[Prasanna Kumar T S M]

On 30-01-2025 15:52, Prasanna Kumar T S M wrote:
>
> On 28-01-2025 03:03, Weinan Liu wrote:
>> This change introduces a kernel space unwinder using sframe table for
>> architectures without ORC unwinder support.
>>
>> The implementation is adapted from Josh's userspace sframe unwinder
>> proposal[1] according to the sframe v2 spec[2].
>>
>> [1] 
>> https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
>> [2] https://sourceware.org/binutils/docs/sframe-spec.html
>>
>> Signed-off-by: Weinan Liu <wnliu@google.com>
>> ---
>>   include/linux/sframe_lookup.h |  43 ++++++++
>>   kernel/Makefile               |   1 +
>>   kernel/sframe_lookup.c        | 196 ++++++++++++++++++++++++++++++++++
Nit: Can this file be placed inside lib/ instead of kernel/ folder?
>>   3 files changed, 240 insertions(+)
>>   create mode 100644 include/linux/sframe_lookup.h
>>   create mode 100644 kernel/sframe_lookup.c
>>
>> diff --git a/include/linux/sframe_lookup.h 
>> b/include/linux/sframe_lookup.h
>> new file mode 100644
>> index 000000000000..1c26cf1f38d4
>> --- /dev/null
>> +++ b/include/linux/sframe_lookup.h
>> @@ -0,0 +1,43 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef _LINUX_SFRAME_LOOKUP_H
>> +#define _LINUX_SFRAME_LOOKUP_H
>> +
>> +/**
>> + * struct sframe_ip_entry - sframe unwind info for given ip
>> + * @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame
>> + *              Pointer(FP) or Stack Pointer(SP)
>> + * @ra_offset: Offset for the Return Address from CFA.
>> + * @fp_offset: Offset for the Frame Pointer (FP) from CFA.
>> + * @use_fp: Use FP to get next CFA or not
>> + */
>> +struct sframe_ip_entry {
>> +    int32_t cfa_offset;
>> +    int32_t ra_offset;
>
> The ra_offset is not present for x86_64 in SFrame FRE as per the spec. 
> I am wondering whether this struct should change based on the 
> architecture or just set ra_offset calculated from cfa_offset for x86_64.
>
>> +    int32_t fp_offset;
>> +    bool use_fp;
>> +};
>> +
>> +/**
>> + * struct sframe_table - sframe struct of a table
>> + * @sfhdr_p: Pointer to sframe header
>> + * @fde_p: Pointer to the first of sframe frame description entry(FDE).
>> + * @fre_p: Pointer to the first of sframe frame row entry(FRE).
>> + */
>> +struct sframe_table {
>> +    struct sframe_header *sfhdr_p;
>> +    struct sframe_fde *fde_p;
>> +    char *fre_p;
>> +};
>> +
>> +#ifdef CONFIG_SFRAME_UNWINDER
>> +void init_sframe_table(void);
>> +int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry);
>> +#else
>> +static inline void init_sframe_table(void) {}
>> +static inline int sframe_find_pc(unsigned long pc, struct 
>> sframe_ip_entry *entry)
>> +{
>> +    return -EINVAL;
>> +}
>> +#endif
>> +
>> +#endif /* _LINUX_SFRAME_LOOKUP_H */
>> diff --git a/kernel/Makefile b/kernel/Makefile
>> index 87866b037fbe..705c9277e5cd 100644
>> --- a/kernel/Makefile
>> +++ b/kernel/Makefile
>> @@ -131,6 +131,7 @@ obj-$(CONFIG_WATCH_QUEUE) += watch_queue.o
>>     obj-$(CONFIG_RESOURCE_KUNIT_TEST) += resource_kunit.o
>>   obj-$(CONFIG_SYSCTL_KUNIT_TEST) += sysctl-test.o
>> +obj-$(CONFIG_SFRAME_UNWINDER) += sframe_lookup.o
>>     CFLAGS_stackleak.o += $(DISABLE_STACKLEAK_PLUGIN)
>>   obj-$(CONFIG_GCC_PLUGIN_STACKLEAK) += stackleak.o
>> diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
>> new file mode 100644
>> index 000000000000..846f1da95d89
>> --- /dev/null
>> +++ b/kernel/sframe_lookup.c
>> @@ -0,0 +1,196 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +#include <linux/module.h>
>> +#include <linux/sort.h>
>> +#include <linux/sframe_lookup.h>
>> +#include <linux/kallsyms.h>
>> +#include "sframe.h"
>> +
>> +#define pr_fmt(fmt)    "sframe: " fmt
>> +
>> +extern char __start_sframe_header[];
>> +extern char __stop_sframe_header[];
>> +
>> +static bool sframe_init __ro_after_init;
>> +static struct sframe_table sftbl;
>> +
>> +#define SFRAME_READ_TYPE(in, out, type)                    \
>> +({                                    \
>> +    type __tmp;                            \
>> +    memcpy(&__tmp, in, sizeof(__tmp));                \
>> +    in += sizeof(__tmp);                        \
>> +    out = __tmp;                            \
>> +})
>> +
>> +#define SFRAME_READ_ROW_ADDR(in_addr, out_addr, type)            \
>> +({                                    \
>> +    switch (type) {                            \
>> +    case SFRAME_FRE_TYPE_ADDR1:                    \
>> +        SFRAME_READ_TYPE(in_addr, out_addr, u8);        \
>> +        break;                            \
>> +    case SFRAME_FRE_TYPE_ADDR2:                    \
>> +        SFRAME_READ_TYPE(in_addr, out_addr, u16);        \
>> +        break;                            \
>> +    case SFRAME_FRE_TYPE_ADDR4:                    \
>> +        SFRAME_READ_TYPE(in_addr, out_addr, u32);        \
>> +        break;                            \
>> +    default:                            \
>> +        break;                            \
>> +    }                                \
>> +})
>> +
>> +#define SFRAME_READ_ROW_OFFSETS(in_addr, out_addr, size) \
>> +({                                    \
>> +    switch (size) {                            \
>> +    case 1:                                \
>> +        SFRAME_READ_TYPE(in_addr, out_addr, s8);        \
>> +        break;                            \
>> +    case 2:                                \
>> +        SFRAME_READ_TYPE(in_addr, out_addr, s16);        \
>> +        break;                            \
>> +    case 4:                                \
>> +        SFRAME_READ_TYPE(in_addr, out_addr, s32);        \
>> +        break;                            \
>> +    default:                            \
>> +        break;                            \
>> +    }                                \
>> +})
>> +
>> +static struct sframe_fde *find_fde(const struct sframe_table *tbl, 
>> unsigned long pc)
>> +{
>> +    int l, r, m, f;
>> +    int32_t ip;
>> +    struct sframe_fde *fdep;
>> +
>> +    if (!tbl || !tbl->sfhdr_p || !tbl->fde_p)
>> +        return NULL;
>> +
>> +    ip = (pc - (unsigned long)tbl->sfhdr_p);
>> +
>> +    /* Do a binary range search to find the rightmost FDE start_addr 
>> < ip */
>> +    l = m = f = 0;
>> +    r = tbl->sfhdr_p->num_fdes;
>> +    while (l < r) {
>> +        m = l + ((r - l) / 2);
>> +        fdep = tbl->fde_p + m;
>> +        if (fdep->start_addr > ip)
>> +            r = m;
>> +        else
>> +            l = m + 1;
>> +    }
>> +    /* use l - 1 because l will be the first item fdep->start_addr > 
>> ip */
>> +    f = l - 1;
>> +    if (f >= tbl->sfhdr_p->num_fdes || f < 0)
>> +        return NULL;
>> +    fdep = tbl->fde_p + f;
>> +    if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
>> +        return NULL;
>> +
>> +    return fdep;
>> +}
>> +
>> +static int find_fre(const struct sframe_table *tbl, unsigned long pc,
>> +        const struct sframe_fde *fdep, struct sframe_ip_entry *entry)
>> +{
>> +    int i, offset_size, offset_count;
>> +    char *fres, *offsets_loc;
>> +    int32_t ip_off;
>> +    uint32_t next_row_ip_off;
>> +    uint8_t fre_info, fde_type = SFRAME_FUNC_FDE_TYPE(fdep->info),
>> +            fre_type = SFRAME_FUNC_FRE_TYPE(fdep->info);
>> +
>> +    fres = tbl->fre_p + fdep->fres_off;
>> +
>> +    /*  Whether PCs in the FREs should be treated as masks or not */
>> +    if (fde_type == SFRAME_FDE_TYPE_PCMASK)
>> +        ip_off = pc % fdep->rep_size;
>> +    else
>> +        ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - 
>> fdep->start_addr;
>> +
>> +    if (ip_off < 0 || ip_off >= fdep->size)
>> +        return -EINVAL;
>> +
>> +    /*
>> +     * FRE structure starts by address of the entry with variants 
>> length. Use
>> +     * two pointers to track current head(fres) and the address of last
>> +     * offset(offsets_loc)
>> +     */
>> +    for (i = 0; i < fdep->fres_num; i++) {
>> +        SFRAME_READ_ROW_ADDR(fres, next_row_ip_off, fre_type);
>> +        if (ip_off < next_row_ip_off)
>> +            break;
>> +        SFRAME_READ_TYPE(fres, fre_info, u8);
>> +        offsets_loc = fres;
>> +        /*
>> +         * jump to the start of next fre
>> +         * fres += fre_offets_cnt*offset_size
>> +         */
>> +        fres += SFRAME_FRE_OFFSET_COUNT(fre_info) << 
>> SFRAME_FRE_OFFSET_SIZE(fre_info);
>> +    }
>> +
>> +    offset_size = 1 << SFRAME_FRE_OFFSET_SIZE(fre_info);
>> +    offset_count = SFRAME_FRE_OFFSET_COUNT(fre_info);
>> +
>> +    if (offset_count > 0) {
>> +        SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->cfa_offset, 
>> offset_size);
>> +        offset_count--;
>> +    }
>> +    if (offset_count > 0 && !entry->ra_offset) {
>> +        SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->ra_offset, 
>> offset_size);
>> +        offset_count--;
>> +    }
>> +    if (offset_count > 0 && !entry->fp_offset) {
>> +        SFRAME_READ_ROW_OFFSETS(offsets_loc, entry->fp_offset, 
>> offset_size);
>> +        offset_count--;
>> +    }
>> +    if (offset_count)
>> +        return -EINVAL;
>> +
>> +    entry->use_fp = SFRAME_FRE_CFA_BASE_REG_ID(fre_info) == 
>> SFRAME_BASE_REG_FP;
>> +
>> +    return 0;
>> +}
>> +
>> +int sframe_find_pc(unsigned long pc, struct sframe_ip_entry *entry)
>> +{
>> +    struct sframe_fde *fdep;
>> +    struct sframe_table *sftbl_p = &sftbl;
>> +    int err;
>> +
>> +    if (!sframe_init)
>> +        return -EINVAL;
>> +
>> +    memset(entry, 0, sizeof(*entry));
>> +    entry->ra_offset = sftbl_p->sfhdr_p->cfa_fixed_ra_offset;
>> +    entry->fp_offset = sftbl_p->sfhdr_p->cfa_fixed_fp_offset;
>> +
>> +    fdep = find_fde(sftbl_p, pc);
>> +    if (!fdep)
>> +        return -EINVAL;
>> +    err = find_fre(sftbl_p, pc, fdep, entry);
>> +    if (err)
>> +        return err;
>> +
>> +    return 0;
>> +}
>> +
>> +void __init init_sframe_table(void)
>> +{
>> +    size_t sframe_size = (void *)__stop_sframe_header - (void 
>> *)__start_sframe_header;
>> +    void *sframe_buf = __start_sframe_header;
>> +
>> +    if (sframe_size <= 0)
>> +        return;
>> +    sftbl.sfhdr_p = sframe_buf;
>> +    if (!sftbl.sfhdr_p || sftbl.sfhdr_p->preamble.magic != 
>> SFRAME_MAGIC ||
>> +        sftbl.sfhdr_p->preamble.version != SFRAME_VERSION_2 ||
>> +        !(sftbl.sfhdr_p->preamble.flags & SFRAME_F_FDE_SORTED)) {
>> +        pr_warn("WARNING: Unable to read sframe header. Disabling 
>> unwinder.\n");
>> +        return;
>> +    }
>> +
>> +    sftbl.fde_p = (struct sframe_fde *)(__start_sframe_header + 
>> SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
>> +                        + sftbl.sfhdr_p->fdes_off);
>> +    sftbl.fre_p = __start_sframe_header + 
>> SFRAME_HDR_SIZE(*sftbl.sfhdr_p)
>> +        + sftbl.sfhdr_p->fres_off;
>> +    sframe_init = true;
>> +}
> Other than the minor suggestion, the code looks good to me.
>
> Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.
>

Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library

[Weinan Liu]

On Thu, Jan 30, 2025 at 2:22 AM Prasanna Kumar T S M <ptsm@linux.microsoft.com> wrote:
> On 28-01-2025 03:03, Weinan Liu wrote:
> > diff --git a/include/linux/sframe_lookup.h b/include/linux/sframe_lookup.h
> > new file mode 100644
> > index 000000000000..1c26cf1f38d4
> > --- /dev/null
> > +++ b/include/linux/sframe_lookup.h
> > @@ -0,0 +1,43 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +#ifndef _LINUX_SFRAME_LOOKUP_H
> > +#define _LINUX_SFRAME_LOOKUP_H
> > +
> > +/**
> > + * struct sframe_ip_entry - sframe unwind info for given ip
> > + * @cfa_offset: Offset for the Canonical Frame Address(CFA) from Frame
> > + *              Pointer(FP) or Stack Pointer(SP)
> > + * @ra_offset: Offset for the Return Address from CFA.
> > + * @fp_offset: Offset for the Frame Pointer (FP) from CFA.
> > + * @use_fp: Use FP to get next CFA or not
> > + */
> > +struct sframe_ip_entry {
> > +	int32_t cfa_offset;
> > +	int32_t ra_offset;
> 
> The ra_offset is not present for x86_64 in SFrame FRE as per the spec. I 
> am wondering whether this struct should change based on the architecture 
> or just set ra_offset calculated from cfa_offset for x86_64.

According to the https://sourceware.org/binutils/docs/sframe-spec.html#AMD64
For x86_64, RA will be stored at a fixed offset from the CFA upon function entry. 
The ra_offset will set to sfhdr_p->cfa_fixed_ra_offset during the initialization
of the sframe_ip_entry within the function sframe_find_pc()

Re: [PATCH 4/8] unwind: Implement generic sframe unwinder library

[Weinan Liu]

On Thu, Jan 30, 2025 at 2:29 AM Prasanna Kumar T S M <ptsm@linux.microsoft.com> wrote:
> On 30-01-2025 15:52, Prasanna Kumar T S M wrote:
> >
> > On 28-01-2025 03:03, Weinan Liu wrote:
> > > This change introduces a kernel space unwinder using sframe table for
> > > architectures without ORC unwinder support.
> > >
> > > The implementation is adapted from Josh's userspace sframe unwinder
> > > proposal[1] according to the sframe v2 spec[2].
> > >
> > > [1]
> > > https://lore.kernel.org/lkml/42c0a99236af65c09c8182e260af7bcf5aa1e158.1730150953.git.jpoimboe@kernel.org/
> > > [2] https://sourceware.org/binutils/docs/sframe-spec.html
> > >
> > > Signed-off-by: Weinan Liu <wnliu@google.com>
> > > ---
> > >   include/linux/sframe_lookup.h |  43 ++++++++
> > >   kernel/Makefile               |   1 +
> > >   kernel/sframe_lookup.c        | 196 ++++++++++++++++++++++++++++++++++
> Nit: Can this file be placed inside lib/ instead of kernel/ folder?
This could be integrated with Josh's proposal in the future.
https://lore.kernel.org/lkml/cover.1737511963.git.jpoimboe@kernel.org/

Either lib/ or kernel/unwind/ are ok to me.

[PATCH 5/8] unwind: arm64: Add sframe unwinder on arm64

[Weinan Liu]

Add unwind_next_frame_sframe() function to unwind by sframe info.
Built with GNU Binutils 2.42 to verify that this sframe unwinder can
backtrace correctly on arm64.

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 arch/arm64/include/asm/stacktrace/common.h |  4 ++
 arch/arm64/kernel/setup.c                  |  2 +
 arch/arm64/kernel/stacktrace.c             | 59 ++++++++++++++++++++++
 3 files changed, 65 insertions(+)

diff --git a/arch/arm64/include/asm/stacktrace/common.h b/arch/arm64/include/asm/stacktrace/common.h
index 821a8fdd31af..19edae8a5b1a 100644
--- a/arch/arm64/include/asm/stacktrace/common.h
+++ b/arch/arm64/include/asm/stacktrace/common.h
@@ -25,6 +25,7 @@ struct stack_info {
  * @stack:       The stack currently being unwound.
  * @stacks:      An array of stacks which can be unwound.
  * @nr_stacks:   The number of stacks in @stacks.
+ * @cfa:         The sp value at the call site of the current function.
  */
 struct unwind_state {
 	unsigned long fp;
@@ -33,6 +34,9 @@ struct unwind_state {
 	struct stack_info stack;
 	struct stack_info *stacks;
 	int nr_stacks;
+#ifdef CONFIG_SFRAME_UNWINDER
+	unsigned long cfa;
+#endif
 };
 
 static inline struct stack_info stackinfo_get_unknown(void)
diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
index 4f613e8e0745..d3ac92b624f3 100644
--- a/arch/arm64/kernel/setup.c
+++ b/arch/arm64/kernel/setup.c
@@ -32,6 +32,7 @@
 #include <linux/sched/task.h>
 #include <linux/scs.h>
 #include <linux/mm.h>
+#include <linux/sframe_lookup.h>
 
 #include <asm/acpi.h>
 #include <asm/fixmap.h>
@@ -377,6 +378,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
 			"This indicates a broken bootloader or old kernel\n",
 			boot_args[1], boot_args[2], boot_args[3]);
 	}
+	init_sframe_table();
 }
 
 static inline bool cpu_can_disable(unsigned int cpu)
diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index 1d9d51d7627f..c035adb8fe8a 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -14,6 +14,7 @@
 #include <linux/sched/debug.h>
 #include <linux/sched/task_stack.h>
 #include <linux/stacktrace.h>
+#include <linux/sframe_lookup.h>
 
 #include <asm/efi.h>
 #include <asm/irq.h>
@@ -242,6 +243,53 @@ kunwind_next_frame_record(struct kunwind_state *state)
 	return 0;
 }
 
+#ifdef CONFIG_SFRAME_UNWINDER
+/*
+ * Unwind to the next frame according to sframe.
+ */
+static __always_inline int
+unwind_next_frame_sframe(struct unwind_state *state)
+{
+	unsigned long fp = state->fp, ip = state->pc;
+	unsigned long base_reg, cfa;
+	unsigned long pc_addr, fp_addr;
+	struct sframe_ip_entry entry;
+	struct stack_info *info;
+	struct frame_record *record = (struct frame_record *)fp;
+
+	int err;
+
+	/* frame record alignment 8 bytes */
+	if (fp & 0x7)
+		return -EINVAL;
+
+	info = unwind_find_stack(state, fp, sizeof(*record));
+	if (!info)
+		return -EINVAL;
+
+	err = sframe_find_pc(ip, &entry);
+	if (err)
+		return -EINVAL;
+
+	unwind_consume_stack(state, info, fp, sizeof(*record));
+
+	base_reg = entry.use_fp ? fp : state->cfa;
+
+	/* Set up the initial CFA using fp based info if CFA is not set */
+	if (!state->cfa)
+		cfa = fp - entry.fp_offset;
+	else
+		cfa = base_reg + entry.cfa_offset;
+	fp_addr = cfa + entry.fp_offset;
+	pc_addr = cfa + entry.ra_offset;
+	state->cfa = cfa;
+	state->fp = READ_ONCE(*(unsigned long *)(fp_addr));
+	state->pc = READ_ONCE(*(unsigned long *)(pc_addr));
+
+	return 0;
+}
+#endif
+
 /*
  * Unwind from one frame record (A) to the next frame record (B).
  *
@@ -261,7 +309,15 @@ kunwind_next(struct kunwind_state *state)
 	case KUNWIND_SOURCE_CALLER:
 	case KUNWIND_SOURCE_TASK:
 	case KUNWIND_SOURCE_REGS_PC:
+#ifdef CONFIG_SFRAME_UNWINDER
+	err = unwind_next_frame_sframe(&state->common);
+
+	/* Fallback to FP based unwinder */
+	if (err)
 		err = kunwind_next_frame_record(state);
+#else
+	err = kunwind_next_frame_record(state);
+#endif
 		break;
 	default:
 		err = -EINVAL;
@@ -347,6 +403,9 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,
 		.common = {
 			.stacks = stacks,
 			.nr_stacks = ARRAY_SIZE(stacks),
+#ifdef CONFIG_SFRAME_UNWINDER
+			.cfa = 0,
+#endif
 		},
 	};
 
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 5/8] unwind: arm64: Add sframe unwinder on arm64

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> Add unwind_next_frame_sframe() function to unwind by sframe info.
> Built with GNU Binutils 2.42 to verify that this sframe unwinder can
> backtrace correctly on arm64.
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   arch/arm64/include/asm/stacktrace/common.h |  4 ++
>   arch/arm64/kernel/setup.c                  |  2 +
>   arch/arm64/kernel/stacktrace.c             | 59 ++++++++++++++++++++++
>   3 files changed, 65 insertions(+)
>
> diff --git a/arch/arm64/include/asm/stacktrace/common.h b/arch/arm64/include/asm/stacktrace/common.h
> index 821a8fdd31af..19edae8a5b1a 100644
> --- a/arch/arm64/include/asm/stacktrace/common.h
> +++ b/arch/arm64/include/asm/stacktrace/common.h
> @@ -25,6 +25,7 @@ struct stack_info {
>    * @stack:       The stack currently being unwound.
>    * @stacks:      An array of stacks which can be unwound.
>    * @nr_stacks:   The number of stacks in @stacks.
> + * @cfa:         The sp value at the call site of the current function.
>    */
>   struct unwind_state {
>   	unsigned long fp;
> @@ -33,6 +34,9 @@ struct unwind_state {
>   	struct stack_info stack;
>   	struct stack_info *stacks;
>   	int nr_stacks;
> +#ifdef CONFIG_SFRAME_UNWINDER
> +	unsigned long cfa;
> +#endif
>   };
>   
>   static inline struct stack_info stackinfo_get_unknown(void)
> diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c
> index 4f613e8e0745..d3ac92b624f3 100644
> --- a/arch/arm64/kernel/setup.c
> +++ b/arch/arm64/kernel/setup.c
> @@ -32,6 +32,7 @@
>   #include <linux/sched/task.h>
>   #include <linux/scs.h>
>   #include <linux/mm.h>
> +#include <linux/sframe_lookup.h>
>   
>   #include <asm/acpi.h>
>   #include <asm/fixmap.h>
> @@ -377,6 +378,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)
>   			"This indicates a broken bootloader or old kernel\n",
>   			boot_args[1], boot_args[2], boot_args[3]);
>   	}
> +	init_sframe_table();
>   }
>   
>   static inline bool cpu_can_disable(unsigned int cpu)
> diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
> index 1d9d51d7627f..c035adb8fe8a 100644
> --- a/arch/arm64/kernel/stacktrace.c
> +++ b/arch/arm64/kernel/stacktrace.c
> @@ -14,6 +14,7 @@
>   #include <linux/sched/debug.h>
>   #include <linux/sched/task_stack.h>
>   #include <linux/stacktrace.h>
> +#include <linux/sframe_lookup.h>
>   
>   #include <asm/efi.h>
>   #include <asm/irq.h>
> @@ -242,6 +243,53 @@ kunwind_next_frame_record(struct kunwind_state *state)
>   	return 0;
>   }
>   
> +#ifdef CONFIG_SFRAME_UNWINDER
> +/*
> + * Unwind to the next frame according to sframe.
> + */
> +static __always_inline int
> +unwind_next_frame_sframe(struct unwind_state *state)
> +{
> +	unsigned long fp = state->fp, ip = state->pc;
> +	unsigned long base_reg, cfa;
> +	unsigned long pc_addr, fp_addr;
> +	struct sframe_ip_entry entry;
> +	struct stack_info *info;
> +	struct frame_record *record = (struct frame_record *)fp;
> +
> +	int err;
> +
> +	/* frame record alignment 8 bytes */
> +	if (fp & 0x7)
> +		return -EINVAL;
> +
> +	info = unwind_find_stack(state, fp, sizeof(*record));
> +	if (!info)
> +		return -EINVAL;
> +
> +	err = sframe_find_pc(ip, &entry);
> +	if (err)
> +		return -EINVAL;
> +
> +	unwind_consume_stack(state, info, fp, sizeof(*record));
> +
> +	base_reg = entry.use_fp ? fp : state->cfa;
> +
> +	/* Set up the initial CFA using fp based info if CFA is not set */
> +	if (!state->cfa)
> +		cfa = fp - entry.fp_offset;
> +	else
> +		cfa = base_reg + entry.cfa_offset;
> +	fp_addr = cfa + entry.fp_offset;
> +	pc_addr = cfa + entry.ra_offset;
> +	state->cfa = cfa;
> +	state->fp = READ_ONCE(*(unsigned long *)(fp_addr));
> +	state->pc = READ_ONCE(*(unsigned long *)(pc_addr));
> +
> +	return 0;
> +}
> +#endif
> +
>   /*
>    * Unwind from one frame record (A) to the next frame record (B).
>    *
> @@ -261,7 +309,15 @@ kunwind_next(struct kunwind_state *state)
>   	case KUNWIND_SOURCE_CALLER:
>   	case KUNWIND_SOURCE_TASK:
>   	case KUNWIND_SOURCE_REGS_PC:
> +#ifdef CONFIG_SFRAME_UNWINDER
> +	err = unwind_next_frame_sframe(&state->common);
> +
> +	/* Fallback to FP based unwinder */
> +	if (err)
>   		err = kunwind_next_frame_record(state);
> +#else
> +	err = kunwind_next_frame_record(state);
> +#endif
>   		break;
>   	default:
>   		err = -EINVAL;
> @@ -347,6 +403,9 @@ kunwind_stack_walk(kunwind_consume_fn consume_state,
>   		.common = {
>   			.stacks = stacks,
>   			.nr_stacks = ARRAY_SIZE(stacks),
> +#ifdef CONFIG_SFRAME_UNWINDER
> +			.cfa = 0,
> +#endif
>   		},
>   	};
>   

Looks good to me.
Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

[PATCH 6/8] unwind: arm64: add reliable stacktrace support for arm64

[Weinan Liu]

To support livepatch, we need to add arch_stack_walk_reliable to
support reliable stacktrace according to
https://docs.kernel.org/livepatch/reliable-stacktrace.html#requirements

report stacktrace is not reliable if we are not able to unwind the stack
by sframe unwinder and fallback to FP based unwinder

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 arch/arm64/include/asm/stacktrace/common.h |  2 +
 arch/arm64/kernel/stacktrace.c             | 47 +++++++++++++++++++++-
 2 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/stacktrace/common.h b/arch/arm64/include/asm/stacktrace/common.h
index 19edae8a5b1a..26449cd402db 100644
--- a/arch/arm64/include/asm/stacktrace/common.h
+++ b/arch/arm64/include/asm/stacktrace/common.h
@@ -26,6 +26,7 @@ struct stack_info {
  * @stacks:      An array of stacks which can be unwound.
  * @nr_stacks:   The number of stacks in @stacks.
  * @cfa:         The sp value at the call site of the current function.
+ * @unreliable:  Stacktrace is unreliable.
  */
 struct unwind_state {
 	unsigned long fp;
@@ -36,6 +37,7 @@ struct unwind_state {
 	int nr_stacks;
 #ifdef CONFIG_SFRAME_UNWINDER
 	unsigned long cfa;
+	bool unreliable;
 #endif
 };
 
diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index c035adb8fe8a..eab16dc05bb5 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -310,11 +310,16 @@ kunwind_next(struct kunwind_state *state)
 	case KUNWIND_SOURCE_TASK:
 	case KUNWIND_SOURCE_REGS_PC:
 #ifdef CONFIG_SFRAME_UNWINDER
-	err = unwind_next_frame_sframe(&state->common);
+	if (!state->common.unreliable)
+		err = unwind_next_frame_sframe(&state->common);
 
 	/* Fallback to FP based unwinder */
-	if (err)
+	if (err || state->common.unreliable) {
 		err = kunwind_next_frame_record(state);
+		/* Mark its stacktrace result as unreliable if it is unwindable via FP */
+		if (!err)
+			state->common.unreliable = true;
+	}
 #else
 	err = kunwind_next_frame_record(state);
 #endif
@@ -446,6 +451,44 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry,
 	kunwind_stack_walk(arch_kunwind_consume_entry, &data, task, regs);
 }
 
+#ifdef CONFIG_SFRAME_UNWINDER
+struct kunwind_reliable_consume_entry_data {
+	stack_trace_consume_fn consume_entry;
+	void *cookie;
+	bool unreliable;
+};
+
+static __always_inline bool
+arch_kunwind_reliable_consume_entry(const struct kunwind_state *state, void *cookie)
+{
+	struct kunwind_reliable_consume_entry_data *data = cookie;
+
+	if (state->common.unreliable) {
+		data->unreliable = true;
+		return false;
+	}
+	return data->consume_entry(data->cookie, state->common.pc);
+}
+
+noinline notrace int arch_stack_walk_reliable(
+				stack_trace_consume_fn consume_entry,
+				void *cookie, struct task_struct *task)
+{
+	struct kunwind_reliable_consume_entry_data data = {
+		.consume_entry = consume_entry,
+		.cookie = cookie,
+		.unreliable = false,
+	};
+
+	kunwind_stack_walk(arch_kunwind_reliable_consume_entry, &data, task, NULL);
+
+	if (data.unreliable)
+		return -EINVAL;
+
+	return 0;
+}
+#endif
+
 struct bpf_unwind_consume_entry_data {
 	bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
 	void *cookie;
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 6/8] unwind: arm64: add reliable stacktrace support for arm64

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> To support livepatch, we need to add arch_stack_walk_reliable to
> support reliable stacktrace according to
> https://docs.kernel.org/livepatch/reliable-stacktrace.html#requirements
>
> report stacktrace is not reliable if we are not able to unwind the stack
> by sframe unwinder and fallback to FP based unwinder
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   arch/arm64/include/asm/stacktrace/common.h |  2 +
>   arch/arm64/kernel/stacktrace.c             | 47 +++++++++++++++++++++-
>   2 files changed, 47 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/stacktrace/common.h b/arch/arm64/include/asm/stacktrace/common.h
> index 19edae8a5b1a..26449cd402db 100644
> --- a/arch/arm64/include/asm/stacktrace/common.h
> +++ b/arch/arm64/include/asm/stacktrace/common.h
> @@ -26,6 +26,7 @@ struct stack_info {
>    * @stacks:      An array of stacks which can be unwound.
>    * @nr_stacks:   The number of stacks in @stacks.
>    * @cfa:         The sp value at the call site of the current function.
> + * @unreliable:  Stacktrace is unreliable.
>    */
>   struct unwind_state {
>   	unsigned long fp;
> @@ -36,6 +37,7 @@ struct unwind_state {
>   	int nr_stacks;
>   #ifdef CONFIG_SFRAME_UNWINDER
>   	unsigned long cfa;
> +	bool unreliable;
>   #endif
>   };
>   
> diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
> index c035adb8fe8a..eab16dc05bb5 100644
> --- a/arch/arm64/kernel/stacktrace.c
> +++ b/arch/arm64/kernel/stacktrace.c
> @@ -310,11 +310,16 @@ kunwind_next(struct kunwind_state *state)
>   	case KUNWIND_SOURCE_TASK:
>   	case KUNWIND_SOURCE_REGS_PC:
>   #ifdef CONFIG_SFRAME_UNWINDER
> -	err = unwind_next_frame_sframe(&state->common);
> +	if (!state->common.unreliable)
> +		err = unwind_next_frame_sframe(&state->common);
>   
>   	/* Fallback to FP based unwinder */
> -	if (err)
> +	if (err || state->common.unreliable) {
>   		err = kunwind_next_frame_record(state);
> +		/* Mark its stacktrace result as unreliable if it is unwindable via FP */
> +		if (!err)
> +			state->common.unreliable = true;
> +	}
>   #else
>   	err = kunwind_next_frame_record(state);
>   #endif
> @@ -446,6 +451,44 @@ noinline noinstr void arch_stack_walk(stack_trace_consume_fn consume_entry,
>   	kunwind_stack_walk(arch_kunwind_consume_entry, &data, task, regs);
>   }
>   
> +#ifdef CONFIG_SFRAME_UNWINDER
> +struct kunwind_reliable_consume_entry_data {
> +	stack_trace_consume_fn consume_entry;
> +	void *cookie;
> +	bool unreliable;
> +};
> +
> +static __always_inline bool
> +arch_kunwind_reliable_consume_entry(const struct kunwind_state *state, void *cookie)
> +{
> +	struct kunwind_reliable_consume_entry_data *data = cookie;
> +
> +	if (state->common.unreliable) {
> +		data->unreliable = true;
> +		return false;
> +	}
> +	return data->consume_entry(data->cookie, state->common.pc);
> +}
> +
> +noinline notrace int arch_stack_walk_reliable(
> +				stack_trace_consume_fn consume_entry,
> +				void *cookie, struct task_struct *task)
> +{
> +	struct kunwind_reliable_consume_entry_data data = {
> +		.consume_entry = consume_entry,
> +		.cookie = cookie,
> +		.unreliable = false,
> +	};
> +
> +	kunwind_stack_walk(arch_kunwind_reliable_consume_entry, &data, task, NULL);
> +
> +	if (data.unreliable)
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +#endif
> +
>   struct bpf_unwind_consume_entry_data {
>   	bool (*consume_entry)(void *cookie, u64 ip, u64 sp, u64 fp);
>   	void *cookie;

Why not fold the previous patch and this into one?

But the code looks good to me.

Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

[PATCH 7/8] arm64: Define TIF_PATCH_PENDING for livepatch

[Weinan Liu]

From: "Madhavan T. Venkataraman" <madvenka@linux.microsoft.com>

	- Define TIF_PATCH_PENDING in arch/arm64/include/asm/thread_info.h
	  for livepatch.

	- Check TIF_PATCH_PENDING in do_notify_resume() to patch the
	  current task for livepatch.

Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
Signed-off-by: Madhavan T. Venkataraman <madvenka@linux.microsoft.com>
Signed-off-by: Weinan Liu <wnliu@google.com>
---
 arch/arm64/include/asm/thread_info.h | 4 +++-
 arch/arm64/kernel/entry-common.c     | 4 ++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
index 1114c1c3300a..3810c2f3914e 100644
--- a/arch/arm64/include/asm/thread_info.h
+++ b/arch/arm64/include/asm/thread_info.h
@@ -64,6 +64,7 @@ void arch_setup_new_exec(void);
 #define TIF_UPROBE		4	/* uprobe breakpoint or singlestep */
 #define TIF_MTE_ASYNC_FAULT	5	/* MTE Asynchronous Tag Check Fault */
 #define TIF_NOTIFY_SIGNAL	6	/* signal notifications exist */
+#define TIF_PATCH_PENDING	7	/* pending live patching update */
 #define TIF_SYSCALL_TRACE	8	/* syscall trace active */
 #define TIF_SYSCALL_AUDIT	9	/* syscall auditing */
 #define TIF_SYSCALL_TRACEPOINT	10	/* syscall tracepoint for ftrace */
@@ -99,11 +100,12 @@ void arch_setup_new_exec(void);
 #define _TIF_MTE_ASYNC_FAULT	(1 << TIF_MTE_ASYNC_FAULT)
 #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_TSC_SIGSEGV	(1 << TIF_TSC_SIGSEGV)
+#define _TIF_PATCH_PENDING	(1 << TIF_PATCH_PENDING)
 
 #define _TIF_WORK_MASK		(_TIF_NEED_RESCHED | _TIF_SIGPENDING | \
 				 _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \
 				 _TIF_UPROBE | _TIF_MTE_ASYNC_FAULT | \
-				 _TIF_NOTIFY_SIGNAL)
+				 _TIF_NOTIFY_SIGNAL | _TIF_PATCH_PENDING)
 
 #define _TIF_SYSCALL_WORK	(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \
 				 _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
index b260ddc4d3e9..b537af333b42 100644
--- a/arch/arm64/kernel/entry-common.c
+++ b/arch/arm64/kernel/entry-common.c
@@ -8,6 +8,7 @@
 #include <linux/context_tracking.h>
 #include <linux/kasan.h>
 #include <linux/linkage.h>
+#include <linux/livepatch.h>
 #include <linux/lockdep.h>
 #include <linux/ptrace.h>
 #include <linux/resume_user_mode.h>
@@ -144,6 +145,9 @@ static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags)
 				       (void __user *)NULL, current);
 		}
 
+		if (thread_flags & _TIF_PATCH_PENDING)
+			klp_update_patch_state(current);
+
 		if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL))
 			do_signal(regs);
 
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 7/8] arm64: Define TIF_PATCH_PENDING for livepatch

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> From: "Madhavan T. Venkataraman" <madvenka@linux.microsoft.com>
>
> 	- Define TIF_PATCH_PENDING in arch/arm64/include/asm/thread_info.h
> 	  for livepatch.
>
> 	- Check TIF_PATCH_PENDING in do_notify_resume() to patch the
> 	  current task for livepatch.
>
> Signed-off-by: Suraj Jitindar Singh <sjitindarsingh@gmail.com>
> Signed-off-by: Madhavan T. Venkataraman <madvenka@linux.microsoft.com>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   arch/arm64/include/asm/thread_info.h | 4 +++-
>   arch/arm64/kernel/entry-common.c     | 4 ++++
>   2 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
> index 1114c1c3300a..3810c2f3914e 100644
> --- a/arch/arm64/include/asm/thread_info.h
> +++ b/arch/arm64/include/asm/thread_info.h
> @@ -64,6 +64,7 @@ void arch_setup_new_exec(void);
>   #define TIF_UPROBE		4	/* uprobe breakpoint or singlestep */
>   #define TIF_MTE_ASYNC_FAULT	5	/* MTE Asynchronous Tag Check Fault */
>   #define TIF_NOTIFY_SIGNAL	6	/* signal notifications exist */
> +#define TIF_PATCH_PENDING	7	/* pending live patching update */
>   #define TIF_SYSCALL_TRACE	8	/* syscall trace active */
>   #define TIF_SYSCALL_AUDIT	9	/* syscall auditing */
>   #define TIF_SYSCALL_TRACEPOINT	10	/* syscall tracepoint for ftrace */
> @@ -99,11 +100,12 @@ void arch_setup_new_exec(void);
>   #define _TIF_MTE_ASYNC_FAULT	(1 << TIF_MTE_ASYNC_FAULT)
>   #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
>   #define _TIF_TSC_SIGSEGV	(1 << TIF_TSC_SIGSEGV)
> +#define _TIF_PATCH_PENDING	(1 << TIF_PATCH_PENDING)
>   
>   #define _TIF_WORK_MASK		(_TIF_NEED_RESCHED | _TIF_SIGPENDING | \
>   				 _TIF_NOTIFY_RESUME | _TIF_FOREIGN_FPSTATE | \
>   				 _TIF_UPROBE | _TIF_MTE_ASYNC_FAULT | \
> -				 _TIF_NOTIFY_SIGNAL)
> +				 _TIF_NOTIFY_SIGNAL | _TIF_PATCH_PENDING)
>   
>   #define _TIF_SYSCALL_WORK	(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \
>   				 _TIF_SYSCALL_TRACEPOINT | _TIF_SECCOMP | \
> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index b260ddc4d3e9..b537af333b42 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -8,6 +8,7 @@
>   #include <linux/context_tracking.h>
>   #include <linux/kasan.h>
>   #include <linux/linkage.h>
> +#include <linux/livepatch.h>
>   #include <linux/lockdep.h>
>   #include <linux/ptrace.h>
>   #include <linux/resume_user_mode.h>
> @@ -144,6 +145,9 @@ static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags)
>   				       (void __user *)NULL, current);
>   		}
>   
> +		if (thread_flags & _TIF_PATCH_PENDING)
> +			klp_update_patch_state(current);
> +
>   		if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL))
>   			do_signal(regs);
>   

Looks good to me.
Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

Re: [PATCH 7/8] arm64: Define TIF_PATCH_PENDING for livepatch

[Miroslav Benes]

Hi,

> diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c
> index b260ddc4d3e9..b537af333b42 100644
> --- a/arch/arm64/kernel/entry-common.c
> +++ b/arch/arm64/kernel/entry-common.c
> @@ -8,6 +8,7 @@
>  #include <linux/context_tracking.h>
>  #include <linux/kasan.h>
>  #include <linux/linkage.h>
> +#include <linux/livepatch.h>
>  #include <linux/lockdep.h>
>  #include <linux/ptrace.h>
>  #include <linux/resume_user_mode.h>
> @@ -144,6 +145,9 @@ static void do_notify_resume(struct pt_regs *regs, unsigned long thread_flags)
>  				       (void __user *)NULL, current);
>  		}
>  
> +		if (thread_flags & _TIF_PATCH_PENDING)
> +			klp_update_patch_state(current);
> +
>  		if (thread_flags & (_TIF_SIGPENDING | _TIF_NOTIFY_SIGNAL))
>  			do_signal(regs);

Just a remark so that it is recorded. Once arm64 is moved to generic 
entry infra, the hunk will not be needed.

Regards,
Miroslav

[PATCH 8/8] arm64: Enable livepatch for ARM64

[Weinan Liu]

Since SFrame is considered as reliable stacktrace, enable livepatch in
arch/arm64/Kconfig

Signed-off-by: Weinan Liu <wnliu@google.com>
---
 arch/arm64/Kconfig | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 100570a048c5..c292bc73b65c 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -271,6 +271,8 @@ config ARM64
 	select HAVE_SOFTIRQ_ON_OWN_STACK
 	select USER_STACKTRACE_SUPPORT
 	select VDSO_GETRANDOM
+	select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER
+	select HAVE_LIVEPATCH		if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE
 	help
 	  ARM 64-bit (AArch64) Linux support.
 
@@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig"
 
 source "arch/arm64/kvm/Kconfig"
 
+source "kernel/livepatch/Kconfig"
-- 
2.48.1.262.g85cc9f2d1e-goog

Re: [PATCH 8/8] arm64: Enable livepatch for ARM64

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> Since SFrame is considered as reliable stacktrace, enable livepatch in
> arch/arm64/Kconfig
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   arch/arm64/Kconfig | 3 +++
>   1 file changed, 3 insertions(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 100570a048c5..c292bc73b65c 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -271,6 +271,8 @@ config ARM64
>   	select HAVE_SOFTIRQ_ON_OWN_STACK
>   	select USER_STACKTRACE_SUPPORT
>   	select VDSO_GETRANDOM
> +	select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER
> +	select HAVE_LIVEPATCH		if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE
>   	help
>   	  ARM 64-bit (AArch64) Linux support.
>   
> @@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig"
>   
>   source "arch/arm64/kvm/Kconfig"
>   
> +source "kernel/livepatch/Kconfig"

Looks good to me.
Reviewed-by: Prasanna Kumar T S M <ptsm@linux.microsoft.com>.

Re: [PATCH 8/8] arm64: Enable livepatch for ARM64

[Prasanna Kumar T S M]

On 28-01-2025 03:03, Weinan Liu wrote:
> Since SFrame is considered as reliable stacktrace, enable livepatch in
> arch/arm64/Kconfig
>
> Signed-off-by: Weinan Liu <wnliu@google.com>
> ---
>   arch/arm64/Kconfig | 3 +++
>   1 file changed, 3 insertions(+)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 100570a048c5..c292bc73b65c 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -271,6 +271,8 @@ config ARM64
>   	select HAVE_SOFTIRQ_ON_OWN_STACK
>   	select USER_STACKTRACE_SUPPORT
>   	select VDSO_GETRANDOM
> +	select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER
> +	select HAVE_LIVEPATCH		if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE
>   	help
>   	  ARM 64-bit (AArch64) Linux support.
>   
> @@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig"
>   
>   source "arch/arm64/kvm/Kconfig"
>   
> +source "kernel/livepatch/Kconfig"

Will this work for ftrace'd (kprobe'd) function as well?

Re: [PATCH 8/8] arm64: Enable livepatch for ARM64

[Steven Rostedt]

On Fri, 31 Jan 2025 21:38:46 +0530
Prasanna Kumar T S M <ptsm@linux.microsoft.com> wrote:

> > --- a/arch/arm64/Kconfig
> > +++ b/arch/arm64/Kconfig
> > @@ -271,6 +271,8 @@ config ARM64
> >   	select HAVE_SOFTIRQ_ON_OWN_STACK
> >   	select USER_STACKTRACE_SUPPORT
> >   	select VDSO_GETRANDOM
> > +	select HAVE_RELIABLE_STACKTRACE if SFRAME_UNWINDER
> > +	select HAVE_LIVEPATCH		if HAVE_DYNAMIC_FTRACE_WITH_ARGS && HAVE_RELIABLE_STACKTRACE
> >   	help
> >   	  ARM 64-bit (AArch64) Linux support.
> >   
> > @@ -2498,3 +2500,4 @@ source "drivers/acpi/Kconfig"
> >   
> >   source "arch/arm64/kvm/Kconfig"
> >   
> > +source "kernel/livepatch/Kconfig"  
> 
> Will this work for ftrace'd (kprobe'd) function as well?

What work? The stack walker? It may require updates to the tracing
trampolines, but it should be doable just like ORC is.

-- Steve

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 1/27/25 1:33 PM, Weinan Liu wrote:
> This patchset implements a generic kernel sframe-based [1] unwinder.
> The main goal is to support reliable stacktraces on arm64.
> 
> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
> required support from objtool: it cannot generate orc unwind tables for
> arm64.
> 
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> Since the sframe unwind table algorithm is similar, these two proposal
> could integrate common functionality in the future.
> 
> There are some incomplete features or challenges:
>    - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>      FRE from kernel modules doesn't appear correct, preventing us from
>      unwinding functions from kernel modules.

I did file https://sourceware.org/bugzilla/show_bug.cgi?id=32589 
earlier.  It is misleading (and inconvenient) to see all 0s in the dump 
of non-relocated SFrame section.

I get the sense that while issue 32589 may be causing confusion that the 
SFrame data is not correct, the problem is elsewhere ?

I can share a fix for 32589 so atleast we can verify that the starting 
point is sane.

>    - Currently, only GCC supports sframe.
> 
> Ref:
> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
> 
> Madhavan T. Venkataraman (1):
>    arm64: Define TIF_PATCH_PENDING for livepatch
> 
> Weinan Liu (7):
>    unwind: build kernel with sframe info
>    arm64: entry: add unwind info for various kernel entries
>    unwind: add sframe v2 header
>    unwind: Implement generic sframe unwinder library
>    unwind: arm64: Add sframe unwinder on arm64
>    unwind: arm64: add reliable stacktrace support for arm64
>    arm64: Enable livepatch for ARM64
> 
>   Makefile                                   |   6 +
>   arch/Kconfig                               |   8 +
>   arch/arm64/Kconfig                         |   3 +
>   arch/arm64/Kconfig.debug                   |  10 +
>   arch/arm64/include/asm/stacktrace/common.h |   6 +
>   arch/arm64/include/asm/thread_info.h       |   4 +-
>   arch/arm64/kernel/entry-common.c           |   4 +
>   arch/arm64/kernel/entry.S                  |  10 +
>   arch/arm64/kernel/setup.c                  |   2 +
>   arch/arm64/kernel/stacktrace.c             | 102 ++++++++++
>   include/asm-generic/vmlinux.lds.h          |  12 ++
>   include/linux/sframe_lookup.h              |  43 +++++
>   kernel/Makefile                            |   1 +
>   kernel/sframe.h                            | 215 +++++++++++++++++++++
>   kernel/sframe_lookup.c                     | 196 +++++++++++++++++++
>   15 files changed, 621 insertions(+), 1 deletion(-)
>   create mode 100644 include/linux/sframe_lookup.h
>   create mode 100644 kernel/sframe.h
>   create mode 100644 kernel/sframe_lookup.c
> 

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

Not only does objdump show that func_start_addr all 0s, but the
func_start_addr in the kernel module's sframe table is also incorrect. I'll
prepare a repro so we can investigate this.

>
> I can share a fix for 32589 so atleast we can verify that the starting
> point is sane.
>

Great! Thanks.
This will be helpful for comparing the values from the sframe table with
the values we obtain at runtime

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

I missed this set before sending my RFC set. If this set works well, we
won't need the other set. I will give this one a try.

Thanks,
Song

On Mon, Jan 27, 2025 at 1:33 PM Weinan Liu <wnliu@google.com> wrote:
>
> This patchset implements a generic kernel sframe-based [1] unwinder.
> The main goal is to support reliable stacktraces on arm64.
>
> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
> required support from objtool: it cannot generate orc unwind tables for
> arm64.
>
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> Since the sframe unwind table algorithm is similar, these two proposal
> could integrate common functionality in the future.
>
> There are some incomplete features or challenges:
>   - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>     FRE from kernel modules doesn't appear correct, preventing us from
>     unwinding functions from kernel modules.
>   - Currently, only GCC supports sframe.
>
> Ref:
> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>
> Madhavan T. Venkataraman (1):
>   arm64: Define TIF_PATCH_PENDING for livepatch
>
> Weinan Liu (7):
>   unwind: build kernel with sframe info
>   arm64: entry: add unwind info for various kernel entries
>   unwind: add sframe v2 header
>   unwind: Implement generic sframe unwinder library
>   unwind: arm64: Add sframe unwinder on arm64
>   unwind: arm64: add reliable stacktrace support for arm64
>   arm64: Enable livepatch for ARM64
>
>  Makefile                                   |   6 +
>  arch/Kconfig                               |   8 +
>  arch/arm64/Kconfig                         |   3 +
>  arch/arm64/Kconfig.debug                   |  10 +
>  arch/arm64/include/asm/stacktrace/common.h |   6 +
>  arch/arm64/include/asm/thread_info.h       |   4 +-
>  arch/arm64/kernel/entry-common.c           |   4 +
>  arch/arm64/kernel/entry.S                  |  10 +
>  arch/arm64/kernel/setup.c                  |   2 +
>  arch/arm64/kernel/stacktrace.c             | 102 ++++++++++
>  include/asm-generic/vmlinux.lds.h          |  12 ++
>  include/linux/sframe_lookup.h              |  43 +++++
>  kernel/Makefile                            |   1 +
>  kernel/sframe.h                            | 215 +++++++++++++++++++++
>  kernel/sframe_lookup.c                     | 196 +++++++++++++++++++
>  15 files changed, 621 insertions(+), 1 deletion(-)
>  create mode 100644 include/linux/sframe_lookup.h
>  create mode 100644 kernel/sframe.h
>  create mode 100644 kernel/sframe_lookup.c
>
> --
> 2.48.1.262.g85cc9f2d1e-goog
>
>

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Thu, Jan 30, 2025 at 9:59 AM Song Liu <song@kernel.org> wrote:
>
> I missed this set before sending my RFC set. If this set works well, we
> won't need the other set. I will give this one a try.

I just realized that llvm doesn't support sframe yet. So we (Meta) still
need some sframe-less approach before llvm supports sframe.

IIRC, Google also uses llvm to compile the kernel. Weinan, would
you mind share your thoughts on how we can adopt this before
llvm supports sframe? (compile arm64 kernel with gcc?)

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Roman Gushchin]

On Thu, Jan 30, 2025 at 10:34:09AM -0800, Song Liu wrote:
> On Thu, Jan 30, 2025 at 9:59 AM Song Liu <song@kernel.org> wrote:
> >
> > I missed this set before sending my RFC set. If this set works well, we
> > won't need the other set. I will give this one a try.
> 
> I just realized that llvm doesn't support sframe yet. So we (Meta) still
> need some sframe-less approach before llvm supports sframe.
> 
> IIRC, Google also uses llvm to compile the kernel. Weinan, would
> you mind share your thoughts on how we can adopt this before
> llvm supports sframe? (compile arm64 kernel with gcc?)

Hi Song,

the plan is to start the work on adding sframe support to llvm
in parallel to landing these changes upstream. From the initial
assessment it shouldn't be too hard.

Thanks

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

Hi Roman,

On Thu, Jan 30, 2025 at 11:01 AM Roman Gushchin
<roman.gushchin@linux.dev> wrote:
>
> On Thu, Jan 30, 2025 at 10:34:09AM -0800, Song Liu wrote:
> > On Thu, Jan 30, 2025 at 9:59 AM Song Liu <song@kernel.org> wrote:
> > >
> > > I missed this set before sending my RFC set. If this set works well, we
> > > won't need the other set. I will give this one a try.
> >
> > I just realized that llvm doesn't support sframe yet. So we (Meta) still
> > need some sframe-less approach before llvm supports sframe.
> >
> > IIRC, Google also uses llvm to compile the kernel. Weinan, would
> > you mind share your thoughts on how we can adopt this before
> > llvm supports sframe? (compile arm64 kernel with gcc?)
>
> Hi Song,
>
> the plan is to start the work on adding sframe support to llvm
> in parallel to landing these changes upstream. From the initial
> assessment it shouldn't be too hard.

Thanks for the information!

Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 5939 bytes --]

Weinan Liu <wnliu@google.com> writes:

> This patchset implements a generic kernel sframe-based [1] unwinder.
> The main goal is to support reliable stacktraces on arm64.
>
> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
> required support from objtool: it cannot generate orc unwind tables for
> arm64.
>
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> Since the sframe unwind table algorithm is similar, these two proposal
> could integrate common functionality in the future.
>
> There are some incomplete features or challenges:
>   - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>     FRE from kernel modules doesn't appear correct, preventing us from
>     unwinding functions from kernel modules.
>   - Currently, only GCC supports sframe.
>
> Ref:
> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>

Hi Weinan,
Thanks for working on this.

I tested this set on my setup and faced some issues, here are the
details:

Here is my setup [on AWS c6gd.16xlarge instance]:
-------------------------------------------------

[root@ip-172-31-32-86 linux-upstream]# uname -a
Linux ip-172-31-32-86.ec2.internal 6.14.0-rc1+ #1 SMP Tue Feb  4 14:15:55 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

[root@ip-172-31-32-86 linux-upstream]# git log --oneline
e9a702365 (HEAD -> master) arm64: Enable livepatch for ARM64
5dedc956e arm64: Define TIF_PATCH_PENDING for livepatch
ba563b31a unwind: arm64: add reliable stacktrace support for arm64
d807d392d unwind: arm64: Add sframe unwinder on arm64
7872f050b unwind: Implement generic sframe unwinder library
03d2ad003 unwind: add sframe v2 header
5e95cc051 arm64: entry: add unwind info for various kernel entries
faff6cbc3 unwind: build kernel with sframe info
0de63bb7d (origin/master, origin/HEAD) Merge tag 'pull-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
902e09c8a fix braino in "9p: fix ->rename_sem exclusion"
f286757b6 Merge tag 'timers-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
a360f3ffd (grafted) Merge tag 'irq-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
bb2784d9a (grafted) jiffies: Cast to unsigned long in secs_to_jiffies() conversion
30d61efe1 (grafted) 9p: fix ->rename_sem exclusion

[root@ip-172-31-32-86 linux-upstream]# grep SFRAME .config
CONFIG_AS_HAS_SFRAME_SUPPORT=y
CONFIG_SFRAME_UNWIND_TABLE=y
CONFIG_SFRAME_UNWINDER=y
[root@ip-172-31-32-86 linux-upstream]# grep LIVEPATCH .config
CONFIG_HAVE_LIVEPATCH=y
CONFIG_LIVEPATCH=y
CONFIG_SAMPLE_LIVEPATCH=m

[root@ip-172-31-32-86 linux-upstream]# as --version
GNU assembler version 2.41-50.al2023.0.2
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `aarch64-amazon-linux'.

[root@ip-172-31-32-86 linux-upstream]# gcc --version
gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Loading the livepatch-sameple module:
-------------------------------------

[root@ip-172-31-32-86 linux-upstream]# kpatch load /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
loading patch module: /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
waiting (up to 15 seconds) for patch transition to complete...
patch transition has stalled!
<4>kpatch: Livepatch process signaling is performed automatically on your system.
<4>kpatch: Skipping manual process signaling.
waiting (up to 60 seconds) for patch transition to complete...

Stalled processes:
340 kdevtmpfs
stack:
[<0>] devtmpfs_work_loop+0x2cc/0x2d8
[<0>] devtmpfsd+0x4c/0x58
[<0>] kthread+0xf0/0x100
[<0>] ret_from_fork+0x10/0x20
module livepatch_sample did not complete its transition, unloading...
disabling patch module: livepatch_sample
waiting (up to 15 seconds) for patch transition to complete...
transition complete (3 seconds)
unloading patch module: livepatch_sample
<4>kpatch: error: failed to load module livepatch_sample (transition stalled)

Useful messages from kernel log [pr_debug enabled]:
---------------------------------------------------

 livepatch: enabling patch 'livepatch_sample'
 livepatch: 'livepatch_sample': initializing patching transition
 livepatch: 'livepatch_sample': starting patching transition
 livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
 livepatch: klp_try_switch_task: insmod:9226 has an unreliable stack
 livepatch: klp_try_switch_task: swapper/63:0 is running
 [......SNIP.......]
 livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
 [......SNIP.......]
 livepatch: signaling remaining tasks
 livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
 livepatch: 'livepatch_sample': reversing transition from patching to unpatching
 livepatch: 'livepatch_sample': starting unpatching transition
 livepatch: klp_try_switch_task: swapper/45:0 is running
 livepatch: 'livepatch_sample': completing unpatching transition
 livepatch: 'livepatch_sample': unpatching complete

Please let me know if you are aware of this already or if this is
expected behaviour with this version. I will try to debug this from my
side as well. Also let me know if you need more details for debugging
this.

P.S. - I also saw multiple build warning like:
ld: warning: orphan section `.eh_frame' from `arch/arm64/kernel/entry.o' being placed in section `.eh_frame'
ld: warning: orphan section `.init.sframe' from `arch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe'


Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 8151 bytes --]

Puranjay Mohan <puranjay@kernel.org> writes:

> Weinan Liu <wnliu@google.com> writes:
>
>> This patchset implements a generic kernel sframe-based [1] unwinder.
>> The main goal is to support reliable stacktraces on arm64.
>>
>> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
>> required support from objtool: it cannot generate orc unwind tables for
>> arm64.
>>
>> Currently, there's already a sframe unwinder proposed for userspace: [2].
>> Since the sframe unwind table algorithm is similar, these two proposal
>> could integrate common functionality in the future.
>>
>> There are some incomplete features or challenges:
>>   - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>>     FRE from kernel modules doesn't appear correct, preventing us from
>>     unwinding functions from kernel modules.
>>   - Currently, only GCC supports sframe.
>>
>> Ref:
>> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
>> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>>
>
> Hi Weinan,
> Thanks for working on this.
>
> I tested this set on my setup and faced some issues, here are the
> details:
>
> Here is my setup [on AWS c6gd.16xlarge instance]:
> -------------------------------------------------
>
> [root@ip-172-31-32-86 linux-upstream]# uname -a
> Linux ip-172-31-32-86.ec2.internal 6.14.0-rc1+ #1 SMP Tue Feb  4 14:15:55 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
>
> [root@ip-172-31-32-86 linux-upstream]# git log --oneline
> e9a702365 (HEAD -> master) arm64: Enable livepatch for ARM64
> 5dedc956e arm64: Define TIF_PATCH_PENDING for livepatch
> ba563b31a unwind: arm64: add reliable stacktrace support for arm64
> d807d392d unwind: arm64: Add sframe unwinder on arm64
> 7872f050b unwind: Implement generic sframe unwinder library
> 03d2ad003 unwind: add sframe v2 header
> 5e95cc051 arm64: entry: add unwind info for various kernel entries
> faff6cbc3 unwind: build kernel with sframe info
> 0de63bb7d (origin/master, origin/HEAD) Merge tag 'pull-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
> 902e09c8a fix braino in "9p: fix ->rename_sem exclusion"
> f286757b6 Merge tag 'timers-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> a360f3ffd (grafted) Merge tag 'irq-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> bb2784d9a (grafted) jiffies: Cast to unsigned long in secs_to_jiffies() conversion
> 30d61efe1 (grafted) 9p: fix ->rename_sem exclusion
>
> [root@ip-172-31-32-86 linux-upstream]# grep SFRAME .config
> CONFIG_AS_HAS_SFRAME_SUPPORT=y
> CONFIG_SFRAME_UNWIND_TABLE=y
> CONFIG_SFRAME_UNWINDER=y
> [root@ip-172-31-32-86 linux-upstream]# grep LIVEPATCH .config
> CONFIG_HAVE_LIVEPATCH=y
> CONFIG_LIVEPATCH=y
> CONFIG_SAMPLE_LIVEPATCH=m
>
> [root@ip-172-31-32-86 linux-upstream]# as --version
> GNU assembler version 2.41-50.al2023.0.2
> Copyright (C) 2023 Free Software Foundation, Inc.
> This program is free software; you may redistribute it under the terms of
> the GNU General Public License version 3 or later.
> This program has absolutely no warranty.
> This assembler was configured for a target of `aarch64-amazon-linux'.
>
> [root@ip-172-31-32-86 linux-upstream]# gcc --version
> gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
> Copyright (C) 2021 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> Loading the livepatch-sameple module:
> -------------------------------------
>
> [root@ip-172-31-32-86 linux-upstream]# kpatch load /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
> loading patch module: /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
> waiting (up to 15 seconds) for patch transition to complete...
> patch transition has stalled!
> <4>kpatch: Livepatch process signaling is performed automatically on your system.
> <4>kpatch: Skipping manual process signaling.
> waiting (up to 60 seconds) for patch transition to complete...
>
> Stalled processes:
> 340 kdevtmpfs
> stack:
> [<0>] devtmpfs_work_loop+0x2cc/0x2d8
> [<0>] devtmpfsd+0x4c/0x58
> [<0>] kthread+0xf0/0x100
> [<0>] ret_from_fork+0x10/0x20
> module livepatch_sample did not complete its transition, unloading...
> disabling patch module: livepatch_sample
> waiting (up to 15 seconds) for patch transition to complete...
> transition complete (3 seconds)
> unloading patch module: livepatch_sample
> <4>kpatch: error: failed to load module livepatch_sample (transition stalled)

After some debugging this is what I found:

devtmpfsd() calls devtmpfs_work_loop() which is marked '__noreturn' and has an
infinite loop. The compiler puts the `bl` to devtmpfs_work_loop() as the the
last instruction in devtmpfsd() and therefore on entry to devtmpfs_work_loop(),
LR points to an instruction beyond devtmpfsd() and this consfuses the unwinder.

ffff800080d9a070 <devtmpfsd>:
ffff800080d9a070:       d503201f        nop
ffff800080d9a074:       d503201f        nop
ffff800080d9a078:       d503233f        paciasp
ffff800080d9a07c:       a9be7bfd        stp     x29, x30, [sp, #-32]!
ffff800080d9a080:       910003fd        mov     x29, sp
ffff800080d9a084:       f9000bf3        str     x19, [sp, #16]
ffff800080d9a088:       943378e8        bl      ffff800081a78428 <devtmpfs_setup>
ffff800080d9a08c:       90006ca1        adrp    x1, ffff800081b2e000 <unique_processor_ids+0x3758>
ffff800080d9a090:       2a0003f3        mov     w19, w0
ffff800080d9a094:       912de021        add     x1, x1, #0xb78
ffff800080d9a098:       91002020        add     x0, x1, #0x8
ffff800080d9a09c:       97cd2a43        bl      ffff8000800e49a8 <complete>
ffff800080d9a0a0:       340000d3        cbz     w19, ffff800080d9a0b8 <devtmpfsd+0x48>
ffff800080d9a0a4:       2a1303e0        mov     w0, w19
ffff800080d9a0a8:       f9400bf3        ldr     x19, [sp, #16]
ffff800080d9a0ac:       a8c27bfd        ldp     x29, x30, [sp], #32
ffff800080d9a0b0:       d50323bf        autiasp
ffff800080d9a0b4:       d65f03c0        ret
ffff800080d9a0b8:       97f06526        bl      ffff8000809b3550 <devtmpfs_work_loop>
ffff800080d9a0bc:       00000000        udf     #0
ffff800080d9a0c0:       d503201f        nop
ffff800080d9a0c4:       d503201f        nop

find_fde() got pc=0xffff800080d9a0bc which is not in [sfde_func_start_address, sfde_func_size)

output for readelf --sframe for devtmpfsd()

func idx [51825]: pc = 0xffff800080d9a070, size = 76 bytes
    STARTPC           CFA       FP        RA
    ffff800080d9a070  sp+0      u         u
    ffff800080d9a07c  sp+0      u         u[s]
    ffff800080d9a080  sp+32     c-32      c-24[s]
    ffff800080d9a0b0  sp+0      u         u[s]
    ffff800080d9a0b4  sp+0      u         u
    ffff800080d9a0b8  sp+32     c-32      c-24[s]

The unwinder and all the related infra is assuming that the return address
will be part of a valid function which is not the case here.

I am not sure which component needs to be fixed here, but the following
patch(which is a hack) fixes the issue by considering the return address as
part of the function descriptor entry.

-- 8< --

diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
index 846f1da95..28bec5064 100644
--- a/kernel/sframe_lookup.c
+++ b/kernel/sframe_lookup.c
@@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned long
        if (f >= tbl->sfhdr_p->num_fdes || f < 0)
                return NULL;
        fdep = tbl->fde_p + f;
-       if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
+       if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)
                return NULL;

        return fdep;
@@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,
        else
                ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - fdep->start_addr;

-       if (ip_off < 0 || ip_off >= fdep->size)
+       if (ip_off < 0 || ip_off > fdep->size)
                return -EINVAL;

        /*

-- >8 --

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

> After some debugging this is what I found:
> 
> devtmpfsd() calls devtmpfs_work_loop() which is marked '__noreturn' and has an
> infinite loop. The compiler puts the `bl` to devtmpfs_work_loop() as the the
> last instruction in devtmpfsd() and therefore on entry to devtmpfs_work_loop(),
> LR points to an instruction beyond devtmpfsd() and this consfuses the unwinder.
> 
> ffff800080d9a070 <devtmpfsd>:
> ffff800080d9a070:       d503201f        nop
> ffff800080d9a074:       d503201f        nop
> ffff800080d9a078:       d503233f        paciasp
> ffff800080d9a07c:       a9be7bfd        stp     x29, x30, [sp, #-32]!
> ffff800080d9a080:       910003fd        mov     x29, sp
> ffff800080d9a084:       f9000bf3        str     x19, [sp, #16]
> ffff800080d9a088:       943378e8        bl      ffff800081a78428 <devtmpfs_setup>
> ffff800080d9a08c:       90006ca1        adrp    x1, ffff800081b2e000 <unique_processor_ids+0x3758>
> ffff800080d9a090:       2a0003f3        mov     w19, w0
> ffff800080d9a094:       912de021        add     x1, x1, #0xb78
> ffff800080d9a098:       91002020        add     x0, x1, #0x8
> ffff800080d9a09c:       97cd2a43        bl      ffff8000800e49a8 <complete>
> ffff800080d9a0a0:       340000d3        cbz     w19, ffff800080d9a0b8 <devtmpfsd+0x48>
> ffff800080d9a0a4:       2a1303e0        mov     w0, w19
> ffff800080d9a0a8:       f9400bf3        ldr     x19, [sp, #16]
> ffff800080d9a0ac:       a8c27bfd        ldp     x29, x30, [sp], #32
> ffff800080d9a0b0:       d50323bf        autiasp
> ffff800080d9a0b4:       d65f03c0        ret
> ffff800080d9a0b8:       97f06526        bl      ffff8000809b3550 <devtmpfs_work_loop>
> ffff800080d9a0bc:       00000000        udf     #0
> ffff800080d9a0c0:       d503201f        nop
> ffff800080d9a0c4:       d503201f        nop
> 
> find_fde() got pc=0xffff800080d9a0bc which is not in [sfde_func_start_address, sfde_func_size)
> 
> output for readelf --sframe for devtmpfsd()
> 
> func idx [51825]: pc = 0xffff800080d9a070, size = 76 bytes
>     STARTPC           CFA       FP        RA
>     ffff800080d9a070  sp+0      u         u
>     ffff800080d9a07c  sp+0      u         u[s]
>     ffff800080d9a080  sp+32     c-32      c-24[s]
>     ffff800080d9a0b0  sp+0      u         u[s]
>     ffff800080d9a0b4  sp+0      u         u
>     ffff800080d9a0b8  sp+32     c-32      c-24[s]
> 
> The unwinder and all the related infra is assuming that the return address
> will be part of a valid function which is not the case here.
> 
> I am not sure which component needs to be fixed here, but the following
> patch(which is a hack) fixes the issue by considering the return address as
> part of the function descriptor entry.
> 
> -- 8< --
> 
> diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
> index 846f1da95..28bec5064 100644
> --- a/kernel/sframe_lookup.c
> +++ b/kernel/sframe_lookup.c
> @@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned long
>         if (f >= tbl->sfhdr_p->num_fdes || f < 0)
>                 return NULL;
>         fdep = tbl->fde_p + f;
> -       if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
> +       if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)
>                 return NULL;
> 
>         return fdep;
> @@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,
>         else
>                 ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - fdep->start_addr;
> 
> -       if (ip_off < 0 || ip_off >= fdep->size)
> +       if (ip_off < 0 || ip_off > fdep->size)
>                 return -EINVAL;
> 
>         /*
> 
> -- >8 --
> 
> Thanks,
> Puranjay

Thank you for reporting this issue.
I just found out that Josh also intentionally uses '>' instead of '>=' for the same reason
https://lore.kernel.org/lkml/20250122225257.h64ftfnorofe7cb4@jpoimboe/T/#m6d70a20ed9f5b3bbe5b24b24b8c5dcc603a79101

QQ, do we need to care the stacktrace after '__noreturn' function?

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 4732 bytes --]

Weinan Liu <wnliu@google.com> writes:

>> After some debugging this is what I found:
>> 
>> devtmpfsd() calls devtmpfs_work_loop() which is marked '__noreturn' and has an
>> infinite loop. The compiler puts the `bl` to devtmpfs_work_loop() as the the
>> last instruction in devtmpfsd() and therefore on entry to devtmpfs_work_loop(),
>> LR points to an instruction beyond devtmpfsd() and this consfuses the unwinder.
>> 
>> ffff800080d9a070 <devtmpfsd>:
>> ffff800080d9a070:       d503201f        nop
>> ffff800080d9a074:       d503201f        nop
>> ffff800080d9a078:       d503233f        paciasp
>> ffff800080d9a07c:       a9be7bfd        stp     x29, x30, [sp, #-32]!
>> ffff800080d9a080:       910003fd        mov     x29, sp
>> ffff800080d9a084:       f9000bf3        str     x19, [sp, #16]
>> ffff800080d9a088:       943378e8        bl      ffff800081a78428 <devtmpfs_setup>
>> ffff800080d9a08c:       90006ca1        adrp    x1, ffff800081b2e000 <unique_processor_ids+0x3758>
>> ffff800080d9a090:       2a0003f3        mov     w19, w0
>> ffff800080d9a094:       912de021        add     x1, x1, #0xb78
>> ffff800080d9a098:       91002020        add     x0, x1, #0x8
>> ffff800080d9a09c:       97cd2a43        bl      ffff8000800e49a8 <complete>
>> ffff800080d9a0a0:       340000d3        cbz     w19, ffff800080d9a0b8 <devtmpfsd+0x48>
>> ffff800080d9a0a4:       2a1303e0        mov     w0, w19
>> ffff800080d9a0a8:       f9400bf3        ldr     x19, [sp, #16]
>> ffff800080d9a0ac:       a8c27bfd        ldp     x29, x30, [sp], #32
>> ffff800080d9a0b0:       d50323bf        autiasp
>> ffff800080d9a0b4:       d65f03c0        ret
>> ffff800080d9a0b8:       97f06526        bl      ffff8000809b3550 <devtmpfs_work_loop>
>> ffff800080d9a0bc:       00000000        udf     #0
>> ffff800080d9a0c0:       d503201f        nop
>> ffff800080d9a0c4:       d503201f        nop
>> 
>> find_fde() got pc=0xffff800080d9a0bc which is not in [sfde_func_start_address, sfde_func_size)
>> 
>> output for readelf --sframe for devtmpfsd()
>> 
>> func idx [51825]: pc = 0xffff800080d9a070, size = 76 bytes
>>     STARTPC           CFA       FP        RA
>>     ffff800080d9a070  sp+0      u         u
>>     ffff800080d9a07c  sp+0      u         u[s]
>>     ffff800080d9a080  sp+32     c-32      c-24[s]
>>     ffff800080d9a0b0  sp+0      u         u[s]
>>     ffff800080d9a0b4  sp+0      u         u
>>     ffff800080d9a0b8  sp+32     c-32      c-24[s]
>> 
>> The unwinder and all the related infra is assuming that the return address
>> will be part of a valid function which is not the case here.
>> 
>> I am not sure which component needs to be fixed here, but the following
>> patch(which is a hack) fixes the issue by considering the return address as
>> part of the function descriptor entry.
>> 
>> -- 8< --
>> 
>> diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
>> index 846f1da95..28bec5064 100644
>> --- a/kernel/sframe_lookup.c
>> +++ b/kernel/sframe_lookup.c
>> @@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned long
>>         if (f >= tbl->sfhdr_p->num_fdes || f < 0)
>>                 return NULL;
>>         fdep = tbl->fde_p + f;
>> -       if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
>> +       if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)
>>                 return NULL;
>> 
>>         return fdep;
>> @@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,
>>         else
>>                 ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - fdep->start_addr;
>> 
>> -       if (ip_off < 0 || ip_off >= fdep->size)
>> +       if (ip_off < 0 || ip_off > fdep->size)
>>                 return -EINVAL;
>> 
>>         /*
>> 
>> -- >8 --
>> 
>> Thanks,
>> Puranjay
>
> Thank you for reporting this issue.
> I just found out that Josh also intentionally uses '>' instead of '>=' for the same reason
> https://lore.kernel.org/lkml/20250122225257.h64ftfnorofe7cb4@jpoimboe/T/#m6d70a20ed9f5b3bbe5b24b24b8c5dcc603a79101
>
> QQ, do we need to care the stacktrace after '__noreturn' function?

Yes, I think we should, but others people could add more to this.

I have been testing this series with Kpatch and created a PR that works
with this unwinder: https://github.com/dynup/kpatch/pull/1439

For the modules, I think we need per module sframe tables that are
initialised when the module is loaded. And the unwinder should use the
module specific table if the IP is in a module's code.

Have you already started working on it? if not I would like to help and
work on that.

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 07, 2025 at 12:16:29PM +0000, Puranjay Mohan wrote:
> Weinan Liu <wnliu@google.com> writes:
> > Thank you for reporting this issue.
> > I just found out that Josh also intentionally uses '>' instead of '>=' for the same reason
> > https://lore.kernel.org/lkml/20250122225257.h64ftfnorofe7cb4@jpoimboe/T/#m6d70a20ed9f5b3bbe5b24b24b8c5dcc603a79101
> >
> > QQ, do we need to care the stacktrace after '__noreturn' function?
> 
> Yes, I think we should, but others people could add more to this.

FYI, here's how ORC handles this:

	/*
	 * Find the orc_entry associated with the text address.
	 *
	 * For a call frame (as opposed to a signal frame), state->ip points to
	 * the instruction after the call.  That instruction's stack layout
	 * could be different from the call instruction's layout, for example
	 * if the call was to a noreturn function.  So get the ORC data for the
	 * call instruction itself.
	 */
	orc = orc_find(state->signal ? state->ip : state->ip - 1);

and state->signal is only set for exceptions/interrupts (including
preemption and page faults) and newly forked tasks which haven't run
yet.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

On Fri, Feb 7, 2025 at 4:16 AM Puranjay Mohan <puranjay@kernel.org> wrote:
>
> Yes, I think we should, but others people could add more to this.
> 
> I have been testing this series with Kpatch and created a PR that works
> with this unwinder: https://github.com/dynup/kpatch/pull/1439
> 
> For the modules, I think we need per module sframe tables that are
> initialised when the module is loaded. And the unwinder should use the
> module specific table if the IP is in a module's code.
> 
> Have you already started working on it? if not I would like to help and
> work on that.
> 
> Thanks,
> Puranjay

Thanks for updating the arm64 kpatch PR so quickly.
So we can use kpatch to test this proposal.

I already have a WIP patch to add sframe support to the kernel module.
However, it is not yet working. I had trouble unwinding frames for the 
kernel module using the current algorithm.

Indu has likely identified the issue and will be addressing it from the
toolchain side.

https://sourceware.org/bugzilla/show_bug.cgi?id=32666

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
> I already have a WIP patch to add sframe support to the kernel module.
> However, it is not yet working. I had trouble unwinding frames for the
> kernel module using the current algorithm.
>
> Indu has likely identified the issue and will be addressing it from the
> toolchain side.
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=32666


I have a working in progress patch that adds sframe support for kernel
module.
https://github.com/heuza/linux/tree/sframe_unwinder.rfc

According to the sframe table values I got during runtime testing, looks
like the offsets are not correct .

When unwind symbols init_module(0xffff80007b155048) from the kernel 
module(livepatch-sample.ko), the start_address of the FDE entries in the
sframe table of the kernel modules appear incorrect.
For instance, the first FDE's start_addr is reported as -20564. Adding
this offset to the module's sframe section address (0xffff80007b15a040)
yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
memory region(It should be larger than 0xffff80007b155000).

Here are the sframe table values of the livepatch-samples.ko that I print
by qemu + gdb.

```
$ /usr/bin/aarch64-linux-gnu-objdump -L --sframe=.sframe ./samples/livepatch/livepatch-sample.ko
./samples/livepatch/livepatch-sample.ko:     file format elf64-littleaarch64

Contents of the SFrame section .sframe:
  Header :

    Version: SFRAME_VERSION_2
    Flags: SFRAME_F_FDE_SORTED
    Num FDEs: 3
    Num FREs: 11

  Function Index :

    func idx [0]: pc = 0x0, size = 12 bytes
    STARTPC         CFA       FP        RA
    0000000000000000  sp+0      u         u

    func idx [1]: pc = 0x0, size = 44 bytes
    STARTPC         CFA       FP        RA
    0000000000000000  sp+0      u         u
    000000000000000c  sp+0      u         u[s]
    0000000000000010  sp+16     c-16      c-8[s]
    0000000000000024  sp+0      u         u[s]
    0000000000000028  sp+0      u         u

    func idx [2]: pc = 0x0, size = 56 bytes
    STARTPC         CFA       FP        RA
    0000000000000000  sp+0      u         u
    000000000000000c  sp+0      u         u[s]
    0000000000000010  sp+16     c-16      c-8[s]
    0000000000000030  sp+0      u         u[s]
    0000000000000034  sp+0      u         u



(gdb) bt
#0  find_fde (tbl=0xffff80007b157708, pc=18446603338286190664) at kernel/sframe_lookup.c:75
#1  0xffff80008031e260 in sframe_find_pc (pc=18446603338286190664, entry=0xffff800086f83800) at kernel/sframe_lookup.c:175
#2  0xffff800080035a48 in unwind_next_frame_sframe (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:270
#3  kunwind_next (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:332
...

(gdb) lx-symbols
loading vmlinux
scanning for modules in /home/wnliu/kernel
loading @0xffff80007b155000: /home/wnliu/kernel/samples/livepatch/livepatch-sample.ko
loading @0xffff80007b14d000: /home/wnliu/kernel/fs/fat/vfat.ko
loading @0xffff80007b130000: /home/wnliu/kernel/fs/fat/fat.ko

(gdb) p/x *tbl->sfhdr_p
$5 = {preamble = {magic = 0xdee2, version = 0x2, flags = 0x1}, abi_arch = 0x2, cfa_fixed_fp_offset = 0x0, cfa_fixed_ra_offset = 0x0, auxhdr_len = 0x0, num_fdes = 0x3, num_fres = 0xb, fre_len = 0x25, fdes_off = 0x0, fres_off = 0x3c}

(gdb) p/x tbl->sfhdr_p
$6 = 0xffff80007b15a040

(gdb) p *tbl->fde_p
$7 = {start_addr = -20564, size = 12, fres_off = 0, fres_num = 1, info = 0 '\000', rep_size = 0 '\000', padding = 0}

(gdb) p *(tbl->fde_p + 1)
$11 = {start_addr = -20552, size = 44, fres_off = 3, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0}

(gdb) p *(tbl->fde_p + 2)
$12 = {start_addr = -20508, size = 56, fres_off = 20, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0}

/* -20564 + 0xffff80007b15a040 = 0xffff80007b154fec */
(gdb) info symbol 0xffff80007b154fec
No symbol matches 0xffff80007b154fec
```

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Tue, Feb 25, 2025 at 01:02:24AM +0000, Weinan Liu wrote:
> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
> > I already have a WIP patch to add sframe support to the kernel module.
> > However, it is not yet working. I had trouble unwinding frames for the
> > kernel module using the current algorithm.
> >
> > Indu has likely identified the issue and will be addressing it from the
> > toolchain side.
> >
> > https://sourceware.org/bugzilla/show_bug.cgi?id=32666
> 
> 
> I have a working in progress patch that adds sframe support for kernel
> module.
> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
> 
> According to the sframe table values I got during runtime testing, looks
> like the offsets are not correct .
> 
> When unwind symbols init_module(0xffff80007b155048) from the kernel 
> module(livepatch-sample.ko), the start_address of the FDE entries in the
> sframe table of the kernel modules appear incorrect.
> For instance, the first FDE's start_addr is reported as -20564. Adding
> this offset to the module's sframe section address (0xffff80007b15a040)
> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
> memory region(It should be larger than 0xffff80007b155000).

I assume kpatch create-diff-object needs to copy over a subset of the
.sframe section.  Similar to what kpatch_regenerate_orc_sections() does.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

On Tue, Feb 25, 2025 at 10:13 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Tue, Feb 25, 2025 at 01:02:24AM +0000, Weinan Liu wrote:
> > On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
> > > I already have a WIP patch to add sframe support to the kernel module.
> > > However, it is not yet working. I had trouble unwinding frames for the
> > > kernel module using the current algorithm.
> > >
> > > Indu has likely identified the issue and will be addressing it from the
> > > toolchain side.
> > >
> > > https://sourceware.org/bugzilla/show_bug.cgi?id=32666
> >
> >
> > I have a working in progress patch that adds sframe support for kernel
> > module.
> > https://github.com/heuza/linux/tree/sframe_unwinder.rfc
> >
> > According to the sframe table values I got during runtime testing, looks
> > like the offsets are not correct .
> >
> > When unwind symbols init_module(0xffff80007b155048) from the kernel
> > module(livepatch-sample.ko), the start_address of the FDE entries in the
> > sframe table of the kernel modules appear incorrect.
> > For instance, the first FDE's start_addr is reported as -20564. Adding
> > this offset to the module's sframe section address (0xffff80007b15a040)
> > yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
> > memory region(It should be larger than 0xffff80007b155000).
>
> I assume kpatch create-diff-object needs to copy over a subset of the
> .sframe section.  Similar to what kpatch_regenerate_orc_sections() does.


You're right that we need to process the sframe section like what
kpatch_regenerate_orc_sections() does when building livepatch by kpatch.

However, livepatch-sample.ko is not generated by kpatch. It is built
directly from samples/livepatch/livepatch-sample.c by gcc during the kernel
build

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
>> I already have a WIP patch to add sframe support to the kernel module.
>> However, it is not yet working. I had trouble unwinding frames for the
>> kernel module using the current algorithm.
>>
>> Indu has likely identified the issue and will be addressing it from the
>> toolchain side.
>>
>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
> 
> I have a working in progress patch that adds sframe support for kernel
> module.
> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
> 
> According to the sframe table values I got during runtime testing, looks
> like the offsets are not correct .
> 

I hope to sanitize the fix for 32666 and post upstream soon (I had to 
address other related issues).  Unless fixed, relocating .sframe 
sections using the .rela.sframe is expected to generate incorrect output.

> When unwind symbols init_module(0xffff80007b155048) from the kernel
> module(livepatch-sample.ko), the start_address of the FDE entries in the
> sframe table of the kernel modules appear incorrect.

init_module will apply the relocations on the .sframe section, isnt it ?

> For instance, the first FDE's start_addr is reported as -20564. Adding
> this offset to the module's sframe section address (0xffff80007b15a040)
> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
> memory region(It should be larger than 0xffff80007b155000).
> 

Hmm..something seems off here.  Having tested a potential fix for 32666 
locally, I do not expect the first FDE to show this symptom.

> Here are the sframe table values of the livepatch-samples.ko that I print
> by qemu + gdb.
> 
> ```
> $ /usr/bin/aarch64-linux-gnu-objdump -L --sframe=.sframe ./samples/livepatch/livepatch-sample.ko
> ./samples/livepatch/livepatch-sample.ko:     file format elf64-littleaarch64
> 
> Contents of the SFrame section .sframe:
>   Header :
> 
>     Version: SFRAME_VERSION_2
>     Flags: SFRAME_F_FDE_SORTED
>     Num FDEs: 3
>     Num FREs: 11
> 
>   Function Index :
> 
>     func idx [0]: pc = 0x0, size = 12 bytes
>     STARTPC         CFA       FP        RA
>     0000000000000000  sp+0      u         u
> 
>     func idx [1]: pc = 0x0, size = 44 bytes
>     STARTPC         CFA       FP        RA
>     0000000000000000  sp+0      u         u
>     000000000000000c  sp+0      u         u[s]
>     0000000000000010  sp+16     c-16      c-8[s]
>     0000000000000024  sp+0      u         u[s]
>     0000000000000028  sp+0      u         u
> 
>     func idx [2]: pc = 0x0, size = 56 bytes
>     STARTPC         CFA       FP        RA
>     0000000000000000  sp+0      u         u
>     000000000000000c  sp+0      u         u[s]
>     0000000000000010  sp+16     c-16      c-8[s]
>     0000000000000030  sp+0      u         u[s]
>     0000000000000034  sp+0      u         u
> 
> 
> 
> (gdb) bt
> #0  find_fde (tbl=0xffff80007b157708, pc=18446603338286190664) at kernel/sframe_lookup.c:75
> #1  0xffff80008031e260 in sframe_find_pc (pc=18446603338286190664, entry=0xffff800086f83800) at kernel/sframe_lookup.c:175
> #2  0xffff800080035a48 in unwind_next_frame_sframe (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:270
> #3  kunwind_next (state=0xffff800086f83828) at arch/arm64/kernel/stacktrace.c:332
> ...
> 
> (gdb) lx-symbols
> loading vmlinux
> scanning for modules in /home/wnliu/kernel
> loading @0xffff80007b155000: /home/wnliu/kernel/samples/livepatch/livepatch-sample.ko
> loading @0xffff80007b14d000: /home/wnliu/kernel/fs/fat/vfat.ko
> loading @0xffff80007b130000: /home/wnliu/kernel/fs/fat/fat.ko
> 
> (gdb) p/x *tbl->sfhdr_p
> $5 = {preamble = {magic = 0xdee2, version = 0x2, flags = 0x1}, abi_arch = 0x2, cfa_fixed_fp_offset = 0x0, cfa_fixed_ra_offset = 0x0, auxhdr_len = 0x0, num_fdes = 0x3, num_fres = 0xb, fre_len = 0x25, fdes_off = 0x0, fres_off = 0x3c}
> 
> (gdb) p/x tbl->sfhdr_p
> $6 = 0xffff80007b15a040
> 
> (gdb) p *tbl->fde_p
> $7 = {start_addr = -20564, size = 12, fres_off = 0, fres_num = 1, info = 0 '\000', rep_size = 0 '\000', padding = 0}
> 
> (gdb) p *(tbl->fde_p + 1)
> $11 = {start_addr = -20552, size = 44, fres_off = 3, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0}
> 
> (gdb) p *(tbl->fde_p + 2)
> $12 = {start_addr = -20508, size = 56, fres_off = 20, fres_num = 5, info = 0 '\000', rep_size = 0 '\000', padding = 0}
> 
> /* -20564 + 0xffff80007b15a040 = 0xffff80007b154fec */
> (gdb) info symbol 0xffff80007b154fec
> No symbol matches 0xffff80007b154fec
> ```

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Weinan Liu]

On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
> >> I already have a WIP patch to add sframe support to the kernel module.
> >> However, it is not yet working. I had trouble unwinding frames for the
> >> kernel module using the current algorithm.
> >>
> >> Indu has likely identified the issue and will be addressing it from the
> >> toolchain side.
> >>
> >> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
> >
> > I have a working in progress patch that adds sframe support for kernel
> > module.
> > https://github.com/heuza/linux/tree/sframe_unwinder.rfc
> >
> > According to the sframe table values I got during runtime testing, looks
> > like the offsets are not correct .
> >
>
> I hope to sanitize the fix for 32666 and post upstream soon (I had to
> address other related issues).  Unless fixed, relocating .sframe
> sections using the .rela.sframe is expected to generate incorrect output.
>
> > When unwind symbols init_module(0xffff80007b155048) from the kernel
> > module(livepatch-sample.ko), the start_address of the FDE entries in the
> > sframe table of the kernel modules appear incorrect.
>
> init_module will apply the relocations on the .sframe section, isnt it ?
>
> > For instance, the first FDE's start_addr is reported as -20564. Adding
> > this offset to the module's sframe section address (0xffff80007b15a040)
> > yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
> > memory region(It should be larger than 0xffff80007b155000).
> >
>
> Hmm..something seems off here.  Having tested a potential fix for 32666
> locally, I do not expect the first FDE to show this symptom.
>

Yes, I think init_module will apply the relocation as well.
To further investigate, here's the relevant relocation and symbol table
information for the kernel module:

Relocation section '.rela.sframe' at offset 0x28350 contains 3 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000001c  000100000105 R_AARCH64_PREL32  0000000000000000 .text + 8
000000000030  000100000105 R_AARCH64_PREL32  0000000000000000 .text + 28
000000000044  000100000105 R_AARCH64_PREL32  0000000000000000 .text + 68

Symbol table '.symtab' contains 68 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 .text
...
    32: 0000000000000008    12 FUNC    LOCAL  DEFAULT    1 livepatch_exit
    33: 0000000000000008     0 NOTYPE  LOCAL  DEFAULT    3 $d
    34: 0000000000000028    44 FUNC    LOCAL  DEFAULT    1 livepatch_init
    35: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT    9 $d
    36: 0000000000000010     0 NOTYPE  LOCAL  DEFAULT    3 $d
    37: 0000000000000068    56 FUNC    LOCAL  DEFAULT    1 livepatch_cmdlin[...]
...
    63: 0000000000000008    12 FUNC    GLOBAL DEFAULT    1 cleanup_module
    64: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND klp_enable_patch
    65: 0000000000000028    44 FUNC    GLOBAL DEFAULT    1 init_module

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/25/25 3:54 PM, Weinan Liu wrote:
> On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>
>> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
>>>> I already have a WIP patch to add sframe support to the kernel module.
>>>> However, it is not yet working. I had trouble unwinding frames for the
>>>> kernel module using the current algorithm.
>>>>
>>>> Indu has likely identified the issue and will be addressing it from the
>>>> toolchain side.
>>>>
>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
>>>
>>> I have a working in progress patch that adds sframe support for kernel
>>> module.
>>> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
>>>
>>> According to the sframe table values I got during runtime testing, looks
>>> like the offsets are not correct .
>>>
>>
>> I hope to sanitize the fix for 32666 and post upstream soon (I had to
>> address other related issues).  Unless fixed, relocating .sframe
>> sections using the .rela.sframe is expected to generate incorrect output.
>>
>>> When unwind symbols init_module(0xffff80007b155048) from the kernel
>>> module(livepatch-sample.ko), the start_address of the FDE entries in the
>>> sframe table of the kernel modules appear incorrect.
>>
>> init_module will apply the relocations on the .sframe section, isnt it ?
>>
>>> For instance, the first FDE's start_addr is reported as -20564. Adding
>>> this offset to the module's sframe section address (0xffff80007b15a040)
>>> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>> memory region(It should be larger than 0xffff80007b155000).
>>>
>>
>> Hmm..something seems off here.  Having tested a potential fix for 32666
>> locally, I do not expect the first FDE to show this symptom.
>>
> 
> Yes, I think init_module will apply the relocation as well.
> To further investigate, here's the relevant relocation and symbol table
> information for the kernel module:
> 
> Relocation section '.rela.sframe' at offset 0x28350 contains 3 entries:
>    Offset          Info           Type           Sym. Value    Sym. Name + Addend
> 00000000001c  000100000105 R_AARCH64_PREL32  0000000000000000 .text + 8
> 000000000030  000100000105 R_AARCH64_PREL32  0000000000000000 .text + 28
> 000000000044  000100000105 R_AARCH64_PREL32  0000000000000000 .text + 68
> 

The offsets look OK..

> Symbol table '.symtab' contains 68 entries:
>     Num:    Value          Size Type    Bind   Vis      Ndx Name
>       0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
>       1: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 .text
> ...
>      32: 0000000000000008    12 FUNC    LOCAL  DEFAULT    1 livepatch_exit
>      33: 0000000000000008     0 NOTYPE  LOCAL  DEFAULT    3 $d
>      34: 0000000000000028    44 FUNC    LOCAL  DEFAULT    1 livepatch_init
>      35: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT    9 $d
>      36: 0000000000000010     0 NOTYPE  LOCAL  DEFAULT    3 $d
>      37: 0000000000000068    56 FUNC    LOCAL  DEFAULT    1 livepatch_cmdlin[...]
> ...
>      63: 0000000000000008    12 FUNC    GLOBAL DEFAULT    1 cleanup_module
>      64: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND klp_enable_patch
>      65: 0000000000000028    44 FUNC    GLOBAL DEFAULT    1 init_module

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 3656 bytes --]

Indu Bhagat <indu.bhagat@oracle.com> writes:

> On 2/25/25 3:54 PM, Weinan Liu wrote:
>> On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>
>>> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
>>>>> I already have a WIP patch to add sframe support to the kernel module.
>>>>> However, it is not yet working. I had trouble unwinding frames for the
>>>>> kernel module using the current algorithm.
>>>>>
>>>>> Indu has likely identified the issue and will be addressing it from the
>>>>> toolchain side.
>>>>>
>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
>>>>
>>>> I have a working in progress patch that adds sframe support for kernel
>>>> module.
>>>> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
>>>>
>>>> According to the sframe table values I got during runtime testing, looks
>>>> like the offsets are not correct .
>>>>
>>>
>>> I hope to sanitize the fix for 32666 and post upstream soon (I had to
>>> address other related issues).  Unless fixed, relocating .sframe
>>> sections using the .rela.sframe is expected to generate incorrect output.
>>>
>>>> When unwind symbols init_module(0xffff80007b155048) from the kernel
>>>> module(livepatch-sample.ko), the start_address of the FDE entries in the
>>>> sframe table of the kernel modules appear incorrect.
>>>
>>> init_module will apply the relocations on the .sframe section, isnt it ?
>>>
>>>> For instance, the first FDE's start_addr is reported as -20564. Adding
>>>> this offset to the module's sframe section address (0xffff80007b15a040)
>>>> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>>> memory region(It should be larger than 0xffff80007b155000).
>>>>
>>>
>>> Hmm..something seems off here.  Having tested a potential fix for 32666
>>> locally, I do not expect the first FDE to show this symptom.
>>>
>> 

Hi,

Sorry for not responding in the past few days.  I was on PTO and was
trying to improve my snowboarding technique, I am back now!!

I think what we are seeing is expected behaviour:

 | For instance, the first FDE's start_addr is reported as -20564. Adding
 | this offset to the module's sframe section address (0xffff80007b15a040)
 | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
 | memory region(It should be larger than 0xffff80007b155000).


Let me explain using a __dummy__ example.

Assume Memory layout before relocation:

 | Address | Element                                 | Relocation
 |  ....   | ....                                    |
 |   60    | init_module (start address)             |
 |   72    | init_module (end address)               |
 |  ....   | .....                                   |
 |   100   | Sframe section header start address     |
 |   128   | First FDE's start address               | RELOC_OP_PREL -> Put init_module address (60) - current address (128)

So, after relocation First FDE's start address has value 60 - 128 = -68

Now, while doing unwinding we Try to add this value to the sframe
section header's start address which is in this example 100,

so 100 + (-68) = 32

So, 32 is not within [60, 72], i.e. within init_module.

You can see that it is possible for this value to be less than the start
address of the module's memory region when this function's address is
very close to the start of the memory region.

The crux is that the offset in the FDE's start address is calculated
based on the address of the FDE's start_address and not based on the
address of the sframe section.


Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/26/25 2:23 AM, Puranjay Mohan wrote:
> Indu Bhagat <indu.bhagat@oracle.com> writes:
> 
>> On 2/25/25 3:54 PM, Weinan Liu wrote:
>>> On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>>
>>>> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
>>>>>> I already have a WIP patch to add sframe support to the kernel module.
>>>>>> However, it is not yet working. I had trouble unwinding frames for the
>>>>>> kernel module using the current algorithm.
>>>>>>
>>>>>> Indu has likely identified the issue and will be addressing it from the
>>>>>> toolchain side.
>>>>>>
>>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
>>>>>
>>>>> I have a working in progress patch that adds sframe support for kernel
>>>>> module.
>>>>> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
>>>>>
>>>>> According to the sframe table values I got during runtime testing, looks
>>>>> like the offsets are not correct .
>>>>>
>>>>
>>>> I hope to sanitize the fix for 32666 and post upstream soon (I had to
>>>> address other related issues).  Unless fixed, relocating .sframe
>>>> sections using the .rela.sframe is expected to generate incorrect output.
>>>>
>>>>> When unwind symbols init_module(0xffff80007b155048) from the kernel
>>>>> module(livepatch-sample.ko), the start_address of the FDE entries in the
>>>>> sframe table of the kernel modules appear incorrect.
>>>>
>>>> init_module will apply the relocations on the .sframe section, isnt it ?
>>>>
>>>>> For instance, the first FDE's start_addr is reported as -20564. Adding
>>>>> this offset to the module's sframe section address (0xffff80007b15a040)
>>>>> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>>>> memory region(It should be larger than 0xffff80007b155000).
>>>>>
>>>>
>>>> Hmm..something seems off here.  Having tested a potential fix for 32666
>>>> locally, I do not expect the first FDE to show this symptom.
>>>>
>>>
> 
> Hi,
> 
> Sorry for not responding in the past few days.  I was on PTO and was
> trying to improve my snowboarding technique, I am back now!!
> 
> I think what we are seeing is expected behaviour:
> 
>   | For instance, the first FDE's start_addr is reported as -20564. Adding
>   | this offset to the module's sframe section address (0xffff80007b15a040)
>   | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>   | memory region(It should be larger than 0xffff80007b155000).
> 
> 
> Let me explain using a __dummy__ example.
> 
> Assume Memory layout before relocation:
> 
>   | Address | Element                                 | Relocation
>   |  ....   | ....                                    |
>   |   60    | init_module (start address)             |
>   |   72    | init_module (end address)               |
>   |  ....   | .....                                   |
>   |   100   | Sframe section header start address     |
>   |   128   | First FDE's start address               | RELOC_OP_PREL -> Put init_module address (60) - current address (128)
> 
> So, after relocation First FDE's start address has value 60 - 128 = -68
> 

For SFrame FDE function start address is :

"Signed 32-bit integral field denoting the virtual memory address of the 
described function, for which the SFrame FDE applies.  The value encoded 
in the ‘sfde_func_start_address’ field is the offset in bytes of the 
function’s start address, from the SFrame section."

So, in your case, after applying the relocations, you will get:
S + A - P = 60 - 128 = -68

This is the distance of the function start address (60) from the current 
location in SFrame section (128)

But what we intend to store is the distance of the function start 
address from the start of the SFrame section.  So we need to do an 
additional step for SFrame FDE:  Value += r_offset

-68 + 28 = -40
Where 28 is the r_offset in the RELA.

So we really expect a -40 in the relocated SFrame section instead of -68 
above.  IOW, the RELAs of SFrame section will need an additional step 
after relocation.

> Now, while doing unwinding we Try to add this value to the sframe
> section header's start address which is in this example 100,
> 
> so 100 + (-68) = 32
> 
> So, 32 is not within [60, 72], i.e. within init_module.
> 
> You can see that it is possible for this value to be less than the start
> address of the module's memory region when this function's address is
> very close to the start of the memory region.
> 
> The crux is that the offset in the FDE's start address is calculated
> based on the address of the FDE's start_address and not based on the
> address of the sframe section.
> 
> 
> Thanks,
> Puranjay

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 4993 bytes --]

Indu Bhagat <indu.bhagat@oracle.com> writes:

> On 2/26/25 2:23 AM, Puranjay Mohan wrote:
>> Indu Bhagat <indu.bhagat@oracle.com> writes:
>> 
>>> On 2/25/25 3:54 PM, Weinan Liu wrote:
>>>> On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>>>
>>>>> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
>>>>>>> I already have a WIP patch to add sframe support to the kernel module.
>>>>>>> However, it is not yet working. I had trouble unwinding frames for the
>>>>>>> kernel module using the current algorithm.
>>>>>>>
>>>>>>> Indu has likely identified the issue and will be addressing it from the
>>>>>>> toolchain side.
>>>>>>>
>>>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
>>>>>>
>>>>>> I have a working in progress patch that adds sframe support for kernel
>>>>>> module.
>>>>>> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
>>>>>>
>>>>>> According to the sframe table values I got during runtime testing, looks
>>>>>> like the offsets are not correct .
>>>>>>
>>>>>
>>>>> I hope to sanitize the fix for 32666 and post upstream soon (I had to
>>>>> address other related issues).  Unless fixed, relocating .sframe
>>>>> sections using the .rela.sframe is expected to generate incorrect output.
>>>>>
>>>>>> When unwind symbols init_module(0xffff80007b155048) from the kernel
>>>>>> module(livepatch-sample.ko), the start_address of the FDE entries in the
>>>>>> sframe table of the kernel modules appear incorrect.
>>>>>
>>>>> init_module will apply the relocations on the .sframe section, isnt it ?
>>>>>
>>>>>> For instance, the first FDE's start_addr is reported as -20564. Adding
>>>>>> this offset to the module's sframe section address (0xffff80007b15a040)
>>>>>> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>>>>> memory region(It should be larger than 0xffff80007b155000).
>>>>>>
>>>>>
>>>>> Hmm..something seems off here.  Having tested a potential fix for 32666
>>>>> locally, I do not expect the first FDE to show this symptom.
>>>>>
>>>>
>> 
>> Hi,
>> 
>> Sorry for not responding in the past few days.  I was on PTO and was
>> trying to improve my snowboarding technique, I am back now!!
>> 
>> I think what we are seeing is expected behaviour:
>> 
>>   | For instance, the first FDE's start_addr is reported as -20564. Adding
>>   | this offset to the module's sframe section address (0xffff80007b15a040)
>>   | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>   | memory region(It should be larger than 0xffff80007b155000).
>> 
>> 
>> Let me explain using a __dummy__ example.
>> 
>> Assume Memory layout before relocation:
>> 
>>   | Address | Element                                 | Relocation
>>   |  ....   | ....                                    |
>>   |   60    | init_module (start address)             |
>>   |   72    | init_module (end address)               |
>>   |  ....   | .....                                   |
>>   |   100   | Sframe section header start address     |
>>   |   128   | First FDE's start address               | RELOC_OP_PREL -> Put init_module address (60) - current address (128)
>> 
>> So, after relocation First FDE's start address has value 60 - 128 = -68
>> 
>
> For SFrame FDE function start address is :
>
> "Signed 32-bit integral field denoting the virtual memory address of the 
> described function, for which the SFrame FDE applies.  The value encoded 
> in the ‘sfde_func_start_address’ field is the offset in bytes of the 
> function’s start address, from the SFrame section."
>
> So, in your case, after applying the relocations, you will get:
> S + A - P = 60 - 128 = -68
>
> This is the distance of the function start address (60) from the current 
> location in SFrame section (128)
>
> But what we intend to store is the distance of the function start 
> address from the start of the SFrame section.  So we need to do an 
> additional step for SFrame FDE:  Value += r_offset

Thanks for the explaination, now it makes sense.

But I couldn't find a relocation type in AARCH64 that does this extra +=
r_offset along with PREL32.

The kernel's module loader is only doing the R_AARCH64_PREL32 which is
why we see this issue.

How is this working even for the kernel itself? or for that matter, any
other binary compiled with sframe?

From my limited undestanding, the way to fix this would be to hack the
relocator to do this additional step while relocating .sframe sections.
Or the 'addend' values in .rela.sframe should already have the +r_offset
added to it, then no change to the relocator would be needed.

> -68 + 28 = -40
> Where 28 is the r_offset in the RELA.
>
> So we really expect a -40 in the relocated SFrame section instead of -68 
> above.  IOW, the RELAs of SFrame section will need an additional step 
> after relocation.
>

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/27/25 1:38 AM, Puranjay Mohan wrote:
> Indu Bhagat <indu.bhagat@oracle.com> writes:
> 
>> On 2/26/25 2:23 AM, Puranjay Mohan wrote:
>>> Indu Bhagat <indu.bhagat@oracle.com> writes:
>>>
>>>> On 2/25/25 3:54 PM, Weinan Liu wrote:
>>>>> On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>>>>>
>>>>>> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> wrote:
>>>>>>>> I already have a WIP patch to add sframe support to the kernel module.
>>>>>>>> However, it is not yet working. I had trouble unwinding frames for the
>>>>>>>> kernel module using the current algorithm.
>>>>>>>>
>>>>>>>> Indu has likely identified the issue and will be addressing it from the
>>>>>>>> toolchain side.
>>>>>>>>
>>>>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
>>>>>>>
>>>>>>> I have a working in progress patch that adds sframe support for kernel
>>>>>>> module.
>>>>>>> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
>>>>>>>
>>>>>>> According to the sframe table values I got during runtime testing, looks
>>>>>>> like the offsets are not correct .
>>>>>>>
>>>>>>
>>>>>> I hope to sanitize the fix for 32666 and post upstream soon (I had to
>>>>>> address other related issues).  Unless fixed, relocating .sframe
>>>>>> sections using the .rela.sframe is expected to generate incorrect output.
>>>>>>
>>>>>>> When unwind symbols init_module(0xffff80007b155048) from the kernel
>>>>>>> module(livepatch-sample.ko), the start_address of the FDE entries in the
>>>>>>> sframe table of the kernel modules appear incorrect.
>>>>>>
>>>>>> init_module will apply the relocations on the .sframe section, isnt it ?
>>>>>>
>>>>>>> For instance, the first FDE's start_addr is reported as -20564. Adding
>>>>>>> this offset to the module's sframe section address (0xffff80007b15a040)
>>>>>>> yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>>>>>> memory region(It should be larger than 0xffff80007b155000).
>>>>>>>
>>>>>>
>>>>>> Hmm..something seems off here.  Having tested a potential fix for 32666
>>>>>> locally, I do not expect the first FDE to show this symptom.
>>>>>>
>>>>>
>>>
>>> Hi,
>>>
>>> Sorry for not responding in the past few days.  I was on PTO and was
>>> trying to improve my snowboarding technique, I am back now!!
>>>
>>> I think what we are seeing is expected behaviour:
>>>
>>>    | For instance, the first FDE's start_addr is reported as -20564. Adding
>>>    | this offset to the module's sframe section address (0xffff80007b15a040)
>>>    | yields 0xffff80007b154fec, which is not within the livepatch-sample.ko
>>>    | memory region(It should be larger than 0xffff80007b155000).
>>>
>>>
>>> Let me explain using a __dummy__ example.
>>>
>>> Assume Memory layout before relocation:
>>>
>>>    | Address | Element                                 | Relocation
>>>    |  ....   | ....                                    |
>>>    |   60    | init_module (start address)             |
>>>    |   72    | init_module (end address)               |
>>>    |  ....   | .....                                   |
>>>    |   100   | Sframe section header start address     |
>>>    |   128   | First FDE's start address               | RELOC_OP_PREL -> Put init_module address (60) - current address (128)
>>>
>>> So, after relocation First FDE's start address has value 60 - 128 = -68
>>>
>>
>> For SFrame FDE function start address is :
>>
>> "Signed 32-bit integral field denoting the virtual memory address of the
>> described function, for which the SFrame FDE applies.  The value encoded
>> in the ‘sfde_func_start_address’ field is the offset in bytes of the
>> function’s start address, from the SFrame section."
>>
>> So, in your case, after applying the relocations, you will get:
>> S + A - P = 60 - 128 = -68
>>
>> This is the distance of the function start address (60) from the current
>> location in SFrame section (128)
>>
>> But what we intend to store is the distance of the function start
>> address from the start of the SFrame section.  So we need to do an
>> additional step for SFrame FDE:  Value += r_offset
> 
> Thanks for the explaination, now it makes sense.
> 
> But I couldn't find a relocation type in AARCH64 that does this extra +=
> r_offset along with PREL32.
> 
> The kernel's module loader is only doing the R_AARCH64_PREL32 which is
> why we see this issue.
> 
> How is this working even for the kernel itself? or for that matter, any
> other binary compiled with sframe?
> 

For the usual executables or shared objects, the calculations are 
applied by ld.bfd at this time.  Hence, the issue manifests in 
relocatable files.

>  From my limited undestanding, the way to fix this would be to hack the
> relocator to do this additional step while relocating .sframe sections.
> Or the 'addend' values in .rela.sframe should already have the +r_offset
> added to it, then no change to the relocator would be needed.
> 

Of the two, adjusting the addend values in .rela.sframe may be a 
reasonable way to go about it.  Let me try it out in GAS and ld.bfd.

>> -68 + 28 = -40
>> Where 28 is the r_offset in the RELA.
>>
>> So we really expect a -40 in the relocated SFrame section instead of -68
>> above.  IOW, the RELAs of SFrame section will need an additional step
>> after relocation.
>>
> 
> Thanks,
> Puranjay

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/27/25 10:47 PM, Indu Bhagat wrote:
> On 2/27/25 1:38 AM, Puranjay Mohan wrote:
>> Indu Bhagat <indu.bhagat@oracle.com> writes:
>>
>>> On 2/26/25 2:23 AM, Puranjay Mohan wrote:
>>>> Indu Bhagat <indu.bhagat@oracle.com> writes:
>>>>
>>>>> On 2/25/25 3:54 PM, Weinan Liu wrote:
>>>>>> On Tue, Feb 25, 2025 at 11:38 AM Indu Bhagat 
>>>>>> <indu.bhagat@oracle.com> wrote:
>>>>>>>
>>>>>>> On Mon, Feb 10, 2025 at 12:30 AM Weinan Liu <wnliu@google.com> 
>>>>>>> wrote:
>>>>>>>>> I already have a WIP patch to add sframe support to the kernel 
>>>>>>>>> module.
>>>>>>>>> However, it is not yet working. I had trouble unwinding frames 
>>>>>>>>> for the
>>>>>>>>> kernel module using the current algorithm.
>>>>>>>>>
>>>>>>>>> Indu has likely identified the issue and will be addressing it 
>>>>>>>>> from the
>>>>>>>>> toolchain side.
>>>>>>>>>
>>>>>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666
>>>>>>>>
>>>>>>>> I have a working in progress patch that adds sframe support for 
>>>>>>>> kernel
>>>>>>>> module.
>>>>>>>> https://github.com/heuza/linux/tree/sframe_unwinder.rfc
>>>>>>>>
>>>>>>>> According to the sframe table values I got during runtime 
>>>>>>>> testing, looks
>>>>>>>> like the offsets are not correct .
>>>>>>>>
>>>>>>>
>>>>>>> I hope to sanitize the fix for 32666 and post upstream soon (I 
>>>>>>> had to
>>>>>>> address other related issues).  Unless fixed, relocating .sframe
>>>>>>> sections using the .rela.sframe is expected to generate incorrect 
>>>>>>> output.
>>>>>>>
>>>>>>>> When unwind symbols init_module(0xffff80007b155048) from the kernel
>>>>>>>> module(livepatch-sample.ko), the start_address of the FDE 
>>>>>>>> entries in the
>>>>>>>> sframe table of the kernel modules appear incorrect.
>>>>>>>
>>>>>>> init_module will apply the relocations on the .sframe section, 
>>>>>>> isnt it ?
>>>>>>>
>>>>>>>> For instance, the first FDE's start_addr is reported as -20564. 
>>>>>>>> Adding
>>>>>>>> this offset to the module's sframe section address 
>>>>>>>> (0xffff80007b15a040)
>>>>>>>> yields 0xffff80007b154fec, which is not within the livepatch- 
>>>>>>>> sample.ko
>>>>>>>> memory region(It should be larger than 0xffff80007b155000).
>>>>>>>>
>>>>>>>
>>>>>>> Hmm..something seems off here.  Having tested a potential fix for 
>>>>>>> 32666
>>>>>>> locally, I do not expect the first FDE to show this symptom.
>>>>>>>
>>>>>>
>>>>
>>>> Hi,
>>>>
>>>> Sorry for not responding in the past few days.  I was on PTO and was
>>>> trying to improve my snowboarding technique, I am back now!!
>>>>
>>>> I think what we are seeing is expected behaviour:
>>>>
>>>>    | For instance, the first FDE's start_addr is reported as -20564. 
>>>> Adding
>>>>    | this offset to the module's sframe section address 
>>>> (0xffff80007b15a040)
>>>>    | yields 0xffff80007b154fec, which is not within the livepatch- 
>>>> sample.ko
>>>>    | memory region(It should be larger than 0xffff80007b155000).
>>>>
>>>>
>>>> Let me explain using a __dummy__ example.
>>>>
>>>> Assume Memory layout before relocation:
>>>>
>>>>    | Address | Element                                 | Relocation
>>>>    |  ....   | ....                                    |
>>>>    |   60    | init_module (start address)             |
>>>>    |   72    | init_module (end address)               |
>>>>    |  ....   | .....                                   |
>>>>    |   100   | Sframe section header start address     |
>>>>    |   128   | First FDE's start address               | 
>>>> RELOC_OP_PREL -> Put init_module address (60) - current address (128)
>>>>
>>>> So, after relocation First FDE's start address has value 60 - 128 = -68
>>>>
>>>
>>> For SFrame FDE function start address is :
>>>
>>> "Signed 32-bit integral field denoting the virtual memory address of the
>>> described function, for which the SFrame FDE applies.  The value encoded
>>> in the ‘sfde_func_start_address’ field is the offset in bytes of the
>>> function’s start address, from the SFrame section."
>>>
>>> So, in your case, after applying the relocations, you will get:
>>> S + A - P = 60 - 128 = -68
>>>
>>> This is the distance of the function start address (60) from the current
>>> location in SFrame section (128)
>>>
>>> But what we intend to store is the distance of the function start
>>> address from the start of the SFrame section.  So we need to do an
>>> additional step for SFrame FDE:  Value += r_offset
>>
>> Thanks for the explaination, now it makes sense.
>>
>> But I couldn't find a relocation type in AARCH64 that does this extra +=
>> r_offset along with PREL32.
>>
>> The kernel's module loader is only doing the R_AARCH64_PREL32 which is
>> why we see this issue.
>>
>> How is this working even for the kernel itself? or for that matter, any
>> other binary compiled with sframe?
>>
> 
> For the usual executables or shared objects, the calculations are 
> applied by ld.bfd at this time.  Hence, the issue manifests in 
> relocatable files.
> 
>>  From my limited undestanding, the way to fix this would be to hack the
>> relocator to do this additional step while relocating .sframe sections.
>> Or the 'addend' values in .rela.sframe should already have the +r_offset
>> added to it, then no change to the relocator would be needed.
>>
> 
> Of the two, adjusting the addend values in .rela.sframe may be a 
> reasonable way to go about it.  Let me try it out in GAS and ld.bfd.
> 

A fix for this is in the works (being discussed on the 
binutils@sourceware list).  I will keep you posted.

Thanks
Indu

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

I run some tests with this set and my RFC set [1]. Most of
the test is done with kpatch-build. I tested both Puranjay's
version [3] and my version [4].

For gcc 14.2.1, I have seen the following issue with this
test [2]. This happens with both upstream and 6.13.2.
The livepatch loaded fine, but the system spilled out the
following warning quickly.

On the other hand, the same test works with LLVM and
my RFC set (LLVM doesn't support SFRAME, and thus
doesn't work with this set yet).

Thanks,
Song


[   81.250437] ------------[ cut here ]------------
[   81.250818] refcount_t: saturated; leaking memory.
[   81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22
refcount_warn_saturate+0x6c/0x140
[   81.251841] Modules linked in: livepatch_special_static(OEK)
[   81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G
OE K    6.13.2-00321-g52d2813b4b07 #49
[   81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
[   81.253503] Hardware name: linux,dummy-virt (DT)
[   81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[   81.254383] pc : refcount_warn_saturate+0x6c/0x140
[   81.254748] lr : refcount_warn_saturate+0x6c/0x140
[   81.255114] sp : ffff800085a6fc00
[   81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180
[   81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608
[   81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140
[   81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff
[   81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006
[   81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20
[   81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4
[   81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
[   81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
[   81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
[   81.260824] Call trace:
[   81.261015]  refcount_warn_saturate+0x6c/0x140 (P)
[   81.261387]  __refcount_add.constprop.0+0x60/0x70
[   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
[   81.262217]  kernel_clone+0x80/0x3e0
[   81.262499]  __do_sys_clone+0x5c/0x88
[   81.262786]  __arm64_sys_clone+0x24/0x38
[   81.263085]  invoke_syscall+0x4c/0x108
[   81.263378]  el0_svc_common.constprop.0+0x44/0xe8
[   81.263734]  do_el0_svc+0x20/0x30
[   81.263993]  el0_svc+0x34/0xf8
[   81.264231]  el0t_64_sync_handler+0x104/0x130
[   81.264561]  el0t_64_sync+0x184/0x188
[   81.264846] ---[ end trace 0000000000000000 ]---
[   82.335559] ------------[ cut here ]------------
[   82.335931] refcount_t: underflow; use-after-free.
[   82.336307] WARNING: CPU: 1 PID: 0 at lib/refcount.c:28
refcount_warn_saturate+0xec/0x140
[   82.336949] Modules linked in: livepatch_special_static(OEK)
[   82.337389] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G
W  OE K    6.13.2-00321-g52d2813b4b07 #49
[   82.338148] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE,
[K]=LIVEPATCH
[   82.338721] Hardware name: linux,dummy-virt (DT)
[   82.339083] pstate: 63400005 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[   82.339617] pc : refcount_warn_saturate+0xec/0x140
[   82.340007] lr : refcount_warn_saturate+0xec/0x140
[   82.340378] sp : ffff80008370fe40
[   82.340637] x29: ffff80008370fe40 x28: 0000000000000000 x27: 0000000000000000
[   82.341188] x26: 000000000000000a x25: ffff0000fdaf7ab8 x24: 0000000000000014
[   82.341737] x23: ffff8000829c8d30 x22: ffff80008370ff28 x21: ffff0000fe020000
[   82.342286] x20: ffff0000c062e140 x19: ffff0000c2e9af80 x18: ffffffffffffffff
[   82.342839] x17: ffff80007b7a0000 x16: ffff800083700000 x15: 0000000000000006
[   82.343389] x14: 0000000000000000 x13: 2e656572662d7265 x12: 7466612d65737520
[   82.343944] x11: ffff8000829f7d70 x10: 000000000000016a x9 : ffff8000801546b4
[   82.344499] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
[   82.345051] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
[   82.345604] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
[   82.346163] Call trace:
[   82.346359]  refcount_warn_saturate+0xec/0x140 (P)
[   82.346736]  __put_task_struct+0x130/0x170
[   82.347063]  delayed_put_task_struct+0xbc/0xe8
[   82.347411]  rcu_core+0x20c/0x5f8
[   82.347680]  rcu_core_si+0x14/0x28
[   82.347952]  handle_softirqs+0x124/0x308
[   82.348260]  __do_softirq+0x18/0x20
[   82.348536]  ____do_softirq+0x14/0x28
[   82.348828]  call_on_irq_stack+0x24/0x30
[   82.349137]  do_softirq_own_stack+0x20/0x38
[   82.349465]  __irq_exit_rcu+0xcc/0x108
[   82.349764]  irq_exit_rcu+0x14/0x28
[   82.350038]  el1_interrupt+0x34/0x50
[   82.350321]  el1h_64_irq_handler+0x14/0x20
[   82.350642]  el1h_64_irq+0x6c/0x70
[   82.350911]  default_idle_call+0x30/0xd0 (P)
[   82.351248]  do_idle+0x1d0/0x200
[   82.351506]  cpu_startup_entry+0x38/0x48
[   82.351818]  secondary_start_kernel+0x124/0x150
[   82.352176]  __secondary_switched+0xac/0xb0
[   82.352505] ---[ end trace 0000000000000000 ]---



[1] SFRAME-less livepatch RFC
https://lore.kernel.org/live-patching/20250129232936.1795412-1-song@kernel.org/
[2] special-static test from kpatch
https://github.com/dynup/kpatch/blob/master/test/integration/linux-6.2.0/special-static.patch
[3] Puranjay's kpatch with arm64 support
https://github.com/puranjaymohan/kpatch/tree/arm64
[4] My version of kpatch with arm64 and LTO support
https://github.com/liu-song-6/kpatch/tree/fb-6.13-v2

On Mon, Jan 27, 2025 at 1:33 PM Weinan Liu <wnliu@google.com> wrote:
>
> This patchset implements a generic kernel sframe-based [1] unwinder.
> The main goal is to support reliable stacktraces on arm64.
>
> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
> required support from objtool: it cannot generate orc unwind tables for
> arm64.
>
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> Since the sframe unwind table algorithm is similar, these two proposal
> could integrate common functionality in the future.
>
> There are some incomplete features or challenges:
>   - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>     FRE from kernel modules doesn't appear correct, preventing us from
>     unwinding functions from kernel modules.
>   - Currently, only GCC supports sframe.
>
> Ref:
> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>
> Madhavan T. Venkataraman (1):
>   arm64: Define TIF_PATCH_PENDING for livepatch
>
> Weinan Liu (7):
>   unwind: build kernel with sframe info
>   arm64: entry: add unwind info for various kernel entries
>   unwind: add sframe v2 header
>   unwind: Implement generic sframe unwinder library
>   unwind: arm64: Add sframe unwinder on arm64
>   unwind: arm64: add reliable stacktrace support for arm64
>   arm64: Enable livepatch for ARM64
>
>  Makefile                                   |   6 +
>  arch/Kconfig                               |   8 +
>  arch/arm64/Kconfig                         |   3 +
>  arch/arm64/Kconfig.debug                   |  10 +
>  arch/arm64/include/asm/stacktrace/common.h |   6 +
>  arch/arm64/include/asm/thread_info.h       |   4 +-
>  arch/arm64/kernel/entry-common.c           |   4 +
>  arch/arm64/kernel/entry.S                  |  10 +
>  arch/arm64/kernel/setup.c                  |   2 +
>  arch/arm64/kernel/stacktrace.c             | 102 ++++++++++
>  include/asm-generic/vmlinux.lds.h          |  12 ++
>  include/linux/sframe_lookup.h              |  43 +++++
>  kernel/Makefile                            |   1 +
>  kernel/sframe.h                            | 215 +++++++++++++++++++++
>  kernel/sframe_lookup.c                     | 196 +++++++++++++++++++
>  15 files changed, 621 insertions(+), 1 deletion(-)
>  create mode 100644 include/linux/sframe_lookup.h
>  create mode 100644 kernel/sframe.h
>  create mode 100644 kernel/sframe_lookup.c
>
> --
> 2.48.1.262.g85cc9f2d1e-goog
>
>

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Wed, Feb 12, 2025 at 03:32:40PM -0800, Song Liu wrote:
> [   81.250437] ------------[ cut here ]------------
> [   81.250818] refcount_t: saturated; leaking memory.
> [   81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22
> refcount_warn_saturate+0x6c/0x140
> [   81.251841] Modules linked in: livepatch_special_static(OEK)
> [   81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G
> OE K    6.13.2-00321-g52d2813b4b07 #49
> [   81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
> [   81.253503] Hardware name: linux,dummy-virt (DT)
> [   81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> [   81.254383] pc : refcount_warn_saturate+0x6c/0x140
> [   81.254748] lr : refcount_warn_saturate+0x6c/0x140
> [   81.255114] sp : ffff800085a6fc00
> [   81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180
> [   81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608
> [   81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140
> [   81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff
> [   81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006
> [   81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20
> [   81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4
> [   81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
> [   81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
> [   81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
> [   81.260824] Call trace:
> [   81.261015]  refcount_warn_saturate+0x6c/0x140 (P)
> [   81.261387]  __refcount_add.constprop.0+0x60/0x70
> [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]

Does that copy_process+0xfdc/0xfd58 resolve to this line in
copy_process()?

			refcount_inc(&current->signal->sigcnt);

Maybe the klp rela reference to 'current' is bogus, or resolving to the
wrong address somehow?

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Wed, Feb 12, 2025 at 3:49 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Wed, Feb 12, 2025 at 03:32:40PM -0800, Song Liu wrote:
> > [   81.250437] ------------[ cut here ]------------
> > [   81.250818] refcount_t: saturated; leaking memory.
> > [   81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22
> > refcount_warn_saturate+0x6c/0x140
> > [   81.251841] Modules linked in: livepatch_special_static(OEK)
> > [   81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G
> > OE K    6.13.2-00321-g52d2813b4b07 #49
> > [   81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
> > [   81.253503] Hardware name: linux,dummy-virt (DT)
> > [   81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> > [   81.254383] pc : refcount_warn_saturate+0x6c/0x140
> > [   81.254748] lr : refcount_warn_saturate+0x6c/0x140
> > [   81.255114] sp : ffff800085a6fc00
> > [   81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180
> > [   81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608
> > [   81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140
> > [   81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff
> > [   81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006
> > [   81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20
> > [   81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4
> > [   81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
> > [   81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
> > [   81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
> > [   81.260824] Call trace:
> > [   81.261015]  refcount_warn_saturate+0x6c/0x140 (P)
> > [   81.261387]  __refcount_add.constprop.0+0x60/0x70
> > [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
>
> Does that copy_process+0xfdc/0xfd58 resolve to this line in
> copy_process()?
>
>                         refcount_inc(&current->signal->sigcnt);
>
> Maybe the klp rela reference to 'current' is bogus, or resolving to the
> wrong address somehow?

It resolves the following line.

p->signal->tty = tty_kref_get(current->signal->tty);

I am not quite sure how 'current' should be resolved.

The size of copy_process (0xfd58) is wrong. It is only about
5.5kB in size. Also, the copy_process function in the .ko file
looks very broken. I will try a few more things.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
> > > [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
> >
> > Does that copy_process+0xfdc/0xfd58 resolve to this line in
> > copy_process()?
> >
> >                         refcount_inc(&current->signal->sigcnt);
> >
> > Maybe the klp rela reference to 'current' is bogus, or resolving to the
> > wrong address somehow?
> 
> It resolves the following line.
> 
> p->signal->tty = tty_kref_get(current->signal->tty);
> 
> I am not quite sure how 'current' should be resolved.

Hm, on arm64 it looks like the value of 'current' is stored in the
SP_EL0 register.  So I guess that shouldn't need any relocations.

> The size of copy_process (0xfd58) is wrong. It is only about
> 5.5kB in size. Also, the copy_process function in the .ko file
> looks very broken. I will try a few more things.

Ah ok, sounds like it's pretty borked.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

[-- Attachment #1: Type: text/plain, Size: 1574 bytes --]

On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
> > > > [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
> > >
> > > Does that copy_process+0xfdc/0xfd58 resolve to this line in
> > > copy_process()?
> > >
> > >                         refcount_inc(&current->signal->sigcnt);
> > >
> > > Maybe the klp rela reference to 'current' is bogus, or resolving to the
> > > wrong address somehow?
> >
> > It resolves the following line.
> >
> > p->signal->tty = tty_kref_get(current->signal->tty);
> >
> > I am not quite sure how 'current' should be resolved.
>
> Hm, on arm64 it looks like the value of 'current' is stored in the
> SP_EL0 register.  So I guess that shouldn't need any relocations.
>
> > The size of copy_process (0xfd58) is wrong. It is only about
> > 5.5kB in size. Also, the copy_process function in the .ko file
> > looks very broken. I will try a few more things.

When I try each step of kpatch-build, the copy_process function
looks reasonable (according to gdb-disassemble) in fork.o and
output.o. However, copy_process looks weird in livepatch-special-static.o,
which is generated by ld:

ld -EL  -maarch64linux -z norelro -z noexecstack
--no-warn-rwx-segments -T ././kpatch.lds  -r -o
livepatch-special-static.o ./patch-hook.o ./output.o

I have attached these files to the email. I am not sure whether
the email server will let them through.

Indu, does this look like an issue with ld?

Thanks,
Song

[-- Attachment #2: patch-hook.o --]
[-- Type: application/octet-stream, Size: 204128 bytes --]

[-- Attachment #3: output.o --]
[-- Type: application/octet-stream, Size: 1214808 bytes --]

[-- Attachment #4: livepatch-special-static.o --]
[-- Type: application/octet-stream, Size: 1417232 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 1866 bytes --]

Song Liu <song@kernel.org> writes:

> On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>>
>> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
>> > > > [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
>> > >
>> > > Does that copy_process+0xfdc/0xfd58 resolve to this line in
>> > > copy_process()?
>> > >
>> > >                         refcount_inc(&current->signal->sigcnt);
>> > >
>> > > Maybe the klp rela reference to 'current' is bogus, or resolving to the
>> > > wrong address somehow?
>> >
>> > It resolves the following line.
>> >
>> > p->signal->tty = tty_kref_get(current->signal->tty);
>> >
>> > I am not quite sure how 'current' should be resolved.
>>
>> Hm, on arm64 it looks like the value of 'current' is stored in the
>> SP_EL0 register.  So I guess that shouldn't need any relocations.
>>
>> > The size of copy_process (0xfd58) is wrong. It is only about
>> > 5.5kB in size. Also, the copy_process function in the .ko file
>> > looks very broken. I will try a few more things.
>
> When I try each step of kpatch-build, the copy_process function
> looks reasonable (according to gdb-disassemble) in fork.o and
> output.o. However, copy_process looks weird in livepatch-special-static.o,
> which is generated by ld:
>
> ld -EL  -maarch64linux -z norelro -z noexecstack
> --no-warn-rwx-segments -T ././kpatch.lds  -r -o
> livepatch-special-static.o ./patch-hook.o ./output.o
>
> I have attached these files to the email. I am not sure whether
> the email server will let them through.

I think, I am missing something here,

I did :

objdump -Dr livepatch-special-static.o | less

and

objdump -Dr output.o | less

and the disassembly of copy_process() looks exactly same.

> Indu, does this look like an issue with ld?
>
> Thanks,
> Song

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Wed, Feb 12, 2025 at 11:46 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>
> Song Liu <song@kernel.org> writes:
>
> > On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >>
> >> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
> >> > > > [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
> >> > >
> >> > > Does that copy_process+0xfdc/0xfd58 resolve to this line in
> >> > > copy_process()?
> >> > >
> >> > >                         refcount_inc(&current->signal->sigcnt);
> >> > >
> >> > > Maybe the klp rela reference to 'current' is bogus, or resolving to the
> >> > > wrong address somehow?
> >> >
> >> > It resolves the following line.
> >> >
> >> > p->signal->tty = tty_kref_get(current->signal->tty);
> >> >
> >> > I am not quite sure how 'current' should be resolved.
> >>
> >> Hm, on arm64 it looks like the value of 'current' is stored in the
> >> SP_EL0 register.  So I guess that shouldn't need any relocations.
> >>
> >> > The size of copy_process (0xfd58) is wrong. It is only about
> >> > 5.5kB in size. Also, the copy_process function in the .ko file
> >> > looks very broken. I will try a few more things.
> >
> > When I try each step of kpatch-build, the copy_process function
> > looks reasonable (according to gdb-disassemble) in fork.o and
> > output.o. However, copy_process looks weird in livepatch-special-static.o,
> > which is generated by ld:
> >
> > ld -EL  -maarch64linux -z norelro -z noexecstack
> > --no-warn-rwx-segments -T ././kpatch.lds  -r -o
> > livepatch-special-static.o ./patch-hook.o ./output.o
> >
> > I have attached these files to the email. I am not sure whether
> > the email server will let them through.
>
> I think, I am missing something here,
>
> I did :
>
> objdump -Dr livepatch-special-static.o | less
>
> and
>
> objdump -Dr output.o | less
>
> and the disassembly of copy_process() looks exactly same.

Yeah, objdump does show the same disassembly. However, if
I open the file with gdb, and do "disassemble copy_process",
the one in livepatch-special-static.o looks very weird.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Thu, Feb 13, 2025 at 11:40:43AM -0800, Song Liu wrote:
> Yeah, objdump does show the same disassembly. However, if
> I open the file with gdb, and do "disassemble copy_process",
> the one in livepatch-special-static.o looks very weird.

The symbol table looks ok.  I'm not sure why gdb is getting confused,
but that could possibly be a red herring.  Maybe it doesn't like the
-ffunction-sections for some reason.

It's really weird the function length reported by kallsyms is so wrong.
Can you share the .ko?

The refcount warning might indicate it's passing some bogus memory to
tty_kref_get().  Any chance you have struct randomization enabled?

Are you sure there's no code or .config mismatch between the built
kernel and the running kernel?

Ignorant arm64 question: is the module's text further away from slab
memory than vmlinux text, thus requiring a different instruction (or
GOT/TOC) to access memory further away in the address space?

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

[-- Attachment #1: Type: text/plain, Size: 1535 bytes --]

On Fri, Feb 14, 2025 at 12:08 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Thu, Feb 13, 2025 at 11:40:43AM -0800, Song Liu wrote:
> > Yeah, objdump does show the same disassembly. However, if
> > I open the file with gdb, and do "disassemble copy_process",
> > the one in livepatch-special-static.o looks very weird.
>
> The symbol table looks ok.  I'm not sure why gdb is getting confused,
> but that could possibly be a red herring.  Maybe it doesn't like the
> -ffunction-sections for some reason.
>
> It's really weird the function length reported by kallsyms is so wrong.
> Can you share the .ko?

Attached is the .ko file.

>
> The refcount warning might indicate it's passing some bogus memory to
> tty_kref_get().  Any chance you have struct randomization enabled?

No, struct randomization is disabled.

> Are you sure there's no code or .config mismatch between the built
> kernel and the running kernel?

The test is done in a vm, so the .config used to build the livepatch is
exactly the same as the testing kernel .

>
> Ignorant arm64 question: is the module's text further away from slab
> memory than vmlinux text, thus requiring a different instruction (or
> GOT/TOC) to access memory further away in the address space?

It appears to me the module text is very close to vmlinux text:

vmlinux: ffff8000800b4b68 T copy_process
module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]

I run tests with CONFIG_PAGE_SIZE_64KB=y.

Thanks,
Song

[-- Attachment #2: livepatch-special-static.ko --]
[-- Type: application/octet-stream, Size: 1550608 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 14, 2025 at 09:51:41AM -0800, Song Liu wrote:
> > Ignorant arm64 question: is the module's text further away from slab
> > memory than vmlinux text, thus requiring a different instruction (or
> > GOT/TOC) to access memory further away in the address space?
> 
> It appears to me the module text is very close to vmlinux text:
> 
> vmlinux: ffff8000800b4b68 T copy_process
> module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]

Hm... the only other thing I can think of is that the klp relas might be
wrong somewhere.  If you share patched.o and .ko files from the same
build I could take a look.

BTW, I realized the wrong function size shown in the WARNING stack trace
is probably just due to a kallsyms quirk.  It calculates a symbol's size
by subtracting its start address from the next symbol's start address.
It doesn't actually use the ELF symbol size.  So the next symbol after
copy_process() in the loaded module's address space might just be far
away.

That kallsyms issue has caused other headaches.  It really needs to be
fixed to use the actual ELF symbol size.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

Hi Josh,

On Fri, Feb 14, 2025 at 11:34 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Fri, Feb 14, 2025 at 09:51:41AM -0800, Song Liu wrote:
> > > Ignorant arm64 question: is the module's text further away from slab
> > > memory than vmlinux text, thus requiring a different instruction (or
> > > GOT/TOC) to access memory further away in the address space?
> >
> > It appears to me the module text is very close to vmlinux text:
> >
> > vmlinux: ffff8000800b4b68 T copy_process
> > module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]
>
> Hm... the only other thing I can think of is that the klp relas might be
> wrong somewhere.  If you share patched.o and .ko files from the same
> build I could take a look.

A tarball with these files is available here:

https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link

> BTW, I realized the wrong function size shown in the WARNING stack trace
> is probably just due to a kallsyms quirk.  It calculates a symbol's size
> by subtracting its start address from the next symbol's start address.
> It doesn't actually use the ELF symbol size.  So the next symbol after
> copy_process() in the loaded module's address space might just be far
> away.

Yeah, I also think kallsyms logic was the issue here. So it is not the same
as the gdb-disassembly issue.

> That kallsyms issue has caused other headaches.  It really needs to be
> fixed to use the actual ELF symbol size.

Maybe we should have a "module_text_end" symbol?

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 14, 2025 at 02:04:17PM -0800, Song Liu wrote:
> Hi Josh,
> 
> On Fri, Feb 14, 2025 at 11:34 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > On Fri, Feb 14, 2025 at 09:51:41AM -0800, Song Liu wrote:
> > > > Ignorant arm64 question: is the module's text further away from slab
> > > > memory than vmlinux text, thus requiring a different instruction (or
> > > > GOT/TOC) to access memory further away in the address space?
> > >
> > > It appears to me the module text is very close to vmlinux text:
> > >
> > > vmlinux: ffff8000800b4b68 T copy_process
> > > module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]
> >
> > Hm... the only other thing I can think of is that the klp relas might be
> > wrong somewhere.  If you share patched.o and .ko files from the same
> > build I could take a look.
> 
> A tarball with these files is available here:
> 
> https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link

Thanks, I'll take a look.

> > That kallsyms issue has caused other headaches.  It really needs to be
> > fixed to use the actual ELF symbol size.
> 
> Maybe we should have a "module_text_end" symbol?

Maybe, though it would be a lot cleaner for kallsyms to just use the
actual ELF sizes.  And actually I'm thinking that would be a pretty
trivial change.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 14, 2025 at 02:04:17PM -0800, Song Liu wrote:
> Hi Josh,
> 
> On Fri, Feb 14, 2025 at 11:34 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > On Fri, Feb 14, 2025 at 09:51:41AM -0800, Song Liu wrote:
> > > > Ignorant arm64 question: is the module's text further away from slab
> > > > memory than vmlinux text, thus requiring a different instruction (or
> > > > GOT/TOC) to access memory further away in the address space?
> > >
> > > It appears to me the module text is very close to vmlinux text:
> > >
> > > vmlinux: ffff8000800b4b68 T copy_process
> > > module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]
> >
> > Hm... the only other thing I can think of is that the klp relas might be
> > wrong somewhere.  If you share patched.o and .ko files from the same
> > build I could take a look.
> 
> A tarball with these files is available here:
> 
> https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link

Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
is looking at all the relocations in order to set up the PLT.  That also
needs to be done for klp relas, or are your patches already doing that?

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Fri, Feb 14, 2025 at 3:23 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Fri, Feb 14, 2025 at 02:04:17PM -0800, Song Liu wrote:
> > Hi Josh,
> >
> > On Fri, Feb 14, 2025 at 11:34 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > >
> > > On Fri, Feb 14, 2025 at 09:51:41AM -0800, Song Liu wrote:
> > > > > Ignorant arm64 question: is the module's text further away from slab
> > > > > memory than vmlinux text, thus requiring a different instruction (or
> > > > > GOT/TOC) to access memory further away in the address space?
> > > >
> > > > It appears to me the module text is very close to vmlinux text:
> > > >
> > > > vmlinux: ffff8000800b4b68 T copy_process
> > > > module: ffff80007b0f06d0 t copy_process [livepatch_always_inline_special_static]
> > >
> > > Hm... the only other thing I can think of is that the klp relas might be
> > > wrong somewhere.  If you share patched.o and .ko files from the same
> > > build I could take a look.
> >
> > A tarball with these files is available here:
> >
> > https://drive.google.com/file/d/1ONB1tC9oK-Z5ShmSXneqWLTjJgC5Xq-C/view?usp=drive_link
>
> Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
> is looking at all the relocations in order to set up the PLT.  That also
> needs to be done for klp relas, or are your patches already doing that?

I don't think either version (this set and my RFC) added logic for PLT.
There is some rela logic on the kpatch-build side. But I am not sure
whether it is sufficient.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Mon, Feb 17, 2025 at 08:38:22PM -0800, Song Liu wrote:
> On Fri, Feb 14, 2025 at 3:23 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
> > is looking at all the relocations in order to set up the PLT.  That also
> > needs to be done for klp relas, or are your patches already doing that?
> 
> I don't think either version (this set and my RFC) added logic for PLT.
> There is some rela logic on the kpatch-build side. But I am not sure
> whether it is sufficient.

The klp relas looked ok.  I didn't see any signs of kpatch-build doing
anything wrong.  AFAICT the problem is that module-plts.c creates PLT
entries for regular relas but not klp relas.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

Hi Josh,

On Mon, Feb 17, 2025 at 10:37 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Mon, Feb 17, 2025 at 08:38:22PM -0800, Song Liu wrote:
> > On Fri, Feb 14, 2025 at 3:23 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > > Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
> > > is looking at all the relocations in order to set up the PLT.  That also
> > > needs to be done for klp relas, or are your patches already doing that?
> >
> > I don't think either version (this set and my RFC) added logic for PLT.
> > There is some rela logic on the kpatch-build side. But I am not sure
> > whether it is sufficient.
>
> The klp relas looked ok.  I didn't see any signs of kpatch-build doing
> anything wrong.  AFAICT the problem is that module-plts.c creates PLT
> entries for regular relas but not klp relas.

In my tests (with printk) module-plts.c processes the .
klp.rela.vmlinux..text.copy_process section just like any other .rela.*
sections. Do we need special handling of the klp.rela.* sections?

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Tue, Feb 18, 2025 at 10:20:10AM -0800, Song Liu wrote:
> Hi Josh,
> 
> On Mon, Feb 17, 2025 at 10:37 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > On Mon, Feb 17, 2025 at 08:38:22PM -0800, Song Liu wrote:
> > > On Fri, Feb 14, 2025 at 3:23 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > > > Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
> > > > is looking at all the relocations in order to set up the PLT.  That also
> > > > needs to be done for klp relas, or are your patches already doing that?
> > >
> > > I don't think either version (this set and my RFC) added logic for PLT.
> > > There is some rela logic on the kpatch-build side. But I am not sure
> > > whether it is sufficient.
> >
> > The klp relas looked ok.  I didn't see any signs of kpatch-build doing
> > anything wrong.  AFAICT the problem is that module-plts.c creates PLT
> > entries for regular relas but not klp relas.
> 
> In my tests (with printk) module-plts.c processes the .
> klp.rela.vmlinux..text.copy_process section just like any other .rela.*
> sections. Do we need special handling of the klp.rela.* sections?

Ok, I see how it works now:

klp_write_section_relocs()
	apply_relocate_add()
		module_emit_plt_entry()
  
If that code is working correctly then I'm fresh out of ideas...

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Tue, Feb 18, 2025 at 10:41 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Tue, Feb 18, 2025 at 10:20:10AM -0800, Song Liu wrote:
> > Hi Josh,
> >
> > On Mon, Feb 17, 2025 at 10:37 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > >
> > > On Mon, Feb 17, 2025 at 08:38:22PM -0800, Song Liu wrote:
> > > > On Fri, Feb 14, 2025 at 3:23 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > > > > Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
> > > > > is looking at all the relocations in order to set up the PLT.  That also
> > > > > needs to be done for klp relas, or are your patches already doing that?
> > > >
> > > > I don't think either version (this set and my RFC) added logic for PLT.
> > > > There is some rela logic on the kpatch-build side. But I am not sure
> > > > whether it is sufficient.
> > >
> > > The klp relas looked ok.  I didn't see any signs of kpatch-build doing
> > > anything wrong.  AFAICT the problem is that module-plts.c creates PLT
> > > entries for regular relas but not klp relas.
> >
> > In my tests (with printk) module-plts.c processes the .
> > klp.rela.vmlinux..text.copy_process section just like any other .rela.*
> > sections. Do we need special handling of the klp.rela.* sections?
>
> Ok, I see how it works now:
>
> klp_write_section_relocs()
>         apply_relocate_add()
>                 module_emit_plt_entry()
>
> If that code is working correctly then I'm fresh out of ideas...

I tried to dump assembly of copy_process, but couldn't find any
clue. I am wondering whether this is an issue with gcc-14.2.1.

Puranjay, could you please try with a different gcc, like some
version of gcc-14?

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Wed, Feb 19, 2025 at 9:44 AM Song Liu <song@kernel.org> wrote:
>
> On Tue, Feb 18, 2025 at 10:41 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >
> > On Tue, Feb 18, 2025 at 10:20:10AM -0800, Song Liu wrote:
> > > Hi Josh,
> > >
> > > On Mon, Feb 17, 2025 at 10:37 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > > >
> > > > On Mon, Feb 17, 2025 at 08:38:22PM -0800, Song Liu wrote:
> > > > > On Fri, Feb 14, 2025 at 3:23 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> > > > > > Poking around the arm64 module code, arch/arm64/kernel/module-plts.c
> > > > > > is looking at all the relocations in order to set up the PLT.  That also
> > > > > > needs to be done for klp relas, or are your patches already doing that?
> > > > >
> > > > > I don't think either version (this set and my RFC) added logic for PLT.
> > > > > There is some rela logic on the kpatch-build side. But I am not sure
> > > > > whether it is sufficient.
> > > >
> > > > The klp relas looked ok.  I didn't see any signs of kpatch-build doing
> > > > anything wrong.  AFAICT the problem is that module-plts.c creates PLT
> > > > entries for regular relas but not klp relas.
> > >
> > > In my tests (with printk) module-plts.c processes the .
> > > klp.rela.vmlinux..text.copy_process section just like any other .rela.*
> > > sections. Do we need special handling of the klp.rela.* sections?
> >
> > Ok, I see how it works now:
> >
> > klp_write_section_relocs()
> >         apply_relocate_add()
> >                 module_emit_plt_entry()
> >
> > If that code is working correctly then I'm fresh out of ideas...
>
> I tried to dump assembly of copy_process, but couldn't find any
> clue. I am wondering whether this is an issue with gcc-14.2.1.
>
> Puranjay, could you please try with a different gcc, like some
> version of gcc-14?

I figured out why this is broken with gcc-14.

The stack trace points to tty_kref_get(), which is eventually
a __refcount_add(). With gcc, there are multiple of constprop
versions of the function.

$ readelf -s -W vmlinux | grep __refcount_add
 19747: ffff800080872eb0   104 FUNC    LOCAL  DEFAULT    2
__refcount_add.constprop.0
 20153: ffff80008001fcd8    88 FUNC    LOCAL  DEFAULT    2
__refcount_add.constprop.0
 36428: ffff800080e62ac0   100 FUNC    LOCAL  DEFAULT    2
__refcount_add.constprop.0
 45055: ffff8000800bf130   100 FUNC    LOCAL  DEFAULT    2
__refcount_add.constprop.0
 54823: ffff8000801c2a40   100 FUNC    LOCAL  DEFAULT    2
__refcount_add.constprop.0
180848: ffff8000810a8cb0   100 FUNC    LOCAL  DEFAULT    2
__refcount_add.constprop.0

The problem is, with gcc-14, these symbols are NOT sorted by
their addresses. OTOH, the order in kallsyms is like:

[root@(none) /test-klp]# grep __refcount_add /proc/kallsyms
ffff80008001fcd8 t __refcount_add.constprop.0
ffff8000800bf130 t __refcount_add.constprop.0
ffff8000801c2a40 t __refcount_add.constprop.0
ffff800080872eb0 t __refcount_add.constprop.0
ffff800080e62ac0 t __refcount_add.constprop.0
ffff8000810a8cb0 t __refcount_add.constprop.0

kpatch-build uses readelf to get the symtab, and then calculate
proper sympos based on this symtab. However, this sympos
is different when the kernel applies this KLP. In this case,
copy_process calls ffff8000800bf130. kpatch-build gives
sympos of 4, based on the output of readelf, but ffff8000800bf130
is actually at position 2 in kallsyms.

gcc-11, OTOH, doesn't have this problem with readelf.

Indu, is this behavior (symbols with same name are not in
sorted order from readelf -s) expected? Or is this a bug?
I am using this gcc:

$ gcc --version
gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1)
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Wed, Feb 19, 2025 at 08:50:09PM -0800, Song Liu wrote:
> Indu, is this behavior (symbols with same name are not in
> sorted order from readelf -s) expected? Or is this a bug?
> I am using this gcc:
> 
> $ gcc --version
> gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1)
> Copyright (C) 2024 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Are you using different binutils versions as well?

It sounds like a linker "issue" to me.  I'm not sure if it qualifies as
a bug, the linker might be free to layout symbols how it wishes.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Thu, Feb 20, 2025 at 10:33 AM Song Liu <song@kernel.org> wrote:
>
>
> On Thu, Feb 20, 2025 at 10:22 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>>
>> On Wed, Feb 19, 2025 at 08:50:09PM -0800, Song Liu wrote:
>> > Indu, is this behavior (symbols with same name are not in
>> > sorted order from readelf -s) expected? Or is this a bug?
>> > I am using this gcc:
>> >
>> > $ gcc --version
>> > gcc (GCC) 14.2.1 20240801 (Red Hat 14.2.1-1)
>> > Copyright (C) 2024 Free Software Foundation, Inc.
>> > This is free software; see the source for copying conditions.  There is NO
>> > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> Are you using different binutils versions as well?
>
>
> I am using binutil 2.41, for both gcc -11 and gcc-14.
>
>>
>> It sounds like a linker "issue" to me.  I'm not sure if it qualifies as
>> a bug, the linker might be free to layout symbols how it wishes.
>
>
> We can probably handle that in kpatch-build.

OK, something like the following gets the proper sympos for
gcc-14 built kernel.

Thanks,
Song


diff --git i/kpatch-build/lookup.c w/kpatch-build/lookup.c
index bd2b732de910..87ac127184c5 100644
--- i/kpatch-build/lookup.c
+++ w/kpatch-build/lookup.c
@@ -479,13 +479,10 @@ static bool lookup_local_symbol(struct
lookup_table *table,
        struct object_symbol *sym;
        unsigned long sympos = 0;
        int i, in_file = 0;
+       bool found = false;

        memset(result, 0, sizeof(*result));
        for_each_obj_symbol(i, sym, table) {
-               if (sym->bind == STB_LOCAL && !strcmp(sym->name,
-                                       lookup_sym->name))
-                       sympos++;
-
                if (lookup_sym->lookup_table_file_sym == sym) {
                        in_file = 1;
                        continue;
@@ -499,20 +496,29 @@ static bool lookup_local_symbol(struct
lookup_table *table,

                if (sym->bind == STB_LOCAL && !strcmp(sym->name,
                                        lookup_sym->name)) {
-                       if (result->objname)
+                       if (found)
                                ERROR("duplicate local symbol found for %s",
                                                lookup_sym->name);

                        result->objname         = table->objname;
                        result->addr            = sym->addr;
                        result->size            = sym->size;
-                       result->sympos          = sympos;
                        result->global          = false;
                        result->exported        = false;
+                       found = true;
                }
        }
+       if (!found)
+               return false;

-       return !!result->objname;
+       for_each_obj_symbol(i, sym, table) {
+               if (sym->bind == STB_LOCAL &&
+                   !strcmp(sym->name, lookup_sym->name) &&
+                   sym->addr < result->addr)
+                       sympos++;
+       }
+       result->sympos = sympos;
+       return true;
 }

 static bool lookup_exported_symbol(struct lookup_table *table, char *name,

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/12/25 11:25 PM, Song Liu wrote:
> On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>>
>> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
>>>>> [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
>>>>
>>>> Does that copy_process+0xfdc/0xfd58 resolve to this line in
>>>> copy_process()?
>>>>
>>>>                          refcount_inc(&current->signal->sigcnt);
>>>>
>>>> Maybe the klp rela reference to 'current' is bogus, or resolving to the
>>>> wrong address somehow?
>>>
>>> It resolves the following line.
>>>
>>> p->signal->tty = tty_kref_get(current->signal->tty);
>>>
>>> I am not quite sure how 'current' should be resolved.
>>
>> Hm, on arm64 it looks like the value of 'current' is stored in the
>> SP_EL0 register.  So I guess that shouldn't need any relocations.
>>
>>> The size of copy_process (0xfd58) is wrong. It is only about
>>> 5.5kB in size. Also, the copy_process function in the .ko file
>>> looks very broken. I will try a few more things.
> 
> When I try each step of kpatch-build, the copy_process function
> looks reasonable (according to gdb-disassemble) in fork.o and
> output.o. However, copy_process looks weird in livepatch-special-static.o,
> which is generated by ld:
> 
> ld -EL  -maarch64linux -z norelro -z noexecstack
> --no-warn-rwx-segments -T ././kpatch.lds  -r -o
> livepatch-special-static.o ./patch-hook.o ./output.o
> 
> I have attached these files to the email. I am not sure whether
> the email server will let them through.
> 
> Indu, does this look like an issue with ld?
> 

Sorry for the delay.

Looks like there has been progress since and issue may be elsewhere, but:

FWIW, I looked at the .sframe and .rela.sframe sections here, the data 
does look OK.  I noted that there is no .sframe for copy_process () in 
output.o... I will take a look into it.

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Thu, Feb 13, 2025 at 3:23 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
> On 2/12/25 11:25 PM, Song Liu wrote:
> > On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> >>
> >> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
> >>>>> [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
> >>>>
> >>>> Does that copy_process+0xfdc/0xfd58 resolve to this line in
> >>>> copy_process()?
> >>>>
> >>>>                          refcount_inc(&current->signal->sigcnt);
> >>>>
> >>>> Maybe the klp rela reference to 'current' is bogus, or resolving to the
> >>>> wrong address somehow?
> >>>
> >>> It resolves the following line.
> >>>
> >>> p->signal->tty = tty_kref_get(current->signal->tty);
> >>>
> >>> I am not quite sure how 'current' should be resolved.
> >>
> >> Hm, on arm64 it looks like the value of 'current' is stored in the
> >> SP_EL0 register.  So I guess that shouldn't need any relocations.
> >>
> >>> The size of copy_process (0xfd58) is wrong. It is only about
> >>> 5.5kB in size. Also, the copy_process function in the .ko file
> >>> looks very broken. I will try a few more things.
> >
> > When I try each step of kpatch-build, the copy_process function
> > looks reasonable (according to gdb-disassemble) in fork.o and
> > output.o. However, copy_process looks weird in livepatch-special-static.o,
> > which is generated by ld:
> >
> > ld -EL  -maarch64linux -z norelro -z noexecstack
> > --no-warn-rwx-segments -T ././kpatch.lds  -r -o
> > livepatch-special-static.o ./patch-hook.o ./output.o
> >
> > I have attached these files to the email. I am not sure whether
> > the email server will let them through.
> >
> > Indu, does this look like an issue with ld?
> >
>
> Sorry for the delay.
>
> Looks like there has been progress since and issue may be elsewhere, but:
>
> FWIW, I looked at the .sframe and .rela.sframe sections here, the data
> does look OK.  I noted that there is no .sframe for copy_process () in
> output.o... I will take a look into it.

That output.o was generated with a version of kpatch-build that blindly
removes all .sframe sections.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 3597 bytes --]

Indu Bhagat <indu.bhagat@oracle.com> writes:

> On 2/12/25 11:25 PM, Song Liu wrote:
>> On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>>>
>>> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
>>>>>> [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
>>>>>
>>>>> Does that copy_process+0xfdc/0xfd58 resolve to this line in
>>>>> copy_process()?
>>>>>
>>>>>                          refcount_inc(&current->signal->sigcnt);
>>>>>
>>>>> Maybe the klp rela reference to 'current' is bogus, or resolving to the
>>>>> wrong address somehow?
>>>>
>>>> It resolves the following line.
>>>>
>>>> p->signal->tty = tty_kref_get(current->signal->tty);
>>>>
>>>> I am not quite sure how 'current' should be resolved.
>>>
>>> Hm, on arm64 it looks like the value of 'current' is stored in the
>>> SP_EL0 register.  So I guess that shouldn't need any relocations.
>>>
>>>> The size of copy_process (0xfd58) is wrong. It is only about
>>>> 5.5kB in size. Also, the copy_process function in the .ko file
>>>> looks very broken. I will try a few more things.
>> 
>> When I try each step of kpatch-build, the copy_process function
>> looks reasonable (according to gdb-disassemble) in fork.o and
>> output.o. However, copy_process looks weird in livepatch-special-static.o,
>> which is generated by ld:
>> 
>> ld -EL  -maarch64linux -z norelro -z noexecstack
>> --no-warn-rwx-segments -T ././kpatch.lds  -r -o
>> livepatch-special-static.o ./patch-hook.o ./output.o
>> 
>> I have attached these files to the email. I am not sure whether
>> the email server will let them through.
>> 
>> Indu, does this look like an issue with ld?
>> 
>
> Sorry for the delay.
>
> Looks like there has been progress since and issue may be elsewhere, but:
>
> FWIW, I looked at the .sframe and .rela.sframe sections here, the data 
> does look OK.  I noted that there is no .sframe for copy_process () in 
> output.o... I will take a look into it.

Hi Indu,

I saw another issue in my kernel build with sframes enabled (-Wa,--gsframe):

ld: warning: orphan section `.init.sframe' from `arch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe'
[... Many more similar warnings (.init.sframe) ...]

So, this orphan sections is generated in the build process.

I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1

Is this section needed for sframes to work? or can we discard
.init.sframe section with a patch like following to the linker script:

-- 8< --

diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 6a437bd08..8e704c0a6 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -1044,9 +1044,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
 # define SANITIZER_DISCARDS
 #endif

+#if defined(CONFIG_SFRAME_UNWIND_TABLE)
+#define DISCARD_INIT_SFRAME *(.init.sframe)
+#else
+#define DISCARD_INIT_SFRAME
+#endif
+
 #define COMMON_DISCARDS                                                        \
        SANITIZER_DISCARDS                                              \
        PATCHABLE_DISCARDS                                              \
+       DISCARD_INIT_SFRAME                                             \
        *(.discard)                                                     \
        *(.discard.*)                                                   \
        *(.export_symbol)                                               \

-- >8 --

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/13/25 11:57 PM, Puranjay Mohan wrote:
> Indu Bhagat <indu.bhagat@oracle.com> writes:
> 
>> On 2/12/25 11:25 PM, Song Liu wrote:
>>> On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>>>>
>>>> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
>>>>>>> [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
>>>>>>
>>>>>> Does that copy_process+0xfdc/0xfd58 resolve to this line in
>>>>>> copy_process()?
>>>>>>
>>>>>>                           refcount_inc(&current->signal->sigcnt);
>>>>>>
>>>>>> Maybe the klp rela reference to 'current' is bogus, or resolving to the
>>>>>> wrong address somehow?
>>>>>
>>>>> It resolves the following line.
>>>>>
>>>>> p->signal->tty = tty_kref_get(current->signal->tty);
>>>>>
>>>>> I am not quite sure how 'current' should be resolved.
>>>>
>>>> Hm, on arm64 it looks like the value of 'current' is stored in the
>>>> SP_EL0 register.  So I guess that shouldn't need any relocations.
>>>>
>>>>> The size of copy_process (0xfd58) is wrong. It is only about
>>>>> 5.5kB in size. Also, the copy_process function in the .ko file
>>>>> looks very broken. I will try a few more things.
>>>
>>> When I try each step of kpatch-build, the copy_process function
>>> looks reasonable (according to gdb-disassemble) in fork.o and
>>> output.o. However, copy_process looks weird in livepatch-special-static.o,
>>> which is generated by ld:
>>>
>>> ld -EL  -maarch64linux -z norelro -z noexecstack
>>> --no-warn-rwx-segments -T ././kpatch.lds  -r -o
>>> livepatch-special-static.o ./patch-hook.o ./output.o
>>>
>>> I have attached these files to the email. I am not sure whether
>>> the email server will let them through.
>>>
>>> Indu, does this look like an issue with ld?
>>>
>>
>> Sorry for the delay.
>>
>> Looks like there has been progress since and issue may be elsewhere, but:
>>
>> FWIW, I looked at the .sframe and .rela.sframe sections here, the data
>> does look OK.  I noted that there is no .sframe for copy_process () in
>> output.o... I will take a look into it.
> 
> Hi Indu,
> 
> I saw another issue in my kernel build with sframes enabled (-Wa,--gsframe):
> 
> ld: warning: orphan section `.init.sframe' from `arch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe'
> [... Many more similar warnings (.init.sframe) ...]
> 
> So, this orphan sections is generated in the build process.
> 
> I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
> 
> Is this section needed for sframes to work? or can we discard

No this should not be discarded.  This looks like a wrongly-named but 
valid SFrame section.

Once correctly named as .sframe, the linker should do the right thing. 
Let me check whats going on..

> .init.sframe section with a patch like following to the linker script:
> 
> -- 8< --
> 
> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
> index 6a437bd08..8e704c0a6 100644
> --- a/include/asm-generic/vmlinux.lds.h
> +++ b/include/asm-generic/vmlinux.lds.h
> @@ -1044,9 +1044,16 @@ defined(CONFIG_AUTOFDO_CLANG) || defined(CONFIG_PROPELLER_CLANG)
>   # define SANITIZER_DISCARDS
>   #endif
> 
> +#if defined(CONFIG_SFRAME_UNWIND_TABLE)
> +#define DISCARD_INIT_SFRAME *(.init.sframe)
> +#else
> +#define DISCARD_INIT_SFRAME
> +#endif
> +
>   #define COMMON_DISCARDS                                                        \
>          SANITIZER_DISCARDS                                              \
>          PATCHABLE_DISCARDS                                              \
> +       DISCARD_INIT_SFRAME                                             \
>          *(.discard)                                                     \
>          *(.discard.*)                                                   \
>          *(.export_symbol)                                               \
> 
> -- >8 --
> 
> Thanks,
> Puranjay

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/14/25 9:39 AM, Indu Bhagat wrote:
> On 2/13/25 11:57 PM, Puranjay Mohan wrote:
>> Indu Bhagat <indu.bhagat@oracle.com> writes:
>>
>>> On 2/12/25 11:25 PM, Song Liu wrote:
>>>> On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> 
>>>> wrote:
>>>>>
>>>>> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
>>>>>>>> [   81.261748]  copy_process+0xfdc/0xfd58 
>>>>>>>> [livepatch_special_static]
>>>>>>>
>>>>>>> Does that copy_process+0xfdc/0xfd58 resolve to this line in
>>>>>>> copy_process()?
>>>>>>>
>>>>>>>                           refcount_inc(&current->signal->sigcnt);
>>>>>>>
>>>>>>> Maybe the klp rela reference to 'current' is bogus, or resolving 
>>>>>>> to the
>>>>>>> wrong address somehow?
>>>>>>
>>>>>> It resolves the following line.
>>>>>>
>>>>>> p->signal->tty = tty_kref_get(current->signal->tty);
>>>>>>
>>>>>> I am not quite sure how 'current' should be resolved.
>>>>>
>>>>> Hm, on arm64 it looks like the value of 'current' is stored in the
>>>>> SP_EL0 register.  So I guess that shouldn't need any relocations.
>>>>>
>>>>>> The size of copy_process (0xfd58) is wrong. It is only about
>>>>>> 5.5kB in size. Also, the copy_process function in the .ko file
>>>>>> looks very broken. I will try a few more things.
>>>>
>>>> When I try each step of kpatch-build, the copy_process function
>>>> looks reasonable (according to gdb-disassemble) in fork.o and
>>>> output.o. However, copy_process looks weird in livepatch-special- 
>>>> static.o,
>>>> which is generated by ld:
>>>>
>>>> ld -EL  -maarch64linux -z norelro -z noexecstack
>>>> --no-warn-rwx-segments -T ././kpatch.lds  -r -o
>>>> livepatch-special-static.o ./patch-hook.o ./output.o
>>>>
>>>> I have attached these files to the email. I am not sure whether
>>>> the email server will let them through.
>>>>
>>>> Indu, does this look like an issue with ld?
>>>>
>>>
>>> Sorry for the delay.
>>>
>>> Looks like there has been progress since and issue may be elsewhere, 
>>> but:
>>>
>>> FWIW, I looked at the .sframe and .rela.sframe sections here, the data
>>> does look OK.  I noted that there is no .sframe for copy_process () in
>>> output.o... I will take a look into it.
>>
>> Hi Indu,
>>
>> I saw another issue in my kernel build with sframes enabled (-Wa,-- 
>> gsframe):
>>
>> ld: warning: orphan section `.init.sframe' from `arch/arm64/kernel/pi/ 
>> lib-fdt.pi.o' being placed in section `.init.sframe'
>> [... Many more similar warnings (.init.sframe) ...]
>>
>> So, this orphan sections is generated in the build process.
>>
>> I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
>>
>> Is this section needed for sframes to work? or can we discard
> 
> No this should not be discarded.  This looks like a wrongly-named but 
> valid SFrame section.
> 

Not wrongly named. --prefix-alloc-sections=.init is expected to do that 
as .sframe is an allocated section.

> Once correctly named as .sframe, the linker should do the right thing. 
> Let me check whats going on..
> 
>> .init.sframe section with a patch like following to the linker script:
>>
>> -- 8< --
>>
>> diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/ 
>> vmlinux.lds.h
>> index 6a437bd08..8e704c0a6 100644
>> --- a/include/asm-generic/vmlinux.lds.h
>> +++ b/include/asm-generic/vmlinux.lds.h
>> @@ -1044,9 +1044,16 @@ defined(CONFIG_AUTOFDO_CLANG) || 
>> defined(CONFIG_PROPELLER_CLANG)
>>   # define SANITIZER_DISCARDS
>>   #endif
>>
>> +#if defined(CONFIG_SFRAME_UNWIND_TABLE)
>> +#define DISCARD_INIT_SFRAME *(.init.sframe)
>> +#else
>> +#define DISCARD_INIT_SFRAME
>> +#endif
>> +
>>   #define 
>> COMMON_DISCARDS                                                        \
>>          
>> SANITIZER_DISCARDS                                              \
>>          
>> PATCHABLE_DISCARDS                                              \
>> +       DISCARD_INIT_SFRAME                                             \
>>          
>> *(.discard)                                                     \
>>          
>> *(.discard.*)                                                   \
>>          
>> *(.export_symbol)                                               \
>>
>> -- >8 --
>>
>> Thanks,
>> Puranjay
> 

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 4716 bytes --]

Indu Bhagat <indu.bhagat@oracle.com> writes:

> On 2/14/25 9:39 AM, Indu Bhagat wrote:
>> On 2/13/25 11:57 PM, Puranjay Mohan wrote:
>>> Indu Bhagat <indu.bhagat@oracle.com> writes:
>>>
>>>> On 2/12/25 11:25 PM, Song Liu wrote:
>>>>> On Wed, Feb 12, 2025 at 6:45 PM Josh Poimboeuf <jpoimboe@kernel.org> 
>>>>> wrote:
>>>>>>
>>>>>> On Wed, Feb 12, 2025 at 06:36:04PM -0800, Song Liu wrote:
>>>>>>>>> [   81.261748]  copy_process+0xfdc/0xfd58 
>>>>>>>>> [livepatch_special_static]
>>>>>>>>
>>>>>>>> Does that copy_process+0xfdc/0xfd58 resolve to this line in
>>>>>>>> copy_process()?
>>>>>>>>
>>>>>>>>                           refcount_inc(&current->signal->sigcnt);
>>>>>>>>
>>>>>>>> Maybe the klp rela reference to 'current' is bogus, or resolving 
>>>>>>>> to the
>>>>>>>> wrong address somehow?
>>>>>>>
>>>>>>> It resolves the following line.
>>>>>>>
>>>>>>> p->signal->tty = tty_kref_get(current->signal->tty);
>>>>>>>
>>>>>>> I am not quite sure how 'current' should be resolved.
>>>>>>
>>>>>> Hm, on arm64 it looks like the value of 'current' is stored in the
>>>>>> SP_EL0 register.  So I guess that shouldn't need any relocations.
>>>>>>
>>>>>>> The size of copy_process (0xfd58) is wrong. It is only about
>>>>>>> 5.5kB in size. Also, the copy_process function in the .ko file
>>>>>>> looks very broken. I will try a few more things.
>>>>>
>>>>> When I try each step of kpatch-build, the copy_process function
>>>>> looks reasonable (according to gdb-disassemble) in fork.o and
>>>>> output.o. However, copy_process looks weird in livepatch-special- 
>>>>> static.o,
>>>>> which is generated by ld:
>>>>>
>>>>> ld -EL  -maarch64linux -z norelro -z noexecstack
>>>>> --no-warn-rwx-segments -T ././kpatch.lds  -r -o
>>>>> livepatch-special-static.o ./patch-hook.o ./output.o
>>>>>
>>>>> I have attached these files to the email. I am not sure whether
>>>>> the email server will let them through.
>>>>>
>>>>> Indu, does this look like an issue with ld?
>>>>>
>>>>
>>>> Sorry for the delay.
>>>>
>>>> Looks like there has been progress since and issue may be elsewhere, 
>>>> but:
>>>>
>>>> FWIW, I looked at the .sframe and .rela.sframe sections here, the data
>>>> does look OK.  I noted that there is no .sframe for copy_process () in
>>>> output.o... I will take a look into it.
>>>
>>> Hi Indu,
>>>
>>> I saw another issue in my kernel build with sframes enabled (-Wa,-- 
>>> gsframe):
>>>
>>> ld: warning: orphan section `.init.sframe' from `arch/arm64/kernel/pi/ 
>>> lib-fdt.pi.o' being placed in section `.init.sframe'
>>> [... Many more similar warnings (.init.sframe) ...]
>>>
>>> So, this orphan sections is generated in the build process.
>>>
>>> I am using GNU ld version 2.41-50 and gcc (GCC) 11.4.1
>>>
>>> Is this section needed for sframes to work? or can we discard
>> 
>> No this should not be discarded.  This looks like a wrongly-named but 
>> valid SFrame section.
>> 
>
> Not wrongly named. --prefix-alloc-sections=.init is expected to do that 
> as .sframe is an allocated section.
>

So, these .init.sframe sections are then moved into .sframe by the
linker? (see linker script line below)

Here are some outputs from the build, do they look correct?

One of the objects that were emitting the warning
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf -SW arch/arm64/kernel/pi/lib-fdt.pi.o | grep sframe
  [47] .init.sframe      PROGBITS        0000000000000000 003c90 000226 00   A  0   0  8
  [48] .rela.init.sframe RELA            0000000000000000 008f08 000180 18   I 51  47  8

Final vmlinux ELF
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf -SW vmlinux | grep sframe
  [ 5] .init.sframe      PROGBITS        ffff80008118c298 119c298 002a88 00   A  0   0  8
  [ 6] .sframe           PROGBITS        ffff80008118ed20 119ed20 247c45 00   A  0   0  8

[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf --sframe=.sframe vmlinux | head
Contents of the SFrame section .sframe:
  Header :

    Version: SFRAME_VERSION_2
    Flags: SFRAME_F_FDE_SORTED
    Num FDEs: 51842
    Num FREs: 321245

  Function Index :

Does this also look correct?
[ec2-user@ip-172-31-32-86 linux-upstream]$ readelf --sframe=.init.sframe vmlinux | head
Contents of the SFrame section .init.sframe:
  Header :

    Version: SFRAME_VERSION_2
    Flags: NONE
    Num FDEs: 16
    Num FREs: 50

  Function Index :


and the linker script has this line:

.sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe)) __stop_sframe_header = .; }

So, do can you suggest the best way to fix these warnings?

Thanks,
Puranjay


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 14, 2025 at 06:58:01PM +0000, Puranjay Mohan wrote:
> and the linker script has this line:
> 
> .sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe)) __stop_sframe_header = .; }
> 
> So, do can you suggest the best way to fix these warnings?

Just add *(.init.sframe) like so:

.sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe) *(.init.sframe)) __stop_sframe_header = .; }

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 14, 2025 at 11:38:22AM -0800, Josh Poimboeuf wrote:
> On Fri, Feb 14, 2025 at 06:58:01PM +0000, Puranjay Mohan wrote:
> > and the linker script has this line:
> > 
> > .sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe)) __stop_sframe_header = .; }
> > 
> > So, do can you suggest the best way to fix these warnings?
> 
> Just add *(.init.sframe) like so:
> 
> .sframe : AT(ADDR(.sframe) - 0) { __start_sframe_header = .; KEEP(*(.sframe) *(.init.sframe)) __stop_sframe_header = .; }

Actually each probably needs its own KEEP:

...  KEEP(*(.sframe)) KEEP(*(.init.sframe)) ...

or so.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Indu Bhagat]

On 2/12/25 3:32 PM, Song Liu wrote:
> I run some tests with this set and my RFC set [1]. Most of
> the test is done with kpatch-build. I tested both Puranjay's
> version [3] and my version [4].
> 
> For gcc 14.2.1, I have seen the following issue with this
> test [2]. This happens with both upstream and 6.13.2.
> The livepatch loaded fine, but the system spilled out the
> following warning quickly.
> 

In presence of the issue 
https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad 
data in SFrame section.  Which may be causing this symptom?

To be clear, the issue affects loaded kernel modules.  I cannot tell for 
certain - is there module loading involved in your test ?

> On the other hand, the same test works with LLVM and
> my RFC set (LLVM doesn't support SFRAME, and thus
> doesn't work with this set yet).
> 
> Thanks,
> Song
> 
> 
> [   81.250437] ------------[ cut here ]------------
> [   81.250818] refcount_t: saturated; leaking memory.
> [   81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22
> refcount_warn_saturate+0x6c/0x140
> [   81.251841] Modules linked in: livepatch_special_static(OEK)
> [   81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G
> OE K    6.13.2-00321-g52d2813b4b07 #49
> [   81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
> [   81.253503] Hardware name: linux,dummy-virt (DT)
> [   81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> [   81.254383] pc : refcount_warn_saturate+0x6c/0x140
> [   81.254748] lr : refcount_warn_saturate+0x6c/0x140
> [   81.255114] sp : ffff800085a6fc00
> [   81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180
> [   81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608
> [   81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140
> [   81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff
> [   81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006
> [   81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20
> [   81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4
> [   81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
> [   81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
> [   81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
> [   81.260824] Call trace:
> [   81.261015]  refcount_warn_saturate+0x6c/0x140 (P)
> [   81.261387]  __refcount_add.constprop.0+0x60/0x70
> [   81.261748]  copy_process+0xfdc/0xfd58 [livepatch_special_static]
> [   81.262217]  kernel_clone+0x80/0x3e0
> [   81.262499]  __do_sys_clone+0x5c/0x88
> [   81.262786]  __arm64_sys_clone+0x24/0x38
> [   81.263085]  invoke_syscall+0x4c/0x108
> [   81.263378]  el0_svc_common.constprop.0+0x44/0xe8
> [   81.263734]  do_el0_svc+0x20/0x30
> [   81.263993]  el0_svc+0x34/0xf8
> [   81.264231]  el0t_64_sync_handler+0x104/0x130
> [   81.264561]  el0t_64_sync+0x184/0x188
> [   81.264846] ---[ end trace 0000000000000000 ]---
> [   82.335559] ------------[ cut here ]------------
> [   82.335931] refcount_t: underflow; use-after-free.
> [   82.336307] WARNING: CPU: 1 PID: 0 at lib/refcount.c:28
> refcount_warn_saturate+0xec/0x140
> [   82.336949] Modules linked in: livepatch_special_static(OEK)
> [   82.337389] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G
> W  OE K    6.13.2-00321-g52d2813b4b07 #49
> [   82.338148] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE,
> [K]=LIVEPATCH
> [   82.338721] Hardware name: linux,dummy-virt (DT)
> [   82.339083] pstate: 63400005 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
> [   82.339617] pc : refcount_warn_saturate+0xec/0x140
> [   82.340007] lr : refcount_warn_saturate+0xec/0x140
> [   82.340378] sp : ffff80008370fe40
> [   82.340637] x29: ffff80008370fe40 x28: 0000000000000000 x27: 0000000000000000
> [   82.341188] x26: 000000000000000a x25: ffff0000fdaf7ab8 x24: 0000000000000014
> [   82.341737] x23: ffff8000829c8d30 x22: ffff80008370ff28 x21: ffff0000fe020000
> [   82.342286] x20: ffff0000c062e140 x19: ffff0000c2e9af80 x18: ffffffffffffffff
> [   82.342839] x17: ffff80007b7a0000 x16: ffff800083700000 x15: 0000000000000006
> [   82.343389] x14: 0000000000000000 x13: 2e656572662d7265 x12: 7466612d65737520
> [   82.343944] x11: ffff8000829f7d70 x10: 000000000000016a x9 : ffff8000801546b4
> [   82.344499] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
> [   82.345051] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
> [   82.345604] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
> [   82.346163] Call trace:
> [   82.346359]  refcount_warn_saturate+0xec/0x140 (P)
> [   82.346736]  __put_task_struct+0x130/0x170
> [   82.347063]  delayed_put_task_struct+0xbc/0xe8
> [   82.347411]  rcu_core+0x20c/0x5f8
> [   82.347680]  rcu_core_si+0x14/0x28
> [   82.347952]  handle_softirqs+0x124/0x308
> [   82.348260]  __do_softirq+0x18/0x20
> [   82.348536]  ____do_softirq+0x14/0x28
> [   82.348828]  call_on_irq_stack+0x24/0x30
> [   82.349137]  do_softirq_own_stack+0x20/0x38
> [   82.349465]  __irq_exit_rcu+0xcc/0x108
> [   82.349764]  irq_exit_rcu+0x14/0x28
> [   82.350038]  el1_interrupt+0x34/0x50
> [   82.350321]  el1h_64_irq_handler+0x14/0x20
> [   82.350642]  el1h_64_irq+0x6c/0x70
> [   82.350911]  default_idle_call+0x30/0xd0 (P)
> [   82.351248]  do_idle+0x1d0/0x200
> [   82.351506]  cpu_startup_entry+0x38/0x48
> [   82.351818]  secondary_start_kernel+0x124/0x150
> [   82.352176]  __secondary_switched+0xac/0xb0
> [   82.352505] ---[ end trace 0000000000000000 ]---
> 
> 
> 
> [1] SFRAME-less livepatch RFC
> https://lore.kernel.org/live-patching/20250129232936.1795412-1-song@kernel.org/
> [2] special-static test from kpatch
> https://github.com/dynup/kpatch/blob/master/test/integration/linux-6.2.0/special-static.patch
> [3] Puranjay's kpatch with arm64 support
> https://github.com/puranjaymohan/kpatch/tree/arm64
> [4] My version of kpatch with arm64 and LTO support
> https://github.com/liu-song-6/kpatch/tree/fb-6.13-v2
> 
> On Mon, Jan 27, 2025 at 1:33 PM Weinan Liu <wnliu@google.com> wrote:
>>
>> This patchset implements a generic kernel sframe-based [1] unwinder.
>> The main goal is to support reliable stacktraces on arm64.
>>
>> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
>> required support from objtool: it cannot generate orc unwind tables for
>> arm64.
>>
>> Currently, there's already a sframe unwinder proposed for userspace: [2].
>> Since the sframe unwind table algorithm is similar, these two proposal
>> could integrate common functionality in the future.
>>
>> There are some incomplete features or challenges:
>>    - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>>      FRE from kernel modules doesn't appear correct, preventing us from
>>      unwinding functions from kernel modules.
>>    - Currently, only GCC supports sframe.
>>
>> Ref:
>> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
>> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>>
>> Madhavan T. Venkataraman (1):
>>    arm64: Define TIF_PATCH_PENDING for livepatch
>>
>> Weinan Liu (7):
>>    unwind: build kernel with sframe info
>>    arm64: entry: add unwind info for various kernel entries
>>    unwind: add sframe v2 header
>>    unwind: Implement generic sframe unwinder library
>>    unwind: arm64: Add sframe unwinder on arm64
>>    unwind: arm64: add reliable stacktrace support for arm64
>>    arm64: Enable livepatch for ARM64
>>
>>   Makefile                                   |   6 +
>>   arch/Kconfig                               |   8 +
>>   arch/arm64/Kconfig                         |   3 +
>>   arch/arm64/Kconfig.debug                   |  10 +
>>   arch/arm64/include/asm/stacktrace/common.h |   6 +
>>   arch/arm64/include/asm/thread_info.h       |   4 +-
>>   arch/arm64/kernel/entry-common.c           |   4 +
>>   arch/arm64/kernel/entry.S                  |  10 +
>>   arch/arm64/kernel/setup.c                  |   2 +
>>   arch/arm64/kernel/stacktrace.c             | 102 ++++++++++
>>   include/asm-generic/vmlinux.lds.h          |  12 ++
>>   include/linux/sframe_lookup.h              |  43 +++++
>>   kernel/Makefile                            |   1 +
>>   kernel/sframe.h                            | 215 +++++++++++++++++++++
>>   kernel/sframe_lookup.c                     | 196 +++++++++++++++++++
>>   15 files changed, 621 insertions(+), 1 deletion(-)
>>   create mode 100644 include/linux/sframe_lookup.h
>>   create mode 100644 kernel/sframe.h
>>   create mode 100644 kernel/sframe_lookup.c
>>
>> --
>> 2.48.1.262.g85cc9f2d1e-goog
>>
>>

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>
> On 2/12/25 3:32 PM, Song Liu wrote:
> > I run some tests with this set and my RFC set [1]. Most of
> > the test is done with kpatch-build. I tested both Puranjay's
> > version [3] and my version [4].
> >
> > For gcc 14.2.1, I have seen the following issue with this
> > test [2]. This happens with both upstream and 6.13.2.
> > The livepatch loaded fine, but the system spilled out the
> > following warning quickly.
> >
>
> In presence of the issue
> https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
> data in SFrame section.  Which may be causing this symptom?
>
> To be clear, the issue affects loaded kernel modules.  I cannot tell for
> certain - is there module loading involved in your test ?

The KLP is a module, I guess that is also affected?

During kpatch-build, we added some logic to drop the .sframe section.
I guess this is wrong, as we need the .sframe section when we apply
the next KLP. However, I don't think the issue is caused by missing
.sframe section.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Wed, Feb 12, 2025 at 06:40:33PM -0800, Song Liu wrote:
> On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
> >
> > On 2/12/25 3:32 PM, Song Liu wrote:
> > > I run some tests with this set and my RFC set [1]. Most of
> > > the test is done with kpatch-build. I tested both Puranjay's
> > > version [3] and my version [4].
> > >
> > > For gcc 14.2.1, I have seen the following issue with this
> > > test [2]. This happens with both upstream and 6.13.2.
> > > The livepatch loaded fine, but the system spilled out the
> > > following warning quickly.
> > >
> >
> > In presence of the issue
> > https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
> > data in SFrame section.  Which may be causing this symptom?
> >
> > To be clear, the issue affects loaded kernel modules.  I cannot tell for
> > certain - is there module loading involved in your test ?
> 
> The KLP is a module, I guess that is also affected?
> 
> During kpatch-build, we added some logic to drop the .sframe section.
> I guess this is wrong, as we need the .sframe section when we apply
> the next KLP. However, I don't think the issue is caused by missing
> .sframe section.

Right, it looks like it unwound through the patch module just fine.

-- 
Josh

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]

Song Liu <song@kernel.org> writes:

> On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>>
>> On 2/12/25 3:32 PM, Song Liu wrote:
>> > I run some tests with this set and my RFC set [1]. Most of
>> > the test is done with kpatch-build. I tested both Puranjay's
>> > version [3] and my version [4].
>> >
>> > For gcc 14.2.1, I have seen the following issue with this
>> > test [2]. This happens with both upstream and 6.13.2.
>> > The livepatch loaded fine, but the system spilled out the
>> > following warning quickly.
>> >
>>
>> In presence of the issue
>> https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
>> data in SFrame section.  Which may be causing this symptom?
>>
>> To be clear, the issue affects loaded kernel modules.  I cannot tell for
>> certain - is there module loading involved in your test ?
>
> The KLP is a module, I guess that is also affected?
>
> During kpatch-build, we added some logic to drop the .sframe section.
> I guess this is wrong, as we need the .sframe section when we apply
> the next KLP. However, I don't think the issue is caused by missing
> .sframe section.

Hi, I did the same testing and did not get the Warning.

I am testing on the 6.12.11 kernel with GCC 11.4.1.

Just to verify, the patch we are testing is:

--- >8 ---

diff -Nupr src.orig/kernel/fork.c src/kernel/fork.c
--- src.orig/kernel/fork.c      2023-01-12 11:20:05.408700033 -0500
+++ src/kernel/fork.c   2023-01-12 11:21:19.186137466 -0500
@@ -1700,10 +1700,18 @@ static void posix_cpu_timers_init_group(
        posix_cputimers_group_init(pct, cpu_limit);
 }

+void kpatch_foo(void)
+{
+       if (!jiffies)
+               printk("kpatch copy signal\n");
+}
+
 static int copy_signal(unsigned long clone_flags, struct task_struct *tsk)
 {
        struct signal_struct *sig;

+       kpatch_foo();
+
        if (clone_flags & CLONE_THREAD)
                return 0;
--- 8< ---

P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for
livepatch module, will add it to the PR after some cleanups.

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Wed, Feb 12, 2025 at 11:26 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>
> Song Liu <song@kernel.org> writes:
>
> > On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
> >>
> >> On 2/12/25 3:32 PM, Song Liu wrote:
> >> > I run some tests with this set and my RFC set [1]. Most of
> >> > the test is done with kpatch-build. I tested both Puranjay's
> >> > version [3] and my version [4].
> >> >
> >> > For gcc 14.2.1, I have seen the following issue with this
> >> > test [2]. This happens with both upstream and 6.13.2.
> >> > The livepatch loaded fine, but the system spilled out the
> >> > following warning quickly.
> >> >
> >>
> >> In presence of the issue
> >> https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
> >> data in SFrame section.  Which may be causing this symptom?
> >>
> >> To be clear, the issue affects loaded kernel modules.  I cannot tell for
> >> certain - is there module loading involved in your test ?
> >
> > The KLP is a module, I guess that is also affected?
> >
> > During kpatch-build, we added some logic to drop the .sframe section.
> > I guess this is wrong, as we need the .sframe section when we apply
> > the next KLP. However, I don't think the issue is caused by missing
> > .sframe section.
>
> Hi, I did the same testing and did not get the Warning.
>
> I am testing on the 6.12.11 kernel with GCC 11.4.1.

Could you please also try kernel 6.13.2?

> Just to verify, the patch we are testing is:

Yes, this is the test patch.
>
> --- >8 ---
[...]
> --- 8< ---
>
> P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for
> livepatch module, will add it to the PR after some cleanups.

Yeah, I think the .sframe section is still needed.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 2250 bytes --]

Song Liu <song@kernel.org> writes:

> On Wed, Feb 12, 2025 at 11:26 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>>
>> Song Liu <song@kernel.org> writes:
>>
>> > On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>> >>
>> >> On 2/12/25 3:32 PM, Song Liu wrote:
>> >> > I run some tests with this set and my RFC set [1]. Most of
>> >> > the test is done with kpatch-build. I tested both Puranjay's
>> >> > version [3] and my version [4].
>> >> >
>> >> > For gcc 14.2.1, I have seen the following issue with this
>> >> > test [2]. This happens with both upstream and 6.13.2.
>> >> > The livepatch loaded fine, but the system spilled out the
>> >> > following warning quickly.
>> >> >
>> >>
>> >> In presence of the issue
>> >> https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
>> >> data in SFrame section.  Which may be causing this symptom?
>> >>
>> >> To be clear, the issue affects loaded kernel modules.  I cannot tell for
>> >> certain - is there module loading involved in your test ?
>> >
>> > The KLP is a module, I guess that is also affected?
>> >
>> > During kpatch-build, we added some logic to drop the .sframe section.
>> > I guess this is wrong, as we need the .sframe section when we apply
>> > the next KLP. However, I don't think the issue is caused by missing
>> > .sframe section.
>>
>> Hi, I did the same testing and did not get the Warning.
>>
>> I am testing on the 6.12.11 kernel with GCC 11.4.1.
>
> Could you please also try kernel 6.13.2?
>
>> Just to verify, the patch we are testing is:
>
> Yes, this is the test patch.
>>
>> --- >8 ---
> [...]
>> --- 8< ---
>>
>> P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for
>> livepatch module, will add it to the PR after some cleanups.
>
> Yeah, I think the .sframe section is still needed.
>

Hi Song,

Can you try with this:
https://github.com/puranjaymohan/kpatch/tree/arm64_wip

This has the .sframe logic patch, but it looks as if I wrote that code
in a 30 minute leetcode interview. I need to refactor it before I send
it for review with the main PR.

Can you test with this branch with your setup?

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Wed, Feb 12, 2025 at 11:53 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>
> Song Liu <song@kernel.org> writes:
>
> > On Wed, Feb 12, 2025 at 11:26 PM Puranjay Mohan <puranjay@kernel.org> wrote:
> >>
> >> Song Liu <song@kernel.org> writes:
> >>
> >> > On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
> >> >>
> >> >> On 2/12/25 3:32 PM, Song Liu wrote:
> >> >> > I run some tests with this set and my RFC set [1]. Most of
> >> >> > the test is done with kpatch-build. I tested both Puranjay's
> >> >> > version [3] and my version [4].
> >> >> >
> >> >> > For gcc 14.2.1, I have seen the following issue with this
> >> >> > test [2]. This happens with both upstream and 6.13.2.
> >> >> > The livepatch loaded fine, but the system spilled out the
> >> >> > following warning quickly.
> >> >> >
> >> >>
> >> >> In presence of the issue
> >> >> https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
> >> >> data in SFrame section.  Which may be causing this symptom?
> >> >>
> >> >> To be clear, the issue affects loaded kernel modules.  I cannot tell for
> >> >> certain - is there module loading involved in your test ?
> >> >
> >> > The KLP is a module, I guess that is also affected?
> >> >
> >> > During kpatch-build, we added some logic to drop the .sframe section.
> >> > I guess this is wrong, as we need the .sframe section when we apply
> >> > the next KLP. However, I don't think the issue is caused by missing
> >> > .sframe section.
> >>
> >> Hi, I did the same testing and did not get the Warning.
> >>
> >> I am testing on the 6.12.11 kernel with GCC 11.4.1.
> >
> > Could you please also try kernel 6.13.2?
> >
> >> Just to verify, the patch we are testing is:
> >
> > Yes, this is the test patch.
> >>
> >> --- >8 ---
> > [...]
> >> --- 8< ---
> >>
> >> P.S. - I have a downstream patch for create-diff-object to generate .sframe sections for
> >> livepatch module, will add it to the PR after some cleanups.
> >
> > Yeah, I think the .sframe section is still needed.
> >
>
> Hi Song,
>
> Can you try with this:
> https://github.com/puranjaymohan/kpatch/tree/arm64_wip
>
> This has the .sframe logic patch, but it looks as if I wrote that code
> in a 30 minute leetcode interview. I need to refactor it before I send
> it for review with the main PR.
>
> Can you test with this branch with your setup?

This branch has the same issue as the other branch.

Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 25191 bytes --]

Song Liu <song@kernel.org> writes:

> On Wed, Feb 12, 2025 at 11:26 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>>
>> Song Liu <song@kernel.org> writes:
>>
>> > On Wed, Feb 12, 2025 at 4:10 PM Indu Bhagat <indu.bhagat@oracle.com> wrote:
>> >>
>> >> On 2/12/25 3:32 PM, Song Liu wrote:
>> >> > I run some tests with this set and my RFC set [1]. Most of
>> >> > the test is done with kpatch-build. I tested both Puranjay's
>> >> > version [3] and my version [4].
>> >> >
>> >> > For gcc 14.2.1, I have seen the following issue with this
>> >> > test [2]. This happens with both upstream and 6.13.2.
>> >> > The livepatch loaded fine, but the system spilled out the
>> >> > following warning quickly.
>> >> >
>> >>
>> >> In presence of the issue
>> >> https://sourceware.org/bugzilla/show_bug.cgi?id=32666, I'd expect bad
>> >> data in SFrame section.  Which may be causing this symptom?
>> >>
>> >> To be clear, the issue affects loaded kernel modules.  I cannot tell for
>> >> certain - is there module loading involved in your test ?
>> >
>> > The KLP is a module, I guess that is also affected?
>> >
>> > During kpatch-build, we added some logic to drop the .sframe section.
>> > I guess this is wrong, as we need the .sframe section when we apply
>> > the next KLP. However, I don't think the issue is caused by missing
>> > .sframe section.
>>
>> Hi, I did the same testing and did not get the Warning.
>>
>> I am testing on the 6.12.11 kernel with GCC 11.4.1.
>
> Could you please also try kernel 6.13.2?
>

I have tested with 6.13.2, here are the results:

[ec2-user@ip-172-31-32-86 ~]$ uname -r
6.13.2+

My tree looks like:

[ec2-user@ip-172-31-32-86 linux-upstream]$ git log --oneline
bdf4086dd (HEAD) arm64: Enable livepatch for ARM64
3904c7058 arm64: Define TIF_PATCH_PENDING for livepatch
b5de442a9 unwind: arm64: add reliable stacktrace support for arm64
fdc243fa8 unwind: arm64: Add sframe unwinder on arm64
87681b7c0 unwind: Implement generic sframe unwinder library
c8833ba50 unwind: add sframe v2 header
f3f3863cf arm64: entry: add unwind info for various kernel entries
57ab97f05 unwind: build kernel with sframe info
0da4b4b84 Linux 6.13.2

I have also fixed the 87681b7c0 ("unwind: Implement generic sframe
unwinder library")

with:

--- >8 ---

diff --git a/kernel/sframe_lookup.c b/kernel/sframe_lookup.c
index 846f1da95d89..28bec5064dc7 100644
--- a/kernel/sframe_lookup.c
+++ b/kernel/sframe_lookup.c
@@ -82,7 +82,7 @@ static struct sframe_fde *find_fde(const struct sframe_table *tbl, unsigned long
        if (f >= tbl->sfhdr_p->num_fdes || f < 0)
                return NULL;
        fdep = tbl->fde_p + f;
-       if (ip < fdep->start_addr || ip >= fdep->start_addr + fdep->size)
+       if (ip < fdep->start_addr || ip > fdep->start_addr + fdep->size)
                return NULL;

        return fdep;
@@ -106,7 +106,7 @@ static int find_fre(const struct sframe_table *tbl, unsigned long pc,
        else
                ip_off = (int32_t)(pc - (unsigned long)tbl->sfhdr_p) - fdep->start_addr;

-       if (ip_off < 0 || ip_off >= fdep->size)
+       if (ip_off < 0 || ip_off > fdep->size)
                return -EINVAL;

        /*

--- 8< ---

GCC is gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)

kpatch is from https://github.com/puranjaymohan/kpatch/tree/arm64_wip

I run the with following:

./kpatch/kpatch-build/kpatch-build --non-replace -d -v \
linux-upstream/vmlinux -s linux-upstream/ -c linux-upstream/.config \
kpatch/test/integration/linux-6.2.0/special-static.patch

and my dmesg output is:

[ec2-user@ip-172-31-32-86 ~]$ sudo dmesg
[  167.202596] livepatch_special_static: loading out-of-tree module taints kernel.
[  167.203217] livepatch_special_static: tainting kernel with TAINT_LIVEPATCH
[  167.203760] livepatch_special_static: module verification failed: signature and/or required key missing - tainting kernel
[  167.205659] livepatch: enabling patch 'livepatch_special_static'
[  167.207358] livepatch: 'livepatch_special_static': starting patching transition
[  168.264901] livepatch: 'livepatch_special_static': patching complete
[  410.641806] livepatch: 'livepatch_special_static': starting unpatching transition
[  412.389369] livepatch: 'livepatch_special_static': unpatching complete

I even ran stress-ng with the livepatch loaded to see if something
happens.


P.S. - The livepatch doesn't have copy_process() but only copy_signal(),
yours had copy_process() somehow.

Here is the symbol table of the .ko

Symbol table '.symtab' contains 169 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 SECTION LOCAL  DEFAULT    1 .note.gnu.build-id
     2: 0000000000000000     0 SECTION LOCAL  DEFAULT    2 .note.Linux
     3: 0000000000000000     0 SECTION LOCAL  DEFAULT    3 .text
     4: 0000000000000000     0 SECTION LOCAL  DEFAULT    5 .exit.text
     5: 0000000000000000     0 SECTION LOCAL  DEFAULT    7 .init.text
     6: 0000000000000000     0 SECTION LOCAL  DEFAULT    9 .text.copy_signal
     7: 0000000000000000     0 SECTION LOCAL  DEFAULT   11 .text.kpatch_foo
     8: 0000000000000000     0 SECTION LOCAL  DEFAULT   13 .altinstructions
     9: 0000000000000000     0 SECTION LOCAL  DEFAULT   15 __patchable_function_entries
    10: 0000000000000000     0 SECTION LOCAL  DEFAULT   17 .codetag.alloc_tags
    11: 0000000000000000     0 SECTION LOCAL  DEFAULT   18 .plt
    12: 0000000000000000     0 SECTION LOCAL  DEFAULT   19 .init.plt
    13: 0000000000000000     0 SECTION LOCAL  DEFAULT   20 .text.ftrace_trampoline
    14: 0000000000000000     0 SECTION LOCAL  DEFAULT   21 .rodata.str1.8
    15: 0000000000000000     0 SECTION LOCAL  DEFAULT   22 .modinfo
    16: 0000000000000000     0 SECTION LOCAL  DEFAULT   23 .sframe
    17: 0000000000000000     0 SECTION LOCAL  DEFAULT   25 .rodata.trace_raw_output_task_newtask.str1.8
    18: 0000000000000000     0 SECTION LOCAL  DEFAULT   26 .rodata.trace_raw_output_task_rename.str1.8
    19: 0000000000000000     0 SECTION LOCAL  DEFAULT   27 .rodata.sighand_ctor.str1.8
    20: 0000000000000000     0 SECTION LOCAL  DEFAULT   28 .rodata.str
    21: 0000000000000000     0 SECTION LOCAL  DEFAULT   29 .rodata.__mmdrop.str1.8
    22: 0000000000000000     0 SECTION LOCAL  DEFAULT   30 .rodata.copy_signal.str1.8
    23: 0000000000000000     0 SECTION LOCAL  DEFAULT   31 .rodata.mm_init.str1.8
    24: 0000000000000000     0 SECTION LOCAL  DEFAULT   32 .rodata.vm_area_alloc.str1.8
    25: 0000000000000000     0 SECTION LOCAL  DEFAULT   33 .rodata.dup_mmap.str1.8
    26: 0000000000000000     0 SECTION LOCAL  DEFAULT   34 .rodata.copy_process.str1.8
    27: 0000000000000000     0 SECTION LOCAL  DEFAULT   35 .rodata.kernel_clone.str1.8
    28: 0000000000000000     0 SECTION LOCAL  DEFAULT   36 .rodata.str__task__trace_system_name
    29: 0000000000000000     0 SECTION LOCAL  DEFAULT   37 .kpatch.strings
    30: 0000000000000000     0 SECTION LOCAL  DEFAULT   38 .kpatch.funcs
    31: 0000000000000000     0 SECTION LOCAL  DEFAULT   40 __versions
    32: 0000000000000000     0 SECTION LOCAL  DEFAULT   41 .bss
    33: 0000000000000000     0 SECTION LOCAL  DEFAULT   42 .bss.__key.4
    34: 0000000000000000     0 SECTION LOCAL  DEFAULT   43 .bss.__key.5
    35: 0000000000000000     0 SECTION LOCAL  DEFAULT   44 .bss.__key.6
    36: 0000000000000000     0 SECTION LOCAL  DEFAULT   45 .note.GNU-stack
    37: 0000000000000000     0 SECTION LOCAL  DEFAULT   46 .comment
    38: 0000000000000000     0 SECTION LOCAL  DEFAULT   47 .data
    39: 0000000000000000     0 SECTION LOCAL  DEFAULT   49 .exit.data
    40: 0000000000000000     0 SECTION LOCAL  DEFAULT   51 .init.data
    41: 0000000000000000     0 SECTION LOCAL  DEFAULT   53 .kpatch.callbacks.pre_patch
    42: 0000000000000000     0 SECTION LOCAL  DEFAULT   54 .kpatch.callbacks.post_patch
    43: 0000000000000000     0 SECTION LOCAL  DEFAULT   55 .kpatch.callbacks.pre_unpatch
    44: 0000000000000000     0 SECTION LOCAL  DEFAULT   56 .kpatch.callbacks.post_unpatch
    45: 0000000000000000     0 SECTION LOCAL  DEFAULT   57 .kpatch.force
    46: 0000000000000000     0 SECTION LOCAL  DEFAULT   58 .gnu.linkonce.this_module
    47: 0000000000000000     0 SECTION LOCAL  DEFAULT   60 .debug_info
    48: 0000000000000000     0 SECTION LOCAL  DEFAULT   62 .debug_abbrev
    49: 0000000000000000     0 SECTION LOCAL  DEFAULT   63 .debug_loc
    50: 0000000000000000     0 SECTION LOCAL  DEFAULT   65 .debug_aranges
    51: 0000000000000000     0 SECTION LOCAL  DEFAULT   67 .debug_ranges
    52: 0000000000000000     0 SECTION LOCAL  DEFAULT   69 .debug_line
    53: 0000000000000000     0 SECTION LOCAL  DEFAULT   71 .debug_str
    54: 0000000000000000     0 SECTION LOCAL  DEFAULT   72 .debug_frame
    55: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS module-common.c
    56: 0000000000000062    52 OBJECT  LOCAL  DEFAULT   22 __UNIQUE_ID_vermagic547
    57: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT    2 $d
    58: 0000000000000000    24 OBJECT  LOCAL  DEFAULT    2 _note_19
    59: 0000000000000018    52 OBJECT  LOCAL  DEFAULT    2 _note_18
    60: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS patch-hook.c
    61: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT    3 $x
    62: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   15 $d
    63: 0000000000000008   104 FUNC    LOCAL  DEFAULT    3 patch_free_livepatch
    64: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT    5 $x
    65: 0000000000000000    36 FUNC    LOCAL  DEFAULT    5 patch_exit
    66: 0000000000000078   260 FUNC    LOCAL  DEFAULT    3 patch_free_scaffold
    67: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   21 $d
    68: 0000000000000188   280 FUNC    LOCAL  DEFAULT    3 patch_find_object_by_name
    69: 00000000000002a8   432 FUNC    LOCAL  DEFAULT    3 add_callbacks_to_patch_objects
    70: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT    7 $x
    71: 0000000000000008   584 FUNC    LOCAL  DEFAULT    7 patch_init
    72: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   47 $d
    73: 0000000000000000    16 OBJECT  LOCAL  DEFAULT   47 patch_objects
    74: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   41 $d
    75: 0000000000000000     8 OBJECT  LOCAL  DEFAULT   41 lpatch
    76: 0000000000000008     4 OBJECT  LOCAL  DEFAULT   41 patch_objects_nr
    77: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   49 $d
    78: 0000000000000000     8 OBJECT  LOCAL  DEFAULT   49 __UNIQUE_ID___addressable_cleanup_module737
    79: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   51 $d
    80: 0000000000000000     8 OBJECT  LOCAL  DEFAULT   51 __UNIQUE_ID___addressable_init_module736
    81: 0000000000000000    12 OBJECT  LOCAL  DEFAULT   22 __UNIQUE_ID_livepatch739
    82: 000000000000000c    12 OBJECT  LOCAL  DEFAULT   22 __UNIQUE_ID_license738
    83: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS fork.c
    84: 0000000000000008   496 FUNC    LOCAL  DEFAULT    9 copy_signal
    85: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   25 $d
    86: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   26 $d
    87: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   27 $d
    88: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   29 $d
    89: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   30 $d
    90: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   31 $d
    91: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   32 $d
    92: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   33 $d
    93: 0000000000000128     0 NOTYPE  LOCAL  DEFAULT   21 $d
    94: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   34 $d
    95: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   35 $d
    96: 0000000000000000     0 OBJECT  LOCAL  DEFAULT   42 __key.4
    97: 0000000000000000     0 OBJECT  LOCAL  DEFAULT   43 __key.5
    98: 0000000000000000     0 OBJECT  LOCAL  DEFAULT   44 __key.6
    99: 0000000000000000     5 OBJECT  LOCAL  DEFAULT   36 str__task__trace_system_name
   100: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   36 $d
   101: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B1
   102: 000000000000000e     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B2
   103: 000000000000001c     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B3
   104: 000000000000002a     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B4
   105: 0000000000000038     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B5
   106: 0000000000000046     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B6
   107: 0000000000000054     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B7
   108: 0000000000000062     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B8
   109: 0000000000000070     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B9
   110: 000000000000007e     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B10
   111: 000000000000008c     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B11
   112: 00000000000000a9     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B12
   113: 00000000000000c1     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B13
   114: 00000000000000d9     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B14
   115: 00000000000000f1     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B15
   116: 0000000000000109     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B16
   117: 0000000000000124     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B17
   118: 0000000000000132     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B18
   119: 000000000000014d     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B19
   120: 0000000000000168     0 NOTYPE  LOCAL  DEFAULT   28 .L14472^B20
   121: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS livepatch-special-static.mod.c
   122: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   58 $d
   123: 0000000000000018    35 OBJECT  LOCAL  DEFAULT   22 __UNIQUE_ID_srcversion549
   124: 000000000000003b     9 OBJECT  LOCAL  DEFAULT   22 __UNIQUE_ID_depends548
   125: 0000000000000044    30 OBJECT  LOCAL  DEFAULT   22 __UNIQUE_ID_name547
   126: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   40 $d
   127: 0000000000000000     0 OBJECT  LOCAL  DEFAULT OS [0xff20] .klp.sym.vmlinux.signal_cachep,1
   128: 0000000000000000  1216 OBJECT  LOCAL  DEFAULT   40 ____versions
   129: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   55 __kpatch_callbacks_pre_unpatch_end
   130: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __init_rwsem
   131: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __list_add_valid_or_report
   132: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __kmalloc_noprof
   133: 0000000000000000  1344 OBJECT  GLOBAL DEFAULT   58 __this_module
   134: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   54 __kpatch_callbacks_post_patch
   135: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   57 __kpatch_force_funcs
   136: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   17 __stop_alloc_tags
   137: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND hrtimer_init
   138: 0000000000000000    36 FUNC    GLOBAL DEFAULT    5 cleanup_module
   139: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND memcpy
   140: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND klp_enable_patch
   141: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND kfree
   142: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   53 __kpatch_callbacks_pre_patch
   143: 0000000000000008   584 FUNC    GLOBAL DEFAULT    7 init_module
   144: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   55 __kpatch_callbacks_pre_unpatch
   145: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   17 __start_alloc_tags
   146: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   38 __kpatch_funcs
   147: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND _printk
   148: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   57 __kpatch_force_funcs_end
   149: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND queued_spin_lock_slowpath
   150: 0000000000000038     0 NOTYPE  GLOBAL DEFAULT   38 __kpatch_funcs_end
   151: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   54 __kpatch_callbacks_post_patch_end
   152: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   53 __kpatch_callbacks_pre_patch_end
   153: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __list_del_entry_valid_or_report
   154: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __mutex_init
   155: 0000000000000008    60 FUNC    GLOBAL DEFAULT   11 kpatch_foo
   156: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   56 __kpatch_callbacks_post_unpatch_end
   157: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND kmem_cache_alloc_noprof
   158: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __init_waitqueue_head
   159: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND strcmp
   160: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND jiffies
   161: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT   56 __kpatch_callbacks_post_unpatch
   162: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND __kmalloc_cache_noprof
   163: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND alt_cb_patch_nops
   164: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND kmalloc_caches
   165: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.it_real_fn,0
   166: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.posix_cputimers_group_init,0
   167: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.tty_audit_fork,0
   168: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT OS [0xff20] .klp.sym.vmlinux.sched_autogroup_fork,0

and Sections:

[ec2-user@ip-172-31-32-86 ~]$ readelf -WS livepatch-special-static.ko
There are 78 section headers, starting at offset 0x1727d8:

Section Headers:
  [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
  [ 1] .note.gnu.build-id NOTE            0000000000000000 000040 000024 00   A  0   0  4
  [ 2] .note.Linux       NOTE            0000000000000000 000064 00004c 00   A  0   0  4
  [ 3] .text             PROGBITS        0000000000000000 0000c0 000458 00  AX  0   0 32
  [ 4] .rela.text        RELA            0000000000000000 000518 000570 18   I 74   3  8
  [ 5] .exit.text        PROGBITS        0000000000000000 000a88 000024 00  AX  0   0  8
  [ 6] .rela.exit.text   RELA            0000000000000000 000ab0 000048 18   I 74   5  8
  [ 7] .init.text        PROGBITS        0000000000000000 000b00 000250 00  AX  0   0 32
  [ 8] .rela.init.text   RELA            0000000000000000 000d50 000348 18   I 74   7  8
  [ 9] .text.copy_signal PROGBITS        0000000000000000 001098 0001f8 00  AX  0   0  8
  [10] .rela.text.copy_signal RELA            0000000000000000 001290 000240 18   I 74   9  8
  [11] .text.kpatch_foo  PROGBITS        0000000000000000 0014d0 000044 00  AX  0   0  8
  [12] .rela.text.kpatch_foo RELA            0000000000000000 001518 000078 18   I 74  11  8
  [13] .altinstructions  PROGBITS        0000000000000000 001590 00000c 00   A  0   0  8
  [14] .rela.altinstructions RELA            0000000000000000 0015a0 000030 18   I 74  13  8
  [15] __patchable_function_entries PROGBITS        0000000000000010 0015d0 000038 00 WAL  3   0  8
  [16] .rela__patchable_function_entries RELA            0000000000000000 001608 0000a8 18   I 74  15  8
  [17] .codetag.alloc_tags PROGBITS        0000000000000540 0016b0 000000 00   W  0   0  1
  [18] .plt              PROGBITS        0000000000000000 0016b0 000001 00  AX  0   0  1
  [19] .init.plt         PROGBITS        0000000000000000 0016b1 000001 00   A  0   0  1
  [20] .text.ftrace_trampoline PROGBITS        0000000000000000 0016b2 000001 00  AX  0   0  1
  [21] .rodata.str1.8    PROGBITS        0000000000000000 0016b8 000455 01 AMS  0   0  8
  [22] .modinfo          PROGBITS        0000000000000000 001b0d 000096 00   A  0   0  1
  [23] .sframe           PROGBITS        0000000000000000 001ba8 0001ad 00   A  0   0  8
  [24] .rela.sframe      RELA            0000000000000000 001d58 0000c0 18   I 74  23  8
  [25] .rodata.trace_raw_output_task_newtask.str1.8 PROGBITS        0000000000000000 001e18 000032 01 AMS  0   0  8
  [26] .rodata.trace_raw_output_task_rename.str1.8 PROGBITS        0000000000000000 001e50 000030 01 AMS  0   0  8
  [27] .rodata.sighand_ctor.str1.8 PROGBITS        0000000000000000 001e80 000017 01 AMS  0   0  8
  [28] .rodata.str       PROGBITS        0000000000000000 001e97 000180 01 AMS  0   0  1
  [29] .rodata.__mmdrop.str1.8 PROGBITS        0000000000000000 002018 00006b 01 AMS  0   0  8
  [30] .rodata.copy_signal.str1.8 PROGBITS        0000000000000000 002088 00005f 01 AMS  0   0  8
  [31] .rodata.mm_init.str1.8 PROGBITS        0000000000000000 0020e8 00000f 01 AMS  0   0  8
  [32] .rodata.vm_area_alloc.str1.8 PROGBITS        0000000000000000 0020f8 000014 01 AMS  0   0  8
  [33] .rodata.dup_mmap.str1.8 PROGBITS        0000000000000000 002110 000054 01 AMS  0   0  8
  [34] .rodata.copy_process.str1.8 PROGBITS        0000000000000000 002168 000017 01 AMS  0   0  8
  [35] .rodata.kernel_clone.str1.8 PROGBITS        0000000000000000 002180 000009 01 AMS  0   0  8
  [36] .rodata.str__task__trace_system_name PROGBITS        0000000000000000 002190 000005 00   A  0   0  8
  [37] .kpatch.strings   PROGBITS        0000000000000000 002195 00006c 00   A  0   0  1
  [38] .kpatch.funcs     PROGBITS        0000000000000000 002208 000038 00   A  0   0  8
  [39] .rela.kpatch.funcs RELA            0000000000000000 002240 000048 18   I 74  38  8
  [40] __versions        PROGBITS        0000000000000000 002288 0004c0 00   A  0   0  8
  [41] .bss              NOBITS          0000000000000000 002748 00000c 00  WA  0   0  8
  [42] .bss.__key.4      NOBITS          0000000000000000 002748 000000 00  WA  0   0  8
  [43] .bss.__key.5      NOBITS          0000000000000000 002748 000000 00  WA  0   0  8
  [44] .bss.__key.6      NOBITS          0000000000000000 002748 000000 00  WA  0   0  8
  [45] .note.GNU-stack   PROGBITS        0000000000000000 002748 000000 00      0   0  1
  [46] .comment          PROGBITS        0000000000000000 002748 00008d 01  MS  0   0  1
  [47] .data             PROGBITS        0000000000000048 0027d8 000010 00  WA  0   0  8
  [48] .rela.data        RELA            0000000000000000 0027e8 000030 18   I 74  47  8
  [49] .exit.data        PROGBITS        0000000000000000 002818 000008 00  WA  0   0  8
  [50] .rela.exit.data   RELA            0000000000000000 002820 000018 18   I 74  49  8
  [51] .init.data        PROGBITS        0000000000000000 002838 000008 00  WA  0   0  8
  [52] .rela.init.data   RELA            0000000000000000 002840 000018 18   I 74  51  8
  [53] .kpatch.callbacks.pre_patch PROGBITS        0000000000000000 002858 000008 00  WA  0   0  1
  [54] .kpatch.callbacks.post_patch PROGBITS        0000000000000000 002860 000008 00  WA  0   0  1
  [55] .kpatch.callbacks.pre_unpatch PROGBITS        0000000000000000 002868 000008 00  WA  0   0  1
  [56] .kpatch.callbacks.post_unpatch PROGBITS        0000000000000000 002870 000008 00  WA  0   0  1
  [57] .kpatch.force     PROGBITS        0000000000000000 002878 000008 00  WA  0   0  1
  [58] .gnu.linkonce.this_module PROGBITS        0000000000000000 002880 000540 00  WA  0   0 64
  [59] .rela.gnu.linkonce.this_module RELA            0000000000000000 002dc0 000030 18   I 74  58  8
  [60] .debug_info       PROGBITS        0000000000000000 002df0 06041d 00      0   0  1
  [61] .rela.debug_info  RELA            0000000000000000 063210 08a390 18   I 74  60  8
  [62] .debug_abbrev     PROGBITS        0000000000000000 0ed5a0 002392 00      0   0  1
  [63] .debug_loc        PROGBITS        0000000000000000 0ef932 031dd6 00      0   0  1
  [64] .rela.debug_loc   RELA            0000000000000000 121708 004e30 18   I 74  63  8
  [65] .debug_aranges    PROGBITS        0000000000000000 126538 000740 00      0   0  1
  [66] .rela.debug_aranges RELA            0000000000000000 126c78 0000d8 18   I 74  65  8
  [67] .debug_ranges     PROGBITS        0000000000000000 126d50 00a870 00      0   0  1
  [68] .rela.debug_ranges RELA            0000000000000000 1315c0 001320 18   I 74  67  8
  [69] .debug_line       PROGBITS        0000000000000000 1328e0 010623 00      0   0  1
  [70] .rela.debug_line  RELA            0000000000000000 142f08 000078 18   I 74  69  8
  [71] .debug_str        PROGBITS        0000000000000000 142f80 02b764 01  MS  0   0  1
  [72] .debug_frame      PROGBITS        0000000000000000 16e6e8 001900 00      0   0  8
  [73] .rela.debug_frame RELA            0000000000000000 16ffe8 000b28 18   I 74  72  8
  [74] .symtab           SYMTAB          0000000000000000 170b10 000fd8 18     75 129  8
  [75] .strtab           STRTAB          0000000000000000 171ae8 0006cc 00      0   0  1
  [76] .shstrtab         STRTAB          0000000000000000 1721b4 000576 00      0   0  1
  [77] .klp.rela.vmlinux..text.copy_signal RELA 0000000000000000 172730 0000a8 18 AIo 74   9  8
  

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <puranjay@kernel.org> wrote:
[...]
>
> P.S. - The livepatch doesn't have copy_process() but only copy_signal(),
> yours had copy_process() somehow.

In my build, copy_signal is inlined to copy_process, unless I add noinline.
If I do add noinline, the issue will not reproduce.

I tried more combinations. The issue doesn't reproduce if I either
1) add noinline to copy_signal, so we are not patching the whole
   copy_process function;
or
2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.

So it appears something in gcc 14.2.1 is causing live patch to fail
for copy_process().

Thanks,
Song

[...]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]

Song Liu <song@kernel.org> writes:

> On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <puranjay@kernel.org> wrote:
> [...]
>>
>> P.S. - The livepatch doesn't have copy_process() but only copy_signal(),
>> yours had copy_process() somehow.
>
> In my build, copy_signal is inlined to copy_process, unless I add noinline.
> If I do add noinline, the issue will not reproduce.
>
> I tried more combinations. The issue doesn't reproduce if I either
> 1) add noinline to copy_signal, so we are not patching the whole
>    copy_process function;
> or
> 2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.
>
> So it appears something in gcc 14.2.1 is causing live patch to fail
> for copy_process().

So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we
can be sure that it is not a sframe problem?

And about having the .sframe section in the livepatch module, I realised
that this set doesn't include support for reading/using sframe data from
any module(livepatches included), so the patch I added for generating
.sframe in kpatch is irrelevant because it is a no-op with the current setup.

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Thu, Feb 13, 2025 at 2:22 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>
> Song Liu <song@kernel.org> writes:
>
> > On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <puranjay@kernel.org> wrote:
> > [...]
> >>
> >> P.S. - The livepatch doesn't have copy_process() but only copy_signal(),
> >> yours had copy_process() somehow.
> >
> > In my build, copy_signal is inlined to copy_process, unless I add noinline.
> > If I do add noinline, the issue will not reproduce.
> >
> > I tried more combinations. The issue doesn't reproduce if I either
> > 1) add noinline to copy_signal, so we are not patching the whole
> >    copy_process function;
> > or
> > 2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.
> >
> > So it appears something in gcc 14.2.1 is causing live patch to fail
> > for copy_process().
>
> So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we
> can be sure that it is not a sframe problem?

My RFC set is the same. No issue with gcc 11.5.0; but hits the same
WARNING with gcc 14.2.1. My previous tests are all with clang. I didn't
see a similar issue there.

> And about having the .sframe section in the livepatch module, I realised
> that this set doesn't include support for reading/using sframe data from
> any module(livepatches included), so the patch I added for generating
> .sframe in kpatch is irrelevant because it is a no-op with the current setup.

Agreed, this issue should be irrelevant to the .sframe section in the .ko file.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

On Thu, Feb 13, 2025 at 2:22 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>
> Song Liu <song@kernel.org> writes:
>
> > On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <puranjay@kernel.org> wrote:
> > [...]
> >>
> >> P.S. - The livepatch doesn't have copy_process() but only copy_signal(),
> >> yours had copy_process() somehow.
> >
> > In my build, copy_signal is inlined to copy_process, unless I add noinline.
> > If I do add noinline, the issue will not reproduce.
> >
> > I tried more combinations. The issue doesn't reproduce if I either
> > 1) add noinline to copy_signal, so we are not patching the whole
> >    copy_process function;
> > or
> > 2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.
> >
> > So it appears something in gcc 14.2.1 is causing live patch to fail
> > for copy_process().
>
> So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we
> can be sure that it is not a sframe problem?
>
> And about having the .sframe section in the livepatch module, I realised
> that this set doesn't include support for reading/using sframe data from
> any module(livepatches included), so the patch I added for generating
> .sframe in kpatch is irrelevant because it is a no-op with the current setup.

Puranjay,

Could you please try the following?

1. Use gcc 11.4.1;
2. Add __always_inline to copy_signal();
3. Build kernel, and livepatch with the same test (we need to
    add __always_inline to the .patch file).
4. Run gdb livepatch-xxx.ko
5. In gdb do disassemble copy_process.

In my tests, both gcc-14.2.1 and gcc-11.5.0 generated a .ko file
that looks weird in gdb-disassemble. Specifically, readels shows
copy_process is about 5.5kB, but gdb-disassemble only shows
140 bytes or so for copy_process. clang doesn't seem to have
this problem.

I am really curious whether you have the same problem in your
setup.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan]

[-- Attachment #1: Type: text/plain, Size: 2367 bytes --]

Song Liu <song@kernel.org> writes:

> On Thu, Feb 13, 2025 at 2:22 PM Puranjay Mohan <puranjay@kernel.org> wrote:
>>
>> Song Liu <song@kernel.org> writes:
>>
>> > On Thu, Feb 13, 2025 at 12:38 AM Puranjay Mohan <puranjay@kernel.org> wrote:
>> > [...]
>> >>
>> >> P.S. - The livepatch doesn't have copy_process() but only copy_signal(),
>> >> yours had copy_process() somehow.
>> >
>> > In my build, copy_signal is inlined to copy_process, unless I add noinline.
>> > If I do add noinline, the issue will not reproduce.
>> >
>> > I tried more combinations. The issue doesn't reproduce if I either
>> > 1) add noinline to copy_signal, so we are not patching the whole
>> >    copy_process function;
>> > or
>> > 2) Switch compiler from gcc 14.2.1 to gcc 11.5.0.
>> >
>> > So it appears something in gcc 14.2.1 is causing live patch to fail
>> > for copy_process().
>>
>> So, can you test your RFC set (without SFRAME) with gcc 14.2.1, so we
>> can be sure that it is not a sframe problem?
>>
>> And about having the .sframe section in the livepatch module, I realised
>> that this set doesn't include support for reading/using sframe data from
>> any module(livepatches included), so the patch I added for generating
>> .sframe in kpatch is irrelevant because it is a no-op with the current setup.
>
> Puranjay,
>
> Could you please try the following?
>
> 1. Use gcc 11.4.1;
> 2. Add __always_inline to copy_signal();
> 3. Build kernel, and livepatch with the same test (we need to
>     add __always_inline to the .patch file).
> 4. Run gdb livepatch-xxx.ko
> 5. In gdb do disassemble copy_process.
>
> In my tests, both gcc-14.2.1 and gcc-11.5.0 generated a .ko file
> that looks weird in gdb-disassemble. Specifically, readels shows
> copy_process is about 5.5kB, but gdb-disassemble only shows
> 140 bytes or so for copy_process. clang doesn't seem to have
> this problem.
>
> I am really curious whether you have the same problem in your
> setup.

Hi Song,

I did this test and found the same issue as you (gdb assembly broken),
but I can see this issue even without the inlining. I think GDB tried to
load the debuginfo and that is somehow broken therefore it fails to
disassemblt properly.

But even with inlining, I couldn't see the warning about the refcount
with my setup.

Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Song Liu]

Hi Puranjay,

Thanks for running the tests.

On Fri, Feb 14, 2025 at 12:56 AM Puranjay Mohan <puranjay@kernel.org> wrote:
[...]
> >
> > I am really curious whether you have the same problem in your
> > setup.
>
> Hi Song,
>
> I did this test and found the same issue as you (gdb assembly broken),
> but I can see this issue even without the inlining. I think GDB tried to
> load the debuginfo and that is somehow broken therefore it fails to
> disassemblt properly.

Yes, this matches my observations: gcc-11 generates the .ko that
confuses gdb.

I tested with two versions of gdb (10.2 and 14.2), both have the
problem. OTOH, lldb is able to disassemble copy_process from a
gcc-compiled .ko file properly.

> But even with inlining, I couldn't see the warning about the refcount
> with my setup.

This also matches my observations. gcc-11 compiled livepatch
works fine.

Thanks,
Song

Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Josh Poimboeuf]

On Fri, Feb 14, 2025 at 08:56:45AM +0000, Puranjay Mohan wrote:
> I did this test and found the same issue as you (gdb assembly broken),
> but I can see this issue even without the inlining. I think GDB tried to
> load the debuginfo and that is somehow broken therefore it fails to
> disassemblt properly.

I had the same theory about the debuginfo, but I stripped debug sections
from that file and gdb still got confused.

Still, the symbol table looked normal, so the gdb issue might be
completely separate from the kernel runtime issues.

-- 
Josh