Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

Song Liu
2025-02-12T23:32:52Z

I run some tests with this set and my RFC set [1]. Most of
the test is done with kpatch-build. I tested both Puranjay's
version [3] and my version [4].

For gcc 14.2.1, I have seen the following issue with this
test [2]. This happens with both upstream and 6.13.2.
The livepatch loaded fine, but the system spilled out the
following warning quickly.

On the other hand, the same test works with LLVM and
my RFC set (LLVM doesn't support SFRAME, and thus
doesn't work with this set yet).

Thanks,
Song

[ 81.250437] ------------[ cut here ]------------
[ 81.250818] refcount_t: saturated; leaking memory.
[ 81.251201] WARNING: CPU: 0 PID: 95 at lib/refcount.c:22
refcount_warn_saturate+0x6c/0x140
[ 81.251841] Modules linked in: livepatch_special_static(OEK)
[ 81.252277] CPU: 0 UID: 0 PID: 95 Comm: bash Tainted: G
OE K 6.13.2-00321-g52d2813b4b07 #49
[ 81.253003] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE, [K]=LIVEPATCH
[ 81.253503] Hardware name: linux,dummy-virt (DT)
[ 81.253856] pstate: 634000c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[ 81.254383] pc : refcount_warn_saturate+0x6c/0x140
[ 81.254748] lr : refcount_warn_saturate+0x6c/0x140
[ 81.255114] sp : ffff800085a6fc00
[ 81.255371] x29: ffff800085a6fc00 x28: 0000000001200000 x27: ffff0000c2966180
[ 81.255918] x26: 0000000000000000 x25: ffff8000829c0000 x24: ffff0000c2e9b608
[ 81.256462] x23: ffff800083351000 x22: ffff0000c2e9af80 x21: ffff0000c062e140
[ 81.257006] x20: ffff0000c1c10c00 x19: ffff800085a6fd80 x18: ffffffffffffffff
[ 81.257544] x17: 0000000000000001 x16: ffffffffffffffff x15: 0000000000000006
[ 81.258083] x14: 0000000000000000 x13: 2e79726f6d656d20 x12: 676e696b61656c20
[ 81.258625] x11: ffff8000829f7d70 x10: 0000000000000147 x9 : ffff8000801546b4
[ 81.259165] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
[ 81.259709] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
[ 81.260257] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
[ 81.260824] Call trace:
[ 81.261015] refcount_warn_saturate+0x6c/0x140 (P)
[ 81.261387] __refcount_add.constprop.0+0x60/0x70
[ 81.261748] copy_process+0xfdc/0xfd58 [livepatch_special_static]
[ 81.262217] kernel_clone+0x80/0x3e0
[ 81.262499] __do_sys_clone+0x5c/0x88
[ 81.262786] __arm64_sys_clone+0x24/0x38
[ 81.263085] invoke_syscall+0x4c/0x108
[ 81.263378] el0_svc_common.constprop.0+0x44/0xe8
[ 81.263734] do_el0_svc+0x20/0x30
[ 81.263993] el0_svc+0x34/0xf8
[ 81.264231] el0t_64_sync_handler+0x104/0x130
[ 81.264561] el0t_64_sync+0x184/0x188
[ 81.264846] ---[ end trace 0000000000000000 ]---
[ 82.335559] ------------[ cut here ]------------
[ 82.335931] refcount_t: underflow; use-after-free.
[ 82.336307] WARNING: CPU: 1 PID: 0 at lib/refcount.c:28
refcount_warn_saturate+0xec/0x140
[ 82.336949] Modules linked in: livepatch_special_static(OEK)
[ 82.337389] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G
W OE K 6.13.2-00321-g52d2813b4b07 #49
[ 82.338148] Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE,
[K]=LIVEPATCH
[ 82.338721] Hardware name: linux,dummy-virt (DT)
[ 82.339083] pstate: 63400005 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--)
[ 82.339617] pc : refcount_warn_saturate+0xec/0x140
[ 82.340007] lr : refcount_warn_saturate+0xec/0x140
[ 82.340378] sp : ffff80008370fe40
[ 82.340637] x29: ffff80008370fe40 x28: 0000000000000000 x27: 0000000000000000
[ 82.341188] x26: 000000000000000a x25: ffff0000fdaf7ab8 x24: 0000000000000014
[ 82.341737] x23: ffff8000829c8d30 x22: ffff80008370ff28 x21: ffff0000fe020000
[ 82.342286] x20: ffff0000c062e140 x19: ffff0000c2e9af80 x18: ffffffffffffffff
[ 82.342839] x17: ffff80007b7a0000 x16: ffff800083700000 x15: 0000000000000006
[ 82.343389] x14: 0000000000000000 x13: 2e656572662d7265 x12: 7466612d65737520
[ 82.343944] x11: ffff8000829f7d70 x10: 000000000000016a x9 : ffff8000801546b4
[ 82.344499] x8 : 00000000fffeffff x7 : 00000000ffff0000 x6 : ffff800082f77d70
[ 82.345051] x5 : 80000000ffff0000 x4 : 0000000000000000 x3 : 0000000000000001
[ 82.345604] x2 : ffff8000829f7a88 x1 : ffff8000829f7a88 x0 : 0000000000000026
[ 82.346163] Call trace:
[ 82.346359] refcount_warn_saturate+0xec/0x140 (P)
[ 82.346736] __put_task_struct+0x130/0x170
[ 82.347063] delayed_put_task_struct+0xbc/0xe8
[ 82.347411] rcu_core+0x20c/0x5f8
[ 82.347680] rcu_core_si+0x14/0x28
[ 82.347952] handle_softirqs+0x124/0x308
[ 82.348260] __do_softirq+0x18/0x20
[ 82.348536] ____do_softirq+0x14/0x28
[ 82.348828] call_on_irq_stack+0x24/0x30
[ 82.349137] do_softirq_own_stack+0x20/0x38
[ 82.349465] __irq_exit_rcu+0xcc/0x108
[ 82.349764] irq_exit_rcu+0x14/0x28
[ 82.350038] el1_interrupt+0x34/0x50
[ 82.350321] el1h_64_irq_handler+0x14/0x20
[ 82.350642] el1h_64_irq+0x6c/0x70
[ 82.350911] default_idle_call+0x30/0xd0 (P)
[ 82.351248] do_idle+0x1d0/0x200
[ 82.351506] cpu_startup_entry+0x38/0x48
[ 82.351818] secondary_start_kernel+0x124/0x150
[ 82.352176] __secondary_switched+0xac/0xb0
[ 82.352505] ---[ end trace 0000000000000000 ]---

[1] SFRAME-less livepatch RFC
https://lore.kernel.org/live-patching/20250129232936.1795412-1-song@kernel.org/
[2] special-static test from kpatch
https://github.com/dynup/kpatch/blob/master/test/integration/linux-6.2.0/special-static.patch
[3] Puranjay's kpatch with arm64 support
https://github.com/puranjaymohan/kpatch/tree/arm64
[4] My version of kpatch with arm64 and LTO support
https://github.com/liu-song-6/kpatch/tree/fb-6.13-v2

On Mon, Jan 27, 2025 at 1:33 PM Weinan Liu <wnliu@google.com> wrote:
>
> This patchset implements a generic kernel sframe-based [1] unwinder.
> The main goal is to support reliable stacktraces on arm64.
>
> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
> required support from objtool: it cannot generate orc unwind tables for
> arm64.
>
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> Since the sframe unwind table algorithm is similar, these two proposal
> could integrate common functionality in the future.
>
> There are some incomplete features or challenges:
> - The unwinder doesn't yet work with kernel modules. The `start_addr` of
> FRE from kernel modules doesn't appear correct, preventing us from
> unwinding functions from kernel modules.
> - Currently, only GCC supports sframe.
>
> Ref:
> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>
> Madhavan T. Venkataraman (1):
> arm64: Define TIF_PATCH_PENDING for livepatch
>
> Weinan Liu (7):
> unwind: build kernel with sframe info
> arm64: entry: add unwind info for various kernel entries
> unwind: add sframe v2 header
> unwind: Implement generic sframe unwinder library
> unwind: arm64: Add sframe unwinder on arm64
> unwind: arm64: add reliable stacktrace support for arm64
> arm64: Enable livepatch for ARM64
>
> Makefile | 6 +
> arch/Kconfig | 8 +
> arch/arm64/Kconfig | 3 +
> arch/arm64/Kconfig.debug | 10 +
> arch/arm64/include/asm/stacktrace/common.h | 6 +
> arch/arm64/include/asm/thread_info.h | 4 +-
> arch/arm64/kernel/entry-common.c | 4 +
> arch/arm64/kernel/entry.S | 10 +
> arch/arm64/kernel/setup.c | 2 +
> arch/arm64/kernel/stacktrace.c | 102 ++++++++++
> include/asm-generic/vmlinux.lds.h | 12 ++
> include/linux/sframe_lookup.h | 43 +++++
> kernel/Makefile | 1 +
> kernel/sframe.h | 215 +++++++++++++++++++++
> kernel/sframe_lookup.c | 196 +++++++++++++++++++
> 15 files changed, 621 insertions(+), 1 deletion(-)
> create mode 100644 include/linux/sframe_lookup.h
> create mode 100644 kernel/sframe.h
> create mode 100644 kernel/sframe_lookup.c
>
> --
> 2.48.1.262.g85cc9f2d1e-goog
>
>