Re: [PATCH 0/8] unwind, arm64: add sframe unwinder for kernel

[Puranjay Mohan <puranjay@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 5939 bytes --]

Weinan Liu <wnliu@google.com> writes:

> This patchset implements a generic kernel sframe-based [1] unwinder.
> The main goal is to support reliable stacktraces on arm64.
>
> On x86 orc unwinder provides reliable stacktraces. But arm64 misses the
> required support from objtool: it cannot generate orc unwind tables for
> arm64.
>
> Currently, there's already a sframe unwinder proposed for userspace: [2].
> Since the sframe unwind table algorithm is similar, these two proposal
> could integrate common functionality in the future.
>
> There are some incomplete features or challenges:
>   - The unwinder doesn't yet work with kernel modules. The `start_addr` of
>     FRE from kernel modules doesn't appear correct, preventing us from
>     unwinding functions from kernel modules.
>   - Currently, only GCC supports sframe.
>
> Ref:
> [1]: https://sourceware.org/binutils/docs/sframe-spec.html
> [2]: https://lore.kernel.org/lkml/cover.1730150953.git.jpoimboe@kernel.org/
>

Hi Weinan,
Thanks for working on this.

I tested this set on my setup and faced some issues, here are the
details:

Here is my setup [on AWS c6gd.16xlarge instance]:
-------------------------------------------------

[root@ip-172-31-32-86 linux-upstream]# uname -a
Linux ip-172-31-32-86.ec2.internal 6.14.0-rc1+ #1 SMP Tue Feb  4 14:15:55 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

[root@ip-172-31-32-86 linux-upstream]# git log --oneline
e9a702365 (HEAD -> master) arm64: Enable livepatch for ARM64
5dedc956e arm64: Define TIF_PATCH_PENDING for livepatch
ba563b31a unwind: arm64: add reliable stacktrace support for arm64
d807d392d unwind: arm64: Add sframe unwinder on arm64
7872f050b unwind: Implement generic sframe unwinder library
03d2ad003 unwind: add sframe v2 header
5e95cc051 arm64: entry: add unwind info for various kernel entries
faff6cbc3 unwind: build kernel with sframe info
0de63bb7d (origin/master, origin/HEAD) Merge tag 'pull-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
902e09c8a fix braino in "9p: fix ->rename_sem exclusion"
f286757b6 Merge tag 'timers-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
a360f3ffd (grafted) Merge tag 'irq-urgent-2025-02-03' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
bb2784d9a (grafted) jiffies: Cast to unsigned long in secs_to_jiffies() conversion
30d61efe1 (grafted) 9p: fix ->rename_sem exclusion

[root@ip-172-31-32-86 linux-upstream]# grep SFRAME .config
CONFIG_AS_HAS_SFRAME_SUPPORT=y
CONFIG_SFRAME_UNWIND_TABLE=y
CONFIG_SFRAME_UNWINDER=y
[root@ip-172-31-32-86 linux-upstream]# grep LIVEPATCH .config
CONFIG_HAVE_LIVEPATCH=y
CONFIG_LIVEPATCH=y
CONFIG_SAMPLE_LIVEPATCH=m

[root@ip-172-31-32-86 linux-upstream]# as --version
GNU assembler version 2.41-50.al2023.0.2
Copyright (C) 2023 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or later.
This program has absolutely no warranty.
This assembler was configured for a target of `aarch64-amazon-linux'.

[root@ip-172-31-32-86 linux-upstream]# gcc --version
gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Loading the livepatch-sameple module:
-------------------------------------

[root@ip-172-31-32-86 linux-upstream]# kpatch load /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
loading patch module: /lib/modules/6.14.0-rc1+/kernel/samples/livepatch/livepatch-sample.ko
waiting (up to 15 seconds) for patch transition to complete...
patch transition has stalled!
<4>kpatch: Livepatch process signaling is performed automatically on your system.
<4>kpatch: Skipping manual process signaling.
waiting (up to 60 seconds) for patch transition to complete...

Stalled processes:
340 kdevtmpfs
stack:
[<0>] devtmpfs_work_loop+0x2cc/0x2d8
[<0>] devtmpfsd+0x4c/0x58
[<0>] kthread+0xf0/0x100
[<0>] ret_from_fork+0x10/0x20
module livepatch_sample did not complete its transition, unloading...
disabling patch module: livepatch_sample
waiting (up to 15 seconds) for patch transition to complete...
transition complete (3 seconds)
unloading patch module: livepatch_sample
<4>kpatch: error: failed to load module livepatch_sample (transition stalled)

Useful messages from kernel log [pr_debug enabled]:
---------------------------------------------------

 livepatch: enabling patch 'livepatch_sample'
 livepatch: 'livepatch_sample': initializing patching transition
 livepatch: 'livepatch_sample': starting patching transition
 livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
 livepatch: klp_try_switch_task: insmod:9226 has an unreliable stack
 livepatch: klp_try_switch_task: swapper/63:0 is running
 [......SNIP.......]
 livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
 [......SNIP.......]
 livepatch: signaling remaining tasks
 livepatch: klp_try_switch_task: kdevtmpfs:340 has an unreliable stack
 livepatch: 'livepatch_sample': reversing transition from patching to unpatching
 livepatch: 'livepatch_sample': starting unpatching transition
 livepatch: klp_try_switch_task: swapper/45:0 is running
 livepatch: 'livepatch_sample': completing unpatching transition
 livepatch: 'livepatch_sample': unpatching complete

Please let me know if you are aware of this already or if this is
expected behaviour with this version. I will try to debug this from my
side as well. Also let me know if you need more details for debugging
this.

P.S. - I also saw multiple build warning like:
ld: warning: orphan section `.eh_frame' from `arch/arm64/kernel/entry.o' being placed in section `.eh_frame'
ld: warning: orphan section `.init.sframe' from `arch/arm64/kernel/pi/lib-fdt.pi.o' being placed in section `.init.sframe'


Thanks,
Puranjay

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]