s390x stack unwinding with perf?

s390x stack unwinding with perf?

[Daan De Meyer]

Hi,

As part of enabling frame pointers system wide for Fedora to get fast
system wide stack unwinding, we're now looking to do the same for
s390x after previously enabling frame pointers for x86-64 and aarch64.
From what we can find, on s390x stack unwinding is done via the
"-mbackchain" compiler option. However, from some early experiments
rebuilding a small part of the distro, this option alone does not seem
to result in usable stack traces in perf (perf record -g => perf
report).

Are there any other compiler options that need to be enabled in order
to get usable stack traces in perf for s390x? Has anyone managed to
get this working before?

Cheers,

Daan De Meyer

Re: s390x stack unwinding with perf?

[Heiko Carstens]

On Thu, Oct 26, 2023 at 02:30:41PM +0200, Daan De Meyer wrote:
> Hi,
> 
> As part of enabling frame pointers system wide for Fedora to get fast
> system wide stack unwinding, we're now looking to do the same for
> s390x after previously enabling frame pointers for x86-64 and aarch64.
> From what we can find, on s390x stack unwinding is done via the
> "-mbackchain" compiler option. However, from some early experiments
> rebuilding a small part of the distro, this option alone does not seem
> to result in usable stack traces in perf (perf record -g => perf
> report).
> 
> Are there any other compiler options that need to be enabled in order
> to get usable stack traces in perf for s390x? Has anyone managed to
> get this working before?

So, are you saying that you want to enable "-mbackchain" for whole user
space for Fedora? If that is the case, what certainly is missing is kernel
support to unwind user space stack frames from the kernel, since until now
no distribution had user space where it would be possible to unwind the
user space stack from the kernel; which is the reason why this was never
implemented.

If this (or some other compiler options) will be enabled for user space, we
need to add kernel support in order to make this work.

Re: s390x stack unwinding with perf?

[Neal Gompa]

On Thu, Oct 26, 2023 at 10:57 AM Heiko Carstens <hca@linux.ibm.com> wrote:
>
> On Thu, Oct 26, 2023 at 02:30:41PM +0200, Daan De Meyer wrote:
> > Hi,
> >
> > As part of enabling frame pointers system wide for Fedora to get fast
> > system wide stack unwinding, we're now looking to do the same for
> > s390x after previously enabling frame pointers for x86-64 and aarch64.
> > From what we can find, on s390x stack unwinding is done via the
> > "-mbackchain" compiler option. However, from some early experiments
> > rebuilding a small part of the distro, this option alone does not seem
> > to result in usable stack traces in perf (perf record -g => perf
> > report).
> >
> > Are there any other compiler options that need to be enabled in order
> > to get usable stack traces in perf for s390x? Has anyone managed to
> > get this working before?
>
> So, are you saying that you want to enable "-mbackchain" for whole user
> space for Fedora? If that is the case, what certainly is missing is kernel
> support to unwind user space stack frames from the kernel, since until now
> no distribution had user space where it would be possible to unwind the
> user space stack from the kernel; which is the reason why this was never
> implemented.
>
> If this (or some other compiler options) will be enabled for user space, we
> need to add kernel support in order to make this work.

Yes, the idea is that we'd want to have this supported in Fedora to
match all the other architectures where "-fno-omit-frame-pointer" is
used.



--
Neal Gompa (FAS: ngompa)

Re: s390x stack unwinding with perf?

[Daan De Meyer]

> > Hi,
> >
> > As part of enabling frame pointers system wide for Fedora to get fast
> > system wide stack unwinding, we're now looking to do the same for
> > s390x after previously enabling frame pointers for x86-64 and aarch64.
> > From what we can find, on s390x stack unwinding is done via the
> > "-mbackchain" compiler option. However, from some early experiments
> > rebuilding a small part of the distro, this option alone does not seem
> > to result in usable stack traces in perf (perf record -g => perf
> > report).
> >
> > Are there any other compiler options that need to be enabled in order
> > to get usable stack traces in perf for s390x? Has anyone managed to
> > get this working before?
>
> So, are you saying that you want to enable "-mbackchain" for whole user
> space for Fedora? If that is the case, what certainly is missing is kernel
> support to unwind user space stack frames from the kernel, since until now
> no distribution had user space where it would be possible to unwind the
> user space stack from the kernel; which is the reason why this was never
> implemented.
>
> If this (or some other compiler options) will be enabled for user space, we
> need to add kernel support in order to make this work.

If the kernel gets support for s390x user space unwinding using the backchain,
we'll propose to enable -mbackchain in the default compilation flags for Fedora
so that s390x on Fedora will have the same profiling experience as x86-64, arm64
and ppc64. For now we'll keep the status quo since compiling with the backchain
doesn't provide any benefit until the kernel unwinder can unwind user
space stacks
using it.

Thanks for clarifying the current state of user space stack unwinding on s390x!

Cheers,

Daan





On Thu, 26 Oct 2023 at 16:57, Heiko Carstens <hca@linux.ibm.com> wrote:
>
> On Thu, Oct 26, 2023 at 02:30:41PM +0200, Daan De Meyer wrote:
> > Hi,
> >
> > As part of enabling frame pointers system wide for Fedora to get fast
> > system wide stack unwinding, we're now looking to do the same for
> > s390x after previously enabling frame pointers for x86-64 and aarch64.
> > From what we can find, on s390x stack unwinding is done via the
> > "-mbackchain" compiler option. However, from some early experiments
> > rebuilding a small part of the distro, this option alone does not seem
> > to result in usable stack traces in perf (perf record -g => perf
> > report).
> >
> > Are there any other compiler options that need to be enabled in order
> > to get usable stack traces in perf for s390x? Has anyone managed to
> > get this working before?
>
> So, are you saying that you want to enable "-mbackchain" for whole user
> space for Fedora? If that is the case, what certainly is missing is kernel
> support to unwind user space stack frames from the kernel, since until now
> no distribution had user space where it would be possible to unwind the
> user space stack from the kernel; which is the reason why this was never
> implemented.
>
> If this (or some other compiler options) will be enabled for user space, we
> need to add kernel support in order to make this work.

Re: s390x stack unwinding with perf?

[Heiko Carstens]

On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> 
> If the kernel gets support for s390x user space unwinding using the backchain,
> we'll propose to enable -mbackchain in the default compilation flags for Fedora
> so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> and ppc64. For now we'll keep the status quo since compiling with the backchain
> doesn't provide any benefit until the kernel unwinder can unwind user
> space stacks
> using it.
> 
> Thanks for clarifying the current state of user space stack unwinding on s390x!

We will implement the missing pieces and let you know when things are
supposed to work.

Re: s390x stack unwinding with perf?

[Neal Gompa]

On Fri, Oct 27, 2023 at 6:10 AM Heiko Carstens <hca@linux.ibm.com> wrote:
>
> On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> >
> > If the kernel gets support for s390x user space unwinding using the backchain,
> > we'll propose to enable -mbackchain in the default compilation flags for Fedora
> > so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> > and ppc64. For now we'll keep the status quo since compiling with the backchain
> > doesn't provide any benefit until the kernel unwinder can unwind user
> > space stacks
> > using it.
> >
> > Thanks for clarifying the current state of user space stack unwinding on s390x!
>
> We will implement the missing pieces and let you know when things are
> supposed to work.

Do you think we could have an initial patch set for implementing the
missing pieces in time for the Linux 6.8 merge window? Then we can
look at enabling this for s390x as a Fedora Linux 40 Change.




--
Neal Gompa (FAS: ngompa)

Re: s390x stack unwinding with perf?

[Heiko Carstens]

On Fri, Oct 27, 2023 at 11:22:42AM -0400, Neal Gompa wrote:
> On Fri, Oct 27, 2023 at 6:10 AM Heiko Carstens <hca@linux.ibm.com> wrote:
> >
> > On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> > >
> > > If the kernel gets support for s390x user space unwinding using the backchain,
> > > we'll propose to enable -mbackchain in the default compilation flags for Fedora
> > > so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> > > and ppc64. For now we'll keep the status quo since compiling with the backchain
> > > doesn't provide any benefit until the kernel unwinder can unwind user
> > > space stacks
> > > using it.
> > >
> > > Thanks for clarifying the current state of user space stack unwinding on s390x!
> >
> > We will implement the missing pieces and let you know when things are
> > supposed to work.
> 
> Do you think we could have an initial patch set for implementing the
> missing pieces in time for the Linux 6.8 merge window? Then we can
> look at enabling this for s390x as a Fedora Linux 40 Change.

This will be very likely the case. Actually the plan is to go with the
patch below. I gave it some testing with Fedora 38 and replaced (only)
glibc with a variant that was compiled with -mbackchain.

It seems to work, including the not avoidable problems described below. So
I guess we will try to bring this upstream with the current merge window.

From 3ba1e255c8af61f172a8b31de2b76d39c7d4c0c7 Mon Sep 17 00:00:00 2001
From: Heiko Carstens <hca@linux.ibm.com>
Date: Mon, 30 Oct 2023 11:34:07 +0100
Subject: [PATCH] s390/perf: implement perf_callchain_user()

Daan De Meyer and Neal Gompa reported that s390 does not support perf user
stack unwinding.

This was never implemented since this requires user space to be compiled
with the -mbackchain compile option, which until now no distribution
did. However this is going to change with Fedora. Therefore provide a
perf_callchain_user() implementation.

Note that due to the way s390 sets up stack frames the provided call chains
can contain invalid values. This is especially true for the first stack
frame, where it is not possible to tell if the return address has been
written to the stack already or not.

Reported-by: Daan De Meyer <daan.j.demeyer@gmail.com>
Reported-by: Neal Gompa <ngompa@fedoraproject.org>
Closes: https://lore.kernel.org/all/CAO8sHcn3+_qrnvp0580aK7jN0Wion5F7KYeBAa4MnCY4mqABPA@mail.gmail.com/
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
---
 arch/s390/kernel/perf_event.c | 47 +++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/arch/s390/kernel/perf_event.c b/arch/s390/kernel/perf_event.c
index c27321cb0969..fdbe7e1d9eea 100644
--- a/arch/s390/kernel/perf_event.c
+++ b/arch/s390/kernel/perf_event.c
@@ -212,6 +212,53 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
 	}
 }
 
+struct stack_frame_user {
+	unsigned long back_chain;
+	unsigned long empty1[5];
+	unsigned long gprs[10];
+	unsigned long empty2[4];
+};
+
+void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
+			 struct pt_regs *regs)
+{
+	struct stack_frame_user __user *sf;
+	unsigned long ip, sp;
+	bool first = true;
+
+	if (perf_guest_state())
+		return;
+	if (is_compat_task())
+		return;
+	perf_callchain_store(entry, instruction_pointer(regs));
+	sf = (void *)user_stack_pointer(regs);
+	pagefault_disable();
+	while (entry->nr < entry->max_stack) {
+		if (__get_user(sp, &sf->back_chain))
+			break;
+		if (__get_user(ip, &sf->gprs[8]))
+			break;
+		if (ip & 0x1) {
+			/*
+			 * If the instruction address is invalid, and this
+			 * is the first stack frame, assume r14 has not
+			 * been written to the stack yet. Otherwise exit.
+			 */
+			if (first && !(regs->gprs[14] & 0x1))
+				ip = regs->gprs[14];
+			else
+				break;
+		}
+		perf_callchain_store(entry, ip);
+		/* Sanity check: ABI requires SP to be aligned 8 bytes. */
+		if (!sp || sp & 0x7)
+			break;
+		sf = (void __user *)sp;
+		first = false;
+	}
+	pagefault_enable();
+}
+
 /* Perf definitions for PMU event attributes in sysfs */
 ssize_t cpumf_events_sysfs_show(struct device *dev,
 				struct device_attribute *attr, char *page)
-- 
2.39.2

Re: s390x stack unwinding with perf?

[Neal Gompa]

On Mon, Oct 30, 2023 at 9:02 AM Heiko Carstens <hca@linux.ibm.com> wrote:
>
> On Fri, Oct 27, 2023 at 11:22:42AM -0400, Neal Gompa wrote:
> > On Fri, Oct 27, 2023 at 6:10 AM Heiko Carstens <hca@linux.ibm.com> wrote:
> > >
> > > On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> > > >
> > > > If the kernel gets support for s390x user space unwinding using the backchain,
> > > > we'll propose to enable -mbackchain in the default compilation flags for Fedora
> > > > so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> > > > and ppc64. For now we'll keep the status quo since compiling with the backchain
> > > > doesn't provide any benefit until the kernel unwinder can unwind user
> > > > space stacks
> > > > using it.
> > > >
> > > > Thanks for clarifying the current state of user space stack unwinding on s390x!
> > >
> > > We will implement the missing pieces and let you know when things are
> > > supposed to work.
> >
> > Do you think we could have an initial patch set for implementing the
> > missing pieces in time for the Linux 6.8 merge window? Then we can
> > look at enabling this for s390x as a Fedora Linux 40 Change.
>
> This will be very likely the case. Actually the plan is to go with the
> patch below. I gave it some testing with Fedora 38 and replaced (only)
> glibc with a variant that was compiled with -mbackchain.
>
> It seems to work, including the not avoidable problems described below. So
> I guess we will try to bring this upstream with the current merge window.
>
> From 3ba1e255c8af61f172a8b31de2b76d39c7d4c0c7 Mon Sep 17 00:00:00 2001
> From: Heiko Carstens <hca@linux.ibm.com>
> Date: Mon, 30 Oct 2023 11:34:07 +0100
> Subject: [PATCH] s390/perf: implement perf_callchain_user()
>
> Daan De Meyer and Neal Gompa reported that s390 does not support perf user
> stack unwinding.
>
> This was never implemented since this requires user space to be compiled
> with the -mbackchain compile option, which until now no distribution
> did. However this is going to change with Fedora. Therefore provide a
> perf_callchain_user() implementation.
>
> Note that due to the way s390 sets up stack frames the provided call chains
> can contain invalid values. This is especially true for the first stack
> frame, where it is not possible to tell if the return address has been
> written to the stack already or not.
>
> Reported-by: Daan De Meyer <daan.j.demeyer@gmail.com>
> Reported-by: Neal Gompa <ngompa@fedoraproject.org>
> Closes: https://lore.kernel.org/all/CAO8sHcn3+_qrnvp0580aK7jN0Wion5F7KYeBAa4MnCY4mqABPA@mail.gmail.com/
> Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
> ---
>  arch/s390/kernel/perf_event.c | 47 +++++++++++++++++++++++++++++++++++
>  1 file changed, 47 insertions(+)
>
> diff --git a/arch/s390/kernel/perf_event.c b/arch/s390/kernel/perf_event.c
> index c27321cb0969..fdbe7e1d9eea 100644
> --- a/arch/s390/kernel/perf_event.c
> +++ b/arch/s390/kernel/perf_event.c
> @@ -212,6 +212,53 @@ void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,
>         }
>  }
>
> +struct stack_frame_user {
> +       unsigned long back_chain;
> +       unsigned long empty1[5];
> +       unsigned long gprs[10];
> +       unsigned long empty2[4];
> +};
> +
> +void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
> +                        struct pt_regs *regs)
> +{
> +       struct stack_frame_user __user *sf;
> +       unsigned long ip, sp;
> +       bool first = true;
> +
> +       if (perf_guest_state())
> +               return;
> +       if (is_compat_task())
> +               return;
> +       perf_callchain_store(entry, instruction_pointer(regs));
> +       sf = (void *)user_stack_pointer(regs);
> +       pagefault_disable();
> +       while (entry->nr < entry->max_stack) {
> +               if (__get_user(sp, &sf->back_chain))
> +                       break;
> +               if (__get_user(ip, &sf->gprs[8]))
> +                       break;
> +               if (ip & 0x1) {
> +                       /*
> +                        * If the instruction address is invalid, and this
> +                        * is the first stack frame, assume r14 has not
> +                        * been written to the stack yet. Otherwise exit.
> +                        */
> +                       if (first && !(regs->gprs[14] & 0x1))
> +                               ip = regs->gprs[14];
> +                       else
> +                               break;
> +               }
> +               perf_callchain_store(entry, ip);
> +               /* Sanity check: ABI requires SP to be aligned 8 bytes. */
> +               if (!sp || sp & 0x7)
> +                       break;
> +               sf = (void __user *)sp;
> +               first = false;
> +       }
> +       pagefault_enable();
> +}
> +
>  /* Perf definitions for PMU event attributes in sysfs */
>  ssize_t cpumf_events_sysfs_show(struct device *dev,
>                                 struct device_attribute *attr, char *page)
> --
> 2.39.2
>

This patch LGTM. I'd love to see it land in Linux 6.7!

Reviewed-by: Neal Gompa <ngompa@fedoraproject.org>



--
Neal Gompa (FAS: ngompa)

Re: s390x stack unwinding with perf?

[Heiko Carstens]

On Mon, Oct 30, 2023 at 09:19:02AM -0400, Neal Gompa wrote:
> On Mon, Oct 30, 2023 at 9:02 AM Heiko Carstens <hca@linux.ibm.com> wrote:
> >
> > On Fri, Oct 27, 2023 at 11:22:42AM -0400, Neal Gompa wrote:
> > > On Fri, Oct 27, 2023 at 6:10 AM Heiko Carstens <hca@linux.ibm.com> wrote:
> > > >
> > > > On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> > > > >
> > > > > If the kernel gets support for s390x user space unwinding using the backchain,
> > > > > we'll propose to enable -mbackchain in the default compilation flags for Fedora
> > > > > so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> > > > > and ppc64. For now we'll keep the status quo since compiling with the backchain
> > > > > doesn't provide any benefit until the kernel unwinder can unwind user
> > > > > space stacks
> > > > > using it.
> > > > >
> > > > > Thanks for clarifying the current state of user space stack unwinding on s390x!
> > > >
> > > > We will implement the missing pieces and let you know when things are
> > > > supposed to work.
> > >
> > > Do you think we could have an initial patch set for implementing the
> > > missing pieces in time for the Linux 6.8 merge window? Then we can
> > > look at enabling this for s390x as a Fedora Linux 40 Change.
> >
> > This will be very likely the case. Actually the plan is to go with the
> > patch below. I gave it some testing with Fedora 38 and replaced (only)
> > glibc with a variant that was compiled with -mbackchain.
...
> This patch LGTM. I'd love to see it land in Linux 6.7!
> 
> Reviewed-by: Neal Gompa <ngompa@fedoraproject.org>

FWIW, this is now upstream and will land in 6.7, together with a similar
patch which adds user stacktrace support:

504b73d00a55 ("s390/perf: implement perf_callchain_user()")
aa44433ac4ee ("s390: add USER_STACKTRACE support")

Please let us know if there are any problems.

Re: s390x stack unwinding with perf?

[Daan De Meyer]

> FWIW, this is now upstream and will land in 6.7, together with a similar
> patch which adds user stacktrace support:

> 504b73d00a55 ("s390/perf: implement perf_callchain_user()")
> aa44433ac4ee ("s390: add USER_STACKTRACE support")

> Please let us know if there are any problems.

I've been trying out the new userspace stacktrace support on Fedora
Rawhide but I'm having some problems getting it to work. I rebuilt
systemd and all of its dependencies (specifically, glibc) with
-mbackchain using copr
(https://copr.fedorainfracloud.org/coprs/daandemeyer/fno-omit-frame-pointer/builds/),
built a Fedora Rawhide image with the rebuilt packages and perf using
mkosi, started an s390x virtual machine and ran "perf record -g
journalctl --verify" followed by "perf report -g 'graph,0.5,caller'".
On x86, this gives me a sensible perf report. On s390x, it seems a lot
of information is still missing.

uname -r: 6.7.0-0.rc6.48.fc40.s390x
cat /usr/lib/modules/6.7.0-0.rc6.48.fc40.s390x/config | grep STACK:
"""
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_STACKLEAK=y
CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK=y
CONFIG_SOFTIRQ_ON_OWN_STACK=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y
CONFIG_RANDOMIZE_KSTACK_OFFSET=y
CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y
CONFIG_BLK_MQ_STACKING=y
CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
CONFIG_NET_EMATCH_STACK=32
# CONFIG_INIT_STACK_NONE is not set
# CONFIG_INIT_STACK_ALL_PATTERN is not set
CONFIG_INIT_STACK_ALL_ZERO=y
CONFIG_ARCH_STACKWALK=y
CONFIG_STACKDEPOT=y
# CONFIG_STACKTRACE_BUILD_ID is not set
# CONFIG_DEBUG_STACK_USAGE is not set
CONFIG_SCHED_STACK_END_CHECK=y
CONFIG_STACKTRACE=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_STACK_TRACER=y
CONFIG_STACKINIT_KUNIT_TEST=m
"""

To reproduce (from a s390x host system):

- git clone https://www.github.com/systemd/mkosi
- cd mkosi
- tee copr.repo <<EOF
[copr:copr.fedorainfracloud.org:daandemeyer:fno-omit-frame-pointer]
name=Copr repo for fno-omit-frame-pointer owned by daandemeyer
baseurl=https://download.copr.fedorainfracloud.org/results/daandemeyer/fno-omit-frame-pointer/fedora-$releasever-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/daandemeyer/fno-omit-frame-pointer/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1
priority=1
EOF
- tee mkosi.local.conf <<EOF
[Distribution]
Distribution=fedora
Release=rawhide
Repositories=fedora-debuginfo
PackageManagerTrees=copr.repo:/etc/yum.repos.d/copr.repo

[Content]
Packages=
        systemd-debuginfo
        systemd-debugsource

[Host]
ToolsTree=default
Incremental=yes
EOF
- bin/mkosi -f qemu"
# You'll end up in a virtual machine running Fedora Rawhide with a
6.7-rc kernel and glibc + systemd built with -mbackchain
- perf record -g journalctl --verify
- perf report -g 'graph,0.5,caller'
# The output will not make a lot of sense whereas it does on x86


On Thu, 9 Nov 2023 at 15:48, Heiko Carstens <hca@linux.ibm.com> wrote:
>
> On Mon, Oct 30, 2023 at 09:19:02AM -0400, Neal Gompa wrote:
> > On Mon, Oct 30, 2023 at 9:02 AM Heiko Carstens <hca@linux.ibm.com> wrote:
> > >
> > > On Fri, Oct 27, 2023 at 11:22:42AM -0400, Neal Gompa wrote:
> > > > On Fri, Oct 27, 2023 at 6:10 AM Heiko Carstens <hca@linux.ibm.com> wrote:
> > > > >
> > > > > On Fri, Oct 27, 2023 at 10:00:53AM +0200, Daan De Meyer wrote:
> > > > > >
> > > > > > If the kernel gets support for s390x user space unwinding using the backchain,
> > > > > > we'll propose to enable -mbackchain in the default compilation flags for Fedora
> > > > > > so that s390x on Fedora will have the same profiling experience as x86-64, arm64
> > > > > > and ppc64. For now we'll keep the status quo since compiling with the backchain
> > > > > > doesn't provide any benefit until the kernel unwinder can unwind user
> > > > > > space stacks
> > > > > > using it.
> > > > > >
> > > > > > Thanks for clarifying the current state of user space stack unwinding on s390x!
> > > > >
> > > > > We will implement the missing pieces and let you know when things are
> > > > > supposed to work.
> > > >
> > > > Do you think we could have an initial patch set for implementing the
> > > > missing pieces in time for the Linux 6.8 merge window? Then we can
> > > > look at enabling this for s390x as a Fedora Linux 40 Change.
> > >
> > > This will be very likely the case. Actually the plan is to go with the
> > > patch below. I gave it some testing with Fedora 38 and replaced (only)
> > > glibc with a variant that was compiled with -mbackchain.
> ...
> > This patch LGTM. I'd love to see it land in Linux 6.7!
> >
> > Reviewed-by: Neal Gompa <ngompa@fedoraproject.org>
>
> FWIW, this is now upstream and will land in 6.7, together with a similar
> patch which adds user stacktrace support:
>
> 504b73d00a55 ("s390/perf: implement perf_callchain_user()")
> aa44433ac4ee ("s390: add USER_STACKTRACE support")
>
> Please let us know if there are any problems.

Re: s390x stack unwinding with perf?

[Heiko Carstens]

On Wed, Dec 20, 2023 at 01:58:36PM +0100, Daan De Meyer wrote:
> > FWIW, this is now upstream and will land in 6.7, together with a similar
> > patch which adds user stacktrace support:
> 
> > 504b73d00a55 ("s390/perf: implement perf_callchain_user()")
> > aa44433ac4ee ("s390: add USER_STACKTRACE support")
> 
> > Please let us know if there are any problems.
> 
> I've been trying out the new userspace stacktrace support on Fedora
> Rawhide but I'm having some problems getting it to work. I rebuilt
> systemd and all of its dependencies (specifically, glibc) with
> -mbackchain using copr
> (https://copr.fedorainfracloud.org/coprs/daandemeyer/fno-omit-frame-pointer/builds/),
> built a Fedora Rawhide image with the rebuilt packages and perf using
> mkosi, started an s390x virtual machine and ran "perf record -g
> journalctl --verify" followed by "perf report -g 'graph,0.5,caller'".
> On x86, this gives me a sensible perf report. On s390x, it seems a lot
> of information is still missing.
...
> To reproduce (from a s390x host system):
...
> - bin/mkosi -f qemu"

This fails with "bootctl not found".

Looking at the binaries in the copr repo it looks like the generated
code does create backchains. So in theory walking backchains should
work.

Could you please give examples what does not work, and what you would
expect? Right now I don't know what to look at.