Linux Perf Users
 help / color / mirror / Atom feed
From: Namhyung Kim <namhyung@kernel.org>
To: Gennady Kupava <gennady.kupava@gmail.com>
Cc: linux-perf-users@vger.kernel.org
Subject: Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
Date: Thu, 27 Aug 2026 14:55:15 -0700	[thread overview]
Message-ID: <apCyQ93c48Dy4H-S@google.com> (raw)
In-Reply-To: <CAPu-DQqF0aF6=GS8Z6KKWeeX_V5LiXeKU_rJQZC+uGg8zuTPNw@mail.gmail.com>

Hello,

Thanks a lot for your detailed report!  It's good but let me jump into
the action items directly.

On Fri, Aug 21, 2026 at 10:49:59PM +0100, Gennady Kupava wrote:
[SNIP]
> Suggestions and questions
> =========================
> 
> 1. Kernel: the fix that was applied to the kernel-side call chain -
>    passing iregs rather than the modified regs - looks like it applies
>    verbatim to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER.  Should
>    those be derived from iregs too when the event requests a user stack
>    dump?  If the precise IP is worth keeping in the sample regardless,
>    should such samples carry a flag, so that userspace knows the IP and the
>    stack do not belong together?  Today nothing distinguishes them.

As Ravi said, you can check the misc field if the IP is from precise
events.

And there's PERF_SAMPLE_REGS_INTR which I believe captures registers
from the interrupt handler.  I don't remember why we have both REGS_USER
and REGS_INTR but it seems REGS_INTR would work for callchains.

> 
> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
>    hardware, so arguably perf should simply not let the two be combined.  I
>    would suggest two rules rather than one, because a blanket refusal would
>    make the common case worse:
> 
>    - for the default event, drop the P when --call-graph dwarf is requested,
>      silently and on PMUs where precision means IBS.  Otherwise a plain
>      "perf record -g --call-graph dwarf" starts failing outright on every AMD
>      box, which is worse than today.  The s390 case in evlist.c suggests this
>      kind of substitution is considered acceptable;
> 
>    - if the user asked for a precise event explicitly and also asked for
>      DWARF call chains, refuse with a message that says why and what to do,
>      instead of quietly producing a useless result.
> 
>    Note that this should be scoped to IBS.  On Intel, PEBS records the whole
>    register set, so precise events and DWARF unwinding work together there
>    and nothing needs restricting.  Frame-pointer call graphs are also fine
>    with precise events - only the leaf frame is off.

I think we can enable both precise IP and dwarf callchains by using
PERF_SAMPLE_REGS_INTR.  For unwinding, it should use PERF_REG_X86_IP
from the REGS_INTR instead of sample.ip

> 
> 3. perf: a single-frame result should probably not count as unwinder success
>    and suppress the fallback.  This one looks like a small, contained fix.

Sounds ok.

> 
> 4. perf: a way to choose the unwinder explicitly - an option or an
>    environment variable - would have saved most of this investigation.

It seems we have --unwind-style option in perf report but not in perf
annotate.  But we can try "unwind.style" config option.  Unfortunately
it doesn't seem to have documentation.

> 
> 5. libdw: the lld layout case is worth fixing, or at least worth recording
>    somewhere, given how much clang output it covers.

Have you checked the latest version?

> 
> 6. Documentation: perf-amd-ibs(1) and perf-record(1) could say that precise
>    events on AMD are incompatible with --call-graph dwarf.  A single sentence
>    would have saved a day here.

We can improve documentation always, but as I said I think we can make
them work.

[SNIP]
> == end of AI description
> 
> I hope it was an interesting read!
> Let me know if I could do anything here, I will be happy to help fixing
> these problems.  Hope this would help anybody, and looking for the feedback.

Thanks, it was a long but interesting read. :)

Sorry for your inconvenience you faced.  I haven't look at dwarf
unwinding for a while and thought libdw was good enough.  But supporting
multiple libraries for the same purpose is hard so we wanted to go with
libdw and deprecated libunwind.  I'm not sure if we need to revisit it.

Thanks,
Namhyung


  parent reply	other threads:[~2026-08-27 21:55 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-08-21 21:49 [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs Gennady Kupava
2026-08-24 13:35 ` Ravi Bangoria
2026-08-25 18:05 ` Ian Rogers
2026-08-27 21:55 ` Namhyung Kim [this message]
2026-08-28 16:47   ` Namhyung Kim
2026-08-28 16:58   ` Namhyung Kim
2026-08-31  5:05     ` Ravi Bangoria
2026-09-01 15:19       ` Namhyung Kim
2026-09-03  8:39         ` Mi, Dapeng
2026-09-03 11:36           ` Ravi Bangoria
2026-09-03 11:59             ` Mi, Dapeng
2026-09-03 16:02               ` Ravi Bangoria
2026-09-04  0:22                 ` Mi, Dapeng
2026-09-05 10:14                   ` Gennady Kupava

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=apCyQ93c48Dy4H-S@google.com \
    --to=namhyung@kernel.org \
    --cc=gennady.kupava@gmail.com \
    --cc=linux-perf-users@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox