* [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
@ 2026-08-21 21:49 Gennady Kupava
2026-08-24 13:35 ` Ravi Bangoria
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Gennady Kupava @ 2026-08-21 21:49 UTC (permalink / raw)
To: linux-perf-users; +Cc: Gennady Kupava
Hello,
I decided to share a sort of horror adventure: my attempt to use a recent
perf with call graphs on recent hardware. I think this story is useful both
to the developers of the tool - there are small bugs to fix here, and a
user experience worth assessing - and to users, who I am pretty sure could
face the same issues. Other than that, I have a feeling this might qualify
as an interesting adventure to follow.
So my idea was to work on optimization of an open-source mapping project
(OsmAnd), for which I am trying to prepare a series of patches - some of
them have landed already, others are still in my head.
I have 15 years of experience using perf-like tools periodically; it was
oprofile before perf, and the first tool of this kind that impressed me a
lot.
So the task here was to take OsmAnd's core library, which is typically built
for Android, build it for amd64, and measure its performance with call
graphs. I am on Debian unstable, very up to date. The idea of this mail is
to explain the pile of problems I faced during this adventure.
The Linux distribution is important, as Debian did not enable frame
pointers - so it is not possible to "just enable FP", call stacks have
to be done using stack dumps.
So I got everything built and ran it, and tried to run perf on it, which
succeeded, but... it quickly turned out that the stack frames were not
complete at all: there was only 2% of callgraphs with deep chain...
And so, together with an AI, I started figuring out what was going on - and
this is where the adventure begins.
Attempt number 1: kernel paranoid. The first suspicious thing found was
kernel.perf_event_paranoid, which is 1 on Debian. I set it to -1, and the
profile did seem to improve a lot - the share of samples carrying a deep
chain went from under 2% to around 44%, which felt like the answer. It was
not. Later it turned out this change did not change anything.
Attempt number 2: the lld issue. Claude found a mail thread on this very
list - "Call graph dwarf unwinding fails with lld", May 2022,
https://www.spinics.net/lists/linux-perf-users/msg19574.html - where it
seemed clear that lld builds are worse than ld ones, so that was the next
candidate to try. So I tried, and indeed, using ld as the linker made the
situation better, but... somehow it still did not work. A few hours of
AI-augmented debugging later there were fewer frames lost, but they were
still lost.
Identified problem number 1: it turned out that the default counters on my
AMD CPU make it impossible to dump the stack and the IP from the same place.
Identified problem number 2: even if the stack dump is correct, libdw cannot
process it, only libunwind can.
Identified problem number 3: even if I compile perf with libunwind, it still
tries to use libdw - and that fails while perf thinks it succeeded.
After identifying all 3 problems, I was able to generate quality stack
traces.
However, it took me half of the day to reach that point, and it is hard to
imagine how people in general are able to use perf - this was way too much
effort just to record call graphs...
From this point I will let the AI explain each of the problems I faced, with
precise technical details (i reviewed it all and it makes sense to me):
The setup
=========
CPU AMD Ryzen 9 9950X3D, family 0x1a, model 0x44, stepping 0,
microcode 0xb404038; ibs_op and ibs_fetch PMUs present
kernel 6.18.5+deb14-amd64 (Debian 6.18.5-1) - all measurements
source linux-source 7.1.8, used for the quotes below; the code there is
unchanged, so none of this is fixed in a newer tree
perf 7.1.8, both the Debian build and a local build
Two workloads appear below. One is a self-contained C reproducer, given at
the end. The other is the real thing: a map tile rasterizer built on Qt5 and
Skia, with deep chains crossing several shared libraries.
The metric throughout is the share of samples whose call chain came back with
a single frame, plus the average chain depth. The counting script is at the
end too. All recordings use --call-graph dwarf,65528.
Problem 1: IP and stack dump mismatch
=====================================
The IP and the stack in one sample describe different moments.
perf record's default event carries the P modifier, so on AMD it is served by
IBS. How one gets there without asking is covered further down.
perf_ibs_handle_irq() copies the whole register set from the NMI frame and
then overrides only the instruction pointer with the one recorded by IBS:
arch/x86/events/amd/ibs.c, perf_ibs_handle_irq()
regs = *iregs;
if (check_rip && (ibs_data.regs[2] & IBS_RIP_INVALID)) {
regs.flags &= ~PERF_EFLAGS_EXACT;
} else {
...
set_linear_ip(®s, ibs_data.regs[1]); /* IbsOpRip */
regs.flags |= PERF_EFLAGS_EXACT;
}
That same pt_regs is what both PERF_SAMPLE_REGS_USER and
PERF_SAMPLE_STACK_USER are derived from:
kernel/events/core.c, perf_sample_regs_user()
if (user_mode(regs)) {
regs_user->abi = perf_reg_abi(current);
regs_user->regs = regs;
kernel/events/core.c, perf_output_sample()
perf_output_sample_ustack(handle, data->stack_user_size,
data->regs_user.regs);
kernel/events/core.c, perf_output_sample_ustack()
sp = perf_user_stack_pointer(regs);
So the IP names the instruction IBS tagged, while SP - and therefore every
byte of the dumped stack - describes wherever the CPU was when the NMI
finally arrived. These are not the same place: in between, the tagged call
has been taken and the callee has built its frame.
Frame-pointer unwinding barely notices. It still returns a full chain, only
with a top frame from a slightly different moment. DWARF unwinding cannot
survive it, because the CFA rule is chosen by IP and then applied to a stack
belonging to somebody else's frame. This is also why the damage concentrates
in the allocator: malloc() and free() are where a lot of cycles are spent
immediately after a call instruction, so that is where the skid lands.
The driver knows about this. A few lines further down, in the same
function, sits this:
/*
* rip recorded by IbsOpRip will not be consistent with rsp and rbp
* recorded as part of interrupt regs. Thus we need to use rip from
* interrupt regs while unwinding call stack.
*/
perf_sample_save_callchain(&data, event, iregs);
throttle = perf_event_overflow(event, &data, ®s);
The kernel-side call chain is deliberately built from iregs - the untouched
interrupt registers - precisely so that the IP agrees with rsp and rbp. The
sample itself, however, is handed ®s, the modified copy - which is
where the user registers and the stack dump come from, as quoted above.
So the inconsistency is known, it is documented in a comment, and it has been
fixed for one of the two consumers. The DWARF path appears to have been
missed.
What the unwinder is actually handed
------------------------------------
I instrumented get_entries() in tools/perf/util/unwind-libunwind-local.c to
print the result of every unw_step() together with the first words of the
recorded stack dump:
DBG init ip=0x...d1d9 sp=0x7ffcc53ff420
DBG [sp+0x18] = 0x40 <- the malloc() argument
DBG [sp+0x28] = 0x55f55cbbd010 <- return address slot: heap
DBG step#1 sret=1 ip=0x55f55cbbd010 <- "returned" into the heap
DBG step#2 sret=-22
and here is the code at the sampled IP:
00000000000011a0 <inner>:
11a0: sub $0x28,%rsp
...
11d0: mov $0x40,%edi
11d5: add $0x8,%rbp
11d9: call 1050 <malloc@plt> <- the IP reported by IBS
11de: mov %rax,-0x8(%rbp)
inner() opens with "sub $0x28,%rsp", so the CFA is rsp+0x30 and the return
address sits at rsp+0x28. The unwinder computed exactly that (0x420 ->
0x450) and read exactly that slot. What it found there is a heap pointer,
because the stack was captured after the call had been taken, and by then the
slot belongs to malloc()'s frame - note malloc()'s own argument, 0x40,
sitting right next to it at [sp+0x18].
I checked the obvious suspects before concluding this: the CFI for that
address is correct, the stack dump is complete (3040 bytes of it), the sample
is a user-mode one, and libdw and libunwind fail on it identically. The
unwinder is not at fault. Its input contradicts itself.
Numbers
-------
Same binary, same run, only the event changes; unwound by libunwind:
event precise_ip single-frame chains avg depth
cpu/cycles/ 0 0.0% 8.0
cpu/cycles/p 1 13.2% 5.1
cpu/cycles/pp 2 15.1% 3.7
cpu/cycles/ppp 3 not available
On the real workload:
cpu/cycles/P 32.1% 4.1
cpu/cycles/ 16.1% 8.8
cpu-clock 16.9% 8.6
(The residue there is Problem 3.)
Why this does not happen on Intel
---------------------------------
setup_pebs_fixed_sample_data() copies the entire GPR set out of the PEBS
record, sp and bp included:
arch/x86/events/intel/ds.c
regs->bp = pebs->bp;
regs->sp = pebs->sp;
so IP and stack agree. IBS records only IbsOpRip; there is no register file
in the record for the driver to copy. The mismatch is therefore rooted in
what the hardware can provide - but, as the comment quoted above shows, the
driver already has an answer for it, and applies that answer to the
kernel-side call chain.
In fairness to IBS, Intel is not flawless here either. The registers come
from the PEBS record, but the stack memory is still copied later, when the
NMI runs, so a sample can be spoiled if the thread returned above the
recorded SP and reused that memory in the meantime. The difference is that
the starting pair is consistent, so the first CFA computation lands on the
right slot; and everything still live on the stack - the frames of callers
that have not returned - cannot have been overwritten, which bounds the
damage to the shallowest frame or two. With IBS the very first step is
already wrong. (That paragraph is reasoning from the code; I have no Intel
machine to check it on.)
How one ends up on IBS without asking for precision
---------------------------------------------------
The default event of perf record is built with the P modifier:
tools/perf/util/evlist.c
while ((pmu = perf_pmus__scan_core(pmu)) != NULL) {
snprintf(buf, sizeof(buf), "%s/cycles/%s", pmu->name,
can_profile_kernel ? "P" : "Pu");
P sets precise_max (tools/perf/util/parse-events.c), which becomes
precise_ip = 3 (tools/perf/util/evsel.c), and evsel__precise_ip_fallback()
then walks it down one step at a time until the kernel agrees:
$ perf record -vv -g -o /tmp/x.data /bin/true
Attempt to add: cpu/cycles/
precise_ip 3
decreasing precise_ip by one (2)
precise_ip 2
precise_ip=2 is accepted here and routed to IBS. In other words, a plain
perf record -g --call-graph dwarf ./prog
on any recent AMD box produces broken call graphs out of the box, silently,
for a user who never asked for precise sampling.
There is already a precedent for special-casing this a few lines up in the
same function: on s390, when call chains are requested, the default event is
a software clock rather than a precise hardware one.
What the workaround costs
-------------------------
Dropping the P is not free, and it should be said plainly. With a
non-precise event the whole sample is taken at one moment, in the interrupt
handler, so the IP and the stack agree and the chain is right - but the leaf
address now carries the usual skid: it names the instruction the CPU happened
to be on when the interrupt arrived, not the one that overflowed the counter.
event leaf instruction address call chain
cycles:P (IBS) exact garbage
cycles skewed by skid correct
On this hardware the two cannot be had at once, unless everything in sight is
built with frame pointers - which on Debian it is not, and which is where
this whole story started. For "where does the time go, by function and by
call path" the trade is the right way round. For perf annotate on a hot loop
it is not, and then there are no call chains at all. That is the part I
would like to see acknowledged somewhere the user can find it.
Problem 2: libdw cannot unwind through lld's default segment layout
===================================================================
This is the one that sent me chasing the linker early on, and it deserves
to be stated carefully, because the linker is not the culprit.
Take a program with a deep call chain - twelve nested noinline functions
around a malloc/free loop, source at the end - and link the same
translation unit twice:
clang -O2 -g -fuse-ld=bfd -o deep_bfd deep-stack.c
clang -O2 -g -fuse-ld=lld -o deep_lld deep-stack.c
Recorded identically with cpu/cycles/, so that Problem 1 is out of the way.
Share of chains with 13 frames or more, and average depth:
binary libdw libunwind
deep_bfd 97.8% (16.7) 97.8% (16.7)
deep_lld 0.3% ( 2.1) 93.9% (16.2)
Same recording, two unwinders, opposite outcomes. And the trigger can be
switched off from the link line, which pins it down:
variant libdw libunwind
lld, as is 0.3% ( 2.1) 93.9% (16.2)
lld -Wl,--no-rosegment 97.7% (16.7) 97.7% (16.7)
lld -Wl,-z,separate-code 95.5% (16.5) 95.5% (16.5)
As far as readelf is concerned nothing is missing from the default lld
output: it has a PT_GNU_EH_FRAME segment and a complete set of FDEs, and
neither binary has .debug_frame.
This is the same ground as llvm-project issue 53156, "perf record
--call-graph dwarf does not support ld.lld's default --rosegment
-z noseparate-code layout" (closed), and the linux-perf-users thread "Call
graph dwarf unwinding fails with lld" from May 2022, where the reporter
wrote plainly "I don't know if this is a bug in lld, perf, or libunwind, only
that it doesn't happen with ld".
What seems to be new here is the missing half of that sentence: libunwind
handles the layout perfectly well. So this is a gap in libdw, not in lld -
and, because of Problem 3 below, a perf built with both unwinders will not
use the one that works.
The practical reach of this is larger than a synthetic test. Every Android
NDK build is linked with lld, and lld is the default in a growing number of
clang configurations.
Problem 3: a one-frame chain counts as success, so the fallback never runs
==========================================================================
tools/perf/util/unwind.c walks a hardcoded list of unwinders, libdw first:
#ifdef HAVE_LIBDW_SUPPORT
symbol_conf.unwind_style[i++] = UNWIND_STYLE_LIBDW;
#endif
#ifdef HAVE_LIBUNWIND_SUPPORT
symbol_conf.unwind_style[i++] = UNWIND_STYLE_LIBUNWIND;
#endif
and stops at the first one that returns anything at all:
if (ret > 0) {
ret = 0;
break;
}
A chain consisting of the leaf frame alone satisfies ret > 0. So whenever
libdw manages to emit the sampled IP and nothing else, libunwind is never
consulted and the user is handed a one-frame "call graph" - which is exactly
what building perf with LIBUNWIND=1 does not fix, and exactly what made this
problem so confusing to chase.
There is no way to influence the choice from the outside: no command line
option, no environment variable, no config knob. To measure the two
unwinders separately I had to add a switch to my own build.
On the real workload, same recording, only the unwinder changes:
event unwinder single-frame avg depth
cycles:P libdw 53.9% 2.2
cycles:P libunwind 7.0% 6.3
cpu-clock libdw 45.6% 2.4
cpu-clock libunwind 0.0% 12.6
cpu-clock default 16.9% 8.6
The last row decomposes as follows: of 2432 samples, libdw returned something
for 904 of them, and 412 of those were a single frame. Those 412 are the
entire residue - for the other 1528 libdw returned nothing at all, the
fallback did run, and libunwind unwound them in full.
One more thing worth mentioning: Debian's perf ships without libunwind at
all.
$ perf version --build-options
dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT
( tip: Deprecated, use LIBUNWIND=1 ... )
So the distribution user has only the weaker unwinder and nothing to fall
back to. Building perf with LIBUNWIND=1 also fails without WERROR=0 (mixed
declarations and code), and the build system calls that path deprecated. On
these workloads the deprecated unwinder is the one that works.
The net effect
==============
Starting point, on a stock Debian perf with the default event: of 2361
samples, 47.6% came back with no frames at all, 28.3% with the leaf frame and
nothing else, and 24.2% with two frames or more. Average depth 2.2. Roughly
three quarters of the profile carried no call path.
Building perf with libunwind but leaving everything else alone gets that to
32.1% single-frame chains at an average depth of 4.1 - better, but the
libdw-first rule is still throwing away the working unwinder's answer.
After switching to a non-precise event and forcing libunwind: 0.0%
single-frame chains, average depth 12.6, every sample carrying a chain.
No frame pointers, no rebuilt system libraries, no debug symbols were needed
to get there - all three of which we tried and measured along the way, and
none of which was the answer. What was needed was knowing about three
unrelated behaviours, none of which is documented and none of which produces
a diagnostic.
Suggestions and questions
=========================
1. Kernel: the fix that was applied to the kernel-side call chain -
passing iregs rather than the modified regs - looks like it applies
verbatim to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER. Should
those be derived from iregs too when the event requests a user stack
dump? If the precise IP is worth keeping in the sample regardless,
should such samples carry a flag, so that userspace knows the IP and the
stack do not belong together? Today nothing distinguishes them.
2. perf: precise events and DWARF call graphs are mutually exclusive on this
hardware, so arguably perf should simply not let the two be combined. I
would suggest two rules rather than one, because a blanket refusal would
make the common case worse:
- for the default event, drop the P when --call-graph dwarf is requested,
silently and on PMUs where precision means IBS. Otherwise a plain
"perf record -g --call-graph dwarf" starts failing outright on every AMD
box, which is worse than today. The s390 case in evlist.c suggests this
kind of substitution is considered acceptable;
- if the user asked for a precise event explicitly and also asked for
DWARF call chains, refuse with a message that says why and what to do,
instead of quietly producing a useless result.
Note that this should be scoped to IBS. On Intel, PEBS records the whole
register set, so precise events and DWARF unwinding work together there
and nothing needs restricting. Frame-pointer call graphs are also fine
with precise events - only the leaf frame is off.
3. perf: a single-frame result should probably not count as unwinder success
and suppress the fallback. This one looks like a small, contained fix.
4. perf: a way to choose the unwinder explicitly - an option or an
environment variable - would have saved most of this investigation.
5. libdw: the lld layout case is worth fixing, or at least worth recording
somewhere, given how much clang output it covers.
6. Documentation: perf-amd-ibs(1) and perf-record(1) could say that precise
events on AMD are incompatible with --call-graph dwarf. A single sentence
would have saved a day here.
Reproducers
===========
Problem 1, the allocator case:
/* gcc -O2 -g -o ibs-repro ibs-repro.c */
#include <stdlib.h>
#include <stdio.h>
static void *keep[200000];
__attribute__((noinline)) static long inner(int n)
{
long c = 0;
int i;
for (i = 0; i < n; i++) {
keep[i] = malloc(64);
c += (long)keep[i];
}
for (i = 0; i < n; i++)
free(keep[i]);
return c;
}
__attribute__((noinline)) static long outer(int n)
{
return inner(n) + 1;
}
int main(void)
{
long s = 0;
int i;
for (i = 0; i < 300; i++)
s += outer(200000);
printf("%ld\n", s);
return 0;
}
perf record -e cpu/cycles/P -F 300 --call-graph dwarf,65528 \
-o p.data ./ibs-repro
perf record -e cpu/cycles/ -F 300 --call-graph dwarf,65528 \
-o n.data ./ibs-repro
Problems 2 and 3, the linker case:
/* clang -O2 -g -fuse-ld=bfd -o deep_bfd deep-stack.c
clang -O2 -g -fuse-ld=lld -o deep_lld deep-stack.c */
#include <stdlib.h>
#include <stdio.h>
static void *keep[4096];
__attribute__((noinline)) static long f12(int n)
{
long c = 0;
for (int i = 0; i < n; i++) {
int k = i & 4095;
free(keep[k]);
keep[k] = malloc(64);
c += (long)keep[k];
}
return c;
}
__attribute__((noinline)) static long f11(int n){ return f12(n)+1; }
__attribute__((noinline)) static long f10(int n){ return f11(n)+2; }
__attribute__((noinline)) static long f9 (int n){ return f10(n)+3; }
__attribute__((noinline)) static long f8 (int n){ return f9 (n)+4; }
__attribute__((noinline)) static long f7 (int n){ return f8 (n)+5; }
__attribute__((noinline)) static long f6 (int n){ return f7 (n)+6; }
__attribute__((noinline)) static long f5 (int n){ return f6 (n)+7; }
__attribute__((noinline)) static long f4 (int n){ return f5 (n)+8; }
__attribute__((noinline)) static long f3 (int n){ return f4 (n)+9; }
__attribute__((noinline)) static long f2 (int n){ return f3 (n)+10; }
__attribute__((noinline)) static long f1 (int n){ return f2 (n)+11; }
int main(void)
{
long s = 0;
for (int i = 0; i < 200; i++)
s += f1(200000);
printf("%ld\n", s);
return 0;
}
perf record -e cpu/cycles/ -F 2000 --call-graph dwarf,65528 \
-o bfd.data ./deep_bfd
perf record -e cpu/cycles/ -F 2000 --call-graph dwarf,65528 \
-o lld.data ./deep_lld
Counting single-frame chains and average depth:
count() { perf script -i "$1" | awk '
/^[^ \t]/ {if (n) {t++; if (n==1) o++; s+=n} n=0; next}
/^[ \t]*[0-9a-f]+ / {n++}
END {printf "%d of %d single-frame (%.1f%%), avg depth %.1f\n",
o, t, 100*o/t, s/t}'; }
To see Problem 1 in isolation a perf built with libunwind is needed, and
Problem 3 has to be worked around; with a libdw-only build all three add up
and the picture is very hard to read. Which, in a sense, is the whole point
of this message.
== end of AI description
I hope it was an interesting read!
Let me know if I could do anything here, I will be happy to help fixing
these problems. Hope this would help anybody, and looking for the feedback.
Regards, Gennady Kupava
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-21 21:49 [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs Gennady Kupava
@ 2026-08-24 13:35 ` Ravi Bangoria
2026-08-25 18:05 ` Ian Rogers
2026-08-27 21:55 ` Namhyung Kim
2 siblings, 0 replies; 14+ messages in thread
From: Ravi Bangoria @ 2026-08-24 13:35 UTC (permalink / raw)
To: Gennady Kupava; +Cc: linux-perf-users
Thanks for the detailed analysis. While I'm yet to digest the whole thread,
> Suggestions and questions
> =========================
>
> 1. Kernel: the fix that was applied to the kernel-side call chain -
> passing iregs rather than the modified regs - looks like it applies
> verbatim to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER. Should
> those be derived from iregs too when the event requests a user stack
> dump? If the precise IP is worth keeping in the sample regardless,
> should such samples carry a flag, so that userspace knows the IP and the
> stack do not belong together? Today nothing distinguishes them.
Samples with the precise IP are tagged as PERF_RECORD_MISC_EXACT_IP.
> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
> hardware, so arguably perf should simply not let the two be combined. I
> would suggest two rules rather than one, because a blanket refusal would
> make the common case worse:
>
> - for the default event, drop the P when --call-graph dwarf is requested,
> silently and on PMUs where precision means IBS. Otherwise a plain
> "perf record -g --call-graph dwarf" starts failing outright on every AMD
> box, which is worse than today. The s390 case in evlist.c suggests this
> kind of substitution is considered acceptable;
>
> - if the user asked for a precise event explicitly and also asked for
> DWARF call chains, refuse with a message that says why and what to do,
> instead of quietly producing a useless result.
Another option would be to report the IBS RIP only via PERF_SAMPLE_IP,
leaving the original pt_regs->ip untouched in PERF_SAMPLE_REGS_USER.
Thanks,
Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-21 21:49 [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs Gennady Kupava
2026-08-24 13:35 ` Ravi Bangoria
@ 2026-08-25 18:05 ` Ian Rogers
2026-08-27 21:55 ` Namhyung Kim
2 siblings, 0 replies; 14+ messages in thread
From: Ian Rogers @ 2026-08-25 18:05 UTC (permalink / raw)
To: Gennady Kupava; +Cc: linux-perf-users
On Fri, Aug 21, 2026 at 2:50 PM Gennady Kupava <gennady.kupava@gmail.com> wrote:
>
> Hello,
>
> I decided to share a sort of horror adventure: my attempt to use a recent
> perf with call graphs on recent hardware. I think this story is useful both
> to the developers of the tool - there are small bugs to fix here, and a
> user experience worth assessing - and to users, who I am pretty sure could
> face the same issues. Other than that, I have a feeling this might qualify
> as an interesting adventure to follow.
>
> So my idea was to work on optimization of an open-source mapping project
> (OsmAnd), for which I am trying to prepare a series of patches - some of
> them have landed already, others are still in my head.
>
> I have 15 years of experience using perf-like tools periodically; it was
> oprofile before perf, and the first tool of this kind that impressed me a
> lot.
>
> So the task here was to take OsmAnd's core library, which is typically built
> for Android, build it for amd64, and measure its performance with call
> graphs. I am on Debian unstable, very up to date. The idea of this mail is
> to explain the pile of problems I faced during this adventure.
>
> The Linux distribution is important, as Debian did not enable frame
> pointers - so it is not possible to "just enable FP", call stacks have
> to be done using stack dumps.
>
> So I got everything built and ran it, and tried to run perf on it, which
> succeeded, but... it quickly turned out that the stack frames were not
> complete at all: there was only 2% of callgraphs with deep chain...
>
> And so, together with an AI, I started figuring out what was going on - and
> this is where the adventure begins.
>
> Attempt number 1: kernel paranoid. The first suspicious thing found was
> kernel.perf_event_paranoid, which is 1 on Debian. I set it to -1, and the
> profile did seem to improve a lot - the share of samples carrying a deep
> chain went from under 2% to around 44%, which felt like the answer. It was
> not. Later it turned out this change did not change anything.
>
> Attempt number 2: the lld issue. Claude found a mail thread on this very
> list - "Call graph dwarf unwinding fails with lld", May 2022,
> https://www.spinics.net/lists/linux-perf-users/msg19574.html - where it
This link was broken for me, but:
https://lore.kernel.org/linux-perf-users/CAOBGo4zjkcX=ZQm1uYRDe9EjYqsTCyZY-Gf1C4XqMNYGFCcF+Q@mail.gmail.com/
worked.
> seemed clear that lld builds are worse than ld ones, so that was the next
> candidate to try. So I tried, and indeed, using ld as the linker made the
> situation better, but... somehow it still did not work. A few hours of
> AI-augmented debugging later there were fewer frames lost, but they were
> still lost.
>
> Identified problem number 1: it turned out that the default counters on my
> AMD CPU make it impossible to dump the stack and the IP from the same place.
>
> Identified problem number 2: even if the stack dump is correct, libdw cannot
> process it, only libunwind can.
>
> Identified problem number 3: even if I compile perf with libunwind, it still
> tries to use libdw - and that fails while perf thinks it succeeded.
So libunwind at this point is pretty much unmaintained. Elsewhere
there has been an adoption of an unwinder in LLVM. The problem with
LLVM is that it is a large dependency generally bringing in far more
than what perf needs, leading to startup latency and binary size
problems. Since the perf tools are built and distributed alongside the
kernel, there is reluctance to add LLVM to the list of dependencies as
most distributions would disable such a large dependency.
Consequently, we've been working to make libdw work.
> After identifying all 3 problems, I was able to generate quality stack
> traces.
>
> However, it took me half of the day to reach that point, and it is hard to
> imagine how people in general are able to use perf - this was way too much
> effort just to record call graphs...
There's no one more entitled than an open source software user :-) But
seriously, thanks for taking the time to provide the feedback.
> From this point I will let the AI explain each of the problems I faced, with
> precise technical details (i reviewed it all and it makes sense to me):
>
>
> The setup
> =========
>
> CPU AMD Ryzen 9 9950X3D, family 0x1a, model 0x44, stepping 0,
> microcode 0xb404038; ibs_op and ibs_fetch PMUs present
> kernel 6.18.5+deb14-amd64 (Debian 6.18.5-1) - all measurements
> source linux-source 7.1.8, used for the quotes below; the code there is
> unchanged, so none of this is fixed in a newer tree
> perf 7.1.8, both the Debian build and a local build
>
> Two workloads appear below. One is a self-contained C reproducer, given at
> the end. The other is the real thing: a map tile rasterizer built on Qt5 and
> Skia, with deep chains crossing several shared libraries.
>
> The metric throughout is the share of samples whose call chain came back with
> a single frame, plus the average chain depth. The counting script is at the
> end too. All recordings use --call-graph dwarf,65528.
>
>
> Problem 1: IP and stack dump mismatch
> =====================================
>
> The IP and the stack in one sample describe different moments.
>
> perf record's default event carries the P modifier, so on AMD it is served by
> IBS. How one gets there without asking is covered further down.
>
> perf_ibs_handle_irq() copies the whole register set from the NMI frame and
> then overrides only the instruction pointer with the one recorded by IBS:
>
> arch/x86/events/amd/ibs.c, perf_ibs_handle_irq()
>
> regs = *iregs;
> if (check_rip && (ibs_data.regs[2] & IBS_RIP_INVALID)) {
> regs.flags &= ~PERF_EFLAGS_EXACT;
> } else {
> ...
> set_linear_ip(®s, ibs_data.regs[1]); /* IbsOpRip */
> regs.flags |= PERF_EFLAGS_EXACT;
> }
>
> That same pt_regs is what both PERF_SAMPLE_REGS_USER and
> PERF_SAMPLE_STACK_USER are derived from:
>
> kernel/events/core.c, perf_sample_regs_user()
>
> if (user_mode(regs)) {
> regs_user->abi = perf_reg_abi(current);
> regs_user->regs = regs;
>
> kernel/events/core.c, perf_output_sample()
>
> perf_output_sample_ustack(handle, data->stack_user_size,
> data->regs_user.regs);
>
> kernel/events/core.c, perf_output_sample_ustack()
>
> sp = perf_user_stack_pointer(regs);
>
> So the IP names the instruction IBS tagged, while SP - and therefore every
> byte of the dumped stack - describes wherever the CPU was when the NMI
> finally arrived. These are not the same place: in between, the tagged call
> has been taken and the callee has built its frame.
This is a known issue with precise samples and call graphs. Even if SP
were sampled, the user stack also needs to be copied.
> Frame-pointer unwinding barely notices. It still returns a full chain, only
> with a top frame from a slightly different moment. DWARF unwinding cannot
> survive it, because the CFA rule is chosen by IP and then applied to a stack
> belonging to somebody else's frame. This is also why the damage concentrates
> in the allocator: malloc() and free() are where a lot of cycles are spent
> immediately after a call instruction, so that is where the skid lands.
There had been hope that frame pointers would become the norm again:
https://www.brendangregg.com/blog/2024-03-17/the-return-of-the-frame-pointers.html
As accurate stack layout is needed for things like live patching,
there was a push for a compressed debug format, ORC, which has now
been generalized as SFrames. As it takes two instructions on x86 to
establish a frame (push rbp, mov rsp->rbp) then this potentially
closes a problem if a sample happens on the push where you could lose
a function as rbp is still the caller's rbp.
The obvious approach that avoids these problems is LBR. Potentially we
could use shadow stacks, but the work for that doesn't exist - maybe
as an RFC patch; I forget.
> The driver knows about this. A few lines further down, in the same
> function, sits this:
>
> /*
> * rip recorded by IbsOpRip will not be consistent with rsp and rbp
> * recorded as part of interrupt regs. Thus we need to use rip from
> * interrupt regs while unwinding call stack.
> */
> perf_sample_save_callchain(&data, event, iregs);
>
> throttle = perf_event_overflow(event, &data, ®s);
>
> The kernel-side call chain is deliberately built from iregs - the untouched
> interrupt registers - precisely so that the IP agrees with rsp and rbp. The
> sample itself, however, is handed ®s, the modified copy - which is
> where the user registers and the stack dump come from, as quoted above.
>
> So the inconsistency is known, it is documented in a comment, and it has been
> fixed for one of the two consumers. The DWARF path appears to have been
> missed.
>
> What the unwinder is actually handed
> ------------------------------------
>
> I instrumented get_entries() in tools/perf/util/unwind-libunwind-local.c to
> print the result of every unw_step() together with the first words of the
> recorded stack dump:
>
> DBG init ip=0x...d1d9 sp=0x7ffcc53ff420
> DBG [sp+0x18] = 0x40 <- the malloc() argument
> DBG [sp+0x28] = 0x55f55cbbd010 <- return address slot: heap
> DBG step#1 sret=1 ip=0x55f55cbbd010 <- "returned" into the heap
> DBG step#2 sret=-22
>
> and here is the code at the sampled IP:
>
> 00000000000011a0 <inner>:
> 11a0: sub $0x28,%rsp
> ...
> 11d0: mov $0x40,%edi
> 11d5: add $0x8,%rbp
> 11d9: call 1050 <malloc@plt> <- the IP reported by IBS
> 11de: mov %rax,-0x8(%rbp)
>
> inner() opens with "sub $0x28,%rsp", so the CFA is rsp+0x30 and the return
> address sits at rsp+0x28. The unwinder computed exactly that (0x420 ->
> 0x450) and read exactly that slot. What it found there is a heap pointer,
> because the stack was captured after the call had been taken, and by then the
> slot belongs to malloc()'s frame - note malloc()'s own argument, 0x40,
> sitting right next to it at [sp+0x18].
>
> I checked the obvious suspects before concluding this: the CFI for that
> address is correct, the stack dump is complete (3040 bytes of it), the sample
> is a user-mode one, and libdw and libunwind fail on it identically. The
> unwinder is not at fault. Its input contradicts itself.
>
> Numbers
> -------
>
> Same binary, same run, only the event changes; unwound by libunwind:
>
> event precise_ip single-frame chains avg depth
> cpu/cycles/ 0 0.0% 8.0
> cpu/cycles/p 1 13.2% 5.1
> cpu/cycles/pp 2 15.1% 3.7
> cpu/cycles/ppp 3 not available
>
> On the real workload:
>
> cpu/cycles/P 32.1% 4.1
> cpu/cycles/ 16.1% 8.8
> cpu-clock 16.9% 8.6
>
> (The residue there is Problem 3.)
>
> Why this does not happen on Intel
> ---------------------------------
>
> setup_pebs_fixed_sample_data() copies the entire GPR set out of the PEBS
> record, sp and bp included:
>
> arch/x86/events/intel/ds.c
>
> regs->bp = pebs->bp;
> regs->sp = pebs->sp;
>
> so IP and stack agree. IBS records only IbsOpRip; there is no register file
> in the record for the driver to copy. The mismatch is therefore rooted in
> what the hardware can provide - but, as the comment quoted above shows, the
> driver already has an answer for it, and applies that answer to the
> kernel-side call chain.
>
> In fairness to IBS, Intel is not flawless here either. The registers come
> from the PEBS record, but the stack memory is still copied later, when the
> NMI runs, so a sample can be spoiled if the thread returned above the
> recorded SP and reused that memory in the meantime. The difference is that
> the starting pair is consistent, so the first CFA computation lands on the
> right slot; and everything still live on the stack - the frames of callers
> that have not returned - cannot have been overwritten, which bounds the
> damage to the shallowest frame or two. With IBS the very first step is
> already wrong. (That paragraph is reasoning from the code; I have no Intel
> machine to check it on.)
Agreed. It is hard to make any of these scenarios work.
> How one ends up on IBS without asking for precision
> ---------------------------------------------------
>
> The default event of perf record is built with the P modifier:
>
> tools/perf/util/evlist.c
>
> while ((pmu = perf_pmus__scan_core(pmu)) != NULL) {
> snprintf(buf, sizeof(buf), "%s/cycles/%s", pmu->name,
> can_profile_kernel ? "P" : "Pu");
>
> P sets precise_max (tools/perf/util/parse-events.c), which becomes
> precise_ip = 3 (tools/perf/util/evsel.c), and evsel__precise_ip_fallback()
> then walks it down one step at a time until the kernel agrees:
>
> $ perf record -vv -g -o /tmp/x.data /bin/true
> Attempt to add: cpu/cycles/
> precise_ip 3
> decreasing precise_ip by one (2)
> precise_ip 2
>
> precise_ip=2 is accepted here and routed to IBS. In other words, a plain
>
> perf record -g --call-graph dwarf ./prog
>
> on any recent AMD box produces broken call graphs out of the box, silently,
> for a user who never asked for precise sampling.
>
> There is already a precedent for special-casing this a few lines up in the
> same function: on s390, when call chains are requested, the default event is
> a software clock rather than a precise hardware one.
So you want precise samples by default as they minimize skid, the
distance between an event happening and what the profiler is telling
you. The max_precise on AMD's core PMU is 0 (see the perf list man
page for the precise number meanings), but there's special effort to
make instructions and cycles precise by moving them onto IBS. Because
of this the perf tool ignores the max_precise value from the PMU files
in sysfs and probes what will work by default.
> What the workaround costs
> -------------------------
>
> Dropping the P is not free, and it should be said plainly. With a
> non-precise event the whole sample is taken at one moment, in the interrupt
> handler, so the IP and the stack agree and the chain is right - but the leaf
> address now carries the usual skid: it names the instruction the CPU happened
> to be on when the interrupt arrived, not the one that overflowed the counter.
>
> event leaf instruction address call chain
> cycles:P (IBS) exact garbage
> cycles skewed by skid correct
>
> On this hardware the two cannot be had at once, unless everything in sight is
> built with frame pointers - which on Debian it is not, and which is where
> this whole story started. For "where does the time go, by function and by
> call path" the trade is the right way round. For perf annotate on a hot loop
> it is not, and then there are no call chains at all. That is the part I
> would like to see acknowledged somewhere the user can find it.
>
>
> Problem 2: libdw cannot unwind through lld's default segment layout
> ===================================================================
>
> This is the one that sent me chasing the linker early on, and it deserves
> to be stated carefully, because the linker is not the culprit.
>
> Take a program with a deep call chain - twelve nested noinline functions
> around a malloc/free loop, source at the end - and link the same
> translation unit twice:
>
> clang -O2 -g -fuse-ld=bfd -o deep_bfd deep-stack.c
> clang -O2 -g -fuse-ld=lld -o deep_lld deep-stack.c
>
> Recorded identically with cpu/cycles/, so that Problem 1 is out of the way.
> Share of chains with 13 frames or more, and average depth:
>
> binary libdw libunwind
> deep_bfd 97.8% (16.7) 97.8% (16.7)
> deep_lld 0.3% ( 2.1) 93.9% (16.2)
>
> Same recording, two unwinders, opposite outcomes. And the trigger can be
> switched off from the link line, which pins it down:
>
> variant libdw libunwind
> lld, as is 0.3% ( 2.1) 93.9% (16.2)
> lld -Wl,--no-rosegment 97.7% (16.7) 97.7% (16.7)
> lld -Wl,-z,separate-code 95.5% (16.5) 95.5% (16.5)
>
> As far as readelf is concerned nothing is missing from the default lld
> output: it has a PT_GNU_EH_FRAME segment and a complete set of FDEs, and
> neither binary has .debug_frame.
>
> This is the same ground as llvm-project issue 53156, "perf record
> --call-graph dwarf does not support ld.lld's default --rosegment
> -z noseparate-code layout" (closed), and the linux-perf-users thread "Call
> graph dwarf unwinding fails with lld" from May 2022, where the reporter
> wrote plainly "I don't know if this is a bug in lld, perf, or libunwind, only
> that it doesn't happen with ld".
>
> What seems to be new here is the missing half of that sentence: libunwind
> handles the layout perfectly well. So this is a gap in libdw, not in lld -
> and, because of Problem 3 below, a perf built with both unwinders will not
> use the one that works.
So when fixing and expanding libunwind support a not unreasonable
alternative would have been to just remove it.
> The practical reach of this is larger than a synthetic test. Every Android
> NDK build is linked with lld, and lld is the default in a growing number of
> clang configurations.
Did you confirm the bug was in perf or in libdw? There are fixes in
libdw and my experiences with newer libdw's have been more positive
than older ones.
> Problem 3: a one-frame chain counts as success, so the fallback never runs
> ==========================================================================
>
> tools/perf/util/unwind.c walks a hardcoded list of unwinders, libdw first:
>
> #ifdef HAVE_LIBDW_SUPPORT
> symbol_conf.unwind_style[i++] = UNWIND_STYLE_LIBDW;
> #endif
> #ifdef HAVE_LIBUNWIND_SUPPORT
> symbol_conf.unwind_style[i++] = UNWIND_STYLE_LIBUNWIND;
> #endif
>
> and stops at the first one that returns anything at all:
>
> if (ret > 0) {
> ret = 0;
> break;
> }
>
> A chain consisting of the leaf frame alone satisfies ret > 0. So whenever
> libdw manages to emit the sampled IP and nothing else, libunwind is never
> consulted and the user is handed a one-frame "call graph" - which is exactly
> what building perf with LIBUNWIND=1 does not fix, and exactly what made this
> problem so confusing to chase.
>
> There is no way to influence the choice from the outside: no command line
> option, no environment variable, no config knob. To measure the two
> unwinders separately I had to add a switch to my own build.
There's a perf config file value of "unwind.style", the code for it
being in the same file you point at. These statements aren't correct.
> On the real workload, same recording, only the unwinder changes:
>
> event unwinder single-frame avg depth
> cycles:P libdw 53.9% 2.2
> cycles:P libunwind 7.0% 6.3
> cpu-clock libdw 45.6% 2.4
> cpu-clock libunwind 0.0% 12.6
> cpu-clock default 16.9% 8.6
>
> The last row decomposes as follows: of 2432 samples, libdw returned something
> for 904 of them, and 412 of those were a single frame. Those 412 are the
> entire residue - for the other 1528 libdw returned nothing at all, the
> fallback did run, and libunwind unwound them in full.
>
> One more thing worth mentioning: Debian's perf ships without libunwind at
> all.
>
> $ perf version --build-options
> dwarf-unwind: [ on ] # HAVE_DWARF_UNWIND_SUPPORT
> libdw-dwarf-unwind: [ on ] # HAVE_LIBDW_SUPPORT
> libunwind: [ OFF ] # HAVE_LIBUNWIND_SUPPORT
> ( tip: Deprecated, use LIBUNWIND=1 ... )
>
> So the distribution user has only the weaker unwinder and nothing to fall
> back to. Building perf with LIBUNWIND=1 also fails without WERROR=0 (mixed
> declarations and code), and the build system calls that path deprecated. On
> these workloads the deprecated unwinder is the one that works.
Yep. It is deprecated because no one is maintaining libunwind. For
example, building with a mixture of libunwind architectures seems
broken, and there is no feedback from the mailing list.
> The net effect
> ==============
>
> Starting point, on a stock Debian perf with the default event: of 2361
> samples, 47.6% came back with no frames at all, 28.3% with the leaf frame and
> nothing else, and 24.2% with two frames or more. Average depth 2.2. Roughly
> three quarters of the profile carried no call path.
>
> Building perf with libunwind but leaving everything else alone gets that to
> 32.1% single-frame chains at an average depth of 4.1 - better, but the
> libdw-first rule is still throwing away the working unwinder's answer.
>
> After switching to a non-precise event and forcing libunwind: 0.0%
> single-frame chains, average depth 12.6, every sample carrying a chain.
>
> No frame pointers, no rebuilt system libraries, no debug symbols were needed
> to get there - all three of which we tried and measured along the way, and
> none of which was the answer. What was needed was knowing about three
> unrelated behaviours, none of which is documented and none of which produces
> a diagnostic.
It seems you are venting as much at the distribution as at the perf tool here.
> Suggestions and questions
> =========================
>
> 1. Kernel: the fix that was applied to the kernel-side call chain -
> passing iregs rather than the modified regs - looks like it applies
> verbatim to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER. Should
> those be derived from iregs too when the event requests a user stack
> dump? If the precise IP is worth keeping in the sample regardless,
> should such samples carry a flag, so that userspace knows the IP and the
> stack do not belong together? Today nothing distinguishes them.
This is valid feedback. In areas like LBR, the sample IP and the
branch stack duplicate values, and they differ and in the tool we make
choices about which to report. Including it in the user regs would be
an ABI change requiring versioning support.
> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
> hardware, so arguably perf should simply not let the two be combined. I
> would suggest two rules rather than one, because a blanket refusal would
> make the common case worse:
>
> - for the default event, drop the P when --call-graph dwarf is requested,
> silently and on PMUs where precision means IBS. Otherwise a plain
> "perf record -g --call-graph dwarf" starts failing outright on every AMD
> box, which is worse than today. The s390 case in evlist.c suggests this
> kind of substitution is considered acceptable;
The s390 change is relatively recent and somewhat in flux. We could
detect AMD and choose events but what is selected currently was done
for AMD (ie we don't use max_precise from the PMU).
> - if the user asked for a precise event explicitly and also asked for
> DWARF call chains, refuse with a message that says why and what to do,
> instead of quietly producing a useless result.
>
> Note that this should be scoped to IBS. On Intel, PEBS records the whole
> register set, so precise events and DWARF unwinding work together there
> and nothing needs restricting. Frame-pointer call graphs are also fine
> with precise events - only the leaf frame is off.
>
> 3. perf: a single-frame result should probably not count as unwinder success
> and suppress the fallback. This one looks like a small, contained fix.
It seems better to fix libdw. Unwinder success means that we think the
unwinder ran and understood the DWARF debug information. Supporting a
mixture of unwinds seems overly complex; moreover, libunwind support
probably shouldn't be there.
> 4. perf: a way to choose the unwinder explicitly - an option or an
> environment variable - would have saved most of this investigation.
This exists.
> 5. libdw: the lld layout case is worth fixing, or at least worth recording
> somewhere, given how much clang output it covers.
Patches welcome.
> 6. Documentation: perf-amd-ibs(1) and perf-record(1) could say that precise
> events on AMD are incompatible with --call-graph dwarf. A single sentence
> would have saved a day here.
It may be profitable here to go to your distribution and complain.
They control how things are built and could, for example, enable frame
pointers.
Thanks,
Ian
> Reproducers
> ===========
>
> Problem 1, the allocator case:
>
> /* gcc -O2 -g -o ibs-repro ibs-repro.c */
> #include <stdlib.h>
> #include <stdio.h>
>
> static void *keep[200000];
>
> __attribute__((noinline)) static long inner(int n)
> {
> long c = 0;
> int i;
>
> for (i = 0; i < n; i++) {
> keep[i] = malloc(64);
> c += (long)keep[i];
> }
> for (i = 0; i < n; i++)
> free(keep[i]);
> return c;
> }
>
> __attribute__((noinline)) static long outer(int n)
> {
> return inner(n) + 1;
> }
>
> int main(void)
> {
> long s = 0;
> int i;
>
> for (i = 0; i < 300; i++)
> s += outer(200000);
> printf("%ld\n", s);
> return 0;
> }
>
> perf record -e cpu/cycles/P -F 300 --call-graph dwarf,65528 \
> -o p.data ./ibs-repro
> perf record -e cpu/cycles/ -F 300 --call-graph dwarf,65528 \
> -o n.data ./ibs-repro
>
> Problems 2 and 3, the linker case:
>
> /* clang -O2 -g -fuse-ld=bfd -o deep_bfd deep-stack.c
> clang -O2 -g -fuse-ld=lld -o deep_lld deep-stack.c */
> #include <stdlib.h>
> #include <stdio.h>
>
> static void *keep[4096];
>
> __attribute__((noinline)) static long f12(int n)
> {
> long c = 0;
> for (int i = 0; i < n; i++) {
> int k = i & 4095;
> free(keep[k]);
> keep[k] = malloc(64);
> c += (long)keep[k];
> }
> return c;
> }
> __attribute__((noinline)) static long f11(int n){ return f12(n)+1; }
> __attribute__((noinline)) static long f10(int n){ return f11(n)+2; }
> __attribute__((noinline)) static long f9 (int n){ return f10(n)+3; }
> __attribute__((noinline)) static long f8 (int n){ return f9 (n)+4; }
> __attribute__((noinline)) static long f7 (int n){ return f8 (n)+5; }
> __attribute__((noinline)) static long f6 (int n){ return f7 (n)+6; }
> __attribute__((noinline)) static long f5 (int n){ return f6 (n)+7; }
> __attribute__((noinline)) static long f4 (int n){ return f5 (n)+8; }
> __attribute__((noinline)) static long f3 (int n){ return f4 (n)+9; }
> __attribute__((noinline)) static long f2 (int n){ return f3 (n)+10; }
> __attribute__((noinline)) static long f1 (int n){ return f2 (n)+11; }
>
> int main(void)
> {
> long s = 0;
> for (int i = 0; i < 200; i++)
> s += f1(200000);
> printf("%ld\n", s);
> return 0;
> }
>
> perf record -e cpu/cycles/ -F 2000 --call-graph dwarf,65528 \
> -o bfd.data ./deep_bfd
> perf record -e cpu/cycles/ -F 2000 --call-graph dwarf,65528 \
> -o lld.data ./deep_lld
>
> Counting single-frame chains and average depth:
>
> count() { perf script -i "$1" | awk '
> /^[^ \t]/ {if (n) {t++; if (n==1) o++; s+=n} n=0; next}
> /^[ \t]*[0-9a-f]+ / {n++}
> END {printf "%d of %d single-frame (%.1f%%), avg depth %.1f\n",
> o, t, 100*o/t, s/t}'; }
>
> To see Problem 1 in isolation a perf built with libunwind is needed, and
> Problem 3 has to be worked around; with a libdw-only build all three add up
> and the picture is very hard to read. Which, in a sense, is the whole point
> of this message.
>
> == end of AI description
>
> I hope it was an interesting read!
> Let me know if I could do anything here, I will be happy to help fixing
> these problems. Hope this would help anybody, and looking for the feedback.
>
> Regards, Gennady Kupava
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-21 21:49 [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs Gennady Kupava
2026-08-24 13:35 ` Ravi Bangoria
2026-08-25 18:05 ` Ian Rogers
@ 2026-08-27 21:55 ` Namhyung Kim
2026-08-28 16:47 ` Namhyung Kim
2026-08-28 16:58 ` Namhyung Kim
2 siblings, 2 replies; 14+ messages in thread
From: Namhyung Kim @ 2026-08-27 21:55 UTC (permalink / raw)
To: Gennady Kupava; +Cc: linux-perf-users
Hello,
Thanks a lot for your detailed report! It's good but let me jump into
the action items directly.
On Fri, Aug 21, 2026 at 10:49:59PM +0100, Gennady Kupava wrote:
[SNIP]
> Suggestions and questions
> =========================
>
> 1. Kernel: the fix that was applied to the kernel-side call chain -
> passing iregs rather than the modified regs - looks like it applies
> verbatim to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER. Should
> those be derived from iregs too when the event requests a user stack
> dump? If the precise IP is worth keeping in the sample regardless,
> should such samples carry a flag, so that userspace knows the IP and the
> stack do not belong together? Today nothing distinguishes them.
As Ravi said, you can check the misc field if the IP is from precise
events.
And there's PERF_SAMPLE_REGS_INTR which I believe captures registers
from the interrupt handler. I don't remember why we have both REGS_USER
and REGS_INTR but it seems REGS_INTR would work for callchains.
>
> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
> hardware, so arguably perf should simply not let the two be combined. I
> would suggest two rules rather than one, because a blanket refusal would
> make the common case worse:
>
> - for the default event, drop the P when --call-graph dwarf is requested,
> silently and on PMUs where precision means IBS. Otherwise a plain
> "perf record -g --call-graph dwarf" starts failing outright on every AMD
> box, which is worse than today. The s390 case in evlist.c suggests this
> kind of substitution is considered acceptable;
>
> - if the user asked for a precise event explicitly and also asked for
> DWARF call chains, refuse with a message that says why and what to do,
> instead of quietly producing a useless result.
>
> Note that this should be scoped to IBS. On Intel, PEBS records the whole
> register set, so precise events and DWARF unwinding work together there
> and nothing needs restricting. Frame-pointer call graphs are also fine
> with precise events - only the leaf frame is off.
I think we can enable both precise IP and dwarf callchains by using
PERF_SAMPLE_REGS_INTR. For unwinding, it should use PERF_REG_X86_IP
from the REGS_INTR instead of sample.ip
>
> 3. perf: a single-frame result should probably not count as unwinder success
> and suppress the fallback. This one looks like a small, contained fix.
Sounds ok.
>
> 4. perf: a way to choose the unwinder explicitly - an option or an
> environment variable - would have saved most of this investigation.
It seems we have --unwind-style option in perf report but not in perf
annotate. But we can try "unwind.style" config option. Unfortunately
it doesn't seem to have documentation.
>
> 5. libdw: the lld layout case is worth fixing, or at least worth recording
> somewhere, given how much clang output it covers.
Have you checked the latest version?
>
> 6. Documentation: perf-amd-ibs(1) and perf-record(1) could say that precise
> events on AMD are incompatible with --call-graph dwarf. A single sentence
> would have saved a day here.
We can improve documentation always, but as I said I think we can make
them work.
[SNIP]
> == end of AI description
>
> I hope it was an interesting read!
> Let me know if I could do anything here, I will be happy to help fixing
> these problems. Hope this would help anybody, and looking for the feedback.
Thanks, it was a long but interesting read. :)
Sorry for your inconvenience you faced. I haven't look at dwarf
unwinding for a while and thought libdw was good enough. But supporting
multiple libraries for the same purpose is hard so we wanted to go with
libdw and deprecated libunwind. I'm not sure if we need to revisit it.
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-27 21:55 ` Namhyung Kim
@ 2026-08-28 16:47 ` Namhyung Kim
2026-08-28 16:58 ` Namhyung Kim
1 sibling, 0 replies; 14+ messages in thread
From: Namhyung Kim @ 2026-08-28 16:47 UTC (permalink / raw)
To: Gennady Kupava; +Cc: linux-perf-users
On Thu, Aug 27, 2026 at 02:55:15PM -0700, Namhyung Kim wrote:
> Hello,
>
> Thanks a lot for your detailed report! It's good but let me jump into
> the action items directly.
>
> On Fri, Aug 21, 2026 at 10:49:59PM +0100, Gennady Kupava wrote:
> [SNIP]
> > Suggestions and questions
> > =========================
> >
> > 1. Kernel: the fix that was applied to the kernel-side call chain -
> > passing iregs rather than the modified regs - looks like it applies
> > verbatim to PERF_SAMPLE_REGS_USER and PERF_SAMPLE_STACK_USER. Should
> > those be derived from iregs too when the event requests a user stack
> > dump? If the precise IP is worth keeping in the sample regardless,
> > should such samples carry a flag, so that userspace knows the IP and the
> > stack do not belong together? Today nothing distinguishes them.
>
> As Ravi said, you can check the misc field if the IP is from precise
> events.
>
> And there's PERF_SAMPLE_REGS_INTR which I believe captures registers
> from the interrupt handler. I don't remember why we have both REGS_USER
> and REGS_INTR but it seems REGS_INTR would work for callchains.
Of course, it won't work when the interrupt was in kernel mode. :(
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-27 21:55 ` Namhyung Kim
2026-08-28 16:47 ` Namhyung Kim
@ 2026-08-28 16:58 ` Namhyung Kim
2026-08-31 5:05 ` Ravi Bangoria
1 sibling, 1 reply; 14+ messages in thread
From: Namhyung Kim @ 2026-08-28 16:58 UTC (permalink / raw)
To: Gennady Kupava; +Cc: linux-perf-users
On Thu, Aug 27, 2026 at 02:55:15PM -0700, Namhyung Kim wrote:
> Hello,
>
> Thanks a lot for your detailed report! It's good but let me jump into
> the action items directly.
>
> On Fri, Aug 21, 2026 at 10:49:59PM +0100, Gennady Kupava wrote:
[SNIP]
> > 2. perf: precise events and DWARF call graphs are mutually exclusive on this
> > hardware, so arguably perf should simply not let the two be combined. I
> > would suggest two rules rather than one, because a blanket refusal would
> > make the common case worse:
> >
> > - for the default event, drop the P when --call-graph dwarf is requested,
> > silently and on PMUs where precision means IBS. Otherwise a plain
> > "perf record -g --call-graph dwarf" starts failing outright on every AMD
> > box, which is worse than today. The s390 case in evlist.c suggests this
> > kind of substitution is considered acceptable;
> >
> > - if the user asked for a precise event explicitly and also asked for
> > DWARF call chains, refuse with a message that says why and what to do,
> > instead of quietly producing a useless result.
> >
> > Note that this should be scoped to IBS. On Intel, PEBS records the whole
> > register set, so precise events and DWARF unwinding work together there
> > and nothing needs restricting. Frame-pointer call graphs are also fine
> > with precise events - only the leaf frame is off.
>
> I think we can enable both precise IP and dwarf callchains by using
> PERF_SAMPLE_REGS_INTR. For unwinding, it should use PERF_REG_X86_IP
> from the REGS_INTR instead of sample.ip
Ok, it seems we already do this in the perf tools but it looks like the
kernel already overwrote the PERF_REG_X86_IP with the precise IP. I
feel like we should fix the kernel.
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-28 16:58 ` Namhyung Kim
@ 2026-08-31 5:05 ` Ravi Bangoria
2026-09-01 15:19 ` Namhyung Kim
0 siblings, 1 reply; 14+ messages in thread
From: Ravi Bangoria @ 2026-08-31 5:05 UTC (permalink / raw)
To: Namhyung Kim, Gennady Kupava; +Cc: linux-perf-users, Ravi Bangoria
>>> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
>>> hardware, so arguably perf should simply not let the two be combined. I
>>> would suggest two rules rather than one, because a blanket refusal would
>>> make the common case worse:
>>>
>>> - for the default event, drop the P when --call-graph dwarf is requested,
>>> silently and on PMUs where precision means IBS. Otherwise a plain
>>> "perf record -g --call-graph dwarf" starts failing outright on every AMD
>>> box, which is worse than today. The s390 case in evlist.c suggests this
>>> kind of substitution is considered acceptable;
>>>
>>> - if the user asked for a precise event explicitly and also asked for
>>> DWARF call chains, refuse with a message that says why and what to do,
>>> instead of quietly producing a useless result.
>>>
>>> Note that this should be scoped to IBS. On Intel, PEBS records the whole
>>> register set, so precise events and DWARF unwinding work together there
>>> and nothing needs restricting. Frame-pointer call graphs are also fine
>>> with precise events - only the leaf frame is off.
>>
>> I think we can enable both precise IP and dwarf callchains by using
>> PERF_SAMPLE_REGS_INTR. For unwinding, it should use PERF_REG_X86_IP
>> from the REGS_INTR instead of sample.ip
>
> Ok, it seems we already do this in the perf tools but it looks like the
> kernel already overwrote the PERF_REG_X86_IP with the precise IP. I
> feel like we should fix the kernel.
This should resolve the DWARF unwinding issue with IBS PMUs:
--- a/arch/x86/events/amd/ibs.c
+++ b/arch/x86/events/amd/ibs.c
@@ -1523,8 +1523,11 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
goto out;
}
- set_linear_ip(®s, ibs_data.regs[1]);
- regs.flags |= PERF_EFLAGS_EXACT;
+ if (event->attr.sample_type & PERF_SAMPLE_IP) {
+ data.ip = ibs_data.regs[1];
+ data.sample_flags |= PERF_SAMPLE_IP;
+ regs.flags |= PERF_EFLAGS_EXACT;
+ }
}
if (((ibs_caps & IBS_CAPS_BIT63_FILTER) ||
---
However, there's no way to address the stack being out of sync with the
IBS RIP, since the IBS HW does not capture GPRs alongside the sample.
Thanks,
Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-08-31 5:05 ` Ravi Bangoria
@ 2026-09-01 15:19 ` Namhyung Kim
2026-09-03 8:39 ` Mi, Dapeng
0 siblings, 1 reply; 14+ messages in thread
From: Namhyung Kim @ 2026-09-01 15:19 UTC (permalink / raw)
To: Ravi Bangoria; +Cc: Gennady Kupava, linux-perf-users
Hello,
On Mon, Aug 31, 2026 at 10:35:06AM +0530, Ravi Bangoria wrote:
> >>> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
> >>> hardware, so arguably perf should simply not let the two be combined. I
> >>> would suggest two rules rather than one, because a blanket refusal would
> >>> make the common case worse:
> >>>
> >>> - for the default event, drop the P when --call-graph dwarf is requested,
> >>> silently and on PMUs where precision means IBS. Otherwise a plain
> >>> "perf record -g --call-graph dwarf" starts failing outright on every AMD
> >>> box, which is worse than today. The s390 case in evlist.c suggests this
> >>> kind of substitution is considered acceptable;
> >>>
> >>> - if the user asked for a precise event explicitly and also asked for
> >>> DWARF call chains, refuse with a message that says why and what to do,
> >>> instead of quietly producing a useless result.
> >>>
> >>> Note that this should be scoped to IBS. On Intel, PEBS records the whole
> >>> register set, so precise events and DWARF unwinding work together there
> >>> and nothing needs restricting. Frame-pointer call graphs are also fine
> >>> with precise events - only the leaf frame is off.
> >>
> >> I think we can enable both precise IP and dwarf callchains by using
> >> PERF_SAMPLE_REGS_INTR. For unwinding, it should use PERF_REG_X86_IP
> >> from the REGS_INTR instead of sample.ip
> >
> > Ok, it seems we already do this in the perf tools but it looks like the
> > kernel already overwrote the PERF_REG_X86_IP with the precise IP. I
> > feel like we should fix the kernel.
>
> This should resolve the DWARF unwinding issue with IBS PMUs:
>
> --- a/arch/x86/events/amd/ibs.c
> +++ b/arch/x86/events/amd/ibs.c
> @@ -1523,8 +1523,11 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
> goto out;
> }
>
> - set_linear_ip(®s, ibs_data.regs[1]);
> - regs.flags |= PERF_EFLAGS_EXACT;
> + if (event->attr.sample_type & PERF_SAMPLE_IP) {
> + data.ip = ibs_data.regs[1];
> + data.sample_flags |= PERF_SAMPLE_IP;
> + regs.flags |= PERF_EFLAGS_EXACT;
> + }
> }
>
> if (((ibs_caps & IBS_CAPS_BIT63_FILTER) ||
> ---
Right, that's what I thought. And I believe we should do similar on
Intel and not update other registers.
>
> However, there's no way to address the stack being out of sync with the
> IBS RIP, since the IBS HW does not capture GPRs alongside the sample.
I think it's ok and we don't need to sync IP and stack. The dwarf
unwind should start from stack and we can see the skid between IP and
the first entry of the callchain.
Thanks,
Namhyung
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-09-01 15:19 ` Namhyung Kim
@ 2026-09-03 8:39 ` Mi, Dapeng
2026-09-03 11:36 ` Ravi Bangoria
0 siblings, 1 reply; 14+ messages in thread
From: Mi, Dapeng @ 2026-09-03 8:39 UTC (permalink / raw)
To: Namhyung Kim, Ravi Bangoria; +Cc: Gennady Kupava, linux-perf-users
On 9/1/2026 11:19 PM, Namhyung Kim wrote:
> Hello,
>
> On Mon, Aug 31, 2026 at 10:35:06AM +0530, Ravi Bangoria wrote:
>>>>> 2. perf: precise events and DWARF call graphs are mutually exclusive on this
>>>>> hardware, so arguably perf should simply not let the two be combined. I
>>>>> would suggest two rules rather than one, because a blanket refusal would
>>>>> make the common case worse:
>>>>>
>>>>> - for the default event, drop the P when --call-graph dwarf is requested,
>>>>> silently and on PMUs where precision means IBS. Otherwise a plain
>>>>> "perf record -g --call-graph dwarf" starts failing outright on every AMD
>>>>> box, which is worse than today. The s390 case in evlist.c suggests this
>>>>> kind of substitution is considered acceptable;
>>>>>
>>>>> - if the user asked for a precise event explicitly and also asked for
>>>>> DWARF call chains, refuse with a message that says why and what to do,
>>>>> instead of quietly producing a useless result.
>>>>>
>>>>> Note that this should be scoped to IBS. On Intel, PEBS records the whole
>>>>> register set, so precise events and DWARF unwinding work together there
>>>>> and nothing needs restricting. Frame-pointer call graphs are also fine
>>>>> with precise events - only the leaf frame is off.
>>>> I think we can enable both precise IP and dwarf callchains by using
>>>> PERF_SAMPLE_REGS_INTR. For unwinding, it should use PERF_REG_X86_IP
>>>> from the REGS_INTR instead of sample.ip
>>> Ok, it seems we already do this in the perf tools but it looks like the
>>> kernel already overwrote the PERF_REG_X86_IP with the precise IP. I
>>> feel like we should fix the kernel.
>> This should resolve the DWARF unwinding issue with IBS PMUs:
>>
>> --- a/arch/x86/events/amd/ibs.c
>> +++ b/arch/x86/events/amd/ibs.c
>> @@ -1523,8 +1523,11 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
>> goto out;
>> }
>>
>> - set_linear_ip(®s, ibs_data.regs[1]);
>> - regs.flags |= PERF_EFLAGS_EXACT;
>> + if (event->attr.sample_type & PERF_SAMPLE_IP) {
>> + data.ip = ibs_data.regs[1];
>> + data.sample_flags |= PERF_SAMPLE_IP;
We may not set PERF_SAMPLE_IP here, otherwise the kernel address leakage
check in perf_instruction_pointer() could be bypassed.
>> + regs.flags |= PERF_EFLAGS_EXACT;
>> + }
>> }
>>
>> if (((ibs_caps & IBS_CAPS_BIT63_FILTER) ||
>> ---
> Right, that's what I thought. And I believe we should do similar on
> Intel and not update other registers.
For Intel PEBS, the call-chain would always use the interrupt regs instead
of the PEBS regs, so there would be no issues for Intel PEBS.
/*
* We must however always use iregs for the unwinder to stay sane; the
* record BP,SP,IP can point into thin air when the record is from a
* previous PMI context or an (I)RET happened between the record and
* PMI.
*/
perf_sample_save_callchain(data, event, iregs);
Thanks.
>
>> However, there's no way to address the stack being out of sync with the
>> IBS RIP, since the IBS HW does not capture GPRs alongside the sample.
> I think it's ok and we don't need to sync IP and stack. The dwarf
> unwind should start from stack and we can see the skid between IP and
> the first entry of the callchain.
>
> Thanks,
> Namhyung
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-09-03 8:39 ` Mi, Dapeng
@ 2026-09-03 11:36 ` Ravi Bangoria
2026-09-03 11:59 ` Mi, Dapeng
0 siblings, 1 reply; 14+ messages in thread
From: Ravi Bangoria @ 2026-09-03 11:36 UTC (permalink / raw)
To: Mi, Dapeng, Namhyung Kim; +Cc: Gennady Kupava, linux-perf-users, Ravi Bangoria
>>> This should resolve the DWARF unwinding issue with IBS PMUs:
>>>
>>> --- a/arch/x86/events/amd/ibs.c
>>> +++ b/arch/x86/events/amd/ibs.c
>>> @@ -1523,8 +1523,11 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
>>> goto out;
>>> }
>>>
>>> - set_linear_ip(®s, ibs_data.regs[1]);
>>> - regs.flags |= PERF_EFLAGS_EXACT;
>>> + if (event->attr.sample_type & PERF_SAMPLE_IP) {
>>> + data.ip = ibs_data.regs[1];
>>> + data.sample_flags |= PERF_SAMPLE_IP;
>
> We may not set PERF_SAMPLE_IP here, otherwise the kernel address leakage
> check in perf_instruction_pointer() could be bypassed.
Yes. I realized this wouldn't be straightforward, since the privilege
level might change between when the HW captures the sample and when the
NMI is delivered. So, any perf code that depends on user_mode() (e.g.
perf_exclude_event(), _REGS_USER, _REGS_INTR, header->misc, etc.) may
regress with the above change. In addition, guest entry/exit occurring
between sample capture and NMI delivery further complicates the problem.
>> Right, that's what I thought. And I believe we should do similar on
>> Intel and not update other registers.
>
> For Intel PEBS, the call-chain would always use the interrupt regs instead
> of the PEBS regs, so there would be no issues for Intel PEBS.
>
> /*
> * We must however always use iregs for the unwinder to stay sane; the
> * record BP,SP,IP can point into thin air when the record is from a
> * previous PMI context or an (I)RET happened between the record and
> * PMI.
> */
> perf_sample_save_callchain(data, event, iregs);
This wouldn't take care of _STACK_USER, right?
Thanks,
Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-09-03 11:36 ` Ravi Bangoria
@ 2026-09-03 11:59 ` Mi, Dapeng
2026-09-03 16:02 ` Ravi Bangoria
0 siblings, 1 reply; 14+ messages in thread
From: Mi, Dapeng @ 2026-09-03 11:59 UTC (permalink / raw)
To: Ravi Bangoria, Namhyung Kim; +Cc: Gennady Kupava, linux-perf-users
On 9/3/2026 7:36 PM, Ravi Bangoria wrote:
>>>> This should resolve the DWARF unwinding issue with IBS PMUs:
>>>>
>>>> --- a/arch/x86/events/amd/ibs.c
>>>> +++ b/arch/x86/events/amd/ibs.c
>>>> @@ -1523,8 +1523,11 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
>>>> goto out;
>>>> }
>>>>
>>>> - set_linear_ip(®s, ibs_data.regs[1]);
>>>> - regs.flags |= PERF_EFLAGS_EXACT;
>>>> + if (event->attr.sample_type & PERF_SAMPLE_IP) {
>>>> + data.ip = ibs_data.regs[1];
>>>> + data.sample_flags |= PERF_SAMPLE_IP;
>> We may not set PERF_SAMPLE_IP here, otherwise the kernel address leakage
>> check in perf_instruction_pointer() could be bypassed.
> Yes. I realized this wouldn't be straightforward, since the privilege
> level might change between when the HW captures the sample and when the
> NMI is delivered. So, any perf code that depends on user_mode() (e.g.
> perf_exclude_event(), _REGS_USER, _REGS_INTR, header->misc, etc.) may
> regress with the above change. In addition, guest entry/exit occurring
> between sample capture and NMI delivery further complicates the problem.
>
>>> Right, that's what I thought. And I believe we should do similar on
>>> Intel and not update other registers.
>> For Intel PEBS, the call-chain would always use the interrupt regs instead
>> of the PEBS regs, so there would be no issues for Intel PEBS.
>>
>> /*
>> * We must however always use iregs for the unwinder to stay sane; the
>> * record BP,SP,IP can point into thin air when the record is from a
>> * previous PMI context or an (I)RET happened between the record and
>> * PMI.
>> */
>> perf_sample_save_callchain(data, event, iregs);
> This wouldn't take care of _STACK_USER, right?
Yes, I just realized this is not fully correct for Intel PEBS after sending
the comments.
On Intel platforms, for SAMPLE_STACK_USER, the IP/SP/BP is consistent and
they all comes from PEBS, so the DWARF unwinding has no issues. But for
SAMPLE_CALLCHAIN, it's not consistent, the IP is overwritten by PEBS while
the BP/SP are still gotten from PMI. So the kernel call-chain unwinding
could not work.
Suppose we need to follow below 2 rules to ensure the correct call-chain
unwinding and register snapshot consistency.
1. All registers including IP should come from either PEBS or PMI, must not
be mixed. This ensures to get an consistent registers snapshot.
2. Either SAMPLE_CALLCHAIN or SAMPLE_STACK_USER is required, the PMI
registers snapshot must be used. This proves the correct DWARF unwinding.
How's your idea?
Thanks.
>
> Thanks,
> Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-09-03 11:59 ` Mi, Dapeng
@ 2026-09-03 16:02 ` Ravi Bangoria
2026-09-04 0:22 ` Mi, Dapeng
0 siblings, 1 reply; 14+ messages in thread
From: Ravi Bangoria @ 2026-09-03 16:02 UTC (permalink / raw)
To: Mi, Dapeng, Namhyung Kim; +Cc: Gennady Kupava, linux-perf-users, Ravi Bangoria
> Suppose we need to follow below 2 rules to ensure the correct call-chain
> unwinding and register snapshot consistency.
>
> 1. All registers including IP should come from either PEBS or PMI, must not
> be mixed. This ensures to get an consistent registers snapshot.
>
> 2. Either SAMPLE_CALLCHAIN or SAMPLE_STACK_USER is required, the PMI
> registers snapshot must be used. This proves the correct DWARF unwinding.
>
> How's your idea?
Looks good. Mind sending the patch?
Thanks,
Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-09-03 16:02 ` Ravi Bangoria
@ 2026-09-04 0:22 ` Mi, Dapeng
2026-09-05 10:14 ` Gennady Kupava
0 siblings, 1 reply; 14+ messages in thread
From: Mi, Dapeng @ 2026-09-04 0:22 UTC (permalink / raw)
To: Ravi Bangoria, Namhyung Kim; +Cc: Gennady Kupava, linux-perf-users
On 9/4/2026 12:02 AM, Ravi Bangoria wrote:
>> Suppose we need to follow below 2 rules to ensure the correct call-chain
>> unwinding and register snapshot consistency.
>>
>> 1. All registers including IP should come from either PEBS or PMI, must not
>> be mixed. This ensures to get an consistent registers snapshot.
>>
>> 2. Either SAMPLE_CALLCHAIN or SAMPLE_STACK_USER is required, the PMI
>> registers snapshot must be used. This proves the correct DWARF unwinding.
>>
>> How's your idea?
> Looks good. Mind sending the patch?
Yep, let me cook the patches. Thanks.
>
> Thanks,
> Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs
2026-09-04 0:22 ` Mi, Dapeng
@ 2026-09-05 10:14 ` Gennady Kupava
0 siblings, 0 replies; 14+ messages in thread
From: Gennady Kupava @ 2026-09-05 10:14 UTC (permalink / raw)
To: Mi, Dapeng; +Cc: Ravi Bangoria, Namhyung Kim, linux-perf-users
Hi, perf people!
It was really interesting to read the conversation and especially nice
to see that there would be a patch.
Yeah, did not notice --unwind-style, and for sure one of the problems i
mentioned should be resolved by that option. May be i could do patch to
document it as a small contribution.
Overall i read all replies to date - not yet deeply understood them yet,
and my plan was to write just about libunwind vs libdw question - basically all
reasoning to remove libunwind makes sense, except that biggest problem
here is that we are switching from something which is working to
something which does not, which is leading to the fact that I as a user
have to go through some challenges using perf to get proper callgraphs.
And i could guess i am not alone with that. I checked, my debian
unstable system contains latest release of libdw. It was one release
behind when i wrote mail originally, but now it is latest.
I learned now about sframes approach and it really makes me really keen
to see it implemented as a way to record callchains. Recording the
entire stack always seems extremely inefficient thing to do - but to be
fair only side-effect of that is that we could do records much less
frequently and sometimes when apps put large amount of data into stack
it doesn't work. LBR somehow never worked for me, may be i need to
explore it more.
Now I am coming to the part which seems to interesting for me, and there
i made quite a discovery which completely reversed my opinion on fp.
My personal opinion on fp always was that 'ok, fine - but how much we
all should pay for operation which even I who is executing it more than
literally any developer I know - in my normal tasks'? So, i did small
test and results been really surprising for me.
Out of the interest, i just built something real - that osmand tile
renderer I was looking at, disabled CPU boost, fixed cpu to 3 GHz, made
40 runs for each fp/no fp and dropped the slowest 10 of each. Note
system and other libs not been built with fp - so not all code paying
cost of fp (by DSO, 71% of the samples fall in the rebuilt binary).
I built my engine with clang-21 and gcc-16.
clang-21 data:
instructions +3.9%
cycles +2.3%
wall time +2.3% (30 runs)
package energy +2.45% (30 runs)
.text +0.35%
L1i load misses -7.3%
prologues added +32k
If I was looking just to clang I would say - hey, 2.5% for 71% of code,
could estimate 3.3% if we rebuilt whole system - my system from 32 core
reduced to 31 core and I would be paying +3.3% energy bills! Debian is
very right not enabling fp!
However then I thought, ok osmand is android project but most people in
most places actually use gcc, what about gcc-16, and ended up in being
in light shock:
instructions +0.8% (1 run)
cycles +0.3% (1 run)
wall time -0.13% (30 runs)
package energy +0.22% (30 runs)
.text -0.54%
L1i load misses -13.7% (1 run)
prologues added +19k
So fp build is consistently faster, less cache misses!
Discussing with my invisible AI friend found the reason:
<begin of AI text>
The code got *smaller*, by 113 KB, and missed the instruction cache 14%
less often. The cause is in the instruction encoding. On x86-64 an
access through rsp needs an extra SIB byte; an access through rbp does
not, but its displacement is negative and often needs four bytes where
rsp's small positive offset fits in one. Measured over the whole binary,
the two effects leave a net saving of 0.19 bytes per stack access:
build via rsp via rbp
no FP 660,629 at 5.41 B 192,803 at 5.55 B
with FP 3,653 at 6.00 B 824,229 at 5.22 B
which works out as -320 KB on stack references, against +77 KB of extra
prologues (19k more functions at four bytes) and +127 KB of other
codegen differences - a net -116 KB, which is what the .text figure
shows.
<end of AI text>
So actually adding fp's to me look more like optimisation (smaller code,
negligible faster runtime with negligible more energy consumed) than
paying a toll for frames, taking in account gcc is still not doing
things in quite optimal way - it could potentially use both rbp and rsp
in different locations. May be I should also suggest fairly
straightforward patch to gcc to win even more with fp enabled.
I decided to ask AI to generate gcc patch to use rsp in case if it would
save instruction code (mixed mode gcc) and did another set of runs.
Legal note: I did not look into that patch, as it seems gcc forbids
non-trivial patches derived or generated by AI, just wanted to see what
would be an effect. I feel being capable enough to write one myself -
may be will try when I have more time, it looks like this worth much
more for all my systems than any kind of perf or osmand contribution.
Data below is different from above - but this is because I've found that
part of the code is using random hashes, which introduced more spread in
the runs above. Runs below are with the random seed fixed.
CODE SIZE:
build .text bytes vs nofp vs fp
nofp 21,467,662 - -
fp 21,351,934 -0.54% -
fp+mixed 21,194,254 -1.27% -0.74%
STACK VAR REFERENCES:
addressing form nofp fp fp+mixed
rbp, 1-byte displacement 92,248 489,962 489,962
rbp, 4-byte displacement 100,555 334,267 257,752
rbp, indexed 4,615 3,284 3,284
rsp, no displacement 47,872 575 3,547
rsp, 1-byte displacement 468,973 1,170 74,713
rsp, 4-byte displacement 143,784 1,908 1,908
rsp, indexed 3,191 66 66
total 861,238 831,232 831,232
INSTRUCTION COUNT:
build run 1 run 2 run 3
nofp 53,817,272,410 53,831,289,203 53,820,709,477
fp 54,747,413,314 54,749,169,058 54,740,499,791
mixed 54,745,775,977 54,748,063,729 54,743,584,686
build mean spread vs nofp vs fp
nofp 53,823,090,363 0.026% - -
fp 54,745,694,054 0.016% +1.71% -
mixed 54,745,808,131 0.008% +1.71% +0.00%
PERFORMANCE, best 10 of 20 by time:
build time ms sd energy J sd vs nofp vs nofp
time energy
nofp 5721.2 48.6 228.4 2.4 - -
fp 5683.6 71.9 227.3 2.9 -0.66% -0.47%
mixed 5728.7 33.0 228.9 1.0 +0.13% +0.22%
Now only question why smaller code runs slower - and further analysis
seems to show that this a bit random - mostly due to cache layout for
this specific task.
So my overall conclusion from this is:
1) enabling fp on amd64 on gcc is net zero cost in performance, less
code size -> net benefit. Very surprising. Nice.
2) makes sense to write and create patch to gcc for mixed addressing
(shifting rbp also makes sense, but that is ABI change)
3) why would anybody need sframes or whatever if fp is actually benefit?
Hope I did not miss anything interesting, will try to understand other
things in thread as next dive =)
Regards,
Gennady
On Fri, 4 Sept 2026 at 01:22, Mi, Dapeng <dapeng1.mi@linux.intel.com> wrote:
>
>
> On 9/4/2026 12:02 AM, Ravi Bangoria wrote:
> >> Suppose we need to follow below 2 rules to ensure the correct call-chain
> >> unwinding and register snapshot consistency.
> >>
> >> 1. All registers including IP should come from either PEBS or PMI, must not
> >> be mixed. This ensures to get an consistent registers snapshot.
> >>
> >> 2. Either SAMPLE_CALLCHAIN or SAMPLE_STACK_USER is required, the PMI
> >> registers snapshot must be used. This proves the correct DWARF unwinding.
> >>
> >> How's your idea?
> > Looks good. Mind sending the patch?
>
> Yep, let me cook the patches. Thanks.
>
>
> >
> > Thanks,
> > Ravi
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-09-05 10:14 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-21 21:49 [DISCUSSION] Three problems behind broken "perf --call-graph dwarf" on AMD: IP and stack dump mismatch, libdw fails on lld's layout, and the unwinder fallback never runs Gennady Kupava
2026-08-24 13:35 ` Ravi Bangoria
2026-08-25 18:05 ` Ian Rogers
2026-08-27 21:55 ` Namhyung Kim
2026-08-28 16:47 ` Namhyung Kim
2026-08-28 16:58 ` Namhyung Kim
2026-08-31 5:05 ` Ravi Bangoria
2026-09-01 15:19 ` Namhyung Kim
2026-09-03 8:39 ` Mi, Dapeng
2026-09-03 11:36 ` Ravi Bangoria
2026-09-03 11:59 ` Mi, Dapeng
2026-09-03 16:02 ` Ravi Bangoria
2026-09-04 0:22 ` Mi, Dapeng
2026-09-05 10:14 ` Gennady Kupava
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox