[PATCH v1] perf machine: Don't unwind the arm64 leaf frame caller without symbols

[PATCH v1] perf machine: Don't unwind the arm64 leaf frame caller without symbols

[Matthew Wood]

On arm64, 'perf record --call-graph fp' also samples LR so that
thread__resolve_callchain_sample() can recover the leaf function's
caller, which is missing from the frame pointer chain. It does so via
get_leaf_frame_caller_aarch64(), which runs a DWARF unwind. The
unwinder looks up the symbol of every frame it visits, so the first
lookup in a DSO loads its full symbol table and demangles every symbol
in it.

Commit 02b2705017d2 ("perf callchain: Allow symbols to be optional when
resolving a callchain") added the symbols argument so that callers which
only need maps can avoid loading symbols, and passed it down to every
add_callchain_ip() call. The leaf frame unwind was missed and still runs
for symbols=false, so 'perf inject' on arm64 has been loading symbols it
doesn't need ever since.

Commit 29be60c93d2d ("perf build-id: Mark DSO in sample callchains")
compounded the issue: process_buildids() now walks the callchain of
every sample at the end of 'perf record' to mark DSOs hit. With -a, that
loads and demangles the symbol tables of every user binary sampled on
the system. This can send things into exponential time and memory
(over 1GB of heap growth in 5 seconds), so

  perf record -a -e cycles:u --call-graph fp -- sleep 2

never finishes writing perf.data after the workload exits.

Callers that pass symbols=false only want the maps that callchain IPs
fall in, and add_callchain_ip() already honours that by using
thread__find_map() instead of thread__find_symbol(). Skip the leaf
frame unwind for them as well. The leaf frame caller is almost always
in the same DSO as the sampled IP or the next frame in the chain, so
build-id marking loses nothing in practice.

Fixes: 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain")
Signed-off-by: Matthew Wood <thepacketgeek@gmail.com>
---
 tools/perf/util/machine.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
index f86b3b7df742..51597cddd239 100644
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -2989,7 +2989,13 @@ static int thread__resolve_callchain_sample(struct thread *thread,

 		usr_idx = callchain_param.order == ORDER_CALLEE ? j-2 : j-1;

-		if (usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {
+		/*
+		 * Finding the leaf frame caller needs a DWARF unwind, and the
+		 * unwinder looks up the symbol of every frame it visits. Skip
+		 * it when the caller asked for maps only, as for build-id
+		 * marking, so that no symbol tables are loaded.
+		 */
+		if (symbols && usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {

 			leaf_frame_caller = get_leaf_frame_caller(sample, thread, usr_idx);

--
2.53.0-Meta

Re: [PATCH v1] perf machine: Don't unwind the arm64 leaf frame caller without symbols

[sashiko-bot]

> On arm64, 'perf record --call-graph fp' also samples LR so that
> thread__resolve_callchain_sample() can recover the leaf function's
> caller, which is missing from the frame pointer chain. It does so via
> get_leaf_frame_caller_aarch64(), which runs a DWARF unwind. The
> unwinder looks up the symbol of every frame it visits, so the first
> lookup in a DSO loads its full symbol table and demangles every symbol
> in it.
> 
> Commit 02b2705017d2 ("perf callchain: Allow symbols to be optional when
> resolving a callchain") added the symbols argument so that callers which
> only need maps can avoid loading symbols, and passed it down to every
> add_callchain_ip() call. The leaf frame unwind was missed and still runs
> for symbols=false, so 'perf inject' on arm64 has been loading symbols it
> doesn't need ever since.
> 
> [ ... ]
> in the same DSO as the sampled IP or the next frame in the chain, so
> build-id marking loses nothing in practice.
> 
> Fixes: 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain")
> Signed-off-by: Matthew Wood <thepacketgeek@gmail.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261014151326.4142140-1-thepacketgeek@gmail.com?part=1

Re: [PATCH v1] perf machine: Don't unwind the arm64 leaf frame caller without symbols

[Ian Rogers]

On Wed, Oct 14, 2026 at 8:13 AM Matthew Wood <thepacketgeek@gmail.com> wrote:
>
> On arm64, 'perf record --call-graph fp' also samples LR so that
> thread__resolve_callchain_sample() can recover the leaf function's
> caller, which is missing from the frame pointer chain. It does so via
> get_leaf_frame_caller_aarch64(), which runs a DWARF unwind. The
> unwinder looks up the symbol of every frame it visits, so the first
> lookup in a DSO loads its full symbol table and demangles every symbol
> in it.
>
> Commit 02b2705017d2 ("perf callchain: Allow symbols to be optional when
> resolving a callchain") added the symbols argument so that callers which
> only need maps can avoid loading symbols, and passed it down to every
> add_callchain_ip() call. The leaf frame unwind was missed and still runs
> for symbols=false, so 'perf inject' on arm64 has been loading symbols it
> doesn't need ever since.
>
> Commit 29be60c93d2d ("perf build-id: Mark DSO in sample callchains")
> compounded the issue: process_buildids() now walks the callchain of
> every sample at the end of 'perf record' to mark DSOs hit. With -a, that
> loads and demangles the symbol tables of every user binary sampled on
> the system. This can send things into exponential time and memory
> (over 1GB of heap growth in 5 seconds), so
>
>   perf record -a -e cycles:u --call-graph fp -- sleep 2
>
> never finishes writing perf.data after the workload exits.
>
> Callers that pass symbols=false only want the maps that callchain IPs
> fall in, and add_callchain_ip() already honours that by using
> thread__find_map() instead of thread__find_symbol(). Skip the leaf
> frame unwind for them as well. The leaf frame caller is almost always
> in the same DSO as the sampled IP or the next frame in the chain, so
> build-id marking loses nothing in practice.
>
> Fixes: 02b2705017d2 ("perf callchain: Allow symbols to be optional when resolving a callchain")
> Signed-off-by: Matthew Wood <thepacketgeek@gmail.com>
> ---
>  tools/perf/util/machine.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/tools/perf/util/machine.c b/tools/perf/util/machine.c
> index f86b3b7df742..51597cddd239 100644
> --- a/tools/perf/util/machine.c
> +++ b/tools/perf/util/machine.c
> @@ -2989,7 +2989,13 @@ static int thread__resolve_callchain_sample(struct thread *thread,
>
>                 usr_idx = callchain_param.order == ORDER_CALLEE ? j-2 : j-1;
>
> -               if (usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {
> +               /*
> +                * Finding the leaf frame caller needs a DWARF unwind, and the
> +                * unwinder looks up the symbol of every frame it visits. Skip
> +                * it when the caller asked for maps only, as for build-id
> +                * marking, so that no symbol tables are loaded.
> +                */
> +               if (symbols && usr_idx >= 0 && chain->ips[usr_idx] == PERF_CONTEXT_USER) {

Hi Matthew, I'm not sure if this is a serious patch or a bug report
about the slowness of DSO marking. Broadly DSO marking is known to be
slow and many perf tests run with -B and -N for this reason. I believe
the intent behind DSO marking as a default is to ensure that perf
provides the best profiling experience. This comes at a cost, but this
fix looks like it will break things because IPs will be lost.

Trying to look a little more at the ARM64 performance problem, I see
that we're trying to unwind 2 frames:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/arm64-frame-pointer-unwind-support.c?h=perf-tools-next#n70
but often just 1 is read. For libunwind the 1 frame case has a special case:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libunwind.c?h=perf-tools-next#n671
I wonder if we could replicate the 1 frame case with libdw to avoid
processing the DSO, and update the caller to be:
```
ret = unwind__get_entries(add_entry, &entries, thread, sample,
callchain_param.order == ORDER_CALLER ? 1 : 2, true);
```

Thanks,
Ian

>
>                         leaf_frame_caller = get_leaf_frame_caller(sample, thread, usr_idx);
>
> --
> 2.53.0-Meta

Re: [PATCH v1] perf machine: Don't unwind the arm64 leaf frame caller without symbols

[Matthew Wood]

On Wed, Oct 14, 2026 at 2:03 PM Ian Rogers <irogers@google.com> wrote:
>
> Hi Matthew, I'm not sure if this is a serious patch or a bug report
> about the slowness of DSO marking. Broadly DSO marking is known to be
> slow and many perf tests run with -B and -N for this reason. I believe
> the intent behind DSO marking as a default is to ensure that perf
> provides the best profiling experience. This comes at a cost, but this
> fix looks like it will break things because IPs will be lost.
>

Hi Ian,

Thanks for looking. It is definitely meant as an attempted fix. On a
128-CPU arm64 host, 'perf record -a -e cycles:u --call-graph fp -- sleep 2'
never finishes writing perf.data, and I'm trying to save some unnecessary work
to solve that. perf sits in __cxa_demangle() on one C++ symbol, with heap
growing by over 1GB every 5 seconds.

You're right about the lost IPs, thanks for mentioning. v1 drops the leaf
frame caller for symbols=false so a DSO that is only reached through that
frame wouldn't be marked.

> Trying to look a little more at the ARM64 performance problem, I see
> that we're trying to unwind 2 frames:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/arm64-frame-pointer-unwind-support.c?h=perf-tools-next#n70
> but often just 1 is read. For libunwind the 1 frame case has a special case:
> https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/unwind-libunwind.c?h=perf-tools-next#n671
> I wonder if we could replicate the 1 frame case with libdw to avoid
> processing the DSO, and update the caller to be:
> ```
> ret = unwind__get_entries(add_entry, &entries, thread, sample,
> callchain_param.order == ORDER_CALLER ? 1 : 2, true);
> ```

I looked at this but I don't think it would avoid the hang, unless I'm
misreading the code:

- The cost is in loading symbols, not in the DWARF unwind itself.
  libunwind's 1 frame case still goes through entry(), which calls
  thread__find_symbol(). libdw's __report_module() does the same for
  every frame. Either way, the DSO's symbol table gets loaded and every
  name in it demangled.

- callchain_param.order defaults to ORDER_CALLEE so process_buildids()
  would still ask for 2 frames.

- The 1 frame case returns only the sample PC. That's the IP we already
  have, not its caller, and get_leaf_frame_caller_aarch64() needs
  entries.length == 2. So in caller order I think it would stop
  recovering the leaf frame caller altogether.

I'm working on and testing a v2 that instead keeps the unwind and frames
as they are, but passes a symbols flag down to unwind__get_entries().
When it's false, libdw and libunwind look up only the map of each frame.
The leaf frame code only uses the addresses, so it never asks for symbols,
and thread__resolve_callchain_unwind() passes its symbols argument through.
That also removes the "Not resolving symbols with an unwinder isn't
currently supported" limitation for 'perf inject' with DWARF
callchains.

On the arm64 host, the 'perf record -a' command above now finishes. To
check that no IPs are lost, I used a small program where one DSO can
only be reached through the leaf frame caller:

  app -> libmid.so:mid() -> libleaf.so:leaf_spin()

leaf_spin() has no frame record, so the kernel's frame pointer chain
goes straight from leaf_spin() to main(). No samples land in mid()
itself, so libmid.so is only marked hit if the leaf frame caller is
kept. Running 'perf record --call-graph fp' without -N and checking
which DSOs end up in the build-id cache:

  unpatched: libmid.so cached
  v1:        libmid.so missing
  v2:        libmid.so cached

'perf script' output is identical between the unpatched perf and v2,
with mid() recovered in the same 2053 samples.

On x86 with libdw, 'perf inject -b' on a --call-graph dwarf profile goes
from 0.24s and 58.5MB max RSS to 0.04s and 15.5MB, with the same
build-ids injected. 'perf report' output and the dwarf unwind test are
unchanged.

I'll send v2 shortly.

Thanks,
Matthew