[PATCH] perf unwind-libdw: Fix unwinding of multi-threaded processes

[PATCH] perf unwind-libdw: Fix unwinding of multi-threaded processes

[Alessio Podda]

The libdw callback API has two levels: dwfl_getthread_frames() first finds
the requested thread using the next_thread() or get_thread() callback and
then walks its stack.

Since perf only has a snapshot of the stack of a single thread, it
provides a stubbed-out API that always returns the pid the Dwfl was
attached with (i.e. whatever was passed to dwfl_attach_state()), rather
than the actual sampled thread's TID.

Commit 6b2658b3f36a ("perf unwind-libdw: Don't discard loaded ELF/DWARF
after every unwind") changed libdw unwinding from recreating the Dwfl
object for each sample to caching it in struct maps, which is shared by
every thread in the process. It left next_thread() unchanged.

Since the pid passed to dwfl_attach_state() is only set at creation, only
the thread of the first sample is ever found. As a result,
dwfl_getthread_frames() fails with ESRCH when asked to unwind a sample
from another thread.

Make next_thread() return the current sample's TID, provide get_thread()
so libdw can find it directly, and pass the process PID expected by
dwfl_attach_state(). This allows libdw to unwind samples from every thread
in a multi-threaded process.

Fixes: 6b2658b3f36a ("perf unwind-libdw: Don't discard loaded ELF/DWARF after every unwind")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Alessio Podda <aleph.pi.gh@gmail.com>
---

A standalone reproducer is available at:

  https://github.com/kryggird/perf-bug-repro/tree/019c8ae97f9e4370ef2265d7badc3269e2a5044f

Run it with the perf binary being tested:

  ./reproduce.sh /path/to/perf

Results:

  perf 7.1.3-201.fc44.x86_64:
  1517 of 1522 samples have no userspace callchain

  patched perf 7.2.rc4.g1590cf032971:
  15 of 1449 samples have no userspace callchain

 tools/perf/util/unwind-libdw.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
index 7f35042be567..3735ef83c8c4 100644
--- a/tools/perf/util/unwind-libdw.c
+++ b/tools/perf/util/unwind-libdw.c
@@ -173,14 +173,30 @@ static int entry(u64 ip, struct unwind_info *ui)
 	return 0;
 }
 
-static pid_t next_thread(Dwfl *dwfl, void *arg, void **thread_argp)
+static pid_t next_thread(Dwfl *dwfl __maybe_unused, void *arg, void **thread_argp)
 {
+	struct dwfl_ui_thread_info *dwfl_ui_ti = arg;
+
 	/* We want only single thread to be processed. */
 	if (*thread_argp != NULL)
 		return 0;
 
+	assert(dwfl_ui_ti->ui != NULL);
 	*thread_argp = arg;
-	return dwfl_pid(dwfl);
+	return thread__tid(dwfl_ui_ti->ui->thread);
+}
+
+static bool get_thread(Dwfl *dwfl __maybe_unused, pid_t tid, void *arg,
+		       void **thread_argp)
+{
+	struct dwfl_ui_thread_info *dwfl_ui_ti = arg;
+
+	assert(dwfl_ui_ti->ui != NULL);
+	if (tid != thread__tid(dwfl_ui_ti->ui->thread))
+		return false;
+
+	*thread_argp = arg;
+	return true;
 }
 
 static int access_dso_mem(struct unwind_info *ui, Dwarf_Addr addr,
@@ -306,6 +322,7 @@ static bool libdw_set_initial_registers(Dwfl_Thread *thread, void *arg)
 
 static const Dwfl_Thread_Callbacks callbacks = {
 	.next_thread           = next_thread,
+	.get_thread            = get_thread,
 	.memory_read           = memory_read,
 	.set_initial_registers = libdw_set_initial_registers,
 };
@@ -400,7 +417,7 @@ int libdw__get_entries(unwind_entry_cb_t cb, void *arg,
 	if (err)
 		goto out;
 
-	dwfl_attach_state(dwfl, /*elf=*/NULL, thread__tid(thread), &callbacks,
+	dwfl_attach_state(dwfl, /*elf=*/NULL, thread__pid(thread), &callbacks,
 			  /* Dwfl thread function argument*/dwfl_ui_ti);
 	// Ignore thread already attached error.
 

base-commit: b95f03f04d475aa6719d15a636ddf32222d55657
-- 
2.55.0

Re: [PATCH] perf unwind-libdw: Fix unwinding of multi-threaded processes

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [Low] Missing explicit POSIX standard header inclusion for `<assert.h>`, breaking compatibility with musl libc.

Pre-existing issues:
- [High] Missing NULL check on `zalloc` allocation for `dwfl_ui_ti` leads to a NULL pointer dereference, and unchecked `dwfl_begin()` failure leads to a memory leak.
- [High] The local pointer `ui` is declared as a static variable but is dynamically allocated and freed within the function, creating a severe data race.
--

--- Patch [1]: [PATCH] perf unwind-libdw: Fix unwinding of multi-threaded processes ---
commit cd2051f3de0d807edd3e4fcc7ed115a6364be27d
Author: Alessio Podda <aleph.pi.gh@gmail.com>

perf unwind-libdw: Fix unwinding of multi-threaded processes

This commit fixes unwinding of multi-threaded processes in perf. It updates
next_thread() to return the current sample's TID and provides get_thread()
so libdw can find the thread directly.

> diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
> index 7f35042be5677..3735ef83c8c48 100644
> --- a/tools/perf/util/unwind-libdw.c
> +++ b/tools/perf/util/unwind-libdw.c
> @@ -173,14 +173,30 @@ static int entry(u64 ip, struct unwind_info *ui)
>  	return 0;
>  }
>  
> -static pid_t next_thread(Dwfl *dwfl, void *arg, void **thread_argp)
> +static pid_t next_thread(Dwfl *dwfl __maybe_unused, void *arg, void **thread_argp)
>  {
> +	struct dwfl_ui_thread_info *dwfl_ui_ti = arg;
> +
>  	/* We want only single thread to be processed. */
>  	if (*thread_argp != NULL)
>  		return 0;
>  
> +	assert(dwfl_ui_ti->ui != NULL);

[Severity: Low]
Does this file need an explicit include for <assert.h>? The perf subsystem
guidelines require direct POSIX header inclusions for compatibility with
musl libc, and it appears the assert macro is being introduced here without
it.

> +	*thread_argp = arg;
> -	return dwfl_pid(dwfl);
> +	return thread__tid(dwfl_ui_ti->ui->thread);
>  }

[ ... ]

> @@ -400,7 +417,7 @@ int libdw__get_entries(unwind_entry_cb_t cb, void *arg,
>  	if (err)
>  		goto out;
>  
> -	dwfl_attach_state(dwfl, /*elf=*/NULL, thread__tid(thread), &callbacks,
> +	dwfl_attach_state(dwfl, /*elf=*/NULL, thread__pid(thread), &callbacks,
>  			  /* Dwfl thread function argument*/dwfl_ui_ti);
>  	// Ignore thread already attached error.

[Severity: High]
This is a pre-existing issue, but can the allocation for dwfl_ui_ti fail
and lead to a NULL pointer dereference?

Looking at earlier code in libdw__get_entries():

	dwfl_ui_ti = zalloc(sizeof(*dwfl_ui_ti));
	dwfl = dwfl_begin(&offline_callbacks);
	if (!dwfl)
		goto out;

	dwfl_ui_ti->dwfl = dwfl;

If zalloc fails, it seems we would dereference the NULL dwfl_ui_ti right
away. Also, if dwfl_begin fails, we jump to out:

out:
...
	dwfl_ui_ti->ui = NULL;

which would also dereference NULL, or leak the allocated dwfl_ui_ti if
zalloc had succeeded. Does this error path need a NULL check and proper
cleanup?

[Severity: High]
This is a pre-existing issue, but does the static declaration of the ui
pointer cause data races?

In libdw__get_entries(), ui is declared as static but dynamically
allocated:

	static struct unwind_info *ui;
...
	ui = zalloc(sizeof(*ui) + sizeof(ui->entries[0]) * max_stack);

And then freed at the end of the same function:

	free(ui);

If multiple threads process events concurrently, could this static pointer
be overwritten, leading to memory leaks, use-after-free, or double-free
issues?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260722102407.3755154-1-aleph.pi.gh@gmail.com?part=1