[PATCH v3 0/7] perf: Add a libdw addr2line implementation

The Linux Kernel Mailing List
 help / color / mirror / Atom feed

* [PATCH v3 0/7] perf: Add a libdw addr2line implementation
@ 2026-01-11  4:13 Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 1/7] perf unwind-libdw: Fix invalid reference counts Ian Rogers
                   ` (7 more replies)
  0 siblings, 8 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

addr2line is a performance bottleneck in perf, add a libdw based
implementation that avoids forking addr2line and caches the decoded
debug information.

Allow the addr2line implementation to be picked via the configuration
file or --addr2line-style with `perf report`.

Test/fix that inline callchains are properly displayed by perf script.

An example:
```
$ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1
[ perf record: Woken up 132 times to write data ]
[ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ]
$ perf script --fields +srcline
...
perf-inlineloop 1814670 293100.228871:     640004 cpu_core/cycles/u: 
            55a11d6e61ee leaf+0x2e
  inlineloop.c:21 (inlined)
            55a11d6e61ee middle+0x2e
  inlineloop.c:27 (inlined)
            55a11d6e61ee parent+0x2e (perf)
  inlineloop.c:32
            55a11d6e629b inlineloop+0x8b (perf)
  inlineloop.c:47
            55a11d69a3bc run_workload+0x5a (perf)
  builtin-test.c:715
            55a11d69aa9f cmd_test+0x417 (perf)
  builtin-test.c:825
            55a11d6155f5 run_builtin+0xd4 (perf)
  perf.c:349
            55a11d61588d handle_internal_command+0xdd (perf)
  perf.c:401
            55a11d6159e6 run_argv+0x35 (perf)
  perf.c:445
            55a11d615d2f main+0x2cb (perf)
  perf.c:553
            7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6)
  libc_start_call_main.h:58
            7fae3d233d64 __libc_start_main_impl+0x84
  libc-start.c:360 (inlined)
            55a11d565f80 _start+0x20 (perf)
  ??:0
...
```

v3: Make the caller inline file and line number accurate in the libdw
    addr2line, rather than using the function's declared location.
    Fix reference counts in unwind-libdw. Add fixes tag for srcline
    inline printing.

v2: Fix bias issue with libdwfl functions. Use cu_walk_functions_at
    from perf's dwarf-aux to fully walk inline functions. Add testing
    that inlined functions are shown in the perf script srcline
    callchain information. Add configurability as to which addr2line
    style to use.
    https://lore.kernel.org/lkml/20260110082647.1487574-1-irogers@google.com/

v1: https://lore.kernel.org/lkml/20251122093934.94971-1-irogers@google.com/

Ian Rogers (7):
  perf unwind-libdw: Fix invalid reference counts
  perf addr2line: Add a libdw implementation
  perf addr2line.c: Rename a2l_style to cmd_a2l_style
  perf srcline: Add configuration support for the addr2line style
  perf callchain: Fix srcline printing with inlines
  perf test workload: Add inlineloop test workload
  perf test: Test addr2line unwinding works with inline functions

 tools/perf/builtin-report.c                 |  10 ++
 tools/perf/tests/builtin-test.c             |   1 +
 tools/perf/tests/shell/addr2line_inlines.sh |  47 ++++++
 tools/perf/tests/tests.h                    |   1 +
 tools/perf/tests/workloads/Build            |   2 +
 tools/perf/tests/workloads/inlineloop.c     |  52 +++++++
 tools/perf/util/Build                       |   1 +
 tools/perf/util/addr2line.c                 |  20 +--
 tools/perf/util/config.c                    |   4 +
 tools/perf/util/dso.c                       |   2 +
 tools/perf/util/dso.h                       |  11 ++
 tools/perf/util/evsel_fprintf.c             |   8 +-
 tools/perf/util/libdw.c                     | 153 ++++++++++++++++++++
 tools/perf/util/libdw.h                     |  60 ++++++++
 tools/perf/util/srcline.c                   | 116 ++++++++++++++-
 tools/perf/util/srcline.h                   |   3 +
 tools/perf/util/symbol_conf.h               |  10 ++
 tools/perf/util/unwind-libdw.c              |   7 +-
 18 files changed, 486 insertions(+), 22 deletions(-)
 create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh
 create mode 100644 tools/perf/tests/workloads/inlineloop.c
 create mode 100644 tools/perf/util/libdw.c
 create mode 100644 tools/perf/util/libdw.h

-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply	[flat|nested] 15+ messages in thread

* [PATCH v3 1/7] perf unwind-libdw: Fix invalid reference counts
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 2/7] perf addr2line: Add a libdw implementation Ian Rogers
                   ` (6 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

The addition of addr_location__exit causes use-after put on the maps
and map references in the unwind info. Add the gets and then add the
map_symbol__exits.

Fixes: 0dd5041c9a0e ("perf addr_location: Add init/exit/copy functions")
Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/util/unwind-libdw.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/unwind-libdw.c b/tools/perf/util/unwind-libdw.c
index ae70fb56a057..3ff427a49e4c 100644
--- a/tools/perf/util/unwind-libdw.c
+++ b/tools/perf/util/unwind-libdw.c
@@ -136,8 +136,8 @@ static int entry(u64 ip, struct unwind_info *ui)
 	}
 
 	e->ip	  = ip;
-	e->ms.maps = al.maps;
-	e->ms.map = al.map;
+	e->ms.maps = maps__get(al.maps);
+	e->ms.map = map__get(al.map);
 	e->ms.sym = al.sym;
 
 	pr_debug("unwind: %s:ip = 0x%" PRIx64 " (0x%" PRIx64 ")\n",
@@ -325,6 +325,9 @@ int unwind__get_entries(unwind_entry_cb_t cb, void *arg,
 	if (err)
 		pr_debug("unwind: failed with '%s'\n", dwfl_errmsg(-1));
 
+	for (i = 0; i < ui->idx; i++)
+		map_symbol__exit(&ui->entries[i].ms);
+
 	dwfl_end(ui->dwfl);
 	free(ui);
 	return 0;
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH v3 2/7] perf addr2line: Add a libdw implementation
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 1/7] perf unwind-libdw: Fix invalid reference counts Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-12 19:52   ` Arnaldo Carvalho de Melo
  2026-01-11  4:13 ` [PATCH v3 3/7] perf addr2line.c: Rename a2l_style to cmd_a2l_style Ian Rogers
                   ` (5 subsequent siblings)
  7 siblings, 1 reply; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

Add an implementation of addr2line that uses libdw. Other addr2line
implementations are slow, particularly in the case of forking
addr2line. Add an implementation that caches the libdw information in
the dso and uses it to find the file and line number
information. Inline information is supported but because
cu_walk_functions_at visits the leaf function last add a
inline_list__append_tail to reverse the lists order.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/util/Build     |   1 +
 tools/perf/util/dso.c     |   2 +
 tools/perf/util/dso.h     |  11 +++
 tools/perf/util/libdw.c   | 153 ++++++++++++++++++++++++++++++++++++++
 tools/perf/util/libdw.h   |  60 +++++++++++++++
 tools/perf/util/srcline.c |  24 ++++++
 tools/perf/util/srcline.h |   1 +
 7 files changed, 252 insertions(+)
 create mode 100644 tools/perf/util/libdw.c
 create mode 100644 tools/perf/util/libdw.h

diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 1c2a43e1dc68..2bed6274e248 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -224,6 +224,7 @@ perf-util-$(CONFIG_LIBDW) += dwarf-regs-powerpc.o
 perf-util-$(CONFIG_LIBDW) += dwarf-regs-x86.o
 perf-util-$(CONFIG_LIBDW) += debuginfo.o
 perf-util-$(CONFIG_LIBDW) += annotate-data.o
+perf-util-$(CONFIG_LIBDW) += libdw.o
 
 perf-util-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
 perf-util-$(CONFIG_LOCAL_LIBUNWIND)    += unwind-libunwind-local.o
diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 344e689567ee..06980844c014 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -32,6 +32,7 @@
 #include "string2.h"
 #include "vdso.h"
 #include "annotate-data.h"
+#include "libdw.h"
 
 static const char * const debuglink_paths[] = {
 	"%.0s%s",
@@ -1605,6 +1606,7 @@ void dso__delete(struct dso *dso)
 	auxtrace_cache__free(RC_CHK_ACCESS(dso)->auxtrace_cache);
 	dso_cache__free(dso);
 	dso__free_a2l(dso);
+	dso__free_a2l_libdw(dso);
 	dso__free_symsrc_filename(dso);
 	nsinfo__zput(RC_CHK_ACCESS(dso)->nsinfo);
 	mutex_destroy(dso__lock(dso));
diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
index f8ccb9816b89..4aee23775054 100644
--- a/tools/perf/util/dso.h
+++ b/tools/perf/util/dso.h
@@ -268,6 +268,7 @@ DECLARE_RC_STRUCT(dso) {
 	const char	 *short_name;
 	const char	 *long_name;
 	void		 *a2l;
+	void		 *a2l_libdw;
 	char		 *symsrc_filename;
 #if defined(__powerpc__)
 	void		*dwfl;			/* DWARF debug info */
@@ -334,6 +335,16 @@ static inline void dso__set_a2l(struct dso *dso, void *val)
 	RC_CHK_ACCESS(dso)->a2l = val;
 }
 
+static inline void *dso__a2l_libdw(const struct dso *dso)
+{
+	return RC_CHK_ACCESS(dso)->a2l_libdw;
+}
+
+static inline void dso__set_a2l_libdw(struct dso *dso, void *val)
+{
+	RC_CHK_ACCESS(dso)->a2l_libdw = val;
+}
+
 static inline unsigned int dso__a2l_fails(const struct dso *dso)
 {
 	return RC_CHK_ACCESS(dso)->a2l_fails;
diff --git a/tools/perf/util/libdw.c b/tools/perf/util/libdw.c
new file mode 100644
index 000000000000..e4bfd52bd172
--- /dev/null
+++ b/tools/perf/util/libdw.c
@@ -0,0 +1,153 @@
+// SPDX-License-Identifier: GPL-2.0
+#include "dso.h"
+#include "libdw.h"
+#include "srcline.h"
+#include "symbol.h"
+#include "dwarf-aux.h"
+#include <fcntl.h>
+#include <unistd.h>
+#include <elfutils/libdwfl.h>
+
+void dso__free_a2l_libdw(struct dso *dso)
+{
+	Dwfl *dwfl = dso__a2l_libdw(dso);
+
+	if (dwfl) {
+		dwfl_end(dwfl);
+		dso__set_a2l_libdw(dso, NULL);
+	}
+}
+
+struct libdw_a2l_cb_args {
+	struct dso *dso;
+	struct symbol *sym;
+	struct inline_node *node;
+	char *leaf_srcline;
+	bool leaf_srcline_used;
+};
+
+static int libdw_a2l_cb(Dwarf_Die *die, void *_args)
+{
+	struct libdw_a2l_cb_args *args  = _args;
+	struct symbol *inline_sym = new_inline_sym(args->dso, args->sym, dwarf_diename(die));
+	const char *call_fname = die_get_call_file(die);
+	char *call_srcline = srcline__unknown;
+	struct inline_list *ilist;
+
+	if (!inline_sym)
+		return -ENOMEM;
+
+	/* Assign caller information to the parent. */
+	if (call_fname)
+		call_srcline = srcline_from_fileline(call_fname, die_get_call_lineno(die));
+
+	list_for_each_entry(ilist, &args->node->val, list) {
+		ilist->srcline =  call_srcline;
+		call_srcline = NULL;
+		break;
+	}
+	if (call_srcline && call_fname)
+		free(call_srcline);
+
+	/* Add this symbol to the chain as the leaf. */
+	inline_list__append_tail(inline_sym, args->leaf_srcline, args->node);
+	args->leaf_srcline_used = true;
+	return 0;
+}
+
+int libdw__addr2line(const char *dso_name, u64 addr,
+		     char **file, unsigned int *line_nr,
+		     struct dso *dso, bool unwind_inlines,
+		     struct inline_node *node, struct symbol *sym)
+{
+	static const Dwfl_Callbacks offline_callbacks = {
+		.find_debuginfo = dwfl_standard_find_debuginfo,
+		.section_address = dwfl_offline_section_address,
+		.find_elf = dwfl_build_id_find_elf,
+	};
+	Dwfl *dwfl = dso__a2l_libdw(dso);
+	Dwfl_Module *mod;
+	Dwfl_Line *dwline;
+	Dwarf_Addr bias;
+	const char *src;
+	int lineno = 0;
+
+	if (!dwfl) {
+		/*
+		 * Initialize Dwfl session.
+		 * We need to open the DSO file to report it to libdw.
+		 */
+		int fd;
+
+		fd = open(dso_name, O_RDONLY);
+		if (fd < 0)
+			return 0;
+
+		dwfl = dwfl_begin(&offline_callbacks);
+		if (!dwfl) {
+			close(fd);
+			return 0;
+		}
+
+		/*
+		 * If the report is successful, the file descriptor fd is consumed
+		 * and closed by the Dwfl. If not, it is not closed.
+		 */
+		mod = dwfl_report_offline(dwfl, dso_name, dso_name, fd);
+		if (!mod) {
+			dwfl_end(dwfl);
+			close(fd);
+			return 0;
+		}
+
+		dwfl_report_end(dwfl, /*removed=*/NULL, /*arg=*/NULL);
+		dso__set_a2l_libdw(dso, dwfl);
+	} else {
+		/* Dwfl session already initialized, get module for address. */
+		mod = dwfl_addrmodule(dwfl, addr);
+	}
+
+	if (!mod)
+		return 0;
+
+	/*
+	 * Get/ignore the dwarf information. Determine the bias, difference
+	 * between the regular ELF addr2line addresses and those to use with
+	 * libdw.
+	 */
+	if (!dwfl_module_getdwarf(mod, &bias))
+		return 0;
+
+	/* Find source line information for the address. */
+	dwline = dwfl_module_getsrc(mod, addr + bias);
+	if (!dwline)
+		return 0;
+
+	/* Get line information. */
+	src = dwfl_lineinfo(dwline, /*addr=*/NULL, &lineno, /*col=*/NULL, /*mtime=*/NULL,
+			    /*length=*/NULL);
+
+	if (file)
+		*file = src ? strdup(src) : NULL;
+	if (line_nr)
+		*line_nr = lineno;
+
+	/* Optionally unwind inline function call chain. */
+	if (unwind_inlines && node) {
+		Dwarf_Addr unused_bias;
+		Dwarf_Die *cudie = dwfl_module_addrdie(mod, addr + bias, &unused_bias);
+		struct libdw_a2l_cb_args args = {
+			.dso = dso,
+			.sym = sym,
+			.node = node,
+			.leaf_srcline = srcline_from_fileline(src ?: "<unknown>", lineno),
+		};
+
+		/* Walk from the parent down to the leaf. */
+		cu_walk_functions_at(cudie, addr, libdw_a2l_cb, &args);
+
+		if (!args.leaf_srcline_used)
+			free(args.leaf_srcline);
+	}
+	return 1;
+}
diff --git a/tools/perf/util/libdw.h b/tools/perf/util/libdw.h
new file mode 100644
index 000000000000..0f8d7b4a11a5
--- /dev/null
+++ b/tools/perf/util/libdw.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef PERF_LIBDW_H
+#define PERF_LIBDW_H
+
+#include <linux/types.h>
+
+struct dso;
+struct inline_node;
+struct symbol;
+
+#ifdef HAVE_LIBDW_SUPPORT
+/*
+ * libdw__addr2line - Convert address to source location using libdw
+ * @dso_name: Name of the DSO
+ * @addr: Address to resolve
+ * @file: Pointer to return filename (caller must free)
+ * @line_nr: Pointer to return line number
+ * @dso: The dso struct
+ * @unwind_inlines: Whether to unwind inline function calls
+ * @node: Inline node list to append to
+ * @sym: The symbol associated with the address
+ *
+ * This function initializes a Dwfl context for the DSO if not already present,
+ * finds the source line information for the given address, and optionally
+ * resolves inline function call chains.
+ *
+ * Returns 1 on success (found), 0 on failure (not found).
+ */
+int libdw__addr2line(const char *dso_name, u64 addr, char **file,
+		     unsigned int *line_nr, struct dso *dso,
+		     bool unwind_inlines, struct inline_node *node,
+		     struct symbol *sym);
+
+/*
+ * dso__free_a2l_libdw - Free libdw resources associated with the DSO
+ * @dso: The dso to free resources for
+ *
+ * This function cleans up the Dwfl context used for addr2line lookups.
+ */
+void dso__free_a2l_libdw(struct dso *dso);
+
+#else /* HAVE_LIBDW_SUPPORT */
+
+static inline int libdw__addr2line(const char *dso_name __maybe_unused,
+				   u64 addr __maybe_unused, char **file __maybe_unused,
+				   unsigned int *line_nr __maybe_unused,
+				   struct dso *dso __maybe_unused,
+				   bool unwind_inlines __maybe_unused,
+				   struct inline_node *node __maybe_unused,
+				   struct symbol *sym __maybe_unused)
+{
+	return 0;
+}
+
+static inline void dso__free_a2l_libdw(struct dso *dso __maybe_unused)
+{
+}
+#endif /* HAVE_LIBDW_SUPPORT */
+
+#endif /* PERF_LIBDW_H */
diff --git a/tools/perf/util/srcline.c b/tools/perf/util/srcline.c
index 27c0966611ab..e2d280678b02 100644
--- a/tools/perf/util/srcline.c
+++ b/tools/perf/util/srcline.c
@@ -6,6 +6,7 @@
 #include "libbfd.h"
 #include "llvm.h"
 #include "symbol.h"
+#include "libdw.h"
 
 #include <inttypes.h>
 #include <string.h>
@@ -51,6 +52,25 @@ int inline_list__append(struct symbol *symbol, char *srcline, struct inline_node
 	return 0;
 }
 
+int inline_list__append_tail(struct symbol *symbol, char *srcline, struct inline_node *node)
+{
+	struct inline_list *ilist;
+
+	ilist = zalloc(sizeof(*ilist));
+	if (ilist == NULL)
+		return -1;
+
+	ilist->symbol = symbol;
+	ilist->srcline = srcline;
+
+	if (callchain_param.order == ORDER_CALLEE)
+		list_add(&ilist->list, &node->val);
+	else
+		list_add_tail(&ilist->list, &node->val);
+
+	return 0;
+}
+
 /* basename version that takes a const input string */
 static const char *gnu_basename(const char *path)
 {
@@ -120,6 +140,10 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *
 {
 	int ret;
 
+	ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
+	if (ret > 0)
+		return ret;
+
 	ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
 	if (ret > 0)
 		return ret;
diff --git a/tools/perf/util/srcline.h b/tools/perf/util/srcline.h
index c36f573cd339..be9f002bf234 100644
--- a/tools/perf/util/srcline.h
+++ b/tools/perf/util/srcline.h
@@ -57,6 +57,7 @@ struct inline_node *inlines__tree_find(struct rb_root_cached *tree, u64 addr);
 void inlines__tree_delete(struct rb_root_cached *tree);
 
 int inline_list__append(struct symbol *symbol, char *srcline, struct inline_node *node);
+int inline_list__append_tail(struct symbol *symbol, char *srcline, struct inline_node *node);
 char *srcline_from_fileline(const char *file, unsigned int line);
 struct symbol *new_inline_sym(struct dso *dso,
 			      struct symbol *base_sym,
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH v3 3/7] perf addr2line.c: Rename a2l_style to cmd_a2l_style
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 1/7] perf unwind-libdw: Fix invalid reference counts Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 2/7] perf addr2line: Add a libdw implementation Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 4/7] perf srcline: Add configuration support for the addr2line style Ian Rogers
                   ` (4 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

The a2l_style is only relevant to the command line version, so rename
to make this clearer.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/util/addr2line.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/tools/perf/util/addr2line.c b/tools/perf/util/addr2line.c
index f2d94a3272d7..0f1499350d47 100644
--- a/tools/perf/util/addr2line.c
+++ b/tools/perf/util/addr2line.c
@@ -90,16 +90,16 @@ static struct child_process *addr2line_subprocess_init(const char *addr2line_pat
 	return a2l;
 }
 
-enum a2l_style {
+enum cmd_a2l_style {
 	BROKEN,
 	GNU_BINUTILS,
 	LLVM,
 };
 
-static enum a2l_style addr2line_configure(struct child_process *a2l, const char *dso_name)
+static enum cmd_a2l_style cmd_addr2line_configure(struct child_process *a2l, const char *dso_name)
 {
 	static bool cached;
-	static enum a2l_style style;
+	static enum cmd_a2l_style style;
 
 	if (!cached) {
 		char buf[128];
@@ -149,7 +149,7 @@ static enum a2l_style addr2line_configure(struct child_process *a2l, const char
 }
 
 static int read_addr2line_record(struct io *io,
-				 enum a2l_style style,
+				 enum cmd_a2l_style style,
 				 const char *dso_name,
 				 u64 addr,
 				 bool first,
@@ -298,7 +298,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,
 	char buf[128];
 	ssize_t written;
 	struct io io = { .eof = false };
-	enum a2l_style a2l_style;
+	enum cmd_a2l_style cmd_a2l_style;
 
 	if (!a2l) {
 		if (!filename__has_section(dso_name, ".debug_line"))
@@ -314,8 +314,8 @@ int cmd__addr2line(const char *dso_name, u64 addr,
 			pr_warning("%s %s: addr2line_subprocess_init failed\n", __func__, dso_name);
 		goto out;
 	}
-	a2l_style = addr2line_configure(a2l, dso_name);
-	if (a2l_style == BROKEN)
+	cmd_a2l_style = cmd_addr2line_configure(a2l, dso_name);
+	if (cmd_a2l_style == BROKEN)
 		goto out;
 
 	/*
@@ -336,7 +336,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,
 	}
 	io__init(&io, a2l->out, buf, sizeof(buf));
 	io.timeout_ms = addr2line_timeout_ms;
-	switch (read_addr2line_record(&io, a2l_style, dso_name, addr, /*first=*/true,
+	switch (read_addr2line_record(&io, cmd_a2l_style, dso_name, addr, /*first=*/true,
 				      &record_function, &record_filename, &record_line_nr)) {
 	case -1:
 		if (!symbol_conf.disable_add2line_warn)
@@ -351,7 +351,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,
 		 * binutils, also force a non-zero address as we're no longer
 		 * reading that record.
 		 */
-		switch (read_addr2line_record(&io, a2l_style, dso_name,
+		switch (read_addr2line_record(&io, cmd_a2l_style, dso_name,
 					      /*addr=*/1, /*first=*/true,
 					      NULL, NULL, NULL)) {
 		case -1:
@@ -397,7 +397,7 @@ int cmd__addr2line(const char *dso_name, u64 addr,
 	 * as we're reading records beyond the first.
 	 */
 	while ((record_status = read_addr2line_record(&io,
-						      a2l_style,
+						      cmd_a2l_style,
 						      dso_name,
 						      /*addr=*/1,
 						      /*first=*/false,
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH v3 4/7] perf srcline: Add configuration support for the addr2line style
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
                   ` (2 preceding siblings ...)
  2026-01-11  4:13 ` [PATCH v3 3/7] perf addr2line.c: Rename a2l_style to cmd_a2l_style Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 5/7] perf callchain: Fix srcline printing with inlines Ian Rogers
                   ` (3 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

Allow the addr2line style to be specified on the `perf report` command
line or in the .perfconfig file.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/builtin-report.c   | 10 ++++
 tools/perf/util/config.c      |  4 ++
 tools/perf/util/srcline.c     | 98 +++++++++++++++++++++++++++++++----
 tools/perf/util/srcline.h     |  2 +
 tools/perf/util/symbol_conf.h | 10 ++++
 5 files changed, 113 insertions(+), 11 deletions(-)

diff --git a/tools/perf/builtin-report.c b/tools/perf/builtin-report.c
index 6c2b4f93ec78..2e936928e8c0 100644
--- a/tools/perf/builtin-report.c
+++ b/tools/perf/builtin-report.c
@@ -1271,6 +1271,13 @@ parse_percent_limit(const struct option *opt, const char *str,
 	return 0;
 }
 
+static int
+report_parse_addr2line_config(const struct option *opt __maybe_unused,
+			      const char *arg, int unset __maybe_unused)
+{
+	return addr2line_configure("addr2line.style", arg, NULL);
+}
+
 static int process_attr(const struct perf_tool *tool __maybe_unused,
 			union perf_event *event,
 			struct evlist **pevlist)
@@ -1447,6 +1454,9 @@ int cmd_report(int argc, const char **argv)
 		   "objdump binary to use for disassembly and annotations"),
 	OPT_STRING(0, "addr2line", &addr2line_path, "path",
 		   "addr2line binary to use for line numbers"),
+	OPT_CALLBACK(0, "addr2line-style", NULL, "addr2line style",
+		     "addr2line styles (libdw,llvm,libbfd,addr2line)",
+		     report_parse_addr2line_config),
 	OPT_BOOLEAN(0, "demangle", &symbol_conf.demangle,
 		    "Symbol demangling. Enabled by default, use --no-demangle to disable."),
 	OPT_BOOLEAN(0, "demangle-kernel", &symbol_conf.demangle_kernel,
diff --git a/tools/perf/util/config.c b/tools/perf/util/config.c
index e0219bc6330a..0452fbc6c085 100644
--- a/tools/perf/util/config.c
+++ b/tools/perf/util/config.c
@@ -20,6 +20,7 @@
 #include "util/stat.h"  /* perf_stat__set_big_num */
 #include "util/evsel.h"  /* evsel__hw_names, evsel__use_bpf_counters */
 #include "util/addr2line.h"  /* addr2line_timeout_ms */
+#include "srcline.h"
 #include "build-id.h"
 #include "debug.h"
 #include "config.h"
@@ -519,6 +520,9 @@ int perf_default_config(const char *var, const char *value,
 	if (strstarts(var, "stat."))
 		return perf_stat_config(var, value);
 
+	if (strstarts(var, "addr2line."))
+		return addr2line_configure(var, value, dummy);
+
 	/* Add other config variables here. */
 	return 0;
 }
diff --git a/tools/perf/util/srcline.c b/tools/perf/util/srcline.c
index e2d280678b02..28fa1abd1fd3 100644
--- a/tools/perf/util/srcline.c
+++ b/tools/perf/util/srcline.c
@@ -7,9 +7,11 @@
 #include "llvm.h"
 #include "symbol.h"
 #include "libdw.h"
+#include "debug.h"
 
 #include <inttypes.h>
 #include <string.h>
+#include <linux/string.h>
 
 bool srcline_full_filename;
 
@@ -138,21 +140,95 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *
 		     struct dso *dso, bool unwind_inlines, struct inline_node *node,
 		     struct symbol *sym)
 {
-	int ret;
+	int ret = 0;
+
+	if (symbol_conf.addr2line_style[0] == A2L_STYLE_UNKNOWN) {
+		int i = 0;
+
+		/* Default addr2line fallback order. */
+#ifdef HAVE_LIBDW_SUPPORT
+		symbol_conf.addr2line_style[i++] = A2L_STYLE_LIBDW;
+#endif
+#ifdef HAVE_LIBLLVM_SUPPORT
+		symbol_conf.addr2line_style[i++] = A2L_STYLE_LLVM;
+#endif
+#ifdef HAVE_LIBBFD_SUPPORT
+		symbol_conf.addr2line_style[i++] = A2L_STYLE_LIBBFD;
+#endif
+		symbol_conf.addr2line_style[i++] = A2L_STYLE_CMD;
+	}
+
+	for (size_t i = 0; i < ARRAY_SIZE(symbol_conf.addr2line_style); i++) {
+		switch (symbol_conf.addr2line_style[i]) {
+		case A2L_STYLE_LIBDW:
+			ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines,
+					       node, sym);
+			break;
+		case A2L_STYLE_LLVM:
+			ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines,
+					      node, sym);
+			break;
+		case A2L_STYLE_LIBBFD:
+			ret = libbfd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines,
+						node, sym);
+			break;
+		case A2L_STYLE_CMD:
+			ret = cmd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines,
+					     node, sym);
+			break;
+		case A2L_STYLE_UNKNOWN:
+		default:
+			break;
+		}
+		if (ret > 0)
+			return ret;
+	}
+
+	return 0;
+}
+
+int addr2line_configure(const char *var, const char *value, void *cb __maybe_unused)
+{
+	static const char * const a2l_style_names[] = {
+		[A2L_STYLE_LIBDW] = "libdw",
+		[A2L_STYLE_LLVM] = "llvm",
+		[A2L_STYLE_LIBBFD] = "libbfd",
+		[A2L_STYLE_CMD] = "addr2line",
+		NULL
+	};
+
+	char *s, *p, *saveptr;
+	size_t i = 0;
 
-	ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
-	if (ret > 0)
-		return ret;
+	if (strcmp(var, "addr2line.style"))
+		return 0;
+
+	if (!value)
+		return -1;
 
-	ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
-	if (ret > 0)
-		return ret;
+	s = strdup(value);
+	if (!s)
+		return -1;
 
-	ret = libbfd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
-	if (ret > 0)
-		return ret;
+	p = strtok_r(s, ",", &saveptr);
+	while (p && i < ARRAY_SIZE(symbol_conf.addr2line_style)) {
+		bool found = false;
+		char *q = strim(p);
+
+		for (size_t j = A2L_STYLE_LIBDW; j < MAX_A2L_STYLE; j++) {
+			if (!strcasecmp(q, a2l_style_names[j])) {
+				symbol_conf.addr2line_style[i++] = j;
+				found = true;
+				break;
+			}
+		}
+		if (!found)
+			pr_warning("Unknown addr2line style: %s\n", q);
+		p = strtok_r(NULL, ",", &saveptr);
+	}
 
-	return cmd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
+	free(s);
+	return 0;
 }
 
 static struct inline_node *addr2inlines(const char *dso_name, u64 addr,
diff --git a/tools/perf/util/srcline.h b/tools/perf/util/srcline.h
index be9f002bf234..7c37b3bf9ce7 100644
--- a/tools/perf/util/srcline.h
+++ b/tools/perf/util/srcline.h
@@ -63,4 +63,6 @@ struct symbol *new_inline_sym(struct dso *dso,
 			      struct symbol *base_sym,
 			      const char *funcname);
 
+int addr2line_configure(const char *var, const char *value, void *cb);
+
 #endif /* PERF_SRCLINE_H */
diff --git a/tools/perf/util/symbol_conf.h b/tools/perf/util/symbol_conf.h
index 7a80d2c14d9b..71bb17372a6c 100644
--- a/tools/perf/util/symbol_conf.h
+++ b/tools/perf/util/symbol_conf.h
@@ -9,6 +9,15 @@
 struct strlist;
 struct intlist;
 
+enum a2l_style {
+	A2L_STYLE_UNKNOWN = 0,
+	A2L_STYLE_LIBDW,
+	A2L_STYLE_LLVM,
+	A2L_STYLE_LIBBFD,
+	A2L_STYLE_CMD,
+};
+#define MAX_A2L_STYLE (A2L_STYLE_CMD + 1)
+
 struct symbol_conf {
 	bool		nanosecs;
 	unsigned short	priv_size;
@@ -70,6 +79,7 @@ struct symbol_conf {
 			*col_width_list_str,
 			*bt_stop_list_str;
 	const char		*addr2line_path;
+	enum a2l_style	addr2line_style[MAX_A2L_STYLE];
 	unsigned long	time_quantum;
        struct strlist	*dso_list,
 			*comm_list,
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH v3 5/7] perf callchain: Fix srcline printing with inlines
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
                   ` (3 preceding siblings ...)
  2026-01-11  4:13 ` [PATCH v3 4/7] perf srcline: Add configuration support for the addr2line style Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 6/7] perf test workload: Add inlineloop test workload Ian Rogers
                   ` (2 subsequent siblings)
  7 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

sample__fprintf_callchain was using map__fprintf_srcline which won't
report inline line numbers. Fix by using the srcline from the
callchain and falling back to the map variant.

Signed-off-by: Ian Rogers <irogers@google.com>
Fixes: 25da4fab5f66 ("perf evsel: Move fprintf methods to separate source file")
---
 tools/perf/util/evsel_fprintf.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/tools/perf/util/evsel_fprintf.c b/tools/perf/util/evsel_fprintf.c
index 10f1a03c2860..5521d00bff2c 100644
--- a/tools/perf/util/evsel_fprintf.c
+++ b/tools/perf/util/evsel_fprintf.c
@@ -185,8 +185,12 @@ int sample__fprintf_callchain(struct perf_sample *sample, int left_alignment,
 			if (print_dso && (!sym || !sym->inlined))
 				printed += map__fprintf_dsoname_dsoff(map, print_dsoff, addr, fp);
 
-			if (print_srcline)
-				printed += map__fprintf_srcline(map, addr, "\n  ", fp);
+			if (print_srcline) {
+				if (node->srcline)
+					printed += fprintf(fp, "\n  %s", node->srcline);
+				else
+					printed += map__fprintf_srcline(map, addr, "\n  ", fp);
+			}
 
 			if (sym && sym->inlined)
 				printed += fprintf(fp, " (inlined)");
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH v3 6/7] perf test workload: Add inlineloop test workload
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
                   ` (4 preceding siblings ...)
  2026-01-11  4:13 ` [PATCH v3 5/7] perf callchain: Fix srcline printing with inlines Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-11  4:13 ` [PATCH v3 7/7] perf test: Test addr2line unwinding works with inline functions Ian Rogers
  2026-01-12 11:18 ` [PATCH v3 0/7] perf: Add a libdw addr2line implementation James Clark
  7 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

The purpose of this workload is to gather samples in an inlined
function. This can be used to test whether inlined addr2line works
correctly.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/builtin-test.c         |  1 +
 tools/perf/tests/tests.h                |  1 +
 tools/perf/tests/workloads/Build        |  2 +
 tools/perf/tests/workloads/inlineloop.c | 52 +++++++++++++++++++++++++
 4 files changed, 56 insertions(+)
 create mode 100644 tools/perf/tests/workloads/inlineloop.c

diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index bd6ffa8e4578..e2490652f030 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -153,6 +153,7 @@ static struct test_workload *workloads[] = {
 	&workload__datasym,
 	&workload__landlock,
 	&workload__traploop,
+	&workload__inlineloop,
 };
 
 #define workloads__for_each(workload) \
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index cb67ddbd0375..1f0f8b267fb1 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -240,6 +240,7 @@ DECLARE_WORKLOAD(brstack);
 DECLARE_WORKLOAD(datasym);
 DECLARE_WORKLOAD(landlock);
 DECLARE_WORKLOAD(traploop);
+DECLARE_WORKLOAD(inlineloop);
 
 extern const char *dso_to_test;
 extern const char *test_objdump_path;
diff --git a/tools/perf/tests/workloads/Build b/tools/perf/tests/workloads/Build
index fb1012cc4fc3..866a00bd14a0 100644
--- a/tools/perf/tests/workloads/Build
+++ b/tools/perf/tests/workloads/Build
@@ -8,9 +8,11 @@ perf-test-y += brstack.o
 perf-test-y += datasym.o
 perf-test-y += landlock.o
 perf-test-y += traploop.o
+perf-test-y += inlineloop.o
 
 CFLAGS_sqrtloop.o         = -g -O0 -fno-inline -U_FORTIFY_SOURCE
 CFLAGS_leafloop.o         = -g -O0 -fno-inline -fno-omit-frame-pointer -U_FORTIFY_SOURCE
 CFLAGS_brstack.o          = -g -O0 -fno-inline -U_FORTIFY_SOURCE
 CFLAGS_datasym.o          = -g -O0 -fno-inline -U_FORTIFY_SOURCE
 CFLAGS_traploop.o         = -g -O0 -fno-inline -U_FORTIFY_SOURCE
+CFLAGS_inlineloop.o       = -g -O2
diff --git a/tools/perf/tests/workloads/inlineloop.c b/tools/perf/tests/workloads/inlineloop.c
new file mode 100644
index 000000000000..bc82dfc7c410
--- /dev/null
+++ b/tools/perf/tests/workloads/inlineloop.c
@@ -0,0 +1,52 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <pthread.h>
+#include <stdlib.h>
+#include <signal.h>
+#include <unistd.h>
+#include <linux/compiler.h>
+#include "../tests.h"
+
+static volatile int a;
+static volatile sig_atomic_t done;
+
+static void sighandler(int sig __maybe_unused)
+{
+	done = 1;
+}
+
+static inline void __attribute__((always_inline)) leaf(int b)
+{
+again:
+	a += b;
+	if (!done)
+		goto again;
+}
+
+static inline void __attribute__((always_inline)) middle(int b)
+{
+	leaf(b);
+}
+
+static noinline void parent(int b)
+{
+	middle(b);
+}
+
+static int inlineloop(int argc, const char **argv)
+{
+	int sec = 1;
+
+	pthread_setname_np(pthread_self(), "perf-inlineloop");
+	if (argc > 0)
+		sec = atoi(argv[0]);
+
+	signal(SIGINT, sighandler);
+	signal(SIGALRM, sighandler);
+	alarm(sec);
+
+	parent(sec);
+
+	return 0;
+}
+
+DEFINE_WORKLOAD(inlineloop);
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* [PATCH v3 7/7] perf test: Test addr2line unwinding works with inline functions
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
                   ` (5 preceding siblings ...)
  2026-01-11  4:13 ` [PATCH v3 6/7] perf test workload: Add inlineloop test workload Ian Rogers
@ 2026-01-11  4:13 ` Ian Rogers
  2026-01-12 11:18 ` [PATCH v3 0/7] perf: Add a libdw addr2line implementation James Clark
  7 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-11  4:13 UTC (permalink / raw)
  To: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Ian Rogers,
	Adrian Hunter, James Clark, Howard Chu, Stephen Brennan,
	linux-kernel, linux-perf-users

Add a test that seeks to see inline functions correctly displayed in
perf script from the inlineloop workload.

Signed-off-by: Ian Rogers <irogers@google.com>
---
 tools/perf/tests/shell/addr2line_inlines.sh | 47 +++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh

diff --git a/tools/perf/tests/shell/addr2line_inlines.sh b/tools/perf/tests/shell/addr2line_inlines.sh
new file mode 100755
index 000000000000..4a5b6f5be23d
--- /dev/null
+++ b/tools/perf/tests/shell/addr2line_inlines.sh
@@ -0,0 +1,47 @@
+#!/bin/bash
+# test addr2line inline unwinding
+# SPDX-License-Identifier: GPL-2.0
+
+set -e
+
+err=0
+test_dir=$(mktemp -d /tmp/perf-test-inline-addr2line.XXXXXXXXXX)
+perf_data="${test_dir}/perf.data"
+perf_script_txt="${test_dir}/perf_script.txt"
+
+cleanup() {
+    rm -rf "${test_dir}"
+    trap - EXIT TERM INT
+}
+
+trap_cleanup() {
+    echo "Unexpected signal in ${FUNCNAME[1]}"
+    cleanup
+    exit 1
+}
+trap trap_cleanup EXIT TERM INT
+
+test_inlinedloop() {
+    echo "Inline unwinding verification test"
+    # Record data. Currently only dwarf callchains support inlined functions.
+    perf record --call-graph dwarf -e task-clock:u -o "${perf_data}" -- perf test -w inlineloop 1
+
+    # Check output with inline (default) and srcline
+    perf script -i "${perf_data}" --fields +srcline > "${perf_script_txt}"
+
+    # Expect the leaf and middle functions to occur on lines in the 20s, with
+    # the non-inlined parent function on a line in the 30s.
+    if grep -q "inlineloop.c:2. (inlined)" "${perf_script_txt}" &&
+       grep -q "inlineloop.c:3.$" "${perf_script_txt}"
+    then
+        echo "Inline unwinding verification test [Success]"
+    else
+        echo "Inline unwinding verification test [Failed missing inlined functions]"
+        err=1
+    fi
+}
+
+test_inlinedloop
+
+cleanup
+exit $err
-- 
2.52.0.457.g6b5491de43-goog


^ permalink raw reply related	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 0/7] perf: Add a libdw addr2line implementation
  2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
                   ` (6 preceding siblings ...)
  2026-01-11  4:13 ` [PATCH v3 7/7] perf test: Test addr2line unwinding works with inline functions Ian Rogers
@ 2026-01-12 11:18 ` James Clark
  2026-01-12 14:49   ` Ian Rogers
  7 siblings, 1 reply; 15+ messages in thread
From: James Clark @ 2026-01-12 11:18 UTC (permalink / raw)
  To: Ian Rogers
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users



On 11/01/2026 4:13 am, Ian Rogers wrote:
> addr2line is a performance bottleneck in perf, add a libdw based
> implementation that avoids forking addr2line and caches the decoded
> debug information.
> 
> Allow the addr2line implementation to be picked via the configuration
> file or --addr2line-style with `perf report`.
> 
> Test/fix that inline callchains are properly displayed by perf script.
> 
> An example:
> ```
> $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1
> [ perf record: Woken up 132 times to write data ]
> [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ]
> $ perf script --fields +srcline
> ...
> perf-inlineloop 1814670 293100.228871:     640004 cpu_core/cycles/u:
>              55a11d6e61ee leaf+0x2e
>    inlineloop.c:21 (inlined)
>              55a11d6e61ee middle+0x2e
>    inlineloop.c:27 (inlined)
>              55a11d6e61ee parent+0x2e (perf)
>    inlineloop.c:32
>              55a11d6e629b inlineloop+0x8b (perf)
>    inlineloop.c:47
>              55a11d69a3bc run_workload+0x5a (perf)
>    builtin-test.c:715
>              55a11d69aa9f cmd_test+0x417 (perf)
>    builtin-test.c:825
>              55a11d6155f5 run_builtin+0xd4 (perf)
>    perf.c:349
>              55a11d61588d handle_internal_command+0xdd (perf)
>    perf.c:401
>              55a11d6159e6 run_argv+0x35 (perf)
>    perf.c:445
>              55a11d615d2f main+0x2cb (perf)
>    perf.c:553
>              7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6)
>    libc_start_call_main.h:58
>              7fae3d233d64 __libc_start_main_impl+0x84
>    libc-start.c:360 (inlined)
>              55a11d565f80 _start+0x20 (perf)
>    ??:0
> ...
> ```
> 
> v3: Make the caller inline file and line number accurate in the libdw
>      addr2line, rather than using the function's declared location.
>      Fix reference counts in unwind-libdw. Add fixes tag for srcline
>      inline printing.
> 
> v2: Fix bias issue with libdwfl functions. Use cu_walk_functions_at
>      from perf's dwarf-aux to fully walk inline functions. Add testing
>      that inlined functions are shown in the perf script srcline
>      callchain information. Add configurability as to which addr2line
>      style to use.
>      https://lore.kernel.org/lkml/20260110082647.1487574-1-irogers@google.com/
> 
> v1: https://lore.kernel.org/lkml/20251122093934.94971-1-irogers@google.com/
> 
> Ian Rogers (7):
>    perf unwind-libdw: Fix invalid reference counts
>    perf addr2line: Add a libdw implementation
>    perf addr2line.c: Rename a2l_style to cmd_a2l_style
>    perf srcline: Add configuration support for the addr2line style
>    perf callchain: Fix srcline printing with inlines
>    perf test workload: Add inlineloop test workload
>    perf test: Test addr2line unwinding works with inline functions
> 
>   tools/perf/builtin-report.c                 |  10 ++
>   tools/perf/tests/builtin-test.c             |   1 +
>   tools/perf/tests/shell/addr2line_inlines.sh |  47 ++++++
>   tools/perf/tests/tests.h                    |   1 +
>   tools/perf/tests/workloads/Build            |   2 +
>   tools/perf/tests/workloads/inlineloop.c     |  52 +++++++
>   tools/perf/util/Build                       |   1 +
>   tools/perf/util/addr2line.c                 |  20 +--
>   tools/perf/util/config.c                    |   4 +
>   tools/perf/util/dso.c                       |   2 +
>   tools/perf/util/dso.h                       |  11 ++
>   tools/perf/util/evsel_fprintf.c             |   8 +-
>   tools/perf/util/libdw.c                     | 153 ++++++++++++++++++++
>   tools/perf/util/libdw.h                     |  60 ++++++++
>   tools/perf/util/srcline.c                   | 116 ++++++++++++++-
>   tools/perf/util/srcline.h                   |   3 +
>   tools/perf/util/symbol_conf.h               |  10 ++
>   tools/perf/util/unwind-libdw.c              |   7 +-
>   18 files changed, 486 insertions(+), 22 deletions(-)
>   create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh
>   create mode 100644 tools/perf/tests/workloads/inlineloop.c
>   create mode 100644 tools/perf/util/libdw.c
>   create mode 100644 tools/perf/util/libdw.h
> 

I don't see the differences to the other addr2line implementations 
anymore, but only because it falls through to the old ones when libdw 
fails now.

For example when building Perf with LLVM it can't get the line in the 
inlineloop workload, and there's still a few things in libc and other 
system libraries it fails on.

But I think it's fine because it doesn't give the wrong line anymore, it 
just falls through to another working addr2line implementation.


Reviewed-by: James Clark <james.clark@linaro.org>


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 0/7] perf: Add a libdw addr2line implementation
  2026-01-12 11:18 ` [PATCH v3 0/7] perf: Add a libdw addr2line implementation James Clark
@ 2026-01-12 14:49   ` Ian Rogers
  2026-01-12 18:29     ` Ian Rogers
  0 siblings, 1 reply; 15+ messages in thread
From: Ian Rogers @ 2026-01-12 14:49 UTC (permalink / raw)
  To: James Clark
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users

On Mon, Jan 12, 2026 at 3:18 AM James Clark <james.clark@linaro.org> wrote:
>
> On 11/01/2026 4:13 am, Ian Rogers wrote:
> > addr2line is a performance bottleneck in perf, add a libdw based
> > implementation that avoids forking addr2line and caches the decoded
> > debug information.
> >
> > Allow the addr2line implementation to be picked via the configuration
> > file or --addr2line-style with `perf report`.
> >
> > Test/fix that inline callchains are properly displayed by perf script.
> >
> > An example:
> > ```
> > $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1
> > [ perf record: Woken up 132 times to write data ]
> > [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ]
> > $ perf script --fields +srcline
> > ...
> > perf-inlineloop 1814670 293100.228871:     640004 cpu_core/cycles/u:
> >              55a11d6e61ee leaf+0x2e
> >    inlineloop.c:21 (inlined)
> >              55a11d6e61ee middle+0x2e
> >    inlineloop.c:27 (inlined)
> >              55a11d6e61ee parent+0x2e (perf)
> >    inlineloop.c:32
> >              55a11d6e629b inlineloop+0x8b (perf)
> >    inlineloop.c:47
> >              55a11d69a3bc run_workload+0x5a (perf)
> >    builtin-test.c:715
> >              55a11d69aa9f cmd_test+0x417 (perf)
> >    builtin-test.c:825
> >              55a11d6155f5 run_builtin+0xd4 (perf)
> >    perf.c:349
> >              55a11d61588d handle_internal_command+0xdd (perf)
> >    perf.c:401
> >              55a11d6159e6 run_argv+0x35 (perf)
> >    perf.c:445
> >              55a11d615d2f main+0x2cb (perf)
> >    perf.c:553
> >              7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6)
> >    libc_start_call_main.h:58
> >              7fae3d233d64 __libc_start_main_impl+0x84
> >    libc-start.c:360 (inlined)
> >              55a11d565f80 _start+0x20 (perf)
> >    ??:0
> > ...
> > ```
> >
> > v3: Make the caller inline file and line number accurate in the libdw
> >      addr2line, rather than using the function's declared location.
> >      Fix reference counts in unwind-libdw. Add fixes tag for srcline
> >      inline printing.
> >
> > v2: Fix bias issue with libdwfl functions. Use cu_walk_functions_at
> >      from perf's dwarf-aux to fully walk inline functions. Add testing
> >      that inlined functions are shown in the perf script srcline
> >      callchain information. Add configurability as to which addr2line
> >      style to use.
> >      https://lore.kernel.org/lkml/20260110082647.1487574-1-irogers@google.com/
> >
> > v1: https://lore.kernel.org/lkml/20251122093934.94971-1-irogers@google.com/
> >
> > Ian Rogers (7):
> >    perf unwind-libdw: Fix invalid reference counts
> >    perf addr2line: Add a libdw implementation
> >    perf addr2line.c: Rename a2l_style to cmd_a2l_style
> >    perf srcline: Add configuration support for the addr2line style
> >    perf callchain: Fix srcline printing with inlines
> >    perf test workload: Add inlineloop test workload
> >    perf test: Test addr2line unwinding works with inline functions
> >
> >   tools/perf/builtin-report.c                 |  10 ++
> >   tools/perf/tests/builtin-test.c             |   1 +
> >   tools/perf/tests/shell/addr2line_inlines.sh |  47 ++++++
> >   tools/perf/tests/tests.h                    |   1 +
> >   tools/perf/tests/workloads/Build            |   2 +
> >   tools/perf/tests/workloads/inlineloop.c     |  52 +++++++
> >   tools/perf/util/Build                       |   1 +
> >   tools/perf/util/addr2line.c                 |  20 +--
> >   tools/perf/util/config.c                    |   4 +
> >   tools/perf/util/dso.c                       |   2 +
> >   tools/perf/util/dso.h                       |  11 ++
> >   tools/perf/util/evsel_fprintf.c             |   8 +-
> >   tools/perf/util/libdw.c                     | 153 ++++++++++++++++++++
> >   tools/perf/util/libdw.h                     |  60 ++++++++
> >   tools/perf/util/srcline.c                   | 116 ++++++++++++++-
> >   tools/perf/util/srcline.h                   |   3 +
> >   tools/perf/util/symbol_conf.h               |  10 ++
> >   tools/perf/util/unwind-libdw.c              |   7 +-
> >   18 files changed, 486 insertions(+), 22 deletions(-)
> >   create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh
> >   create mode 100644 tools/perf/tests/workloads/inlineloop.c
> >   create mode 100644 tools/perf/util/libdw.c
> >   create mode 100644 tools/perf/util/libdw.h
> >
>
> I don't see the differences to the other addr2line implementations
> anymore, but only because it falls through to the old ones when libdw
> fails now.
>
> For example when building Perf with LLVM it can't get the line in the
> inlineloop workload, and there's still a few things in libc and other
> system libraries it fails on.

Hmm.. I wonder what the issue is. I was looking at the dwarf output
from my gcc builds with llvm-dwarfdump. I wonder if LLVM builds are
doing something to confuse libdw? I'll try to investigate. There are
quite a few levels of libdw: there's the raw libdw, libdwfl (frontend
to libdw) that does the parsing and tries to give things like nested
debug scopes (libdwfl is the one needing addresses with a module bias
rather than raw file offsets), and then there is the dwarf-aux.c that
is in perf and is used by things like probe finding (I believe this
doesn't need biases addresses). Anyway, with the biases there are
things I can screw up (like in the v1 patch) but maybe the LLVM issue
is just a libdw and dwarf-5 kind of thing. Maybe it is ARM specific
:-/

> But I think it's fine because it doesn't give the wrong line anymore, it
> just falls through to another working addr2line implementation.

Just to confirm that with gcc builds it isn't failing now? ie it isn't
just an addr2line implementation that falls through all the time? I
was seeing things working/testing on x86 with gcc.

> Reviewed-by: James Clark <james.clark@linaro.org>

Thanks,
Ian

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 0/7] perf: Add a libdw addr2line implementation
  2026-01-12 14:49   ` Ian Rogers
@ 2026-01-12 18:29     ` Ian Rogers
  2026-01-13 12:03       ` James Clark
  0 siblings, 1 reply; 15+ messages in thread
From: Ian Rogers @ 2026-01-12 18:29 UTC (permalink / raw)
  To: James Clark
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users

On Mon, Jan 12, 2026 at 6:49 AM Ian Rogers <irogers@google.com> wrote:
>
> On Mon, Jan 12, 2026 at 3:18 AM James Clark <james.clark@linaro.org> wrote:
> >
> > On 11/01/2026 4:13 am, Ian Rogers wrote:
> > > addr2line is a performance bottleneck in perf, add a libdw based
> > > implementation that avoids forking addr2line and caches the decoded
> > > debug information.
> > >
> > > Allow the addr2line implementation to be picked via the configuration
> > > file or --addr2line-style with `perf report`.
> > >
> > > Test/fix that inline callchains are properly displayed by perf script.
> > >
> > > An example:
> > > ```
> > > $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1
> > > [ perf record: Woken up 132 times to write data ]
> > > [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ]
> > > $ perf script --fields +srcline
> > > ...
> > > perf-inlineloop 1814670 293100.228871:     640004 cpu_core/cycles/u:
> > >              55a11d6e61ee leaf+0x2e
> > >    inlineloop.c:21 (inlined)
> > >              55a11d6e61ee middle+0x2e
> > >    inlineloop.c:27 (inlined)
> > >              55a11d6e61ee parent+0x2e (perf)
> > >    inlineloop.c:32
> > >              55a11d6e629b inlineloop+0x8b (perf)
> > >    inlineloop.c:47
> > >              55a11d69a3bc run_workload+0x5a (perf)
> > >    builtin-test.c:715
> > >              55a11d69aa9f cmd_test+0x417 (perf)
> > >    builtin-test.c:825
> > >              55a11d6155f5 run_builtin+0xd4 (perf)
> > >    perf.c:349
> > >              55a11d61588d handle_internal_command+0xdd (perf)
> > >    perf.c:401
> > >              55a11d6159e6 run_argv+0x35 (perf)
> > >    perf.c:445
> > >              55a11d615d2f main+0x2cb (perf)
> > >    perf.c:553
> > >              7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6)
> > >    libc_start_call_main.h:58
> > >              7fae3d233d64 __libc_start_main_impl+0x84
> > >    libc-start.c:360 (inlined)
> > >              55a11d565f80 _start+0x20 (perf)
> > >    ??:0
> > > ...
> > > ```
> > >
> > > v3: Make the caller inline file and line number accurate in the libdw
> > >      addr2line, rather than using the function's declared location.
> > >      Fix reference counts in unwind-libdw. Add fixes tag for srcline
> > >      inline printing.
> > >
> > > v2: Fix bias issue with libdwfl functions. Use cu_walk_functions_at
> > >      from perf's dwarf-aux to fully walk inline functions. Add testing
> > >      that inlined functions are shown in the perf script srcline
> > >      callchain information. Add configurability as to which addr2line
> > >      style to use.
> > >      https://lore.kernel.org/lkml/20260110082647.1487574-1-irogers@google.com/
> > >
> > > v1: https://lore.kernel.org/lkml/20251122093934.94971-1-irogers@google.com/
> > >
> > > Ian Rogers (7):
> > >    perf unwind-libdw: Fix invalid reference counts
> > >    perf addr2line: Add a libdw implementation
> > >    perf addr2line.c: Rename a2l_style to cmd_a2l_style
> > >    perf srcline: Add configuration support for the addr2line style
> > >    perf callchain: Fix srcline printing with inlines
> > >    perf test workload: Add inlineloop test workload
> > >    perf test: Test addr2line unwinding works with inline functions
> > >
> > >   tools/perf/builtin-report.c                 |  10 ++
> > >   tools/perf/tests/builtin-test.c             |   1 +
> > >   tools/perf/tests/shell/addr2line_inlines.sh |  47 ++++++
> > >   tools/perf/tests/tests.h                    |   1 +
> > >   tools/perf/tests/workloads/Build            |   2 +
> > >   tools/perf/tests/workloads/inlineloop.c     |  52 +++++++
> > >   tools/perf/util/Build                       |   1 +
> > >   tools/perf/util/addr2line.c                 |  20 +--
> > >   tools/perf/util/config.c                    |   4 +
> > >   tools/perf/util/dso.c                       |   2 +
> > >   tools/perf/util/dso.h                       |  11 ++
> > >   tools/perf/util/evsel_fprintf.c             |   8 +-
> > >   tools/perf/util/libdw.c                     | 153 ++++++++++++++++++++
> > >   tools/perf/util/libdw.h                     |  60 ++++++++
> > >   tools/perf/util/srcline.c                   | 116 ++++++++++++++-
> > >   tools/perf/util/srcline.h                   |   3 +
> > >   tools/perf/util/symbol_conf.h               |  10 ++
> > >   tools/perf/util/unwind-libdw.c              |   7 +-
> > >   18 files changed, 486 insertions(+), 22 deletions(-)
> > >   create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh
> > >   create mode 100644 tools/perf/tests/workloads/inlineloop.c
> > >   create mode 100644 tools/perf/util/libdw.c
> > >   create mode 100644 tools/perf/util/libdw.h
> > >
> >
> > I don't see the differences to the other addr2line implementations
> > anymore, but only because it falls through to the old ones when libdw
> > fails now.
> >
> > For example when building Perf with LLVM it can't get the line in the
> > inlineloop workload, and there's still a few things in libc and other
> > system libraries it fails on.
>
> Hmm.. I wonder what the issue is. I was looking at the dwarf output
> from my gcc builds with llvm-dwarfdump. I wonder if LLVM builds are
> doing something to confuse libdw? I'll try to investigate. There are
> quite a few levels of libdw: there's the raw libdw, libdwfl (frontend
> to libdw) that does the parsing and tries to give things like nested
> debug scopes (libdwfl is the one needing addresses with a module bias
> rather than raw file offsets), and then there is the dwarf-aux.c that
> is in perf and is used by things like probe finding (I believe this
> doesn't need biases addresses). Anyway, with the biases there are
> things I can screw up (like in the v1 patch) but maybe the LLVM issue
> is just a libdw and dwarf-5 kind of thing. Maybe it is ARM specific
> :-/

Testing with clang/llvm on x86-64 (dwarf5):
```
$ make -C tools/perf O=/tmp/perf DEBUG=1 CC=clang CXX=clang++
HOSTCC=clang clean all
...
$ llvm-dwarfdump /tmp/perf/perf
...
0x0014f852: Compile Unit: length = 0x00000294, format = DWARF32,
version = 0x0005, unit_type = DW_UT_compile,
abbr_offset = 0x1879a, addr_size = 0x08 (next unit at 0x0014faea)

0x0014f85e: DW_TAG_compile_unit
             DW_AT_producer    ("Debian clang version 19.1.7 (3+build5)")
             DW_AT_language    (DW_LANG_C11)
             DW_AT_name        ("tests/workloads/inlineloop.c")
             DW_AT_str_offsets_base    (0x0004a550)
             DW_AT_stmt_list   (0x0008c3f2)
             DW_AT_comp_dir    ("linux/tools/perf")
             DW_AT_low_pc      (0x00000000001e61c0)
             DW_AT_high_pc     (0x00000000001e62e9)
             DW_AT_addr_base   (0x00022248)
             DW_AT_loclists_base       (0x0000018a)
...
$  sudo /tmp/perf/perf record --call-graph dwarf -e cycles:u --
/tmp/perf/perf test -w inlineloop 1
...
$  sudo /tmp/perf/perf script --fields +srcline
...
perf-inlineloop 2284167 423038.015394:     569917 cpu_core/cycles/u:
           56390020d2c6 leaf+0x26
 inlineloop.c:21 (inlined)
           56390020d2c6 middle+0x26
 inlineloop.c:27 (inlined)
           56390020d2c6 parent+0x26 (/tmp/perf/perf)
...
```
I ran inside of gdb and confirmed that the libdw code is creating the
inlined information (breakpoint on libdw_a2l_cb, etc.). So I'm not
able to reproduce the LLVM issue for now on x86-64.

Thanks,
Ian

> > But I think it's fine because it doesn't give the wrong line anymore, it
> > just falls through to another working addr2line implementation.
>
> Just to confirm that with gcc builds it isn't failing now? ie it isn't
> just an addr2line implementation that falls through all the time? I
> was seeing things working/testing on x86 with gcc.
>
> > Reviewed-by: James Clark <james.clark@linaro.org>
>
> Thanks,
> Ian

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 2/7] perf addr2line: Add a libdw implementation
  2026-01-11  4:13 ` [PATCH v3 2/7] perf addr2line: Add a libdw implementation Ian Rogers
@ 2026-01-12 19:52   ` Arnaldo Carvalho de Melo
  2026-01-12 21:23     ` Ian Rogers
  0 siblings, 1 reply; 15+ messages in thread
From: Arnaldo Carvalho de Melo @ 2026-01-12 19:52 UTC (permalink / raw)
  To: Ian Rogers
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Namhyung Kim,
	Alexander Shishkin, Jiri Olsa, Adrian Hunter, James Clark,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users

On Sat, Jan 10, 2026 at 08:13:33PM -0800, Ian Rogers wrote:
> Add an implementation of addr2line that uses libdw. Other addr2line
> implementations are slow, particularly in the case of forking
> addr2line. Add an implementation that caches the libdw information in
> the dso and uses it to find the file and line number
> information. Inline information is supported but because
> cu_walk_functions_at visits the leaf function last add a
> inline_list__append_tail to reverse the lists order.

So while testing I moved the inlineloop 'perf test' workload to the
front to test without this new libdw implementation and then with it,
and tested using perf probe on it:

root@number:/home/acme# perf probe -x ~/bin/perf libdw__addr2line%return 'ret=$retval'
Added new event:
  probe_perf:libdw_addr2line__return (on libdw__addr2line%return in /home/acme/bin/perf with ret=$retval)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:libdw_addr2line__return -aR sleep 1

root@number:/home/acme# perf trace -e probe_perf:libdw_addr2line__return perf report -f --dso perf --stdio -s srcfile,srcline
     0.000 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
     0.007 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.680 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.685 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.689 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.692 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.696 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.699 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.702 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.705 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.709 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.712 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.715 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.718 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.721 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.724 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.727 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.729 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.732 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.734 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.737 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.740 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
     0.771 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    13.563 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    13.766 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
    14.099 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
    14.103 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.105 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.134 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.143 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.242 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.244 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.245 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.246 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.248 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.250 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.251 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.252 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.258 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.260 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.265 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.267 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.268 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
    14.270 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 4K of event 'cpu/cycles/Pu'
# Event count (approx.): 5535180842
#
# Overhead  Source File   Source:Line    
# ........  ............  ...............
#
    99.04%  inlineloop.c  inlineloop.c:21
     0.46%  inlineloop.c  inlineloop.c:20


#
# (Tip: Save output of perf stat using: perf stat record <target workload>)
#
root@number:/home/acme#

Putting more probes to see the fallbacks...

root@number:/home/acme# perf probe -l
  probe_perf:cmd_addr2line (on cmd__addr2line@util/addr2line.c in /home/acme/bin/perf)
  probe_perf:libbfd_addr2line (on libbfd__addr2line@util/libbfd.h in /home/acme/bin/perf)
  probe_perf:libdw_addr2line (on libdw__addr2line@util/libdw.c in /home/acme/bin/perf)
  probe_perf:llvm_addr2line (on llvm__addr2line@util/llvm.c in /home/acme/bin/perf)
  probe_perf:libdw_addr2line__return (on libdw__addr2line%return@util/libdw.c in /home/acme/bin/perf with ret)
root@number:/home/acme#

As we have:

static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *line_nr,
                     struct dso *dso, bool unwind_inlines, struct inline_node *node,
                     struct symbol *sym)
{
        int ret;

        ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
        if (ret > 0)
                return ret;

        ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
        if (ret > 0)
                return ret;

        ret = libbfd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
        if (ret > 0)
                return ret;

        return cmd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
}

root@number:/home/acme# perf stat -e probe_perf:*_addr2line perf report -f --dso perf --stdio -s srcfile,srcline
# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 4K of event 'cpu/cycles/Pu'
# Event count (approx.): 5535180842
#
# Overhead  Source File   Source:Line    
# ........  ............  ...............
#
    99.04%  inlineloop.c  inlineloop.c:21
     0.46%  inlineloop.c  inlineloop.c:20


#
# (Tip: To set sampling period of individual events use perf record -e cpu/cpu-cycles,period=100001/,cpu/branches,period=10001/ ...)
#

 Performance counter stats for 'perf report -f --dso perf --stdio -s srcfile,srcline':

                44      probe_perf:libdw_addr2line                                            
                23      probe_perf:llvm_addr2line                                             
                 0      probe_perf:libbfd_addr2line                                           
                 0      probe_perf:cmd_addr2line                                              

       0.036554827 seconds time elapsed

       0.030570000 seconds user
       0.006049000 seconds sys


root@number:/home/acme#

Maybe asking for the dso_name may help understand why it is falling back
23 times to the llvm based one...

root@number:/home/acme# perf probe -x ~/bin/perf llvm__addr2line dso_name:string
Target program is compiled without optimization. Skipping prologue.
Probe on address 0x5f73e4 to force probing at the function entry.

Added new event:
  probe_perf:llvm_addr2line (on llvm__addr2line in /home/acme/bin/perf with dso_name:string)

You can now use it in all perf tools, such as:

	perf record -e probe_perf:llvm_addr2line -aR sleep 1

root@number:/home/acme# perf trace --libtrace -e probe_perf:llvm_addr2line perf report -f --dso perf --stdio -s srcfile,srcline
     0.000 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
     0.697 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
     0.704 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
     0.708 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
     0.712 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
<SNIP>

root@number:/home/acme# readelf -wi /root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf | head -30
Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x20 (32-bit)
   Version:       5
   Unit Type:     DW_UT_partial (3)
   Abbrev Offset: 0xad87
   Pointer Size:  8
 <0><c>: Abbrev Number: 38 (DW_TAG_partial_unit)
    <d>   DW_AT_stmt_list   : 0
    <11>   DW_AT_comp_dir    : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf
 <1><15>: Abbrev Number: 57 (DW_TAG_base_type)
    <16>   DW_AT_byte_size   : 8
    <17>   DW_AT_encoding    : 7	(unsigned)
    <18>   DW_AT_name        : (indirect string, offset: 0x9bf1): long unsigned int
 <1><1c>: Abbrev Number: 57 (DW_TAG_base_type)
    <1d>   DW_AT_byte_size   : 1
    <1e>   DW_AT_encoding    : 6	(signed char)
    <1f>   DW_AT_name        : (indirect string, offset: 0x7bab): char
 <1><23>: Abbrev Number: 0
  Compilation Unit @ offset 0x24:
   Length:        0x20 (32-bit)
   Version:       5
   Unit Type:     DW_UT_partial (3)
   Abbrev Offset: 0xad87
   Pointer Size:  8
 <0><30>: Abbrev Number: 38 (DW_TAG_partial_unit)
    <31>   DW_AT_stmt_list   : 0
    <35>   DW_AT_comp_dir    : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf
 <1><39>: Abbrev Number: 57 (DW_TAG_base_type)
root@number:/home/acme#

Some partial unit glibc DWARF 5. I think we need some more libdw setup
to handle those?

But then we can merge what James already reviewed and go on from here,
right?

- Arnaldo
 
> Signed-off-by: Ian Rogers <irogers@google.com>
> ---
>  tools/perf/util/Build     |   1 +
>  tools/perf/util/dso.c     |   2 +
>  tools/perf/util/dso.h     |  11 +++
>  tools/perf/util/libdw.c   | 153 ++++++++++++++++++++++++++++++++++++++
>  tools/perf/util/libdw.h   |  60 +++++++++++++++
>  tools/perf/util/srcline.c |  24 ++++++
>  tools/perf/util/srcline.h |   1 +
>  7 files changed, 252 insertions(+)
>  create mode 100644 tools/perf/util/libdw.c
>  create mode 100644 tools/perf/util/libdw.h
> 
> diff --git a/tools/perf/util/Build b/tools/perf/util/Build
> index 1c2a43e1dc68..2bed6274e248 100644
> --- a/tools/perf/util/Build
> +++ b/tools/perf/util/Build
> @@ -224,6 +224,7 @@ perf-util-$(CONFIG_LIBDW) += dwarf-regs-powerpc.o
>  perf-util-$(CONFIG_LIBDW) += dwarf-regs-x86.o
>  perf-util-$(CONFIG_LIBDW) += debuginfo.o
>  perf-util-$(CONFIG_LIBDW) += annotate-data.o
> +perf-util-$(CONFIG_LIBDW) += libdw.o
>  
>  perf-util-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
>  perf-util-$(CONFIG_LOCAL_LIBUNWIND)    += unwind-libunwind-local.o
> diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> index 344e689567ee..06980844c014 100644
> --- a/tools/perf/util/dso.c
> +++ b/tools/perf/util/dso.c
> @@ -32,6 +32,7 @@
>  #include "string2.h"
>  #include "vdso.h"
>  #include "annotate-data.h"
> +#include "libdw.h"
>  
>  static const char * const debuglink_paths[] = {
>  	"%.0s%s",
> @@ -1605,6 +1606,7 @@ void dso__delete(struct dso *dso)
>  	auxtrace_cache__free(RC_CHK_ACCESS(dso)->auxtrace_cache);
>  	dso_cache__free(dso);
>  	dso__free_a2l(dso);
> +	dso__free_a2l_libdw(dso);
>  	dso__free_symsrc_filename(dso);
>  	nsinfo__zput(RC_CHK_ACCESS(dso)->nsinfo);
>  	mutex_destroy(dso__lock(dso));
> diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
> index f8ccb9816b89..4aee23775054 100644
> --- a/tools/perf/util/dso.h
> +++ b/tools/perf/util/dso.h
> @@ -268,6 +268,7 @@ DECLARE_RC_STRUCT(dso) {
>  	const char	 *short_name;
>  	const char	 *long_name;
>  	void		 *a2l;
> +	void		 *a2l_libdw;
>  	char		 *symsrc_filename;
>  #if defined(__powerpc__)
>  	void		*dwfl;			/* DWARF debug info */
> @@ -334,6 +335,16 @@ static inline void dso__set_a2l(struct dso *dso, void *val)
>  	RC_CHK_ACCESS(dso)->a2l = val;
>  }
>  
> +static inline void *dso__a2l_libdw(const struct dso *dso)
> +{
> +	return RC_CHK_ACCESS(dso)->a2l_libdw;
> +}
> +
> +static inline void dso__set_a2l_libdw(struct dso *dso, void *val)
> +{
> +	RC_CHK_ACCESS(dso)->a2l_libdw = val;
> +}
> +
>  static inline unsigned int dso__a2l_fails(const struct dso *dso)
>  {
>  	return RC_CHK_ACCESS(dso)->a2l_fails;
> diff --git a/tools/perf/util/libdw.c b/tools/perf/util/libdw.c
> new file mode 100644
> index 000000000000..e4bfd52bd172
> --- /dev/null
> +++ b/tools/perf/util/libdw.c
> @@ -0,0 +1,153 @@
> +// SPDX-License-Identifier: GPL-2.0
> +#include "dso.h"
> +#include "libdw.h"
> +#include "srcline.h"
> +#include "symbol.h"
> +#include "dwarf-aux.h"
> +#include <fcntl.h>
> +#include <unistd.h>
> +#include <elfutils/libdwfl.h>
> +
> +void dso__free_a2l_libdw(struct dso *dso)
> +{
> +	Dwfl *dwfl = dso__a2l_libdw(dso);
> +
> +	if (dwfl) {
> +		dwfl_end(dwfl);
> +		dso__set_a2l_libdw(dso, NULL);
> +	}
> +}
> +
> +struct libdw_a2l_cb_args {
> +	struct dso *dso;
> +	struct symbol *sym;
> +	struct inline_node *node;
> +	char *leaf_srcline;
> +	bool leaf_srcline_used;
> +};
> +
> +static int libdw_a2l_cb(Dwarf_Die *die, void *_args)
> +{
> +	struct libdw_a2l_cb_args *args  = _args;
> +	struct symbol *inline_sym = new_inline_sym(args->dso, args->sym, dwarf_diename(die));
> +	const char *call_fname = die_get_call_file(die);
> +	char *call_srcline = srcline__unknown;
> +	struct inline_list *ilist;
> +
> +	if (!inline_sym)
> +		return -ENOMEM;
> +
> +	/* Assign caller information to the parent. */
> +	if (call_fname)
> +		call_srcline = srcline_from_fileline(call_fname, die_get_call_lineno(die));
> +
> +	list_for_each_entry(ilist, &args->node->val, list) {
> +		ilist->srcline =  call_srcline;
> +		call_srcline = NULL;
> +		break;
> +	}
> +	if (call_srcline && call_fname)
> +		free(call_srcline);
> +
> +	/* Add this symbol to the chain as the leaf. */
> +	inline_list__append_tail(inline_sym, args->leaf_srcline, args->node);
> +	args->leaf_srcline_used = true;
> +	return 0;
> +}
> +
> +int libdw__addr2line(const char *dso_name, u64 addr,
> +		     char **file, unsigned int *line_nr,
> +		     struct dso *dso, bool unwind_inlines,
> +		     struct inline_node *node, struct symbol *sym)
> +{
> +	static const Dwfl_Callbacks offline_callbacks = {
> +		.find_debuginfo = dwfl_standard_find_debuginfo,
> +		.section_address = dwfl_offline_section_address,
> +		.find_elf = dwfl_build_id_find_elf,
> +	};
> +	Dwfl *dwfl = dso__a2l_libdw(dso);
> +	Dwfl_Module *mod;
> +	Dwfl_Line *dwline;
> +	Dwarf_Addr bias;
> +	const char *src;
> +	int lineno = 0;
> +
> +	if (!dwfl) {
> +		/*
> +		 * Initialize Dwfl session.
> +		 * We need to open the DSO file to report it to libdw.
> +		 */
> +		int fd;
> +
> +		fd = open(dso_name, O_RDONLY);
> +		if (fd < 0)
> +			return 0;
> +
> +		dwfl = dwfl_begin(&offline_callbacks);
> +		if (!dwfl) {
> +			close(fd);
> +			return 0;
> +		}
> +
> +		/*
> +		 * If the report is successful, the file descriptor fd is consumed
> +		 * and closed by the Dwfl. If not, it is not closed.
> +		 */
> +		mod = dwfl_report_offline(dwfl, dso_name, dso_name, fd);
> +		if (!mod) {
> +			dwfl_end(dwfl);
> +			close(fd);
> +			return 0;
> +		}
> +
> +		dwfl_report_end(dwfl, /*removed=*/NULL, /*arg=*/NULL);
> +		dso__set_a2l_libdw(dso, dwfl);
> +	} else {
> +		/* Dwfl session already initialized, get module for address. */
> +		mod = dwfl_addrmodule(dwfl, addr);
> +	}
> +
> +	if (!mod)
> +		return 0;
> +
> +	/*
> +	 * Get/ignore the dwarf information. Determine the bias, difference
> +	 * between the regular ELF addr2line addresses and those to use with
> +	 * libdw.
> +	 */
> +	if (!dwfl_module_getdwarf(mod, &bias))
> +		return 0;
> +
> +	/* Find source line information for the address. */
> +	dwline = dwfl_module_getsrc(mod, addr + bias);
> +	if (!dwline)
> +		return 0;
> +
> +	/* Get line information. */
> +	src = dwfl_lineinfo(dwline, /*addr=*/NULL, &lineno, /*col=*/NULL, /*mtime=*/NULL,
> +			    /*length=*/NULL);
> +
> +	if (file)
> +		*file = src ? strdup(src) : NULL;
> +	if (line_nr)
> +		*line_nr = lineno;
> +
> +	/* Optionally unwind inline function call chain. */
> +	if (unwind_inlines && node) {
> +		Dwarf_Addr unused_bias;
> +		Dwarf_Die *cudie = dwfl_module_addrdie(mod, addr + bias, &unused_bias);
> +		struct libdw_a2l_cb_args args = {
> +			.dso = dso,
> +			.sym = sym,
> +			.node = node,
> +			.leaf_srcline = srcline_from_fileline(src ?: "<unknown>", lineno),
> +		};
> +
> +		/* Walk from the parent down to the leaf. */
> +		cu_walk_functions_at(cudie, addr, libdw_a2l_cb, &args);
> +
> +		if (!args.leaf_srcline_used)
> +			free(args.leaf_srcline);
> +	}
> +	return 1;
> +}
> diff --git a/tools/perf/util/libdw.h b/tools/perf/util/libdw.h
> new file mode 100644
> index 000000000000..0f8d7b4a11a5
> --- /dev/null
> +++ b/tools/perf/util/libdw.h
> @@ -0,0 +1,60 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef PERF_LIBDW_H
> +#define PERF_LIBDW_H
> +
> +#include <linux/types.h>
> +
> +struct dso;
> +struct inline_node;
> +struct symbol;
> +
> +#ifdef HAVE_LIBDW_SUPPORT
> +/*
> + * libdw__addr2line - Convert address to source location using libdw
> + * @dso_name: Name of the DSO
> + * @addr: Address to resolve
> + * @file: Pointer to return filename (caller must free)
> + * @line_nr: Pointer to return line number
> + * @dso: The dso struct
> + * @unwind_inlines: Whether to unwind inline function calls
> + * @node: Inline node list to append to
> + * @sym: The symbol associated with the address
> + *
> + * This function initializes a Dwfl context for the DSO if not already present,
> + * finds the source line information for the given address, and optionally
> + * resolves inline function call chains.
> + *
> + * Returns 1 on success (found), 0 on failure (not found).
> + */
> +int libdw__addr2line(const char *dso_name, u64 addr, char **file,
> +		     unsigned int *line_nr, struct dso *dso,
> +		     bool unwind_inlines, struct inline_node *node,
> +		     struct symbol *sym);
> +
> +/*
> + * dso__free_a2l_libdw - Free libdw resources associated with the DSO
> + * @dso: The dso to free resources for
> + *
> + * This function cleans up the Dwfl context used for addr2line lookups.
> + */
> +void dso__free_a2l_libdw(struct dso *dso);
> +
> +#else /* HAVE_LIBDW_SUPPORT */
> +
> +static inline int libdw__addr2line(const char *dso_name __maybe_unused,
> +				   u64 addr __maybe_unused, char **file __maybe_unused,
> +				   unsigned int *line_nr __maybe_unused,
> +				   struct dso *dso __maybe_unused,
> +				   bool unwind_inlines __maybe_unused,
> +				   struct inline_node *node __maybe_unused,
> +				   struct symbol *sym __maybe_unused)
> +{
> +	return 0;
> +}
> +
> +static inline void dso__free_a2l_libdw(struct dso *dso __maybe_unused)
> +{
> +}
> +#endif /* HAVE_LIBDW_SUPPORT */
> +
> +#endif /* PERF_LIBDW_H */
> diff --git a/tools/perf/util/srcline.c b/tools/perf/util/srcline.c
> index 27c0966611ab..e2d280678b02 100644
> --- a/tools/perf/util/srcline.c
> +++ b/tools/perf/util/srcline.c
> @@ -6,6 +6,7 @@
>  #include "libbfd.h"
>  #include "llvm.h"
>  #include "symbol.h"
> +#include "libdw.h"
>  
>  #include <inttypes.h>
>  #include <string.h>
> @@ -51,6 +52,25 @@ int inline_list__append(struct symbol *symbol, char *srcline, struct inline_node
>  	return 0;
>  }
>  
> +int inline_list__append_tail(struct symbol *symbol, char *srcline, struct inline_node *node)
> +{
> +	struct inline_list *ilist;
> +
> +	ilist = zalloc(sizeof(*ilist));
> +	if (ilist == NULL)
> +		return -1;
> +
> +	ilist->symbol = symbol;
> +	ilist->srcline = srcline;
> +
> +	if (callchain_param.order == ORDER_CALLEE)
> +		list_add(&ilist->list, &node->val);
> +	else
> +		list_add_tail(&ilist->list, &node->val);
> +
> +	return 0;
> +}
> +
>  /* basename version that takes a const input string */
>  static const char *gnu_basename(const char *path)
>  {
> @@ -120,6 +140,10 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *
>  {
>  	int ret;
>  
> +	ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
> +	if (ret > 0)
> +		return ret;
> +
>  	ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
>  	if (ret > 0)
>  		return ret;
> diff --git a/tools/perf/util/srcline.h b/tools/perf/util/srcline.h
> index c36f573cd339..be9f002bf234 100644
> --- a/tools/perf/util/srcline.h
> +++ b/tools/perf/util/srcline.h
> @@ -57,6 +57,7 @@ struct inline_node *inlines__tree_find(struct rb_root_cached *tree, u64 addr);
>  void inlines__tree_delete(struct rb_root_cached *tree);
>  
>  int inline_list__append(struct symbol *symbol, char *srcline, struct inline_node *node);
> +int inline_list__append_tail(struct symbol *symbol, char *srcline, struct inline_node *node);
>  char *srcline_from_fileline(const char *file, unsigned int line);
>  struct symbol *new_inline_sym(struct dso *dso,
>  			      struct symbol *base_sym,
> -- 
> 2.52.0.457.g6b5491de43-goog
> 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 2/7] perf addr2line: Add a libdw implementation
  2026-01-12 19:52   ` Arnaldo Carvalho de Melo
@ 2026-01-12 21:23     ` Ian Rogers
  0 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-12 21:23 UTC (permalink / raw)
  To: Arnaldo Carvalho de Melo, Masami Hiramatsu
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Namhyung Kim,
	Alexander Shishkin, Jiri Olsa, Adrian Hunter, James Clark,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users

On Mon, Jan 12, 2026 at 11:52 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Sat, Jan 10, 2026 at 08:13:33PM -0800, Ian Rogers wrote:
> > Add an implementation of addr2line that uses libdw. Other addr2line
> > implementations are slow, particularly in the case of forking
> > addr2line. Add an implementation that caches the libdw information in
> > the dso and uses it to find the file and line number
> > information. Inline information is supported but because
> > cu_walk_functions_at visits the leaf function last add a
> > inline_list__append_tail to reverse the lists order.
>
> So while testing I moved the inlineloop 'perf test' workload to the
> front to test without this new libdw implementation and then with it,
> and tested using perf probe on it:
>
> root@number:/home/acme# perf probe -x ~/bin/perf libdw__addr2line%return 'ret=$retval'
> Added new event:
>   probe_perf:libdw_addr2line__return (on libdw__addr2line%return in /home/acme/bin/perf with ret=$retval)
>
> You can now use it in all perf tools, such as:
>
>         perf record -e probe_perf:libdw_addr2line__return -aR sleep 1
>
> root@number:/home/acme# perf trace -e probe_perf:libdw_addr2line__return perf report -f --dso perf --stdio -s srcfile,srcline
>      0.000 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>      0.007 :1204769/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.680 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.685 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.689 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.692 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.696 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.699 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.702 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.705 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.709 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.712 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.715 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.718 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.721 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.724 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.727 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.729 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.732 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.734 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.737 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.740 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>      0.771 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     13.563 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     13.766 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>     14.099 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548)
>     14.103 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.105 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.134 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.143 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.242 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.244 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.245 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.246 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.248 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.250 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.251 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.252 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.258 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.260 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.265 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.267 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.268 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
>     14.270 perf/1204769 probe_perf:libdw_addr2line__return(__probe_func: 7698362, __probe_ret_ip: 7030548, ret: 1)
> # To display the perf.data header info, please use --header/--header-only options.
> #
> #
> # Total Lost Samples: 0
> #
> # Samples: 4K of event 'cpu/cycles/Pu'
> # Event count (approx.): 5535180842
> #
> # Overhead  Source File   Source:Line
> # ........  ............  ...............
> #
>     99.04%  inlineloop.c  inlineloop.c:21
>      0.46%  inlineloop.c  inlineloop.c:20
>
>
> #
> # (Tip: Save output of perf stat using: perf stat record <target workload>)
> #
> root@number:/home/acme#
>
> Putting more probes to see the fallbacks...
>
> root@number:/home/acme# perf probe -l
>   probe_perf:cmd_addr2line (on cmd__addr2line@util/addr2line.c in /home/acme/bin/perf)
>   probe_perf:libbfd_addr2line (on libbfd__addr2line@util/libbfd.h in /home/acme/bin/perf)
>   probe_perf:libdw_addr2line (on libdw__addr2line@util/libdw.c in /home/acme/bin/perf)
>   probe_perf:llvm_addr2line (on llvm__addr2line@util/llvm.c in /home/acme/bin/perf)
>   probe_perf:libdw_addr2line__return (on libdw__addr2line%return@util/libdw.c in /home/acme/bin/perf with ret)
> root@number:/home/acme#
>
> As we have:
>
> static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *line_nr,
>                      struct dso *dso, bool unwind_inlines, struct inline_node *node,
>                      struct symbol *sym)
> {
>         int ret;
>
>         ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
>         if (ret > 0)
>                 return ret;
>
>         ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
>         if (ret > 0)
>                 return ret;
>
>         ret = libbfd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
>         if (ret > 0)
>                 return ret;
>
>         return cmd__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
> }
>
> root@number:/home/acme# perf stat -e probe_perf:*_addr2line perf report -f --dso perf --stdio -s srcfile,srcline
> # To display the perf.data header info, please use --header/--header-only options.
> #
> #
> # Total Lost Samples: 0
> #
> # Samples: 4K of event 'cpu/cycles/Pu'
> # Event count (approx.): 5535180842
> #
> # Overhead  Source File   Source:Line
> # ........  ............  ...............
> #
>     99.04%  inlineloop.c  inlineloop.c:21
>      0.46%  inlineloop.c  inlineloop.c:20
>
>
> #
> # (Tip: To set sampling period of individual events use perf record -e cpu/cpu-cycles,period=100001/,cpu/branches,period=10001/ ...)
> #
>
>  Performance counter stats for 'perf report -f --dso perf --stdio -s srcfile,srcline':
>
>                 44      probe_perf:libdw_addr2line
>                 23      probe_perf:llvm_addr2line
>                  0      probe_perf:libbfd_addr2line
>                  0      probe_perf:cmd_addr2line
>
>        0.036554827 seconds time elapsed
>
>        0.030570000 seconds user
>        0.006049000 seconds sys
>
>
> root@number:/home/acme#
>
> Maybe asking for the dso_name may help understand why it is falling back
> 23 times to the llvm based one...
>
> root@number:/home/acme# perf probe -x ~/bin/perf llvm__addr2line dso_name:string
> Target program is compiled without optimization. Skipping prologue.
> Probe on address 0x5f73e4 to force probing at the function entry.
>
> Added new event:
>   probe_perf:llvm_addr2line (on llvm__addr2line in /home/acme/bin/perf with dso_name:string)
>
> You can now use it in all perf tools, such as:
>
>         perf record -e probe_perf:llvm_addr2line -aR sleep 1
>
> root@number:/home/acme# perf trace --libtrace -e probe_perf:llvm_addr2line perf report -f --dso perf --stdio -s srcfile,srcline
>      0.000 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
>      0.697 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
>      0.704 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
>      0.708 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
>      0.712 perf/1205478 probe_perf:llvm_addr2line((5f7416) dso_name_string="/root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf")
> <SNIP>
>
> root@number:/home/acme# readelf -wi /root/.debug/.build-id/cc/315df11334de01fb444465f08f6bca45a12bd2/elf | head -30
> Contents of the .debug_info section:
>
>   Compilation Unit @ offset 0:
>    Length:        0x20 (32-bit)
>    Version:       5
>    Unit Type:     DW_UT_partial (3)
>    Abbrev Offset: 0xad87
>    Pointer Size:  8
>  <0><c>: Abbrev Number: 38 (DW_TAG_partial_unit)
>     <d>   DW_AT_stmt_list   : 0
>     <11>   DW_AT_comp_dir    : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf
>  <1><15>: Abbrev Number: 57 (DW_TAG_base_type)
>     <16>   DW_AT_byte_size   : 8
>     <17>   DW_AT_encoding    : 7        (unsigned)
>     <18>   DW_AT_name        : (indirect string, offset: 0x9bf1): long unsigned int
>  <1><1c>: Abbrev Number: 57 (DW_TAG_base_type)
>     <1d>   DW_AT_byte_size   : 1
>     <1e>   DW_AT_encoding    : 6        (signed char)
>     <1f>   DW_AT_name        : (indirect string, offset: 0x7bab): char
>  <1><23>: Abbrev Number: 0
>   Compilation Unit @ offset 0x24:
>    Length:        0x20 (32-bit)
>    Version:       5
>    Unit Type:     DW_UT_partial (3)
>    Abbrev Offset: 0xad87
>    Pointer Size:  8
>  <0><30>: Abbrev Number: 38 (DW_TAG_partial_unit)
>     <31>   DW_AT_stmt_list   : 0
>     <35>   DW_AT_comp_dir    : (indirect line string, offset: 0xc9c): /usr/src/debug/glibc-2.42-5.fc43.x86_64/elf
>  <1><39>: Abbrev Number: 57 (DW_TAG_base_type)
> root@number:/home/acme#
>
> Some partial unit glibc DWARF 5. I think we need some more libdw setup
> to handle those?

Could be. Thanks for the analysis!

> But then we can merge what James already reviewed and go on from here,
> right?

+Masami Hiramatsu

I believe so. I'm not aware of anything outstanding on the patches -
v2 was using the function's declaration file:line rather than its
calling file:line, but that's all resolved in v3 and I'm not aware of
anything else left outstanding. Perhaps James can comment more on the
(non-blocking) issues he faced with LLVM and perhaps Masami can
comment on the libdw usage (I just checked that he did a lot of the
dwarf-aux.c work).

Thanks,
Ian

> - Arnaldo
>
> > Signed-off-by: Ian Rogers <irogers@google.com>
> > ---
> >  tools/perf/util/Build     |   1 +
> >  tools/perf/util/dso.c     |   2 +
> >  tools/perf/util/dso.h     |  11 +++
> >  tools/perf/util/libdw.c   | 153 ++++++++++++++++++++++++++++++++++++++
> >  tools/perf/util/libdw.h   |  60 +++++++++++++++
> >  tools/perf/util/srcline.c |  24 ++++++
> >  tools/perf/util/srcline.h |   1 +
> >  7 files changed, 252 insertions(+)
> >  create mode 100644 tools/perf/util/libdw.c
> >  create mode 100644 tools/perf/util/libdw.h
> >
> > diff --git a/tools/perf/util/Build b/tools/perf/util/Build
> > index 1c2a43e1dc68..2bed6274e248 100644
> > --- a/tools/perf/util/Build
> > +++ b/tools/perf/util/Build
> > @@ -224,6 +224,7 @@ perf-util-$(CONFIG_LIBDW) += dwarf-regs-powerpc.o
> >  perf-util-$(CONFIG_LIBDW) += dwarf-regs-x86.o
> >  perf-util-$(CONFIG_LIBDW) += debuginfo.o
> >  perf-util-$(CONFIG_LIBDW) += annotate-data.o
> > +perf-util-$(CONFIG_LIBDW) += libdw.o
> >
> >  perf-util-$(CONFIG_LIBDW_DWARF_UNWIND) += unwind-libdw.o
> >  perf-util-$(CONFIG_LOCAL_LIBUNWIND)    += unwind-libunwind-local.o
> > diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
> > index 344e689567ee..06980844c014 100644
> > --- a/tools/perf/util/dso.c
> > +++ b/tools/perf/util/dso.c
> > @@ -32,6 +32,7 @@
> >  #include "string2.h"
> >  #include "vdso.h"
> >  #include "annotate-data.h"
> > +#include "libdw.h"
> >
> >  static const char * const debuglink_paths[] = {
> >       "%.0s%s",
> > @@ -1605,6 +1606,7 @@ void dso__delete(struct dso *dso)
> >       auxtrace_cache__free(RC_CHK_ACCESS(dso)->auxtrace_cache);
> >       dso_cache__free(dso);
> >       dso__free_a2l(dso);
> > +     dso__free_a2l_libdw(dso);
> >       dso__free_symsrc_filename(dso);
> >       nsinfo__zput(RC_CHK_ACCESS(dso)->nsinfo);
> >       mutex_destroy(dso__lock(dso));
> > diff --git a/tools/perf/util/dso.h b/tools/perf/util/dso.h
> > index f8ccb9816b89..4aee23775054 100644
> > --- a/tools/perf/util/dso.h
> > +++ b/tools/perf/util/dso.h
> > @@ -268,6 +268,7 @@ DECLARE_RC_STRUCT(dso) {
> >       const char       *short_name;
> >       const char       *long_name;
> >       void             *a2l;
> > +     void             *a2l_libdw;
> >       char             *symsrc_filename;
> >  #if defined(__powerpc__)
> >       void            *dwfl;                  /* DWARF debug info */
> > @@ -334,6 +335,16 @@ static inline void dso__set_a2l(struct dso *dso, void *val)
> >       RC_CHK_ACCESS(dso)->a2l = val;
> >  }
> >
> > +static inline void *dso__a2l_libdw(const struct dso *dso)
> > +{
> > +     return RC_CHK_ACCESS(dso)->a2l_libdw;
> > +}
> > +
> > +static inline void dso__set_a2l_libdw(struct dso *dso, void *val)
> > +{
> > +     RC_CHK_ACCESS(dso)->a2l_libdw = val;
> > +}
> > +
> >  static inline unsigned int dso__a2l_fails(const struct dso *dso)
> >  {
> >       return RC_CHK_ACCESS(dso)->a2l_fails;
> > diff --git a/tools/perf/util/libdw.c b/tools/perf/util/libdw.c
> > new file mode 100644
> > index 000000000000..e4bfd52bd172
> > --- /dev/null
> > +++ b/tools/perf/util/libdw.c
> > @@ -0,0 +1,153 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +#include "dso.h"
> > +#include "libdw.h"
> > +#include "srcline.h"
> > +#include "symbol.h"
> > +#include "dwarf-aux.h"
> > +#include <fcntl.h>
> > +#include <unistd.h>
> > +#include <elfutils/libdwfl.h>
> > +
> > +void dso__free_a2l_libdw(struct dso *dso)
> > +{
> > +     Dwfl *dwfl = dso__a2l_libdw(dso);
> > +
> > +     if (dwfl) {
> > +             dwfl_end(dwfl);
> > +             dso__set_a2l_libdw(dso, NULL);
> > +     }
> > +}
> > +
> > +struct libdw_a2l_cb_args {
> > +     struct dso *dso;
> > +     struct symbol *sym;
> > +     struct inline_node *node;
> > +     char *leaf_srcline;
> > +     bool leaf_srcline_used;
> > +};
> > +
> > +static int libdw_a2l_cb(Dwarf_Die *die, void *_args)
> > +{
> > +     struct libdw_a2l_cb_args *args  = _args;
> > +     struct symbol *inline_sym = new_inline_sym(args->dso, args->sym, dwarf_diename(die));
> > +     const char *call_fname = die_get_call_file(die);
> > +     char *call_srcline = srcline__unknown;
> > +     struct inline_list *ilist;
> > +
> > +     if (!inline_sym)
> > +             return -ENOMEM;
> > +
> > +     /* Assign caller information to the parent. */
> > +     if (call_fname)
> > +             call_srcline = srcline_from_fileline(call_fname, die_get_call_lineno(die));
> > +
> > +     list_for_each_entry(ilist, &args->node->val, list) {
> > +             ilist->srcline =  call_srcline;
> > +             call_srcline = NULL;
> > +             break;
> > +     }
> > +     if (call_srcline && call_fname)
> > +             free(call_srcline);
> > +
> > +     /* Add this symbol to the chain as the leaf. */
> > +     inline_list__append_tail(inline_sym, args->leaf_srcline, args->node);
> > +     args->leaf_srcline_used = true;
> > +     return 0;
> > +}
> > +
> > +int libdw__addr2line(const char *dso_name, u64 addr,
> > +                  char **file, unsigned int *line_nr,
> > +                  struct dso *dso, bool unwind_inlines,
> > +                  struct inline_node *node, struct symbol *sym)
> > +{
> > +     static const Dwfl_Callbacks offline_callbacks = {
> > +             .find_debuginfo = dwfl_standard_find_debuginfo,
> > +             .section_address = dwfl_offline_section_address,
> > +             .find_elf = dwfl_build_id_find_elf,
> > +     };
> > +     Dwfl *dwfl = dso__a2l_libdw(dso);
> > +     Dwfl_Module *mod;
> > +     Dwfl_Line *dwline;
> > +     Dwarf_Addr bias;
> > +     const char *src;
> > +     int lineno = 0;
> > +
> > +     if (!dwfl) {
> > +             /*
> > +              * Initialize Dwfl session.
> > +              * We need to open the DSO file to report it to libdw.
> > +              */
> > +             int fd;
> > +
> > +             fd = open(dso_name, O_RDONLY);
> > +             if (fd < 0)
> > +                     return 0;
> > +
> > +             dwfl = dwfl_begin(&offline_callbacks);
> > +             if (!dwfl) {
> > +                     close(fd);
> > +                     return 0;
> > +             }
> > +
> > +             /*
> > +              * If the report is successful, the file descriptor fd is consumed
> > +              * and closed by the Dwfl. If not, it is not closed.
> > +              */
> > +             mod = dwfl_report_offline(dwfl, dso_name, dso_name, fd);
> > +             if (!mod) {
> > +                     dwfl_end(dwfl);
> > +                     close(fd);
> > +                     return 0;
> > +             }
> > +
> > +             dwfl_report_end(dwfl, /*removed=*/NULL, /*arg=*/NULL);
> > +             dso__set_a2l_libdw(dso, dwfl);
> > +     } else {
> > +             /* Dwfl session already initialized, get module for address. */
> > +             mod = dwfl_addrmodule(dwfl, addr);
> > +     }
> > +
> > +     if (!mod)
> > +             return 0;
> > +
> > +     /*
> > +      * Get/ignore the dwarf information. Determine the bias, difference
> > +      * between the regular ELF addr2line addresses and those to use with
> > +      * libdw.
> > +      */
> > +     if (!dwfl_module_getdwarf(mod, &bias))
> > +             return 0;
> > +
> > +     /* Find source line information for the address. */
> > +     dwline = dwfl_module_getsrc(mod, addr + bias);
> > +     if (!dwline)
> > +             return 0;
> > +
> > +     /* Get line information. */
> > +     src = dwfl_lineinfo(dwline, /*addr=*/NULL, &lineno, /*col=*/NULL, /*mtime=*/NULL,
> > +                         /*length=*/NULL);
> > +
> > +     if (file)
> > +             *file = src ? strdup(src) : NULL;
> > +     if (line_nr)
> > +             *line_nr = lineno;
> > +
> > +     /* Optionally unwind inline function call chain. */
> > +     if (unwind_inlines && node) {
> > +             Dwarf_Addr unused_bias;
> > +             Dwarf_Die *cudie = dwfl_module_addrdie(mod, addr + bias, &unused_bias);
> > +             struct libdw_a2l_cb_args args = {
> > +                     .dso = dso,
> > +                     .sym = sym,
> > +                     .node = node,
> > +                     .leaf_srcline = srcline_from_fileline(src ?: "<unknown>", lineno),
> > +             };
> > +
> > +             /* Walk from the parent down to the leaf. */
> > +             cu_walk_functions_at(cudie, addr, libdw_a2l_cb, &args);
> > +
> > +             if (!args.leaf_srcline_used)
> > +                     free(args.leaf_srcline);
> > +     }
> > +     return 1;
> > +}
> > diff --git a/tools/perf/util/libdw.h b/tools/perf/util/libdw.h
> > new file mode 100644
> > index 000000000000..0f8d7b4a11a5
> > --- /dev/null
> > +++ b/tools/perf/util/libdw.h
> > @@ -0,0 +1,60 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +#ifndef PERF_LIBDW_H
> > +#define PERF_LIBDW_H
> > +
> > +#include <linux/types.h>
> > +
> > +struct dso;
> > +struct inline_node;
> > +struct symbol;
> > +
> > +#ifdef HAVE_LIBDW_SUPPORT
> > +/*
> > + * libdw__addr2line - Convert address to source location using libdw
> > + * @dso_name: Name of the DSO
> > + * @addr: Address to resolve
> > + * @file: Pointer to return filename (caller must free)
> > + * @line_nr: Pointer to return line number
> > + * @dso: The dso struct
> > + * @unwind_inlines: Whether to unwind inline function calls
> > + * @node: Inline node list to append to
> > + * @sym: The symbol associated with the address
> > + *
> > + * This function initializes a Dwfl context for the DSO if not already present,
> > + * finds the source line information for the given address, and optionally
> > + * resolves inline function call chains.
> > + *
> > + * Returns 1 on success (found), 0 on failure (not found).
> > + */
> > +int libdw__addr2line(const char *dso_name, u64 addr, char **file,
> > +                  unsigned int *line_nr, struct dso *dso,
> > +                  bool unwind_inlines, struct inline_node *node,
> > +                  struct symbol *sym);
> > +
> > +/*
> > + * dso__free_a2l_libdw - Free libdw resources associated with the DSO
> > + * @dso: The dso to free resources for
> > + *
> > + * This function cleans up the Dwfl context used for addr2line lookups.
> > + */
> > +void dso__free_a2l_libdw(struct dso *dso);
> > +
> > +#else /* HAVE_LIBDW_SUPPORT */
> > +
> > +static inline int libdw__addr2line(const char *dso_name __maybe_unused,
> > +                                u64 addr __maybe_unused, char **file __maybe_unused,
> > +                                unsigned int *line_nr __maybe_unused,
> > +                                struct dso *dso __maybe_unused,
> > +                                bool unwind_inlines __maybe_unused,
> > +                                struct inline_node *node __maybe_unused,
> > +                                struct symbol *sym __maybe_unused)
> > +{
> > +     return 0;
> > +}
> > +
> > +static inline void dso__free_a2l_libdw(struct dso *dso __maybe_unused)
> > +{
> > +}
> > +#endif /* HAVE_LIBDW_SUPPORT */
> > +
> > +#endif /* PERF_LIBDW_H */
> > diff --git a/tools/perf/util/srcline.c b/tools/perf/util/srcline.c
> > index 27c0966611ab..e2d280678b02 100644
> > --- a/tools/perf/util/srcline.c
> > +++ b/tools/perf/util/srcline.c
> > @@ -6,6 +6,7 @@
> >  #include "libbfd.h"
> >  #include "llvm.h"
> >  #include "symbol.h"
> > +#include "libdw.h"
> >
> >  #include <inttypes.h>
> >  #include <string.h>
> > @@ -51,6 +52,25 @@ int inline_list__append(struct symbol *symbol, char *srcline, struct inline_node
> >       return 0;
> >  }
> >
> > +int inline_list__append_tail(struct symbol *symbol, char *srcline, struct inline_node *node)
> > +{
> > +     struct inline_list *ilist;
> > +
> > +     ilist = zalloc(sizeof(*ilist));
> > +     if (ilist == NULL)
> > +             return -1;
> > +
> > +     ilist->symbol = symbol;
> > +     ilist->srcline = srcline;
> > +
> > +     if (callchain_param.order == ORDER_CALLEE)
> > +             list_add(&ilist->list, &node->val);
> > +     else
> > +             list_add_tail(&ilist->list, &node->val);
> > +
> > +     return 0;
> > +}
> > +
> >  /* basename version that takes a const input string */
> >  static const char *gnu_basename(const char *path)
> >  {
> > @@ -120,6 +140,10 @@ static int addr2line(const char *dso_name, u64 addr, char **file, unsigned int *
> >  {
> >       int ret;
> >
> > +     ret = libdw__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
> > +     if (ret > 0)
> > +             return ret;
> > +
> >       ret = llvm__addr2line(dso_name, addr, file, line_nr, dso, unwind_inlines, node, sym);
> >       if (ret > 0)
> >               return ret;
> > diff --git a/tools/perf/util/srcline.h b/tools/perf/util/srcline.h
> > index c36f573cd339..be9f002bf234 100644
> > --- a/tools/perf/util/srcline.h
> > +++ b/tools/perf/util/srcline.h
> > @@ -57,6 +57,7 @@ struct inline_node *inlines__tree_find(struct rb_root_cached *tree, u64 addr);
> >  void inlines__tree_delete(struct rb_root_cached *tree);
> >
> >  int inline_list__append(struct symbol *symbol, char *srcline, struct inline_node *node);
> > +int inline_list__append_tail(struct symbol *symbol, char *srcline, struct inline_node *node);
> >  char *srcline_from_fileline(const char *file, unsigned int line);
> >  struct symbol *new_inline_sym(struct dso *dso,
> >                             struct symbol *base_sym,
> > --
> > 2.52.0.457.g6b5491de43-goog
> >

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 0/7] perf: Add a libdw addr2line implementation
  2026-01-12 18:29     ` Ian Rogers
@ 2026-01-13 12:03       ` James Clark
  2026-01-13 23:47         ` Ian Rogers
  0 siblings, 1 reply; 15+ messages in thread
From: James Clark @ 2026-01-13 12:03 UTC (permalink / raw)
  To: Ian Rogers
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users



On 12/01/2026 6:29 pm, Ian Rogers wrote:
> On Mon, Jan 12, 2026 at 6:49 AM Ian Rogers <irogers@google.com> wrote:
>>
>> On Mon, Jan 12, 2026 at 3:18 AM James Clark <james.clark@linaro.org> wrote:
>>>
>>> On 11/01/2026 4:13 am, Ian Rogers wrote:
>>>> addr2line is a performance bottleneck in perf, add a libdw based
>>>> implementation that avoids forking addr2line and caches the decoded
>>>> debug information.
>>>>
>>>> Allow the addr2line implementation to be picked via the configuration
>>>> file or --addr2line-style with `perf report`.
>>>>
>>>> Test/fix that inline callchains are properly displayed by perf script.
>>>>
>>>> An example:
>>>> ```
>>>> $ perf record --call-graph dwarf -e cycles:u -- perf test -w inlineloop 1
>>>> [ perf record: Woken up 132 times to write data ]
>>>> [ perf record: Captured and wrote 32.814 MB perf.data (4074 samples) ]
>>>> $ perf script --fields +srcline
>>>> ...
>>>> perf-inlineloop 1814670 293100.228871:     640004 cpu_core/cycles/u:
>>>>               55a11d6e61ee leaf+0x2e
>>>>     inlineloop.c:21 (inlined)
>>>>               55a11d6e61ee middle+0x2e
>>>>     inlineloop.c:27 (inlined)
>>>>               55a11d6e61ee parent+0x2e (perf)
>>>>     inlineloop.c:32
>>>>               55a11d6e629b inlineloop+0x8b (perf)
>>>>     inlineloop.c:47
>>>>               55a11d69a3bc run_workload+0x5a (perf)
>>>>     builtin-test.c:715
>>>>               55a11d69aa9f cmd_test+0x417 (perf)
>>>>     builtin-test.c:825
>>>>               55a11d6155f5 run_builtin+0xd4 (perf)
>>>>     perf.c:349
>>>>               55a11d61588d handle_internal_command+0xdd (perf)
>>>>     perf.c:401
>>>>               55a11d6159e6 run_argv+0x35 (perf)
>>>>     perf.c:445
>>>>               55a11d615d2f main+0x2cb (perf)
>>>>     perf.c:553
>>>>               7fae3d233ca7 __libc_start_call_main+0x77 (libc.so.6)
>>>>     libc_start_call_main.h:58
>>>>               7fae3d233d64 __libc_start_main_impl+0x84
>>>>     libc-start.c:360 (inlined)
>>>>               55a11d565f80 _start+0x20 (perf)
>>>>     ??:0
>>>> ...
>>>> ```
>>>>
>>>> v3: Make the caller inline file and line number accurate in the libdw
>>>>       addr2line, rather than using the function's declared location.
>>>>       Fix reference counts in unwind-libdw. Add fixes tag for srcline
>>>>       inline printing.
>>>>
>>>> v2: Fix bias issue with libdwfl functions. Use cu_walk_functions_at
>>>>       from perf's dwarf-aux to fully walk inline functions. Add testing
>>>>       that inlined functions are shown in the perf script srcline
>>>>       callchain information. Add configurability as to which addr2line
>>>>       style to use.
>>>>       https://lore.kernel.org/lkml/20260110082647.1487574-1-irogers@google.com/
>>>>
>>>> v1: https://lore.kernel.org/lkml/20251122093934.94971-1-irogers@google.com/
>>>>
>>>> Ian Rogers (7):
>>>>     perf unwind-libdw: Fix invalid reference counts
>>>>     perf addr2line: Add a libdw implementation
>>>>     perf addr2line.c: Rename a2l_style to cmd_a2l_style
>>>>     perf srcline: Add configuration support for the addr2line style
>>>>     perf callchain: Fix srcline printing with inlines
>>>>     perf test workload: Add inlineloop test workload
>>>>     perf test: Test addr2line unwinding works with inline functions
>>>>
>>>>    tools/perf/builtin-report.c                 |  10 ++
>>>>    tools/perf/tests/builtin-test.c             |   1 +
>>>>    tools/perf/tests/shell/addr2line_inlines.sh |  47 ++++++
>>>>    tools/perf/tests/tests.h                    |   1 +
>>>>    tools/perf/tests/workloads/Build            |   2 +
>>>>    tools/perf/tests/workloads/inlineloop.c     |  52 +++++++
>>>>    tools/perf/util/Build                       |   1 +
>>>>    tools/perf/util/addr2line.c                 |  20 +--
>>>>    tools/perf/util/config.c                    |   4 +
>>>>    tools/perf/util/dso.c                       |   2 +
>>>>    tools/perf/util/dso.h                       |  11 ++
>>>>    tools/perf/util/evsel_fprintf.c             |   8 +-
>>>>    tools/perf/util/libdw.c                     | 153 ++++++++++++++++++++
>>>>    tools/perf/util/libdw.h                     |  60 ++++++++
>>>>    tools/perf/util/srcline.c                   | 116 ++++++++++++++-
>>>>    tools/perf/util/srcline.h                   |   3 +
>>>>    tools/perf/util/symbol_conf.h               |  10 ++
>>>>    tools/perf/util/unwind-libdw.c              |   7 +-
>>>>    18 files changed, 486 insertions(+), 22 deletions(-)
>>>>    create mode 100755 tools/perf/tests/shell/addr2line_inlines.sh
>>>>    create mode 100644 tools/perf/tests/workloads/inlineloop.c
>>>>    create mode 100644 tools/perf/util/libdw.c
>>>>    create mode 100644 tools/perf/util/libdw.h
>>>>
>>>
>>> I don't see the differences to the other addr2line implementations
>>> anymore, but only because it falls through to the old ones when libdw
>>> fails now.
>>>
>>> For example when building Perf with LLVM it can't get the line in the
>>> inlineloop workload, and there's still a few things in libc and other
>>> system libraries it fails on.
>>
>> Hmm.. I wonder what the issue is. I was looking at the dwarf output
>> from my gcc builds with llvm-dwarfdump. I wonder if LLVM builds are

I see some issues in libc on Ubuntu though, which I assume is compiled 
with GCC, although there's no .comment section in it so I can't be sure. 
So it's not exclusively LLVM but it does seem like LLVM builds cause a 
lot more failures.

>> doing something to confuse libdw? I'll try to investigate. There are
>> quite a few levels of libdw: there's the raw libdw, libdwfl (frontend
>> to libdw) that does the parsing and tries to give things like nested
>> debug scopes (libdwfl is the one needing addresses with a module bias
>> rather than raw file offsets), and then there is the dwarf-aux.c that
>> is in perf and is used by things like probe finding (I believe this
>> doesn't need biases addresses). Anyway, with the biases there are
>> things I can screw up (like in the v1 patch) but maybe the LLVM issue
>> is just a libdw and dwarf-5 kind of thing. Maybe it is ARM specific
>> :-/

Actually I get the same behavior on Arm and x86.

> 
> Testing with clang/llvm on x86-64 (dwarf5):
> ```
> $ make -C tools/perf O=/tmp/perf DEBUG=1 CC=clang CXX=clang++
> HOSTCC=clang clean all
> ...
> $ llvm-dwarfdump /tmp/perf/perf
> ...
> 0x0014f852: Compile Unit: length = 0x00000294, format = DWARF32,
> version = 0x0005, unit_type = DW_UT_compile,
> abbr_offset = 0x1879a, addr_size = 0x08 (next unit at 0x0014faea)
> 
> 0x0014f85e: DW_TAG_compile_unit
>               DW_AT_producer    ("Debian clang version 19.1.7 (3+build5)")
>               DW_AT_language    (DW_LANG_C11)
>               DW_AT_name        ("tests/workloads/inlineloop.c")
>               DW_AT_str_offsets_base    (0x0004a550)
>               DW_AT_stmt_list   (0x0008c3f2)
>               DW_AT_comp_dir    ("linux/tools/perf")
>               DW_AT_low_pc      (0x00000000001e61c0)
>               DW_AT_high_pc     (0x00000000001e62e9)
>               DW_AT_addr_base   (0x00022248)
>               DW_AT_loclists_base       (0x0000018a)
> ...
> $  sudo /tmp/perf/perf record --call-graph dwarf -e cycles:u --
> /tmp/perf/perf test -w inlineloop 1
> ...
> $  sudo /tmp/perf/perf script --fields +srcline
> ...
> perf-inlineloop 2284167 423038.015394:     569917 cpu_core/cycles/u:
>             56390020d2c6 leaf+0x26
>   inlineloop.c:21 (inlined)
>             56390020d2c6 middle+0x26
>   inlineloop.c:27 (inlined)
>             56390020d2c6 parent+0x26 (/tmp/perf/perf)
> ...
> ```
> I ran inside of gdb and confirmed that the libdw code is creating the
> inlined information (breakpoint on libdw_a2l_cb, etc.). So I'm not
> able to reproduce the LLVM issue for now on x86-64.
> 
> Thanks,
> Ian
> 

If I set this in ~/.perfconfig so the fallback is disabled:

  [addr2line]
	style = libdw

Then:

  $ make LLVM=1 -C tools/perf DEBUG=1 clean all
  $ perf record --delay 1000 -- perf test -w inlineloop 2
  $ perf script --fields ip,srcline
      6012b5957b70
   perf[1f7b70]
      6012b5957b70
   perf[1f7b70]
   ...


x86:

  $ clang -v
  Ubuntu clang version 15.0.7

Arm:

  $ clang -v
  Ubuntu clang version 18.1.8 (11~20.04.2)

Disabling the ~/.perfconfig to re-enable the LLVM fallback works:

(x86)
$ perf script --fields ip,srcline
      6012b5957b70
   inlineloop.c:20
      6012b5957b70
   inlineloop.c:20

Interestingly, on Arm this results in zeros for line numbers. This is a 
completely different issue though which I didn't notice before because I 
built with GCC. It falls all the way back to A2L_STYLE_CMD:

(Arm)
$ perf script --fields ip,srcline
      aaaad0a7828c
   inlineloop.c:0
      aaaad0a7828c
   inlineloop.c:0

$ addr2line -e `which perf` -a -i -f aaaad0a7828c
0x0000aaaad0a7828c
??
??:0

Probably shouldn't get sidetracked by that here though. It's at least 
working when compiled with GCC, and neither LLVM or libdw work, so it's 
no worse.

>>> But I think it's fine because it doesn't give the wrong line anymore, it
>>> just falls through to another working addr2line implementation.
>>
>> Just to confirm that with gcc builds it isn't failing now? ie it isn't
>> just an addr2line implementation that falls through all the time? I
>> was seeing things working/testing on x86 with gcc.
>>

No, the GCC Perf build always works with libdw as far as I can see. Just 
the occasional fall through to LLVM with some libc addresses.

>>> Reviewed-by: James Clark <james.clark@linaro.org>
>>
>> Thanks,
>> Ian


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [PATCH v3 0/7] perf: Add a libdw addr2line implementation
  2026-01-13 12:03       ` James Clark
@ 2026-01-13 23:47         ` Ian Rogers
  0 siblings, 0 replies; 15+ messages in thread
From: Ian Rogers @ 2026-01-13 23:47 UTC (permalink / raw)
  To: James Clark
  Cc: Tony Jones, Peter Zijlstra, Ingo Molnar, Arnaldo Carvalho de Melo,
	Namhyung Kim, Alexander Shishkin, Jiri Olsa, Adrian Hunter,
	Howard Chu, Stephen Brennan, linux-kernel, linux-perf-users

Thanks Arnaldo for landing the series in perf-tools-next!

James mentioned frame pointer unwinding lacking inlines. I had a look
at and I think this patch may suffice (although on some quick testing
I wasn't able to get inlines other than at the leaf):
```
--- a/tools/perf/util/machine.c
+++ b/tools/perf/util/machine.c
@@ -2090,6 +2090,8 @@ struct iterations {
       u64 cycles;
};

+static int append_inlines(struct callchain_cursor *cursor, struct
map_symbol *ms, u64 ip);
+
static int add_callchain_ip(struct thread *thread,
                           struct callchain_cursor *cursor,
                           struct symbol **parent,
@@ -2170,6 +2172,10 @@ static int add_callchain_ip(struct thread *thread,
       ms.maps = maps__get(al.maps);
       ms.map = map__get(al.map);
       ms.sym = al.sym;
+
+       if (append_inlines(cursor, &ms, ip) == 0)
+               goto out;
+
       srcline = callchain_srcline(&ms, al.addr);
       err = callchain_cursor_append(cursor, ip, &ms,
                                     branch, flags, nr_loop_iter,
```

Having inline information seems like a good thing regardless of the
stack trace format, so it'd be nice to move a patch like this forward.

Thanks,
Ian

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2026-01-13 23:47 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-01-11  4:13 [PATCH v3 0/7] perf: Add a libdw addr2line implementation Ian Rogers
2026-01-11  4:13 ` [PATCH v3 1/7] perf unwind-libdw: Fix invalid reference counts Ian Rogers
2026-01-11  4:13 ` [PATCH v3 2/7] perf addr2line: Add a libdw implementation Ian Rogers
2026-01-12 19:52   ` Arnaldo Carvalho de Melo
2026-01-12 21:23     ` Ian Rogers
2026-01-11  4:13 ` [PATCH v3 3/7] perf addr2line.c: Rename a2l_style to cmd_a2l_style Ian Rogers
2026-01-11  4:13 ` [PATCH v3 4/7] perf srcline: Add configuration support for the addr2line style Ian Rogers
2026-01-11  4:13 ` [PATCH v3 5/7] perf callchain: Fix srcline printing with inlines Ian Rogers
2026-01-11  4:13 ` [PATCH v3 6/7] perf test workload: Add inlineloop test workload Ian Rogers
2026-01-11  4:13 ` [PATCH v3 7/7] perf test: Test addr2line unwinding works with inline functions Ian Rogers
2026-01-12 11:18 ` [PATCH v3 0/7] perf: Add a libdw addr2line implementation James Clark
2026-01-12 14:49   ` Ian Rogers
2026-01-12 18:29     ` Ian Rogers
2026-01-13 12:03       ` James Clark
2026-01-13 23:47         ` Ian Rogers

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox