* [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
2026-10-12 0:30 [PATCH perf-tools-next v3 0/5] perf ftrace: Support inlined functions and display enhancements in function graph tracer Aaron Tomlin
@ 2026-10-12 0:30 ` Aaron Tomlin
2026-10-12 0:41 ` sashiko-bot
2026-10-12 0:30 ` [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments Aaron Tomlin
` (3 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Aaron Tomlin @ 2026-10-12 0:30 UTC (permalink / raw)
To: peterz, mingo, acme, namhyung
Cc: mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter,
atomlin, linux-perf-users, linux-kernel
In __cmd_ftrace(), incoming trace data from trace_pipe was historically
buffered in a fixed 4096-byte array allocated directly on the stack
frame. Furthermore, reads were written directly to stdout via fwrite()
in arbitrary chunk sizes, rather than as discrete, well-delimited
trace lines.
Refactor __cmd_ftrace() to employ dynamic heap allocation for the read
buffer alongside a dynamic struct strbuf to accumulate streamed
characters into whole lines. Pre-allocating the strbuf ensures that
line processing incurs zero heap reallocations during steady-state
polling, while the stack frame of __cmd_ftrace() is reduced by over 96%
(down to ~160 bytes).
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
tools/perf/builtin-ftrace.c | 57 ++++++++++++++++++++++++++++++-------
1 file changed, 46 insertions(+), 11 deletions(-)
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index 6017493ff179..d6604be593a3 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -41,7 +41,10 @@
#include "util/units.h"
#include "util/parse-sublevel-options.h"
+#include "util/strbuf.h"
+
#define DEFAULT_TRACER "function_graph"
+#define TRACE_BUF_SIZE 4096
static volatile sig_atomic_t workload_exec_errno;
static volatile sig_atomic_t done;
@@ -738,7 +741,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
{
char *trace_file;
int trace_fd;
- char buf[4096];
+ char *buf = NULL;
+ struct strbuf linebuf = STRBUF_INIT;
struct pollfd pollfd = {
.events = POLLIN,
};
@@ -767,10 +771,17 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
setup_pager();
+ buf = malloc(TRACE_BUF_SIZE);
+ if (!buf) {
+ pr_err("failed to allocate trace buffer\n");
+ goto out_reset;
+ }
+ strbuf_init(&linebuf, 512);
+
trace_file = get_tracing_instance_file("trace_pipe");
if (!trace_file) {
pr_err("failed to open trace_pipe\n");
- goto out_reset;
+ goto out_free;
}
trace_fd = open(trace_file, O_RDONLY);
@@ -779,7 +790,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
if (trace_fd < 0) {
pr_err("failed to open trace_pipe\n");
- goto out_reset;
+ goto out_free;
}
fcntl(trace_fd, F_SETFL, O_NONBLOCK);
@@ -810,11 +821,19 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
break;
if (pollfd.revents & POLLIN) {
- int n = read(trace_fd, buf, sizeof(buf));
- if (n < 0)
- break;
- if (fwrite(buf, n, 1, stdout) != 1)
+ int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
+ if (n <= 0)
break;
+ for (int i = 0; i < n; i++) {
+ if (buf[i] == '\n') {
+ fprintf(stdout, "%s\n", linebuf.buf);
+ strbuf_setlen(&linebuf, 0);
+ } else {
+ if (strbuf_addch(&linebuf, buf[i]) < 0)
+ goto out_close_fd;
+ }
+ }
/* flush output since stdout is in full buffering mode due to pager */
fflush(stdout);
}
@@ -823,7 +842,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
write_tracing_file("tracing_on", "0");
if (workload_exec_errno) {
- const char *emsg = str_error_r(workload_exec_errno, buf, sizeof(buf));
+ const char *emsg = str_error_r(workload_exec_errno, buf, TRACE_BUF_SIZE);
/* flush stdout first so below error msg appears at the end. */
fflush(stdout);
pr_err("workload failed: %s\n", emsg);
@@ -832,15 +851,31 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
/* read remaining buffer contents */
while (true) {
- int n = read(trace_fd, buf, sizeof(buf));
+ int n = read(trace_fd, buf, TRACE_BUF_SIZE);
+
if (n <= 0)
break;
- if (fwrite(buf, n, 1, stdout) != 1)
- break;
+ for (int i = 0; i < n; i++) {
+ if (buf[i] == '\n') {
+ fprintf(stdout, "%s\n", linebuf.buf);
+ strbuf_setlen(&linebuf, 0);
+ } else {
+ if (strbuf_addch(&linebuf, buf[i]) < 0)
+ goto out_close_fd;
+ }
+ }
+ }
+
+ if (linebuf.len > 0) {
+ fprintf(stdout, "%s\n", linebuf.buf);
+ strbuf_setlen(&linebuf, 0);
}
out_close_fd:
close(trace_fd);
+out_free:
+ free(buf);
+ strbuf_release(&linebuf);
out_reset:
exit_tracing_instance();
out:
--
2.55.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
2026-10-12 0:30 ` [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming Aaron Tomlin
@ 2026-10-12 0:41 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-12 0:41 UTC (permalink / raw)
To: Aaron Tomlin; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] perf: dropped stdout error check causes infinite loop on EPIPE
- [Medium] perf: uninitialized heap memory read in __cmd_ftrace() strbuf
--
commit 0a06afad8e6a1e6bf21a4b8a3c19582ed3d99cd1
Author: Aaron Tomlin <atomlin@atomlin.com>
perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming
This commit refactors __cmd_ftrace() to use dynamic heap allocation
for the read buffer and a strbuf to accumulate streamed characters
into lines. This reduces the stack frame size by over 96%.
> diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
> index 6017493ff1797..d6604be593a34 100644
> --- a/tools/perf/builtin-ftrace.c
> +++ b/tools/perf/builtin-ftrace.c
[ ... ]
> @@ -810,11 +821,19 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
> break;
>
> if (pollfd.revents & POLLIN) {
> - int n = read(trace_fd, buf, sizeof(buf));
> - if (n < 0)
> - break;
> - if (fwrite(buf, n, 1, stdout) != 1)
> + int n = read(trace_fd, buf, TRACE_BUF_SIZE);
> +
> + if (n <= 0)
> break;
> + for (int i = 0; i < n; i++) {
> + if (buf[i] == '\n') {
> + fprintf(stdout, "%s\n", linebuf.buf);
[Severity: Medium]
Can this print uninitialized heap memory?
Earlier in this function, strbuf_init() allocates the buffer but does not
explicitly null-terminate it:
tools/perf/util/strbuf.c:strbuf_grow() {
...
buf = realloc(sb->alloc ? sb->buf : NULL, nr * sizeof(*buf));
if (!buf)
return -ENOMEM;
sb->buf = buf;
sb->alloc = nr;
return 0;
}
If the very first character read from trace_pipe is '\n' (for example, from an
empty trace line), linebuf.buf is passed to fprintf() as a string before any
null byte has been written to it.
[Severity: Medium]
Does the removal of the stdout error check cause an infinite loop?
The original code explicitly checked if fwrite() failed to gracefully exit
the polling loop if stdout closes (for example, when the pager exits or
when piped to head).
Because trace_pipe provides endless continuous data while tracing is active,
ignoring the fprintf() error forces the tool to endlessly read trace data
and discard it, never terminating.
> + strbuf_setlen(&linebuf, 0);
> + } else {
> + if (strbuf_addch(&linebuf, buf[i]) < 0)
> + goto out_close_fd;
> + }
> + }
> /* flush output since stdout is in full buffering mode due to pager */
> fflush(stdout);
> }
[ ... ]
> @@ -832,15 +851,31 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
>
> /* read remaining buffer contents */
> while (true) {
> - int n = read(trace_fd, buf, sizeof(buf));
> + int n = read(trace_fd, buf, TRACE_BUF_SIZE);
> +
> if (n <= 0)
> break;
> - if (fwrite(buf, n, 1, stdout) != 1)
> - break;
> + for (int i = 0; i < n; i++) {
> + if (buf[i] == '\n') {
> + fprintf(stdout, "%s\n", linebuf.buf);
[Severity: Medium]
Does the lack of a return value check on fprintf() also cause an infinite
loop here on a broken pipe?
The previous code broke out of the loop if fwrite() failed. Without
checking the return value of fprintf(), an error writing to stdout is
ignored.
> + strbuf_setlen(&linebuf, 0);
> + } else {
> + if (strbuf_addch(&linebuf, buf[i]) < 0)
> + goto out_close_fd;
> + }
> + }
> + }
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261012003016.263063-1-atomlin@atomlin.com?part=1
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments
2026-10-12 0:30 [PATCH perf-tools-next v3 0/5] perf ftrace: Support inlined functions and display enhancements in function graph tracer Aaron Tomlin
2026-10-12 0:30 ` [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming Aaron Tomlin
@ 2026-10-12 0:30 ` Aaron Tomlin
2026-10-12 0:38 ` sashiko-bot
2026-10-12 0:30 ` [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer Aaron Tomlin
` (2 subsequent siblings)
4 siblings, 1 reply; 11+ messages in thread
From: Aaron Tomlin @ 2026-10-12 0:30 UTC (permalink / raw)
To: peterz, mingo, acme, namhyung
Cc: mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter,
atomlin, linux-perf-users, linux-kernel
When tracing with the Linux kernel's function graph tracer, enabling
funcgraph-retaddr records the return address of each function entry as
a comment in the trace stream (e.g. /* <-load_elf_phdrs+0x6c/0xb0 */).
While valuable for tracing execution origins, these trailing comments
can add considerable visual clutter to the call graph, particularly
when examining deeply nested call trees or when synthesising higher-level
abstractions.
Introduce support for filtering out return address comments from
function graph trace output. Provide a standalone --filter-retaddr
flag.
Key aspects of this implementation:
1. Return address parsing and filtering
Implement ftrace_parse_retaddr() and ftrace_filter_retaddr() in
tools/perf/util/ftrace.c to robustly match return address
comments across diverse output styles (standard format, return
values, and bracketed addresses) and strip them in-place with
memmove(3).
2. Command-line interface
Expose --filter-retaddr in
tools/perf/builtin-ftrace.c, updating the documentation in
tools/perf/Documentation/perf-ftrace.txt accordingly.
3. Automated verification
Add a dedicated unit test suite ("Ftrace return address
processing") in tools/perf/tests/ftrace.c to validate comment
parsing and string filtering across all supported formats.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
tools/perf/Documentation/perf-ftrace.txt | 5 +-
tools/perf/builtin-ftrace.c | 9 +++
tools/perf/tests/Build | 1 +
tools/perf/tests/builtin-test.c | 1 +
tools/perf/tests/ftrace.c | 80 ++++++++++++++++++++++
tools/perf/tests/tests.h | 1 +
tools/perf/util/Build | 1 +
tools/perf/util/ftrace.c | 87 ++++++++++++++++++++++++
tools/perf/util/ftrace.h | 5 ++
9 files changed, 189 insertions(+), 1 deletion(-)
create mode 100644 tools/perf/tests/ftrace.c
create mode 100644 tools/perf/util/ftrace.c
diff --git a/tools/perf/Documentation/perf-ftrace.txt b/tools/perf/Documentation/perf-ftrace.txt
index 3f3808e513fe..3dbb7e8e783e 100644
--- a/tools/perf/Documentation/perf-ftrace.txt
+++ b/tools/perf/Documentation/perf-ftrace.txt
@@ -1,5 +1,4 @@
perf-ftrace(1)
-==============
NAME
----
@@ -134,6 +133,10 @@ OPTIONS for 'perf ftrace trace'
- depth=<n> - Set max depth for function graph tracer to follow.
- tail - Print function name at the end.
+--filter-retaddr::
+ Filter out function return address comments (`/* <-caller+offset */`)
+ from function_graph output.
+
OPTIONS for 'perf ftrace latency'
---------------------------------
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index d6604be593a3..f3221089a479 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -827,6 +827,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
break;
for (int i = 0; i < n; i++) {
if (buf[i] == '\n') {
+ if (ftrace->filter_retaddr)
+ ftrace_filter_retaddr(linebuf.buf);
fprintf(stdout, "%s\n", linebuf.buf);
strbuf_setlen(&linebuf, 0);
} else {
@@ -857,6 +859,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
break;
for (int i = 0; i < n; i++) {
if (buf[i] == '\n') {
+ if (ftrace->filter_retaddr)
+ ftrace_filter_retaddr(linebuf.buf);
fprintf(stdout, "%s\n", linebuf.buf);
strbuf_setlen(&linebuf, 0);
} else {
@@ -867,6 +871,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
}
if (linebuf.len > 0) {
+ if (ftrace->filter_retaddr)
+ ftrace_filter_retaddr(linebuf.buf);
fprintf(stdout, "%s\n", linebuf.buf);
strbuf_setlen(&linebuf, 0);
}
@@ -1827,11 +1833,14 @@ int cmd_ftrace(int argc, const char **argv)
"Set nograph filter on given functions", parse_filter_func),
OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
"Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
+ "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
parse_graph_tracer_opts),
OPT_CALLBACK('m', "buffer-size", &ftrace.percpu_buffer_size, "size",
"Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
OPT_BOOLEAN(0, "inherit", &ftrace.inherit,
"Trace children processes"),
+ OPT_BOOLEAN_SET(0, "filter-retaddr", &ftrace.filter_retaddr, &ftrace.filter_retaddr_set,
+ "Filter out funcgraph return address comments"),
OPT_INTEGER('D', "delay", &ftrace.target.initial_delay,
"Number of milliseconds to wait before starting tracing after program start"),
OPT_PARENT(common_options),
diff --git a/tools/perf/tests/Build b/tools/perf/tests/Build
index 395f2866094a..136138932fcb 100644
--- a/tools/perf/tests/Build
+++ b/tools/perf/tests/Build
@@ -72,6 +72,7 @@ perf-test-y += hwmon_pmu.o
perf-test-y += tool_pmu.o
perf-test-y += subcmd-help.o
perf-test-y += kallsyms-split.o
+perf-test-y += ftrace.o
ifeq ($(SRCARCH),$(filter $(SRCARCH),x86 arm arm64 powerpc riscv))
perf-test-$(CONFIG_DWARF_UNWIND) += dwarf-unwind.o
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index d2f594921e25..9663f48afb92 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -137,6 +137,7 @@ static struct test_suite *generic_tests[] = {
&suite__perf_hooks,
&suite__unit_number__scnprint,
&suite__mem2node,
+ &suite__ftrace,
&suite__time_utils,
&suite__jit_write_elf,
&suite__pfm,
diff --git a/tools/perf/tests/ftrace.c b/tools/perf/tests/ftrace.c
new file mode 100644
index 000000000000..596cecee844e
--- /dev/null
+++ b/tools/perf/tests/ftrace.c
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/compiler.h>
+#include <linux/kernel.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include "tests.h"
+#include "debug.h"
+#include "util/ftrace.h"
+
+static int test_parse_retaddr(void)
+{
+ char sym[128];
+ u64 offset = 0;
+
+ /* Standard format */
+ TEST_ASSERT_VAL("parse standard retaddr",
+ ftrace_parse_retaddr("kernel_read() { /* <-load_elf_phdrs+0x6c/0xb0 */",
+ sym, sizeof(sym), &offset));
+ TEST_ASSERT_EQUAL("sym name", strcmp(sym, "load_elf_phdrs"), 0);
+ TEST_ASSERT_VAL("offset", offset == 0x6c);
+
+ /* With return value */
+ TEST_ASSERT_VAL("parse retaddr with ret",
+ ftrace_parse_retaddr(
+ "__cond_resched(); /* <-load_elf_phdrs+0x6c/0xb0 ret=0x0 */",
+ sym, sizeof(sym), &offset));
+ TEST_ASSERT_EQUAL("sym name", strcmp(sym, "load_elf_phdrs"), 0);
+ TEST_ASSERT_VAL("offset", offset == 0x6c);
+
+ /* Bracketed address */
+ TEST_ASSERT_VAL("parse bracketed retaddr",
+ ftrace_parse_retaddr("foo() { /* <-[0xffffffff818534c8] bar+0x20/0x40 */",
+ sym, sizeof(sym), &offset));
+ TEST_ASSERT_EQUAL("sym name", strcmp(sym, "bar"), 0);
+ TEST_ASSERT_VAL("offset", offset == 0x20);
+
+ /* No return address */
+ TEST_ASSERT_VAL("no retaddr",
+ !ftrace_parse_retaddr("do_filp_open();", sym, sizeof(sym), &offset));
+
+ return TEST_OK;
+}
+
+static int test_filter_retaddr(void)
+{
+ char line1[] = "kernel_read() { /* <-load_elf_phdrs+0x6c/0xb0 */";
+ char line2[] = "__cond_resched(); /* <-load_elf_phdrs+0x6c/0xb0 ret=0x0 */";
+ char line3[] = "do_filp_open();";
+
+ ftrace_filter_retaddr(line1);
+ TEST_ASSERT_EQUAL("filter standard retaddr", strcmp(line1, "kernel_read() {"), 0);
+
+ ftrace_filter_retaddr(line2);
+ TEST_ASSERT_EQUAL("filter retaddr with ret", strcmp(line2, "__cond_resched();"), 0);
+
+ ftrace_filter_retaddr(line3);
+ TEST_ASSERT_EQUAL("filter line without retaddr unchanged",
+ strcmp(line3, "do_filp_open();"), 0);
+
+ return TEST_OK;
+}
+
+static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
+{
+ int ret;
+
+ ret = test_parse_retaddr();
+ if (ret != TEST_OK)
+ return ret;
+
+ ret = test_filter_retaddr();
+ if (ret != TEST_OK)
+ return ret;
+
+ return TEST_OK;
+}
+
+DEFINE_SUITE("Ftrace return address processing", ftrace);
diff --git a/tools/perf/tests/tests.h b/tools/perf/tests/tests.h
index 9c96f33483d1..1aec32e094c8 100644
--- a/tools/perf/tests/tests.h
+++ b/tools/perf/tests/tests.h
@@ -163,6 +163,7 @@ DECLARE_SUITE(perf_hooks);
DECLARE_SUITE(unit_number__scnprint);
DECLARE_SUITE(mem2node);
DECLARE_SUITE(maps);
+DECLARE_SUITE(ftrace);
DECLARE_SUITE(time_utils);
DECLARE_SUITE(jit_write_elf);
DECLARE_SUITE(api_io);
diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index 2c1f880c4c47..5b082d09006d 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -169,6 +169,7 @@ perf-util-y += list_sort.o
perf-util-y += mutex.o
perf-util-y += sharded_mutex.o
perf-util-y += intel-tpebs.o
+perf-util-y += ftrace.o
perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter.o
perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_counter_cgroup.o
diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
new file mode 100644
index 000000000000..344d36293f5c
--- /dev/null
+++ b/tools/perf/util/ftrace.c
@@ -0,0 +1,87 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/compiler.h>
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <ctype.h>
+#include "util/ftrace.h"
+
+/**
+ * ftrace_parse_retaddr - Parse return address comment from trace line
+ * @str: The trace line containing the comment
+ * @sym_name: Buffer to store the parsed symbol name
+ * @sym_len: Length of the @sym_name buffer
+ * @offset: Pointer to store the parsed offset
+ *
+ * Extract the function name and offset from a funcgraph-retaddr comment
+ * (e.g. "<-load_elf_phdrs+0x6c/0xb0").
+ *
+ * Return: true if successfully parsed, false otherwise.
+ */
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset)
+{
+ const char *p = strstr(str, "<-");
+ const char *plus;
+ const char *bracket;
+ char *endptr;
+ size_t name_len;
+
+ if (!p)
+ return false;
+
+ p += 2;
+ while (*p == ' ')
+ p++;
+
+ if (*p == '[') {
+ bracket = strchr(p, ']');
+ if (bracket)
+ p = bracket + 1;
+ while (*p == ' ')
+ p++;
+ }
+
+ plus = strchr(p, '+');
+ if (!plus)
+ return false;
+
+ name_len = plus - p;
+ if (name_len == 0 || name_len >= sym_len)
+ return false;
+
+ memcpy(sym_name, p, name_len);
+ sym_name[name_len] = '\0';
+
+ *offset = strtoull(plus + 1, &endptr, 16);
+ return true;
+}
+
+/**
+ * ftrace_filter_retaddr - Remove return address comment from trace line
+ * @str: The trace line to modify
+ *
+ * Find and strip the funcgraph-retaddr comment from the trace line
+ * in-place, removing trailing spaces where applicable.
+ */
+void ftrace_filter_retaddr(char *str)
+{
+ char *start = strstr(str, "/* <-");
+ char *end;
+
+ if (!start)
+ return;
+
+ end = strstr(start, "*/");
+ if (!end)
+ return;
+
+ end += 2; /* skip end-of-comment delimiter */
+
+ /* Also backtrack any spaces before comment */
+ while (start > str && *(start - 1) == ' ')
+ start--;
+
+ memmove(start, end, strlen(end) + 1);
+}
diff --git a/tools/perf/util/ftrace.h b/tools/perf/util/ftrace.h
index 950f2efafad2..50216d3fef64 100644
--- a/tools/perf/util/ftrace.h
+++ b/tools/perf/util/ftrace.h
@@ -39,6 +39,8 @@ struct perf_ftrace {
int graph_verbose;
int graph_thresh;
int graph_tail;
+ bool filter_retaddr;
+ bool filter_retaddr_set;
};
struct filter_entry {
@@ -93,4 +95,7 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
#endif /* HAVE_BPF_SKEL */
+bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
+void ftrace_filter_retaddr(char *str);
+
#endif /* __PERF_FTRACE_H__ */
--
2.55.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments
2026-10-12 0:30 ` [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments Aaron Tomlin
@ 2026-10-12 0:38 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-12 0:38 UTC (permalink / raw)
To: Aaron Tomlin; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Low] perf: build failure in cmd_ftrace() due to extra OPT_CALLBACK argument
- [Low] perf: ftrace_parse_retaddr() is dead code and never used for filtering
- [Low] perf: missing document title underline breaks asciidoc generation
--
commit 5dab431981f96da9cbf4caa8e862a4c0e461a1ff
Author: Aaron Tomlin <atomlin@atomlin.com>
perf ftrace: Support filtering return address comments
This commit introduces support for filtering out return address comments from
function graph trace output via a standalone --filter-retaddr flag. It adds
parsing and filtering logic, updates the command-line interface, and includes
a dedicated unit test suite.
> diff --git a/tools/perf/Documentation/perf-ftrace.txt b/tools/perf/Documentation/perf-ftrace.txt
> index 3f3808e513fe8..3dbb7e8e783e9 100644
> --- a/tools/perf/Documentation/perf-ftrace.txt
> +++ b/tools/perf/Documentation/perf-ftrace.txt
> @@ -1,5 +1,4 @@
> perf-ftrace(1)
> -==============
[Severity: Low]
Does removing the document title underline break the asciidoc generation?
Asciidoctor strictly requires the document title to be underlined with equal
signs of at least the same length as the text. Removing it renders the
document malformed.
[ ... ]
> diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
> index d6604be593a34..f3221089a4793 100644
> --- a/tools/perf/builtin-ftrace.c
> +++ b/tools/perf/builtin-ftrace.c
[ ... ]
> @@ -1827,11 +1833,14 @@ int cmd_ftrace(int argc, const char **argv)
> "Set nograph filter on given functions", parse_filter_func),
> OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
> "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
> + "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
> parse_graph_tracer_opts),
[Severity: Low]
Does this extra string literal cause a build failure?
The OPT_CALLBACK macro for graph-opts was modified by adding a new help
string literal without removing the old one. Because they are separated by a
comma, the C preprocessor treats them as two distinct arguments, causing the
macro to be invoked with 7 arguments instead of 6.
[ ... ]
> diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
> new file mode 100644
> index 0000000000000..344d36293f5ca
> --- /dev/null
> +++ b/tools/perf/util/ftrace.c
[ ... ]
> +bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset)
> +{
[Severity: Low]
Is ftrace_parse_retaddr() ever used outside of the test suite?
The commit message claims this function is implemented to robustly match
return address comments before stripping them. However, looking at the
implementation of ftrace_filter_retaddr() below:
[ ... ]
> +void ftrace_filter_retaddr(char *str)
> +{
> + char *start = strstr(str, "/* <-");
> + char *end;
> +
> + if (!start)
> + return;
> +
> + end = strstr(start, "*/");
It appears ftrace_filter_retaddr() performs string replacement using strstr
and memmove directly, bypassing ftrace_parse_retaddr() entirely. This leaves
the parsing function unused in the actual execution path.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261012003016.263063-1-atomlin@atomlin.com?part=2
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer
2026-10-12 0:30 [PATCH perf-tools-next v3 0/5] perf ftrace: Support inlined functions and display enhancements in function graph tracer Aaron Tomlin
2026-10-12 0:30 ` [PATCH perf-tools-next v3 1/5] perf ftrace: Optimise __cmd_ftrace() stack frame with strbuf streaming Aaron Tomlin
2026-10-12 0:30 ` [PATCH perf-tools-next v3 2/5] perf ftrace: Support filtering return address comments Aaron Tomlin
@ 2026-10-12 0:30 ` Aaron Tomlin
2026-10-12 0:43 ` sashiko-bot
2026-10-12 0:30 ` [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration " Aaron Tomlin
2026-10-12 0:30 ` [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions " Aaron Tomlin
4 siblings, 1 reply; 11+ messages in thread
From: Aaron Tomlin @ 2026-10-12 0:30 UTC (permalink / raw)
To: peterz, mingo, acme, namhyung
Cc: mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter,
atomlin, linux-perf-users, linux-kernel
The Linux kernel's function graph tracer operates at the machine
instruction level via compiler instrumentation
(-fpatchable-function-entry), recording entry and return events for
physical function calls. Consequently, functions inlined by the compiler
are invisible in the resulting call graph, as no discrete call or return
instructions are emitted for them. While technically expected, this
omission frequently obscures the logical execution flow when analysing
kernel subsystems heavily reliant upon inlining.
Address this by introducing the --inline option to perf ftrace for the
function_graph tracer.
When the kernel is configured with CONFIG_FUNCTION_GRAPH_RETADDR=y, the
funcgraph-retaddr trace option records the caller's return address on
each function entry, manifested in the trace stream as a comment
(e.g. /* <-irq_work_queue_on+0x81/0xa0 */). By interrogating DWARF debug
information from the kernel image (vmlinux) utilising libdw, perf ftrace
resolves this return address to its inlined callchain and synthesises the
intermediate inlined frames directly into the streamed call graph.
Key aspects of this implementation:
1. Inlined frame presentation
Synthesised inlined functions are rendered with an explicit
/* (inline) */ annotation at both entry and exit:
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | wake_up_new_task() {
0) | activate_task() { /* (inline) */
0) 1.234 us | enqueue_task();
0) | } /* activate_task (inline) */
0) 2.567 us | }
This affords immediate visual clarity to the reader upon
entering an inlined scope, without necessitating a search for
the closing delimiter.
2. Call graph reconstruction and dynamic stack tracking
Maintains a per-CPU stack tracking real and inlined frames,
synthesising entry and exit delimiters to accurately reconstruct
the execution hierarchy. Inlined frames are structured strictly
root-to-leaf and validated against the active stack to prevent
un-traced callers from attaching orphan inlines. Bare closing
braces without trailing function names (the default when
funcgraph-tail is disabled) are correctly matched against
preceding frames on the stack. Furthermore, stack depth
saturation guards ensure that inlined frame depth tracking
remains clamped and indentation cannot drift even under deep
call trees exceeding FTRACE_MAX_STACK.
3. Option handling and diagnostics
Adds -k or --vmlinux to designate an explicit debug image, and
provides clear diagnostic guidance should --inline be invoked on
a kernel lacking CONFIG_FUNCTION_GRAPH_RETADDR. Automatically
enables return address comment filtering unless explicitly
requested.
4. Automated verification
Expands the unit test suite in tools/perf/tests/ftrace.c to
validate DWARF inline symbol resolution, comment filtering
toggles, bare closing braces handling, and stack saturation
recovery.
Assisted-by: Antigravity:gemini-3.1-pro
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
tools/perf/Documentation/perf-ftrace.txt | 17 +-
tools/perf/builtin-ftrace.c | 56 +-
tools/perf/tests/ftrace.c | 303 ++++++++++-
tools/perf/util/ftrace.c | 621 ++++++++++++++++++++++-
tools/perf/util/ftrace.h | 11 +
5 files changed, 990 insertions(+), 18 deletions(-)
diff --git a/tools/perf/Documentation/perf-ftrace.txt b/tools/perf/Documentation/perf-ftrace.txt
index 3dbb7e8e783e..86306caf569b 100644
--- a/tools/perf/Documentation/perf-ftrace.txt
+++ b/tools/perf/Documentation/perf-ftrace.txt
@@ -133,9 +133,24 @@ OPTIONS for 'perf ftrace trace'
- depth=<n> - Set max depth for function graph tracer to follow.
- tail - Print function name at the end.
+--inline::
+ Show inlined functions in function_graph tracer. This requires
+ kernel support for the funcgraph-retaddr option and a vmlinux with
+ debug symbols. Inlined functions are displayed in the call graph
+ hierarchy, and their inlined status is indicated with `/* (inline) */`
+ on entry and exit, e.g. `func() { /* (inline) */` and `} /* func (inline) */`.
+ By default, raw return address comments (`/* <-caller+offset */`)
+ are hidden in the output; specify `--graph-opts retaddr` to show them.
+
--filter-retaddr::
Filter out function return address comments (`/* <-caller+offset */`)
- from function_graph output.
+ from function_graph output. This is enabled by default when `--inline`
+ is used; use `--no-filter-retaddr` to disable it.
+
+-k::
+--vmlinux=::
+ Path to the vmlinux file containing debug symbols for resolving
+ inlined functions.
OPTIONS for 'perf ftrace latency'
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index f3221089a479..504007064e24 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -574,9 +574,14 @@ static int set_tracing_funcgraph_retval(struct perf_ftrace *ftrace)
static int set_tracing_funcgraph_retaddr(struct perf_ftrace *ftrace)
{
- if (ftrace->graph_retaddr) {
- if (write_tracing_option_file("funcgraph-retaddr", "1") < 0)
+ if (ftrace->graph_retaddr || ftrace->use_inline) {
+ if (write_tracing_option_file("funcgraph-retaddr", "1") < 0) {
+ if (ftrace->use_inline) {
+ pr_err("failed to set funcgraph-retaddr option in tracing\n");
+ pr_err("Hint: ensure kernel is compiled with CONFIG_FUNCTION_GRAPH_RETADDR=y\n");
+ }
return -1;
+ }
}
return 0;
@@ -769,6 +774,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
goto out_reset;
}
+ if (perf_ftrace__setup_inlines(ftrace) < 0)
+ goto out_reset;
+
setup_pager();
buf = malloc(TRACE_BUF_SIZE);
@@ -827,9 +835,10 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
break;
for (int i = 0; i < n; i++) {
if (buf[i] == '\n') {
- if (ftrace->filter_retaddr)
- ftrace_filter_retaddr(linebuf.buf);
- fprintf(stdout, "%s\n", linebuf.buf);
+ if (ftrace_process_fgraph_line(ftrace,
+ linebuf.buf,
+ stdout) < 0)
+ goto out_close_fd;
strbuf_setlen(&linebuf, 0);
} else {
if (strbuf_addch(&linebuf, buf[i]) < 0)
@@ -859,9 +868,10 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
break;
for (int i = 0; i < n; i++) {
if (buf[i] == '\n') {
- if (ftrace->filter_retaddr)
- ftrace_filter_retaddr(linebuf.buf);
- fprintf(stdout, "%s\n", linebuf.buf);
+ if (ftrace_process_fgraph_line(ftrace,
+ linebuf.buf,
+ stdout) < 0)
+ goto out_close_fd;
strbuf_setlen(&linebuf, 0);
} else {
if (strbuf_addch(&linebuf, buf[i]) < 0)
@@ -871,9 +881,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
}
if (linebuf.len > 0) {
- if (ftrace->filter_retaddr)
- ftrace_filter_retaddr(linebuf.buf);
- fprintf(stdout, "%s\n", linebuf.buf);
+ if (ftrace_process_fgraph_line(ftrace, linebuf.buf,
+ stdout) < 0)
+ goto out_close_fd;
strbuf_setlen(&linebuf, 0);
}
@@ -883,6 +893,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
free(buf);
strbuf_release(&linebuf);
out_reset:
+ perf_ftrace__cleanup_inlines(ftrace);
exit_tracing_instance();
out:
return (done && !workload_exec_errno) ? 0 : -1;
@@ -1839,8 +1850,13 @@ int cmd_ftrace(int argc, const char **argv)
"Size of per cpu buffer, needs to use a B, K, M or G suffix.", parse_buffer_size),
OPT_BOOLEAN(0, "inherit", &ftrace.inherit,
"Trace children processes"),
- OPT_BOOLEAN_SET(0, "filter-retaddr", &ftrace.filter_retaddr, &ftrace.filter_retaddr_set,
- "Filter out funcgraph return address comments"),
+ OPT_BOOLEAN(0, "inline", &ftrace.use_inline,
+ "Show inlined functions in function_graph tracer"),
+ OPT_BOOLEAN_SET(0, "filter-retaddr", &ftrace.filter_retaddr,
+ &ftrace.filter_retaddr_set,
+ "Filter out funcgraph return address comments (default with --inline)"),
+ OPT_STRING('k', "vmlinux", &symbol_conf.vmlinux_name, "file",
+ "vmlinux pathname"),
OPT_INTEGER('D', "delay", &ftrace.target.initial_delay,
"Number of milliseconds to wait before starting tracing after program start"),
OPT_PARENT(common_options),
@@ -1956,6 +1972,20 @@ int cmd_ftrace(int argc, const char **argv)
switch (subcmd) {
case PERF_FTRACE_TRACE:
+ if (ftrace.use_inline) {
+ if (ftrace.tracer && strcmp(ftrace.tracer, "function_graph") != 0) {
+ pr_err("Error: --inline option is only supported with function_graph tracer\n");
+ ret = -EINVAL;
+ goto out_delete_filters;
+ }
+ ftrace.tracer = "function_graph";
+ }
+ if (!ftrace.filter_retaddr_set) {
+ if (ftrace.use_inline && !ftrace.graph_retaddr)
+ ftrace.filter_retaddr = true;
+ else
+ ftrace.filter_retaddr = false;
+ }
cmd_func = __cmd_ftrace;
break;
case PERF_FTRACE_LATENCY:
diff --git a/tools/perf/tests/ftrace.c b/tools/perf/tests/ftrace.c
index 596cecee844e..e856bed00700 100644
--- a/tools/perf/tests/ftrace.c
+++ b/tools/perf/tests/ftrace.c
@@ -7,6 +7,10 @@
#include <unistd.h>
#include "tests.h"
#include "debug.h"
+#include "dso.h"
+#include "machine.h"
+#include "map.h"
+#include "symbol.h"
#include "util/ftrace.h"
static int test_parse_retaddr(void)
@@ -62,6 +66,291 @@ static int test_filter_retaddr(void)
return TEST_OK;
}
+#define TEST_CHECK(text, cond) \
+do { \
+ if (!(cond)) { \
+ pr_debug("FAILED %s:%d %s\n", __FILE__, __LINE__, text); \
+ ret = TEST_FAIL; \
+ goto out_cleanup; \
+ } \
+} while (0)
+
+static int test_inline_processing(void)
+{
+ static const char line_entry[] = " 0) | wake_up_new_task() {";
+ static const char line_enqueue[] =
+ " 0) | enqueue_task() { /* <-wake_up_new_task+0x1d1/0x3e0 */";
+ static const char line_exit_enqueue[] = " 0) + 10.000 us | } /* enqueue_task */";
+ static const char line_exit_task[] = " 0) + 20.000 us | } /* wake_up_new_task */";
+ static const char line_bare_enqueue[] = " 0) + 10.000 us | }";
+ static const char line_bare_task[] = " 0) + 20.000 us | }";
+ static const char line_unmatched[] = " 0) + 5.000 us | } /* unrelated_func */";
+ struct perf_ftrace ftrace;
+ char *out_buf = NULL;
+ size_t out_len = 0;
+ FILE *out;
+ const char *unmatched_pos;
+ const char *inline_exit_pos;
+ int ret = TEST_OK;
+
+ memset(&ftrace, 0, sizeof(ftrace));
+ ftrace.use_inline = true;
+ ftrace.filter_retaddr = 1; /* Default when --graph-opts retaddr is not specified */
+ ftrace.tracer = "function_graph";
+
+ /* If vmlinux is present, test resolving symbols */
+ if (access("vmlinux", R_OK) == 0) {
+ symbol_conf.vmlinux_name = "vmlinux";
+ symbol_conf.ignore_vmlinux_buildid = true;
+ ret = perf_ftrace__setup_inlines(&ftrace);
+ if (ret < 0 || !ftrace.machine) {
+ ret = TEST_SKIP;
+ goto out_cleanup;
+ }
+
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_cleanup;
+ }
+
+ /* Feed simulated function graph trace lines */
+ ftrace_process_fgraph_line(&ftrace, line_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ pr_debug("Processed output:\n%s\n", out_buf);
+
+ /* Verify activate_task entry with inline hint */
+ TEST_CHECK("contains activate_task entry with inline hint",
+ strstr(out_buf, "activate_task() { /* (inline) */") != NULL);
+
+ /* Verify activate_task exit with inline hint */
+ TEST_CHECK("contains activate_task exit with inline hint",
+ strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+ /* Verify return address comment was filtered out by default */
+ TEST_CHECK("retaddr comment filtered out",
+ strstr(out_buf, "/* <-wake_up_new_task") == NULL);
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* Now test with filter_retaddr = 0 (user specified --graph-opts retaddr) */
+ ftrace.filter_retaddr = 0;
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_cleanup;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ /* Verify return address comment IS preserved when filter_retaddr = 0 */
+ TEST_CHECK("retaddr comment preserved when requested",
+ strstr(out_buf, "/* <-wake_up_new_task") != NULL);
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* Test with bare closing braces (funcgraph-tail disabled by default) */
+ ftrace.filter_retaddr = 1;
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_cleanup;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_bare_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_bare_task, out);
+
+ fclose(out);
+
+ TEST_CHECK("bare braces inline entry",
+ strstr(out_buf, "activate_task() { /* (inline) */") != NULL);
+ TEST_CHECK("bare braces inline exit",
+ strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* Test that unmatched exit events do not pop active inlined frames */
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_cleanup;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_enqueue, out);
+ /* Unmatched exit event while activate_task is still active */
+ ftrace_process_fgraph_line(&ftrace, line_unmatched, out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+ fclose(out);
+
+ unmatched_pos = strstr(out_buf, "} /* unrelated_func */");
+ inline_exit_pos = strstr(out_buf, "} /* activate_task (inline) */");
+
+ TEST_CHECK("unmatched exit present", unmatched_pos != NULL);
+ TEST_CHECK("inline exit present", inline_exit_pos != NULL);
+ TEST_CHECK("inline exit not prematurely popped before unmatched exit",
+ inline_exit_pos > unmatched_pos);
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* Test recovery when an intermediate real function exit event is dropped */
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_cleanup;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_enqueue, out);
+ /* line_exit_enqueue is dropped; outer wake_up_new_task exits directly */
+ ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+ fclose(out);
+
+ TEST_CHECK("inline exit synthesized despite dropped callee exit",
+ strstr(out_buf, "} /* activate_task (inline) */") != NULL);
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* Test that callers not on active stack do not synthesise inlines */
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_cleanup;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_entry, out);
+ /* housekeeping_any_cpu called by untraced ring_buffer_unlock_commit */
+ ftrace_process_fgraph_line(&ftrace,
+ " 0) | housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+ out);
+ ftrace_process_fgraph_line(&ftrace, line_exit_task, out);
+
+ fclose(out);
+
+ TEST_CHECK("untraced caller inlines not synthesised",
+ strstr(out_buf, "rb_wakeups") == NULL &&
+ strstr(out_buf, "rb_irq_work_queue") == NULL);
+ }
+
+out_cleanup:
+ free(out_buf);
+ perf_ftrace__cleanup_inlines(&ftrace);
+ return ret;
+}
+#undef TEST_CHECK
+
+static int test_stack_saturation(void)
+{
+ struct perf_ftrace ftrace;
+ char *out_buf = NULL;
+ size_t out_len = 0;
+ FILE *out;
+ int i;
+
+ memset(&ftrace, 0, sizeof(ftrace));
+ ftrace.use_inline = true;
+ ftrace_cpu_states__clear();
+
+ out = open_memstream(&out_buf, &out_len);
+ if (!out)
+ return TEST_FAIL;
+
+ /* Push more than FTRACE_MAX_STACK (256) frames */
+ for (i = 0; i < 300; i++)
+ ftrace_process_fgraph_line(&ftrace, " 0) | sub_func() {", out);
+
+ /* Pop all 300 frames */
+ for (i = 0; i < 300; i++)
+ ftrace_process_fgraph_line(&ftrace, " 0) | }", out);
+
+ /* Feed one more function entry - it must have base indentation (no indentation drift) */
+ ftrace_process_fgraph_line(&ftrace, " 0) | base_func() {", out);
+
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ /* Verify base_func starts at standard 2 spaces indentation without drift */
+ TEST_ASSERT_VAL("no indentation drift after stack saturation",
+ strstr(out_buf, "| base_func() {") != NULL);
+
+ free(out_buf);
+ return TEST_OK;
+}
+
+static int test_column_layouts(void)
+{
+ static const struct {
+ const char *desc;
+ bool verbose;
+ const char *in;
+ const char *expected;
+ } cases[] = {
+ {
+ "default layout", false,
+ " 7) | wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
+ " 7) | wake_up_new_task() {\n",
+ },
+ {
+ "verbose layout", true,
+ " 1234.567890 | 7) bash-1234 | | wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
+ " 1234.567890 | 7) bash-1234 | | wake_up_new_task() {\n",
+ },
+ };
+ struct perf_ftrace ftrace;
+ char *out_buf = NULL;
+ size_t out_len = 0;
+ FILE *out;
+ size_t i;
+
+ for (i = 0; i < ARRAY_SIZE(cases); i++) {
+ memset(&ftrace, 0, sizeof(ftrace));
+ ftrace.use_inline = true;
+ ftrace.filter_retaddr = true;
+ ftrace.graph_verbose = cases[i].verbose;
+
+ out = open_memstream(&out_buf, &out_len);
+ if (!out)
+ return TEST_FAIL;
+
+ ftrace_process_fgraph_line(&ftrace, cases[i].in, out);
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ if (strcmp(out_buf, cases[i].expected)) {
+ pr_debug("%s: expected '%s', got '%s'\n",
+ cases[i].desc, cases[i].expected, out_buf);
+ free(out_buf);
+ return TEST_FAIL;
+ }
+ free(out_buf);
+ out_buf = NULL;
+ }
+
+ return TEST_OK;
+}
+
static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret;
@@ -74,7 +363,19 @@ static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __ma
if (ret != TEST_OK)
return ret;
+ ret = test_stack_saturation();
+ if (ret != TEST_OK)
+ return ret;
+
+ ret = test_column_layouts();
+ if (ret != TEST_OK)
+ return ret;
+
+ ret = test_inline_processing();
+ if (ret != TEST_OK)
+ return ret;
+
return TEST_OK;
}
-DEFINE_SUITE("Ftrace return address processing", ftrace);
+DEFINE_SUITE("Ftrace inline processing", ftrace);
diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
index 344d36293f5c..8737bb52921e 100644
--- a/tools/perf/util/ftrace.c
+++ b/tools/perf/util/ftrace.c
@@ -1,13 +1,82 @@
// SPDX-License-Identifier: GPL-2.0
-#include <linux/compiler.h>
-#include <linux/kernel.h>
-#include <linux/string.h>
+/*
+ * ftrace.c - Ftrace utilities and inlined function processing
+ *
+ * Copyright (C) 2026 Aaron Tomlin <atomlin@atomlin.com>
+ */
#include <stdio.h>
#include <stdlib.h>
+#include <errno.h>
#include <string.h>
#include <ctype.h>
+#include <stdbool.h>
+#include <linux/kernel.h>
+#include <linux/string.h>
+#include "debug.h"
+#include "dso.h"
+#include "machine.h"
+#include "map.h"
+#include "srcline.h"
+#include "symbol.h"
#include "util/ftrace.h"
+#define FTRACE_MAX_CPUS 4096
+#define FTRACE_MAX_STACK 256
+#define FTRACE_MAX_INLINES 32
+
+struct ftrace_stack_frame {
+ char *name;
+ bool inlined;
+ int indent;
+};
+
+struct ftrace_cpu_state {
+ struct ftrace_stack_frame stack[FTRACE_MAX_STACK];
+ int depth;
+ int inlined_depth;
+};
+
+static struct ftrace_cpu_state *ftrace_cpu_states[FTRACE_MAX_CPUS];
+
+/**
+ * ftrace_cpu_states__clear - Free allocated CPU stack states
+ *
+ * Free all memory associated with the per-CPU stack tracking states
+ * used for inlined function resolution, including the strings
+ * allocated for stack frames.
+ */
+void ftrace_cpu_states__clear(void)
+{
+ int i, j;
+
+ for (i = 0; i < FTRACE_MAX_CPUS; i++) {
+ if (ftrace_cpu_states[i]) {
+ for (j = 0; j < ftrace_cpu_states[i]->depth; j++)
+ free(ftrace_cpu_states[i]->stack[j].name);
+ free(ftrace_cpu_states[i]);
+ ftrace_cpu_states[i] = NULL;
+ }
+ }
+}
+
+/**
+ * get_cpu_state - Get or allocate stack state for a given CPU
+ * @cpu: The CPU number
+ *
+ * Return: A pointer to the CPU's stack state, or NULL on error or
+ * out-of-bounds CPU index.
+ */
+static struct ftrace_cpu_state *get_cpu_state(int cpu)
+{
+ if (cpu < 0 || cpu >= FTRACE_MAX_CPUS)
+ return NULL;
+
+ if (!ftrace_cpu_states[cpu])
+ ftrace_cpu_states[cpu] = zalloc(sizeof(struct ftrace_cpu_state));
+
+ return ftrace_cpu_states[cpu];
+}
+
/**
* ftrace_parse_retaddr - Parse return address comment from trace line
* @str: The trace line containing the comment
@@ -85,3 +154,549 @@ void ftrace_filter_retaddr(char *str)
memmove(start, end, strlen(end) + 1);
}
+
+/**
+ * ftrace_resolve_inlines - Resolve inlined functions for a return address
+ * @machine: Perf machine context
+ * @sym_name: Name of the caller function
+ * @offset: Instruction offset within the caller function
+ * @inlined_names: Array to store the resolved inline function names
+ * @max_inlines: Maximum number of inline frames to resolve
+ *
+ * Use DWARF debug information from the kernel image to resolve a caller's
+ * return address to its inlined callchain.
+ *
+ * Return: The number of inline functions successfully resolved.
+ */
+int ftrace_resolve_inlines(struct machine *machine,
+ const char *sym_name, u64 offset,
+ const char **inlined_names, int max_inlines)
+{
+ struct map *map = NULL;
+ struct symbol *sym;
+ struct dso *dso;
+ struct inline_node *node;
+ struct inline_list *ilist;
+ u64 ip, addr;
+ int count = 0;
+
+ if (!machine)
+ return 0;
+
+ sym = machine__find_kernel_symbol_by_name(machine, sym_name, &map);
+ if (!sym || !map)
+ return 0;
+
+ dso = map__dso(map);
+ if (!dso) {
+ map__put(map);
+ return 0;
+ }
+
+ ip = sym->start + offset;
+ addr = map__rip_2objdump(map, ip);
+
+ node = inlines__tree_find(dso__inlined_nodes(dso), addr);
+ if (!node) {
+ node = dso__parse_addr_inlines(dso, addr, sym);
+ if (node)
+ inlines__tree_insert(dso__inlined_nodes(dso), node);
+ }
+
+ if (!node) {
+ map__put(map);
+ return 0;
+ }
+
+ list_for_each_entry_reverse(ilist, &node->val, list) {
+ if (ilist->symbol && symbol__inlined(ilist->symbol)) {
+ if (count < max_inlines) {
+ char *name = strdup(ilist->symbol->name);
+
+ if (name)
+ inlined_names[count++] = name;
+ }
+ }
+ }
+
+ map__put(map);
+ return count;
+}
+
+/**
+ * extract_entry_func_name - Extract the function name from an entry record
+ * @str: The trace line's function column string
+ * @buf: Buffer to store the extracted name
+ * @size: Size of the @buf
+ *
+ * Parse the function name out of an entry trace record by stripping away
+ * parentheses or braces.
+ */
+static void extract_entry_func_name(const char *str, char *buf, size_t size)
+{
+ size_t i = 0;
+
+ while (*str && *str != '(' && *str != '{' && !isspace(*str) && i < size - 1)
+ buf[i++] = *str++;
+ buf[i] = '\0';
+}
+
+/**
+ * extract_exit_func_name - Extract the function name from an exit record
+ * @str: The trace line's function column string
+ * @buf: Buffer to store the extracted name
+ * @size: Size of the @buf
+ *
+ * Parse the original function name out of the trailing comment in an
+ * exit trace record.
+ */
+static void extract_exit_func_name(const char *str, char *buf, size_t size)
+{
+ const char *p = strchr(str, '}');
+ size_t i = 0;
+
+ buf[0] = '\0';
+ if (!p)
+ return;
+
+ p = strstr(p, "/*");
+ if (!p)
+ return;
+
+ p = skip_spaces(p + 2);
+ while (*p && !isspace(*p) && *p != '*' && *p != '/' && i < size - 1)
+ buf[i++] = *p++;
+ buf[i] = '\0';
+}
+
+/**
+ * make_blank_prefix - Create a blank prefix mirroring a trace record's context
+ * @line: The original trace record
+ * @pipe_char: Pointer to the final delimiter character of the context columns
+ * @buf: Buffer to store the blank prefix
+ * @size: Size of the @buf
+ *
+ * Generate a prefix matching the context columns of @line (e.g. CPU, duration)
+ * but with the duration field replaced by spaces.
+ */
+static void make_blank_prefix(const char *line, const char *pipe_char, char *buf, size_t size)
+{
+ size_t len = pipe_char - line + 1;
+ const char *paren = strchr(line, ')');
+ char *c;
+
+ if (len >= size)
+ len = size - 1;
+
+ memcpy(buf, line, len);
+ buf[len] = '\0';
+
+ /* Replace duration between ')' and '|' with spaces */
+ if (paren && paren < line + len) {
+ char *end = buf + (pipe_char - line);
+
+ if (end > buf + len)
+ end = buf + len;
+ for (c = buf + (paren - line) + 1; c < end; c++)
+ *c = ' ';
+ }
+}
+
+/**
+ * find_func_column - Locate the function call column in a trace record
+ * @ftrace: Perf ftrace context
+ * @line: The trace record to parse
+ * @cpu: Pointer to store the CPU number
+ *
+ * Locate the context columns that precede the function call column of a
+ * function_graph trace record. Depending on the funcgraph-* options set
+ * up by perf, a record looks like:
+ *
+ * [ abstime |] CPU) [ comm-pid |] [ duration |] func
+ *
+ * The abstime and comm-pid columns are only present with --graph-opts
+ * verbose.
+ *
+ * Return: A pointer to the last character of the context columns (the final
+ * '|' delimiter). Returns NULL if @line is not a trace record.
+ */
+static const char *find_func_column(const struct perf_ftrace *ftrace,
+ const char *line, int *cpu)
+{
+ const char *p = line;
+ char *endptr;
+ long val;
+
+ if (ftrace->graph_verbose) {
+ p = strchr(p, '|');
+ if (!p)
+ return NULL;
+ p++;
+ }
+
+ p = skip_spaces(p);
+ val = strtol(p, &endptr, 10);
+ if (endptr == p || *endptr != ')')
+ return NULL;
+ p = endptr;
+
+ if (ftrace->graph_verbose) {
+ p = strchr(p, '|');
+ if (!p)
+ return NULL;
+ }
+
+ p = strchr(p + 1, '|');
+ if (!p)
+ return NULL;
+
+ *cpu = (int)val;
+ return p;
+}
+
+/**
+ * print_line - Output a trace line verbatim, filtering return addresses if needed
+ * @ftrace: Perf ftrace context
+ * @line: The trace line to output
+ * @out: File stream to output to
+ *
+ * Output @line to @out. If ftrace->filter_retaddr is set, return address
+ * comments are filtered out first.
+ *
+ * Return: 0 on success, or a negative error code (e.g. -ENOMEM) on failure.
+ */
+static int print_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
+ char *filtered;
+
+ if (!ftrace->filter_retaddr) {
+ fprintf(out, "%s\n", line);
+ return 0;
+ }
+
+ filtered = strdup(line);
+ if (!filtered) {
+ pr_err("Not enough memory\n");
+ return -ENOMEM;
+ }
+
+ ftrace_filter_retaddr(filtered);
+ fprintf(out, "%s\n", filtered);
+ free(filtered);
+ return 0;
+}
+
+/**
+ * ftrace_process_fgraph_line - Process and output a function graph trace line
+ * @ftrace: Perf ftrace context
+ * @line: The trace line to process
+ * @out: File stream to output to
+ *
+ * Process a trace line from the function_graph tracer. If --inline is set,
+ * this function parses the return address comment, resolves it to its
+ * inlined callchain, and synthesises intermediate frames into the streamed
+ * call graph.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
+{
+ char caller_sym[128];
+ char blank_prefix[128];
+ const char *inlined_names[FTRACE_MAX_INLINES];
+ const char *pipe_char;
+ const char *after_pipe;
+ const char *func_start;
+ struct ftrace_cpu_state *cs;
+ u64 caller_offset = 0;
+ int num_inlines = 0;
+ int cpu = -1;
+ int leading_spaces;
+ int extra_indent;
+ int caller_idx;
+ int active_inlines;
+ int common;
+ int match_idx;
+ int i;
+ bool is_exit;
+ bool has_open_brace;
+ bool has_semicolon;
+ bool has_retaddr = false;
+
+ if (!ftrace->use_inline)
+ return print_line(ftrace, line, out);
+
+ pipe_char = find_func_column(ftrace, line, &cpu);
+ if (!pipe_char)
+ return print_line(ftrace, line, out);
+
+ cs = get_cpu_state(cpu);
+ if (!cs)
+ return print_line(ftrace, line, out);
+
+ make_blank_prefix(line, pipe_char, blank_prefix, sizeof(blank_prefix));
+
+ after_pipe = pipe_char + 1;
+ func_start = skip_spaces(after_pipe);
+ leading_spaces = func_start - after_pipe;
+
+ if (*func_start == '\0')
+ return print_line(ftrace, line, out);
+
+ is_exit = (*func_start == '}');
+
+ if (is_exit) {
+ char exit_name[128];
+
+ extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
+
+ /* Find matching frame on stack */
+ match_idx = -1;
+ for (i = cs->depth - 1; i >= 0; i--) {
+ if (!cs->stack[i].inlined) {
+ if (exit_name[0] == '\0' ||
+ (cs->stack[i].name &&
+ strcmp(cs->stack[i].name, exit_name) == 0)) {
+ match_idx = i;
+ break;
+ }
+ }
+ }
+
+ if (match_idx >= 0) {
+ /* Close all inlined and orphaned frames above match_idx */
+ while (cs->depth > 0 && cs->depth - 1 > match_idx) {
+ struct ftrace_stack_frame *top = &cs->stack[cs->depth - 1];
+
+ if (top->inlined) {
+ int exit_indent = top->indent;
+
+ if (cs->inlined_depth > 0)
+ cs->inlined_depth--;
+ fprintf(out, "%s%*s} /* %s (inline) */\n",
+ blank_prefix, exit_indent, "", top->name);
+ }
+ free(top->name);
+ cs->depth--;
+ }
+
+ /* Pop the matched real frame */
+ if (cs->depth > 0 && match_idx == cs->depth - 1 &&
+ !cs->stack[cs->depth - 1].inlined) {
+ free(cs->stack[cs->depth - 1].name);
+ cs->depth--;
+ }
+
+ extra_indent = cs->inlined_depth * 2;
+ } else {
+ extra_indent = 0;
+ }
+
+ fprintf(out, "%.*s%*s%s\n",
+ (int)(pipe_char - line + 1), line,
+ leading_spaces + extra_indent, "", func_start);
+ return 0;
+ }
+
+ /* It's an entry or a leaf call */
+ has_open_brace = (strchr(func_start, '{') != NULL);
+ has_semicolon = (strchr(func_start, ';') != NULL);
+
+ if (ftrace_parse_retaddr(func_start, caller_sym, sizeof(caller_sym), &caller_offset)) {
+ has_retaddr = true;
+ num_inlines = ftrace_resolve_inlines(ftrace->machine, caller_sym,
+ caller_offset, inlined_names,
+ FTRACE_MAX_INLINES);
+ }
+
+ /* Determine which inlined frames are currently open above the caller */
+ caller_idx = -1;
+ if (has_retaddr) {
+ for (i = cs->depth - 1; i >= 0; i--) {
+ if (!cs->stack[i].inlined && cs->stack[i].name &&
+ strcmp(cs->stack[i].name, caller_sym) == 0) {
+ caller_idx = i;
+ break;
+ }
+ }
+
+ /* If caller is not on the tracked stack, do not synthesise inlines */
+ if (caller_idx < 0) {
+ for (i = 0; i < num_inlines; i++)
+ free((void *)inlined_names[i]);
+ num_inlines = 0;
+ }
+ }
+
+ active_inlines = 0;
+ if (caller_idx >= 0) {
+ for (i = caller_idx + 1; i < cs->depth; i++) {
+ if (cs->stack[i].inlined)
+ active_inlines++;
+ else
+ break;
+ }
+ }
+
+ /* Find common prefix of inlined functions */
+ common = 0;
+ if (caller_idx >= 0) {
+ while (common < active_inlines && common < num_inlines &&
+ cs->stack[caller_idx + 1 + common].name &&
+ !strcmp(cs->stack[caller_idx + 1 + common].name,
+ inlined_names[common])) {
+ common++;
+ }
+ }
+
+ /* Close inlined frames that are no longer active */
+ while (active_inlines > common && cs->depth > 0) {
+ if (cs->stack[cs->depth - 1].inlined) {
+ struct ftrace_stack_frame *top = &cs->stack[cs->depth - 1];
+ int exit_indent = top->indent;
+
+ if (cs->inlined_depth > 0)
+ cs->inlined_depth--;
+ fprintf(out, "%s%*s} /* %s (inline) */\n",
+ blank_prefix, exit_indent, "", top->name);
+ free(top->name);
+ cs->depth--;
+ active_inlines--;
+ } else {
+ break;
+ }
+ }
+
+ /* Open new inlined frames */
+ for (i = common; i < num_inlines; i++) {
+ int entry_indent = leading_spaces + cs->inlined_depth * 2;
+
+ if (!inlined_names[i]) {
+ pr_err("Not enough memory\n");
+ for (int j = 0; j < num_inlines; j++)
+ free((void *)inlined_names[j]);
+ return -ENOMEM;
+ }
+ if (cs->depth >= FTRACE_MAX_STACK)
+ break;
+
+ fprintf(out, "%s%*s%s() { /* (inline) */\n",
+ blank_prefix, entry_indent, "", inlined_names[i]);
+ cs->stack[cs->depth].name = strdup(inlined_names[i]);
+ if (!cs->stack[cs->depth].name) {
+ pr_err("Not enough memory\n");
+ for (int j = 0; j < num_inlines; j++)
+ free((void *)inlined_names[j]);
+ return -ENOMEM;
+ }
+ cs->stack[cs->depth].inlined = true;
+ cs->stack[cs->depth].indent = entry_indent;
+ cs->depth++;
+ cs->inlined_depth++;
+ }
+
+ /* Print current line with updated indentation */
+ extra_indent = cs->inlined_depth * 2;
+
+ if (ftrace->filter_retaddr) {
+ char *filtered = strdup(func_start);
+
+ if (filtered) {
+ ftrace_filter_retaddr(filtered);
+ fprintf(out, "%.*s%*s%s\n",
+ (int)(pipe_char - line + 1), line,
+ leading_spaces + extra_indent, "", filtered);
+ free(filtered);
+ } else {
+ pr_err("Not enough memory\n");
+ for (i = 0; i < num_inlines; i++)
+ free((void *)inlined_names[i]);
+ return -ENOMEM;
+ }
+ } else {
+ fprintf(out, "%.*s%*s%s\n",
+ (int)(pipe_char - line + 1), line,
+ leading_spaces + extra_indent, "", func_start);
+ }
+
+ /* If non-leaf entry, push onto stack */
+ if (has_open_brace && !has_semicolon) {
+ char entry_name[128];
+
+ extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
+ if (cs->depth < FTRACE_MAX_STACK) {
+ cs->stack[cs->depth].name = strdup(entry_name);
+ if (!cs->stack[cs->depth].name) {
+ pr_err("Not enough memory\n");
+ for (i = 0; i < num_inlines; i++)
+ free((void *)inlined_names[i]);
+ return -ENOMEM;
+ }
+ cs->stack[cs->depth].inlined = false;
+ cs->stack[cs->depth].indent = leading_spaces + extra_indent;
+ cs->depth++;
+ }
+ }
+
+ for (i = 0; i < num_inlines; i++)
+ free((void *)inlined_names[i]);
+ return 0;
+}
+
+/**
+ * perf_ftrace__setup_inlines - Initialise symbol resolution for inlines
+ * @ftrace: Perf ftrace context
+ *
+ * Initialise symbol tracking and load the kernel image (vmlinux) for
+ * DWARF debug information parsing.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace)
+{
+ struct map *kmap;
+
+ if (!ftrace->use_inline)
+ return 0;
+
+ symbol_conf.try_vmlinux_path = (symbol_conf.vmlinux_name == NULL);
+ symbol_conf.inline_name = true;
+ if (symbol_conf.vmlinux_name)
+ symbol_conf.ignore_vmlinux_buildid = true;
+
+ if (symbol__init(NULL) < 0) {
+ pr_warning("Failed to initialize symbols for inlined functions\n");
+ return -1;
+ }
+
+ ftrace->machine = machine__new_host(NULL);
+ if (!ftrace->machine) {
+ pr_warning("Failed to create host machine for symbols\n");
+ return -1;
+ }
+
+ if (symbol_conf.vmlinux_name) {
+ kmap = machine__kernel_map(ftrace->machine);
+ if (kmap)
+ dso__load_vmlinux(map__dso(kmap), kmap, symbol_conf.vmlinux_name, false);
+ } else {
+ machine__load_vmlinux_path(ftrace->machine);
+ }
+
+ return 0;
+}
+
+/**
+ * perf_ftrace__cleanup_inlines - Clean up resources used for inline resolution
+ * @ftrace: Perf ftrace context
+ *
+ * Free the machine context and the CPU stack tracking states.
+ */
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace)
+{
+ if (ftrace->machine) {
+ machine__delete(ftrace->machine);
+ ftrace->machine = NULL;
+ }
+ ftrace_cpu_states__clear();
+}
diff --git a/tools/perf/util/ftrace.h b/tools/perf/util/ftrace.h
index 50216d3fef64..4140902d9de6 100644
--- a/tools/perf/util/ftrace.h
+++ b/tools/perf/util/ftrace.h
@@ -41,6 +41,8 @@ struct perf_ftrace {
int graph_tail;
bool filter_retaddr;
bool filter_retaddr_set;
+ bool use_inline;
+ struct machine *machine;
};
struct filter_entry {
@@ -95,7 +97,16 @@ perf_ftrace__latency_cleanup_bpf(struct perf_ftrace *ftrace __maybe_unused)
#endif /* HAVE_BPF_SKEL */
+struct machine;
+
bool ftrace_parse_retaddr(const char *str, char *sym_name, size_t sym_len, u64 *offset);
void ftrace_filter_retaddr(char *str);
+int ftrace_resolve_inlines(struct machine *machine,
+ const char *sym_name, u64 offset,
+ const char **inlined_names, int max_inlines);
+int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FILE *out);
+void ftrace_cpu_states__clear(void);
+int perf_ftrace__setup_inlines(struct perf_ftrace *ftrace);
+void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace);
#endif /* __PERF_FTRACE_H__ */
--
2.55.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer
2026-10-12 0:30 ` [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer Aaron Tomlin
@ 2026-10-12 0:43 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-12 0:43 UTC (permalink / raw)
To: Aaron Tomlin; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] perf: machine__delete() used instead of machine__put() violating refcount API
- [Low] perf: missing explicit POSIX libc headers in tools/perf/tests/ftrace.c
- [Low] perf ftrace: make_blank_prefix clobbers comm-pid column in verbose mode
--
commit 9964df54e91214dc84e330cbac056548e3af8bc4
Author: Aaron Tomlin <atomlin@atomlin.com>
perf ftrace: Support display of inlined functions in function graph tracer
This commit introduces the --inline option to perf ftrace for the function
graph tracer. It utilizes DWARF debug information to dynamically resolve and
synthesize inlined function frames directly into the streamed call graph.
> diff --git a/tools/perf/tests/ftrace.c b/tools/perf/tests/ftrace.c
> index 596cecee844e3..e856bed00700a 100644
> --- a/tools/perf/tests/ftrace.c
> +++ b/tools/perf/tests/ftrace.c
> @@ -7,6 +7,10 @@
> #include <unistd.h>
> #include "tests.h"
> #include "debug.h"
> +#include "dso.h"
> +#include "machine.h"
> +#include "map.h"
> +#include "symbol.h"
> #include "util/ftrace.h"
[ ... ]
> +static int test_inline_processing(void)
> +{
> + static const char line_entry[] = " 0) | wake_up_new_task() {";
[ ... ]
> + struct perf_ftrace ftrace;
> + char *out_buf = NULL;
> + size_t out_len = 0;
> + FILE *out;
> + const char *unmatched_pos;
> + const char *inline_exit_pos;
> + int ret = TEST_OK;
> +
> + memset(&ftrace, 0, sizeof(ftrace));
> + ftrace.use_inline = true;
> + ftrace.filter_retaddr = 1; /* Default when --graph-opts retaddr is not specified */
> + ftrace.tracer = "function_graph";
[ ... ]
> + out = open_memstream(&out_buf, &out_len);
[Severity: Low]
Is it possible that test_inline_processing() and other functions in this test
file will fail to compile under musl libc?
The perf subsystem guidelines require files using POSIX standard libc functions,
constants, or variables to directly include the corresponding headers. This code
makes heavy use of string/memory functions like memset() and file streams like
open_memstream(), but the file does not include <string.h> or <stdio.h>. Could
these headers be explicitly included to avoid relying on implicit glibc
namespace pollution?
[ ... ]
> diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
> index 344d36293f5ca..8737bb52921e1 100644
> --- a/tools/perf/util/ftrace.c
> +++ b/tools/perf/util/ftrace.c
> @@ -1,13 +1,82 @@
[ ... ]
> +static void make_blank_prefix(const char *line, const char *pipe_char, char *buf, size_t size)
> +{
> + size_t len = pipe_char - line + 1;
> + const char *paren = strchr(line, ')');
> + char *c;
> +
> + if (len >= size)
> + len = size - 1;
> +
> + memcpy(buf, line, len);
> + buf[len] = '\0';
> +
> + /* Replace duration between ')' and '|' with spaces */
> + if (paren && paren < line + len) {
> + char *end = buf + (pipe_char - line);
> +
> + if (end > buf + len)
> + end = buf + len;
> + for (c = buf + (paren - line) + 1; c < end; c++)
> + *c = ' ';
> + }
> +}
[Severity: Low]
Does make_blank_prefix() inadvertently erase the comm-pid string when in
verbose mode?
In function_graph verbose mode (--graph-opts verbose), the trace output
includes a comm-pid column and a duration column between the CPU number and
the function name. This loop overwrites everything between the closing
parenthesis of the CPU number and the final pipe delimiter before the function
name with spaces. While this correctly blanks out the duration column, it
looks like it might unintentionally clobber the task comm-pid string and its
pipe delimiter as well, breaking visual alignment.
[ ... ]
> +void perf_ftrace__cleanup_inlines(struct perf_ftrace *ftrace)
> +{
> + if (ftrace->machine) {
> + machine__delete(ftrace->machine);
> + ftrace->machine = NULL;
> + }
> + ftrace_cpu_states__clear();
> +}
[Severity: Medium]
Should machine__put() be used here instead of machine__delete()?
The machine reference is allocated using machine__new_host() earlier in
perf_ftrace__setup_inlines(). The perf subsystem guidelines mandate strict
reference count checking, stating that every reference handle allocated via
_new() must be strictly paired with a matching _put(). Bypassing the refcount
API by calling machine__delete() directly can lead to memory leaks,
use-after-free traps, and LSAN failures.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261012003016.263063-1-atomlin@atomlin.com?part=3
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration in function graph tracer
2026-10-12 0:30 [PATCH perf-tools-next v3 0/5] perf ftrace: Support inlined functions and display enhancements in function graph tracer Aaron Tomlin
` (2 preceding siblings ...)
2026-10-12 0:30 ` [PATCH perf-tools-next v3 3/5] perf ftrace: Support display of inlined functions in function graph tracer Aaron Tomlin
@ 2026-10-12 0:30 ` Aaron Tomlin
2026-10-12 0:39 ` sashiko-bot
2026-10-12 0:30 ` [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions " Aaron Tomlin
4 siblings, 1 reply; 11+ messages in thread
From: Aaron Tomlin @ 2026-10-12 0:30 UTC (permalink / raw)
To: peterz, mingo, acme, namhyung
Cc: mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter,
atomlin, linux-perf-users, linux-kernel
The Linux kernel's function graph tracer displays the execution duration
of completed functions by default via the funcgraph-duration trace
option. Whilst informative, in scenarios where the user is solely
interested in the call graph topology, call flow sequence, or inlined
hierarchy, the duration column can add visual clutter and widen each
trace line.
Introduce support for omitting function duration by exposing the
noduration sub-option under --graph-opts. Consistent with the existing
nosleep-time and noirqs sub-options, it accepts an optional value, so
noduration=0 explicitly retains the duration column; when specified
more than once, the last occurrence takes effect. When enabled, perf
ftrace configures the kernel's tracing instance option
funcgraph-duration to 0, and restores it to 1 upon resetting tracing
options.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
tools/perf/Documentation/perf-ftrace.txt | 1 +
tools/perf/builtin-ftrace.c | 28 +++++++++++++++++++++++-
tools/perf/tests/ftrace.c | 25 +++++++++++++++++----
tools/perf/util/ftrace.c | 15 +++++++++----
tools/perf/util/ftrace.h | 1 +
5 files changed, 61 insertions(+), 9 deletions(-)
diff --git a/tools/perf/Documentation/perf-ftrace.txt b/tools/perf/Documentation/perf-ftrace.txt
index 86306caf569b..a584583936bb 100644
--- a/tools/perf/Documentation/perf-ftrace.txt
+++ b/tools/perf/Documentation/perf-ftrace.txt
@@ -128,6 +128,7 @@ OPTIONS for 'perf ftrace trace'
- retaddr - Show function return address.
- nosleep-time - Measure on-CPU time only for function_graph tracer.
- noirqs - Ignore functions that happen inside interrupt.
+ - noduration - Do not print function duration.
- verbose - Show process names, PIDs, timestamps, etc.
- thresh=<n> - Setup trace duration threshold in microseconds.
- depth=<n> - Set max depth for function graph tracer to follow.
diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
index 504007064e24..926726c5d9a8 100644
--- a/tools/perf/builtin-ftrace.c
+++ b/tools/perf/builtin-ftrace.c
@@ -299,6 +299,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
write_tracing_option_file("func_stack_trace", "0");
write_tracing_option_file("sleep-time", "1");
write_tracing_option_file("funcgraph-irqs", "1");
+ write_tracing_option_file("funcgraph-duration", "1");
write_tracing_option_file("funcgraph-proc", "0");
write_tracing_option_file("funcgraph-abstime", "0");
write_tracing_option_file("funcgraph-tail", "0");
@@ -598,6 +599,26 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
return 0;
}
+/**
+ * set_tracing_funcgraph_duration - Set funcgraph-duration tracing option
+ * @ftrace: Perf ftrace context
+ *
+ * Configure the kernel's funcgraph-duration tracing option based on the
+ * ftrace->graph_noduration flag.
+ *
+ * Return: 0 on success, or a negative error code on failure.
+ */
+static int set_tracing_funcgraph_duration(struct perf_ftrace *ftrace)
+{
+ if (!ftrace->graph_noduration)
+ return 0;
+
+ if (write_tracing_option_file("funcgraph-duration", "0") < 0)
+ return -1;
+
+ return 0;
+}
+
static int set_tracing_funcgraph_verbose(struct perf_ftrace *ftrace)
{
if (!ftrace->graph_verbose)
@@ -707,6 +728,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)
return -1;
}
+ if (set_tracing_funcgraph_duration(ftrace) < 0) {
+ pr_err("failed to set tracing option funcgraph-duration\n");
+ return -1;
+ }
+
if (set_tracing_funcgraph_verbose(ftrace) < 0) {
pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime\n");
return -1;
@@ -1754,6 +1780,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
{ .name = "retaddr", .value_ptr = &ftrace->graph_retaddr },
{ .name = "nosleep-time", .value_ptr = &ftrace->graph_nosleep_time },
{ .name = "noirqs", .value_ptr = &ftrace->graph_noirqs },
+ { .name = "noduration", .value_ptr = &ftrace->graph_noduration },
{ .name = "verbose", .value_ptr = &ftrace->graph_verbose },
{ .name = "thresh", .value_ptr = &ftrace->graph_thresh },
{ .name = "depth", .value_ptr = &ftrace->graph_depth },
@@ -1843,7 +1870,6 @@ int cmd_ftrace(int argc, const char **argv)
OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
"Set nograph filter on given functions", parse_filter_func),
OPT_CALLBACK(0, "graph-opts", &ftrace, "options",
- "Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,verbose,thresh=<n>,depth=<n>",
"Graph tracer options, available options: args,retval,retval-hex,retaddr,nosleep-time,noirqs,noduration,verbose,thresh=<n>,depth=<n>,tail",
parse_graph_tracer_opts),
OPT_CALLBACK('m', "buffer-size", &ftrace.percpu_buffer_size, "size",
diff --git a/tools/perf/tests/ftrace.c b/tools/perf/tests/ftrace.c
index e856bed00700..cb212339e666 100644
--- a/tools/perf/tests/ftrace.c
+++ b/tools/perf/tests/ftrace.c
@@ -303,20 +303,36 @@ static int test_column_layouts(void)
{
static const struct {
const char *desc;
- bool verbose;
+ bool noduration;
+ bool verbose;
const char *in;
const char *expected;
} cases[] = {
{
- "default layout", false,
+ "default layout", false, false,
" 7) | wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
" 7) | wake_up_new_task() {\n",
},
{
- "verbose layout", true,
+ "noduration layout", true, false,
+ " 7) housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+ " 7) housekeeping_any_cpu();\n",
+ },
+ {
+ "verbose layout", false, true,
" 1234.567890 | 7) bash-1234 | | wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
" 1234.567890 | 7) bash-1234 | | wake_up_new_task() {\n",
},
+ {
+ "verbose noduration layout", true, true,
+ " 1234.567890 | 7) bash-1234 | wake_up_new_task() { /* <-kernel_clone+0x160/0x510 */",
+ " 1234.567890 | 7) bash-1234 | wake_up_new_task() {\n",
+ },
+ {
+ "noduration context switch marker", true, false,
+ " 7) bash-1234 => sh-5678",
+ " 7) bash-1234 => sh-5678\n",
+ },
};
struct perf_ftrace ftrace;
char *out_buf = NULL;
@@ -328,7 +344,8 @@ static int test_column_layouts(void)
memset(&ftrace, 0, sizeof(ftrace));
ftrace.use_inline = true;
ftrace.filter_retaddr = true;
- ftrace.graph_verbose = cases[i].verbose;
+ ftrace.graph_noduration = cases[i].noduration;
+ ftrace.graph_verbose = cases[i].verbose;
out = open_memstream(&out_buf, &out_len);
if (!out)
diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
index 8737bb52921e..5cae01a28529 100644
--- a/tools/perf/util/ftrace.c
+++ b/tools/perf/util/ftrace.c
@@ -315,10 +315,12 @@ static void make_blank_prefix(const char *line, const char *pipe_char, char *buf
* [ abstime |] CPU) [ comm-pid |] [ duration |] func
*
* The abstime and comm-pid columns are only present with --graph-opts
- * verbose.
+ * verbose, whereas the duration column, and thus its '|' delimiter, is
+ * omitted with --graph-opts noduration.
*
* Return: A pointer to the last character of the context columns (the final
- * '|' delimiter). Returns NULL if @line is not a trace record.
+ * '|' delimiter, or the CPU ')' when no delimiter follows it). Returns NULL
+ * if @line is not a trace record (e.g. a header or a context switch marker).
*/
static const char *find_func_column(const struct perf_ftrace *ftrace,
const char *line, int *cpu)
@@ -346,9 +348,14 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
return NULL;
}
- p = strchr(p + 1, '|');
- if (!p)
+ if (!ftrace->graph_noduration) {
+ p = strchr(p + 1, '|');
+ if (!p)
+ return NULL;
+ } else if (strstr(p, " => ")) {
+ /* Context switch marker: " CPU) comm-pid => comm-pid" */
return NULL;
+ }
*cpu = (int)val;
return p;
diff --git a/tools/perf/util/ftrace.h b/tools/perf/util/ftrace.h
index 4140902d9de6..d202a09d112f 100644
--- a/tools/perf/util/ftrace.h
+++ b/tools/perf/util/ftrace.h
@@ -36,6 +36,7 @@ struct perf_ftrace {
int graph_retaddr;
int graph_nosleep_time;
int graph_noirqs;
+ int graph_noduration;
int graph_verbose;
int graph_thresh;
int graph_tail;
--
2.55.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration in function graph tracer
2026-10-12 0:30 ` [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration " Aaron Tomlin
@ 2026-10-12 0:39 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-12 0:39 UTC (permalink / raw)
To: Aaron Tomlin; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Low] perf: formatting corruption in make_blank_prefix() with verbose enabled
- [Low] perf: profile subcommand silently drops records with noduration
--
commit d0f3a0ada6164d2372b897fe5b75d0f425aca5e4
Author: Aaron Tomlin <atomlin@atomlin.com>
perf ftrace: Support omitting execution duration in function graph tracer
The Linux kernel's function graph tracer displays the execution duration
of completed functions by default via the funcgraph-duration trace
option. Whilst informative, in scenarios where the user is solely
interested in the call graph topology, call flow sequence, or inlined
hierarchy, the duration column can add visual clutter and widen each
trace line.
[ ... ]
> diff --git a/tools/perf/builtin-ftrace.c b/tools/perf/builtin-ftrace.c
> index 504007064e240..926726c5d9a88 100644
> --- a/tools/perf/builtin-ftrace.c
> +++ b/tools/perf/builtin-ftrace.c
[ ... ]
> @@ -1754,6 +1780,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
> { .name = "retaddr", .value_ptr = &ftrace->graph_retaddr },
> { .name = "nosleep-time", .value_ptr = &ftrace->graph_nosleep_time },
> { .name = "noirqs", .value_ptr = &ftrace->graph_noirqs },
> + { .name = "noduration", .value_ptr = &ftrace->graph_noduration },
[Severity: Low]
Does adding noduration here allow it to be used with the profile subcommand?
The profile subcommand inherently requires function duration to compute
statistics. When the duration column is omitted, parse_func_duration()
sees no " us" suffix and silently drops the record:
tools/perf/builtin-ftrace.c:parse_func_duration() {
...
if (strncmp(p, " us", 3)) {
pr_debug("non-usec time found.. ignoring\n");
return 0;
}
...
}
This results in an empty output instead of rejecting the invalid option
combination. Should the option be rejected when used with profile?
> { .name = "verbose", .value_ptr = &ftrace->graph_verbose },
> { .name = "thresh", .value_ptr = &ftrace->graph_thresh },
> { .name = "depth", .value_ptr = &ftrace->graph_depth },
[ ... ]
> diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
> index 8737bb52921e1..5cae01a28529f 100644
> --- a/tools/perf/util/ftrace.c
> +++ b/tools/perf/util/ftrace.c
[ ... ]
> @@ -346,9 +348,14 @@ static const char *find_func_column(const struct perf_ftrace *ftrace,
> return NULL;
> }
>
> - p = strchr(p + 1, '|');
> - if (!p)
> + if (!ftrace->graph_noduration) {
> + p = strchr(p + 1, '|');
> + if (!p)
> + return NULL;
> + } else if (strstr(p, " => ")) {
> + /* Context switch marker: " CPU) comm-pid => comm-pid" */
> return NULL;
> + }
[Severity: Low]
Will this corrupt the comm-pid column when verbose is enabled alongside
noduration?
When both verbose and noduration are enabled, find_func_column() returns
the CPU ')' or the separator after it. make_blank_prefix() uses the final
pipe_char and overwrites everything after the ')' with spaces:
tools/perf/util/ftrace.c:make_blank_prefix() {
...
/* Replace duration between ')' and '|' with spaces */
if (paren && paren < line + len) {
char *end = buf + (pipe_char - line);
...
for (c = buf + (paren - line) + 1; c < end; c++)
*c = ' ';
}
...
}
Since there is no duration field, does this blindly overwrite the comm-pid
column and its trailing separator?
>
> *cpu = (int)val;
> return p;
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261012003016.263063-1-atomlin@atomlin.com?part=4
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions in function graph tracer
2026-10-12 0:30 [PATCH perf-tools-next v3 0/5] perf ftrace: Support inlined functions and display enhancements in function graph tracer Aaron Tomlin
` (3 preceding siblings ...)
2026-10-12 0:30 ` [PATCH perf-tools-next v3 4/5] perf ftrace: Support omitting execution duration " Aaron Tomlin
@ 2026-10-12 0:30 ` Aaron Tomlin
2026-10-12 0:37 ` sashiko-bot
4 siblings, 1 reply; 11+ messages in thread
From: Aaron Tomlin @ 2026-10-12 0:30 UTC (permalink / raw)
To: peterz, mingo, acme, namhyung
Cc: mark.rutland, alexander.shishkin, jolsa, irogers, adrian.hunter,
atomlin, linux-perf-users, linux-kernel
When tracing with the function graph tracer using a targeted filter
(-G or --graph-funcs), users occasionally observe unrelated functions in
the trace output:
1. Top-level functions (e.g. most notably
finish_task_switch.isra.0()) appearing before or alongside the
requested target function. This occurs because clearing and
rewriting set_graph_function creates a transient window where
remote CPUs still observe an empty filter (tracing all
functions) before RCU synchronisation completes, coupled with
perf ftrace sleeping in poll() on trace_pipe right as tracing
starts.
2. Detached/orphan calls inside the target callgraph (e.g.
housekeeping_any_cpu() and arch_irq_work_raise()) appearing as
child frames under the target function. These routines are
invoked by rb_wakeups() to notify perf's poll() on trace_pipe
whenever an event is committed, and execute while the target
function is on the CPU.
Resolve both issues deterministically in userspace:
1. Track active target callgraph state per CPU in
tools/perf/util/ftrace.c. Any top-level entry not matching the
requested -G targets is suppressed along with its nested
children until its scope exits.
2. Within an active target callgraph, validate that child entries
descend from an active frame on the target stack using their
return addresses (CONFIG_FUNCTION_GRAPH_RETADDR). Any detached
calls originating outside the target hierarchy (such as
ring-buffer wakeups) are suppressed along with their subtrees.
For example:
# tracer: function_graph
#
# CPU FUNCTION CALLS
# | | | | |
3) activate_task() {
3) enqueue_task() {
3) psi_enqueue() { /* (inline) */
3) arch_static_branch_jump() { /* (inline) */
3) enqueue_task_fair() {
3) enqueue_hierarchy() { /* (inline) */
3) update_curr();
3) enqueue_entity() {
3) account_entity_enqueue() { /* (inline) */
3) account_llc_enqueue();
3) } /* account_entity_enqueue (inline) */
3) }
3) } /* enqueue_hierarchy (inline) */
3) place_entity() {
3) avg_vruntime();
3) }
3) __enqueue_entity();
3) dl_server_start();
3) }
3) } /* arch_static_branch_jump (inline) */
3) } /* psi_enqueue (inline) */
3) sched_info_enqueue() { /* (inline) */
3) psi_task_change() {
3) psi_write_begin() { /* (inline) */
3) psi_flags_change();
3) } /* psi_write_begin (inline) */
3) psi_group_change() {
3) record_times();
3) }
3) }
3) } /* sched_info_enqueue (inline) */
3) }
3) }
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
---
tools/perf/tests/ftrace.c | 191 ++++++++++++++++++++++++++++++++++++++
tools/perf/util/ftrace.c | 174 ++++++++++++++++++++++++++++++++--
2 files changed, 359 insertions(+), 6 deletions(-)
diff --git a/tools/perf/tests/ftrace.c b/tools/perf/tests/ftrace.c
index cb212339e666..0b88feb97a1e 100644
--- a/tools/perf/tests/ftrace.c
+++ b/tools/perf/tests/ftrace.c
@@ -368,6 +368,193 @@ static int test_column_layouts(void)
return TEST_OK;
}
+static int test_graph_target_filter(void)
+{
+ static const char line_idle[] = " 7) 3.000 us | sched_idle_set_state();";
+ static const char line_sw_entry[] = " 6) | finish_task_switch.isra.0() {";
+ static const char line_sw_child[] = " 6) 1.000 us | _raw_spin_unlock();";
+ static const char line_sw_exit[] = " 6) 2.000 us | }";
+ static const char line_tgt_entry[] = " 2) | activate_task() {";
+ static const char line_tgt_child[] = " 2) 1.500 us | enqueue_task();";
+ static const char line_tgt_exit[] = " 2) 2.500 us | }";
+ struct perf_ftrace ftrace;
+ struct filter_entry *entry;
+ struct filter_entry *pos, *tmp;
+ char *out_buf = NULL;
+ size_t out_len = 0;
+ FILE *out;
+ int ret = TEST_OK;
+
+ memset(&ftrace, 0, sizeof(ftrace));
+ INIT_LIST_HEAD(&ftrace.graph_funcs);
+
+ entry = malloc(sizeof(*entry) + strlen("activate_task") + 1);
+ if (!entry)
+ return TEST_FAIL;
+ strcpy(entry->name, "activate_task");
+ list_add_tail(&entry->list, &ftrace.graph_funcs);
+
+ /* 1. Test without --inline */
+ ftrace.use_inline = false;
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_idle, out);
+ ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+ ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ if (strstr(out_buf, "sched_idle_set_state")) {
+ pr_debug("FAILED: sched_idle_set_state not suppressed\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (strstr(out_buf, "finish_task_switch")) {
+ pr_debug("FAILED: finish_task_switch not suppressed\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (strstr(out_buf, "_raw_spin_unlock")) {
+ pr_debug("FAILED: _raw_spin_unlock not suppressed\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (!strstr(out_buf, "activate_task() {")) {
+ pr_debug("FAILED: activate_task entry missing\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (!strstr(out_buf, "enqueue_task();")) {
+ pr_debug("FAILED: enqueue_task child missing\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* 2. Test with --inline */
+ ftrace.use_inline = true;
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_idle, out);
+ ftrace_process_fgraph_line(&ftrace, line_sw_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_sw_child, out);
+ ftrace_process_fgraph_line(&ftrace, line_sw_exit, out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_child, out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ pr_debug("Target filter output with --inline:\n%s\n", out_buf);
+
+ if (strstr(out_buf, "finish_task_switch")) {
+ pr_debug("FAILED: finish_task_switch not suppressed with --inline\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (!strstr(out_buf, "activate_task() {")) {
+ pr_debug("FAILED: activate_task entry missing with --inline\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+
+ free(out_buf);
+ out_buf = NULL;
+
+ /* 3. Test suppressing detached orphan calls within target graph */
+ ftrace.use_inline = true;
+ ftrace.filter_retaddr = 1;
+ out = open_memstream(&out_buf, &out_len);
+ if (!out) {
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+
+ ftrace_process_fgraph_line(&ftrace, line_tgt_entry, out);
+ /* Leaf orphan call from ring_buffer_unlock_commit */
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) | housekeeping_any_cpu(); /* <-ring_buffer_unlock_commit+0x1a0/0x1f0 */",
+ out);
+ /* Non-leaf orphan call from irq_work_queue_on */
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) | arch_irq_work_raise() { /* <-irq_work_queue_on+0x81/0xa0 */",
+ out);
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) | apic_wait_icr_idle() { /* (inline) */",
+ out);
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) | x2apic_send_IPI_self();",
+ out);
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) | } /* apic_wait_icr_idle (inline) */",
+ out);
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) 1.000 us | }",
+ out);
+ /* Genuine child call from activate_task */
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) | enqueue_task() { /* <-activate_task+0x1d1/0x3e0 */",
+ out);
+ ftrace_process_fgraph_line(&ftrace,
+ " 2) 1.500 us | }",
+ out);
+ ftrace_process_fgraph_line(&ftrace, line_tgt_exit, out);
+
+ fclose(out);
+ ftrace_cpu_states__clear();
+
+ pr_debug("Orphan call filter output:\n%s\n", out_buf);
+
+ if (strstr(out_buf, "housekeeping_any_cpu")) {
+ pr_debug("FAILED: housekeeping_any_cpu orphan not suppressed\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (strstr(out_buf, "arch_irq_work_raise")) {
+ pr_debug("FAILED: arch_irq_work_raise orphan not suppressed\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (strstr(out_buf, "x2apic_send_IPI_self")) {
+ pr_debug("FAILED: x2apic_send_IPI_self not suppressed\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+ if (!strstr(out_buf, "enqueue_task() {")) {
+ pr_debug("FAILED: genuine enqueue_task missing\n");
+ ret = TEST_FAIL;
+ goto out_free_filter;
+ }
+
+ free(out_buf);
+ out_buf = NULL;
+
+out_free_filter:
+ if (out_buf)
+ free(out_buf);
+ list_for_each_entry_safe(pos, tmp, &ftrace.graph_funcs, list) {
+ list_del(&pos->list);
+ free(pos);
+ }
+ return ret;
+}
+
static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __maybe_unused)
{
int ret;
@@ -388,6 +575,10 @@ static int test__ftrace(struct test_suite *test __maybe_unused, int subtest __ma
if (ret != TEST_OK)
return ret;
+ ret = test_graph_target_filter();
+ if (ret != TEST_OK)
+ return ret;
+
ret = test_inline_processing();
if (ret != TEST_OK)
return ret;
diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
index 5cae01a28529..96a41077dccb 100644
--- a/tools/perf/util/ftrace.c
+++ b/tools/perf/util/ftrace.c
@@ -34,6 +34,9 @@ struct ftrace_cpu_state {
struct ftrace_stack_frame stack[FTRACE_MAX_STACK];
int depth;
int inlined_depth;
+ bool in_target;
+ int target_root_depth;
+ int orphan_depth;
};
static struct ftrace_cpu_state *ftrace_cpu_states[FTRACE_MAX_CPUS];
@@ -393,6 +396,39 @@ static int print_line(struct perf_ftrace *ftrace, const char *line, FILE *out)
return 0;
}
+static inline bool ftrace_has_graph_funcs(const struct perf_ftrace *ftrace)
+{
+ return ftrace->graph_funcs.next != NULL && !list_empty(&ftrace->graph_funcs);
+}
+
+/**
+ * ftrace_is_graph_func - Check if a function matches the graph_funcs target filter
+ * @ftrace: Perf ftrace context
+ * @func_name: The function name to test
+ *
+ * Return: true if ftrace->graph_funcs is empty or @func_name matches a
+ * target filter, false otherwise.
+ */
+static bool ftrace_is_graph_func(const struct perf_ftrace *ftrace, const char *func_name)
+{
+ struct filter_entry *pos;
+
+ if (!ftrace_has_graph_funcs(ftrace))
+ return true;
+
+ list_for_each_entry(pos, &ftrace->graph_funcs, list) {
+ size_t len = strlen(pos->name);
+
+ if (strcmp(pos->name, func_name) == 0)
+ return true;
+ if (!strncmp(pos->name, func_name, len) &&
+ (func_name[len] == '\0' || func_name[len] == '.'))
+ return true;
+ }
+
+ return false;
+}
+
/**
* ftrace_process_fgraph_line - Process and output a function graph trace line
* @ftrace: Perf ftrace context
@@ -429,8 +465,9 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
bool has_open_brace;
bool has_semicolon;
bool has_retaddr = false;
+ bool has_filter = ftrace_has_graph_funcs(ftrace);
- if (!ftrace->use_inline)
+ if (!has_filter && !ftrace->use_inline)
return print_line(ftrace, line, out);
pipe_char = find_func_column(ftrace, line, &cpu);
@@ -455,6 +492,11 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
if (is_exit) {
char exit_name[128];
+ if (cs->orphan_depth > 0) {
+ cs->orphan_depth--;
+ return 0;
+ }
+
extract_exit_func_name(func_start, exit_name, sizeof(exit_name));
/* Find matching frame on stack */
@@ -480,8 +522,9 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
if (cs->inlined_depth > 0)
cs->inlined_depth--;
- fprintf(out, "%s%*s} /* %s (inline) */\n",
- blank_prefix, exit_indent, "", top->name);
+ if (!has_filter || cs->in_target)
+ fprintf(out, "%s%*s} /* %s (inline) */\n",
+ blank_prefix, exit_indent, "", top->name);
}
free(top->name);
cs->depth--;
@@ -499,9 +542,21 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
extra_indent = 0;
}
- fprintf(out, "%.*s%*s%s\n",
- (int)(pipe_char - line + 1), line,
- leading_spaces + extra_indent, "", func_start);
+ if (!has_filter || cs->in_target) {
+ if (!ftrace->use_inline) {
+ print_line(ftrace, line, out);
+ } else {
+ fprintf(out, "%.*s%*s%s\n",
+ (int)(pipe_char - line + 1), line,
+ leading_spaces + extra_indent, "", func_start);
+ }
+ }
+
+ if (has_filter && cs->in_target && cs->depth <= cs->target_root_depth) {
+ cs->in_target = false;
+ cs->target_root_depth = 0;
+ cs->orphan_depth = 0;
+ }
return 0;
}
@@ -509,6 +564,92 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
has_open_brace = (strchr(func_start, '{') != NULL);
has_semicolon = (strchr(func_start, ';') != NULL);
+ if (cs->orphan_depth > 0) {
+ if (has_open_brace && !has_semicolon)
+ cs->orphan_depth++;
+ return 0;
+ }
+
+ if (has_filter) {
+ if (cs->depth == 0 || !cs->in_target) {
+ char entry_name[128];
+
+ extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
+ if (ftrace_is_graph_func(ftrace, entry_name)) {
+ cs->in_target = true;
+ cs->target_root_depth = cs->depth;
+ cs->orphan_depth = 0;
+ } else if (cs->depth == 0) {
+ cs->in_target = false;
+ cs->orphan_depth = 0;
+ }
+ }
+
+ if (!cs->in_target) {
+ /* Non-matching call tree; track depth if entering scope, suppress output */
+ if (has_open_brace && !has_semicolon) {
+ char entry_name[128];
+
+ extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
+ if (cs->depth < FTRACE_MAX_STACK) {
+ cs->stack[cs->depth].name = strdup(entry_name);
+ if (!cs->stack[cs->depth].name) {
+ pr_err("Not enough memory\n");
+ return -ENOMEM;
+ }
+ cs->stack[cs->depth].inlined = false;
+ cs->stack[cs->depth].indent = leading_spaces;
+ cs->depth++;
+ }
+ }
+ return 0;
+ }
+ }
+
+ if (!ftrace->use_inline) {
+ if (has_filter && cs->in_target && cs->depth > cs->target_root_depth) {
+ if (ftrace_parse_retaddr(func_start, caller_sym,
+ sizeof(caller_sym), &caller_offset)) {
+ caller_idx = -1;
+ for (i = cs->depth - 1; i >= 0; i--) {
+ if (!cs->stack[i].inlined && cs->stack[i].name &&
+ strcmp(cs->stack[i].name, caller_sym) == 0) {
+ caller_idx = i;
+ break;
+ }
+ }
+ if (caller_idx < cs->target_root_depth) {
+ if (has_open_brace && !has_semicolon)
+ cs->orphan_depth = 1;
+ return 0;
+ }
+ }
+ }
+
+ if (has_open_brace && !has_semicolon) {
+ char entry_name[128];
+
+ extract_entry_func_name(func_start, entry_name, sizeof(entry_name));
+ if (cs->depth < FTRACE_MAX_STACK) {
+ cs->stack[cs->depth].name = strdup(entry_name);
+ if (!cs->stack[cs->depth].name) {
+ pr_err("Not enough memory\n");
+ return -ENOMEM;
+ }
+ cs->stack[cs->depth].inlined = false;
+ cs->stack[cs->depth].indent = leading_spaces;
+ cs->depth++;
+ }
+ }
+ print_line(ftrace, line, out);
+ if (has_filter && cs->in_target && cs->depth == 0 &&
+ has_semicolon && !has_open_brace) {
+ cs->in_target = false;
+ cs->orphan_depth = 0;
+ }
+ return 0;
+ }
+
if (ftrace_parse_retaddr(func_start, caller_sym, sizeof(caller_sym), &caller_offset)) {
has_retaddr = true;
num_inlines = ftrace_resolve_inlines(ftrace->machine, caller_sym,
@@ -535,6 +676,21 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
}
}
+ /*
+ * When inside a target graph filter, child functions must descend
+ * from an active frame on the target stack. Suppress detached
+ * calls originating outside the target hierarchy.
+ */
+ if (has_filter && cs->in_target && cs->depth > cs->target_root_depth) {
+ if (has_retaddr && caller_idx < cs->target_root_depth) {
+ if (has_open_brace && !has_semicolon)
+ cs->orphan_depth = 1;
+ for (i = 0; i < num_inlines; i++)
+ free((void *)inlined_names[i]);
+ return 0;
+ }
+ }
+
active_inlines = 0;
if (caller_idx >= 0) {
for (i = caller_idx + 1; i < cs->depth; i++) {
@@ -645,6 +801,12 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
}
}
+ if (has_filter && cs->in_target && cs->depth == 0 &&
+ has_semicolon && !has_open_brace) {
+ cs->in_target = false;
+ cs->orphan_depth = 0;
+ }
+
for (i = 0; i < num_inlines; i++)
free((void *)inlined_names[i]);
return 0;
--
2.55.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions in function graph tracer
2026-10-12 0:30 ` [PATCH perf-tools-next v3 5/5] perf ftrace: Filter extraneous functions " Aaron Tomlin
@ 2026-10-12 0:37 ` sashiko-bot
0 siblings, 0 replies; 11+ messages in thread
From: sashiko-bot @ 2026-10-12 0:37 UTC (permalink / raw)
To: Aaron Tomlin; +Cc: linux-perf-users
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] perf ftrace: Target filter state leaks for leaf functions at non-zero depth
--
commit 1481567e7a5dda207ce5b61e39c1b59803ca97c4
Author: Aaron Tomlin <atomlin@atomlin.com>
perf ftrace: Filter extraneous functions in function graph tracer
This commit enhances perf ftrace to filter out extraneous top-level functions
and detached orphan calls from the function graph output when using a target
filter.
> diff --git a/tools/perf/util/ftrace.c b/tools/perf/util/ftrace.c
> index 5cae01a28529f..96a41077dccbf 100644
> --- a/tools/perf/util/ftrace.c
> +++ b/tools/perf/util/ftrace.c
[ ... ]
> @@ -509,6 +564,92 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
> has_open_brace = (strchr(func_start, '{') != NULL);
> has_semicolon = (strchr(func_start, ';') != NULL);
>
[ ... ]
> + print_line(ftrace, line, out);
> + if (has_filter && cs->in_target && cs->depth == 0 &&
> + has_semicolon && !has_open_brace) {
> + cs->in_target = false;
> + cs->orphan_depth = 0;
> + }
> + return 0;
> + }
[Severity: Medium]
When a targeted function is a leaf call (no braces, only a semicolon) and
occurs at a call stack depth greater than 0, doesn't the hardcoded check for
cs->depth == 0 prevent the filter state from resetting?
It appears that cs->in_target will remain true indefinitely if the target is
not at the root depth, causing subsequent non-targeted sibling functions and
their descendants within the same parent frame to bypass the filter and
incorrectly appear in the trace output.
[ ... ]
> @@ -645,6 +801,12 @@ int ftrace_process_fgraph_line(struct perf_ftrace *ftrace, const char *line, FIL
> }
> }
>
> + if (has_filter && cs->in_target && cs->depth == 0 &&
> + has_semicolon && !has_open_brace) {
> + cs->in_target = false;
> + cs->orphan_depth = 0;
> + }
> +
> for (i = 0; i < num_inlines; i++)
> free((void *)inlined_names[i]);
> return 0;
[Severity: Medium]
The identical logic exists here for the use_inline code path. Should these
checks verify cs->depth == cs->target_root_depth instead of cs->depth == 0
to properly reset the filter for leaf targets at non-zero depths?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261012003016.263063-1-atomlin@atomlin.com?part=5
^ permalink raw reply [flat|nested] 11+ messages in thread