From: Adrian Hunter <adrian.hunter@intel.com>
To: Leo Yan <leo.yan@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>,
Arnaldo Carvalho de Melo <acme@kernel.org>,
Namhyung Kim <namhyung@kernel.org>, Jiri Olsa <jolsa@kernel.org>,
Ian Rogers <irogers@google.com>, KP Singh <kpsingh@kernel.org>,
Matt Bobrowski <mattbobrowski@google.com>,
Song Liu <song@kernel.org>, Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Andrii Nakryiko <andrii@kernel.org>,
Martin KaFai Lau <martin.lau@linux.dev>,
"Eduard Zingerman" <eddyz87@gmail.com>,
Yonghong Song <yonghong.song@linux.dev>,
"John Fastabend" <john.fastabend@gmail.com>,
Stanislav Fomichev <sdf@fomichev.me>, Hao Luo <haoluo@google.com>,
Steven Rostedt <rostedt@goodmis.org>,
"Masami Hiramatsu" <mhiramat@kernel.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
James Clark <james.clark@linaro.org>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Mike Leach <mike.leach@linaro.org>,
<linux-perf-users@vger.kernel.org>,
<linux-kernel@vger.kernel.org>, <bpf@vger.kernel.org>,
<linux-trace-kernel@vger.kernel.org>
Subject: Re: [PATCH v3 0/6] perf auxtrace: Support AUX pause and resume with BPF
Date: Tue, 5 Aug 2025 22:16:29 +0300 [thread overview]
Message-ID: <0a0ed9d4-6511-4f0b-868f-22a3f95697f8@intel.com> (raw)
In-Reply-To: <20250730182623.GE143191@e132581.arm.com>
On 30/07/2025 21:26, Leo Yan wrote:
> Hi Adrian,
>
> On Mon, Jul 28, 2025 at 08:02:51PM +0300, Adrian Hunter wrote:
>> On 25/07/2025 12:59, Leo Yan wrote:
>>> This series extends Perf for fine-grained tracing by using BPF program
>>> to pause and resume AUX tracing. The BPF program can be attached to
>>> tracepoints (including ftrace tracepoints and dynamic tracepoints, like
>>> kprobe, kretprobe, uprobe and uretprobe).
>>
>> Using eBPF to pause/resume AUX tracing seems like a great idea.
>>
>> AFAICT with this patch set, there is just support for pause/resume
>> much like what could be done directly without eBPF, so I wonder if you
>> could share a bit more on how you see this evolving, and what your
>> future plans are?
>
> IIUC, here you mean the tool can use `perf probe` to firstly create
> probes, then enable tracepoints as PMU event for AUX pause and resume.
Yes, like:
$ sudo perf probe 'do_sys_openat2 how->flags how->mode'
Added new event:
probe:do_sys_openat2 (on do_sys_openat2 with flags=how->flags mode=how->mode)
You can now use it in all perf tools, such as:
perf record -e probe:do_sys_openat2 -aR sleep 1
$ sudo perf probe do_sys_openat2%return
Added new event:
probe:do_sys_openat2__return (on do_sys_openat2%return)
You can now use it in all perf tools, such as:
perf record -e probe:do_sys_openat2__return -aR sleep 1
$ sudo perf record --kcore -e intel_pt/aux-action=start-paused/k -e probe:do_sys_openat2/aux-action=resume/ --filter='flags==0x98800' -e probe:do_sys_openat2__return/aux-action=pause/ -- ls
arch certs CREDITS cscope.out drivers fs include io_uring Kbuild kernel LICENSES Makefile mm perf.data README samples security tools virt
block COPYING crypto Documentation init ipc Kconfig lib MAINTAINERS net rust scripts sound usr
[ perf record: Woken up 2 times to write data ]
[ perf record: Captured and wrote 0.067 MB perf.data ]
$ sudo perf script --itrace=qi | grep -B1 instructions ls 37607 [003] 36109.137560: probe:do_sys_openat2: (ffffffff9d2276a0) flags=0x98800 mode=0x0
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9cdc3834 native_write_msr+0x4 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9cdc3836 native_write_msr+0x6 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9cd26728 pt_config_start+0x58 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9cd27727 pt_event_start+0x107 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d0d5a04 perf_event_aux_pause+0x114 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d0d80f7 __perf_event_overflow+0x197 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d0d844d perf_swevent_event+0x12d ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d0d8738 perf_tp_event+0x188 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d00fad6 kprobe_perf_func+0x256 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d00fbbd kprobe_dispatcher+0x6d ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9cf80582 aggr_pre_handler+0x42 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9cdbcbb2 kprobe_ftrace_handler+0x152 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffffc12440f5 ftrace_trampoline+0xf5 ([kernel.kallsyms])
ls 37607 [003] 36109.137562: 1 instructions:k: ffffffff9d2276a5 do_sys_openat2+0x5 ([kernel.kallsyms])
ls 37607 [003] 36109.137563: 1 instructions:k: ffffffff9d4c3d60 hook_file_alloc_security+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137564: 1 instructions:k: ffffffff9d4a5050 apparmor_file_alloc_security+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137565: 1 instructions:k: ffffffff9d42d400 cap_capable+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137565: 1 instructions:k: ffffffff9d4a4b70 apparmor_capable+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137566: 1 instructions:k: ffffffff9d42d400 cap_capable+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137566: 1 instructions:k: ffffffff9d4a4b70 apparmor_capable+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137567: 1 instructions:k: ffffffff9d4c4e80 hook_file_open+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137567: 1 instructions:k: ffffffff9d4a5aa0 apparmor_file_open+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137567: 1 instructions:k: ffffffff9d31fb10 ext4_dir_open+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137567: 1 instructions:k: ffffffff9d4cc740 ima_file_check+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137567: 1 instructions:k: ffffffff9d4a5960 apparmor_current_getlsmprop_subj+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137568: 1 instructions:k: ffffffff9cdb76c0 arch_rethook_trampoline+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137568: 1 instructions:k: ffffffff9cf80670 kretprobe_rethook_handler+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137568: 1 instructions:k: ffffffff9d00fe90 kretprobe_dispatcher+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137568: 1 instructions:k: ffffffff9cd282c0 pt_event_stop+0x0 ([kernel.kallsyms])
ls 37607 [003] 36109.137569: 1 instructions:k: ffffffff9cdc3834 native_write_msr+0x4 ([kernel.kallsyms])
>
> I would say a benefit from this series is users can use a single
> command to create probes and bind eBPF program for AUX pause and
> resume in one go.
>
> To be honest, at current stage, I don't have clear idea for expanding
> this feature. But a clear requirement is: AUX trace data usually is
> quite huge, after initial analysis, developers might want to focus
> on specific function profiling (based on function entry and exit) or
> specific period (E.g., start tracing when hit a tracepoing and stop when
> hit another tracepoint).
>
> eBPF program is powerful. Basically, we can extend it in two different
> dimensions. One direction is we can easily attach the eBPF program to more
> kernel modules, like networking, storage, etc. Another direction is to
> improve the eBPF program itself as a filter for better fine-grained
> tracing, so far we only support limited filtering based on CPU ID or PID,
> we also can extend the filtering based on time, event types, etc.
next prev parent reply other threads:[~2025-08-05 19:16 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-07-25 9:59 [PATCH v3 0/6] perf auxtrace: Support AUX pause and resume with BPF Leo Yan
2025-07-25 9:59 ` [PATCH PATCH v2 v3 1/6] perf/core: Make perf_event_aux_pause() as external function Leo Yan
2025-07-25 9:59 ` [PATCH PATCH v2 v3 2/6] bpf: Add bpf_perf_event_aux_pause kfunc Leo Yan
2025-07-25 9:59 ` [PATCH PATCH v2 v3 3/6] perf: auxtrace: Control AUX pause and resume with BPF Leo Yan
2025-07-25 9:59 ` [PATCH PATCH v2 v3 4/6] perf: auxtrace: Add BPF userspace program for AUX pause and resume Leo Yan
2025-07-25 9:59 ` [PATCH PATCH v2 v3 5/6] perf record: Support AUX pause and resume with BPF Leo Yan
2025-07-25 9:59 ` [PATCH PATCH v2 v3 6/6] perf docs: Document " Leo Yan
2025-07-25 10:16 ` [PATCH v3 0/6] perf auxtrace: Support " Leo Yan
2025-07-28 17:02 ` Adrian Hunter
2025-07-30 18:26 ` Leo Yan
2025-08-05 19:16 ` Adrian Hunter [this message]
2025-08-08 11:47 ` Leo Yan
2025-09-04 16:38 ` Arnaldo Carvalho de Melo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=0a0ed9d4-6511-4f0b-868f-22a3f95697f8@intel.com \
--to=adrian.hunter@intel.com \
--cc=acme@kernel.org \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=eddyz87@gmail.com \
--cc=haoluo@google.com \
--cc=irogers@google.com \
--cc=james.clark@linaro.org \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kpsingh@kernel.org \
--cc=leo.yan@arm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=martin.lau@linux.dev \
--cc=mathieu.desnoyers@efficios.com \
--cc=mattbobrowski@google.com \
--cc=mhiramat@kernel.org \
--cc=mike.leach@linaro.org \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=sdf@fomichev.me \
--cc=song@kernel.org \
--cc=suzuki.poulose@arm.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox