[PATCH v1 0/7] perf auxtrace: Support AUX pause with BPF backend

[PATCH v1 0/7] perf auxtrace: Support AUX pause with BPF backend

[Leo Yan]

This series extends Perf's ability for fine-grained tracing by attaching
specific trace events to eBPF programs. As the first step, this series
supports kprobe, kretprobe, and tracepoints for dynamically pausing and
resuming AUX trace.

The first two patches expose a BPF API from kernel so the AUX pause and
resume can be invoked from a BPF kernel program.

Syncing UAPI headers between kernel and tools is finished in patch 03.

The main changes in the Perf tool for implementing eBPF skeleton
program, hooking BPF program in a perf record session, and attaching
trace events with BPF programs are finished in patches 04 ~ 06.

The patch 07 updates documentation for usage of the new introduced
option '--bpf-aux-pause'.

This series has been tested on TC platform with ETE / TRBE with
commands:

  perf record -e cs_etm/aux-action=start-paused/ \
  --bpf-aux-pause="kretprobe:__arm64_sys_openat:p,kprobe:__arm64_sys_openat:r,tp:sched:sched_switch:r" -a -- ls

  perf record -e cs_etm/aux-action=start-paused/ \
  --bpf-aux-pause="kretprobe:__arm64_sys_openat:p,kprobe:__arm64_sys_openat:r,tp:sched:sched_switch:r" -i -- ls

Note, as the AUX pause operation cannot be inherited by child tasks, it
requires to specify the '-i' option for default trace mode and
per-thread mode.


Leo Yan (7):
  perf/core: Make perf_event_aux_pause() as external function
  bpf: Add bpf_perf_event_aux_pause kfunc
  bpf: Sync bpf_perf_event_aux_pause in tools UAPI bpf.h
  perf: auxtrace: Introduce eBPF program for AUX pause
  perf: auxtrace: Support BPF backend for AUX pause
  perf record: Support AUX pause with BPF
  perf docs: Document AUX pause with BPF

 include/linux/perf_event.h                    |   1 +
 include/uapi/linux/bpf.h                      |  21 +
 kernel/bpf/verifier.c                         |   2 +
 kernel/events/core.c                          |   2 +-
 kernel/trace/bpf_trace.c                      |  52 +++
 tools/include/uapi/linux/bpf.h                |  21 +
 tools/perf/Documentation/perf-record.txt      |  40 ++
 tools/perf/Makefile.perf                      |   1 +
 tools/perf/builtin-record.c                   |  18 +-
 tools/perf/util/Build                         |   4 +
 tools/perf/util/auxtrace.h                    |  43 ++
 tools/perf/util/bpf_auxtrace_pause.c          | 385 ++++++++++++++++++
 tools/perf/util/bpf_skel/auxtrace_pause.bpf.c | 135 ++++++
 tools/perf/util/evsel.c                       |   6 +
 tools/perf/util/record.h                      |   1 +
 15 files changed, 730 insertions(+), 2 deletions(-)
 create mode 100644 tools/perf/util/bpf_auxtrace_pause.c
 create mode 100644 tools/perf/util/bpf_skel/auxtrace_pause.bpf.c

-- 
2.34.1

[PATCH v1 1/7] perf/core: Make perf_event_aux_pause() as external function

[Leo Yan]

Expose perf_event_aux_pause() as an external function, this will be used
by eBPF APIs in sequential changes.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 include/linux/perf_event.h | 1 +
 kernel/events/core.c       | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index cb99ec8c9e96..890b7ba4a729 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1162,6 +1162,7 @@ int perf_event_read_local(struct perf_event *event, u64 *value,
 			  u64 *enabled, u64 *running);
 extern u64 perf_event_read_value(struct perf_event *event,
 				 u64 *enabled, u64 *running);
+extern void perf_event_aux_pause(struct perf_event *event, bool pause);
 
 extern struct perf_callchain_entry *perf_callchain(struct perf_event *event, struct pt_regs *regs);
 
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 065f9188b44a..8deb356a915e 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8032,7 +8032,7 @@ static void __perf_event_aux_pause(struct perf_event *event, bool pause)
 	}
 }
 
-static void perf_event_aux_pause(struct perf_event *event, bool pause)
+void perf_event_aux_pause(struct perf_event *event, bool pause)
 {
 	struct perf_buffer *rb;
 
-- 
2.34.1

[PATCH v1 2/7] bpf: Add bpf_perf_event_aux_pause kfunc

[Leo Yan]

The bpf_perf_event_aux_pause kfunc will be used to control the Perf AUX
area to pause or resume.

An example use-case is attaching eBPF to Ftrace tracepoints.  When a
tracepoint is hit, the associated eBPF program will be executed.  The
eBPF program can invoke bpf_perf_event_aux_pause() to pause or resume
AUX trace.  This is useful for fine-grained tracing by combining
Perf and eBPF.

This commit implements the bpf_perf_event_aux_pause kfunc, and make it
pass the eBPF verifier.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 include/uapi/linux/bpf.h | 21 ++++++++++++++++
 kernel/bpf/verifier.c    |  2 ++
 kernel/trace/bpf_trace.c | 52 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 75 insertions(+)

diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
index 4162afc6b5d0..678278c91ce2 100644
--- a/include/uapi/linux/bpf.h
+++ b/include/uapi/linux/bpf.h
@@ -5795,6 +5795,26 @@ union bpf_attr {
  *		0 on success.
  *
  *		**-ENOENT** if the bpf_local_storage cannot be found.
+ *
+ * long bpf_perf_event_aux_pause(struct bpf_map *map, u64 flags, u32 pause)
+ *	Description
+ *		Pause or resume an AUX area trace associated to the perf event.
+ *
+ *		The *flags* argument is specified as the key value for
+ *		retrieving event pointer from the passed *map*.
+ *
+ *		The *pause* argument controls AUX trace pause or resume.
+ *		Non-zero values (true) are to pause the AUX trace and the zero
+ *		value (false) is for re-enabling the AUX trace.
+ *	Return
+ *		0 on success.
+ *
+ *		**-ENOENT** if not found event in the events map.
+ *
+ *		**-E2BIG** if the event index passed in the *flags* parameter
+ *		is out-of-range of the map.
+ *
+ *		**-EINVAL** if the flags passed is an invalid value.
  */
 #define ___BPF_FUNC_MAPPER(FN, ctx...)			\
 	FN(unspec, 0, ##ctx)				\
@@ -6009,6 +6029,7 @@ union bpf_attr {
 	FN(user_ringbuf_drain, 209, ##ctx)		\
 	FN(cgrp_storage_get, 210, ##ctx)		\
 	FN(cgrp_storage_delete, 211, ##ctx)		\
+	FN(perf_event_aux_pause, 212, ##ctx)		\
 	/* */
 
 /* backwards-compatibility macros for users of __BPF_FUNC_MAPPER that don't
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 09f7fa635f67..1f3acd8a7de3 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -9315,6 +9315,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
 		    func_id != BPF_FUNC_perf_event_output &&
 		    func_id != BPF_FUNC_skb_output &&
 		    func_id != BPF_FUNC_perf_event_read_value &&
+		    func_id != BPF_FUNC_perf_event_aux_pause &&
 		    func_id != BPF_FUNC_xdp_output)
 			goto error;
 		break;
@@ -9443,6 +9444,7 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env,
 	case BPF_FUNC_perf_event_read:
 	case BPF_FUNC_perf_event_output:
 	case BPF_FUNC_perf_event_read_value:
+	case BPF_FUNC_perf_event_aux_pause:
 	case BPF_FUNC_skb_output:
 	case BPF_FUNC_xdp_output:
 		if (map->map_type != BPF_MAP_TYPE_PERF_EVENT_ARRAY)
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 949a3870946c..a3b857f6cab4 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -617,6 +617,56 @@ static const struct bpf_func_proto bpf_perf_event_read_value_proto = {
 	.arg4_type	= ARG_CONST_SIZE,
 };
 
+BPF_CALL_3(bpf_perf_event_aux_pause, struct bpf_map *, map, u64, flags,
+	   u32, pause)
+{
+	unsigned long irq_flags;
+	struct bpf_array *array = container_of(map, struct bpf_array, map);
+	unsigned int cpu = smp_processor_id();
+	u64 index = flags & BPF_F_INDEX_MASK;
+	struct bpf_event_entry *ee;
+	int ret = 0;
+
+	/*
+	 * Disabling interrupts avoids scheduling and race condition with
+	 * perf event enabling and disabling flow.
+	 */
+	local_irq_save(irq_flags);
+
+	if (unlikely(flags & ~(BPF_F_INDEX_MASK))) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	if (index == BPF_F_CURRENT_CPU)
+		index = cpu;
+	if (unlikely(index >= array->map.max_entries)) {
+		ret = -E2BIG;
+		goto out;
+	}
+
+	ee = READ_ONCE(array->ptrs[index]);
+	if (!ee) {
+		ret = -ENOENT;
+		goto out;
+	}
+
+	perf_event_aux_pause(ee->event, pause);
+
+out:
+	local_irq_restore(irq_flags);
+	return ret;
+}
+
+static const struct bpf_func_proto bpf_perf_event_aux_pause_proto = {
+	.func		= bpf_perf_event_aux_pause,
+	.gpl_only	= true,
+	.ret_type	= RET_INTEGER,
+	.arg1_type	= ARG_CONST_MAP_PTR,
+	.arg2_type	= ARG_ANYTHING,
+	.arg3_type	= ARG_ANYTHING,
+};
+
 static __always_inline u64
 __bpf_perf_event_output(struct pt_regs *regs, struct bpf_map *map,
 			u64 flags, struct perf_sample_data *sd)
@@ -1565,6 +1615,8 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
 		return &bpf_find_vma_proto;
 	case BPF_FUNC_trace_vprintk:
 		return bpf_get_trace_vprintk_proto();
+	case BPF_FUNC_perf_event_aux_pause:
+		return &bpf_perf_event_aux_pause_proto;
 	default:
 		return bpf_base_func_proto(func_id, prog);
 	}
-- 
2.34.1

Re: [PATCH v1 2/7] bpf: Add bpf_perf_event_aux_pause kfunc

[Yonghong Song]

On 12/15/24 11:34 AM, Leo Yan wrote:
> The bpf_perf_event_aux_pause kfunc will be used to control the Perf AUX
> area to pause or resume.
>
> An example use-case is attaching eBPF to Ftrace tracepoints.  When a
> tracepoint is hit, the associated eBPF program will be executed.  The
> eBPF program can invoke bpf_perf_event_aux_pause() to pause or resume
> AUX trace.  This is useful for fine-grained tracing by combining
> Perf and eBPF.
>
> This commit implements the bpf_perf_event_aux_pause kfunc, and make it
> pass the eBPF verifier.

The subject and commit message mentions to implement a kfunc,
but actually you implemented a uapi helper. Please implement a kfunc
instead (searching __bpf_kfunc in kernel/bpf directory).

>
> Signed-off-by: Leo Yan <leo.yan@arm.com>
> ---
>   include/uapi/linux/bpf.h | 21 ++++++++++++++++
>   kernel/bpf/verifier.c    |  2 ++
>   kernel/trace/bpf_trace.c | 52 ++++++++++++++++++++++++++++++++++++++++
>   3 files changed, 75 insertions(+)
>
> diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h
> index 4162afc6b5d0..678278c91ce2 100644
> --- a/include/uapi/linux/bpf.h
> +++ b/include/uapi/linux/bpf.h
> @@ -5795,6 +5795,26 @@ union bpf_attr {
>    *		0 on success.
>    *
>    *		**-ENOENT** if the bpf_local_storage cannot be found.
> + *
> + * long bpf_perf_event_aux_pause(struct bpf_map *map, u64 flags, u32 pause)
> + *	Description
> + *		Pause or resume an AUX area trace associated to the perf event.
> + *
> + *		The *flags* argument is specified as the key value for
> + *		retrieving event pointer from the passed *map*.
> + *
> + *		The *pause* argument controls AUX trace pause or resume.
> + *		Non-zero values (true) are to pause the AUX trace and the zero
> + *		value (false) is for re-enabling the AUX trace.
> + *	Return
> + *		0 on success.
> + *
> + *		**-ENOENT** if not found event in the events map.
> + *
> + *		**-E2BIG** if the event index passed in the *flags* parameter
> + *		is out-of-range of the map.
> + *
> + *		**-EINVAL** if the flags passed is an invalid value.
>    */
>   #define ___BPF_FUNC_MAPPER(FN, ctx...)			\
>   	FN(unspec, 0, ##ctx)				\
> @@ -6009,6 +6029,7 @@ union bpf_attr {
>   	FN(user_ringbuf_drain, 209, ##ctx)		\
>   	FN(cgrp_storage_get, 210, ##ctx)		\
>   	FN(cgrp_storage_delete, 211, ##ctx)		\
> +	FN(perf_event_aux_pause, 212, ##ctx)		\
>   	/* */
>   
>   /* backwards-compatibility macros for users of __BPF_FUNC_MAPPER that don't
>
[...]

Re: [PATCH v1 2/7] bpf: Add bpf_perf_event_aux_pause kfunc

[Leo Yan]

Hi Yonghong,

Really sorry for the long delay. Now I am restarting this work.

On Mon, Dec 16, 2024 at 09:21:15AM -0800, Yonghong Song wrote:
> On 12/15/24 11:34 AM, Leo Yan wrote:
> > The bpf_perf_event_aux_pause kfunc will be used to control the Perf AUX
> > area to pause or resume.
> > 
> > An example use-case is attaching eBPF to Ftrace tracepoints.  When a
> > tracepoint is hit, the associated eBPF program will be executed.  The
> > eBPF program can invoke bpf_perf_event_aux_pause() to pause or resume
> > AUX trace.  This is useful for fine-grained tracing by combining
> > Perf and eBPF.
> > 
> > This commit implements the bpf_perf_event_aux_pause kfunc, and make it
> > pass the eBPF verifier.
> 
> The subject and commit message mentions to implement a kfunc,
> but actually you implemented a uapi helper. Please implement a kfunc
> instead (searching __bpf_kfunc in kernel/bpf directory).

After some research, my understanding is that kfunc is flexible for
exposing APIs via BTF, whereas BPF_CALL is typically used for core BPF
features - such as accessing BPF maps.

Coming back to this patch: it exposes a function with the following
definition:

  int bpf_perf_event_aux_pause(struct bpf_map *map, u64 flags, u32 pause);

I'm not certain whether using __bpf_kfunc is appropriate here, or if I
should stick to BPF_CALL to ensure support for accessing bpf_map
pointers?

Thanks,
Leo

Re: [PATCH v1 2/7] bpf: Add bpf_perf_event_aux_pause kfunc

[Yonghong Song]

On 7/14/25 10:45 AM, Leo Yan wrote:
> Hi Yonghong,
>
> Really sorry for the long delay. Now I am restarting this work.
>
> On Mon, Dec 16, 2024 at 09:21:15AM -0800, Yonghong Song wrote:
>> On 12/15/24 11:34 AM, Leo Yan wrote:
>>> The bpf_perf_event_aux_pause kfunc will be used to control the Perf AUX
>>> area to pause or resume.
>>>
>>> An example use-case is attaching eBPF to Ftrace tracepoints.  When a
>>> tracepoint is hit, the associated eBPF program will be executed.  The
>>> eBPF program can invoke bpf_perf_event_aux_pause() to pause or resume
>>> AUX trace.  This is useful for fine-grained tracing by combining
>>> Perf and eBPF.
>>>
>>> This commit implements the bpf_perf_event_aux_pause kfunc, and make it
>>> pass the eBPF verifier.
>> The subject and commit message mentions to implement a kfunc,
>> but actually you implemented a uapi helper. Please implement a kfunc
>> instead (searching __bpf_kfunc in kernel/bpf directory).
> After some research, my understanding is that kfunc is flexible for
> exposing APIs via BTF, whereas BPF_CALL is typically used for core BPF
> features - such as accessing BPF maps.
>
> Coming back to this patch: it exposes a function with the following
> definition:
>
>    int bpf_perf_event_aux_pause(struct bpf_map *map, u64 flags, u32 pause);
>
> I'm not certain whether using __bpf_kfunc is appropriate here, or if I
> should stick to BPF_CALL to ensure support for accessing bpf_map
> pointers?

Using helpers (BPF_CALL) is not an option as the whole bpf ecosystem
moves to kfunc mechanism. You can certainly use kfunc with 'struct bpf_map *'
as the argument. For example the following kfunc:
   __bpf_kfunc s64 bpf_map_sum_elem_count(const struct bpf_map *map)
in kernel/bpf/map_iter.c
   

>
> Thanks,
> Leo

Re: [PATCH v1 2/7] bpf: Add bpf_perf_event_aux_pause kfunc

[Leo Yan]

Hi Yonghong,

On Tue, Jul 15, 2025 at 10:12:02AM -0700, Yonghong Song wrote:

[...]

> > I'm not certain whether using __bpf_kfunc is appropriate here, or if I
> > should stick to BPF_CALL to ensure support for accessing bpf_map
> > pointers?
> 
> Using helpers (BPF_CALL) is not an option as the whole bpf ecosystem
> moves to kfunc mechanism. You can certainly use kfunc with 'struct bpf_map *'
> as the argument. For example the following kfunc:
>   __bpf_kfunc s64 bpf_map_sum_elem_count(const struct bpf_map *map)
> in kernel/bpf/map_iter.c

Thanks a lot for suggestion. I followed the idea to refactor the patch
with kfunc, see the new version:
https://lore.kernel.org/linux-perf-users/20250718-perf_aux_pause_resume_bpf_rebase-v2-0-992557b8fb16@arm.com/T/#m27a72255c93fa672e164cb87a322b979fe8f9408

Just clarify one thing, I defined the kfunc in new patch:

  int bpf_perf_event_aux_pause(void *p__map, u64 flags, u32 pause)

Unlike your suggestion, I defined the first parameter as "void
*p__map" (I refers to bpf_arena_alloc_pages()) rather than
"struct bpf_map *map". This is because the BPF program will pass a
variable from the map section, rather than passing a map pointer.

TBH, I do not watch closely the BPF mailing list, so I may not be
fully following the conventions. If anything is incorrect, please
correct it as needed.

Thank you,

Leo

Re: [PATCH v1 2/7] bpf: Add bpf_perf_event_aux_pause kfunc

[Eduard Zingerman]

On Fri, 2025-07-18 at 16:38 +0100, Leo Yan wrote:

[...]

> Just clarify one thing, I defined the kfunc in new patch:
> 
>   int bpf_perf_event_aux_pause(void *p__map, u64 flags, u32 pause)
> 
> Unlike your suggestion, I defined the first parameter as "void
> *p__map" (I refers to bpf_arena_alloc_pages()) rather than
> "struct bpf_map *map". This is because the BPF program will pass a
> variable from the map section, rather than passing a map pointer.

This is correct,
see commit 8d94f1357c00 ("bpf: Recognize '__map' suffix in kfunc arguments")

[...]

[PATCH v1 3/7] bpf: Sync bpf_perf_event_aux_pause in tools UAPI bpf.h

[Leo Yan]

As the new API bpf_perf_event_aux_pause has been added in the kernel
UAPI bpf.h.  Sync with tools UAPI bpf.h.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/include/uapi/linux/bpf.h | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 4162afc6b5d0..678278c91ce2 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -5795,6 +5795,26 @@ union bpf_attr {
  *		0 on success.
  *
  *		**-ENOENT** if the bpf_local_storage cannot be found.
+ *
+ * long bpf_perf_event_aux_pause(struct bpf_map *map, u64 flags, u32 pause)
+ *	Description
+ *		Pause or resume an AUX area trace associated to the perf event.
+ *
+ *		The *flags* argument is specified as the key value for
+ *		retrieving event pointer from the passed *map*.
+ *
+ *		The *pause* argument controls AUX trace pause or resume.
+ *		Non-zero values (true) are to pause the AUX trace and the zero
+ *		value (false) is for re-enabling the AUX trace.
+ *	Return
+ *		0 on success.
+ *
+ *		**-ENOENT** if not found event in the events map.
+ *
+ *		**-E2BIG** if the event index passed in the *flags* parameter
+ *		is out-of-range of the map.
+ *
+ *		**-EINVAL** if the flags passed is an invalid value.
  */
 #define ___BPF_FUNC_MAPPER(FN, ctx...)			\
 	FN(unspec, 0, ##ctx)				\
@@ -6009,6 +6029,7 @@ union bpf_attr {
 	FN(user_ringbuf_drain, 209, ##ctx)		\
 	FN(cgrp_storage_get, 210, ##ctx)		\
 	FN(cgrp_storage_delete, 211, ##ctx)		\
+	FN(perf_event_aux_pause, 212, ##ctx)		\
 	/* */
 
 /* backwards-compatibility macros for users of __BPF_FUNC_MAPPER that don't
-- 
2.34.1

[PATCH v1 4/7] perf: auxtrace: Introduce eBPF program for AUX pause

[Leo Yan]

This commit introduces eBPF program as backend to trigger AUX pause and
resume.

The eBPF programs are prepared for attaching kprobe, kretprobe and
tracepoints.  When a eBPF program is invoked, it calls the eBPF API
bpf_perf_event_aux_pause() for controlling AUX pause and resume.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/perf/Makefile.perf                      |   1 +
 tools/perf/util/bpf_skel/auxtrace_pause.bpf.c | 135 ++++++++++++++++++
 2 files changed, 136 insertions(+)
 create mode 100644 tools/perf/util/bpf_skel/auxtrace_pause.bpf.c

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index d74241a15131..14ac29094eb5 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -1173,6 +1173,7 @@ SKELETONS += $(SKEL_OUT)/kwork_trace.skel.h $(SKEL_OUT)/sample_filter.skel.h
 SKELETONS += $(SKEL_OUT)/kwork_top.skel.h
 SKELETONS += $(SKEL_OUT)/bench_uprobe.skel.h
 SKELETONS += $(SKEL_OUT)/augmented_raw_syscalls.skel.h
+SKELETONS += $(SKEL_OUT)/auxtrace_pause.skel.h
 
 $(SKEL_TMP_OUT) $(LIBAPI_OUTPUT) $(LIBBPF_OUTPUT) $(LIBPERF_OUTPUT) $(LIBSUBCMD_OUTPUT) $(LIBSYMBOL_OUTPUT):
 	$(Q)$(MKDIR) -p $@
diff --git a/tools/perf/util/bpf_skel/auxtrace_pause.bpf.c b/tools/perf/util/bpf_skel/auxtrace_pause.bpf.c
new file mode 100644
index 000000000000..02c211e30e37
--- /dev/null
+++ b/tools/perf/util/bpf_skel/auxtrace_pause.bpf.c
@@ -0,0 +1,135 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+// Copyright 2024 Arm Limited
+#include "vmlinux.h"
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u8));
+	__uint(max_entries, 1);
+} cpu_filter SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_HASH);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(__u8));
+	__uint(max_entries, 1);
+} task_filter SEC(".maps");
+
+struct {
+	__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
+	__uint(key_size, sizeof(__u32));
+	__uint(value_size, sizeof(int));
+	__uint(max_entries, 1);
+} events SEC(".maps");
+
+int enabled = 0;
+
+const volatile int has_cpu = 0;
+const volatile int has_task = 0;
+
+static int event_aux_pause(void)
+{
+	__u64 key;
+
+	if (!enabled)
+		return 0;
+
+	if (has_cpu) {
+		__u32 cpu = bpf_get_smp_processor_id();
+		__u8 *ok;
+
+		ok = bpf_map_lookup_elem(&cpu_filter, &cpu);
+		if (!ok)
+			return 0;
+
+		key = cpu;
+	}
+
+	if (has_task) {
+		__u32 pid = bpf_get_current_pid_tgid() & 0xffffffff;
+		__u8 *ok;
+
+		ok = bpf_map_lookup_elem(&task_filter, &pid);
+		if (!ok)
+			return 0;
+
+		key = 0;
+	}
+
+	bpf_perf_event_aux_pause(&events, key, 1);
+	return 0;
+}
+
+static int event_aux_resume(void)
+{
+	__u64 key;
+
+	if (!enabled)
+		return 0;
+
+	if (has_cpu) {
+		__u32 cpu = bpf_get_smp_processor_id();
+		__u8 *ok;
+
+		ok = bpf_map_lookup_elem(&cpu_filter, &cpu);
+		if (!ok)
+			return 0;
+
+		key = cpu;
+	}
+
+	if (has_task) {
+		__u32 pid = bpf_get_current_pid_tgid() & 0xffffffff;
+		__u8 *ok;
+
+		ok = bpf_map_lookup_elem(&task_filter, &pid);
+		if (!ok)
+			return 0;
+
+		key = 0;
+	}
+
+	bpf_perf_event_aux_pause(&events, key, 0);
+	return 0;
+}
+
+SEC("kprobe/func_pause")
+int BPF_PROG(kprobe_event_pause)
+{
+	return event_aux_pause();
+}
+
+SEC("kretprobe/func_pause")
+int BPF_PROG(kretprobe_event_pause)
+{
+	return event_aux_pause();
+}
+
+SEC("tp/func_pause")
+int BPF_PROG(tp_event_pause)
+{
+	return event_aux_pause();
+}
+
+SEC("kprobe/func_resume")
+int BPF_PROG(kprobe_event_resume)
+{
+	return event_aux_resume();
+}
+
+SEC("kretprobe/func_resume")
+int BPF_PROG(kretprobe_event_resume)
+{
+	return event_aux_resume();
+}
+
+SEC("tp/func_resume")
+int BPF_PROG(tp_event_resume)
+{
+	return event_aux_resume();
+}
+
+char LICENSE[] SEC("license") = "Dual BSD/GPL";
-- 
2.34.1

[PATCH v1 5/7] perf: auxtrace: Support BPF backend for AUX pause

[Leo Yan]

This commit support BPF backend for AUX pause.

It uses a list to maintain the trigger points for AUX pause and resume.
The trigger points (e.g. kprobe, tracepoints) will be attached to BPF
programs based on the trigger type.

It also provides an API auxtrace__update_bpf_map() for updating perf AUX
event into the BPF map.  The information will be used afterwards by BPF
program in kernel to fetch the event handler for controlling AUX trace.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/perf/util/Build                |   4 +
 tools/perf/util/auxtrace.h           |  43 +++
 tools/perf/util/bpf_auxtrace_pause.c | 385 +++++++++++++++++++++++++++
 3 files changed, 432 insertions(+)
 create mode 100644 tools/perf/util/bpf_auxtrace_pause.c

diff --git a/tools/perf/util/Build b/tools/perf/util/Build
index c06d2ee9024c..adb6d8b11e4c 100644
--- a/tools/perf/util/Build
+++ b/tools/perf/util/Build
@@ -178,6 +178,10 @@ ifeq ($(CONFIG_LIBTRACEEVENT),y)
   perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_kwork_top.o
 endif
 
+ifeq ($(CONFIG_AUXTRACE),y)
+  perf-util-$(CONFIG_PERF_BPF_SKEL) += bpf_auxtrace_pause.o
+endif
+
 perf-util-$(CONFIG_LIBELF) += symbol-elf.o
 perf-util-$(CONFIG_LIBELF) += probe-file.o
 perf-util-$(CONFIG_LIBELF) += probe-event.o
diff --git a/tools/perf/util/auxtrace.h b/tools/perf/util/auxtrace.h
index b0db84d27b25..52831e501dea 100644
--- a/tools/perf/util/auxtrace.h
+++ b/tools/perf/util/auxtrace.h
@@ -907,4 +907,47 @@ void itrace_synth_opts__clear_time_range(struct itrace_synth_opts *opts
 
 #endif
 
+#if defined(HAVE_AUXTRACE_SUPPORT) && defined(HAVE_BPF_SKEL)
+
+int auxtrace__prepare_bpf(struct auxtrace_record *itr, const char *str);
+int auxtrace__set_bpf_filter(struct evlist *evlist, struct record_opts *opts);
+int auxtrace__enable_bpf(void);
+int auxtrace__cleanup_bpf(void);
+int auxtrace__update_bpf_map(struct evsel *evsel, int cpu_map_idx, int fd);
+
+#else	/* HAVE_AUXTRACE_SUPPORT && HAVE_BPF_SKEL */
+
+static inline int auxtrace__prepare_bpf(struct auxtrace_record *itr
+					__maybe_unused,
+					const char *str __maybe_unused)
+{
+	return -EINVAL;
+}
+
+static inline int auxtrace__set_bpf_filter(struct evlist *evlist __maybe_unused,
+					   struct record_opts *opts
+					   __maybe_unused)
+{
+	return -EINVAL;
+}
+
+static inline int auxtrace__enable_bpf(void)
+{
+	return -EINVAL;
+}
+
+static inline int auxtrace__cleanup_bpf(void)
+{
+	return -EINVAL;
+}
+
+static int auxtrace__update_bpf_map(struct evsel *evsel __maybe_unused,
+				    int cpu_map_idx __maybe_unused,
+				    int fd __maybe_unused)
+{
+	return -EINVAL;
+}
+
+#endif
+
 #endif
diff --git a/tools/perf/util/bpf_auxtrace_pause.c b/tools/perf/util/bpf_auxtrace_pause.c
new file mode 100644
index 000000000000..1012e0d62dbc
--- /dev/null
+++ b/tools/perf/util/bpf_auxtrace_pause.c
@@ -0,0 +1,385 @@
+// SPDX-License-Identifier: GPL-2.0
+
+/* Copyright 2024 Arm Limited */
+
+#include <stdio.h>
+#include <fcntl.h>
+#include <stdint.h>
+#include <stdlib.h>
+
+#include <linux/err.h>
+
+#include "util/auxtrace.h"
+#include "util/cpumap.h"
+#include "util/thread_map.h"
+#include "util/debug.h"
+#include "util/evlist.h"
+#include "util/bpf_counter.h"
+#include "util/record.h"
+#include "util/target.h"
+
+#include "util/bpf_skel/auxtrace_pause.skel.h"
+
+/* The valid controlling type is "p" (pause) and "r" (resume) */
+#define is_attach_kprobe(str)		\
+	(!strcmp((str), "kprobe") || !strcmp((str), "kretprobe"))
+#define is_attach_tracepoint(str)	\
+	(!strcmp((str), "tp") || !strcmp((str), "tracepoint"))
+
+/* The valid controlling type is "p" (pause) and "r" (resume) */
+#define is_valid_ctrl_type(str)	\
+	(!strcmp((str), "p") || !strcmp((str), "r"))
+
+static struct auxtrace_pause_bpf *skel;
+
+struct trigger_entry {
+	struct list_head list;
+	char *arg0;
+	char *arg1;
+	char *arg2;
+	char *arg3;
+};
+
+static int trigger_entry_num;
+static LIST_HEAD(trigger_list);
+static struct bpf_link **trigger_links;
+
+static void auxtrace__free_bpf_trigger_list(void)
+{
+	struct trigger_entry *entry, *next;
+
+	list_for_each_entry_safe(entry, next, &trigger_list, list) {
+		free(entry->arg0);
+		free(entry->arg1);
+		free(entry->arg2);
+		free(entry->arg3);
+		free(entry);
+	}
+
+	trigger_entry_num = 0;
+}
+
+static int auxtrace__alloc_bpf_trigger_list(const char *str)
+{
+	char *cmd_str;
+	char *substr, *saveptr1;
+	struct trigger_entry *entry;
+	int ret = 0;
+
+	if (!str)
+		return -EINVAL;
+
+	cmd_str = strdup(str);
+	if (!cmd_str)
+		return -ENOMEM;
+
+	substr = strtok_r(cmd_str, ",", &saveptr1);
+	for ( ; substr != NULL; substr = strtok_r(NULL, ",", &saveptr1)) {
+		char *fmt1_str, *fmt2_str, *fmt3_str, *fmt4_str, *fmt;
+
+		entry = zalloc(sizeof(*entry));
+		if (!entry) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		/*
+		 * A trigger is expressed with several fields with separator ":".
+		 * The first field is specified for attach types, it can be one
+		 * of types listed below:
+		 *   kprobe
+		 *   kretprobe
+		 *   tp
+		 *   tracepoint
+		 *
+		 * Kprobe and kretprobe use the second field to express the
+		 * probe point. So far, it only support a simple expression with
+		 * kernel function name.  Kprobe/kretprobe trigger format is:
+		 *   {kprobe|kretprobe}:function_name:{p|r}
+		 *
+		 * Tracepoint use the second field for the trace category and
+		 * the third field for the tracepoint name.  Tracepoint trigger
+		 * format is:
+		 *   {tp|tracepoint}:category:tracepint_name:{p|r}
+		 *
+		 * The last field is used to express the controlling type: "p"
+		 * means aux pause and "r" is for aux resume.
+		 */
+		fmt1_str = strtok_r(substr, ":", &fmt);
+		fmt2_str = strtok_r(NULL, ":", &fmt);
+		fmt3_str = strtok_r(NULL, ":", &fmt);
+		if (!fmt1_str || !fmt2_str || !fmt3_str) {
+			pr_err("Failed to parse bpf aux pause string: %s\n",
+				substr);
+			ret = -EINVAL;
+			goto out;
+		}
+
+		entry->arg0 = strdup(fmt1_str);
+		entry->arg1 = strdup(fmt2_str);
+		entry->arg2 = strdup(fmt3_str);
+		if (!entry->arg0 || !entry->arg1 || !entry->arg2) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		if (!is_attach_kprobe(entry->arg0) &&
+		    !is_attach_tracepoint(entry->arg0)) {
+			pr_err("Failed to support bpf aux pause attach: %s\n",
+			       entry->arg0);
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (is_attach_kprobe(entry->arg0)) {
+			if (!is_valid_ctrl_type(entry->arg2)) {
+				pr_err("Failed to support bpf aux pause ctrl: %s\n",
+				       entry->arg2);
+				ret = -EINVAL;
+				goto out;
+			}
+		} else {
+			fmt4_str = strtok_r(NULL, ":", &fmt);
+			if (!fmt4_str) {
+				ret = -ENOMEM;
+				goto out;
+			}
+
+			entry->arg3 = strdup(fmt4_str);
+			if (!entry->arg3) {
+				ret = -ENOMEM;
+				goto out;
+			}
+
+			if (!is_valid_ctrl_type(entry->arg3)) {
+				pr_err("Failed to support bpf aux pause ctrl: %s\n",
+				       entry->arg3);
+				ret = -EINVAL;
+				goto out;
+			}
+		}
+
+		if (ret)
+			goto out;
+
+		list_add(&entry->list, &trigger_list);
+		trigger_entry_num++;
+	}
+
+	free(cmd_str);
+	return 0;
+
+out:
+	free(cmd_str);
+	if (entry) {
+		free(entry->arg0);
+		free(entry->arg1);
+		free(entry->arg2);
+		free(entry->arg3);
+		free(entry);
+	}
+	auxtrace__free_bpf_trigger_list();
+	return ret;
+}
+
+int auxtrace__prepare_bpf(struct auxtrace_record *itr, const char *str)
+{
+	int ret;
+
+	if (!itr || !str)
+		return 0;
+
+	skel = auxtrace_pause_bpf__open();
+	if (!skel) {
+		pr_err("Failed to open func latency skeleton\n");
+		return -1;
+	}
+
+	ret = auxtrace__alloc_bpf_trigger_list(str);
+	if (ret) {
+		auxtrace_pause_bpf__destroy(skel);
+		skel = NULL;
+		return ret;
+	}
+
+	return 0;
+}
+
+static struct bpf_link *auxtrace__attach_bpf_prog(struct trigger_entry *entry)
+{
+	struct bpf_link *link = NULL;
+
+	if (!strcmp(entry->arg0, "kprobe")) {
+		if (!strcmp(entry->arg2, "p")) {
+			link = bpf_program__attach_kprobe(
+					skel->progs.kprobe_event_pause,
+					false, entry->arg1);
+		} else if (!strcmp(entry->arg2, "r")) {
+			link = bpf_program__attach_kprobe(
+					skel->progs.kprobe_event_resume,
+					false, entry->arg1);
+		}
+	} else if (!strcmp(entry->arg0, "kretprobe")) {
+		if (!strcmp(entry->arg2, "p")) {
+			link = bpf_program__attach_kprobe(
+					skel->progs.kretprobe_event_pause,
+					true, entry->arg1);
+		} else if (!strcmp(entry->arg2, "r")) {
+			link = bpf_program__attach_kprobe(
+					skel->progs.kretprobe_event_resume,
+					true, entry->arg1);
+		}
+	} else if (is_attach_tracepoint(entry->arg0)) {
+		if (!strcmp(entry->arg3, "p")) {
+			link = bpf_program__attach_tracepoint(
+					skel->progs.tp_event_pause,
+					entry->arg1, entry->arg2);
+		} else if (!strcmp(entry->arg3, "r")) {
+			link = bpf_program__attach_tracepoint(
+					skel->progs.tp_event_resume,
+					entry->arg1, entry->arg2);
+		}
+	}
+
+	return link;
+}
+
+int auxtrace__set_bpf_filter(struct evlist *evlist, struct record_opts *opts)
+{
+	int fd, err;
+	int i, ncpus = 1, ntasks = 1;
+	struct trigger_entry *trigger_entry;
+	struct target *target;
+
+	if (!skel)
+		return 0;
+
+	if (!opts)
+		return -EINVAL;
+
+	target = &opts->target;
+
+	if (target__has_cpu(target)) {
+		ncpus = perf_cpu_map__nr(evlist->core.user_requested_cpus);
+		bpf_map__set_max_entries(skel->maps.cpu_filter, ncpus);
+		skel->rodata->has_cpu = 1;
+	}
+
+	if (target__has_task(target) || target__none(target)) {
+		ntasks = perf_thread_map__nr(evlist->core.threads);
+		bpf_map__set_max_entries(skel->maps.task_filter, ntasks);
+		skel->rodata->has_task = 1;
+	}
+
+	bpf_map__set_max_entries(skel->maps.events, libbpf_num_possible_cpus());
+
+	err = auxtrace_pause_bpf__load(skel);
+	if (err) {
+		pr_err("Failed to load func latency skeleton: %d\n", err);
+		goto out;
+	}
+
+	if (target__has_cpu(target)) {
+		u32 cpu;
+		u8 val = 1;
+
+		fd = bpf_map__fd(skel->maps.cpu_filter);
+
+		for (i = 0; i < ncpus; i++) {
+			cpu = perf_cpu_map__cpu(evlist->core.user_requested_cpus, i).cpu;
+			bpf_map_update_elem(fd, &cpu, &val, BPF_ANY);
+		}
+	}
+
+	if (target__has_task(target) || target__none(target)) {
+		u32 pid;
+		u8 val = 1;
+
+		fd = bpf_map__fd(skel->maps.task_filter);
+
+		for (i = 0; i < ntasks; i++) {
+			pid = perf_thread_map__pid(evlist->core.threads, i);
+			bpf_map_update_elem(fd, &pid, &val, BPF_ANY);
+		}
+	}
+
+	trigger_links = zalloc(sizeof(*trigger_links) * trigger_entry_num);
+	if (!trigger_links)
+		return -ENOMEM;
+
+	i = 0;
+	list_for_each_entry(trigger_entry, &trigger_list, list) {
+		trigger_links[i] = auxtrace__attach_bpf_prog(trigger_entry);
+		err = libbpf_get_error(trigger_links[i]);
+		if (err) {
+			pr_err("Failed to attach bpf program to aux pause entry\n");
+			pr_err("  arg0=%s arg1=%s arg2=%s arg3=%s\n",
+			       trigger_entry->arg0, trigger_entry->arg1,
+			       trigger_entry->arg2, trigger_entry->arg3);
+			trigger_links[i] = NULL;
+			goto out;
+		}
+		i++;
+	}
+
+	return 0;
+
+out:
+	for (i = 0; i < trigger_entry_num; i++) {
+		if (!trigger_links[i])
+			continue;
+		bpf_link__destroy(trigger_links[i]);
+	}
+
+	return err;
+}
+
+int auxtrace__enable_bpf(void)
+{
+	if (!skel)
+		return 0;
+
+	skel->bss->enabled = 1;
+	return 0;
+}
+
+int auxtrace__cleanup_bpf(void)
+{
+	int i;
+
+	if (!skel)
+		return 0;
+
+	for (i = 0; i < trigger_entry_num; i++) {
+		if (!trigger_links[i])
+			continue;
+		bpf_link__destroy(trigger_links[i]);
+	}
+
+	auxtrace__free_bpf_trigger_list();
+	auxtrace_pause_bpf__destroy(skel);
+	return 0;
+}
+
+int auxtrace__update_bpf_map(struct evsel *evsel, int cpu_map_idx, int fd)
+{
+	int ret;
+
+	if (!skel)
+		return 0;
+
+	if (!evsel__is_aux_event(evsel))
+		return 0;
+
+	ret = bpf_map_update_elem(bpf_map__fd(skel->maps.events),
+				  &cpu_map_idx, &fd, BPF_ANY);
+	if (ret) {
+		pr_err("Failed to update BPF map for auxtrace: %s.\n",
+			strerror(errno));
+		if (errno == EOPNOTSUPP)
+			pr_err("  Try to disable inherit mode with option '-i'.\n");
+		return ret;
+	}
+
+	return 0;
+}
-- 
2.34.1

[PATCH v1 6/7] perf record: Support AUX pause with BPF

[Leo Yan]

This commit enables the BPF backend in a perf record session for AUX
pause.

It introduces an option "--bpf-aux-pause" for specifying kprobe and
tracepoint events for triggering AUX pause and resume.  The BPF
program will be loaded after the option is set, and the events will be
updated into the BPF map, so the BPF program in kernel knows to
control AUX pause and resume based on the event handler.

After:

  perf record -e cs_etm/aux-action=start-paused/ \
    --bpf-aux-pause="kretprobe:__arm64_sys_openat:p,kprobe:__arm64_sys_openat:r,tp:sched:sched_switch:r" \
    -a -- ls

  perf record -e cs_etm/aux-action=start-paused/ \
    --bpf-aux-pause="kretprobe:__arm64_sys_openat:p,kprobe:__arm64_sys_openat:r,tp:sched:sched_switch:r" \
    --per-thread -i -- ls

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/perf/builtin-record.c | 18 +++++++++++++++++-
 tools/perf/util/evsel.c     |  6 ++++++
 tools/perf/util/record.h    |  1 +
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/tools/perf/builtin-record.c b/tools/perf/builtin-record.c
index 49361c5b2251..ae6bb23e0233 100644
--- a/tools/perf/builtin-record.c
+++ b/tools/perf/builtin-record.c
@@ -864,7 +864,12 @@ static int record__auxtrace_init(struct record *rec)
 	if (err)
 		return err;
 
-	return auxtrace_parse_filters(rec->evlist);
+	err = auxtrace_parse_filters(rec->evlist);
+	if (err)
+		return err;
+
+	return auxtrace__prepare_bpf(rec->itr,
+				     rec->opts.auxtrace_bpf_aux_pause_opts);
 }
 
 #else
@@ -2486,6 +2491,10 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 
 	evlist__config(rec->evlist, opts, &callchain_param);
 
+	err = auxtrace__set_bpf_filter(rec->evlist, opts);
+	if (err)
+		goto out_free_threads;
+
 	/* Debug message used by test scripts */
 	pr_debug3("perf record opening and mmapping events\n");
 	if (record__open(rec) != 0) {
@@ -2556,6 +2565,9 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	if (record__start_threads(rec))
 		goto out_free_threads;
 
+	if (auxtrace__enable_bpf())
+		goto out_free_threads;
+
 	/*
 	 * When perf is starting the traced process, all the events
 	 * (apart from group members) have enable_on_exec=1 set,
@@ -2875,6 +2887,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)
 	}
 
 out_delete_session:
+	auxtrace__cleanup_bpf();
 #ifdef HAVE_EVENTFD_SUPPORT
 	if (done_fd >= 0) {
 		fd = done_fd;
@@ -3566,6 +3579,9 @@ static struct option __record_options[] = {
 	OPT_BOOLEAN(0, "off-cpu", &record.off_cpu, "Enable off-cpu analysis"),
 	OPT_STRING(0, "setup-filter", &record.filter_action, "pin|unpin",
 		   "BPF filter action"),
+	OPT_STRING_OPTARG(0, "bpf-aux-pause", &record.opts.auxtrace_bpf_aux_pause_opts,
+			  "{kprobe|kretprobe|tp|tracepoint}:{category}:trace_name:{p|r}",
+			  "Enable AUX pause with BPF backend", ""),
 	OPT_END()
 };
 
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index f6a5284ed5f9..a77053e546bc 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -2586,6 +2586,12 @@ static int evsel__open_cpu(struct evsel *evsel, struct perf_cpu_map *cpus,
 
 			bpf_counter__install_pe(evsel, idx, fd);
 
+			/* Update event info into BPF map for AUX trace */
+			if (auxtrace__update_bpf_map(evsel, idx, fd)) {
+				err = -EINVAL;
+				goto out_close;
+			}
+
 			if (unlikely(test_attr__enabled())) {
 				test_attr__open(&evsel->core.attr, pid, cpu,
 						fd, group_fd, evsel->open_flags);
diff --git a/tools/perf/util/record.h b/tools/perf/util/record.h
index a6566134e09e..051a4b71721b 100644
--- a/tools/perf/util/record.h
+++ b/tools/perf/util/record.h
@@ -64,6 +64,7 @@ struct record_opts {
 	size_t	      auxtrace_snapshot_size;
 	const char    *auxtrace_snapshot_opts;
 	const char    *auxtrace_sample_opts;
+	const char    *auxtrace_bpf_aux_pause_opts;
 	bool	      sample_transaction;
 	bool	      use_clockid;
 	clockid_t     clockid;
-- 
2.34.1

[PATCH v1 7/7] perf docs: Document AUX pause with BPF

[Leo Yan]

Documents the usage of the --bpf-aux-pause option and provides
examples.

Signed-off-by: Leo Yan <leo.yan@arm.com>
---
 tools/perf/Documentation/perf-record.txt | 40 ++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/tools/perf/Documentation/perf-record.txt b/tools/perf/Documentation/perf-record.txt
index 80686d590de2..25cf993b7a50 100644
--- a/tools/perf/Documentation/perf-record.txt
+++ b/tools/perf/Documentation/perf-record.txt
@@ -535,6 +535,46 @@ must be an AUX area event. Samples on other events will be created containing
 data from the AUX area. Optionally sample size may be specified, otherwise it
 defaults to 4KiB.
 
+--bpf-aux-pause=[=OPTIONS]::
+Specify trace events for triggering AUX pause with a BPF program. A trace event
+can be a kprobe, kretprobe or tracepoint. This option must be enabled in
+combination with the "aux-action=start-paused" configuration in an AUX event.
+
+For attaching a kprobe or kretprobe event, the format is:
+
+  {kprobe|kretprobe}:function_name:{p|r}
+
+For attaching a tracepoint, the format is:
+
+  {tp|tracepoint}:category:tracepint_name:{p|r}
+
+The first field is for the trace event type. It supports three types: kprobe,
+kretprobe, and tracepoint ('tp' is also supported as an abbreviation for
+"tracepoint"). The last field specifies whether the action is pause ("p") or
+resume ("r").
+
+For kprobe and kretprobe, the "function_name" field is for specifying a kernel
+function name. In the case of a tracepoint, the "category" and "tracepoint_name"
+fields are used together for providing complete tracepoint info.
+
+The '--bpf-aux-pause' option does not support inherit mode.  In the default
+trace mode or per-thread mode, it needs to be combined with the '-i' or
+'--no-inherit' option to disable inherit mode.
+
+The syntax supports multiple trace events, with each separated by a comma (,).
+For example, users can set up AUX pause on a kernel function with kretprobe and
+AUX resume on a tracepoint with the syntax below:
+
+  For default trace mode (with inherit mode disabled):
+  perf record -e cs_etm/aux-action=start-paused/ \
+  --bpf-aux-pause="kretprobe:__arm64_sys_openat:p,tp:sched:sched_switch:r" \
+  -i ...
+
+  For system wide trace mode:
+  perf record -e cs_etm/aux-action=start-paused/ \
+  --bpf-aux-pause="kretprobe:__arm64_sys_openat:p,tp:sched:sched_switch:r" \
+  -a ...
+
 --proc-map-timeout::
 When processing pre-existing threads /proc/XXX/mmap, it may take a long time,
 because the file may be huge. A time out is needed in such cases.
-- 
2.34.1