[RFC PATCH v2 0/2] libbpf: Add support for dynamic tracepoints

[RFC PATCH v2 0/2] libbpf: Add support for dynamic tracepoints

[Yafang Shao]

The primary goal of this change is to enable tracing of inlined kernel
functions with BPF programs.

Dynamic tracepoints can be created using tools like perf-probe, debugfs, or
similar utilities. For example:

  $ perf probe -a 'tcp_listendrop sk'
  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Here, tcp_listendrop() is an example of an inlined kernel function.

While these dynamic tracepoints are functional, they cannot be easily
attached to BPF programs. For instance, attempting to use them with
bpftrace results in the following error:

  $ bpftrace -l 'tracepoint:probe:*'
  tracepoint:probe:tcp_listendrop

  $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
  Attaching 1 probe...
  ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
  ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop

The issue lies in how these dynamic tracepoints are implemented: despite
being exposed as tracepoints, they remain kprobe events internally. As a
result, loading them as a tracepoint program fails. Instead, they must be
loaded as kprobe programs.

This change introduces support for such use cases in libbpf by adding a
new section: SEC("kprobe/SUBSYSTEM/PROBE")

- Future work
  Extend support for dynamic tracepoints in bpftrace.

Changes:
v1->v2:
- Use a new SEC("kprobe/SUBSYSTEM/PROBE") instead (Jiri)

v1: https://lore.kernel.org/bpf/20250105124403.991-1-laoar.shao@gmail.com/

Yafang Shao (2):
  libbpf: Add support for dynamic tracepoint
  selftests/bpf: Add selftest for dynamic tracepoint

 tools/lib/bpf/libbpf.c                        | 29 ++++++++-
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 3 files changed, 119 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c

-- 
2.43.5

[RFC PATCH v2 1/2] libbpf: Add support for dynamic tracepoint

[Yafang Shao]

Dynamic tracepoints can be created using debugfs, perf or similar tools.
For example:

  $ perf probe -a 'tcp_listendrop sk'

This command creates a new tracepoint under debugfs:

  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

Notably, the probed function tcp_listendrop() is an inlined kernel function.

Although this dynamic tracepoint appears as a tracepoint, it is internally
implemented as a kprobe. Therefore, if we want to attach a bpf prog to
it, the bpf prog must be loaded as a kprobe prog.

The primary motivation for adding support for dynamic tracepoints is to
simplify tracing of inlined kernel functions using BPF tools, such as
bpftrace. By leveraging tools like perf, users can create a dynamic
tracepoint for an inlined kernel function and then attach a BPF program to
it.

To achieve this, a new section, SEC("kprobe/SUBSYSTEM/PROBE"), has been
introduced.

Suggested-by: Jiri Olsa <olsajiri@gmail.com>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Cc: Daniel Xu <dxu@dxuuu.xyz>
---
 tools/lib/bpf/libbpf.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 66173ddb5a2d..23ea9272491b 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -11600,11 +11600,34 @@ bpf_program__attach_kprobe_multi_opts(const struct bpf_program *prog,
 	return libbpf_err_ptr(err);
 }
 
+/* A dynamic tracepoint: "kprobe/SUBSYSTEM/PROBE" */
+static int attach_dynamic_tracepoint(const struct bpf_program *prog, const char *func_name,
+				     struct bpf_link **link)
+{
+	char *tp_subsys, *tp_name;
+
+	tp_subsys = strdup(func_name);
+	if (!tp_subsys)
+		return -ENOMEM;
+
+	tp_name = strchr(tp_subsys, '/');
+	if (!tp_name) {
+		free(tp_subsys);
+		return -EINVAL;
+	}
+
+	*tp_name = '\0';
+	tp_name++;
+	*link = bpf_program__attach_tracepoint(prog, tp_subsys, tp_name);
+	free(tp_subsys);
+	return libbpf_get_error(*link);
+}
+
 static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link)
 {
 	DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts);
+	const char *func_name, *dynamic_tp;
 	unsigned long offset = 0;
-	const char *func_name;
 	char *func;
 	int n;
 
@@ -11620,6 +11643,10 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
 	else
 		func_name = prog->sec_name + sizeof("kprobe/") - 1;
 
+	dynamic_tp = strchr(func_name, '/');
+	if (dynamic_tp)
+		return attach_dynamic_tracepoint(prog, func_name, link);
+
 	n = sscanf(func_name, "%m[a-zA-Z0-9_.]+%li", &func, &offset);
 	if (n < 1) {
 		pr_warn("kprobe name is invalid: %s\n", func_name);
-- 
2.43.5

[RFC PATCH v2 2/2] selftests/bpf: Add selftest for dynamic tracepoint

[Yafang Shao]

The result is as follows,

  $ tools/testing/selftests/bpf/test_progs --name=dynamic_tp
  #85      dynamic_tp:OK
  Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c

diff --git a/tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c b/tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
new file mode 100644
index 000000000000..c205e0c8e3e3
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <test_progs.h>
+#include <bpf/btf.h>
+#include <bpf/bpf.h>
+
+#include "dynamic_tp.skel.h"
+
+int dynamic_tp(const char *cmd)
+{
+	const char *kprobe_file = "/sys/kernel/debug/tracing/kprobe_events";
+	ssize_t bytes_written;
+	int fd, err;
+
+	fd = open(kprobe_file, O_WRONLY | O_APPEND);
+	if (!ASSERT_GE(fd, 0, "open kprobe_events"))
+		return -1;
+
+	bytes_written = write(fd, cmd, strlen(cmd));
+	if (!ASSERT_GT(bytes_written, 0, "write kprobe_events")) {
+		close(fd);
+		return -1;
+	}
+
+	err = close(fd);
+	if (!ASSERT_OK(err, "close kprobe_events"))
+		return -1;
+	return 0;
+}
+
+void test_dynamic_tp(void)
+{
+	struct dynamic_tp *skel;
+	pid_t child_pid;
+	int status, err;
+
+	/* create a dynamic tracepoint */
+	err = dynamic_tp("p:my_dynamic_tp kernel_clone");
+	if (!ASSERT_OK(err, "create dynamic tp"))
+		return;
+
+	skel = dynamic_tp__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "load progs"))
+		goto remove_tp;
+	skel->bss->pid = getpid();
+	err = dynamic_tp__attach(skel);
+	if (!ASSERT_OK(err, "attach progs"))
+		goto cleanup;
+
+	/* trigger the dynamic tracepoint */
+	child_pid = fork();
+	if (!ASSERT_GT(child_pid, -1, "child_pid"))
+		goto cleanup;
+	if (child_pid == 0)
+		_exit(0);
+	waitpid(child_pid, &status, 0);
+
+	ASSERT_EQ(skel->bss->result, 1, "result");
+
+cleanup:
+	dynamic_tp__destroy(skel);
+remove_tp:
+	dynamic_tp("-:my_dynamic_tp kernel_clone");
+}
diff --git a/tools/testing/selftests/bpf/progs/dynamic_tp.c b/tools/testing/selftests/bpf/progs/dynamic_tp.c
new file mode 100644
index 000000000000..d3be37c220f3
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/dynamic_tp.c
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+char _license[] SEC("license") = "GPL";
+
+#define MAX_STACK_TRACE_DEPTH 32
+unsigned long entries[MAX_STACK_TRACE_DEPTH] = {};
+#define SIZE_OF_ULONG (sizeof(unsigned long))
+
+int result, pid;
+
+SEC("kprobe/kprobes/my_dynamic_tp")
+int dynamic_tp(struct pt_regs *ctx)
+{
+	int ret;
+
+	ret = bpf_get_stack(ctx, entries, MAX_STACK_TRACE_DEPTH * SIZE_OF_ULONG, 0);
+	if (ret < 0) {
+		result = -1;
+		return ret;
+	}
+	if (bpf_get_current_pid_tgid() >> 32 == pid)
+		result = 1;
+	return 0;
+}
-- 
2.43.5

Re: [RFC PATCH v2 0/2] libbpf: Add support for dynamic tracepoints

[Andrii Nakryiko]

On Sat, Jan 11, 2025 at 10:45 PM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> The primary goal of this change is to enable tracing of inlined kernel
> functions with BPF programs.
>
> Dynamic tracepoints can be created using tools like perf-probe, debugfs, or
> similar utilities. For example:
>
>   $ perf probe -a 'tcp_listendrop sk'
>   $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
>   enable  filter  format  hist  id  trigger
>
> Here, tcp_listendrop() is an example of an inlined kernel function.
>
> While these dynamic tracepoints are functional, they cannot be easily
> attached to BPF programs. For instance, attempting to use them with
> bpftrace results in the following error:
>
>   $ bpftrace -l 'tracepoint:probe:*'
>   tracepoint:probe:tcp_listendrop
>
>   $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
>   Attaching 1 probe...
>   ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
>   ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop
>
> The issue lies in how these dynamic tracepoints are implemented: despite
> being exposed as tracepoints, they remain kprobe events internally. As a
> result, loading them as a tracepoint program fails. Instead, they must be
> loaded as kprobe programs.
>
> This change introduces support for such use cases in libbpf by adding a
> new section: SEC("kprobe/SUBSYSTEM/PROBE")
>
> - Future work
>   Extend support for dynamic tracepoints in bpftrace.

Seems like the primary motivation is bpftrace support, so let's start
there. I'm hesitant to include this in libbpf right now. The whole
SEC("kprobe") that calls "attach_tracepoint()" underneath makes me
uncomfortable.

You can still attach to such dynamic probe today with pure libbpf
(e.g., if bpftrace needs to do this, for example) by creating
perf_event FD from tracefs' id, and then using
bpf_program__attach_perf_event_opts() to attach to it. It will be on
the user to use either tracepoint or kprobe BPF program for such
attachment.

Yes, this doesn't work "declaratively" with a nice SEC("...") syntax,
but at least it's doable programmatically, and that's what matters for
bpftrace.



>
> Changes:
> v1->v2:
> - Use a new SEC("kprobe/SUBSYSTEM/PROBE") instead (Jiri)
>
> v1: https://lore.kernel.org/bpf/20250105124403.991-1-laoar.shao@gmail.com/
>
> Yafang Shao (2):
>   libbpf: Add support for dynamic tracepoint
>   selftests/bpf: Add selftest for dynamic tracepoint
>
>  tools/lib/bpf/libbpf.c                        | 29 ++++++++-
>  .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
>  .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
>  3 files changed, 119 insertions(+), 1 deletion(-)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
>  create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c
>
> --
> 2.43.5
>

Re: [RFC PATCH v2 0/2] libbpf: Add support for dynamic tracepoints

[Yafang Shao]

On Wed, Jan 15, 2025 at 6:32 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Sat, Jan 11, 2025 at 10:45 PM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > The primary goal of this change is to enable tracing of inlined kernel
> > functions with BPF programs.
> >
> > Dynamic tracepoints can be created using tools like perf-probe, debugfs, or
> > similar utilities. For example:
> >
> >   $ perf probe -a 'tcp_listendrop sk'
> >   $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
> >   enable  filter  format  hist  id  trigger
> >
> > Here, tcp_listendrop() is an example of an inlined kernel function.
> >
> > While these dynamic tracepoints are functional, they cannot be easily
> > attached to BPF programs. For instance, attempting to use them with
> > bpftrace results in the following error:
> >
> >   $ bpftrace -l 'tracepoint:probe:*'
> >   tracepoint:probe:tcp_listendrop
> >
> >   $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
> >   Attaching 1 probe...
> >   ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
> >   ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop
> >
> > The issue lies in how these dynamic tracepoints are implemented: despite
> > being exposed as tracepoints, they remain kprobe events internally. As a
> > result, loading them as a tracepoint program fails. Instead, they must be
> > loaded as kprobe programs.
> >
> > This change introduces support for such use cases in libbpf by adding a
> > new section: SEC("kprobe/SUBSYSTEM/PROBE")
> >
> > - Future work
> >   Extend support for dynamic tracepoints in bpftrace.
>
> Seems like the primary motivation is bpftrace support, so let's start
> there.

I believe we should extend support for this feature in BCC as well.
Since this is a common and widely applicable feature, wouldn't it make
sense to include it directly in libbpf?

> I'm hesitant to include this in libbpf right now. The whole
> SEC("kprobe") that calls "attach_tracepoint()" underneath makes me
> uncomfortable.

I reused the bpf_program__attach_tracepoint() function, but if we
examine its implementation closely, we can see that it actually
creates a DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); rather
than a bpf_tracepoint_opts.

If naming is a significant concern, it wouldn’t be difficult to
reimplement the same logic with a more appropriate name, such as
attach_kprobe_based_tracepoint().

>
> You can still attach to such dynamic probe today with pure libbpf
> (e.g., if bpftrace needs to do this, for example) by creating
> perf_event FD from tracefs' id, and then using
> bpf_program__attach_perf_event_opts() to attach to it. It will be on
> the user to use either tracepoint or kprobe BPF program for such
> attachment.

You're right—we can manually attach it in the tools, but if we can
make it auto-attachable in libbpf, why not take advantage of that and
simplify the process?


>
> Yes, this doesn't work "declaratively" with a nice SEC("...") syntax,
> but at least it's doable programmatically, and that's what matters for
> bpftrace.


--
Regards
Yafang

Re: [RFC PATCH v2 0/2] libbpf: Add support for dynamic tracepoints

[Andrii Nakryiko]

On Tue, Jan 14, 2025 at 7:13 PM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> On Wed, Jan 15, 2025 at 6:32 AM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Sat, Jan 11, 2025 at 10:45 PM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > The primary goal of this change is to enable tracing of inlined kernel
> > > functions with BPF programs.
> > >
> > > Dynamic tracepoints can be created using tools like perf-probe, debugfs, or
> > > similar utilities. For example:
> > >
> > >   $ perf probe -a 'tcp_listendrop sk'
> > >   $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
> > >   enable  filter  format  hist  id  trigger
> > >
> > > Here, tcp_listendrop() is an example of an inlined kernel function.
> > >
> > > While these dynamic tracepoints are functional, they cannot be easily
> > > attached to BPF programs. For instance, attempting to use them with
> > > bpftrace results in the following error:
> > >
> > >   $ bpftrace -l 'tracepoint:probe:*'
> > >   tracepoint:probe:tcp_listendrop
> > >
> > >   $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
> > >   Attaching 1 probe...
> > >   ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
> > >   ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop
> > >
> > > The issue lies in how these dynamic tracepoints are implemented: despite
> > > being exposed as tracepoints, they remain kprobe events internally. As a
> > > result, loading them as a tracepoint program fails. Instead, they must be
> > > loaded as kprobe programs.
> > >
> > > This change introduces support for such use cases in libbpf by adding a
> > > new section: SEC("kprobe/SUBSYSTEM/PROBE")
> > >
> > > - Future work
> > >   Extend support for dynamic tracepoints in bpftrace.
> >
> > Seems like the primary motivation is bpftrace support, so let's start
> > there.
>
> I believe we should extend support for this feature in BCC as well.
> Since this is a common and widely applicable feature, wouldn't it make
> sense to include it directly in libbpf?
>
> > I'm hesitant to include this in libbpf right now. The whole
> > SEC("kprobe") that calls "attach_tracepoint()" underneath makes me
> > uncomfortable.
>
> I reused the bpf_program__attach_tracepoint() function, but if we
> examine its implementation closely, we can see that it actually
> creates a DECLARE_LIBBPF_OPTS(bpf_perf_event_opts, pe_opts); rather
> than a bpf_tracepoint_opts.
>
> If naming is a significant concern, it wouldn’t be difficult to
> reimplement the same logic with a more appropriate name, such as
> attach_kprobe_based_tracepoint().

"kprobe based tracepoint" is exactly the thing that makes me unwilling
to include it right now. Let's put this on hold for a little bit and
see how bpftrace users use it first, we can always add something to
libbpf later.

>
> >
> > You can still attach to such dynamic probe today with pure libbpf
> > (e.g., if bpftrace needs to do this, for example) by creating
> > perf_event FD from tracefs' id, and then using
> > bpf_program__attach_perf_event_opts() to attach to it. It will be on
> > the user to use either tracepoint or kprobe BPF program for such
> > attachment.
>
> You're right—we can manually attach it in the tools, but if we can
> make it auto-attachable in libbpf, why not take advantage of that and
> simplify the process?
>
>
> >
> > Yes, this doesn't work "declaratively" with a nice SEC("...") syntax,
> > but at least it's doable programmatically, and that's what matters for
> > bpftrace.
>
>
> --
> Regards
> Yafang