[RFC PATCH bpf-next 0/2] libbpf: Add support for dynamic tracepoint

[RFC PATCH bpf-next 0/2] libbpf: Add support for dynamic tracepoint

[Yafang Shao]

Dynamic tracepoints can be created using tools like perf-probe, debugfs, or
similar. For example:

- perf probe
  $ perf probe -a 'tcp_listendrop sk'
  $ ls /sys/kernel/debug/tracing/events/probe/tcp_listendrop/
  enable  filter  format  hist  id  trigger

- debugfs
  $ echo 'p:myprobe kernel_clone' >> /sys/kernel/debug/tracing/kprobe_events
  $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
  enable  filter  format  hist  id  trigger

While these dynamic tracepoints are functional, they cannot be easily
attached to BPF programs. For instance, attempting to use them with
bpftrace results in the following error:

  $ bpftrace -e 'tracepoint:probe:tcp_listendrop {print(comm)}'
  Attaching 1 probe...
  ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument
  ERROR: Error attaching probe: tracepoint:probe:tcp_listendrop

The issue lies in how these dynamic tracepoints are implemented: despite
being exposed as tracepoints, they remain kprobe events internally. As a
result, loading them as a tracepoint program fails. Instead, they must be
loaded as kprobe programs but attached as tracepoints.

This patchset addresses the limitation, enabling seamless attachment of
such tracepoints with BPF. It simplifies tracing inlined kernel functions
using BPF.
 
Yafang Shao (2):
  libbpf: Add support for dynamic tracepoint
  selftests/bpf: Add selftest for dynamic tracepoint

 tools/lib/bpf/libbpf.c                        |  3 +
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 3 files changed, 94 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c

-- 
2.43.5

[RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Yafang Shao]

Dynamic tracepoints can be created using debugfs. For example:

   echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events

This command creates a new tracepoint under debugfs:

  $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
  enable  filter  format  hist  id  trigger

Although this dynamic tracepoint appears as a tracepoint, it is internally
implemented as a kprobe. However, it must be attached as a tracepoint to
function correctly in certain contexts.

This update adds support in libbpf for handling such tracepoints,
simplifying their usage and integration in BPF workflows.

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 tools/lib/bpf/libbpf.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 66173ddb5a2d..077bec761ebf 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -9504,6 +9504,7 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("struct_ops.s+",	STRUCT_OPS, 0, SEC_SLEEPABLE),
 	SEC_DEF("sk_lookup",		SK_LOOKUP, BPF_SK_LOOKUP, SEC_ATTACHABLE),
 	SEC_DEF("netfilter",		NETFILTER, BPF_NETFILTER, SEC_NONE),
+	SEC_DEF("dynamic_tp+",          KPROBE, 0, SEC_NONE, attach_tp),
 };
 
 int libbpf_register_prog_handler(const char *sec,
@@ -12500,6 +12501,8 @@ static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_lin
 	/* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */
 	if (str_has_pfx(prog->sec_name, "tp/"))
 		tp_cat = sec_name + sizeof("tp/") - 1;
+	else if (str_has_pfx(prog->sec_name, "dynamic_tp/"))
+		tp_cat = sec_name + sizeof("dynamic_tp/") - 1;
 	else
 		tp_cat = sec_name + sizeof("tracepoint/") - 1;
 	tp_name = strchr(tp_cat, '/');
-- 
2.43.5

[RFC PATCH bpf-next 2/2] selftests/bpf: Add selftest for dynamic tracepoint

[Yafang Shao]

The result is as follows,

  $ tools/testing/selftests/bpf/test_progs --name=dynamic_tp
  #85      dynamic_tp:OK
  Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 .../bpf/prog_tests/test_dynamic_tp.c          | 64 +++++++++++++++++++
 .../testing/selftests/bpf/progs/dynamic_tp.c  | 27 ++++++++
 2 files changed, 91 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
 create mode 100644 tools/testing/selftests/bpf/progs/dynamic_tp.c

diff --git a/tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c b/tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
new file mode 100644
index 000000000000..b3d46a3db03a
--- /dev/null
+++ b/tools/testing/selftests/bpf/prog_tests/test_dynamic_tp.c
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <test_progs.h>
+#include <bpf/btf.h>
+#include <bpf/bpf.h>
+
+#include "dynamic_tp.skel.h"
+
+int dynamic_tp(const char *cmd)
+{
+	const char *kprobe_file = "/sys/kernel/debug/tracing/kprobe_events";
+	ssize_t bytes_written;
+	int fd, err;
+
+	fd = open(kprobe_file, O_WRONLY | O_APPEND);
+	if (!ASSERT_GE(fd, 0, "open kprobe_events"))
+		return -1;
+
+	bytes_written = write(fd, cmd, strlen(cmd));
+	if (!ASSERT_GT(bytes_written, 0, "write kprobe_events")) {
+		close(fd);
+		return -1;
+	}
+
+	err = close(fd);
+	if (!ASSERT_OK(err, "close kprobe_events"))
+		return -1;
+	return 0;
+}
+
+void test_dynamic_tp(void)
+{
+	struct dynamic_tp *skel;
+	pid_t child_pid;
+	int status, err;
+
+	/* create a dynamic tracepoint */
+	err = dynamic_tp("p:my_dynamic_tp kernel_clone");
+	if (!ASSERT_OK(err, "create dynamic tp"))
+		return;
+
+	skel = dynamic_tp__open_and_load();
+	if (!ASSERT_OK_PTR(skel, "load progs"))
+		goto cleanup;
+	skel->bss->pid = getpid();
+	err = dynamic_tp__attach(skel);
+	if (!ASSERT_OK(err, "attach progs"))
+		goto cleanup;
+
+	/* trigger the dynamic tracepoint */
+	child_pid = fork();
+	if (!ASSERT_GT(child_pid, -1, "child_pid"))
+		goto cleanup;
+	if (child_pid == 0)
+		_exit(0);
+	waitpid(child_pid, &status, 0);
+
+	ASSERT_EQ(skel->bss->result, 1, "result");
+
+cleanup:
+	dynamic_tp__destroy(skel);
+	/* remove the dynamic tracepoint */
+	dynamic_tp("-:my_dynamic_tp kernel_clone");
+}
diff --git a/tools/testing/selftests/bpf/progs/dynamic_tp.c b/tools/testing/selftests/bpf/progs/dynamic_tp.c
new file mode 100644
index 000000000000..0aec0cd48547
--- /dev/null
+++ b/tools/testing/selftests/bpf/progs/dynamic_tp.c
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <vmlinux.h>
+#include <bpf/bpf_helpers.h>
+
+char _license[] SEC("license") = "GPL";
+
+#define MAX_STACK_TRACE_DEPTH 32
+unsigned long entries[MAX_STACK_TRACE_DEPTH] = {};
+#define SIZE_OF_ULONG (sizeof(unsigned long))
+
+int result, pid;
+
+SEC("dynamic_tp/kprobes/my_dynamic_tp")
+int dynamic_tp(struct pt_regs *ctx)
+{
+	int ret;
+
+	ret = bpf_get_stack(ctx, entries, MAX_STACK_TRACE_DEPTH * SIZE_OF_ULONG, 0);
+	if (ret < 0) {
+		result = -1;
+		return ret;
+	}
+	if (bpf_get_current_pid_tgid() >> 32 == pid)
+		result = 1;
+	return 0;
+}
-- 
2.43.5

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Alexei Starovoitov]

On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> Dynamic tracepoints can be created using debugfs. For example:
>
>    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
>
> This command creates a new tracepoint under debugfs:
>
>   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
>   enable  filter  format  hist  id  trigger
>
> Although this dynamic tracepoint appears as a tracepoint, it is internally
> implemented as a kprobe. However, it must be attached as a tracepoint to
> function correctly in certain contexts.

Nack.
There are multiple mechanisms to create kprobe/tp via text interfaces.
We're not going to mix them with the programmatic libbpf api.

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Yafang Shao]

On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > Dynamic tracepoints can be created using debugfs. For example:
> >
> >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> >
> > This command creates a new tracepoint under debugfs:
> >
> >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> >   enable  filter  format  hist  id  trigger
> >
> > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > implemented as a kprobe. However, it must be attached as a tracepoint to
> > function correctly in certain contexts.
>
> Nack.
> There are multiple mechanisms to create kprobe/tp via text interfaces.
> We're not going to mix them with the programmatic libbpf api.

It appears that bpftrace still lacks support for adding a kprobe/tp
and then attaching to it directly. Is that correct?
What do you think about introducing this mechanism into bpftrace? With
such a feature, we could easily attach to inlined kernel functions
using bpftrace.

--
Regards
Yafang

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Alexei Starovoitov]

On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <laoar.shao@gmail.com> wrote:
>
> On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > Dynamic tracepoints can be created using debugfs. For example:
> > >
> > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > >
> > > This command creates a new tracepoint under debugfs:
> > >
> > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > >   enable  filter  format  hist  id  trigger
> > >
> > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > function correctly in certain contexts.
> >
> > Nack.
> > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > We're not going to mix them with the programmatic libbpf api.
>
> It appears that bpftrace still lacks support for adding a kprobe/tp
> and then attaching to it directly. Is that correct?

what do you mean?
bpftrace supports both kprobe attaching and tp too.

> What do you think about introducing this mechanism into bpftrace? With
> such a feature, we could easily attach to inlined kernel functions
> using bpftrace.

Attaching to inlined funcs also sort-of works. It relies on dwarf,
and there is work in progress to add a special section to vmlinux
to annotate inlined sites, so it can work without dwarf.

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Yafang Shao]

On Tue, Jan 7, 2025 at 6:33 AM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <laoar.shao@gmail.com> wrote:
> >
> > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > >
> > > > Dynamic tracepoints can be created using debugfs. For example:
> > > >
> > > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > > >
> > > > This command creates a new tracepoint under debugfs:
> > > >
> > > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > > >   enable  filter  format  hist  id  trigger
> > > >
> > > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > > function correctly in certain contexts.
> > >
> > > Nack.
> > > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > > We're not going to mix them with the programmatic libbpf api.
> >
> > It appears that bpftrace still lacks support for adding a kprobe/tp
> > and then attaching to it directly. Is that correct?
>
> what do you mean?

Take the inlined kernel function tcp_listendrop() as an example:

$ perf probe -a 'tcp_listendrop sk'
Added new events:
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)
  probe:tcp_listendrop (on tcp_listendrop with sk)

You can now use it in all perf tools, such as:

        perf record -e probe:tcp_listendrop -aR sleep 1

Similarly, we can also use bpftrace to trace inlined kernel functions.
For example:

- add a dynamic tracepoint
  $ bpftrace probe -a 'tcp_listendrop sk'

- trace the dynamic tracepoint
  $ bpftrace probe -e 'probe:tcp_listendrop {print(args->sk)}'

> bpftrace supports both kprobe attaching and tp too.

The dynamic tracepoint is not supported yet.

>
> > What do you think about introducing this mechanism into bpftrace? With
> > such a feature, we could easily attach to inlined kernel functions
> > using bpftrace.
>
> Attaching to inlined funcs also sort-of works. It relies on dwarf,
> and there is work in progress to add a special section to vmlinux
> to annotate inlined sites, so it can work without dwarf.

What’s the benefit of doing this? Why not simply read the DWARF
information directly from vmlinux?

$ readelf -S /boot/vmlinux  | grep debug_info
  [63] .debug_info       PROGBITS         0000000000000000  03e2bc20

The DWARF information embedded in vmlinux makes it straightforward to
trace inlined functions without requiring any kernel modifications.
This approach allows all existing kernel releases to immediately take
advantage of the functionality, eliminating the need for kernel
recompilation or patching.


--
Regards
Yafang

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Jiri Olsa]

On Mon, Jan 06, 2025 at 10:32:15AM +0800, Yafang Shao wrote:
> On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > Dynamic tracepoints can be created using debugfs. For example:
> > >
> > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > >
> > > This command creates a new tracepoint under debugfs:
> > >
> > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > >   enable  filter  format  hist  id  trigger
> > >
> > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > function correctly in certain contexts.
> >
> > Nack.
> > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > We're not going to mix them with the programmatic libbpf api.
> 
> It appears that bpftrace still lacks support for adding a kprobe/tp
> and then attaching to it directly. Is that correct?
> What do you think about introducing this mechanism into bpftrace? With
> such a feature, we could easily attach to inlined kernel functions
> using bpftrace.

so with the 'echo .. > kprobe_events' you create kprobe which will be
exported through tracefs together with other tracepoints and bpftrace
sees it as another tracepoint.. but it's a kprobe :-\

how about we add support for kprobe section like SEC("kprobe/SUBSYSTEM/PROBE"),
so in your case above it'd be SEC("kprobe/kprobes/myprobe")

then attach_kprobe would parse that out and use new new probe_attach_mode
for bpf_program__attach_kprobe_opts to attach it correctly

cc-ing Viktor

jirka

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Yafang Shao]

On Tue, Jan 7, 2025 at 8:16 PM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Mon, Jan 06, 2025 at 10:32:15AM +0800, Yafang Shao wrote:
> > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > >
> > > > Dynamic tracepoints can be created using debugfs. For example:
> > > >
> > > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > > >
> > > > This command creates a new tracepoint under debugfs:
> > > >
> > > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > > >   enable  filter  format  hist  id  trigger
> > > >
> > > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > > function correctly in certain contexts.
> > >
> > > Nack.
> > > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > > We're not going to mix them with the programmatic libbpf api.
> >
> > It appears that bpftrace still lacks support for adding a kprobe/tp
> > and then attaching to it directly. Is that correct?
> > What do you think about introducing this mechanism into bpftrace? With
> > such a feature, we could easily attach to inlined kernel functions
> > using bpftrace.
>
> so with the 'echo .. > kprobe_events' you create kprobe which will be
> exported through tracefs together with other tracepoints and bpftrace
> sees it as another tracepoint.. but it's a kprobe :-\

exactly.

>
> how about we add support for kprobe section like SEC("kprobe/SUBSYSTEM/PROBE"),
> so in your case above it'd be SEC("kprobe/kprobes/myprobe")

This is similar to what I'm currently proposing:

  SEC("dynamic_tp/kprobes/my_dynamic_tp")

My proposal requires only a 3-line change. In contrast, if we
implement it as you suggested, it may require significantly more code
changes. I prefer to introduce a new section, such as
SEC("dynamic_tracepoint/"), SEC("kprobe_tracepoint/"), or something
similar, for this special type of kprobe. However, if you believe
SEC("kprobe/SUBSYSTEM/PROBE") is a better approach, I’m happy to
implement it that way.

>
> then attach_kprobe would parse that out and use new new probe_attach_mode
> for bpf_program__attach_kprobe_opts to attach it correctly

Yes, that would be a great enhancement for tracing inlined kernel functions.

--
Regards
Yafang

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Daniel Xu]

Hi Yafang,

On Mon, Jan 06, 2025 at 10:32:15AM +0800, Yafang Shao wrote:
> On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > Dynamic tracepoints can be created using debugfs. For example:
> > >
> > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > >
> > > This command creates a new tracepoint under debugfs:
> > >
> > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > >   enable  filter  format  hist  id  trigger
> > >
> > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > function correctly in certain contexts.
> >
> > Nack.
> > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > We're not going to mix them with the programmatic libbpf api.
> 
> It appears that bpftrace still lacks support for adding a kprobe/tp
> and then attaching to it directly. Is that correct?
> What do you think about introducing this mechanism into bpftrace? With
> such a feature, we could easily attach to inlined kernel functions
> using bpftrace.

Is the idea to have some other application create dynamic tracepoints
based on kernel debuginfo?

FWIW bpftrace has some initial support for probing inlined kernel
functions w/ DWARF. I don't believe it's enabled by default yet, though
- there's a few limitations. I'll comment in thread below with more
details.

Thanks,
Daniel

Re: [RFC PATCH bpf-next 1/2] libbpf: Add support for dynamic tracepoint

[Daniel Xu]

On Tue, Jan 07, 2025 at 10:41:46AM +0800, Yafang Shao wrote:
> On Tue, Jan 7, 2025 at 6:33 AM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Sun, Jan 5, 2025 at 6:32 PM Yafang Shao <laoar.shao@gmail.com> wrote:
> > >
> > > On Mon, Jan 6, 2025 at 8:16 AM Alexei Starovoitov
> > > <alexei.starovoitov@gmail.com> wrote:
> > > >
> > > > On Sun, Jan 5, 2025 at 4:44 AM Yafang Shao <laoar.shao@gmail.com> wrote:
> > > > >
> > > > > Dynamic tracepoints can be created using debugfs. For example:
> > > > >
> > > > >    echo 'p:myprobe kernel_clone args' >> /sys/kernel/debug/tracing/kprobe_events
> > > > >
> > > > > This command creates a new tracepoint under debugfs:
> > > > >
> > > > >   $ ls /sys/kernel/debug/tracing/events/kprobes/myprobe/
> > > > >   enable  filter  format  hist  id  trigger
> > > > >
> > > > > Although this dynamic tracepoint appears as a tracepoint, it is internally
> > > > > implemented as a kprobe. However, it must be attached as a tracepoint to
> > > > > function correctly in certain contexts.
> > > >
> > > > Nack.
> > > > There are multiple mechanisms to create kprobe/tp via text interfaces.
> > > > We're not going to mix them with the programmatic libbpf api.
> > >
> > > It appears that bpftrace still lacks support for adding a kprobe/tp
> > > and then attaching to it directly. Is that correct?
> >
> > what do you mean?
> 
> Take the inlined kernel function tcp_listendrop() as an example:
> 
> $ perf probe -a 'tcp_listendrop sk'
> Added new events:
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
>   probe:tcp_listendrop (on tcp_listendrop with sk)
> 
> You can now use it in all perf tools, such as:
> 
>         perf record -e probe:tcp_listendrop -aR sleep 1

Cool, I'm guessing perf-probe can speak DWARF and will parse all the
inline information.

> 
> Similarly, we can also use bpftrace to trace inlined kernel functions.
> For example:
> 
> - add a dynamic tracepoint
>   $ bpftrace probe -a 'tcp_listendrop sk'
> 
> - trace the dynamic tracepoint
>   $ bpftrace probe -e 'probe:tcp_listendrop {print(args->sk)}'
> 
> > bpftrace supports both kprobe attaching and tp too.
> 
> The dynamic tracepoint is not supported yet.
> 
> >
> > > What do you think about introducing this mechanism into bpftrace? With
> > > such a feature, we could easily attach to inlined kernel functions
> > > using bpftrace.
> >
> > Attaching to inlined funcs also sort-of works. It relies on dwarf,
> > and there is work in progress to add a special section to vmlinux
> > to annotate inlined sites, so it can work without dwarf.
> 
> What’s the benefit of doing this? Why not simply read the DWARF
> information directly from vmlinux?
> 
> $ readelf -S /boot/vmlinux  | grep debug_info
>   [63] .debug_info       PROGBITS         0000000000000000  03e2bc20
> 
> The DWARF information embedded in vmlinux makes it straightforward to
> trace inlined functions without requiring any kernel modifications.
> This approach allows all existing kernel releases to immediately take
> advantage of the functionality, eliminating the need for kernel
> recompilation or patching.

I'd disagree that this approach works with all existing kernels. Kernel
debuginfo is usually not available by default. On some distros, it's not
available at all.

This is particularly relevant for partial inlining - where compiler
inlines some callsites but leaves the symbol in. In these cases, users
trying to probe a symbol will succeed in attaching but then silently
lose events. There is no obvious way for user to know to install
debuginfo. Or to create dynamic tracepoints.

This is the motivation for always-available metadata. Something small
enough where distros can leave it on by default. Similar to motivation
for BTF. There's also overhead involved w/ parsing DWARF. A more compact
representation helps reduce overhead.