Re: [RFC PATCH v2 0/8] fprobe: Introduce fprobe function entry/exit probe

Andrii Nakryiko
2022-01-15T01:08:45Z

On Wed, Jan 12, 2022 at 6:02 AM Masami Hiramatsu <mhiramat@kernel.org> wrote:
>
> Hi Jiri and Alexei,
>
> Here is the 2nd version of fprobe. This version uses the
> ftrace_set_filter_ips() for reducing the registering overhead.
> Note that this also drops per-probe point private data, which
> is not used anyway.

This per-probe private data is necessary for the feature called BPF
cookie, in which each attachment has a unique user-provided u64 value
associated to it, accessible at runtime through
bpf_get_attach_cookie() helper. One way or another we'll need to
support this to make these multi-attach BPF programs really useful for
generic tracing applications.

Jiri,

We've discussed with Alexei this week how cookies can be supported for
multi-attach fentry (where it seems even more challenging than in
kprobe case), and agreed on rather simple solution, which roughly goes
like this. When multi-attaching either fentry/fexit program, save
sorted array of IP addresses and then sorted in the same order as IPs
list of u64 cookies. At runtime, bpf_get_attach_cookie() helper should
somehow get access to these two arrays and functions IP (that we
already have with bpf_get_func_ip()), perform binary search and locate
necessary cookie. This offloads the overhead of finding this cookie to
actual call site of bpf_get_attach_cookie() (and it's a log(N), so not
bad at all, especially if BPF program can be optimized to call this
helper just once).

I think something like that should be doable for Masami's fprobe-based
multi-attach kprobes, right? That would allow to have super-fast
attachment, but still support BPF cookie per each individual IP/kernel
function attachment. I haven't looked at code thoroughly, though,
please let me know if I'm missing something fundamental.

>
> This introduces the fprobe, the function entry/exit probe with
> multiple probe point support. This also introduces the rethook
> for hooking function return as same as kretprobe does. This
> abstraction will help us to generalize the fgraph tracer,
> because we can just switch it from rethook in fprobe, depending
> on the kernel configuration.
>
> The patch [1/8] and [7/8] are from your series[1]. Other libbpf
> patches will not be affected by this change.
>
> [1] https://lore.kernel.org/all/20220104080943.113249-1-jolsa@kernel.org/T/#u
>
> I also added an out-of-tree (just for testing) patch at the
> end of this series ([8/8]) for adding a wildcard support to
> the sample program. With that patch, it shows how long the
> registration will take;
>
> # time insmod fprobe_example.ko symbol='btrfs_*'
> [ 36.130947] fprobe_init: 1028 symbols found
> [ 36.177901] fprobe_init: Planted fprobe at btrfs_*
> real 0m 0.08s
> user 0m 0.00s
> sys 0m 0.07s
>
> Thank you,
>
> ---
>
> Jiri Olsa (2):
> ftrace: Add ftrace_set_filter_ips function
> bpf: Add kprobe link for attaching raw kprobes
>
> Masami Hiramatsu (6):
> fprobe: Add ftrace based probe APIs
> rethook: Add a generic return hook
> rethook: x86: Add rethook x86 implementation
> fprobe: Add exit_handler support
> fprobe: Add sample program for fprobe
> [DO NOT MERGE] Out-of-tree: Support wildcard symbol option to sample
>
>
> arch/x86/Kconfig | 1
> arch/x86/kernel/Makefile | 1
> arch/x86/kernel/rethook.c | 115 ++++++++++++++++++++
> include/linux/bpf_types.h | 1
> include/linux/fprobe.h | 57 ++++++++++
> include/linux/ftrace.h | 3 +
> include/linux/rethook.h | 74 +++++++++++++
> include/linux/sched.h | 3 +
> include/uapi/linux/bpf.h | 12 ++
> kernel/bpf/syscall.c | 195 +++++++++++++++++++++++++++++++++-
> kernel/exit.c | 2
> kernel/fork.c | 3 +
> kernel/kallsyms.c | 1
> kernel/trace/Kconfig | 22 ++++
> kernel/trace/Makefile | 2
> kernel/trace/fprobe.c | 168 +++++++++++++++++++++++++++++
> kernel/trace/ftrace.c | 54 ++++++++-
> kernel/trace/rethook.c | 226 +++++++++++++++++++++++++++++++++++++++
> samples/Kconfig | 7 +
> samples/Makefile | 1
> samples/fprobe/Makefile | 3 +
> samples/fprobe/fprobe_example.c | 154 +++++++++++++++++++++++++++
> tools/include/uapi/linux/bpf.h | 12 ++
> 23 files changed, 1103 insertions(+), 14 deletions(-)
> create mode 100644 arch/x86/kernel/rethook.c
> create mode 100644 include/linux/fprobe.h
> create mode 100644 include/linux/rethook.h
> create mode 100644 kernel/trace/fprobe.c
> create mode 100644 kernel/trace/rethook.c
> create mode 100644 samples/fprobe/Makefile
> create mode 100644 samples/fprobe/fprobe_example.c
>
> --
> Masami Hiramatsu (Linaro) <mhiramat@kernel.org>