Re: [PATCH v14 00/19] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph

[Masami Hiramatsu (Google) <mhiramat@kernel.org>]

On Thu, 12 Sep 2024 18:55:40 -0700
Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Thu, Sep 12, 2024 at 4:54 PM Masami Hiramatsu <mhiramat@kernel.org> wrote:
> >
> > On Thu, 12 Sep 2024 11:41:17 -0700
> > Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> >
> > > + BPF ML
> > >
> > > On Thu, Sep 12, 2024 at 8:35 AM <bot+bpf-ci@kernel.org> wrote:
> > > >
> > > > Dear patch submitter,
> > > >
> > > > CI has tested the following submission:
> > > > Status:     FAILURE
> > > > Name:       [v14,00/19] tracing: fprobe: function_graph: Multi-function graph and fprobe on fgraph
> > > > Patchwork:  https://patchwork.kernel.org/project/netdevbpf/list/?series=889822&state=*
> > > > Matrix:     https://github.com/kernel-patches/bpf/actions/runs/10833792984
> > > >
> > > > Failed jobs:
> > > > test_progs-aarch64-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061791397
> > > > test_progs_no_alu32-aarch64-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061791836
> > > > test_progs-s390x-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061757062
> > > > test_progs_no_alu32-s390x-gcc: https://github.com/kernel-patches/bpf/actions/runs/10833792984/job/30061757809
> > > >
> > > > First test_progs failure (test_progs-aarch64-gcc):
> > > > #132 kprobe_multi_testmod_test
> > > > serial_test_kprobe_multi_testmod_test:PASS:load_kallsyms_local 0 nsec
> > > > #132/1 kprobe_multi_testmod_test/testmod_attach_api_syms
> > > > test_testmod_attach_api:PASS:fentry_raw_skel_load 0 nsec
> > > > trigger_module_test_read:PASS:testmod_file_open 0 nsec
> > > > test_testmod_attach_api:PASS:trigger_read 0 nsec
> > > > kprobe_multi_testmod_check:FAIL:kprobe_test1_result unexpected kprobe_test1_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kprobe_test2_result unexpected kprobe_test2_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kprobe_test3_result unexpected kprobe_test3_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test1_result unexpected kretprobe_test1_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test2_result unexpected kretprobe_test2_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test3_result unexpected kretprobe_test3_result: actual 0 != expected 1
> > > > #132/2 kprobe_multi_testmod_test/testmod_attach_api_addrs
> > > > test_testmod_attach_api_addrs:PASS:ksym_get_addr_local 0 nsec
> > > > test_testmod_attach_api_addrs:PASS:ksym_get_addr_local 0 nsec
> > > > test_testmod_attach_api_addrs:PASS:ksym_get_addr_local 0 nsec
> > > > test_testmod_attach_api:PASS:fentry_raw_skel_load 0 nsec
> > > > trigger_module_test_read:PASS:testmod_file_open 0 nsec
> > > > test_testmod_attach_api:PASS:trigger_read 0 nsec
> > > > kprobe_multi_testmod_check:FAIL:kprobe_test1_result unexpected kprobe_test1_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kprobe_test2_result unexpected kprobe_test2_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kprobe_test3_result unexpected kprobe_test3_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test1_result unexpected kretprobe_test1_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test2_result unexpected kretprobe_test2_result: actual 0 != expected 1
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test3_result unexpected kretprobe_test3_result: actual 0 != expected 1
> > > >
> > >
> > > Seems like this selftest is still broken. Please let me know if you
> > > need help with building and running BPF selftests to reproduce this
> > > locally.
> >
> > Thanks, It will be helpful. Also I would like to know, is there any
> > debug mode (like print more debug logs)?
> 
> So first of all, the assumption is that you will build a most recent
> kernel with Kconfig values set from
> tools/testing/selftests/bpf/config, so make sure you  append that to
> your kernel config. Then build the kernel, BPF selftests' Makefile
> tries to find latest built kernel (according to KBUILD_OUTPUT/O/etc).
> 
> Now to building BPF selftests:
> 
> $ cd tools/testing/selftests/bpf
> $ make -j$(nproc)
> 
> You'll need decently recent Clang and a few dependent packages. At
> least elfutils-devel, libzstd-devel, but there might be a few more
> which I never can remember.
> 
> Once everything is built, you can run the failing test with
> 
> $ sudo ./test_progs -t kprobe_multi_testmod_test -v
> 
> The source code for user space part for that test is in
> prog_tests/kprobe_multi_testmod_test.c and BPF-side is in
> progs/kprobe_multi.c.

Thanks for the information!

> 
> Taking failing output from the test:
> 
> > > > kprobe_multi_testmod_check:FAIL:kretprobe_test3_result unexpected kretprobe_test3_result: actual 0 != expected 1
> 
> kretprobe_test3_result is a sort of identifier for a test condition,
> you can find a corresponding line in user space .c file grepping for
> that:
> 
> ASSERT_EQ(skel->bss->kretprobe_testmod_test3_result, 1,
> "kretprobe_test3_result");
> 
> Most probably the problem is in:
> 
> __u64 addr = bpf_get_func_ip(ctx);

Yeah, and as I replyed to another thread, the problem is
that the ftrace entry_ip is not symbol ip.

We have ftrace_call_adjust() arch function for reverse
direction (symbol ip to ftrace entry ip) but what we need
here is the reverse translate function (ftrace entry to symbol)

The easiest way is to use kallsyms to find it, but this is
a bit costful (but it just increase bsearch in several levels).
Other possible options are

 - Change bpf_kprobe_multi_addrs_cmp() to accept a range
   of address. [sym_addr, sym_addr + offset) returns true,
   bpf_kprobe_multi_cookie() can find the entry address.
   The offset depends on arch, but 16 is enough.

 - Change bpf_kprobe_multi_addrs_cmp() to call
   ftrace_call_adjust() before comparing. This may take a
   cost but find actual entry address.

 - (Cached method) when making link->addrs, make a link->adj_addrs
   array too, where adj_addrs[i] == ftrace_call_adjust(addrs[i]).

Let me try the 3rd one. It may consume more memory but the
fastest solution.

Thank you,

> 
> which is then checked as
> 
> if ((const void *) addr == &bpf_testmod_fentry_test3)
> 
> 
> With your patches this doesn't match anymore.

It actually enables the fprobe on those architectures, so
without my series, those test should be skipped.

Thank you,

> 
> 
> Hopefully the above gives you some pointers, let me know if you run
> into any problems.
> 
> >
> > Thank you,
> >
> > >
> > > >
> > > > Please note: this email is coming from an unmonitored mailbox. If you have
> > > > questions or feedback, please reach out to the Meta Kernel CI team at
> > > > kernel-ci@meta.com.
> >
> >
> > --
> > Masami Hiramatsu (Google) <mhiramat@kernel.org>


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>