Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Namhyung Kim <namhyung@kernel.org>]

On Fri, Jan 31, 2025 at 04:14:09PM -0800, Howard Chu wrote:
> Hello Namhyung,
> 
> Sorry for somehow missing this reply...
> 
> On Thu, Jan 23, 2025 at 3:53 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > On Tue, Jan 21, 2025 at 04:19:35PM -0800, Howard Chu wrote:
> > > Hello Namhyung,
> > >
> > > On Sat, Jan 18, 2025 at 10:28 AM Namhyung Kim <namhyung@kernel.org> wrote:
> > > >
> > > > On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > > > > > Hello Arnaldo,
> > > > > >
> > > > > > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > > > > > <acme@kernel.org> wrote:
> > > > > > >
> > > > > > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > > > > > Changes in v4:
> > > > > > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > > > > > >    running the test.
> > > > > > > > > >
> > > > > > > > > > Changes in v3:
> > > > > > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > > > > > >
> > > > > > > > > > v1, v2:
> > > > > > > > > >
> > > > > > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > > > > > > of POSIX shell to pass Shellcheck.
> > > > > > > > > >
> > > > > > > > > > This patch series also adds documentation for the new option
> > > > > > > > > > --force-btf, which is used in the tests.
> > > > > > > > > >
> > > > > > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > > > > > >
> > > > > > > > > > Howard Chu (2):
> > > > > > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > > > > > >   perf docs: Add documentation for --force-btf option
> > > > > > > > >
> > > > > > > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > > > > > > >
> > > > > > > > Thanks, applied to perf-tools-next,
> > > > > > >
> > > > > > > It fails when running in parallel mode, sometimes:
> > > > > > >
> > > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > >
> > > > > > > Then:
> > > > > > >
> > > > > > > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > >
> > > > > > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > > > > > >
> > > > > > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > > > > > >
> > > > > > > root@number:~#
> > > > > > >
> > > > > > >
> > > > > > > So its not just when running in paralell, anyway, its merged, we can go
> > > > > > > on from what we got there.
> > > > > >
> > > > > > It never fails on my machine, I think the reason is my machine is not
> > > > > > fully-loaded. Can you please run
> > > > > > ```
> > > > > > perf stat --null -r 10 perf test -vv "BTF general"
> > > > > > ```
> > > > > > And provide the output?
> > > > >
> > > > > root@number:~# grep -m1 'model name' /proc/cpuinfo
> > > > > model name    : Intel(R) Core(TM) i7-14700K
> > > > > root@number:~#
> > > > >
> > > > > root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
> > > > > 106: perf trace BTF general tests:
> > > > > --- start ---
> > > > > test child forked, pid 47824
> > > > > Checking if vmlinux BTF exists
> > > > > Testing perf trace's string augmentation
> > > > > String augmentation test failed
> > > > > ---- end(-1) ----
> > > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > > 106: perf trace BTF general tests:
> > > > > --- start ---
> > > > > test child forked, pid 47842
> > > > > Checking if vmlinux BTF exists
> > > > > Testing perf trace's string augmentation
> > > > > String augmentation test failed
> > > > > ---- end(-1) ----
> > > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > > ...
> > > >
> > > > I also see the failure on not fully-loaded machines (both Intel and
> > > > AMD).
> > > >
> > > >   $ sudo ./perf test -v BTF
> > > >   --- start ---
> > > >   test child forked, pid 525025
> > > >   Checking if vmlinux BTF exists
> > > >   Testing perf trace's string augmentation
> > > >   String augmentation test failed
> > > >   ---- end(-1) ----
> > > >   106: perf trace BTF general tests                                    : FAILED!
> > > >
> > > > Thanks,
> > > > Namhyung
> > > >
> > >
> > > Just wondering, did you and Arnaldo apply this patch:
> > > https://lore.kernel.org/linux-perf-users/20241213023047.541218-1-howardchu95@gmail.com/
> >
> > I don't think we applied it, will add to perf-tools.
> >
> > >
> > > This seems to solve the problem for me...
> >
> > But I'm not sure if it fixes the problem.  IIUC the patch fixes the
> > loading problem.
> 
> Yes. However, if the machine is unable to load the BPF program, the
> error message will be:
> 
> perf $ sudo ./perf test general -v
> --- start ---
> test child forked, pid 173990
> Checking if vmlinux BTF exists
> Testing perf trace's string augmentation
> String augmentation test failed
> ---- end(-1) ----
> 106: perf trace BTF general tests                                    : FAILED!
> 
> Which is indistinguishable from an actual string augmentation error
> that has a string mismatch...

Ok, that's not good.

> 
> > And for some reason, I don't see the problem any more.
> > But I remember this sometimes happened in the past.
> 
> Do you mean that perf trace breaks occasionally? This sounds urgent.
> Could you please provide me with your kernel version and Clang
> version? Thank you very much! :)

Yep, sometimes it works and sometimes it failed to load the BPF.

The kernel version of this laptop is v6.10.11 and clang version is
16.0.6.

Thanks,
Namhyung