[PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Howard Chu]

Changes in v4:
 - Use if -f to check the existence of vmlinux BTF, and exit if it
   doesn't, so trace_test_string will not overwrite $err, and keep
   running the test.

Changes in v3:
 - Add vmlinux BTF check, and skip the tests if it doesn't exist

v1, v2:

The previous version of the perf trace BTF general augmentation tests
didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
<acme@kernel.org> for pointing this out), this version uses bash instead
of POSIX shell to pass Shellcheck.

This patch series also adds documentation for the new option
--force-btf, which is used in the tests.

Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/

Howard Chu (2):
  perf trace: Add tests for BTF general augmentation
  perf docs: Add documentation for --force-btf option

 tools/perf/Documentation/perf-trace.txt     |  5 ++
 tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++
 2 files changed, 98 insertions(+)
 create mode 100755 tools/perf/tests/shell/trace_btf_general.sh

-- 
2.43.0

[PATCH v4 1/2] perf trace: Add tests for BTF general augmentation

[Howard Chu]

Currently, we only have perf trace augmentation tests for enum
arguments. This patch adds tests for more general syscall arguments,
such as struct pointers, strings, and buffers.

These tests utilize the perf config system to configure the perf trace
output, as suggested by Arnaldo Carvalho de Melo <acme@kernel.org>

Suggested-by: Arnaldo Carvalho de Melo <acme@kernel.org>
Signed-off-by: Howard Chu <howardchu95@gmail.com>
---
 tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++
 1 file changed, 93 insertions(+)
 create mode 100755 tools/perf/tests/shell/trace_btf_general.sh

diff --git a/tools/perf/tests/shell/trace_btf_general.sh b/tools/perf/tests/shell/trace_btf_general.sh
new file mode 100755
index 000000000000..005c93e6c770
--- /dev/null
+++ b/tools/perf/tests/shell/trace_btf_general.sh
@@ -0,0 +1,93 @@
+#!/bin/bash
+# perf trace BTF general tests
+# SPDX-License-Identifier: GPL-2.0
+
+err=0
+set -e
+
+. "$(dirname $0)"/lib/probe.sh
+
+file1=$(mktemp /tmp/file1_XXXX)
+file2=$(echo $file1 | sed 's/file1/file2/g')
+
+buffer="buffer content"
+perf_config_tmp=$(mktemp /tmp/.perfconfig_XXXXX)
+
+trap cleanup EXIT TERM INT HUP
+
+check_vmlinux() {
+  echo "Checking if vmlinux BTF exists"
+  if [ ! -f /sys/kernel/btf/vmlinux ]
+  then
+    echo "Skipped due to missing vmlinux BTF"
+    return 2
+  fi
+  return 0
+}
+
+trace_test_string() {
+  echo "Testing perf trace's string augmentation"
+  if ! perf trace -e renameat* --max-events=1 -- mv ${file1} ${file2} 2>&1 | \
+    grep -q -E "^mv/[0-9]+ renameat(2)?\(.*, \"${file1}\", .*, \"${file2}\", .*\) += +[0-9]+$"
+  then
+    echo "String augmentation test failed"
+    err=1
+  fi
+}
+
+trace_test_buffer() {
+  echo "Testing perf trace's buffer augmentation"
+  # echo will insert a newline (\10) at the end of the buffer
+  if ! perf trace -e write --max-events=1 -- echo "${buffer}" 2>&1 | \
+    grep -q -E "^echo/[0-9]+ write\([0-9]+, ${buffer}.*, [0-9]+\) += +[0-9]+$"
+  then
+    echo "Buffer augmentation test failed"
+    err=1
+  fi
+}
+
+trace_test_struct_btf() {
+  echo "Testing perf trace's struct augmentation"
+  if ! perf trace -e clock_nanosleep --force-btf --max-events=1 -- sleep 1 2>&1 | \
+    grep -q -E "^sleep/[0-9]+ clock_nanosleep\(0, 0, \{1,\}, 0x[0-9a-f]+\) += +[0-9]+$"
+  then
+    echo "BTF struct augmentation test failed"
+    err=1
+  fi
+}
+
+cleanup() {
+  rm -rf ${file1} ${file2} ${perf_config_tmp}
+}
+
+trap_cleanup() {
+  echo "Unexpected signal in ${FUNCNAME[1]}"
+  cleanup
+  exit 1
+}
+
+# don't overwrite user's perf config
+trace_config() {
+  export PERF_CONFIG=${perf_config_tmp}
+  perf config trace.show_arg_names=false trace.show_duration=false \
+    trace.show_timestamp=false trace.args_alignment=0
+}
+
+skip_if_no_perf_trace || exit 2
+check_vmlinux || exit 2
+
+trace_config
+
+trace_test_string
+
+if [ $err = 0 ]; then
+  trace_test_buffer
+fi
+
+if [ $err = 0 ]; then
+  trace_test_struct_btf
+fi
+
+cleanup
+
+exit $err
-- 
2.43.0

[PATCH v4 2/2] perf docs: Add documentation for --force-btf option

[Howard Chu]

The --force-btf option is intended for debugging purposes and is
currently undocumented. Add documentation for it.

Signed-off-by: Howard Chu <howardchu95@gmail.com>
---
 tools/perf/Documentation/perf-trace.txt | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tools/perf/Documentation/perf-trace.txt b/tools/perf/Documentation/perf-trace.txt
index 6e0cc50bbc13..fb3d2af33844 100644
--- a/tools/perf/Documentation/perf-trace.txt
+++ b/tools/perf/Documentation/perf-trace.txt
@@ -241,6 +241,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs.
 	printing using the existing 'perf trace' syscall arg beautifiers to map integer
 	arguments to strings (pid to comm, syscall id to syscall name, etc).
 
+--force-btf::
+	Use btf_dump to pretty print syscall argument data, instead of using hand-crafted pretty
+	printers. This option is intended for testing BTF integration in perf trace. btf_dump-based
+	pretty-printing serves as a fallback to hand-crafted pretty printers, as the latter can
+	better pretty-print integer flags and struct pointers.
 
 PAGEFAULTS
 ----------
-- 
2.43.0

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Namhyung Kim]

On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> Changes in v4:
>  - Use if -f to check the existence of vmlinux BTF, and exit if it
>    doesn't, so trace_test_string will not overwrite $err, and keep
>    running the test.
> 
> Changes in v3:
>  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> 
> v1, v2:
> 
> The previous version of the perf trace BTF general augmentation tests
> didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> <acme@kernel.org> for pointing this out), this version uses bash instead
> of POSIX shell to pass Shellcheck.
> 
> This patch series also adds documentation for the new option
> --force-btf, which is used in the tests.
> 
> Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> 
> Howard Chu (2):
>   perf trace: Add tests for BTF general augmentation
>   perf docs: Add documentation for --force-btf option

Acked-by: Namhyung Kim <namhyung@kernel.org>

Thanks,
Namhyung

> 
>  tools/perf/Documentation/perf-trace.txt     |  5 ++
>  tools/perf/tests/shell/trace_btf_general.sh | 93 +++++++++++++++++++++
>  2 files changed, 98 insertions(+)
>  create mode 100755 tools/perf/tests/shell/trace_btf_general.sh
> 
> -- 
> 2.43.0
> 

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Arnaldo Carvalho de Melo]

On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > Changes in v4:
> >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> >    doesn't, so trace_test_string will not overwrite $err, and keep
> >    running the test.
> > 
> > Changes in v3:
> >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > 
> > v1, v2:
> > 
> > The previous version of the perf trace BTF general augmentation tests
> > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > <acme@kernel.org> for pointing this out), this version uses bash instead
> > of POSIX shell to pass Shellcheck.
> > 
> > This patch series also adds documentation for the new option
> > --force-btf, which is used in the tests.
> > 
> > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > 
> > Howard Chu (2):
> >   perf trace: Add tests for BTF general augmentation
> >   perf docs: Add documentation for --force-btf option
> 
> Acked-by: Namhyung Kim <namhyung@kernel.org>

Thanks, applied to perf-tools-next,

- Arnaldo

Re: [PATCH v4 2/2] perf docs: Add documentation for --force-btf option

[Arnaldo Carvalho de Melo]

On Sun, Dec 15, 2024 at 11:07:11AM -0800, Howard Chu wrote:
> The --force-btf option is intended for debugging purposes and is
> currently undocumented. Add documentation for it.
> 
> Signed-off-by: Howard Chu <howardchu95@gmail.com>
> ---
>  tools/perf/Documentation/perf-trace.txt | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/tools/perf/Documentation/perf-trace.txt b/tools/perf/Documentation/perf-trace.txt
> index 6e0cc50bbc13..fb3d2af33844 100644
> --- a/tools/perf/Documentation/perf-trace.txt
> +++ b/tools/perf/Documentation/perf-trace.txt
> @@ -241,6 +241,11 @@ the thread executes on the designated CPUs. Default is to monitor all CPUs.
>  	printing using the existing 'perf trace' syscall arg beautifiers to map integer
>  	arguments to strings (pid to comm, syscall id to syscall name, etc).
>  
> +--force-btf::
> +	Use btf_dump to pretty print syscall argument data, instead of using hand-crafted pretty
> +	printers. This option is intended for testing BTF integration in perf trace. btf_dump-based
> +	pretty-printing serves as a fallback to hand-crafted pretty printers, as the latter can
> +	better pretty-print integer flags and struct pointers.

Applied, but for the casual reader the "btf_dump" reference is vague, it
would be interesting to mention that it is part of libbpf and uses
only what is available in the BTF format.

Then talking about the "hand-crafted" pretty printers, it is not all the
time "hand crafted", but automated by all those shell scripts that
convert kernel source code that doesn't get converted into BTF info
(defines) to create tables that then get associated with those syscall
args or even struct members.

But that is something for a followup paches, I'm applying it as is as it
improves the current documentation.

- Arnaldo

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Arnaldo Carvalho de Melo]

On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > Changes in v4:
> > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > >    running the test.
> > > 
> > > Changes in v3:
> > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > 
> > > v1, v2:
> > > 
> > > The previous version of the perf trace BTF general augmentation tests
> > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > of POSIX shell to pass Shellcheck.
> > > 
> > > This patch series also adds documentation for the new option
> > > --force-btf, which is used in the tests.
> > > 
> > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > 
> > > Howard Chu (2):
> > >   perf trace: Add tests for BTF general augmentation
> > >   perf docs: Add documentation for --force-btf option
> > 
> > Acked-by: Namhyung Kim <namhyung@kernel.org>
> 
> Thanks, applied to perf-tools-next,

It fails when running in parallel mode, sometimes:

109: perf trace BTF general tests                                    : FAILED!

Then:

root@number:~# perf stat --null -r 10 perf test "BTF general"
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : FAILED!
109: perf trace BTF general tests                                    : Ok
109: perf trace BTF general tests                                    : Ok

 Performance counter stats for 'perf test BTF general' (10 runs):

             2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )

root@number:~#


So its not just when running in paralell, anyway, its merged, we can go
on from what we got there.

- Arnaldo

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Howard Chu]

Hello Arnaldo,

On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > Changes in v4:
> > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > >    running the test.
> > > >
> > > > Changes in v3:
> > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > >
> > > > v1, v2:
> > > >
> > > > The previous version of the perf trace BTF general augmentation tests
> > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > of POSIX shell to pass Shellcheck.
> > > >
> > > > This patch series also adds documentation for the new option
> > > > --force-btf, which is used in the tests.
> > > >
> > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > >
> > > > Howard Chu (2):
> > > >   perf trace: Add tests for BTF general augmentation
> > > >   perf docs: Add documentation for --force-btf option
> > >
> > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> >
> > Thanks, applied to perf-tools-next,
>
> It fails when running in parallel mode, sometimes:
>
> 109: perf trace BTF general tests                                    : FAILED!
>
> Then:
>
> root@number:~# perf stat --null -r 10 perf test "BTF general"
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : FAILED!
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : FAILED!
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
>
>  Performance counter stats for 'perf test BTF general' (10 runs):
>
>              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
>
> root@number:~#
>
>
> So its not just when running in paralell, anyway, its merged, we can go
> on from what we got there.

Thanks, I will resolve this issue and send v5 along with the
documentation changes you suggested.

Thanks,
Howard
>
> - Arnaldo

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Arnaldo Carvalho de Melo]

On Mon, Dec 23, 2024 at 06:03:11PM -0800, Howard Chu wrote:
> On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > So its not just when running in paralell, anyway, its merged, we can go
> > on from what we got there.
 
> Thanks, I will resolve this issue and send v5 along with the
> documentation changes you suggested.

I got v4 merged meanwhile, we can go on from there, with further patches
polishihg those aspects.

Thanks,

- Arnaldo

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Howard Chu]

Hello Arnaldo,

On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
<acme@kernel.org> wrote:
>
> On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > Changes in v4:
> > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > >    running the test.
> > > >
> > > > Changes in v3:
> > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > >
> > > > v1, v2:
> > > >
> > > > The previous version of the perf trace BTF general augmentation tests
> > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > of POSIX shell to pass Shellcheck.
> > > >
> > > > This patch series also adds documentation for the new option
> > > > --force-btf, which is used in the tests.
> > > >
> > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > >
> > > > Howard Chu (2):
> > > >   perf trace: Add tests for BTF general augmentation
> > > >   perf docs: Add documentation for --force-btf option
> > >
> > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> >
> > Thanks, applied to perf-tools-next,
>
> It fails when running in parallel mode, sometimes:
>
> 109: perf trace BTF general tests                                    : FAILED!
>
> Then:
>
> root@number:~# perf stat --null -r 10 perf test "BTF general"
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : FAILED!
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : FAILED!
> 109: perf trace BTF general tests                                    : Ok
> 109: perf trace BTF general tests                                    : Ok
>
>  Performance counter stats for 'perf test BTF general' (10 runs):
>
>              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
>
> root@number:~#
>
>
> So its not just when running in paralell, anyway, its merged, we can go
> on from what we got there.

It never fails on my machine, I think the reason is my machine is not
fully-loaded. Can you please run
```
perf stat --null -r 10 perf test -vv "BTF general"
```
And provide the output?

Thanks,
Howard
>
> - Arnaldo

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Arnaldo Carvalho de Melo]

On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> Hello Arnaldo,
> 
> On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> <acme@kernel.org> wrote:
> >
> > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > Changes in v4:
> > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > >    running the test.
> > > > >
> > > > > Changes in v3:
> > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > >
> > > > > v1, v2:
> > > > >
> > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > of POSIX shell to pass Shellcheck.
> > > > >
> > > > > This patch series also adds documentation for the new option
> > > > > --force-btf, which is used in the tests.
> > > > >
> > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > >
> > > > > Howard Chu (2):
> > > > >   perf trace: Add tests for BTF general augmentation
> > > > >   perf docs: Add documentation for --force-btf option
> > > >
> > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > >
> > > Thanks, applied to perf-tools-next,
> >
> > It fails when running in parallel mode, sometimes:
> >
> > 109: perf trace BTF general tests                                    : FAILED!
> >
> > Then:
> >
> > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : FAILED!
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : FAILED!
> > 109: perf trace BTF general tests                                    : Ok
> > 109: perf trace BTF general tests                                    : Ok
> >
> >  Performance counter stats for 'perf test BTF general' (10 runs):
> >
> >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> >
> > root@number:~#
> >
> >
> > So its not just when running in paralell, anyway, its merged, we can go
> > on from what we got there.
> 
> It never fails on my machine, I think the reason is my machine is not
> fully-loaded. Can you please run
> ```
> perf stat --null -r 10 perf test -vv "BTF general"
> ```
> And provide the output?

root@number:~# grep -m1 'model name' /proc/cpuinfo 
model name	: Intel(R) Core(TM) i7-14700K
root@number:~#

root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47824
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47842
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47860
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47878
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47896
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Buffer augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47917
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Buffer augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47938
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Testing perf trace's struct augmentation
---- end(0) ----
106: perf trace BTF general tests                                    : Ok
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47962
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!
106: perf trace BTF general tests:
--- start ---
test child forked, pid 47980
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Testing perf trace's struct augmentation
---- end(0) ----
106: perf trace BTF general tests                                    : Ok
106: perf trace BTF general tests:
--- start ---
test child forked, pid 48004
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
Testing perf trace's buffer augmentation
Testing perf trace's struct augmentation
---- end(0) ----
106: perf trace BTF general tests                                    : Ok

 Performance counter stats for 'perf test -vv BTF general' (10 runs):

             1.166 +- 0.272 seconds time elapsed  ( +- 23.34% )

root@number:~#

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Namhyung Kim]

On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > Hello Arnaldo,
> > 
> > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > <acme@kernel.org> wrote:
> > >
> > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > Changes in v4:
> > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > >    running the test.
> > > > > >
> > > > > > Changes in v3:
> > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > >
> > > > > > v1, v2:
> > > > > >
> > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > > of POSIX shell to pass Shellcheck.
> > > > > >
> > > > > > This patch series also adds documentation for the new option
> > > > > > --force-btf, which is used in the tests.
> > > > > >
> > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > >
> > > > > > Howard Chu (2):
> > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > >   perf docs: Add documentation for --force-btf option
> > > > >
> > > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > > >
> > > > Thanks, applied to perf-tools-next,
> > >
> > > It fails when running in parallel mode, sometimes:
> > >
> > > 109: perf trace BTF general tests                                    : FAILED!
> > >
> > > Then:
> > >
> > > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : FAILED!
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : FAILED!
> > > 109: perf trace BTF general tests                                    : Ok
> > > 109: perf trace BTF general tests                                    : Ok
> > >
> > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > >
> > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > >
> > > root@number:~#
> > >
> > >
> > > So its not just when running in paralell, anyway, its merged, we can go
> > > on from what we got there.
> > 
> > It never fails on my machine, I think the reason is my machine is not
> > fully-loaded. Can you please run
> > ```
> > perf stat --null -r 10 perf test -vv "BTF general"
> > ```
> > And provide the output?
> 
> root@number:~# grep -m1 'model name' /proc/cpuinfo 
> model name	: Intel(R) Core(TM) i7-14700K
> root@number:~#
> 
> root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
> 106: perf trace BTF general tests:
> --- start ---
> test child forked, pid 47824
> Checking if vmlinux BTF exists
> Testing perf trace's string augmentation
> String augmentation test failed
> ---- end(-1) ----
> 106: perf trace BTF general tests                                    : FAILED!
> 106: perf trace BTF general tests:
> --- start ---
> test child forked, pid 47842
> Checking if vmlinux BTF exists
> Testing perf trace's string augmentation
> String augmentation test failed
> ---- end(-1) ----
> 106: perf trace BTF general tests                                    : FAILED!
> ...

I also see the failure on not fully-loaded machines (both Intel and
AMD).

  $ sudo ./perf test -v BTF
  --- start ---
  test child forked, pid 525025
  Checking if vmlinux BTF exists
  Testing perf trace's string augmentation
  String augmentation test failed
  ---- end(-1) ----
  106: perf trace BTF general tests                                    : FAILED!

Thanks,
Namhyung

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Howard Chu]

Hello Namhyung,

On Sat, Jan 18, 2025 at 10:28 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > > Hello Arnaldo,
> > >
> > > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > > <acme@kernel.org> wrote:
> > > >
> > > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > > Changes in v4:
> > > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > > >    running the test.
> > > > > > >
> > > > > > > Changes in v3:
> > > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > > >
> > > > > > > v1, v2:
> > > > > > >
> > > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > > > of POSIX shell to pass Shellcheck.
> > > > > > >
> > > > > > > This patch series also adds documentation for the new option
> > > > > > > --force-btf, which is used in the tests.
> > > > > > >
> > > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > > >
> > > > > > > Howard Chu (2):
> > > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > > >   perf docs: Add documentation for --force-btf option
> > > > > >
> > > > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > > > >
> > > > > Thanks, applied to perf-tools-next,
> > > >
> > > > It fails when running in parallel mode, sometimes:
> > > >
> > > > 109: perf trace BTF general tests                                    : FAILED!
> > > >
> > > > Then:
> > > >
> > > > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > 109: perf trace BTF general tests                                    : Ok
> > > > 109: perf trace BTF general tests                                    : Ok
> > > >
> > > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > > >
> > > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > > >
> > > > root@number:~#
> > > >
> > > >
> > > > So its not just when running in paralell, anyway, its merged, we can go
> > > > on from what we got there.
> > >
> > > It never fails on my machine, I think the reason is my machine is not
> > > fully-loaded. Can you please run
> > > ```
> > > perf stat --null -r 10 perf test -vv "BTF general"
> > > ```
> > > And provide the output?
> >
> > root@number:~# grep -m1 'model name' /proc/cpuinfo
> > model name    : Intel(R) Core(TM) i7-14700K
> > root@number:~#
> >
> > root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
> > 106: perf trace BTF general tests:
> > --- start ---
> > test child forked, pid 47824
> > Checking if vmlinux BTF exists
> > Testing perf trace's string augmentation
> > String augmentation test failed
> > ---- end(-1) ----
> > 106: perf trace BTF general tests                                    : FAILED!
> > 106: perf trace BTF general tests:
> > --- start ---
> > test child forked, pid 47842
> > Checking if vmlinux BTF exists
> > Testing perf trace's string augmentation
> > String augmentation test failed
> > ---- end(-1) ----
> > 106: perf trace BTF general tests                                    : FAILED!
> > ...
>
> I also see the failure on not fully-loaded machines (both Intel and
> AMD).
>
>   $ sudo ./perf test -v BTF
>   --- start ---
>   test child forked, pid 525025
>   Checking if vmlinux BTF exists
>   Testing perf trace's string augmentation
>   String augmentation test failed
>   ---- end(-1) ----
>   106: perf trace BTF general tests                                    : FAILED!
>
> Thanks,
> Namhyung
>

Just wondering, did you and Arnaldo apply this patch:
https://lore.kernel.org/linux-perf-users/20241213023047.541218-1-howardchu95@gmail.com/

This seems to solve the problem for me...

Thanks,
Howard

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Namhyung Kim]

On Tue, Jan 21, 2025 at 04:19:35PM -0800, Howard Chu wrote:
> Hello Namhyung,
> 
> On Sat, Jan 18, 2025 at 10:28 AM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> > > On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > > > Hello Arnaldo,
> > > >
> > > > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > > > <acme@kernel.org> wrote:
> > > > >
> > > > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > > > Changes in v4:
> > > > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > > > >    running the test.
> > > > > > > >
> > > > > > > > Changes in v3:
> > > > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > > > >
> > > > > > > > v1, v2:
> > > > > > > >
> > > > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > > > > of POSIX shell to pass Shellcheck.
> > > > > > > >
> > > > > > > > This patch series also adds documentation for the new option
> > > > > > > > --force-btf, which is used in the tests.
> > > > > > > >
> > > > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > > > >
> > > > > > > > Howard Chu (2):
> > > > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > > > >   perf docs: Add documentation for --force-btf option
> > > > > > >
> > > > > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > > > > >
> > > > > > Thanks, applied to perf-tools-next,
> > > > >
> > > > > It fails when running in parallel mode, sometimes:
> > > > >
> > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > >
> > > > > Then:
> > > > >
> > > > > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > 109: perf trace BTF general tests                                    : Ok
> > > > >
> > > > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > > > >
> > > > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > > > >
> > > > > root@number:~#
> > > > >
> > > > >
> > > > > So its not just when running in paralell, anyway, its merged, we can go
> > > > > on from what we got there.
> > > >
> > > > It never fails on my machine, I think the reason is my machine is not
> > > > fully-loaded. Can you please run
> > > > ```
> > > > perf stat --null -r 10 perf test -vv "BTF general"
> > > > ```
> > > > And provide the output?
> > >
> > > root@number:~# grep -m1 'model name' /proc/cpuinfo
> > > model name    : Intel(R) Core(TM) i7-14700K
> > > root@number:~#
> > >
> > > root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
> > > 106: perf trace BTF general tests:
> > > --- start ---
> > > test child forked, pid 47824
> > > Checking if vmlinux BTF exists
> > > Testing perf trace's string augmentation
> > > String augmentation test failed
> > > ---- end(-1) ----
> > > 106: perf trace BTF general tests                                    : FAILED!
> > > 106: perf trace BTF general tests:
> > > --- start ---
> > > test child forked, pid 47842
> > > Checking if vmlinux BTF exists
> > > Testing perf trace's string augmentation
> > > String augmentation test failed
> > > ---- end(-1) ----
> > > 106: perf trace BTF general tests                                    : FAILED!
> > > ...
> >
> > I also see the failure on not fully-loaded machines (both Intel and
> > AMD).
> >
> >   $ sudo ./perf test -v BTF
> >   --- start ---
> >   test child forked, pid 525025
> >   Checking if vmlinux BTF exists
> >   Testing perf trace's string augmentation
> >   String augmentation test failed
> >   ---- end(-1) ----
> >   106: perf trace BTF general tests                                    : FAILED!
> >
> > Thanks,
> > Namhyung
> >
> 
> Just wondering, did you and Arnaldo apply this patch:
> https://lore.kernel.org/linux-perf-users/20241213023047.541218-1-howardchu95@gmail.com/

I don't think we applied it, will add to perf-tools.

> 
> This seems to solve the problem for me...

But I'm not sure if it fixes the problem.  IIUC the patch fixes the
loading problem.  And for some reason, I don't see the problem any more.
But I remember this sometimes happened in the past.

Thanks,
Namhyung

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Howard Chu]

Hello Namhyung,

Sorry for somehow missing this reply...

On Thu, Jan 23, 2025 at 3:53 PM Namhyung Kim <namhyung@kernel.org> wrote:
>
> On Tue, Jan 21, 2025 at 04:19:35PM -0800, Howard Chu wrote:
> > Hello Namhyung,
> >
> > On Sat, Jan 18, 2025 at 10:28 AM Namhyung Kim <namhyung@kernel.org> wrote:
> > >
> > > On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > > > > Hello Arnaldo,
> > > > >
> > > > > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > > > > <acme@kernel.org> wrote:
> > > > > >
> > > > > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > > > > Changes in v4:
> > > > > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > > > > >    running the test.
> > > > > > > > >
> > > > > > > > > Changes in v3:
> > > > > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > > > > >
> > > > > > > > > v1, v2:
> > > > > > > > >
> > > > > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > > > > > of POSIX shell to pass Shellcheck.
> > > > > > > > >
> > > > > > > > > This patch series also adds documentation for the new option
> > > > > > > > > --force-btf, which is used in the tests.
> > > > > > > > >
> > > > > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > > > > >
> > > > > > > > > Howard Chu (2):
> > > > > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > > > > >   perf docs: Add documentation for --force-btf option
> > > > > > > >
> > > > > > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > > > > > >
> > > > > > > Thanks, applied to perf-tools-next,
> > > > > >
> > > > > > It fails when running in parallel mode, sometimes:
> > > > > >
> > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > >
> > > > > > Then:
> > > > > >
> > > > > > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > >
> > > > > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > > > > >
> > > > > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > > > > >
> > > > > > root@number:~#
> > > > > >
> > > > > >
> > > > > > So its not just when running in paralell, anyway, its merged, we can go
> > > > > > on from what we got there.
> > > > >
> > > > > It never fails on my machine, I think the reason is my machine is not
> > > > > fully-loaded. Can you please run
> > > > > ```
> > > > > perf stat --null -r 10 perf test -vv "BTF general"
> > > > > ```
> > > > > And provide the output?
> > > >
> > > > root@number:~# grep -m1 'model name' /proc/cpuinfo
> > > > model name    : Intel(R) Core(TM) i7-14700K
> > > > root@number:~#
> > > >
> > > > root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
> > > > 106: perf trace BTF general tests:
> > > > --- start ---
> > > > test child forked, pid 47824
> > > > Checking if vmlinux BTF exists
> > > > Testing perf trace's string augmentation
> > > > String augmentation test failed
> > > > ---- end(-1) ----
> > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > 106: perf trace BTF general tests:
> > > > --- start ---
> > > > test child forked, pid 47842
> > > > Checking if vmlinux BTF exists
> > > > Testing perf trace's string augmentation
> > > > String augmentation test failed
> > > > ---- end(-1) ----
> > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > ...
> > >
> > > I also see the failure on not fully-loaded machines (both Intel and
> > > AMD).
> > >
> > >   $ sudo ./perf test -v BTF
> > >   --- start ---
> > >   test child forked, pid 525025
> > >   Checking if vmlinux BTF exists
> > >   Testing perf trace's string augmentation
> > >   String augmentation test failed
> > >   ---- end(-1) ----
> > >   106: perf trace BTF general tests                                    : FAILED!
> > >
> > > Thanks,
> > > Namhyung
> > >
> >
> > Just wondering, did you and Arnaldo apply this patch:
> > https://lore.kernel.org/linux-perf-users/20241213023047.541218-1-howardchu95@gmail.com/
>
> I don't think we applied it, will add to perf-tools.
>
> >
> > This seems to solve the problem for me...
>
> But I'm not sure if it fixes the problem.  IIUC the patch fixes the
> loading problem.

Yes. However, if the machine is unable to load the BPF program, the
error message will be:

perf $ sudo ./perf test general -v
--- start ---
test child forked, pid 173990
Checking if vmlinux BTF exists
Testing perf trace's string augmentation
String augmentation test failed
---- end(-1) ----
106: perf trace BTF general tests                                    : FAILED!

Which is indistinguishable from an actual string augmentation error
that has a string mismatch...

> And for some reason, I don't see the problem any more.
> But I remember this sometimes happened in the past.

Do you mean that perf trace breaks occasionally? This sounds urgent.
Could you please provide me with your kernel version and Clang
version? Thank you very much! :)

Thanks,
Howard
>
> Thanks,
> Namhyung
>

Re: [PATCH v4 0/2] perf trace: Add more tests for BTF-augmented perf trace

[Namhyung Kim]

On Fri, Jan 31, 2025 at 04:14:09PM -0800, Howard Chu wrote:
> Hello Namhyung,
> 
> Sorry for somehow missing this reply...
> 
> On Thu, Jan 23, 2025 at 3:53 PM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > On Tue, Jan 21, 2025 at 04:19:35PM -0800, Howard Chu wrote:
> > > Hello Namhyung,
> > >
> > > On Sat, Jan 18, 2025 at 10:28 AM Namhyung Kim <namhyung@kernel.org> wrote:
> > > >
> > > > On Mon, Jan 13, 2025 at 12:57:23PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > On Fri, Jan 10, 2025 at 04:52:08PM -0800, Howard Chu wrote:
> > > > > > Hello Arnaldo,
> > > > > >
> > > > > > On Mon, Dec 23, 2024 at 11:40 AM Arnaldo Carvalho de Melo
> > > > > > <acme@kernel.org> wrote:
> > > > > > >
> > > > > > > On Mon, Dec 23, 2024 at 04:02:43PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > > On Mon, Dec 16, 2024 at 11:58:46AM -0800, Namhyung Kim wrote:
> > > > > > > > > On Sun, Dec 15, 2024 at 11:07:09AM -0800, Howard Chu wrote:
> > > > > > > > > > Changes in v4:
> > > > > > > > > >  - Use if -f to check the existence of vmlinux BTF, and exit if it
> > > > > > > > > >    doesn't, so trace_test_string will not overwrite $err, and keep
> > > > > > > > > >    running the test.
> > > > > > > > > >
> > > > > > > > > > Changes in v3:
> > > > > > > > > >  - Add vmlinux BTF check, and skip the tests if it doesn't exist
> > > > > > > > > >
> > > > > > > > > > v1, v2:
> > > > > > > > > >
> > > > > > > > > > The previous version of the perf trace BTF general augmentation tests
> > > > > > > > > > didn't pass Shellcheck (thanks to Arnaldo Carvalho de Melo
> > > > > > > > > > <acme@kernel.org> for pointing this out), this version uses bash instead
> > > > > > > > > > of POSIX shell to pass Shellcheck.
> > > > > > > > > >
> > > > > > > > > > This patch series also adds documentation for the new option
> > > > > > > > > > --force-btf, which is used in the tests.
> > > > > > > > > >
> > > > > > > > > > Link: https://lore.kernel.org/linux-perf-users/Zt9yiQq-n-W6I274@x1/
> > > > > > > > > >
> > > > > > > > > > Howard Chu (2):
> > > > > > > > > >   perf trace: Add tests for BTF general augmentation
> > > > > > > > > >   perf docs: Add documentation for --force-btf option
> > > > > > > > >
> > > > > > > > > Acked-by: Namhyung Kim <namhyung@kernel.org>
> > > > > > > >
> > > > > > > > Thanks, applied to perf-tools-next,
> > > > > > >
> > > > > > > It fails when running in parallel mode, sometimes:
> > > > > > >
> > > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > >
> > > > > > > Then:
> > > > > > >
> > > > > > > root@number:~# perf stat --null -r 10 perf test "BTF general"
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : FAILED!
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > > 109: perf trace BTF general tests                                    : Ok
> > > > > > >
> > > > > > >  Performance counter stats for 'perf test BTF general' (10 runs):
> > > > > > >
> > > > > > >              2.148 +- 0.293 seconds time elapsed  ( +- 13.63% )
> > > > > > >
> > > > > > > root@number:~#
> > > > > > >
> > > > > > >
> > > > > > > So its not just when running in paralell, anyway, its merged, we can go
> > > > > > > on from what we got there.
> > > > > >
> > > > > > It never fails on my machine, I think the reason is my machine is not
> > > > > > fully-loaded. Can you please run
> > > > > > ```
> > > > > > perf stat --null -r 10 perf test -vv "BTF general"
> > > > > > ```
> > > > > > And provide the output?
> > > > >
> > > > > root@number:~# grep -m1 'model name' /proc/cpuinfo
> > > > > model name    : Intel(R) Core(TM) i7-14700K
> > > > > root@number:~#
> > > > >
> > > > > root@number:~# perf stat --null -r 10 perf test -vv "BTF general"
> > > > > 106: perf trace BTF general tests:
> > > > > --- start ---
> > > > > test child forked, pid 47824
> > > > > Checking if vmlinux BTF exists
> > > > > Testing perf trace's string augmentation
> > > > > String augmentation test failed
> > > > > ---- end(-1) ----
> > > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > > 106: perf trace BTF general tests:
> > > > > --- start ---
> > > > > test child forked, pid 47842
> > > > > Checking if vmlinux BTF exists
> > > > > Testing perf trace's string augmentation
> > > > > String augmentation test failed
> > > > > ---- end(-1) ----
> > > > > 106: perf trace BTF general tests                                    : FAILED!
> > > > > ...
> > > >
> > > > I also see the failure on not fully-loaded machines (both Intel and
> > > > AMD).
> > > >
> > > >   $ sudo ./perf test -v BTF
> > > >   --- start ---
> > > >   test child forked, pid 525025
> > > >   Checking if vmlinux BTF exists
> > > >   Testing perf trace's string augmentation
> > > >   String augmentation test failed
> > > >   ---- end(-1) ----
> > > >   106: perf trace BTF general tests                                    : FAILED!
> > > >
> > > > Thanks,
> > > > Namhyung
> > > >
> > >
> > > Just wondering, did you and Arnaldo apply this patch:
> > > https://lore.kernel.org/linux-perf-users/20241213023047.541218-1-howardchu95@gmail.com/
> >
> > I don't think we applied it, will add to perf-tools.
> >
> > >
> > > This seems to solve the problem for me...
> >
> > But I'm not sure if it fixes the problem.  IIUC the patch fixes the
> > loading problem.
> 
> Yes. However, if the machine is unable to load the BPF program, the
> error message will be:
> 
> perf $ sudo ./perf test general -v
> --- start ---
> test child forked, pid 173990
> Checking if vmlinux BTF exists
> Testing perf trace's string augmentation
> String augmentation test failed
> ---- end(-1) ----
> 106: perf trace BTF general tests                                    : FAILED!
> 
> Which is indistinguishable from an actual string augmentation error
> that has a string mismatch...

Ok, that's not good.

> 
> > And for some reason, I don't see the problem any more.
> > But I remember this sometimes happened in the past.
> 
> Do you mean that perf trace breaks occasionally? This sounds urgent.
> Could you please provide me with your kernel version and Clang
> version? Thank you very much! :)

Yep, sometimes it works and sometimes it failed to load the BPF.

The kernel version of this laptop is v6.10.11 and clang version is
16.0.6.

Thanks,
Namhyung