[PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Arnaldo Carvalho de Melo]

Howard reported problems using perf features that use BPF:

  perf $ clang -v
  Debian clang version 15.0.6
  Target: x86_64-pc-linux-gnu
  Thread model: posix
  InstalledDir: /bin
  Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
  Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
  Candidate multilib: .;@m64
  Selected multilib: .;@m64
  perf $ ./perf trace -e write --max-events=1
  libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
  libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
  0: R1=ctx() R10=fp0

But it works with:

  perf $ clang -v
  Debian clang version 16.0.6 (15~deb12u1)
  Target: x86_64-pc-linux-gnu
  Thread model: posix
  InstalledDir: /bin
  Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
  Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
  Candidate multilib: .;@m64
  Selected multilib: .;@m64
  perf $ ./perf trace -e write --max-events=1
       0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
  perf $

So lets make that the required version, if you happen to have a slightly
older version where this work, please report so that we can adjust the
minimum required version.

Reported-by: Howard Chu <howardchu95@gmail.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alan Maguire <alan.maguire@oracle.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
---
 tools/perf/Makefile.config | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/perf/Makefile.config b/tools/perf/Makefile.config
index 37e3eee2986e580a..4c1f75e326e2c1d2 100644
--- a/tools/perf/Makefile.config
+++ b/tools/perf/Makefile.config
@@ -705,8 +705,8 @@ ifeq ($(BUILD_BPF_SKEL),1)
     BUILD_BPF_SKEL := 0
   else
     CLANG_VERSION := $(shell $(CLANG) --version | head -1 | sed 's/.*clang version \([[:digit:]]\+.[[:digit:]]\+.[[:digit:]]\+\).*/\1/g')
-    ifeq ($(call version-lt3,$(CLANG_VERSION),12.0.1),1)
-      $(warning Warning: Disabled BPF skeletons as reliable BTF generation needs at least $(CLANG) version 12.0.1)
+    ifeq ($(call version-lt3,$(CLANG_VERSION),16.0.6),1)
+      $(warning Warning: Disabled BPF skeletons as at least $(CLANG) version 16.0.6 is reported to be a working setup with the current of BPF based perf features)
       BUILD_BPF_SKEL := 0
     endif
   endif
-- 
2.46.0

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[James Clark]

On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
> Howard reported problems using perf features that use BPF:
> 
>    perf $ clang -v
>    Debian clang version 15.0.6
>    Target: x86_64-pc-linux-gnu
>    Thread model: posix
>    InstalledDir: /bin
>    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>    Candidate multilib: .;@m64
>    Selected multilib: .;@m64
>    perf $ ./perf trace -e write --max-events=1
>    libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
>    libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
>    0: R1=ctx() R10=fp0
> 
> But it works with:
> 
>    perf $ clang -v
>    Debian clang version 16.0.6 (15~deb12u1)
>    Target: x86_64-pc-linux-gnu
>    Thread model: posix
>    InstalledDir: /bin
>    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>    Candidate multilib: .;@m64
>    Selected multilib: .;@m64
>    perf $ ./perf trace -e write --max-events=1
>         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
>    perf $
> 
> So lets make that the required version, if you happen to have a slightly
> older version where this work, please report so that we can adjust the
> minimum required version.
> 

I wasn't able to reproduce the issue with either of these versions. But 
I suppose it could be an issue with only 15.0.6.

$ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64


Ubuntu clang version 14.0.0-1ubuntu1.1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Arnaldo Carvalho de Melo]

On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
> > Howard reported problems using perf features that use BPF:

> >    perf $ clang -v
> >    Debian clang version 15.0.6
> >    Target: x86_64-pc-linux-gnu
> >    Thread model: posix
> >    InstalledDir: /bin
> >    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >    Candidate multilib: .;@m64
> >    Selected multilib: .;@m64
> >    perf $ ./perf trace -e write --max-events=1
> >    libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> >    libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> >    0: R1=ctx() R10=fp0
> > 
> > But it works with:
> > 
> >    perf $ clang -v
> >    Debian clang version 16.0.6 (15~deb12u1)
> >    Target: x86_64-pc-linux-gnu
> >    Thread model: posix
> >    InstalledDir: /bin
> >    Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >    Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >    Candidate multilib: .;@m64
> >    Selected multilib: .;@m64
> >    perf $ ./perf trace -e write --max-events=1
> >         0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
> >    perf $
> > 
> > So lets make that the required version, if you happen to have a slightly
> > older version where this work, please report so that we can adjust the
> > minimum required version.
 
> I wasn't able to reproduce the issue with either of these versions. But I
> suppose it could be an issue with only 15.0.6.

Interesting, that complicates things, probably the best way then is to
try to build it, if it fails, mention that 15.0.6 is known to be
problematic and suggest working versions?

- Arnaldo
 
> $ clang -v
> Ubuntu clang version 15.0.7
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
> Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
> Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
> Candidate multilib: .;@m64
> Selected multilib: .;@m64
> 
> 
> Ubuntu clang version 14.0.0-1ubuntu1.1
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/bin
> Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
> Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
> Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
> Candidate multilib: .;@m64
> Selected multilib: .;@m64

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[James Clark]

On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
> On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
>> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
>>> Howard reported problems using perf features that use BPF:
> 
>>>     perf $ clang -v
>>>     Debian clang version 15.0.6
>>>     Target: x86_64-pc-linux-gnu
>>>     Thread model: posix
>>>     InstalledDir: /bin
>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Candidate multilib: .;@m64
>>>     Selected multilib: .;@m64
>>>     perf $ ./perf trace -e write --max-events=1
>>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
>>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
>>>     0: R1=ctx() R10=fp0
>>>
>>> But it works with:
>>>
>>>     perf $ clang -v
>>>     Debian clang version 16.0.6 (15~deb12u1)
>>>     Target: x86_64-pc-linux-gnu
>>>     Thread model: posix
>>>     InstalledDir: /bin
>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Candidate multilib: .;@m64
>>>     Selected multilib: .;@m64
>>>     perf $ ./perf trace -e write --max-events=1
>>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
>>>     perf $
>>>
>>> So lets make that the required version, if you happen to have a slightly
>>> older version where this work, please report so that we can adjust the
>>> minimum required version.
>   
>> I wasn't able to reproduce the issue with either of these versions. But I
>> suppose it could be an issue with only 15.0.6.
> 
> Interesting, that complicates things, probably the best way then is to
> try to build it, if it fails, mention that 15.0.6 is known to be
> problematic and suggest working versions?
> 
> - Arnaldo
>   

I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so 
making 16 the minimum could be an issue.

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[James Clark]

On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
> On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
>> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
>>> Howard reported problems using perf features that use BPF:
> 
>>>     perf $ clang -v
>>>     Debian clang version 15.0.6
>>>     Target: x86_64-pc-linux-gnu
>>>     Thread model: posix
>>>     InstalledDir: /bin
>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Candidate multilib: .;@m64
>>>     Selected multilib: .;@m64
>>>     perf $ ./perf trace -e write --max-events=1
>>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
>>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
>>>     0: R1=ctx() R10=fp0
>>>
>>> But it works with:
>>>
>>>     perf $ clang -v
>>>     Debian clang version 16.0.6 (15~deb12u1)
>>>     Target: x86_64-pc-linux-gnu
>>>     Thread model: posix
>>>     InstalledDir: /bin
>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>     Candidate multilib: .;@m64
>>>     Selected multilib: .;@m64
>>>     perf $ ./perf trace -e write --max-events=1
>>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
>>>     perf $
>>>
>>> So lets make that the required version, if you happen to have a slightly
>>> older version where this work, please report so that we can adjust the
>>> minimum required version.
>   
>> I wasn't able to reproduce the issue with either of these versions. But I
>> suppose it could be an issue with only 15.0.6.
> 
> Interesting, that complicates things, probably the best way then is to
> try to build it, if it fails, mention that 15.0.6 is known to be
> problematic and suggest working versions?
> 
> - Arnaldo

I still wasn't able to reproduce it with 15.0.6. And I double checked 
with V=1 that the build was using the right clang. I suppose it could be 
a build configuration issue, or maybe with a different kernel version?

$  uname --kernel-release
6.8.0-76060800daily20240311-generic

$ ../../llvm-project/build/bin/clang -v
clang version 15.0.6 (https://github.com/llvm/llvm-project.git 
088f33605d8a61ff519c580a71b1dd57d16a03f8)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64

$ git log
commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796 
(perf-tools-next/perf-tools-next)

$ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
   tools/perf

$ perf trace -e write --max-events=1

      0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
        0x7fffa102d9b0, count: 8)                           = 8

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Howard Chu]

Hello James and Arnaldo,

On Fri, Sep 13, 2024 at 3:42 AM Arnaldo Carvalho de Melo
<arnaldo.melo@gmail.com> wrote:
>
>
>
> On Fri, Sep 13, 2024, 7:20 AM James Clark <james.clark@linaro.org> wrote:
>>
>>
>>
>> On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
>> > On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
>> >> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
>> >>> Howard reported problems using perf features that use BPF:
>> >
>> >>>     perf $ clang -v
>> >>>     Debian clang version 15.0.6
>> >>>     Target: x86_64-pc-linux-gnu
>> >>>     Thread model: posix
>> >>>     InstalledDir: /bin
>> >>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>> >>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>> >>>     Candidate multilib: .;@m64
>> >>>     Selected multilib: .;@m64
>> >>>     perf $ ./perf trace -e write --max-events=1
>> >>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
>> >>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
>> >>>     0: R1=ctx() R10=fp0
>> >>>
>> >>> But it works with:
>> >>>
>> >>>     perf $ clang -v
>> >>>     Debian clang version 16.0.6 (15~deb12u1)
>> >>>     Target: x86_64-pc-linux-gnu
>> >>>     Thread model: posix
>> >>>     InstalledDir: /bin
>> >>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>> >>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>> >>>     Candidate multilib: .;@m64
>> >>>     Selected multilib: .;@m64
>> >>>     perf $ ./perf trace -e write --max-events=1
>> >>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
>> >>>     perf $
>> >>>
>> >>> So lets make that the required version, if you happen to have a slightly
>> >>> older version where this work, please report so that we can adjust the
>> >>> minimum required version.
>> >
>> >> I wasn't able to reproduce the issue with either of these versions. But I
>> >> suppose it could be an issue with only 15.0.6.
>> >
>> > Interesting, that complicates things, probably the best way then is to
>> > try to build it, if it fails, mention that 15.0.6 is known to be
>> > problematic and suggest working versions?
>> >
>> > - Arnaldo
>>
>> I still wasn't able to reproduce it with 15.0.6. And I double checked
>> with V=1 that the build was using the right clang. I suppose it could be
>> a build configuration issue, or maybe with a different kernel version?
>
>
> Howard? If not reproducible we can revert it.
>
> - Arnaldo
>
>
>>
>> $  uname --kernel-release
>> 6.8.0-76060800daily20240311-generic
>>
>> $ ../../llvm-project/build/bin/clang -v
>> clang version 15.0.6 (https://github.com/llvm/llvm-project.git
>> 088f33605d8a61ff519c580a71b1dd57d16a03f8)
>> Target: x86_64-unknown-linux-gnu
>> Thread model: posix
>> InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
>> Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
>> Candidate multilib: .;@m64
>> Selected multilib: .;@m64
>>
>> $ git log
>> commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
>> (perf-tools-next/perf-tools-next)
>>
>> $ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
>>    tools/perf
>>
>> $ perf trace -e write --max-events=1
>>
>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
>>         0x7fffa102d9b0, count: 8)                           = 8

Thanks for the effort! But this output seems to be unaugmented? I
think you are testing the perf/perf-tools-next branch that's slightly
older (perf mem: Fix the wrong reference in parse_record_events()
003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't
have the new perf trace feature yet...

That's why the perf trace output looks like:

>> $ perf trace -e write --max-events=1
>>
>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
>>         0x7fffa102d9b0, count: 8)

With the new buffer augmentation it should look like this:

perf $ ./perf trace -e write --max-events=1
     0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
count: 8)                         =

The new perf trace can be found on
https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
perf-tools-next

perf $ git log
commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD ->
struct-aug-arnaldo-2, perf/perf-tools-next)

Also, if you want to test the clang, you may want to checkout to the
commit before the clang-16 guard (otherwise you can't test different
clang version).

perf $ git log --oneline
1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
trace: Mark the 'head' arg in the set_robust_list syscall as coming
from user space
0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as
coming from user space
edf3ce0ed38e perf env: Find correct branch counter info on hybrid
9953807c9e01 perf evlist: Print hint for group
eb9b9a6f5ab3 tools: Drop nonsensical -O6
89c0a55e550e perf pmu: To info add event_type_desc
f08cc258431d perf evsel: Add accessor for tool_event
925320737ae2 perf pmus: Fake PMU clean up
d3d5c1a00fcd perf list: Avoid potential out of bounds memory read
4ae354d73a8e perf help: Fix a typo ("bellow")
74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system
83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '='
from probe line number regex
9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF
skeletons. **********[[[[[[[[[[[[!!!!(This is the clang
guard)]]]]]]]]]]*****
4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume
it is coming _from_ userspace
e37b315c17df perf parse-events: Remove duplicated include in parse-events.c

My machine is debian 12, and I cannot build perf trace with the new
augmentation feature on clang-13, clang-14, and clang-15.

perf $ uname -r
6.11.0-061100rc7-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)

perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @
augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()


perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
...
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @
augmented_raw_syscalls.bpf.c:509


perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0


perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
     0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
count: 8)                         =


Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you
mentioned.

Thanks,
Howard


>>

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Howard Chu]

Hello,

I'm able to reproduce it on Ubuntu 24.

perf $ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

perf $ uname -r
6.11.0-061100-generic

perf $ git log
commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
Author: Arnaldo Carvalho de Melo <acme@redhat.com>
Date:   Tue Sep 10 13:54:23 2024 -0300

    perf trace: If a syscall arg is marked as 'const', assume it is
coming _from_ userspace


perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter_rename(struct syscall_enter_args *args) @
augmented_raw_syscalls.bpf.c:275


With clang-16:

perf $ clang -v
Ubuntu clang version 16.0.6 (23ubuntu4)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Candidate multilib: .;@m64
Selected multilib: .;@m64

perf $ ./perf trace -e write --max-events=1
     0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:
\27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =

> I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so
> making 16 the minimum could be an issue.

Yes, I think disabling any distro that doesn't come with clang 16
(released in 2023) is not a good idea. I'll try to tame the BPF
verifier.

Thanks,
Howard

On Tue, Sep 17, 2024 at 10:37 AM Howard Chu <howardchu95@gmail.com> wrote:
>
> Hello James and Arnaldo,
>
> On Fri, Sep 13, 2024 at 3:42 AM Arnaldo Carvalho de Melo
> <arnaldo.melo@gmail.com> wrote:
> >
> >
> >
> > On Fri, Sep 13, 2024, 7:20 AM James Clark <james.clark@linaro.org> wrote:
> >>
> >>
> >>
> >> On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
> >> > On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
> >> >> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
> >> >>> Howard reported problems using perf features that use BPF:
> >> >
> >> >>>     perf $ clang -v
> >> >>>     Debian clang version 15.0.6
> >> >>>     Target: x86_64-pc-linux-gnu
> >> >>>     Thread model: posix
> >> >>>     InstalledDir: /bin
> >> >>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >> >>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >> >>>     Candidate multilib: .;@m64
> >> >>>     Selected multilib: .;@m64
> >> >>>     perf $ ./perf trace -e write --max-events=1
> >> >>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> >> >>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> >> >>>     0: R1=ctx() R10=fp0
> >> >>>
> >> >>> But it works with:
> >> >>>
> >> >>>     perf $ clang -v
> >> >>>     Debian clang version 16.0.6 (15~deb12u1)
> >> >>>     Target: x86_64-pc-linux-gnu
> >> >>>     Thread model: posix
> >> >>>     InstalledDir: /bin
> >> >>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >> >>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >> >>>     Candidate multilib: .;@m64
> >> >>>     Selected multilib: .;@m64
> >> >>>     perf $ ./perf trace -e write --max-events=1
> >> >>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
> >> >>>     perf $
> >> >>>
> >> >>> So lets make that the required version, if you happen to have a slightly
> >> >>> older version where this work, please report so that we can adjust the
> >> >>> minimum required version.
> >> >
> >> >> I wasn't able to reproduce the issue with either of these versions. But I
> >> >> suppose it could be an issue with only 15.0.6.
> >> >
> >> > Interesting, that complicates things, probably the best way then is to
> >> > try to build it, if it fails, mention that 15.0.6 is known to be
> >> > problematic and suggest working versions?
> >> >
> >> > - Arnaldo
> >>
> >> I still wasn't able to reproduce it with 15.0.6. And I double checked
> >> with V=1 that the build was using the right clang. I suppose it could be
> >> a build configuration issue, or maybe with a different kernel version?
> >
> >
> > Howard? If not reproducible we can revert it.
> >
> > - Arnaldo
> >
> >
> >>
> >> $  uname --kernel-release
> >> 6.8.0-76060800daily20240311-generic
> >>
> >> $ ../../llvm-project/build/bin/clang -v
> >> clang version 15.0.6 (https://github.com/llvm/llvm-project.git
> >> 088f33605d8a61ff519c580a71b1dd57d16a03f8)
> >> Target: x86_64-unknown-linux-gnu
> >> Thread model: posix
> >> InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
> >> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
> >> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> >> Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> >> Candidate multilib: .;@m64
> >> Selected multilib: .;@m64
> >>
> >> $ git log
> >> commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
> >> (perf-tools-next/perf-tools-next)
> >>
> >> $ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
> >>    tools/perf
> >>
> >> $ perf trace -e write --max-events=1
> >>
> >>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> >>         0x7fffa102d9b0, count: 8)                           = 8
>
> Thanks for the effort! But this output seems to be unaugmented? I
> think you are testing the perf/perf-tools-next branch that's slightly
> older (perf mem: Fix the wrong reference in parse_record_events()
> 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't
> have the new perf trace feature yet...
>
> That's why the perf trace output looks like:
>
> >> $ perf trace -e write --max-events=1
> >>
> >>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> >>         0x7fffa102d9b0, count: 8)
>
> With the new buffer augmentation it should look like this:
>
> perf $ ./perf trace -e write --max-events=1
>      0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> count: 8)                         =
>
> The new perf trace can be found on
> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
> perf-tools-next
>
> perf $ git log
> commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD ->
> struct-aug-arnaldo-2, perf/perf-tools-next)
>
> Also, if you want to test the clang, you may want to checkout to the
> commit before the clang-16 guard (otherwise you can't test different
> clang version).
>
> perf $ git log --oneline
> 1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
> trace: Mark the 'head' arg in the set_robust_list syscall as coming
> from user space
> 0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as
> coming from user space
> edf3ce0ed38e perf env: Find correct branch counter info on hybrid
> 9953807c9e01 perf evlist: Print hint for group
> eb9b9a6f5ab3 tools: Drop nonsensical -O6
> 89c0a55e550e perf pmu: To info add event_type_desc
> f08cc258431d perf evsel: Add accessor for tool_event
> 925320737ae2 perf pmus: Fake PMU clean up
> d3d5c1a00fcd perf list: Avoid potential out of bounds memory read
> 4ae354d73a8e perf help: Fix a typo ("bellow")
> 74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system
> 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '='
> from probe line number regex
> 9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF
> skeletons. **********[[[[[[[[[[[[!!!!(This is the clang
> guard)]]]]]]]]]]*****
> 4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume
> it is coming _from_ userspace
> e37b315c17df perf parse-events: Remove duplicated include in parse-events.c
>
> My machine is debian 12, and I cannot build perf trace with the new
> augmentation feature on clang-13, clang-14, and clang-15.
>
> perf $ uname -r
> 6.11.0-061100rc7-generic
>
> perf $ git log
> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
>
> perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
> ...
> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> 0: R1=ctx() R10=fp0
> ; int sys_enter(struct syscall_enter_args *args) @
> augmented_raw_syscalls.bpf.c:509
> 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
>
>
> perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
> ...
> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> 0: R1=ctx() R10=fp0
> ; int sys_enter(struct syscall_enter_args *args) @
> augmented_raw_syscalls.bpf.c:509
>
>
> perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> 0: R1=ctx() R10=fp0
>
>
> perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
>      0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> count: 8)                         =
>
>
> Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you
> mentioned.
>
> Thanks,
> Howard
>
>
> >>

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Howard Chu]

Hello,

I got some good news:

Not being able to pass the BPF verifier is solely (or bi-solely)
caused by these two functions:

SEC("tp/syscalls/sys_enter_rename")
int sys_enter_rename(struct syscall_enter_args *args)

SEC("tp/syscalls/sys_enter_renameat2")
int sys_enter_renameat2(struct syscall_enter_args *args)

The problem with it is double-string augmentation, but I haven't
figured out how to solve it. Can you please take a look please,
Arnaldo? Now I just commented these two BPF functions and perf trace
runs no problem, built by clang that's older than clang 16, for
example clang 15.0.7.

perf $ clang -v
Ubuntu clang version 15.0.7
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
     0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
count: 8)                         =

But plot twist, it won't build on clang-14

perf $ clang -v
Ubuntu clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /bin
Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
Candidate multilib: .;@m64
Selected multilib: .;@m64
perf $ ./perf trace -e write --max-events=1
libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int sys_enter(struct syscall_enter_args *args) @
augmented_raw_syscalls.bpf.c:509
0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()

This time the problem is my code in BTF general collector:

; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497

I'll resolve this.

Thanks,
Howard

On Wed, Sep 18, 2024 at 8:13 PM Howard Chu <howardchu95@gmail.com> wrote:
>
> Hello,
>
> I'm able to reproduce it on Ubuntu 24.
>
> perf $ cat /etc/os-release
> PRETTY_NAME="Ubuntu 24.04.1 LTS"
> NAME="Ubuntu"
> VERSION_ID="24.04"
> VERSION="24.04.1 LTS (Noble Numbat)"
> VERSION_CODENAME=noble
> ID=ubuntu
> ID_LIKE=debian
> HOME_URL="https://www.ubuntu.com/"
> SUPPORT_URL="https://help.ubuntu.com/"
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> UBUNTU_CODENAME=noble
> LOGO=ubuntu-logo
>
> perf $ uname -r
> 6.11.0-061100-generic
>
> perf $ git log
> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
> Author: Arnaldo Carvalho de Melo <acme@redhat.com>
> Date:   Tue Sep 10 13:54:23 2024 -0300
>
>     perf trace: If a syscall arg is marked as 'const', assume it is
> coming _from_ userspace
>
>
> perf $ clang -v
> Ubuntu clang version 15.0.7
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /bin
> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Candidate multilib: .;@m64
> Selected multilib: .;@m64
>
> perf $ ./perf trace -e write --max-events=1
> libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> 0: R1=ctx() R10=fp0
> ; int sys_enter_rename(struct syscall_enter_args *args) @
> augmented_raw_syscalls.bpf.c:275
>
>
> With clang-16:
>
> perf $ clang -v
> Ubuntu clang version 16.0.6 (23ubuntu4)
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /bin
> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Candidate multilib: .;@m64
> Selected multilib: .;@m64
>
> perf $ ./perf trace -e write --max-events=1
>      0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:
> \27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
>
> > I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so
> > making 16 the minimum could be an issue.
>
> Yes, I think disabling any distro that doesn't come with clang 16
> (released in 2023) is not a good idea. I'll try to tame the BPF
> verifier.
>
> Thanks,
> Howard
>
> On Tue, Sep 17, 2024 at 10:37 AM Howard Chu <howardchu95@gmail.com> wrote:
> >
> > Hello James and Arnaldo,
> >
> > On Fri, Sep 13, 2024 at 3:42 AM Arnaldo Carvalho de Melo
> > <arnaldo.melo@gmail.com> wrote:
> > >
> > >
> > >
> > > On Fri, Sep 13, 2024, 7:20 AM James Clark <james.clark@linaro.org> wrote:
> > >>
> > >>
> > >>
> > >> On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
> > >> > On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
> > >> >> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
> > >> >>> Howard reported problems using perf features that use BPF:
> > >> >
> > >> >>>     perf $ clang -v
> > >> >>>     Debian clang version 15.0.6
> > >> >>>     Target: x86_64-pc-linux-gnu
> > >> >>>     Thread model: posix
> > >> >>>     InstalledDir: /bin
> > >> >>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >> >>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >> >>>     Candidate multilib: .;@m64
> > >> >>>     Selected multilib: .;@m64
> > >> >>>     perf $ ./perf trace -e write --max-events=1
> > >> >>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> > >> >>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> > >> >>>     0: R1=ctx() R10=fp0
> > >> >>>
> > >> >>> But it works with:
> > >> >>>
> > >> >>>     perf $ clang -v
> > >> >>>     Debian clang version 16.0.6 (15~deb12u1)
> > >> >>>     Target: x86_64-pc-linux-gnu
> > >> >>>     Thread model: posix
> > >> >>>     InstalledDir: /bin
> > >> >>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >> >>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >> >>>     Candidate multilib: .;@m64
> > >> >>>     Selected multilib: .;@m64
> > >> >>>     perf $ ./perf trace -e write --max-events=1
> > >> >>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
> > >> >>>     perf $
> > >> >>>
> > >> >>> So lets make that the required version, if you happen to have a slightly
> > >> >>> older version where this work, please report so that we can adjust the
> > >> >>> minimum required version.
> > >> >
> > >> >> I wasn't able to reproduce the issue with either of these versions. But I
> > >> >> suppose it could be an issue with only 15.0.6.
> > >> >
> > >> > Interesting, that complicates things, probably the best way then is to
> > >> > try to build it, if it fails, mention that 15.0.6 is known to be
> > >> > problematic and suggest working versions?
> > >> >
> > >> > - Arnaldo
> > >>
> > >> I still wasn't able to reproduce it with 15.0.6. And I double checked
> > >> with V=1 that the build was using the right clang. I suppose it could be
> > >> a build configuration issue, or maybe with a different kernel version?
> > >
> > >
> > > Howard? If not reproducible we can revert it.
> > >
> > > - Arnaldo
> > >
> > >
> > >>
> > >> $  uname --kernel-release
> > >> 6.8.0-76060800daily20240311-generic
> > >>
> > >> $ ../../llvm-project/build/bin/clang -v
> > >> clang version 15.0.6 (https://github.com/llvm/llvm-project.git
> > >> 088f33605d8a61ff519c580a71b1dd57d16a03f8)
> > >> Target: x86_64-unknown-linux-gnu
> > >> Thread model: posix
> > >> InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
> > >> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
> > >> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> > >> Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> > >> Candidate multilib: .;@m64
> > >> Selected multilib: .;@m64
> > >>
> > >> $ git log
> > >> commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
> > >> (perf-tools-next/perf-tools-next)
> > >>
> > >> $ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
> > >>    tools/perf
> > >>
> > >> $ perf trace -e write --max-events=1
> > >>
> > >>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> > >>         0x7fffa102d9b0, count: 8)                           = 8
> >
> > Thanks for the effort! But this output seems to be unaugmented? I
> > think you are testing the perf/perf-tools-next branch that's slightly
> > older (perf mem: Fix the wrong reference in parse_record_events()
> > 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't
> > have the new perf trace feature yet...
> >
> > That's why the perf trace output looks like:
> >
> > >> $ perf trace -e write --max-events=1
> > >>
> > >>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> > >>         0x7fffa102d9b0, count: 8)
> >
> > With the new buffer augmentation it should look like this:
> >
> > perf $ ./perf trace -e write --max-events=1
> >      0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> > count: 8)                         =
> >
> > The new perf trace can be found on
> > https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
> > perf-tools-next
> >
> > perf $ git log
> > commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD ->
> > struct-aug-arnaldo-2, perf/perf-tools-next)
> >
> > Also, if you want to test the clang, you may want to checkout to the
> > commit before the clang-16 guard (otherwise you can't test different
> > clang version).
> >
> > perf $ git log --oneline
> > 1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
> > trace: Mark the 'head' arg in the set_robust_list syscall as coming
> > from user space
> > 0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as
> > coming from user space
> > edf3ce0ed38e perf env: Find correct branch counter info on hybrid
> > 9953807c9e01 perf evlist: Print hint for group
> > eb9b9a6f5ab3 tools: Drop nonsensical -O6
> > 89c0a55e550e perf pmu: To info add event_type_desc
> > f08cc258431d perf evsel: Add accessor for tool_event
> > 925320737ae2 perf pmus: Fake PMU clean up
> > d3d5c1a00fcd perf list: Avoid potential out of bounds memory read
> > 4ae354d73a8e perf help: Fix a typo ("bellow")
> > 74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system
> > 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '='
> > from probe line number regex
> > 9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF
> > skeletons. **********[[[[[[[[[[[[!!!!(This is the clang
> > guard)]]]]]]]]]]*****
> > 4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume
> > it is coming _from_ userspace
> > e37b315c17df perf parse-events: Remove duplicated include in parse-events.c
> >
> > My machine is debian 12, and I cannot build perf trace with the new
> > augmentation feature on clang-13, clang-14, and clang-15.
> >
> > perf $ uname -r
> > 6.11.0-061100rc7-generic
> >
> > perf $ git log
> > commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
> >
> > perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
> > ...
> > libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > 0: R1=ctx() R10=fp0
> > ; int sys_enter(struct syscall_enter_args *args) @
> > augmented_raw_syscalls.bpf.c:509
> > 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
> >
> >
> > perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
> > ...
> > libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > 0: R1=ctx() R10=fp0
> > ; int sys_enter(struct syscall_enter_args *args) @
> > augmented_raw_syscalls.bpf.c:509
> >
> >
> > perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
> > libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > 0: R1=ctx() R10=fp0
> >
> >
> > perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
> >      0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> > count: 8)                         =
> >
> >
> > Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you
> > mentioned.
> >
> > Thanks,
> > Howard
> >
> >
> > >>

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Alan Maguire]

On 24/09/2024 03:00, Howard Chu wrote:
> Hello,
> 
> I got some good news:
> 
> Not being able to pass the BPF verifier is solely (or bi-solely)
> caused by these two functions:
> 
> SEC("tp/syscalls/sys_enter_rename")
> int sys_enter_rename(struct syscall_enter_args *args)
> 
> SEC("tp/syscalls/sys_enter_renameat2")
> int sys_enter_renameat2(struct syscall_enter_args *args)
> 
> The problem with it is double-string augmentation, but I haven't
> figured out how to solve it. Can you please take a look please,
> Arnaldo? Now I just commented these two BPF functions and perf trace
> runs no problem, built by clang that's older than clang 16, for
> example clang 15.0.7.
>

hi Howard,

I could be wrong - and I don't know this code at all - but I _think_ the
problem is here:

	oldpath_len = augmented_arg__read_str(&augmented_args->arg,
oldpath_arg, sizeof(augmented_args->arg.value));
        augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
        len += augmented_args->arg.size;

        struct augmented_arg *arg2 = (void *)&augmented_args->arg.value
+ augmented_args->arg.size;

        newpath_len = augmented_arg__read_str(arg2, newpath_arg,
sizeof(augmented_args->arg.value));


Notice that these strings share the augmented_args->arg.value for
storage, but in the second case (reading arg2) we have added
augmented_args->arg.size to the offset where we store the second string.
However at the same time we have also told augmented_arg__read_str()
that it has

sizeof(augmented_args->arg.value)


...to work with as arg_len, where in reality we should have specified

sizeof(augmented_args->arg.value) - augmented_args->arg.size

...and ensured that after doing that subtraction, there is still space
to work with (i.e. the above is > 0).

I _think_ the verifier is likely complaining that we can write past the
end of augmented_args->arg.value , does that sound right?

Fixing the arg_len parameter for cases where we record more than one
augmented string by reducing arg_len available should solve this if so.
Hope this helps,

Alan

> perf $ clang -v
> Ubuntu clang version 15.0.7
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /bin
> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Candidate multilib: .;@m64
> Selected multilib: .;@m64
> perf $ ./perf trace -e write --max-events=1
>      0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> count: 8)                         =
> 
> But plot twist, it won't build on clang-14
> 
> perf $ clang -v
> Ubuntu clang version 14.0.6
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /bin
> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> Candidate multilib: .;@m64
> Selected multilib: .;@m64
> perf $ ./perf trace -e write --max-events=1
> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> 0: R1=ctx() R10=fp0
> ; int sys_enter(struct syscall_enter_args *args) @
> augmented_raw_syscalls.bpf.c:509
> 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
> 
> This time the problem is my code in BTF general collector:
> 
> ; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497
> 
> I'll resolve this.
> 
> Thanks,
> Howard
> 
> On Wed, Sep 18, 2024 at 8:13 PM Howard Chu <howardchu95@gmail.com> wrote:
>>
>> Hello,
>>
>> I'm able to reproduce it on Ubuntu 24.
>>
>> perf $ cat /etc/os-release
>> PRETTY_NAME="Ubuntu 24.04.1 LTS"
>> NAME="Ubuntu"
>> VERSION_ID="24.04"
>> VERSION="24.04.1 LTS (Noble Numbat)"
>> VERSION_CODENAME=noble
>> ID=ubuntu
>> ID_LIKE=debian
>> HOME_URL="https://www.ubuntu.com/"
>> SUPPORT_URL="https://help.ubuntu.com/"
>> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
>> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
>> UBUNTU_CODENAME=noble
>> LOGO=ubuntu-logo
>>
>> perf $ uname -r
>> 6.11.0-061100-generic
>>
>> perf $ git log
>> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
>> Author: Arnaldo Carvalho de Melo <acme@redhat.com>
>> Date:   Tue Sep 10 13:54:23 2024 -0300
>>
>>     perf trace: If a syscall arg is marked as 'const', assume it is
>> coming _from_ userspace
>>
>>
>> perf $ clang -v
>> Ubuntu clang version 15.0.7
>> Target: x86_64-pc-linux-gnu
>> Thread model: posix
>> InstalledDir: /bin
>> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
>> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
>> Candidate multilib: .;@m64
>> Selected multilib: .;@m64
>>
>> perf $ ./perf trace -e write --max-events=1
>> libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
>> libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
>> 0: R1=ctx() R10=fp0
>> ; int sys_enter_rename(struct syscall_enter_args *args) @
>> augmented_raw_syscalls.bpf.c:275
>>
>>
>> With clang-16:
>>
>> perf $ clang -v
>> Ubuntu clang version 16.0.6 (23ubuntu4)
>> Target: x86_64-pc-linux-gnu
>> Thread model: posix
>> InstalledDir: /bin
>> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
>> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
>> Candidate multilib: .;@m64
>> Selected multilib: .;@m64
>>
>> perf $ ./perf trace -e write --max-events=1
>>      0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:
>> \27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
>>
>>> I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so
>>> making 16 the minimum could be an issue.
>>
>> Yes, I think disabling any distro that doesn't come with clang 16
>> (released in 2023) is not a good idea. I'll try to tame the BPF
>> verifier.
>>
>> Thanks,
>> Howard
>>
>> On Tue, Sep 17, 2024 at 10:37 AM Howard Chu <howardchu95@gmail.com> wrote:
>>>
>>> Hello James and Arnaldo,
>>>
>>> On Fri, Sep 13, 2024 at 3:42 AM Arnaldo Carvalho de Melo
>>> <arnaldo.melo@gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> On Fri, Sep 13, 2024, 7:20 AM James Clark <james.clark@linaro.org> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
>>>>>> On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
>>>>>>> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
>>>>>>>> Howard reported problems using perf features that use BPF:
>>>>>>
>>>>>>>>     perf $ clang -v
>>>>>>>>     Debian clang version 15.0.6
>>>>>>>>     Target: x86_64-pc-linux-gnu
>>>>>>>>     Thread model: posix
>>>>>>>>     InstalledDir: /bin
>>>>>>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>>>>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>>>>>>     Candidate multilib: .;@m64
>>>>>>>>     Selected multilib: .;@m64
>>>>>>>>     perf $ ./perf trace -e write --max-events=1
>>>>>>>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
>>>>>>>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
>>>>>>>>     0: R1=ctx() R10=fp0
>>>>>>>>
>>>>>>>> But it works with:
>>>>>>>>
>>>>>>>>     perf $ clang -v
>>>>>>>>     Debian clang version 16.0.6 (15~deb12u1)
>>>>>>>>     Target: x86_64-pc-linux-gnu
>>>>>>>>     Thread model: posix
>>>>>>>>     InstalledDir: /bin
>>>>>>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>>>>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
>>>>>>>>     Candidate multilib: .;@m64
>>>>>>>>     Selected multilib: .;@m64
>>>>>>>>     perf $ ./perf trace -e write --max-events=1
>>>>>>>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
>>>>>>>>     perf $
>>>>>>>>
>>>>>>>> So lets make that the required version, if you happen to have a slightly
>>>>>>>> older version where this work, please report so that we can adjust the
>>>>>>>> minimum required version.
>>>>>>
>>>>>>> I wasn't able to reproduce the issue with either of these versions. But I
>>>>>>> suppose it could be an issue with only 15.0.6.
>>>>>>
>>>>>> Interesting, that complicates things, probably the best way then is to
>>>>>> try to build it, if it fails, mention that 15.0.6 is known to be
>>>>>> problematic and suggest working versions?
>>>>>>
>>>>>> - Arnaldo
>>>>>
>>>>> I still wasn't able to reproduce it with 15.0.6. And I double checked
>>>>> with V=1 that the build was using the right clang. I suppose it could be
>>>>> a build configuration issue, or maybe with a different kernel version?
>>>>
>>>>
>>>> Howard? If not reproducible we can revert it.
>>>>
>>>> - Arnaldo
>>>>
>>>>
>>>>>
>>>>> $  uname --kernel-release
>>>>> 6.8.0-76060800daily20240311-generic
>>>>>
>>>>> $ ../../llvm-project/build/bin/clang -v
>>>>> clang version 15.0.6 (https://github.com/llvm/llvm-project.git
>>>>> 088f33605d8a61ff519c580a71b1dd57d16a03f8)
>>>>> Target: x86_64-unknown-linux-gnu
>>>>> Thread model: posix
>>>>> InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
>>>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
>>>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
>>>>> Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
>>>>> Candidate multilib: .;@m64
>>>>> Selected multilib: .;@m64
>>>>>
>>>>> $ git log
>>>>> commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
>>>>> (perf-tools-next/perf-tools-next)
>>>>>
>>>>> $ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
>>>>>    tools/perf
>>>>>
>>>>> $ perf trace -e write --max-events=1
>>>>>
>>>>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
>>>>>         0x7fffa102d9b0, count: 8)                           = 8
>>>
>>> Thanks for the effort! But this output seems to be unaugmented? I
>>> think you are testing the perf/perf-tools-next branch that's slightly
>>> older (perf mem: Fix the wrong reference in parse_record_events()
>>> 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't
>>> have the new perf trace feature yet...
>>>
>>> That's why the perf trace output looks like:
>>>
>>>>> $ perf trace -e write --max-events=1
>>>>>
>>>>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
>>>>>         0x7fffa102d9b0, count: 8)
>>>
>>> With the new buffer augmentation it should look like this:
>>>
>>> perf $ ./perf trace -e write --max-events=1
>>>      0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
>>> count: 8)                         =
>>>
>>> The new perf trace can be found on
>>> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
>>> perf-tools-next
>>>
>>> perf $ git log
>>> commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD ->
>>> struct-aug-arnaldo-2, perf/perf-tools-next)
>>>
>>> Also, if you want to test the clang, you may want to checkout to the
>>> commit before the clang-16 guard (otherwise you can't test different
>>> clang version).
>>>
>>> perf $ git log --oneline
>>> 1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
>>> trace: Mark the 'head' arg in the set_robust_list syscall as coming
>>> from user space
>>> 0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as
>>> coming from user space
>>> edf3ce0ed38e perf env: Find correct branch counter info on hybrid
>>> 9953807c9e01 perf evlist: Print hint for group
>>> eb9b9a6f5ab3 tools: Drop nonsensical -O6
>>> 89c0a55e550e perf pmu: To info add event_type_desc
>>> f08cc258431d perf evsel: Add accessor for tool_event
>>> 925320737ae2 perf pmus: Fake PMU clean up
>>> d3d5c1a00fcd perf list: Avoid potential out of bounds memory read
>>> 4ae354d73a8e perf help: Fix a typo ("bellow")
>>> 74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system
>>> 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '='
>>> from probe line number regex
>>> 9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF
>>> skeletons. **********[[[[[[[[[[[[!!!!(This is the clang
>>> guard)]]]]]]]]]]*****
>>> 4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume
>>> it is coming _from_ userspace
>>> e37b315c17df perf parse-events: Remove duplicated include in parse-events.c
>>>
>>> My machine is debian 12, and I cannot build perf trace with the new
>>> augmentation feature on clang-13, clang-14, and clang-15.
>>>
>>> perf $ uname -r
>>> 6.11.0-061100rc7-generic
>>>
>>> perf $ git log
>>> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
>>>
>>> perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
>>> ...
>>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
>>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
>>> 0: R1=ctx() R10=fp0
>>> ; int sys_enter(struct syscall_enter_args *args) @
>>> augmented_raw_syscalls.bpf.c:509
>>> 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
>>>
>>>
>>> perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
>>> ...
>>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
>>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
>>> 0: R1=ctx() R10=fp0
>>> ; int sys_enter(struct syscall_enter_args *args) @
>>> augmented_raw_syscalls.bpf.c:509
>>>
>>>
>>> perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
>>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
>>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
>>> 0: R1=ctx() R10=fp0
>>>
>>>
>>> perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
>>>      0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
>>> count: 8)                         =
>>>
>>>
>>> Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you
>>> mentioned.
>>>
>>> Thanks,
>>> Howard
>>>
>>>
>>>>>

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Howard Chu]

Hello Alan,

On Tue, Sep 24, 2024 at 2:30 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 24/09/2024 03:00, Howard Chu wrote:
> > Hello,
> >
> > I got some good news:
> >
> > Not being able to pass the BPF verifier is solely (or bi-solely)
> > caused by these two functions:
> >
> > SEC("tp/syscalls/sys_enter_rename")
> > int sys_enter_rename(struct syscall_enter_args *args)
> >
> > SEC("tp/syscalls/sys_enter_renameat2")
> > int sys_enter_renameat2(struct syscall_enter_args *args)
> >
> > The problem with it is double-string augmentation, but I haven't
> > figured out how to solve it. Can you please take a look please,
> > Arnaldo? Now I just commented these two BPF functions and perf trace
> > runs no problem, built by clang that's older than clang 16, for
> > example clang 15.0.7.
> >
>
> hi Howard,
>
> I could be wrong - and I don't know this code at all - but I _think_ the
> problem is here:
>
>         oldpath_len = augmented_arg__read_str(&augmented_args->arg,
> oldpath_arg, sizeof(augmented_args->arg.value));
>         augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
>         len += augmented_args->arg.size;
>
>         struct augmented_arg *arg2 = (void *)&augmented_args->arg.value
> + augmented_args->arg.size;
>
>         newpath_len = augmented_arg__read_str(arg2, newpath_arg,
> sizeof(augmented_args->arg.value));
>
>
> Notice that these strings share the augmented_args->arg.value for
> storage, but in the second case (reading arg2) we have added
> augmented_args->arg.size to the offset where we store the second string.
> However at the same time we have also told augmented_arg__read_str()
> that it has
>
> sizeof(augmented_args->arg.value)
>
>
> ...to work with as arg_len, where in reality we should have specified
>
> sizeof(augmented_args->arg.value) - augmented_args->arg.size
>
> ...and ensured that after doing that subtraction, there is still space
> to work with (i.e. the above is > 0).
>
> I _think_ the verifier is likely complaining that we can write past the
> end of augmented_args->arg.value , does that sound right?
>
> Fixing the arg_len parameter for cases where we record more than one
> augmented string by reducing arg_len available should solve this if so.
> Hope this helps,


Thanks for the help, but unfortunately it didn't work... But that's a
valid analysis, I had the same idea as you, but I haven't been able to
solve it. I'll give it more tries.

Thanks,
Howard

>
> Alan
>
> > perf $ clang -v
> > Ubuntu clang version 15.0.7
> > Target: x86_64-pc-linux-gnu
> > Thread model: posix
> > InstalledDir: /bin
> > Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > Candidate multilib: .;@m64
> > Selected multilib: .;@m64
> > perf $ ./perf trace -e write --max-events=1
> >      0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> > count: 8)                         =
> >
> > But plot twist, it won't build on clang-14
> >
> > perf $ clang -v
> > Ubuntu clang version 14.0.6
> > Target: x86_64-pc-linux-gnu
> > Thread model: posix
> > InstalledDir: /bin
> > Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > Candidate multilib: .;@m64
> > Selected multilib: .;@m64
> > perf $ ./perf trace -e write --max-events=1
> > libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > 0: R1=ctx() R10=fp0
> > ; int sys_enter(struct syscall_enter_args *args) @
> > augmented_raw_syscalls.bpf.c:509
> > 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
> >
> > This time the problem is my code in BTF general collector:
> >
> > ; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497
> >
> > I'll resolve this.
> >
> > Thanks,
> > Howard
> >
> > On Wed, Sep 18, 2024 at 8:13 PM Howard Chu <howardchu95@gmail.com> wrote:
> >>
> >> Hello,
> >>
> >> I'm able to reproduce it on Ubuntu 24.
> >>
> >> perf $ cat /etc/os-release
> >> PRETTY_NAME="Ubuntu 24.04.1 LTS"
> >> NAME="Ubuntu"
> >> VERSION_ID="24.04"
> >> VERSION="24.04.1 LTS (Noble Numbat)"
> >> VERSION_CODENAME=noble
> >> ID=ubuntu
> >> ID_LIKE=debian
> >> HOME_URL="https://www.ubuntu.com/"
> >> SUPPORT_URL="https://help.ubuntu.com/"
> >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> >> UBUNTU_CODENAME=noble
> >> LOGO=ubuntu-logo
> >>
> >> perf $ uname -r
> >> 6.11.0-061100-generic
> >>
> >> perf $ git log
> >> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
> >> Author: Arnaldo Carvalho de Melo <acme@redhat.com>
> >> Date:   Tue Sep 10 13:54:23 2024 -0300
> >>
> >>     perf trace: If a syscall arg is marked as 'const', assume it is
> >> coming _from_ userspace
> >>
> >>
> >> perf $ clang -v
> >> Ubuntu clang version 15.0.7
> >> Target: x86_64-pc-linux-gnu
> >> Thread model: posix
> >> InstalledDir: /bin
> >> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> >> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> >> Candidate multilib: .;@m64
> >> Selected multilib: .;@m64
> >>
> >> perf $ ./perf trace -e write --max-events=1
> >> libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> >> libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> >> 0: R1=ctx() R10=fp0
> >> ; int sys_enter_rename(struct syscall_enter_args *args) @
> >> augmented_raw_syscalls.bpf.c:275
> >>
> >>
> >> With clang-16:
> >>
> >> perf $ clang -v
> >> Ubuntu clang version 16.0.6 (23ubuntu4)
> >> Target: x86_64-pc-linux-gnu
> >> Thread model: posix
> >> InstalledDir: /bin
> >> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> >> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> >> Candidate multilib: .;@m64
> >> Selected multilib: .;@m64
> >>
> >> perf $ ./perf trace -e write --max-events=1
> >>      0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:
> >> \27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
> >>
> >>> I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so
> >>> making 16 the minimum could be an issue.
> >>
> >> Yes, I think disabling any distro that doesn't come with clang 16
> >> (released in 2023) is not a good idea. I'll try to tame the BPF
> >> verifier.
> >>
> >> Thanks,
> >> Howard
> >>
> >> On Tue, Sep 17, 2024 at 10:37 AM Howard Chu <howardchu95@gmail.com> wrote:
> >>>
> >>> Hello James and Arnaldo,
> >>>
> >>> On Fri, Sep 13, 2024 at 3:42 AM Arnaldo Carvalho de Melo
> >>> <arnaldo.melo@gmail.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Sep 13, 2024, 7:20 AM James Clark <james.clark@linaro.org> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
> >>>>>> On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
> >>>>>>> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
> >>>>>>>> Howard reported problems using perf features that use BPF:
> >>>>>>
> >>>>>>>>     perf $ clang -v
> >>>>>>>>     Debian clang version 15.0.6
> >>>>>>>>     Target: x86_64-pc-linux-gnu
> >>>>>>>>     Thread model: posix
> >>>>>>>>     InstalledDir: /bin
> >>>>>>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >>>>>>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >>>>>>>>     Candidate multilib: .;@m64
> >>>>>>>>     Selected multilib: .;@m64
> >>>>>>>>     perf $ ./perf trace -e write --max-events=1
> >>>>>>>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> >>>>>>>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> >>>>>>>>     0: R1=ctx() R10=fp0
> >>>>>>>>
> >>>>>>>> But it works with:
> >>>>>>>>
> >>>>>>>>     perf $ clang -v
> >>>>>>>>     Debian clang version 16.0.6 (15~deb12u1)
> >>>>>>>>     Target: x86_64-pc-linux-gnu
> >>>>>>>>     Thread model: posix
> >>>>>>>>     InstalledDir: /bin
> >>>>>>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >>>>>>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> >>>>>>>>     Candidate multilib: .;@m64
> >>>>>>>>     Selected multilib: .;@m64
> >>>>>>>>     perf $ ./perf trace -e write --max-events=1
> >>>>>>>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
> >>>>>>>>     perf $
> >>>>>>>>
> >>>>>>>> So lets make that the required version, if you happen to have a slightly
> >>>>>>>> older version where this work, please report so that we can adjust the
> >>>>>>>> minimum required version.
> >>>>>>
> >>>>>>> I wasn't able to reproduce the issue with either of these versions. But I
> >>>>>>> suppose it could be an issue with only 15.0.6.
> >>>>>>
> >>>>>> Interesting, that complicates things, probably the best way then is to
> >>>>>> try to build it, if it fails, mention that 15.0.6 is known to be
> >>>>>> problematic and suggest working versions?
> >>>>>>
> >>>>>> - Arnaldo
> >>>>>
> >>>>> I still wasn't able to reproduce it with 15.0.6. And I double checked
> >>>>> with V=1 that the build was using the right clang. I suppose it could be
> >>>>> a build configuration issue, or maybe with a different kernel version?
> >>>>
> >>>>
> >>>> Howard? If not reproducible we can revert it.
> >>>>
> >>>> - Arnaldo
> >>>>
> >>>>
> >>>>>
> >>>>> $  uname --kernel-release
> >>>>> 6.8.0-76060800daily20240311-generic
> >>>>>
> >>>>> $ ../../llvm-project/build/bin/clang -v
> >>>>> clang version 15.0.6 (https://github.com/llvm/llvm-project.git
> >>>>> 088f33605d8a61ff519c580a71b1dd57d16a03f8)
> >>>>> Target: x86_64-unknown-linux-gnu
> >>>>> Thread model: posix
> >>>>> InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
> >>>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
> >>>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> >>>>> Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> >>>>> Candidate multilib: .;@m64
> >>>>> Selected multilib: .;@m64
> >>>>>
> >>>>> $ git log
> >>>>> commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
> >>>>> (perf-tools-next/perf-tools-next)
> >>>>>
> >>>>> $ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
> >>>>>    tools/perf
> >>>>>
> >>>>> $ perf trace -e write --max-events=1
> >>>>>
> >>>>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> >>>>>         0x7fffa102d9b0, count: 8)                           = 8
> >>>
> >>> Thanks for the effort! But this output seems to be unaugmented? I
> >>> think you are testing the perf/perf-tools-next branch that's slightly
> >>> older (perf mem: Fix the wrong reference in parse_record_events()
> >>> 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't
> >>> have the new perf trace feature yet...
> >>>
> >>> That's why the perf trace output looks like:
> >>>
> >>>>> $ perf trace -e write --max-events=1
> >>>>>
> >>>>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> >>>>>         0x7fffa102d9b0, count: 8)
> >>>
> >>> With the new buffer augmentation it should look like this:
> >>>
> >>> perf $ ./perf trace -e write --max-events=1
> >>>      0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> >>> count: 8)                         =
> >>>
> >>> The new perf trace can be found on
> >>> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
> >>> perf-tools-next
> >>>
> >>> perf $ git log
> >>> commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD ->
> >>> struct-aug-arnaldo-2, perf/perf-tools-next)
> >>>
> >>> Also, if you want to test the clang, you may want to checkout to the
> >>> commit before the clang-16 guard (otherwise you can't test different
> >>> clang version).
> >>>
> >>> perf $ git log --oneline
> >>> 1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
> >>> trace: Mark the 'head' arg in the set_robust_list syscall as coming
> >>> from user space
> >>> 0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as
> >>> coming from user space
> >>> edf3ce0ed38e perf env: Find correct branch counter info on hybrid
> >>> 9953807c9e01 perf evlist: Print hint for group
> >>> eb9b9a6f5ab3 tools: Drop nonsensical -O6
> >>> 89c0a55e550e perf pmu: To info add event_type_desc
> >>> f08cc258431d perf evsel: Add accessor for tool_event
> >>> 925320737ae2 perf pmus: Fake PMU clean up
> >>> d3d5c1a00fcd perf list: Avoid potential out of bounds memory read
> >>> 4ae354d73a8e perf help: Fix a typo ("bellow")
> >>> 74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system
> >>> 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '='
> >>> from probe line number regex
> >>> 9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF
> >>> skeletons. **********[[[[[[[[[[[[!!!!(This is the clang
> >>> guard)]]]]]]]]]]*****
> >>> 4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume
> >>> it is coming _from_ userspace
> >>> e37b315c17df perf parse-events: Remove duplicated include in parse-events.c
> >>>
> >>> My machine is debian 12, and I cannot build perf trace with the new
> >>> augmentation feature on clang-13, clang-14, and clang-15.
> >>>
> >>> perf $ uname -r
> >>> 6.11.0-061100rc7-generic
> >>>
> >>> perf $ git log
> >>> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
> >>>
> >>> perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
> >>> ...
> >>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> >>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> >>> 0: R1=ctx() R10=fp0
> >>> ; int sys_enter(struct syscall_enter_args *args) @
> >>> augmented_raw_syscalls.bpf.c:509
> >>> 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
> >>>
> >>>
> >>> perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
> >>> ...
> >>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> >>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> >>> 0: R1=ctx() R10=fp0
> >>> ; int sys_enter(struct syscall_enter_args *args) @
> >>> augmented_raw_syscalls.bpf.c:509
> >>>
> >>>
> >>> perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
> >>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> >>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> >>> 0: R1=ctx() R10=fp0
> >>>
> >>>
> >>> perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
> >>>      0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> >>> count: 8)                         =
> >>>
> >>>
> >>> Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you
> >>> mentioned.
> >>>
> >>> Thanks,
> >>> Howard
> >>>
> >>>
> >>>>>
>

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Howard Chu]

Hi Alan, Arnaldo and James,

This problem was solved in [PATCH 0/2] perf trace: Fix support for the
new BPF feature in clang 12 (Link:
https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-howardchu95@gmail.com/T/#t),
sorry I forgot to cc you two.

Alan's thought was correct. Thank you so much! :)

Thanks,
Howard


On Tue, Sep 24, 2024 at 11:45 AM Howard Chu <howardchu95@gmail.com> wrote:
>
> Hello Alan,
>
> On Tue, Sep 24, 2024 at 2:30 AM Alan Maguire <alan.maguire@oracle.com> wrote:
> >
> > On 24/09/2024 03:00, Howard Chu wrote:
> > > Hello,
> > >
> > > I got some good news:
> > >
> > > Not being able to pass the BPF verifier is solely (or bi-solely)
> > > caused by these two functions:
> > >
> > > SEC("tp/syscalls/sys_enter_rename")
> > > int sys_enter_rename(struct syscall_enter_args *args)
> > >
> > > SEC("tp/syscalls/sys_enter_renameat2")
> > > int sys_enter_renameat2(struct syscall_enter_args *args)
> > >
> > > The problem with it is double-string augmentation, but I haven't
> > > figured out how to solve it. Can you please take a look please,
> > > Arnaldo? Now I just commented these two BPF functions and perf trace
> > > runs no problem, built by clang that's older than clang 16, for
> > > example clang 15.0.7.
> > >
> >
> > hi Howard,
> >
> > I could be wrong - and I don't know this code at all - but I _think_ the
> > problem is here:
> >
> >         oldpath_len = augmented_arg__read_str(&augmented_args->arg,
> > oldpath_arg, sizeof(augmented_args->arg.value));
> >         augmented_args->arg.size = PERF_ALIGN(oldpath_len + 1, sizeof(u64));
> >         len += augmented_args->arg.size;
> >
> >         struct augmented_arg *arg2 = (void *)&augmented_args->arg.value
> > + augmented_args->arg.size;
> >
> >         newpath_len = augmented_arg__read_str(arg2, newpath_arg,
> > sizeof(augmented_args->arg.value));
> >
> >
> > Notice that these strings share the augmented_args->arg.value for
> > storage, but in the second case (reading arg2) we have added
> > augmented_args->arg.size to the offset where we store the second string.
> > However at the same time we have also told augmented_arg__read_str()
> > that it has
> >
> > sizeof(augmented_args->arg.value)
> >
> >
> > ...to work with as arg_len, where in reality we should have specified
> >
> > sizeof(augmented_args->arg.value) - augmented_args->arg.size
> >
> > ...and ensured that after doing that subtraction, there is still space
> > to work with (i.e. the above is > 0).
> >
> > I _think_ the verifier is likely complaining that we can write past the
> > end of augmented_args->arg.value , does that sound right?
> >
> > Fixing the arg_len parameter for cases where we record more than one
> > augmented string by reducing arg_len available should solve this if so.
> > Hope this helps,
>
>
> Thanks for the help, but unfortunately it didn't work... But that's a
> valid analysis, I had the same idea as you, but I haven't been able to
> solve it. I'll give it more tries.
>
> Thanks,
> Howard
>
> >
> > Alan
> >
> > > perf $ clang -v
> > > Ubuntu clang version 15.0.7
> > > Target: x86_64-pc-linux-gnu
> > > Thread model: posix
> > > InstalledDir: /bin
> > > Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > > Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > > Candidate multilib: .;@m64
> > > Selected multilib: .;@m64
> > > perf $ ./perf trace -e write --max-events=1
> > >      0.000 ( 0.008 ms): gmain/2173 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> > > count: 8)                         =
> > >
> > > But plot twist, it won't build on clang-14
> > >
> > > perf $ clang -v
> > > Ubuntu clang version 14.0.6
> > > Target: x86_64-pc-linux-gnu
> > > Thread model: posix
> > > InstalledDir: /bin
> > > Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > > Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > > Candidate multilib: .;@m64
> > > Selected multilib: .;@m64
> > > perf $ ./perf trace -e write --max-events=1
> > > libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > > libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > > 0: R1=ctx() R10=fp0
> > > ; int sys_enter(struct syscall_enter_args *args) @
> > > augmented_raw_syscalls.bpf.c:509
> > > 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
> > >
> > > This time the problem is my code in BTF general collector:
> > >
> > > ; payload_offset += written; @ augmented_raw_syscalls.bpf.c:497
> > >
> > > I'll resolve this.
> > >
> > > Thanks,
> > > Howard
> > >
> > > On Wed, Sep 18, 2024 at 8:13 PM Howard Chu <howardchu95@gmail.com> wrote:
> > >>
> > >> Hello,
> > >>
> > >> I'm able to reproduce it on Ubuntu 24.
> > >>
> > >> perf $ cat /etc/os-release
> > >> PRETTY_NAME="Ubuntu 24.04.1 LTS"
> > >> NAME="Ubuntu"
> > >> VERSION_ID="24.04"
> > >> VERSION="24.04.1 LTS (Noble Numbat)"
> > >> VERSION_CODENAME=noble
> > >> ID=ubuntu
> > >> ID_LIKE=debian
> > >> HOME_URL="https://www.ubuntu.com/"
> > >> SUPPORT_URL="https://help.ubuntu.com/"
> > >> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> > >> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> > >> UBUNTU_CODENAME=noble
> > >> LOGO=ubuntu-logo
> > >>
> > >> perf $ uname -r
> > >> 6.11.0-061100-generic
> > >>
> > >> perf $ git log
> > >> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
> > >> Author: Arnaldo Carvalho de Melo <acme@redhat.com>
> > >> Date:   Tue Sep 10 13:54:23 2024 -0300
> > >>
> > >>     perf trace: If a syscall arg is marked as 'const', assume it is
> > >> coming _from_ userspace
> > >>
> > >>
> > >> perf $ clang -v
> > >> Ubuntu clang version 15.0.7
> > >> Target: x86_64-pc-linux-gnu
> > >> Thread model: posix
> > >> InstalledDir: /bin
> > >> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > >> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > >> Candidate multilib: .;@m64
> > >> Selected multilib: .;@m64
> > >>
> > >> perf $ ./perf trace -e write --max-events=1
> > >> libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> > >> libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> > >> 0: R1=ctx() R10=fp0
> > >> ; int sys_enter_rename(struct syscall_enter_args *args) @
> > >> augmented_raw_syscalls.bpf.c:275
> > >>
> > >>
> > >> With clang-16:
> > >>
> > >> perf $ clang -v
> > >> Ubuntu clang version 16.0.6 (23ubuntu4)
> > >> Target: x86_64-pc-linux-gnu
> > >> Thread model: posix
> > >> InstalledDir: /bin
> > >> Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > >> Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/13
> > >> Candidate multilib: .;@m64
> > >> Selected multilib: .;@m64
> > >>
> > >> perf $ ./perf trace -e write --max-events=1
> > >>      0.000 ( 0.021 ms): sudo/4741 write(fd: 8, buf:
> > >> \27[?25l\27[37m\27[48;5;96m\27[H[trace] , count: 205) =
> > >>
> > >>> I'll try to build it and see. But Ubuntu 22 only has clang 11 to 15 so
> > >>> making 16 the minimum could be an issue.
> > >>
> > >> Yes, I think disabling any distro that doesn't come with clang 16
> > >> (released in 2023) is not a good idea. I'll try to tame the BPF
> > >> verifier.
> > >>
> > >> Thanks,
> > >> Howard
> > >>
> > >> On Tue, Sep 17, 2024 at 10:37 AM Howard Chu <howardchu95@gmail.com> wrote:
> > >>>
> > >>> Hello James and Arnaldo,
> > >>>
> > >>> On Fri, Sep 13, 2024 at 3:42 AM Arnaldo Carvalho de Melo
> > >>> <arnaldo.melo@gmail.com> wrote:
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Fri, Sep 13, 2024, 7:20 AM James Clark <james.clark@linaro.org> wrote:
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On 12/09/2024 15:50, Arnaldo Carvalho de Melo wrote:
> > >>>>>> On Thu, Sep 12, 2024 at 03:40:32PM +0100, James Clark wrote:
> > >>>>>>> On 11/09/2024 13:24, Arnaldo Carvalho de Melo wrote:
> > >>>>>>>> Howard reported problems using perf features that use BPF:
> > >>>>>>
> > >>>>>>>>     perf $ clang -v
> > >>>>>>>>     Debian clang version 15.0.6
> > >>>>>>>>     Target: x86_64-pc-linux-gnu
> > >>>>>>>>     Thread model: posix
> > >>>>>>>>     InstalledDir: /bin
> > >>>>>>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >>>>>>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >>>>>>>>     Candidate multilib: .;@m64
> > >>>>>>>>     Selected multilib: .;@m64
> > >>>>>>>>     perf $ ./perf trace -e write --max-events=1
> > >>>>>>>>     libbpf: prog 'sys_enter_rename': BPF program load failed: Permission denied
> > >>>>>>>>     libbpf: prog 'sys_enter_rename': -- BEGIN PROG LOAD LOG --
> > >>>>>>>>     0: R1=ctx() R10=fp0
> > >>>>>>>>
> > >>>>>>>> But it works with:
> > >>>>>>>>
> > >>>>>>>>     perf $ clang -v
> > >>>>>>>>     Debian clang version 16.0.6 (15~deb12u1)
> > >>>>>>>>     Target: x86_64-pc-linux-gnu
> > >>>>>>>>     Thread model: posix
> > >>>>>>>>     InstalledDir: /bin
> > >>>>>>>>     Found candidate GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >>>>>>>>     Selected GCC installation: /bin/../lib/gcc/x86_64-linux-gnu/12
> > >>>>>>>>     Candidate multilib: .;@m64
> > >>>>>>>>     Selected multilib: .;@m64
> > >>>>>>>>     perf $ ./perf trace -e write --max-events=1
> > >>>>>>>>          0.000 ( 0.009 ms): gmain/1448 write(fd: 4, buf: \1\0\0\0\0\0\0\0, count: 8)                         = 8 (kworker/0:0-eve)
> > >>>>>>>>     perf $
> > >>>>>>>>
> > >>>>>>>> So lets make that the required version, if you happen to have a slightly
> > >>>>>>>> older version where this work, please report so that we can adjust the
> > >>>>>>>> minimum required version.
> > >>>>>>
> > >>>>>>> I wasn't able to reproduce the issue with either of these versions. But I
> > >>>>>>> suppose it could be an issue with only 15.0.6.
> > >>>>>>
> > >>>>>> Interesting, that complicates things, probably the best way then is to
> > >>>>>> try to build it, if it fails, mention that 15.0.6 is known to be
> > >>>>>> problematic and suggest working versions?
> > >>>>>>
> > >>>>>> - Arnaldo
> > >>>>>
> > >>>>> I still wasn't able to reproduce it with 15.0.6. And I double checked
> > >>>>> with V=1 that the build was using the right clang. I suppose it could be
> > >>>>> a build configuration issue, or maybe with a different kernel version?
> > >>>>
> > >>>>
> > >>>> Howard? If not reproducible we can revert it.
> > >>>>
> > >>>> - Arnaldo
> > >>>>
> > >>>>
> > >>>>>
> > >>>>> $  uname --kernel-release
> > >>>>> 6.8.0-76060800daily20240311-generic
> > >>>>>
> > >>>>> $ ../../llvm-project/build/bin/clang -v
> > >>>>> clang version 15.0.6 (https://github.com/llvm/llvm-project.git
> > >>>>> 088f33605d8a61ff519c580a71b1dd57d16a03f8)
> > >>>>> Target: x86_64-unknown-linux-gnu
> > >>>>> Thread model: posix
> > >>>>> InstalledDir: /home/james/workspace/linux/linux/../../llvm-project/build/bin
> > >>>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
> > >>>>> Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> > >>>>> Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
> > >>>>> Candidate multilib: .;@m64
> > >>>>> Selected multilib: .;@m64
> > >>>>>
> > >>>>> $ git log
> > >>>>> commit 003265bb6f028d7bcd7cbd92d6ba2b4e26382796
> > >>>>> (perf-tools-next/perf-tools-next)
> > >>>>>
> > >>>>> $ make O=../build/local/ CLANG=../../llvm-project/build/bin/clang -C \
> > >>>>>    tools/perf
> > >>>>>
> > >>>>> $ perf trace -e write --max-events=1
> > >>>>>
> > >>>>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> > >>>>>         0x7fffa102d9b0, count: 8)                           = 8
> > >>>
> > >>> Thanks for the effort! But this output seems to be unaugmented? I
> > >>> think you are testing the perf/perf-tools-next branch that's slightly
> > >>> older (perf mem: Fix the wrong reference in parse_record_events()
> > >>> 003265bb6f028d7bcd7cbd92d6ba2b4e26382796), and I think that doesn't
> > >>> have the new perf trace feature yet...
> > >>>
> > >>> That's why the perf trace output looks like:
> > >>>
> > >>>>> $ perf trace -e write --max-events=1
> > >>>>>
> > >>>>>       0.000 ( 0.026 ms): gnome-shell/5454 write(fd: 5, buf:
> > >>>>>         0x7fffa102d9b0, count: 8)
> > >>>
> > >>> With the new buffer augmentation it should look like this:
> > >>>
> > >>> perf $ ./perf trace -e write --max-events=1
> > >>>      0.000 ( 0.011 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> > >>> count: 8)                         =
> > >>>
> > >>> The new perf trace can be found on
> > >>> https://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/
> > >>> perf-tools-next
> > >>>
> > >>> perf $ git log
> > >>> commit 1de5b5dcb8353f36581c963df2d359a5f151a0be (HEAD ->
> > >>> struct-aug-arnaldo-2, perf/perf-tools-next)
> > >>>
> > >>> Also, if you want to test the clang, you may want to checkout to the
> > >>> commit before the clang-16 guard (otherwise you can't test different
> > >>> clang version).
> > >>>
> > >>> perf $ git log --oneline
> > >>> 1de5b5dcb835 (HEAD -> struct-aug-arnaldo-2, perf/perf-tools-next) perf
> > >>> trace: Mark the 'head' arg in the set_robust_list syscall as coming
> > >>> from user space
> > >>> 0c1019e3463b perf trace: Mark the 'rseq' arg in the rseq syscall as
> > >>> coming from user space
> > >>> edf3ce0ed38e perf env: Find correct branch counter info on hybrid
> > >>> 9953807c9e01 perf evlist: Print hint for group
> > >>> eb9b9a6f5ab3 tools: Drop nonsensical -O6
> > >>> 89c0a55e550e perf pmu: To info add event_type_desc
> > >>> f08cc258431d perf evsel: Add accessor for tool_event
> > >>> 925320737ae2 perf pmus: Fake PMU clean up
> > >>> d3d5c1a00fcd perf list: Avoid potential out of bounds memory read
> > >>> 4ae354d73a8e perf help: Fix a typo ("bellow")
> > >>> 74298dd8acb8 perf ftrace: Detect whether ftrace is enabled on system
> > >>> 83420d5f5863 perf test shell probe_vfs_getname: Remove extraneous '='
> > >>> from probe line number regex
> > >>> 9327f0ecad48 perf build: Require at least clang 16.0.6 to build BPF
> > >>> skeletons. **********[[[[[[[[[[[[!!!!(This is the clang
> > >>> guard)]]]]]]]]]]*****
> > >>> 4c1af9bf97eb perf trace: If a syscall arg is marked as 'const', assume
> > >>> it is coming _from_ userspace
> > >>> e37b315c17df perf parse-events: Remove duplicated include in parse-events.c
> > >>>
> > >>> My machine is debian 12, and I cannot build perf trace with the new
> > >>> augmentation feature on clang-13, clang-14, and clang-15.
> > >>>
> > >>> perf $ uname -r
> > >>> 6.11.0-061100rc7-generic
> > >>>
> > >>> perf $ git log
> > >>> commit 4c1af9bf97eb56d069421c3233ce61608458d5c8 (HEAD)
> > >>>
> > >>> perf $ make CLANG=/bin/clang-13 && ./perf trace -e write --max-events=1
> > >>> ...
> > >>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > >>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > >>> 0: R1=ctx() R10=fp0
> > >>> ; int sys_enter(struct syscall_enter_args *args) @
> > >>> augmented_raw_syscalls.bpf.c:509
> > >>> 0: (bf) r7 = r1                       ; R1=ctx() R7_w=ctx()
> > >>>
> > >>>
> > >>> perf $ make CLANG=/bin/clang-14 && ./perf trace -e write --max-events=1
> > >>> ...
> > >>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > >>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > >>> 0: R1=ctx() R10=fp0
> > >>> ; int sys_enter(struct syscall_enter_args *args) @
> > >>> augmented_raw_syscalls.bpf.c:509
> > >>>
> > >>>
> > >>> perf $ make CLANG=/bin/clang-15 && ./perf trace -e write --max-events=1
> > >>> libbpf: prog 'sys_enter': BPF program load failed: Invalid argument
> > >>> libbpf: prog 'sys_enter': -- BEGIN PROG LOAD LOG --
> > >>> 0: R1=ctx() R10=fp0
> > >>>
> > >>>
> > >>> perf $ make CLANG=/bin/clang-16 && ./perf trace -e write --max-events=1
> > >>>      0.000 ( 0.010 ms): gmain/1408 write(fd: 4, buf: \1\0\0\0\0\0\0\0,
> > >>> count: 8)                         =
> > >>>
> > >>>
> > >>> Anyway thank you so much, I'll try to reproduce it on Ubuntu, like you
> > >>> mentioned.
> > >>>
> > >>> Thanks,
> > >>> Howard
> > >>>
> > >>>
> > >>>>>
> >

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Namhyung Kim]

On Tue, Oct 08, 2024 at 12:27:24AM -0700, Howard Chu wrote:
> Hi Alan, Arnaldo and James,
> 
> This problem was solved in [PATCH 0/2] perf trace: Fix support for the
> new BPF feature in clang 12 (Link:
> https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-howardchu95@gmail.com/T/#t),
> sorry I forgot to cc you two.
> 
> Alan's thought was correct. Thank you so much! :)

It'd be great if any of you can test this change.  Now I only have
machines with clang 16.

Thanks,
Namhyung

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[James Clark]

On 10/10/2024 2:20 am, Arnaldo Carvalho de Melo wrote:
> On Wed, Oct 9, 2024, 10:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
> 
>> On Tue, Oct 08, 2024 at 12:27:24AM -0700, Howard Chu wrote:
>>> Hi Alan, Arnaldo and James,
>>>
>>> This problem was solved in [PATCH 0/2] perf trace: Fix support for the
>>> new BPF feature in clang 12 (Link:
>>>
>> https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-howardchu95@gmail.com/T/#t
>> ),
>>> sorry I forgot to cc you two.
>>>
>>> Alan's thought was correct. Thank you so much! :)
>>
>> It'd be great if any of you can test this change.  Now I only have
>> machines with clang 16.
>>
> 
> I'll test this tomorrow.
> 
> - Arnaldo
> 
>>
>> Thanks,
>> Namhyung
>>
>>
> 

Tested with clang 15:

$ sudo perf trace -e write --max-events=100 -- echo hello

   0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6) 

                                            =

Tested-by: James Clark <james.clark@linaro.org>


Unrelated to this change, I noticed that with older clangs or with 
BUILD_BPF_SKEL=0 that commit b257fac12f38 ("perf trace: Pretty print 
buffer data") changes the buffer address to print nothing, and the '6' 
return value is missing. Not sure if this was intended or not:

  $ sudo perf trace -e write --max-events=100 -- echo hello

Before:
      0.000 ( 0.009 ms): echo/772951 write(fd: 1, buf: 0x58c415257440,
         count: 6)                           = 6

After:
      0.000 ( 0.009 ms): echo/760370 write(fd: 1, buf: , count: 6) 

                                             =

Thanks
James

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[James Clark]

On 10/10/2024 10:00 am, James Clark wrote:
> 
> 
> On 10/10/2024 2:20 am, Arnaldo Carvalho de Melo wrote:
>> On Wed, Oct 9, 2024, 10:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
>>
>>> On Tue, Oct 08, 2024 at 12:27:24AM -0700, Howard Chu wrote:
>>>> Hi Alan, Arnaldo and James,
>>>>
>>>> This problem was solved in [PATCH 0/2] perf trace: Fix support for the
>>>> new BPF feature in clang 12 (Link:
>>>>
>>> https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1- 
>>> howardchu95@gmail.com/T/#t
>>> ),
>>>> sorry I forgot to cc you two.
>>>>
>>>> Alan's thought was correct. Thank you so much! :)
>>>
>>> It'd be great if any of you can test this change.  Now I only have
>>> machines with clang 16.
>>>
>>
>> I'll test this tomorrow.
>>
>> - Arnaldo
>>
>>>
>>> Thanks,
>>> Namhyung
>>>
>>>
>>
> 
> Tested with clang 15:
> 
> $ sudo perf trace -e write --max-events=100 -- echo hello
> 
>    0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6)
>                                             =
> 
> Tested-by: James Clark <james.clark@linaro.org>
> 

Should have left this tag on the actual patch. Will re-post it there.

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Namhyung Kim]

On Thu, Oct 10, 2024 at 10:11:35AM +0100, James Clark wrote:
> 
> 
> On 10/10/2024 10:00 am, James Clark wrote:
> > 
> > 
> > On 10/10/2024 2:20 am, Arnaldo Carvalho de Melo wrote:
> > > On Wed, Oct 9, 2024, 10:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > > 
> > > > On Tue, Oct 08, 2024 at 12:27:24AM -0700, Howard Chu wrote:
> > > > > Hi Alan, Arnaldo and James,
> > > > > 
> > > > > This problem was solved in [PATCH 0/2] perf trace: Fix support for the
> > > > > new BPF feature in clang 12 (Link:
> > > > > 
> > > > https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-
> > > > howardchu95@gmail.com/T/#t
> > > > ),
> > > > > sorry I forgot to cc you two.
> > > > > 
> > > > > Alan's thought was correct. Thank you so much! :)
> > > > 
> > > > It'd be great if any of you can test this change.  Now I only have
> > > > machines with clang 16.
> > > > 
> > > 
> > > I'll test this tomorrow.
> > > 
> > > - Arnaldo
> > > 
> > > > 
> > > > Thanks,
> > > > Namhyung
> > > > 
> > > > 
> > > 
> > 
> > Tested with clang 15:
> > 
> > $ sudo perf trace -e write --max-events=100 -- echo hello
> > 
> >    0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6)
> >                                             =
> > 
> > Tested-by: James Clark <james.clark@linaro.org>
> > 
> 
> Should have left this tag on the actual patch. Will re-post it there.
 
Right, thanks for taking care of this!
Namhyung

Re: [PATCH 1/1] perf build: Require at least clang 16.0.6 to build BPF skeletons

[Arnaldo Carvalho de Melo]

On Thu, Oct 10, 2024 at 10:00:41AM +0100, James Clark wrote:
> 
> 
> On 10/10/2024 2:20 am, Arnaldo Carvalho de Melo wrote:
> > On Wed, Oct 9, 2024, 10:01 PM Namhyung Kim <namhyung@kernel.org> wrote:
> > 
> > > On Tue, Oct 08, 2024 at 12:27:24AM -0700, Howard Chu wrote:
> > > > Hi Alan, Arnaldo and James,
> > > > 
> > > > This problem was solved in [PATCH 0/2] perf trace: Fix support for the
> > > > new BPF feature in clang 12 (Link:
> > > > 
> > > https://lore.kernel.org/linux-perf-users/20241007051414.2995674-1-howardchu95@gmail.com/T/#t
> > > ),
> > > > sorry I forgot to cc you two.
> > > > 
> > > > Alan's thought was correct. Thank you so much! :)
> > > 
> > > It'd be great if any of you can test this change.  Now I only have
> > > machines with clang 16.
> > > 
> > 
> > I'll test this tomorrow.
> > 
> > - Arnaldo
> > 
> > > 
> > > Thanks,
> > > Namhyung
> > > 
> > > 
> > 
> 
> Tested with clang 15:
> 
> $ sudo perf trace -e write --max-events=100 -- echo hello
> 
>   0.000 ( 0.014 ms): echo/834165 write(fd: 1, buf: hello\10, count: 6)
> 
>                                            =
> 
> Tested-by: James Clark <james.clark@linaro.org>
> 
> 
> Unrelated to this change, I noticed that with older clangs or with
> BUILD_BPF_SKEL=0 that commit b257fac12f38 ("perf trace: Pretty print buffer
> data") changes the buffer address to print nothing, and the '6' return value
> is missing. Not sure if this was intended or not:
> 
>  $ sudo perf trace -e write --max-events=100 -- echo hello
> 
> Before:
>      0.000 ( 0.009 ms): echo/772951 write(fd: 1, buf: 0x58c415257440,
>         count: 6)                           = 6
> 
> After:
>      0.000 ( 0.009 ms): echo/760370 write(fd: 1, buf: , count: 6)
> 

I noticed this with fedora:40, and bisected it to:

⬢[acme@toolbox perf-tools]$ git bisect good
b257fac12f38d7f503b932313d704cee21092350 is the first bad commit
commit b257fac12f38d7f503b932313d704cee21092350
Author: Howard Chu <howardchu95@gmail.com>
Date:   Sun Aug 25 00:33:19 2024 +0800

    perf trace: Pretty print buffer data
    
    Define TRACE_AUG_MAX_BUF in trace_augment.h data, which is the maximum
    buffer size we can augment. BPF will include this header too.
    
    Print buffer in a way that's different than just printing a string, we
    print all the control characters in \digits (such as \0 for null, and
    \10 for newline, LF).
    
    For character that has a bigger value than 127, we print the digits
    instead of the character itself as well.
    
    Committer notes:
    
    Simplified the buffer scnprintf to avoid using multiple buffers as
    discussed in the patch review thread.
    
    We can't really all 'buf' args to SCA_BUF as we're collecting so far
    just on the sys_enter path, so we would be printing the previous 'read'
    arg buffer contents, not what the kernel puts there.
    
    So instead of:
       static int syscall_fmt__cmp(const void *name, const void *fmtp)
      @@ -1987,8 +1989,6 @@ syscall_arg_fmt__init_array(struct syscall_arg_fmt *arg, struct tep_format_field
      -               else if (strstr(field->type, "char *") && strstr(field->name, "buf"))
      -                       arg->scnprintf = SCA_BUF;
    
    Do:
    
    static const struct syscall_fmt syscall_fmts[] = {
      +       { .name     = "write",      .errpid = true,
      +         .arg = { [1] = { .scnprintf = SCA_BUF /* buf */, from_user = true, }, }, },
    
    Signed-off-by: Howard Chu <howardchu95@gmail.com>
    Cc: Adrian Hunter <adrian.hunter@intel.com>
    Cc: Ian Rogers <irogers@google.com>
    Cc: Jiri Olsa <jolsa@kernel.org>
    Cc: Kan Liang <kan.liang@linux.intel.com>
    Cc: Namhyung Kim <namhyung@kernel.org>
    Link: https://lore.kernel.org/r/20240815013626.935097-8-howardchu95@gmail.com
    Link: https://lore.kernel.org/r/20240824163322.60796-6-howardchu95@gmail.com
    Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>


Just did a:

⬢[acme@toolbox perf-tools]$ git checkout b257fac12f38d7f503b932313d704cee21092350
Updating files: 100% (12326/12326), done.
Note: switching to 'b257fac12f38d7f503b932313d704cee21092350'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
<SNIP>
HEAD is now at b257fac12f38d7f5 perf trace: Pretty print buffer


make clean + rebuild and the returns are gone, if I do:

⬢[acme@toolbox perf-tools]$ git checkout b257fac12f38d7f503b932313d704cee21092350^
Previous HEAD position was b257fac12f38d7f5 perf trace: Pretty print buffer data
HEAD is now at cb32035214b9a09d perf trace: Pretty print struct data
⬢[acme@toolbox perf-tools]$ 

clean + rebuild instead I get those returns back:

Now staring at it...


- Arnaldo