Problem with BTF generation on mips64el

Problem with BTF generation on mips64el

[Tony Ambardar]

[-- Attachment #1: Type: text/plain, Size: 5491 bytes --]

Hello,

For some time now I'm seeing multiple issues during BTF generation while
building recent kernels targeting mips64el, and would appreciate some help
to understand and fix the problems.

Some relate to resolve_btfids:

>   LD      vmlinux
>   BTFIDS  vmlinux
> WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_key_put
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>   NM      System.map
>   SORTTAB vmlinux
>   OBJCOPY vmlinux.32

These do not appear to be #ifdef-related and have similar past reports [1].

I also see many pahole failures during BTF encoding of modules, such as:

>   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
>   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
>   CC [M]  net/psample/psample.mod.o
>   LD [M]  crypto/cmac.ko
>   BTF [M] crypto/cmac.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
>   LD [M]  lib/test_bpf.ko
>   BTF [M] lib/test_bpf.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
>   LD [M]  lib/crc-ccitt.ko
>   BTF [M] lib/crc-ccitt.ko
> die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> die__process_unit: tag not supported 0x11 (compile_unit)!
> die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
>   LD [M]  lib/libcrc32c.ko
>   BTF [M] lib/libcrc32c.ko
> die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> die__process_unit: tag not supported 0x11 (compile_unit)!
> die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
>   LD [M]  lib/ts_kmp.ko
>   BTF [M] lib/ts_kmp.ko
>   LD [M]  lib/ts_bm.ko
>   BTF [M] lib/ts_bm.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!

I have seen reports of various similar "die__" messages on the dwarves
list and repo, with the hint of an elfutils connection [2] but nothing
conclusive.

Details of the git commit and build environment are as follows:

> $ git log -1 --oneline  bpf/master
> 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> test_sk_storage_tracing.c
> 
> $ lsb_release -a
> Description:    Ubuntu 22.04.4 LTS
> 
> $ cat gcc-compile.txt
> ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> 
> $ mips64el-linux-gnuabi64-gcc --version
> mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> 
> $ mips64el-linux-gnuabi64-ld --version
> GNU ld (GNU Binutils for Ubuntu) 2.38
> 
> $ pahole --version
> v1.26
> 
> $ ldd $(which pahole)
>         linux-vdso.so.1 (0x00007fff16f3f000)
>         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
>         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
>         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
>         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> 
> $ dpkg -s elfutils
> Package: elfutils
> ...
> Version: 0.186-1build1
> Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)

For reference, I also attached the full .config and build log from the
above.

I should add this is not only a problem with the latest bpf/master but
also appears to affect the 6.6.x LTS kernel, which I tested while building
a mips64el OpenWrt distro image. That build environment employs the latest
gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.

Not only do I see similar warnings from resolve_btfids and pahole, but
while running the distro image I encounter module loading failures that
suggest ELF corruption in some module .ko files, based on the following:

> root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> ...
> init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> ...

> $ man init_module
> ...
> The following errors may additionally occur for init_module():
> ...
>      EINVAL param_values is invalid, or some part of the ELF image in
>      module_image contains inconsistencies.
> ...

I'd be grateful if some of the BTF/pahole experts could please review this
issue and share next steps or other details I might provide.

Thanks,
Tony Ambardar

Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
Link: https://github.com/acmel/dwarves/issues/45 [2]

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 22159 bytes --]

[-- Attachment #3: mips64el-linux-gnuabi64.log.gz --]
[-- Type: application/gzip, Size: 20880 bytes --]

Re: Problem with BTF generation on mips64el

[Hengqi Chen]

Hi Tony,

On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> Hello,
>
> For some time now I'm seeing multiple issues during BTF generation while
> building recent kernels targeting mips64el, and would appreciate some help
> to understand and fix the problems.
>
> Some relate to resolve_btfids:
>
> >   LD      vmlinux
> >   BTFIDS  vmlinux
> > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> >   NM      System.map
> >   SORTTAB vmlinux
> >   OBJCOPY vmlinux.32
>
> These do not appear to be #ifdef-related and have similar past reports [1].
>
> I also see many pahole failures during BTF encoding of modules, such as:
>
> >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> >   CC [M]  net/psample/psample.mod.o
> >   LD [M]  crypto/cmac.ko
> >   BTF [M] crypto/cmac.ko
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!

The issue seems to be related to elfutils. Have you tried build from
the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the
reloc_simple_type hook.

> >   LD [M]  lib/test_bpf.ko
> >   BTF [M] lib/test_bpf.ko
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!
> >   LD [M]  lib/crc-ccitt.ko
> >   BTF [M] lib/crc-ccitt.ko
> > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> > die__process_unit: tag not supported 0x11 (compile_unit)!
> > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> >   LD [M]  lib/libcrc32c.ko
> >   BTF [M] lib/libcrc32c.ko
> > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> > die__process_unit: tag not supported 0x11 (compile_unit)!
> > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> >   LD [M]  lib/ts_kmp.ko
> >   BTF [M] lib/ts_kmp.ko
> >   LD [M]  lib/ts_bm.ko
> >   BTF [M] lib/ts_bm.ko
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!
>
> I have seen reports of various similar "die__" messages on the dwarves
> list and repo, with the hint of an elfutils connection [2] but nothing
> conclusive.
>
> Details of the git commit and build environment are as follows:
>
> > $ git log -1 --oneline  bpf/master
> > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > test_sk_storage_tracing.c
> >
> > $ lsb_release -a
> > Description:    Ubuntu 22.04.4 LTS
> >
> > $ cat gcc-compile.txt
> > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> >
> > $ mips64el-linux-gnuabi64-gcc --version
> > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> >
> > $ mips64el-linux-gnuabi64-ld --version
> > GNU ld (GNU Binutils for Ubuntu) 2.38
> >
> > $ pahole --version
> > v1.26
> >
> > $ ldd $(which pahole)
> >         linux-vdso.so.1 (0x00007fff16f3f000)
> >         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
> >         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
> >         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
> >         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
> >         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
> >         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
> >         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> >
> > $ dpkg -s elfutils
> > Package: elfutils
> > ...
> > Version: 0.186-1build1
> > Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> > libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
>
> For reference, I also attached the full .config and build log from the
> above.
>
> I should add this is not only a problem with the latest bpf/master but
> also appears to affect the 6.6.x LTS kernel, which I tested while building
> a mips64el OpenWrt distro image. That build environment employs the latest
> gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
>
> Not only do I see similar warnings from resolve_btfids and pahole, but
> while running the distro image I encounter module loading failures that
> suggest ELF corruption in some module .ko files, based on the following:
>
> > root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> > ...
> > init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> > ...
>
> > $ man init_module
> > ...
> > The following errors may additionally occur for init_module():
> > ...
> >      EINVAL param_values is invalid, or some part of the ELF image in
> >      module_image contains inconsistencies.
> > ...
>
> I'd be grateful if some of the BTF/pahole experts could please review this
> issue and share next steps or other details I might provide.
>
> Thanks,
> Tony Ambardar
>
> Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> Link: https://github.com/acmel/dwarves/issues/45 [2]

Cheers,
Hengqi

Re: Problem with BTF generation on mips64el

[Jiri Olsa]

On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> Hi Tony,
> 
> On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > Hello,
> >
> > For some time now I'm seeing multiple issues during BTF generation while
> > building recent kernels targeting mips64el, and would appreciate some help
> > to understand and fix the problems.
> >
> > Some relate to resolve_btfids:
> >
> > >   LD      vmlinux
> > >   BTFIDS  vmlinux
> > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > >   NM      System.map
> > >   SORTTAB vmlinux
> > >   OBJCOPY vmlinux.32
> >
> > These do not appear to be #ifdef-related and have similar past reports [1].

I can reproduce the warning just for bpf_session_cookie,
which has fix in progress:
  https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t

> >
> > I also see many pahole failures during BTF encoding of modules, such as:
> >
> > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > >   CC [M]  net/psample/psample.mod.o
> > >   LD [M]  crypto/cmac.ko
> > >   BTF [M] crypto/cmac.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> The issue seems to be related to elfutils. Have you tried build from
> the latest elfutils source ?
> I saw the latest MIPS backend in elfutils already implemented the
> reloc_simple_type hook.

hi,
+1, could you also check the pahole version you used?

jirka

> 
> > >   LD [M]  lib/test_bpf.ko
> > >   BTF [M] lib/test_bpf.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > >   LD [M]  lib/crc-ccitt.ko
> > >   BTF [M] lib/crc-ccitt.ko
> > > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> > > die__process_unit: tag not supported 0x11 (compile_unit)!
> > > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> > >   LD [M]  lib/libcrc32c.ko
> > >   BTF [M] lib/libcrc32c.ko
> > > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> > > die__process_unit: tag not supported 0x11 (compile_unit)!
> > > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> > >   LD [M]  lib/ts_kmp.ko
> > >   BTF [M] lib/ts_kmp.ko
> > >   LD [M]  lib/ts_bm.ko
> > >   BTF [M] lib/ts_bm.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> >
> > I have seen reports of various similar "die__" messages on the dwarves
> > list and repo, with the hint of an elfutils connection [2] but nothing
> > conclusive.
> >
> > Details of the git commit and build environment are as follows:
> >
> > > $ git log -1 --oneline  bpf/master
> > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > test_sk_storage_tracing.c
> > >
> > > $ lsb_release -a
> > > Description:    Ubuntu 22.04.4 LTS
> > >
> > > $ cat gcc-compile.txt
> > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > >
> > > $ mips64el-linux-gnuabi64-gcc --version
> > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > >
> > > $ mips64el-linux-gnuabi64-ld --version
> > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > >
> > > $ pahole --version
> > > v1.26
> > >
> > > $ ldd $(which pahole)
> > >         linux-vdso.so.1 (0x00007fff16f3f000)
> > >         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
> > >         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
> > >         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
> > >         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
> > >         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
> > >         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
> > >         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> > >
> > > $ dpkg -s elfutils
> > > Package: elfutils
> > > ...
> > > Version: 0.186-1build1
> > > Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> > > libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
> >
> > For reference, I also attached the full .config and build log from the
> > above.
> >
> > I should add this is not only a problem with the latest bpf/master but
> > also appears to affect the 6.6.x LTS kernel, which I tested while building
> > a mips64el OpenWrt distro image. That build environment employs the latest
> > gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
> >
> > Not only do I see similar warnings from resolve_btfids and pahole, but
> > while running the distro image I encounter module loading failures that
> > suggest ELF corruption in some module .ko files, based on the following:
> >
> > > root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> > > ...
> > > init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> > > ...
> >
> > > $ man init_module
> > > ...
> > > The following errors may additionally occur for init_module():
> > > ...
> > >      EINVAL param_values is invalid, or some part of the ELF image in
> > >      module_image contains inconsistencies.
> > > ...
> >
> > I'd be grateful if some of the BTF/pahole experts could please review this
> > issue and share next steps or other details I might provide.
> >
> > Thanks,
> > Tony Ambardar
> >
> > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > Link: https://github.com/acmel/dwarves/issues/45 [2]
> 
> Cheers,
> Hengqi
> 

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

Hi Jiri,

On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > Hi Tony,
> > 
> > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > For some time now I'm seeing multiple issues during BTF generation while
> > > building recent kernels targeting mips64el, and would appreciate some help
> > > to understand and fix the problems.
> > >
> > > Some relate to resolve_btfids:
> > >
> > > >   LD      vmlinux
> > > >   BTFIDS  vmlinux
> > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > >   NM      System.map
> > > >   SORTTAB vmlinux
> > > >   OBJCOPY vmlinux.32
> > >
> > > These do not appear to be #ifdef-related and have similar past reports [1].
> 
> I can reproduce the warning just for bpf_session_cookie,
> which has fix in progress:
>   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t

I gather there are different root causes for these warnings. From the link,
the issue with bpf_session_cookie seems related to conditional compilation,
which have come up before on the mailing list IIRC.

In comparison, consider my above warning for bpf_key_put, which is defined
in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
enabled in my config and so not the issue. I can in fact see the global
text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.

So I suspect the warnings might come from linker or optimization problems,
or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?

> 
> > >
> > > I also see many pahole failures during BTF encoding of modules, such as:
> > >
> > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > >   CC [M]  net/psample/psample.mod.o
> > > >   LD [M]  crypto/cmac.ko
> > > >   BTF [M] crypto/cmac.ko
> > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > 
> > The issue seems to be related to elfutils. Have you tried build from
> > the latest elfutils source ?
> > I saw the latest MIPS backend in elfutils already implemented the
> > reloc_simple_type hook.
> 
> hi,
> +1, could you also check the pahole version you used?

No luck I'm afraid with using the latest elfutils as suggested.

I used pahole v1.26 in my original testing for this bug report, as noted
below (might have been buried a bit).

> 
> jirka
> 
> SNIP
>
> > > Details of the git commit and build environment are as follows:
> > >
> > > > $ git log -1 --oneline  bpf/master
> > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > test_sk_storage_tracing.c
> > > >
> > > > $ lsb_release -a
> > > > Description:    Ubuntu 22.04.4 LTS
> > > >
> > > > $ cat gcc-compile.txt
> > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > >
> > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > >
> > > > $ mips64el-linux-gnuabi64-ld --version
> > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > >
> > > > $ pahole --version
> > > > v1.26
> 
> SNIP
>
> > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > issue and share next steps or other details I might provide.
> > >
> > > Thanks,
> > > Tony Ambardar
> > >
> > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > 
> > Cheers,
> > Hengqi

Thanks,
Tony

Re: Problem with BTF generation on mips64el

[Jiri Olsa]

On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> Hi Jiri,
> 
> On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > > Some relate to resolve_btfids:
> > > >
> > > > >   LD      vmlinux
> > > > >   BTFIDS  vmlinux
> > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > >   NM      System.map
> > > > >   SORTTAB vmlinux
> > > > >   OBJCOPY vmlinux.32
> > > >
> > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > 
> > I can reproduce the warning just for bpf_session_cookie,
> > which has fix in progress:
> >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> 
> I gather there are different root causes for these warnings. From the link,
> the issue with bpf_session_cookie seems related to conditional compilation,
> which have come up before on the mailing list IIRC.
> 
> In comparison, consider my above warning for bpf_key_put, which is defined
> in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> enabled in my config and so not the issue. I can in fact see the global
> text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.

yes, that would be a problem, the resolve_btfids goes through the BTF
lists and needs to resolve the function in vmlinux, when it's not there,
it will output the warning above

> 
> So I suspect the warnings might come from linker or optimization problems,
> or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?

I can reproduce now resolve_btfids warnings:

  LD      vmlinux
  BTFIDS  vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_is_return
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages

when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
the __bpf_kfunc flags do not work properly on this cross compile chain..
and linker just optimize it away?

#define __bpf_kfunc __used noinline

thanks,
jirka

> 
> > 
> > > >
> > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > 
> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > hi,
> > +1, could you also check the pahole version you used?
> 
> No luck I'm afraid with using the latest elfutils as suggested.
> 
> I used pahole v1.26 in my original testing for this bug report, as noted
> below (might have been buried a bit).
> 
> > 
> > jirka
> > 
> > SNIP
> >
> > > > Details of the git commit and build environment are as follows:
> > > >
> > > > > $ git log -1 --oneline  bpf/master
> > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > test_sk_storage_tracing.c
> > > > >
> > > > > $ lsb_release -a
> > > > > Description:    Ubuntu 22.04.4 LTS
> > > > >
> > > > > $ cat gcc-compile.txt
> > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > >
> > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > >
> > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > >
> > > > > $ pahole --version
> > > > > v1.26
> > 
> > SNIP
> >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> 
> Thanks,
> Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Fri, May 31, 2024 at 04:21:48PM +0200, Jiri Olsa wrote:
> On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> > Hi Jiri,
> > 
> > On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > Hi Tony,
> > > > 
> > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > to understand and fix the problems.
> > > > >
> > > > > Some relate to resolve_btfids:
> > > > >
> > > > > >   LD      vmlinux
> > > > > >   BTFIDS  vmlinux
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > > >   NM      System.map
> > > > > >   SORTTAB vmlinux
> > > > > >   OBJCOPY vmlinux.32
> > > > >
> > > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > > 
> > > I can reproduce the warning just for bpf_session_cookie,
> > > which has fix in progress:
> > >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> > 
> > I gather there are different root causes for these warnings. From the link,
> > the issue with bpf_session_cookie seems related to conditional compilation,
> > which have come up before on the mailing list IIRC.
> > 
> > In comparison, consider my above warning for bpf_key_put, which is defined
> > in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> > enabled in my config and so not the issue. I can in fact see the global
> > text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
> 
> yes, that would be a problem, the resolve_btfids goes through the BTF
> lists and needs to resolve the function in vmlinux, when it's not there,
> it will output the warning above
> 
> > 
> > So I suspect the warnings might come from linker or optimization problems,
> > or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
> 
> I can reproduce now resolve_btfids warnings:
> 

That's great to hear! Did you also use mips64el? And were you able to see
the various "die__proc:" errors from pahole during BTF generation?

>   LD      vmlinux
>   BTFIDS  vmlinux
> WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> WARN: resolve_btfids: unresolved symbol bpf_session_is_return
> WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_key_put
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> 
> when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
> the __bpf_kfunc flags do not work properly on this cross compile chain..
> and linker just optimize it away?

Yes, I verified the problem occurs during link-time garbage cleaning, when
LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.

> 
> #define __bpf_kfunc __used noinline
> 

After researching, I believe this isn't a mips problem specifically, rather
the __bpf_kfunc tag only partly works. It appears the __used macro works at
the compiler and IR-level but is ignored by the linker. I have a working
fix that also handles the linker level, so I'll post the patches shortly.

Unfortunately, that still leaves the pahole errors generating BTF, which
are serious as they seem to break module loading.

> thanks,
> jirka

Cheers,
Tony
> 
> > 
> > > 
> > > > >
> > > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > > >
> > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > >   LD [M]  crypto/cmac.ko
> > > > > >   BTF [M] crypto/cmac.ko
> > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > > 
> > > > The issue seems to be related to elfutils. Have you tried build from
> > > > the latest elfutils source ?
> > > > I saw the latest MIPS backend in elfutils already implemented the
> > > > reloc_simple_type hook.
> > > 
> > > hi,
> > > +1, could you also check the pahole version you used?
> > 
> > No luck I'm afraid with using the latest elfutils as suggested.
> > 
> > I used pahole v1.26 in my original testing for this bug report, as noted
> > below (might have been buried a bit).
> > 
> > > 
> > > jirka
> > > 
> > > SNIP
> > >
> > > > > Details of the git commit and build environment are as follows:
> > > > >
> > > > > > $ git log -1 --oneline  bpf/master
> > > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > > test_sk_storage_tracing.c
> > > > > >
> > > > > > $ lsb_release -a
> > > > > > Description:    Ubuntu 22.04.4 LTS
> > > > > >
> > > > > > $ cat gcc-compile.txt
> > > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > > >
> > > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > > >
> > > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > > >
> > > > > > $ pahole --version
> > > > > > v1.26
> > > 
> > > SNIP
> > >
> > > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > > issue and share next steps or other details I might provide.
> > > > >
> > > > > Thanks,
> > > > > Tony Ambardar
> > > > >
> > > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > > 
> > > > Cheers,
> > > > Hengqi
> > 
> > Thanks,
> > Tony

Re: Problem with BTF generation on mips64el

[Jiri Olsa]

On Mon, Jun 03, 2024 at 02:02:45AM -0700, Tony Ambardar wrote:
> On Fri, May 31, 2024 at 04:21:48PM +0200, Jiri Olsa wrote:
> > On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> > > Hi Jiri,
> > > 
> > > On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > > Hi Tony,
> > > > > 
> > > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > > to understand and fix the problems.
> > > > > >
> > > > > > Some relate to resolve_btfids:
> > > > > >
> > > > > > >   LD      vmlinux
> > > > > > >   BTFIDS  vmlinux
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > > > >   NM      System.map
> > > > > > >   SORTTAB vmlinux
> > > > > > >   OBJCOPY vmlinux.32
> > > > > >
> > > > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > > > 
> > > > I can reproduce the warning just for bpf_session_cookie,
> > > > which has fix in progress:
> > > >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> > > 
> > > I gather there are different root causes for these warnings. From the link,
> > > the issue with bpf_session_cookie seems related to conditional compilation,
> > > which have come up before on the mailing list IIRC.
> > > 
> > > In comparison, consider my above warning for bpf_key_put, which is defined
> > > in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> > > enabled in my config and so not the issue. I can in fact see the global
> > > text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
> > 
> > yes, that would be a problem, the resolve_btfids goes through the BTF
> > lists and needs to resolve the function in vmlinux, when it's not there,
> > it will output the warning above
> > 
> > > 
> > > So I suspect the warnings might come from linker or optimization problems,
> > > or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
> > 
> > I can reproduce now resolve_btfids warnings:
> > 
> 
> That's great to hear! Did you also use mips64el? And were you able to see

nope, just mips64, couldn't find mips64el on fedora

> the various "die__proc:" errors from pahole during BTF generation?

yep, I can see them as well

> 
> >   LD      vmlinux
> >   BTFIDS  vmlinux
> > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > WARN: resolve_btfids: unresolved symbol bpf_session_is_return
> > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > 
> > when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
> > the __bpf_kfunc flags do not work properly on this cross compile chain..
> > and linker just optimize it away?
> 
> Yes, I verified the problem occurs during link-time garbage cleaning, when
> LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.
> 
> > 
> > #define __bpf_kfunc __used noinline
> > 
> 
> After researching, I believe this isn't a mips problem specifically, rather
> the __bpf_kfunc tag only partly works. It appears the __used macro works at
> the compiler and IR-level but is ignored by the linker. I have a working
> fix that also handles the linker level, so I'll post the patches shortly.

great, I was wondering how to fix that

thanks,
jirka

> 
> Unfortunately, that still leaves the pahole errors generating BTF, which
> are serious as they seem to break module loading.
> 
> > thanks,
> > jirka
> 
> Cheers,
> Tony
> > 
> > > 
> > > > 
> > > > > >
> > > > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > > > >
> > > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > > >   LD [M]  crypto/cmac.ko
> > > > > > >   BTF [M] crypto/cmac.ko
> > > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > > > 
> > > > > The issue seems to be related to elfutils. Have you tried build from
> > > > > the latest elfutils source ?
> > > > > I saw the latest MIPS backend in elfutils already implemented the
> > > > > reloc_simple_type hook.
> > > > 
> > > > hi,
> > > > +1, could you also check the pahole version you used?
> > > 
> > > No luck I'm afraid with using the latest elfutils as suggested.
> > > 
> > > I used pahole v1.26 in my original testing for this bug report, as noted
> > > below (might have been buried a bit).
> > > 
> > > > 
> > > > jirka
> > > > 
> > > > SNIP
> > > >
> > > > > > Details of the git commit and build environment are as follows:
> > > > > >
> > > > > > > $ git log -1 --oneline  bpf/master
> > > > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > > > test_sk_storage_tracing.c
> > > > > > >
> > > > > > > $ lsb_release -a
> > > > > > > Description:    Ubuntu 22.04.4 LTS
> > > > > > >
> > > > > > > $ cat gcc-compile.txt
> > > > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > > > >
> > > > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > > > >
> > > > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > > > >
> > > > > > > $ pahole --version
> > > > > > > v1.26
> > > > 
> > > > SNIP
> > > >
> > > > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > > > issue and share next steps or other details I might provide.
> > > > > >
> > > > > > Thanks,
> > > > > > Tony Ambardar
> > > > > >
> > > > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > > > 
> > > > > Cheers,
> > > > > Hengqi
> > > 
> > > Thanks,
> > > Tony

[PATCH bpf v1 0/2] bpf: Fix linker optimization removing kfuncs

[Tony Ambardar]

This patch series fixes unwanted stripping of kernel kfuncs during linker
optimization, as indicated by build warnings from resolve_btfids e.g.
"WARN: resolve_btfids: unresolved symbol ...". This can happen because the
__bpf_kfunc macro annotating kfunc declarations is ignored during linking.

Patch 1 adds support for the compiler attribute "__retain__", used to
avoid linker garbage cleanup. Patch 2 then updates __bpf_kfunc to use this
attribute when LTO builds are enabled.


Tony Ambardar (2):
  Compiler Attributes: Add __retain macro
  bpf: Harden __bpf_kfunc tag against linker kfunc removal

 include/linux/btf.h                 |  2 +-
 include/linux/compiler_attributes.h | 14 ++++++++++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

-- 
2.34.1

[PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[Tony Ambardar]

Some code includes the __used macro to prevent functions and data from
being optimized out. This macro implements __attribute__((__used__)), which
operates at the compiler and IR-level, and so still allows a linker to
remove objects intended to be kept.

Compilers supporting __attribute__((__retain__)) can address this gap by
setting the flag SHF_GNU_RETAIN on the section of a function/variable,
indicating to the linker the object should be retained. This attribute is
available since gcc 11, clang 13, and binutils 2.36.

Provide a __retain macro implementing __attribute__((__retain__)), whose
first user will be the '__bpf_kfunc' tag.

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/compiler_attributes.h | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h
index 32284cd26d52..1c22e1a734dc 100644
--- a/include/linux/compiler_attributes.h
+++ b/include/linux/compiler_attributes.h
@@ -326,6 +326,20 @@
  */
 #define __pure                          __attribute__((__pure__))
 
+/*
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
 /*
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-section-function-attribute
  *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Variable-Attributes.html#index-section-variable-attribute
-- 
2.34.1

Re: [PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[Miguel Ojeda]

On Mon, Jun 3, 2024 at 2:16 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> +#if __has_attribute(__retain__) && \
> +       (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> +        defined(CONFIG_LTO_CLANG))

Since this attribute depends on the configuration, please move it to
`compiler_types.h` instead.

Cheers,
Miguel

Re: [PATCH bpf v1 1/2] Compiler Attributes: Add __retain macro

[Tony Ambardar]

On Mon, Jun 03, 2024 at 03:57:38PM +0200, Miguel Ojeda wrote:
> On Mon, Jun 3, 2024 at 2:16 PM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > +#if __has_attribute(__retain__) && \
> > +       (defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> > +        defined(CONFIG_LTO_CLANG))
> 
> Since this attribute depends on the configuration, please move it to
> `compiler_types.h` instead.

Noted and thanks for clarifying; I'll change in v2.

Cheers,
Tony

> 
> Cheers,
> Miguel

[PATCH bpf v1 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Tony Ambardar]

BPF kfuncs are often not directly referenced and may be inadvertently
removed by optimization steps during kernel builds, thus the __bpf_kfunc
tag mitigates against this removal by including the __used macro. However,
this macro alone does not prevent removal during linking, and may still
yield build warnings (e.g. on mips64el):

    LD      vmlinux
    BTFIDS  vmlinux
  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
  WARN: resolve_btfids: unresolved symbol bpf_key_put
  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
    NM      System.map
    SORTTAB vmlinux
    OBJCOPY vmlinux.32

Update the __bpf_kfunc tag to better guard against linker optimization by
including the new __retain compiler macro, which fixes the warnings above.

Verify the __retain macro with readelf by checking object flags for 'R':

  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index f9e56fd12a9f..7c3e40c3295e 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \
-- 
2.34.1

[PATCH bpf v2 0/2] bpf: Fix linker optimization removing kfuncs

[Tony Ambardar]

This patch series fixes unwanted stripping of kernel kfuncs during linker
optimization, as indicated by build warnings from resolve_btfids e.g.
"WARN: resolve_btfids: unresolved symbol ...". This can happen because the
__bpf_kfunc macro annotating kfunc declarations is ignored during linking.

Patch 1 adds support for the compiler attribute "__retain__", used to
avoid linker garbage cleanup. Patch 2 then updates __bpf_kfunc to use this
attribute when LTO builds are enabled.

Build-tested locally against mips64el with gcc 13.3, and run through
upstream kernel-patches/bpf CI.

Tony Ambardar (2):
  compiler_types.h: Define __retain for __attribute__((__retain__))
  bpf: Harden __bpf_kfunc tag against linker kfunc removal

 include/linux/btf.h            |  2 +-
 include/linux/compiler_types.h | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)

-- 

v2:
- move __retain macro to compiler_types.h (Miguel Ojeda)

-- 
2.34.1

[PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Tony Ambardar]

Some code includes the __used macro to prevent functions and data from
being optimized out. This macro implements __attribute__((__used__)), which
operates at the compiler and IR-level, and so still allows a linker to
remove objects intended to be kept.

Compilers supporting __attribute__((__retain__)) can address this gap by
setting the flag SHF_GNU_RETAIN on the section of a function/variable,
indicating to the linker the object should be retained. This attribute is
available since gcc 11, clang 13, and binutils 2.36.

Provide a __retain macro implementing __attribute__((__retain__)), whose
first user will be the '__bpf_kfunc' tag.

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/compiler_types.h | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index 93600de3800b..f14c275950b5 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
 # define __preserve_most
 #endif
 
+/*
+ * Annotating a function/variable with __retain tells the compiler to place
+ * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
+ * instructs the linker to retain the object during garbage-cleanup or LTO
+ * phases.
+ *
+ * Note that the __used macro is also used to prevent functions or data
+ * being optimized out, but operates at the compiler/IR-level and may still
+ * allow unintended removal of objects during linking.
+ *
+ * Optional: only supported since gcc >= 11, clang >= 13
+ *
+ *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
+ * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
+ */
+#if __has_attribute(__retain__) && \
+	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
+	 defined(CONFIG_LTO_CLANG))
+# define __retain			__attribute__((__retain__))
+#else
+# define __retain
+#endif
+
 /* Compiler specific macros. */
 #ifdef __clang__
 #include <linux/compiler-clang.h>
-- 
2.34.1

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Yonghong Song]

On 6/3/24 10:23 PM, Tony Ambardar wrote:
> Some code includes the __used macro to prevent functions and data from
> being optimized out. This macro implements __attribute__((__used__)), which
> operates at the compiler and IR-level, and so still allows a linker to
> remove objects intended to be kept.
>
> Compilers supporting __attribute__((__retain__)) can address this gap by
> setting the flag SHF_GNU_RETAIN on the section of a function/variable,
> indicating to the linker the object should be retained. This attribute is
> available since gcc 11, clang 13, and binutils 2.36.
>
> Provide a __retain macro implementing __attribute__((__retain__)), whose
> first user will be the '__bpf_kfunc' tag.
>
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> ---
>   include/linux/compiler_types.h | 23 +++++++++++++++++++++++
>   1 file changed, 23 insertions(+)
>
> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> index 93600de3800b..f14c275950b5 100644
> --- a/include/linux/compiler_types.h
> +++ b/include/linux/compiler_types.h
> @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
>   # define __preserve_most
>   #endif
>   
> +/*
> + * Annotating a function/variable with __retain tells the compiler to place
> + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
> + * instructs the linker to retain the object during garbage-cleanup or LTO
> + * phases.
> + *
> + * Note that the __used macro is also used to prevent functions or data
> + * being optimized out, but operates at the compiler/IR-level and may still
> + * allow unintended removal of objects during linking.
> + *
> + * Optional: only supported since gcc >= 11, clang >= 13
> + *
> + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
> + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
> + */
> +#if __has_attribute(__retain__) && \
> +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> +	 defined(CONFIG_LTO_CLANG))

Could you explain why CONFIG_LTO_CLANG is added here?
IIUC, the __used macro permits garbage collection at section
level, so CLANG_LTO_CLANG without
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
shuold not change final section dynamics, right?

> +# define __retain			__attribute__((__retain__))
> +#else
> +# define __retain
> +#endif
> +
>   /* Compiler specific macros. */
>   #ifdef __clang__
>   #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Tony Ambardar]

On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
> 
> On 6/3/24 10:23 PM, Tony Ambardar wrote:
> > Some code includes the __used macro to prevent functions and data from
> > being optimized out. This macro implements __attribute__((__used__)), which
> > operates at the compiler and IR-level, and so still allows a linker to
> > remove objects intended to be kept.
> > 
> > Compilers supporting __attribute__((__retain__)) can address this gap by
> > setting the flag SHF_GNU_RETAIN on the section of a function/variable,
> > indicating to the linker the object should be retained. This attribute is
> > available since gcc 11, clang 13, and binutils 2.36.
> > 
> > Provide a __retain macro implementing __attribute__((__retain__)), whose
> > first user will be the '__bpf_kfunc' tag.
> > 
> > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > Cc: stable@vger.kernel.org # v6.6+
> > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> > ---
> >   include/linux/compiler_types.h | 23 +++++++++++++++++++++++
> >   1 file changed, 23 insertions(+)
> > 
> > diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> > index 93600de3800b..f14c275950b5 100644
> > --- a/include/linux/compiler_types.h
> > +++ b/include/linux/compiler_types.h
> > @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
> >   # define __preserve_most
> >   #endif
> > +/*
> > + * Annotating a function/variable with __retain tells the compiler to place
> > + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
> > + * instructs the linker to retain the object during garbage-cleanup or LTO
> > + * phases.
> > + *
> > + * Note that the __used macro is also used to prevent functions or data
> > + * being optimized out, but operates at the compiler/IR-level and may still
> > + * allow unintended removal of objects during linking.
> > + *
> > + * Optional: only supported since gcc >= 11, clang >= 13
> > + *
> > + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
> > + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
> > + */
> > +#if __has_attribute(__retain__) && \
> > +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> > +	 defined(CONFIG_LTO_CLANG))
> 
> Could you explain why CONFIG_LTO_CLANG is added here?
> IIUC, the __used macro permits garbage collection at section
> level, so CLANG_LTO_CLANG without
> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
> shuold not change final section dynamics, right?

Hi Yonghong,

I included the conditional guard to ensure consistent behaviour between
__retain and other features forcing split sections. In particular, the same
guard is used in vmlinux.lds.h to merge split sections where needed. For
example, using __retain in llvm builds without CONFIG_LTO was failing CI
tests on kernel-patches/bpf because the kernel didn't boot properly. And in
further testing, the kernel had no issues loading BPF kfunc modules with
such split sections, so I left the module (partial) linking scripts alone.

Maybe I misunderstand you question re: __used?

Thanks,
Tony
> 
> > +# define __retain			__attribute__((__retain__))
> > +#else
> > +# define __retain
> > +#endif
> > +
> >   /* Compiler specific macros. */
> >   #ifdef __clang__
> >   #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Yonghong Song]

On 6/10/24 3:56 PM, Tony Ambardar wrote:
> On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
>> On 6/3/24 10:23 PM, Tony Ambardar wrote:
>>> Some code includes the __used macro to prevent functions and data from
>>> being optimized out. This macro implements __attribute__((__used__)), which
>>> operates at the compiler and IR-level, and so still allows a linker to
>>> remove objects intended to be kept.
>>>
>>> Compilers supporting __attribute__((__retain__)) can address this gap by
>>> setting the flag SHF_GNU_RETAIN on the section of a function/variable,
>>> indicating to the linker the object should be retained. This attribute is
>>> available since gcc 11, clang 13, and binutils 2.36.
>>>
>>> Provide a __retain macro implementing __attribute__((__retain__)), whose
>>> first user will be the '__bpf_kfunc' tag.
>>>
>>> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
>>> Cc: stable@vger.kernel.org # v6.6+
>>> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
>>> ---
>>>    include/linux/compiler_types.h | 23 +++++++++++++++++++++++
>>>    1 file changed, 23 insertions(+)
>>>
>>> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
>>> index 93600de3800b..f14c275950b5 100644
>>> --- a/include/linux/compiler_types.h
>>> +++ b/include/linux/compiler_types.h
>>> @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
>>>    # define __preserve_most
>>>    #endif
>>> +/*
>>> + * Annotating a function/variable with __retain tells the compiler to place
>>> + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
>>> + * instructs the linker to retain the object during garbage-cleanup or LTO
>>> + * phases.
>>> + *
>>> + * Note that the __used macro is also used to prevent functions or data
>>> + * being optimized out, but operates at the compiler/IR-level and may still
>>> + * allow unintended removal of objects during linking.
>>> + *
>>> + * Optional: only supported since gcc >= 11, clang >= 13
>>> + *
>>> + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
>>> + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
>>> + */
>>> +#if __has_attribute(__retain__) && \
>>> +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
>>> +	 defined(CONFIG_LTO_CLANG))
>> Could you explain why CONFIG_LTO_CLANG is added here?
>> IIUC, the __used macro permits garbage collection at section
>> level, so CLANG_LTO_CLANG without
>> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
>> shuold not change final section dynamics, right?
> Hi Yonghong,
>
> I included the conditional guard to ensure consistent behaviour between
> __retain and other features forcing split sections. In particular, the same
> guard is used in vmlinux.lds.h to merge split sections where needed. For
> example, using __retain in llvm builds without CONFIG_LTO was failing CI
> tests on kernel-patches/bpf because the kernel didn't boot properly. And in
> further testing, the kernel had no issues loading BPF kfunc modules with
> such split sections, so I left the module (partial) linking scripts alone.

I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
in .config file. The following are all occurances in Kconfig:

$ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION

Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
for x86?

I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
diff --git a/init/Kconfig b/init/Kconfig
index 72404c1f2157..adf8718e2f5b 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
  endchoice
  
  config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
-       bool
+       def_bool y
         help
           This requires that the arch annotates or otherwise protects
           its external entry points from being discarded. Linker scripts

But with the above, I cannot boot the kernel.


Did I miss anything?

>
> Maybe I misunderstand you question re: __used?
>
> Thanks,
> Tony
>>> +# define __retain			__attribute__((__retain__))
>>> +#else
>>> +# define __retain
>>> +#endif
>>> +
>>>    /* Compiler specific macros. */
>>>    #ifdef __clang__
>>>    #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Tony Ambardar]

On Fri, Jun 14, 2024 at 11:47:19AM -0700, Yonghong Song wrote:
> 
> On 6/10/24 3:56 PM, Tony Ambardar wrote:
> > On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
> > > On 6/3/24 10:23 PM, Tony Ambardar wrote:
> > > > Some code includes the __used macro to prevent functions and data from
> > > > being optimized out. This macro implements __attribute__((__used__)), which
> > > > operates at the compiler and IR-level, and so still allows a linker to
> > > > remove objects intended to be kept.
> > > > 
> > > > Compilers supporting __attribute__((__retain__)) can address this gap by
> > > > setting the flag SHF_GNU_RETAIN on the section of a function/variable,
> > > > indicating to the linker the object should be retained. This attribute is
> > > > available since gcc 11, clang 13, and binutils 2.36.
> > > > 
> > > > Provide a __retain macro implementing __attribute__((__retain__)), whose
> > > > first user will be the '__bpf_kfunc' tag.
> > > > 
> > > > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > > > Cc: stable@vger.kernel.org # v6.6+
> > > > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> > > > ---
> > > >    include/linux/compiler_types.h | 23 +++++++++++++++++++++++
> > > >    1 file changed, 23 insertions(+)
> > > > 
> > > > diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
> > > > index 93600de3800b..f14c275950b5 100644
> > > > --- a/include/linux/compiler_types.h
> > > > +++ b/include/linux/compiler_types.h
> > > > @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
> > > >    # define __preserve_most
> > > >    #endif
> > > > +/*
> > > > + * Annotating a function/variable with __retain tells the compiler to place
> > > > + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
> > > > + * instructs the linker to retain the object during garbage-cleanup or LTO
> > > > + * phases.
> > > > + *
> > > > + * Note that the __used macro is also used to prevent functions or data
> > > > + * being optimized out, but operates at the compiler/IR-level and may still
> > > > + * allow unintended removal of objects during linking.
> > > > + *
> > > > + * Optional: only supported since gcc >= 11, clang >= 13
> > > > + *
> > > > + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
> > > > + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
> > > > + */
> > > > +#if __has_attribute(__retain__) && \
> > > > +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
> > > > +	 defined(CONFIG_LTO_CLANG))
> > > Could you explain why CONFIG_LTO_CLANG is added here?
> > > IIUC, the __used macro permits garbage collection at section
> > > level, so CLANG_LTO_CLANG without
> > > CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
> > > shuold not change final section dynamics, right?
> > Hi Yonghong,
> > 
> > I included the conditional guard to ensure consistent behaviour between
> > __retain and other features forcing split sections. In particular, the same
> > guard is used in vmlinux.lds.h to merge split sections where needed. For
> > example, using __retain in llvm builds without CONFIG_LTO was failing CI
> > tests on kernel-patches/bpf because the kernel didn't boot properly. And in
> > further testing, the kernel had no issues loading BPF kfunc modules with
> > such split sections, so I left the module (partial) linking scripts alone.
> 
> I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
> in .config file. The following are all occurances in Kconfig:

My understanding is one doesn't directly set HAVE_LD_DEAD_CODE_...; it's a
per-arch capability flag which guards setting LD_DEAD_CODE_DATA_ELIMINATION
but only targets "small systems" (i.e. embedded), so no surprise x86 isn't
in the arch list below.

> 
> $ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
> arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
> init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> 
> Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> for x86?

I doubt it given the target arches above, but curious what's the need for
x86 support? Only x86_32? My patches were motivated seeing resolve_btfids
and pahole errors for a couple years on MIPS routers. I don't recall seeing
the same for x86 builds, so my testing focussed more on preserving x86
builds rather than adding/testing the arch flag for x86.

> 
> I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
> diff --git a/init/Kconfig b/init/Kconfig
> index 72404c1f2157..adf8718e2f5b 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
>  endchoice
>  config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> -       bool
> +       def_bool y
>         help
>           This requires that the arch annotates or otherwise protects
>           its external entry points from being discarded. Linker scripts
> 
> But with the above, I cannot boot the kernel.

OK, interesting exercise. Setting HAVE_LD_DEAD_CODE_DATA_ELIMINATION
shouldn't change anything itself so I suppose you are also setting
LD_DEAD_CODE_DATA_ELIMINATION? From previous testing on kernel-patches/CI,
first guess would be vmlinux linker script doing section merges unaware of
some x86 quirk. Or x86-specific linker script unhappy with split sections.

> 
> 
> Did I miss anything?
> 
> > 
> > Maybe I misunderstand you question re: __used?
> > 
> > Thanks,
> > Tony
> > > > +# define __retain			__attribute__((__retain__))
> > > > +#else
> > > > +# define __retain
> > > > +#endif
> > > > +
> > > >    /* Compiler specific macros. */
> > > >    #ifdef __clang__
> > > >    #include <linux/compiler-clang.h>

Re: [PATCH bpf v2 1/2] compiler_types.h: Define __retain for __attribute__((__retain__))

[Yonghong Song]

On 6/14/24 11:57 PM, Tony Ambardar wrote:
> On Fri, Jun 14, 2024 at 11:47:19AM -0700, Yonghong Song wrote:
>> On 6/10/24 3:56 PM, Tony Ambardar wrote:
>>> On Tue, Jun 04, 2024 at 10:55:39PM -0700, Yonghong Song wrote:
>>>> On 6/3/24 10:23 PM, Tony Ambardar wrote:
>>>>> Some code includes the __used macro to prevent functions and data from
>>>>> being optimized out. This macro implements __attribute__((__used__)), which
>>>>> operates at the compiler and IR-level, and so still allows a linker to
>>>>> remove objects intended to be kept.
>>>>>
>>>>> Compilers supporting __attribute__((__retain__)) can address this gap by
>>>>> setting the flag SHF_GNU_RETAIN on the section of a function/variable,
>>>>> indicating to the linker the object should be retained. This attribute is
>>>>> available since gcc 11, clang 13, and binutils 2.36.
>>>>>
>>>>> Provide a __retain macro implementing __attribute__((__retain__)), whose
>>>>> first user will be the '__bpf_kfunc' tag.
>>>>>
>>>>> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
>>>>> Cc: stable@vger.kernel.org # v6.6+
>>>>> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
>>>>> ---
>>>>>     include/linux/compiler_types.h | 23 +++++++++++++++++++++++
>>>>>     1 file changed, 23 insertions(+)
>>>>>
>>>>> diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
>>>>> index 93600de3800b..f14c275950b5 100644
>>>>> --- a/include/linux/compiler_types.h
>>>>> +++ b/include/linux/compiler_types.h
>>>>> @@ -143,6 +143,29 @@ static inline void __chk_io_ptr(const volatile void __iomem *ptr) { }
>>>>>     # define __preserve_most
>>>>>     #endif
>>>>> +/*
>>>>> + * Annotating a function/variable with __retain tells the compiler to place
>>>>> + * the object in its own section and set the flag SHF_GNU_RETAIN. This flag
>>>>> + * instructs the linker to retain the object during garbage-cleanup or LTO
>>>>> + * phases.
>>>>> + *
>>>>> + * Note that the __used macro is also used to prevent functions or data
>>>>> + * being optimized out, but operates at the compiler/IR-level and may still
>>>>> + * allow unintended removal of objects during linking.
>>>>> + *
>>>>> + * Optional: only supported since gcc >= 11, clang >= 13
>>>>> + *
>>>>> + *   gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-retain-function-attribute
>>>>> + * clang: https://clang.llvm.org/docs/AttributeReference.html#retain
>>>>> + */
>>>>> +#if __has_attribute(__retain__) && \
>>>>> +	(defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || \
>>>>> +	 defined(CONFIG_LTO_CLANG))
>>>> Could you explain why CONFIG_LTO_CLANG is added here?
>>>> IIUC, the __used macro permits garbage collection at section
>>>> level, so CLANG_LTO_CLANG without
>>>> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
>>>> shuold not change final section dynamics, right?
>>> Hi Yonghong,
>>>
>>> I included the conditional guard to ensure consistent behaviour between
>>> __retain and other features forcing split sections. In particular, the same
>>> guard is used in vmlinux.lds.h to merge split sections where needed. For
>>> example, using __retain in llvm builds without CONFIG_LTO was failing CI
>>> tests on kernel-patches/bpf because the kernel didn't boot properly. And in
>>> further testing, the kernel had no issues loading BPF kfunc modules with
>>> such split sections, so I left the module (partial) linking scripts alone.
>> I tried with both bpf and bpf-next tree and I cannot make CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y
>> in .config file. The following are all occurances in Kconfig:
> My understanding is one doesn't directly set HAVE_LD_DEAD_CODE_...; it's a
> per-arch capability flag which guards setting LD_DEAD_CODE_DATA_ELIMINATION
> but only targets "small systems" (i.e. embedded), so no surprise x86 isn't
> in the arch list below.

I see. Yes, mips should support it but not x86. No wonder why I cannot reproduce.

>
>> $ egrep -r HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> arch/mips/Kconfig:      select HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> arch/powerpc/Kconfig:   select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if HAVE_OBJTOOL_MCOUNT && (!ARCH_USING_PATCHABLE_FUNCTION_ENTRY || (!CC_IS_GCC || GCC_VERSION >= 110100))
>> arch/riscv/Kconfig:     select HAVE_LD_DEAD_CODE_DATA_ELIMINATION if !LD_IS_LLD
>> init/Kconfig:config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> init/Kconfig:   depends on HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>>
>> Are there some pending patches to enable HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> for x86?
> I doubt it given the target arches above, but curious what's the need for
> x86 support? Only x86_32? My patches were motivated seeing resolve_btfids
> and pahole errors for a couple years on MIPS routers. I don't recall seeing
> the same for x86 builds, so my testing focussed more on preserving x86
> builds rather than adding/testing the arch flag for x86.
>> I could foce CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION=y with the following hack:
>> diff --git a/init/Kconfig b/init/Kconfig
>> index 72404c1f2157..adf8718e2f5b 100644
>> --- a/init/Kconfig
>> +++ b/init/Kconfig
>> @@ -1402,7 +1402,7 @@ config CC_OPTIMIZE_FOR_SIZE
>>   endchoice
>>   config HAVE_LD_DEAD_CODE_DATA_ELIMINATION
>> -       bool
>> +       def_bool y
>>          help
>>            This requires that the arch annotates or otherwise protects
>>            its external entry points from being discarded. Linker scripts
>>
>> But with the above, I cannot boot the kernel.
> OK, interesting exercise. Setting HAVE_LD_DEAD_CODE_DATA_ELIMINATION
> shouldn't change anything itself so I suppose you are also setting
> LD_DEAD_CODE_DATA_ELIMINATION? From previous testing on kernel-patches/CI,
> first guess would be vmlinux linker script doing section merges unaware of
> some x86 quirk. Or x86-specific linker script unhappy with split sections.

I guess x86 needs additional change to make HAVE_LD_DEAD_CODE_DATA_ELIMINATION
work. I still curious about why CONFIG_LTO_CLANG is necessary.

In asm-generic/vmlinux.lds.h,

/*
  * LD_DEAD_CODE_DATA_ELIMINATION option enables -fdata-sections, which
  * generates .data.identifier sections, which need to be pulled in with
  * .data. We don't want to pull in .data..other sections, which Linux
  * has defined. Same for text and bss.
  *
  * With LTO_CLANG, the linker also splits sections by default, so we need
  * these macros to combine the sections during the final link.
  *
  * RODATA_MAIN is not used because existing code already defines .rodata.x
  * sections to be brought in with rodata.
  */
#if defined(CONFIG_LD_DEAD_CODE_DATA_ELIMINATION) || defined(CONFIG_LTO_CLANG)
#define TEXT_MAIN .text .text.[0-9a-zA-Z_]*
#define DATA_MAIN .data .data.[0-9a-zA-Z_]* .data..L* .data..compoundliteral* .data.$__unnamed_* .data.$L*
#define SDATA_MAIN .sdata .sdata.[0-9a-zA-Z_]*
#define RODATA_MAIN .rodata .rodata.[0-9a-zA-Z_]* .rodata..L*
#define BSS_MAIN .bss .bss.[0-9a-zA-Z_]* .bss..compoundliteral*
#define SBSS_MAIN .sbss .sbss.[0-9a-zA-Z_]*
#else
#define TEXT_MAIN .text
#define DATA_MAIN .data
#define SDATA_MAIN .sdata
#define RODATA_MAIN .rodata
#define BSS_MAIN .bss
#define SBSS_MAIN .sbss
#endif

If CONFIG_LTO_CLANG is defined and CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is
not defined, it is not clear whether __used functions will get eliminated
or not. I tried with thinlto with a simple example on x86 with some unused
function marked with __used, and that function survived in the final binary.

But your patch won't hurt, so I am okay with it.

>
>>
>> Did I miss anything?
>>
>>> Maybe I misunderstand you question re: __used?
>>>
>>> Thanks,
>>> Tony
>>>>> +# define __retain			__attribute__((__retain__))
>>>>> +#else
>>>>> +# define __retain
>>>>> +#endif
>>>>> +
>>>>>     /* Compiler specific macros. */
>>>>>     #ifdef __clang__
>>>>>     #include <linux/compiler-clang.h>

[PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Tony Ambardar]

BPF kfuncs are often not directly referenced and may be inadvertently
removed by optimization steps during kernel builds, thus the __bpf_kfunc
tag mitigates against this removal by including the __used macro. However,
this macro alone does not prevent removal during linking, and may still
yield build warnings (e.g. on mips64el):

    LD      vmlinux
    BTFIDS  vmlinux
  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
  WARN: resolve_btfids: unresolved symbol bpf_key_put
  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
    NM      System.map
    SORTTAB vmlinux
    OBJCOPY vmlinux.32

Update the __bpf_kfunc tag to better guard against linker optimization by
including the new __retain compiler macro, which fixes the warnings above.

Verify the __retain macro with readelf by checking object flags for 'R':

  $ readelf -Wa kernel/trace/bpf_trace.o
  Section Headers:
    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
  ...
    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
  ...
  Key to Flags:
  ...
    R (retain), D (mbind), p (processor specific)

Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
Cc: stable@vger.kernel.org # v6.6+
Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
---
 include/linux/btf.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/btf.h b/include/linux/btf.h
index f9e56fd12a9f..7c3e40c3295e 100644
--- a/include/linux/btf.h
+++ b/include/linux/btf.h
@@ -82,7 +82,7 @@
  * as to avoid issues such as the compiler inlining or eliding either a static
  * kfunc, or a global kfunc in an LTO build.
  */
-#define __bpf_kfunc __used noinline
+#define __bpf_kfunc __used __retain noinline
 
 #define __bpf_kfunc_start_defs()					       \
 	__diag_push();							       \
-- 
2.34.1

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Jiri Olsa]

On Mon, Jun 03, 2024 at 10:23:16PM -0700, Tony Ambardar wrote:
> BPF kfuncs are often not directly referenced and may be inadvertently
> removed by optimization steps during kernel builds, thus the __bpf_kfunc
> tag mitigates against this removal by including the __used macro. However,
> this macro alone does not prevent removal during linking, and may still
> yield build warnings (e.g. on mips64el):
> 
>     LD      vmlinux
>     BTFIDS  vmlinux
>   WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
>   WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
>   WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
>   WARN: resolve_btfids: unresolved symbol bpf_key_put
>   WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
>   WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
>   WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
>   WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
>   WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
>   WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
>   WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
>   WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>     NM      System.map
>     SORTTAB vmlinux
>     OBJCOPY vmlinux.32
> 
> Update the __bpf_kfunc tag to better guard against linker optimization by
> including the new __retain compiler macro, which fixes the warnings above.
> 
> Verify the __retain macro with readelf by checking object flags for 'R':
> 
>   $ readelf -Wa kernel/trace/bpf_trace.o
>   Section Headers:
>     [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
>   ...
>     [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
>   ...
>   Key to Flags:
>   ...
>     R (retain), D (mbind), p (processor specific)
> 
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>

tested on mips64 cross build and the warnings are gone
and related functions are in the vmlinux

patchset looks good to me

Tested-by: Jiri Olsa <jolsa@kernel.org>
Reviewed-by: Jiri Olsa <jolsa@kernel.org>

thanks,
jirka

> ---
>  include/linux/btf.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/linux/btf.h b/include/linux/btf.h
> index f9e56fd12a9f..7c3e40c3295e 100644
> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -82,7 +82,7 @@
>   * as to avoid issues such as the compiler inlining or eliding either a static
>   * kfunc, or a global kfunc in an LTO build.
>   */
> -#define __bpf_kfunc __used noinline
> +#define __bpf_kfunc __used __retain noinline
>  
>  #define __bpf_kfunc_start_defs()					       \
>  	__diag_push();							       \
> -- 
> 2.34.1
> 

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Geert Uytterhoeven]

[-- Attachment #1: Type: text/plain, Size: 4091 bytes --]

 	Hi Tony,

On Mon, 3 Jun 2024, Tony Ambardar wrote:
> BPF kfuncs are often not directly referenced and may be inadvertently
> removed by optimization steps during kernel builds, thus the __bpf_kfunc
> tag mitigates against this removal by including the __used macro. However,
> this macro alone does not prevent removal during linking, and may still
> yield build warnings (e.g. on mips64el):
>
>    LD      vmlinux
>    BTFIDS  vmlinux
>  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
>  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
>  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
>  WARN: resolve_btfids: unresolved symbol bpf_key_put
>  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
>  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
>  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
>  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
>  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
>  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>    NM      System.map
>    SORTTAB vmlinux
>    OBJCOPY vmlinux.32
>
> Update the __bpf_kfunc tag to better guard against linker optimization by
> including the new __retain compiler macro, which fixes the warnings above.
>
> Verify the __retain macro with readelf by checking object flags for 'R':
>
>  $ readelf -Wa kernel/trace/bpf_trace.o
>  Section Headers:
>    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
>  ...
>    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
>  ...
>  Key to Flags:
>  ...
>    R (retain), D (mbind), p (processor specific)
>
> Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> Reported-by: kernel test robot <lkp@intel.com>
> Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> Cc: stable@vger.kernel.org # v6.6+
> Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>

Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
v6.10-rc5.

This is causing build failures on ARM with
CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:

     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11859 | {
           | ^
     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11872 | {
           | ^
     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11885 | {
           | ^
     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     11906 | {
           | ^
     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
     12092 | {
           | ^
     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       713 | {
           | ^
     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       736 | {
           | ^
     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
       769 | {
           | ^
     [...]

My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

> --- a/include/linux/btf.h
> +++ b/include/linux/btf.h
> @@ -82,7 +82,7 @@
>  * as to avoid issues such as the compiler inlining or eliding either a static
>  * kfunc, or a global kfunc in an LTO build.
>  */
> -#define __bpf_kfunc __used noinline
> +#define __bpf_kfunc __used __retain noinline
>
> #define __bpf_kfunc_start_defs()					       \
> 	__diag_push();							       \

Gr{oetje,eeting}s,

 						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
 							    -- Linus Torvalds

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Jiri Olsa]

On Tue, Jun 25, 2024 at 12:46:48PM +0200, Geert Uytterhoeven wrote:
> 	Hi Tony,
> 
> On Mon, 3 Jun 2024, Tony Ambardar wrote:
> > BPF kfuncs are often not directly referenced and may be inadvertently
> > removed by optimization steps during kernel builds, thus the __bpf_kfunc
> > tag mitigates against this removal by including the __used macro. However,
> > this macro alone does not prevent removal during linking, and may still
> > yield build warnings (e.g. on mips64el):
> > 
> >    LD      vmlinux
> >    BTFIDS  vmlinux
> >  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> >  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> >  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> >  WARN: resolve_btfids: unresolved symbol bpf_key_put
> >  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> >  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> >  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> >  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> >  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> >    NM      System.map
> >    SORTTAB vmlinux
> >    OBJCOPY vmlinux.32
> > 
> > Update the __bpf_kfunc tag to better guard against linker optimization by
> > including the new __retain compiler macro, which fixes the warnings above.
> > 
> > Verify the __retain macro with readelf by checking object flags for 'R':
> > 
> >  $ readelf -Wa kernel/trace/bpf_trace.o
> >  Section Headers:
> >    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
> >  ...
> >    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
> >  ...
> >  Key to Flags:
> >  ...
> >    R (retain), D (mbind), p (processor specific)
> > 
> > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > Reported-by: kernel test robot <lkp@intel.com>
> > Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> > Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> > Cc: stable@vger.kernel.org # v6.6+
> > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> 
> Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
> ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
> v6.10-rc5.
> 
> This is causing build failures on ARM with
> CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
> 
>     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11859 | {
>           | ^
>     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11872 | {
>           | ^
>     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11885 | {
>           | ^
>     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     11906 | {
>           | ^
>     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>     12092 | {
>           | ^
>     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       713 | {
>           | ^
>     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       736 | {
>           | ^
>     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
>       769 | {
>           | ^
>     [...]
> 
> My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).

hum, so it'd mean __has_attribute(__retain__) returns true while gcc still
ignores the retain attribute.. like in this bug which seems similar:
  https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
but not sure how it got fixed.. any chance you can upgrade gcc and retest?

jirka

> 
> > --- a/include/linux/btf.h
> > +++ b/include/linux/btf.h
> > @@ -82,7 +82,7 @@
> >  * as to avoid issues such as the compiler inlining or eliding either a static
> >  * kfunc, or a global kfunc in an LTO build.
> >  */
> > -#define __bpf_kfunc __used noinline
> > +#define __bpf_kfunc __used __retain noinline
> > 
> > #define __bpf_kfunc_start_defs()					       \
> > 	__diag_push();							       \
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

Re: [PATCH bpf v2 2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal

[Geert Uytterhoeven]

Hi Jiri,

On Wed, Jun 26, 2024 at 11:52 AM Jiri Olsa <olsajiri@gmail.com> wrote:
> On Tue, Jun 25, 2024 at 12:46:48PM +0200, Geert Uytterhoeven wrote:
> > On Mon, 3 Jun 2024, Tony Ambardar wrote:
> > > BPF kfuncs are often not directly referenced and may be inadvertently
> > > removed by optimization steps during kernel builds, thus the __bpf_kfunc
> > > tag mitigates against this removal by including the __used macro. However,
> > > this macro alone does not prevent removal during linking, and may still
> > > yield build warnings (e.g. on mips64el):
> > >
> > >    LD      vmlinux
> > >    BTFIDS  vmlinux
> > >  WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > >  WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > >  WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > >  WARN: resolve_btfids: unresolved symbol bpf_key_put
> > >  WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > >  WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > >  WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > >  WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > >  WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > >  WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > >    NM      System.map
> > >    SORTTAB vmlinux
> > >    OBJCOPY vmlinux.32
> > >
> > > Update the __bpf_kfunc tag to better guard against linker optimization by
> > > including the new __retain compiler macro, which fixes the warnings above.
> > >
> > > Verify the __retain macro with readelf by checking object flags for 'R':
> > >
> > >  $ readelf -Wa kernel/trace/bpf_trace.o
> > >  Section Headers:
> > >    [Nr]  Name              Type     Address  Off  Size ES Flg Lk Inf Al
> > >  ...
> > >    [178] .text.bpf_key_put PROGBITS 00000000 6420 0050 00 AXR  0   0  8
> > >  ...
> > >  Key to Flags:
> > >  ...
> > >    R (retain), D (mbind), p (processor specific)
> > >
> > > Link: https://lore.kernel.org/bpf/ZlmGoT9KiYLZd91S@krava/T/
> > > Reported-by: kernel test robot <lkp@intel.com>
> > > Closes: https://lore.kernel.org/r/202401211357.OCX9yllM-lkp@intel.com/
> > > Fixes: 57e7c169cd6a ("bpf: Add __bpf_kfunc tag for marking kernel functions as kfuncs")
> > > Cc: stable@vger.kernel.org # v6.6+
> > > Signed-off-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> >
> > Thanks for your patch, which is now commit 7bdcedd5c8fb88e7
> > ("bpf: Harden __bpf_kfunc tag against linker kfunc removal") in
> > v6.10-rc5.
> >
> > This is causing build failures on ARM with
> > CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y:
> >
> >     net/core/filter.c:11859:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11859 | {
> >           | ^
> >     net/core/filter.c:11872:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11872 | {
> >           | ^
> >     net/core/filter.c:11885:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11885 | {
> >           | ^
> >     net/core/filter.c:11906:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     11906 | {
> >           | ^
> >     net/core/filter.c:12092:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >     12092 | {
> >           | ^
> >     net/core/xdp.c:713:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       713 | {
> >           | ^
> >     net/core/xdp.c:736:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       736 | {
> >           | ^
> >     net/core/xdp.c:769:1: error: ‘retain’ attribute ignored [-Werror=attributes]
> >       769 | {
> >           | ^
> >     [...]
> >
> > My compiler is arm-linux-gnueabihf-gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04).
>
> hum, so it'd mean __has_attribute(__retain__) returns true while gcc still
> ignores the retain attribute.. like in this bug which seems similar:
>   https://gcc.gnu.org/bugzilla/show_bug.cgi?id=99587
> but not sure how it got fixed.. any chance you can upgrade gcc and retest?

Indeed, __has_attribute(__retain__) returns true, while the attribute
is not supported.

My test program:

cat > /tmp/a.c <<EOF
#if __has_attribute(__retain__)
#warning __retain__ OK
#else
#warning No __retain__
#endif

int x __attribute__((__retain__));
EOF

$ arm-linux-gnueabihf-gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu
11.4.0-1ubuntu1~22.04))

/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~
/tmp/a.c:7:1: warning: ‘retain’ attribute ignored [-Wattributes]
    7 | int x __attribute__((__retain__));
      | ^~~

Oops

$ arm-linux-gnueabihf-gcc-12 -c /tmp/a.c # gcc version 12.3.0 (Ubuntu
12.3.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

Fixed

It works fine with the native gcc-11:

$ gcc-11 -c /tmp/a.c # gcc version 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
/tmp/a.c:2:2: warning: #warning __retain__ OK [-Wcpp]
    2 | #warning __retain__ OK
      |  ^~~~~~~

I gave it a try on all installed gcc-11 compilers.

/usr/bin/aarch64-linux-gnu-gcc-11
/usr/bin/alpha-linux-gnu-gcc-11
/usr/bin/arm-linux-gnueabi-gcc-11
/usr/bin/arm-linux-gnueabihf-gcc-11
/usr/bin/hppa64-linux-gnu-gcc-11
/usr/bin/hppa-linux-gnu-gcc-11
/usr/bin/m68k-linux-gnu-gcc-11
/usr/bin/powerpc64le-linux-gnu-gcc-11
/usr/bin/powerpc64-linux-gnu-gcc-11
/usr/bin/powerpc-linux-gnu-gcc-11
/usr/bin/riscv64-linux-gnu-gcc-11
/usr/bin/s390x-linux-gnu-gcc-11
/usr/bin/sh4-linux-gnu-gcc-11
/usr/bin/sparc64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnu-gcc-11
/usr/bin/x86_64-linux-gnux32-gcc-11

All of them failed (incl. x32), except for the native x86_64-linux-gnu-gcc-11.

It works fine with all installed gcc-12 compilers
(arm-linux-gnueabihf-gcc-12, m68k-linux-gnu-gcc-12, x86_64-linux-gnu-gcc-12).

With gcc-9, the absence of __retain__ is detected correctly.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH bpf v2 0/2] bpf: Fix linker optimization removing kfuncs

[patchwork-bot+netdevbpf]

Hello:

This series was applied to bpf/bpf.git (master)
by Daniel Borkmann <daniel@iogearbox.net>:

On Mon,  3 Jun 2024 22:23:14 -0700 you wrote:
> This patch series fixes unwanted stripping of kernel kfuncs during linker
> optimization, as indicated by build warnings from resolve_btfids e.g.
> "WARN: resolve_btfids: unresolved symbol ...". This can happen because the
> __bpf_kfunc macro annotating kfunc declarations is ignored during linking.
> 
> Patch 1 adds support for the compiler attribute "__retain__", used to
> avoid linker garbage cleanup. Patch 2 then updates __bpf_kfunc to use this
> attribute when LTO builds are enabled.
> 
> [...]

Here is the summary with links:
  - [bpf,v2,1/2] compiler_types.h: Define __retain for __attribute__((__retain__))
    https://git.kernel.org/bpf/bpf/c/0a5d3258d7c9
  - [bpf,v2,2/2] bpf: Harden __bpf_kfunc tag against linker kfunc removal
    https://git.kernel.org/bpf/bpf/c/7bdcedd5c8fb

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

Hello Hengqi,

On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> Hi Tony,
> 
> On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > Hello,
> >
> > For some time now I'm seeing multiple issues during BTF generation while
> > building recent kernels targeting mips64el, and would appreciate some help
> > to understand and fix the problems.
> >
> SNIP
> >
> > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > >   CC [M]  net/psample/psample.mod.o
> > >   LD [M]  crypto/cmac.ko
> > >   BTF [M] crypto/cmac.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> The issue seems to be related to elfutils. Have you tried build from
> the latest elfutils source ?
> I saw the latest MIPS backend in elfutils already implemented the
> reloc_simple_type hook.

Good idea. I tried rebuilding elfutils from the latest upstream commit:
https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508

I then linked this elfutils with pahole built from the latest pahole/next:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993

I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
the kernel with the same config. Unfortunately, the warnings/errors from
resolve_btfids and pahole still occur.

> SNIP
> >
> > I'd be grateful if some of the BTF/pahole experts could please review this
> > issue and share next steps or other details I might provide.
> >
> > Thanks,
> > Tony Ambardar
> >
> > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > Link: https://github.com/acmel/dwarves/issues/45 [2]
> 
> Cheers,
> Hengqi

Thanks for the suggestion,
Tony

Re: Problem with BTF generation on mips64el

[Arnaldo Carvalho de Melo]

On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> Hello Hengqi,
> 
> On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > Hi Tony,
> > 
> > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > For some time now I'm seeing multiple issues during BTF generation while
> > > building recent kernels targeting mips64el, and would appreciate some help
> > > to understand and fix the problems.
> > >
> > SNIP
> > >
> > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > >   CC [M]  net/psample/psample.mod.o
> > > >   LD [M]  crypto/cmac.ko
> > > >   BTF [M] crypto/cmac.ko
> > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > or DW_TAG_skeleton_unit expected got member (0xd)!

Can you check the kernel CONFIG_ variables related to DEBUG information
and post them here? I have this on fedora:

[acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
[acme@nine linux]$

If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
fedora config.

- Arnaldo

> > The issue seems to be related to elfutils. Have you tried build from
> > the latest elfutils source ?
> > I saw the latest MIPS backend in elfutils already implemented the
> > reloc_simple_type hook.
> 
> Good idea. I tried rebuilding elfutils from the latest upstream commit:
> https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> 
> I then linked this elfutils with pahole built from the latest pahole/next:
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> 
> I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> the kernel with the same config. Unfortunately, the warnings/errors from
> resolve_btfids and pahole still occur.
> 
> > SNIP
> > >
> > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > issue and share next steps or other details I might provide.
> > >
> > > Thanks,
> > > Tony Ambardar
> > >
> > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > 
> > Cheers,
> > Hengqi
> 
> Thanks for the suggestion,
> Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > Hello Hengqi,
> > 
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > SNIP
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> Can you check the kernel CONFIG_ variables related to DEBUG information
> and post them here? I have this on fedora:
> 
> [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> [acme@nine linux]$
> 
> If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> fedora config.
> 
> - Arnaldo
> 

OK, I have those 3 settings as expected on bpf/master branch:

kodidev:~/linux$ grep CONFIG_DEBUG_INFO .config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_NONE is not set
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_COMPRESSED_NONE=y
# CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

I also attached the full .config in my original mail if you think other
config options might be suspect.

Is there anything else I can check? What do you make of the many pahole
"die__process:" warnings? Could it be related to your comment here:
https://github.com/acmel/dwarves/issues/45#issuecomment-1974004606

Thanks,
Tony

> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > Good idea. I tried rebuilding elfutils from the latest upstream commit:
> > https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> > 
> > I then linked this elfutils with pahole built from the latest pahole/next:
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> > 
> > I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> > the kernel with the same config. Unfortunately, the warnings/errors from
> > resolve_btfids and pahole still occur.
> > 
> > > SNIP
> > > >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> > 
> > Thanks for the suggestion,
> > Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > Hello Hengqi,
> > 
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > SNIP
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> Can you check the kernel CONFIG_ variables related to DEBUG information
> and post them here? I have this on fedora:
> 
> [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> [acme@nine linux]$
> 
> If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> fedora config.

One more useful observation: the pahole BTF generation problems appear only
for mips64. If I build the same config for mips32be I see none of those
"die__process: errors" from pahole and the processed modules load properly.

I also tried running pahole under gdb to look further, but lack knowledge
of pahole and libdw internals, so couldn't narrow things down to a pahole,
elfutils or compiler-output problem.

To help troubleshoot further, I packaged my base vmlinux, some module .ko
files showing problems, and a reproducer script. I've uploaded the tarball
here:

https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

Thanks,
Tony

> 
> - Arnaldo
> 
> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > Good idea. I tried rebuilding elfutils from the latest upstream commit:
> > https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> > 
> > I then linked this elfutils with pahole built from the latest pahole/next:
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> > 
> > I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> > the kernel with the same config. Unfortunately, the warnings/errors from
> > resolve_btfids and pahole still occur.
> > 
> > > SNIP
> > > >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> > 
> > Thanks for the suggestion,
> > Tony

Re: Problem with BTF generation on mips64el

[Arnaldo Carvalho de Melo]

On Mon, Jun 03, 2024 at 04:20:01AM -0700, Tony Ambardar wrote:
> On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > to understand and fix the problems.

> > > > SNIP

> > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > >   LD [M]  crypto/cmac.ko
> > > > > >   BTF [M] crypto/cmac.ko
> > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!

> > Can you check the kernel CONFIG_ variables related to DEBUG information
> > and post them here? I have this on fedora:

> > [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> > CONFIG_DEBUG_INFO=y
> > # CONFIG_DEBUG_INFO_REDUCED is not set
> > # CONFIG_DEBUG_INFO_COMPRESSED is not set
> > # CONFIG_DEBUG_INFO_SPLIT is not set
> > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> > # CONFIG_DEBUG_INFO_DWARF4 is not set
> > # CONFIG_DEBUG_INFO_DWARF5 is not set
> > CONFIG_DEBUG_INFO_BTF=y
> > CONFIG_DEBUG_INFO_BTF_MODULES=y
> > [acme@nine linux]$

> > If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> > CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> > fedora config.

> One more useful observation: the pahole BTF generation problems appear only
> for mips64. If I build the same config for mips32be I see none of those
> "die__process: errors" from pahole and the processed modules load properly.

> I also tried running pahole under gdb to look further, but lack knowledge
> of pahole and libdw internals, so couldn't narrow things down to a pahole,
> elfutils or compiler-output problem.
> 
> To help troubleshoot further, I packaged my base vmlinux, some module .ko
> files showing problems, and a reproducer script. I've uploaded the tarball
> here:
> 
> https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

So, with that vmlinux, on a Fedora 39 x86_64 notebook:

⬢[acme@toolbox repro_die__process]$ readelf -wi vmlinux | head -45
Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x3f (32-bit)
   Version:       4
   Abbrev Offset: 0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_stmt_list   : 0
    <10>   DW_AT_ranges      : 0
    <14>   DW_AT_name        : (indirect string, offset: 0): arch/mips/kernel/head.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2c): GNU AS 2.42
    <20>   DW_AT_language    : 32769	(MIPS assembler)
 <1><22>: Abbrev Number: 2 (DW_TAG_subprogram)
    <23>   DW_AT_name        : (indirect string, offset: 0x38): kernel_entry
    <27>   DW_AT_external    : 1
    <27>   DW_AT_type        : <0x41>
    <28>   DW_AT_low_pc      : 0xffffffff80b0ce68
    <30>   DW_AT_high_pc     : 172
 <1><32>: Abbrev Number: 2 (DW_TAG_subprogram)
    <33>   DW_AT_name        : (indirect string, offset: 0x45): smp_bootstrap
    <37>   DW_AT_external    : 1
    <37>   DW_AT_type        : <0x41>
    <38>   DW_AT_low_pc      : 0xffffffff80b0cf14
    <40>   DW_AT_high_pc     : 44
 <1><41>: Abbrev Number: 3 (DW_TAG_unspecified_type)
 <1><42>: Abbrev Number: 0
  Compilation Unit @ offset 0x43:
   Length:        0x22ed9 (32-bit)
   Version:       4
   Abbrev Offset: 0x26
   Pointer Size:  8
 <0><4e>: Abbrev Number: 192 (DW_TAG_compile_unit)
    <50>   DW_AT_producer    : (indirect string, offset: 0x8147): GNU C11 13.3.0 -G 0 -mel -mno-check-zero-division -mabi=64 -mno-abicalls -msoft-float -march=mips64r2 -msym32 -mllsc -mips64r2 -mno-shared -g -gdwarf-4 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-pic -ffreestanding -fstack-check=no -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector -fno-stack-clash-protection -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -ffunction-sections -fdata-sections
    <54>   DW_AT_language    : 12	(ANSI C99)
    <55>   DW_AT_name        : (indirect string, offset: 0xce66): init/main.c
    <59>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <5d>   DW_AT_ranges      : 0x1350
    <61>   DW_AT_low_pc      : 0
    <69>   DW_AT_stmt_list   : 0x80
 <1><6d>: Abbrev Number: 91 (DW_TAG_base_type)
    <6e>   DW_AT_byte_size   : 8
    <6f>   DW_AT_encoding    : 7	(unsigned)
    <70>   DW_AT_name        : (indirect string, offset: 0xb1c7): long unsigned int
⬢[acme@toolbox repro_die__process]$

⬢[acme@toolbox repro_die__process]$ time pahole -F dwarf vmlinux | wc -l
84377

real	0m10.992s
user	0m9.526s
sys	0m1.445s
⬢[acme@toolbox repro_die__process]$

No warnings, 84377 lines of types printed from DWARF that file, which is
comparable to the BTF in it:

⬢[acme@toolbox repro_die__process]$ time pahole -F btf vmlinux | wc -l
83994

real	0m0.121s
user	0m0.095s
sys	0m0.031s
⬢[acme@toolbox repro_die__process]$

There are a few diffs from types generated from DWARF to those generated
from BTF (unexpected ones, that is) as noticed by btfdiff (found in the
pahole git repo), that its just some print ordering issue:

⬢[acme@toolbox repro_die__process]$ btfdiff vmlinux 
--- /tmp/btfdiff.dwarf.oX71hp	2024-06-03 11:42:32.228618100 -0300
+++ /tmp/btfdiff.btf.u762WM	2024-06-03 11:42:18.042748246 -0300
@@ -10027,6 +10027,24 @@ struct bio_map_data {
 };
 struct bio_post_read_ctx {
 	struct bio *               bio;                                                  /*     0     8 */
+	struct work_struct {
+		/* typedef atomic_long_t -> atomic64_t */ struct {
+			/* typedef s64 -> __s64 */ long long int counter;                /*     8     8 */
+		} data; /*     8     8 */
+		struct list_head {
+			struct list_head * next;                                         /*    16     8 */
+			struct list_head * prev;                                         /*    24     8 */
+		} entry; /*    16    16 */
+		/* typedef work_func_t */ void               (*func)(struct work_struct *); /*    32     8 */
+	} work; /*     8    32 */
+	unsigned int               cur_step;                                             /*    40     4 */
+	unsigned int               enabled_steps;                                        /*    44     4 */
+
+	/* size: 48, cachelines: 1, members: 4 */
+	/* last cacheline: 48 bytes */
+};
+struct bio_post_read_ctx {
+	struct bio *               bio;                                                  /*     0     8 */
 	struct f2fs_sb_info *      sbi;                                                  /*     8     8 */
 	struct work_struct {
 		/* typedef atomic_long_t -> atomic64_t */ struct {
@@ -10049,24 +10067,6 @@ struct bio_post_read_ctx {
 	/* sum members: 57, holes: 1, sum holes: 3 */
 	/* padding: 4 */
 };
-struct bio_post_read_ctx {
-	struct bio *               bio;                                                  /*     0     8 */
-	struct work_struct {
-		/* typedef atomic_long_t -> atomic64_t */ struct {
-			/* typedef s64 -> __s64 */ long long int counter;                /*     8     8 */
-		} data; /*     8     8 */
-		struct list_head {
-			struct list_head * next;                                         /*    16     8 */
-			struct list_head * prev;                                         /*    24     8 */
-		} entry; /*    16    16 */
-		/* typedef work_func_t */ void               (*func)(struct work_struct *); /*    32     8 */
-	} work; /*     8    32 */
-	unsigned int               cur_step;                                             /*    40     4 */
-	unsigned int               enabled_steps;                                        /*    44     4 */
-
-	/* size: 48, cachelines: 1, members: 4 */
-	/* last cacheline: 48 bytes */
-};
 struct bio_set {
 	struct kmem_cache *        bio_slab;                                             /*     0     8 */
 	unsigned int               front_pad;                                            /*     8     4 */
@@ -43044,6 +43044,17 @@ struct dma_block {
 	/* last cacheline: 16 bytes */
 };
 struct dma_chan {
+	int                        lock;                                                 /*     0     4 */
+
+	/* XXX 4 bytes hole, try to pack */
+
+	const char  *              device_id;                                            /*     8     8 */
+
+	/* size: 16, cachelines: 1, members: 2 */
+	/* sum members: 12, holes: 1, sum holes: 4 */
+	/* last cacheline: 16 bytes */
+};
+struct dma_chan {
 	struct dma_device *        device;                                               /*     0     8 */
 	struct device *            slave;                                                /*     8     8 */
 	/* typedef dma_cookie_t -> s32 -> __s32 */ int                        cookie;    /*    16     4 */
@@ -43071,17 +43082,6 @@ struct dma_chan {
 	/* sum members: 108, holes: 1, sum holes: 4 */
 	/* last cacheline: 48 bytes */
 };
-struct dma_chan {
-	int                        lock;                                                 /*     0     4 */
-
-	/* XXX 4 bytes hole, try to pack */
-
-	const char  *              device_id;                                            /*     8     8 */
-
-	/* size: 16, cachelines: 1, members: 2 */
-	/* sum members: 12, holes: 1, sum holes: 4 */
-	/* last cacheline: 16 bytes */
-};
 struct dma_chan_dev {
 	struct dma_chan *          chan;                                                 /*     0     8 */
 	struct device {
@@ -125786,12 +125786,11 @@ struct perf_aux_event {
 		/* typedef __u16 */ short unsigned int misc;                             /*     4     2 */
 		/* typedef __u16 */ short unsigned int size;                             /*     6     2 */
 	} header; /*     0     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     offset;                    /*     8     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     size;                      /*    16     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     flags;                     /*    24     8 */
+	/* typedef u32 -> __u32 */ unsigned int               pid;                       /*     8     4 */
+	/* typedef u32 -> __u32 */ unsigned int               tid;                       /*    12     4 */
 
-	/* size: 32, cachelines: 1, members: 4 */
-	/* last cacheline: 32 bytes */
+	/* size: 16, cachelines: 1, members: 3 */
+	/* last cacheline: 16 bytes */
 };
 struct perf_aux_event {
 	struct perf_event_header {
@@ -125799,11 +125798,12 @@ struct perf_aux_event {
 		/* typedef __u16 */ short unsigned int misc;                             /*     4     2 */
 		/* typedef __u16 */ short unsigned int size;                             /*     6     2 */
 	} header; /*     0     8 */
-	/* typedef u32 -> __u32 */ unsigned int               pid;                       /*     8     4 */
-	/* typedef u32 -> __u32 */ unsigned int               tid;                       /*    12     4 */
+	/* typedef u64 -> __u64 */ long long unsigned int     offset;                    /*     8     8 */
+	/* typedef u64 -> __u64 */ long long unsigned int     size;                      /*    16     8 */
+	/* typedef u64 -> __u64 */ long long unsigned int     flags;                     /*    24     8 */
 
-	/* size: 16, cachelines: 1, members: 3 */
-	/* last cacheline: 16 bytes */
+	/* size: 32, cachelines: 1, members: 4 */
+	/* last cacheline: 32 bytes */
 };
 struct perf_bpf_event {
 	struct bpf_prog *          prog;                                                 /*     0     8 */
@@ -163037,10 +163037,11 @@ struct syscall_tp_t {
 
 	/* XXX 4 bytes hole, try to pack */
 
-	long unsigned int          args[6];                                              /*    16    48 */
+	long unsigned int          ret;                                                  /*    16     8 */
 
-	/* size: 64, cachelines: 1, members: 3 */
-	/* sum members: 60, holes: 1, sum holes: 4 */
+	/* size: 24, cachelines: 1, members: 3 */
+	/* sum members: 20, holes: 1, sum holes: 4 */
+	/* last cacheline: 24 bytes */
 };
 struct syscall_tp_t {
 	struct trace_entry {
@@ -163053,11 +163054,10 @@ struct syscall_tp_t {
 
 	/* XXX 4 bytes hole, try to pack */
 
-	long unsigned int          ret;                                                  /*    16     8 */
+	long unsigned int          args[6];                                              /*    16    48 */
 
-	/* size: 24, cachelines: 1, members: 3 */
-	/* sum members: 20, holes: 1, sum holes: 4 */
-	/* last cacheline: 24 bytes */
+	/* size: 64, cachelines: 1, members: 3 */
+	/* sum members: 60, holes: 1, sum holes: 4 */
 };
 struct syscall_trace_enter {
 	struct trace_entry {
⬢[acme@toolbox repro_die__process]$ 

But if we do it manually and without expanding types as btfdiff does:

⬢[acme@toolbox repro_die__process]$ pahole -F btf -C dma_chan vmlinux 
struct dma_chan {
	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */

	/* XXX 4 bytes hole, try to pack */

	struct dma_chan_dev *      dev;                  /*    32     8 */
	const char  *              name;                 /*    40     8 */
	char *                     dbg_client_name;      /*    48     8 */
	struct list_head           device_node;          /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
	struct dma_chan_percpu *   local;                /*    72     8 */
	int                        client_count;         /*    80     4 */
	int                        table_count;          /*    84     4 */
	struct dma_router *        router;               /*    88     8 */
	void *                     route_data;           /*    96     8 */
	void *                     private;              /*   104     8 */

	/* size: 112, cachelines: 2, members: 15 */
	/* sum members: 108, holes: 1, sum holes: 4 */
	/* last cacheline: 48 bytes */
};

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf -C dma_chan vmlinux 
struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */

	/* size: 16, cachelines: 1, members: 2 */
	/* sum members: 12, holes: 1, sum holes: 4 */
	/* last cacheline: 16 bytes */
};

⬢[acme@toolbox repro_die__process]$

⬢[acme@toolbox perf-tools]$ git grep 'struct dma_chan {'
arch/mips/include/asm/mach-au1x00/au1000_dma.h:struct dma_chan {
include/linux/dmaengine.h:struct dma_chan {
kernel/dma.c:struct dma_chan {
⬢[acme@toolbox perf-tools]$

So we have this 'struct dma_chan' in the common kernel in include/linux/dmaengine.h

struct dma_chan {
        struct dma_device *device;
        struct device *slave;
        dma_cookie_t cookie;
        dma_cookie_t completed_cookie;

        /* sysfs */
        int chan_id;
        struct dma_chan_dev *dev;
        const char *name;       
#ifdef CONFIG_DEBUG_FS
        char *dbg_client_name;
#endif

        struct list_head device_node;
        struct dma_chan_percpu __percpu *local;
        int client_count;       
        int table_count;

        /* DMA router */
        struct dma_router *router;
        void *route_data;

        void *private;
};


And also this one in kernel/dma.c

struct dma_chan {
        int  lock;
        const char *device_id;
};

If we don't specify the name but grep for it:

⬢[acme@toolbox repro_die__process]$ pahole -F btf vmlinux | grep 'struct dma_chan {' -A5
struct dma_chan {
	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */
--
struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */
⬢[acme@toolbox repro_die__process]$ pahole -F dwarf vmlinux | grep 'struct dma_chan {' -A5
struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */

--
struct dma_chan {
	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */
⬢[acme@toolbox repro_die__process]$

So the vmlinux seems to have its BTF section successfully generated,
lemme check the .ko files.

- Arnaldo

elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Arnaldo Carvalho de Melo]

On Mon, Jun 03, 2024 at 11:56:43AM -0300, Arnaldo Carvalho de Melo wrote:
> So the vmlinux seems to have its BTF section successfully generated,
> lemme check the .ko files.

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf netdevsim.ko  | wc -l
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
11448
⬢[acme@toolbox repro_die__process]$ pahole -F btf netdevsim.ko  | wc -l
libbpf: Invalid BTF string section
pahole: file 'netdevsim.ko' has no btf type information.
0
⬢[acme@toolbox repro_die__process]$

We manage to load a lot of DWARF types but then we hit that warning, no
BTF encoded.

Humm,

⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo netdevsim.ko | wc -l
0
⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | wc -l
779837
⬢[acme@toolbox repro_die__process]$

It seems this is some problem with elfutils...

After I added:

⬢[acme@toolbox pahole]$ git diff
diff --git a/dwarf_loader.c b/dwarf_loader.c
index f59477b44dfea026..b832c93cc2194eaf 100644
--- a/dwarf_loader.c
+++ b/dwarf_loader.c
@@ -2847,8 +2847,8 @@ static int die__process(Dwarf_Die *die, struct cu *cu, struct conf_load *conf)
        }
 
        if (tag != DW_TAG_compile_unit && tag != DW_TAG_type_unit) {
-               fprintf(stderr, "%s: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got %s (0x%x)!\n",
-                       __FUNCTION__, dwarf_tag_name(tag), tag);
+               fprintf(stderr, "%s: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got %s (0x%x) @ %llx!\n",
+                       __FUNCTION__, dwarf_tag_name(tag), tag, (unsigned long long)dwarf_dieoffset(die));
                return -EINVAL;
        }
 
⬢[acme@toolbox pahole]$

We get:

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf netdevsim.ko > /dev/null
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd) @ 529de!
⬢[acme@toolbox repro_die__process]$

And if we look at binutils' readeld DWARF dump we get:

⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep '<529de>' -B5 -A8
  Compilation Unit @ offset 0x529d3:
   Length:        0x2132e (32-bit)
   Version:       4
   Abbrev Offset: 0x1d0e
   Pointer Size:  8
 <0><529de>: Abbrev Number: 108 (DW_TAG_compile_unit)
    <529df>   DW_AT_producer    : (indirect string, offset: 0x41389): GNU C11 13.3.0 -G 0 -mel -mno-check-zero-division -mabi=64 -mno-abicalls -msoft-float -march=mips64r2 -msym32 -mlong-calls -mllsc -mips64r2 -mno-shared -g -gdwarf-4 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-pic -ffreestanding -fstack-check=no -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector -fno-stack-clash-protection -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack
    <529e3>   DW_AT_language    : 12	(ANSI C99)
    <529e4>   DW_AT_name        : (indirect string, offset: 0x44ce0): drivers/net/netdevsim/ethtool.c
    <529e8>   DW_AT_comp_dir    : (indirect string, offset: 0x3f407): /home/kodidev/linux
    <529ec>   DW_AT_low_pc      : 0x5270
    <529f4>   DW_AT_high_pc     : 0x65c
    <529fc>   DW_AT_stmt_list   : 0x709d
 <1><52a00>: Abbrev Number: 57 (DW_TAG_base_type)
⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep '<529de>' -B5 -A8

And at that <529de> die it _is_ a DW_TAG_compile_unit, and we managed to
process two other DW_TAG_compile_units up to that point:

⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep -w DW_TAG_compile_unit
 <0><b>: Abbrev Number: 188 (DW_TAG_compile_unit)
 <0><26c16>: Abbrev Number: 188 (DW_TAG_compile_unit)
 <0><529de>: Abbrev Number: 108 (DW_TAG_compile_unit)
 <0><73d10>: Abbrev Number: 204 (DW_TAG_compile_unit)
 <0><a43a8>: Abbrev Number: 160 (DW_TAG_compile_unit)
 <0><c7e66>: Abbrev Number: 110 (DW_TAG_compile_unit)
 <0><e8f21>: Abbrev Number: 158 (DW_TAG_compile_unit)
 <0><10c525>: Abbrev Number: 115 (DW_TAG_compile_unit)
 <0><12da49>: Abbrev Number: 167 (DW_TAG_compile_unit)
 <0><1546b9>: Abbrev Number: 140 (DW_TAG_compile_unit)
 <0><17656a>: Abbrev Number: 1 (DW_TAG_compile_unit)
⬢[acme@toolbox repro_die__process]$


And eu-readelf also doesn't work for that mips vmlinux:

⬢[acme@toolbox repro_die__process]$ file vmlinux 
vmlinux: ELF 64-bit LSB executable, MIPS, MIPS64 rel2 version 1 (SYSV), statically linked, BuildID[sha1]=19c8dafc6aa33a2e63b87487dfb10bf215bdf3a3, with debug_info, not stripped
⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo vmlinux 
⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo+ vmlinux 
⬢[acme@toolbox repro_die__process]$ readelf -wi vmlinux | head -30
Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x3f (32-bit)
   Version:       4
   Abbrev Offset: 0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_stmt_list   : 0
    <10>   DW_AT_ranges      : 0
    <14>   DW_AT_name        : (indirect string, offset: 0): arch/mips/kernel/head.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2c): GNU AS 2.42
    <20>   DW_AT_language    : 32769	(MIPS assembler)
 <1><22>: Abbrev Number: 2 (DW_TAG_subprogram)
    <23>   DW_AT_name        : (indirect string, offset: 0x38): kernel_entry
    <27>   DW_AT_external    : 1
    <27>   DW_AT_type        : <0x41>
    <28>   DW_AT_low_pc      : 0xffffffff80b0ce68
    <30>   DW_AT_high_pc     : 172
 <1><32>: Abbrev Number: 2 (DW_TAG_subprogram)
    <33>   DW_AT_name        : (indirect string, offset: 0x45): smp_bootstrap
    <37>   DW_AT_external    : 1
    <37>   DW_AT_type        : <0x41>
    <38>   DW_AT_low_pc      : 0xffffffff80b0cf14
    <40>   DW_AT_high_pc     : 44
 <1><41>: Abbrev Number: 3 (DW_TAG_unspecified_type)
 <1><42>: Abbrev Number: 0
  Compilation Unit @ offset 0x43:
   Length:        0x22ed9 (32-bit)
⬢[acme@toolbox repro_die__process]$

Couldn't find a way to ask eu-readelf for more verbose output, where we
could perhaps get some clue as to why it produces nothing while binutils
readelf manages to grok it, Mark, do you know some other way to ask
eu-readelf to produce more debug output?

I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
that then made eu-readelf to not be able to process it while pahole,
that uses eltuils' libraries, was able to process the first two CUs for
a kernel module and all the CUs for the vmlinux file :-\

Mark, the whole thread is available at:

https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u

Now tryint to look at elfutils source to see if I can figure out
something there..

- Arnaldo

⬢[acme@toolbox repro_die__process]$ ltrace eu-readelf -winfo netdevsim.ko
<SNIP>
strlen(".gdb_index")                                                                 = 10
strncmp(".mdebug.abi64", ".gdb_index", 10)                                           = 6
elf_nextscn(0x55fddd480470, 0x55fddd4820e8, 10, 103)                                 = 0x55fddd4821b8
gelf_getshdr(0x55fddd4821b8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_strptr(0x55fddd480470, 54, 349, 0x55fddd4831f8)                                  = 0x7f0970925879
strlen(".note.GNU-stack")                                                            = 15
strncmp(".note.GNU-stack", ".debug_abbrev", 13)                                      = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_addr")                                                                = 11
strncmp(".note.GNU-stack", ".debug_addr", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_aranges")                                                             = 14
strncmp(".note.GNU-stack", ".debug_aranges", 14)                                     = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_frame")                                                               = 12
strncmp(".note.GNU-stack", ".debug_frame", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_info")                                                                = 11
strncmp(".note.GNU-stack", ".debug_info", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_types")                                                               = 12
strncmp(".note.GNU-stack", ".debug_types", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_line")                                                                = 11
strncmp(".note.GNU-stack", ".debug_line", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_loc")                                                                 = 10
strncmp(".note.GNU-stack", ".debug_loc", 10)                                         = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_loclists")                                                            = 15
strncmp(".note.GNU-stack", ".debug_loclists", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_pubnames")                                                            = 15
strncmp(".note.GNU-stack", ".debug_pubnames", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_str")                                                                 = 10
strncmp(".note.GNU-stack", ".debug_str", 10)                                         = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_line_str")                                                            = 15
strncmp(".note.GNU-stack", ".debug_line_str", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_str_offsets")                                                         = 18
strncmp(".note.GNU-stack", ".debug_str_offsets", 18)                                 = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_macinfo")                                                             = 14
strncmp(".note.GNU-stack", ".debug_macinfo", 14)                                     = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_macro")                                                               = 12
strncmp(".note.GNU-stack", ".debug_macro", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_ranges")                                                              = 13
strncmp(".note.GNU-stack", ".debug_ranges", 13)                                      = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_rnglists")                                                            = 15
strncmp(".note.GNU-stack", ".debug_rnglists", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".eh_frame")                                                                  = 9
strncmp(".note.GNU-stack", ".eh_frame", 9)                                           = 9
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".eh_frame_hdr")                                                              = 13
strncmp(".note.GNU-stack", ".eh_frame_hdr", 13)                                      = 9
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".gcc_except_table")                                                          = 17
strncmp(".note.GNU-stack", ".gcc_except_table", 17)                                  = 7
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".gdb_index")                                                                 = 10
strncmp(".note.GNU-stack", ".gdb_index", 10)                                         = 7
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
elf_nextscn(0x55fddd480470, 0x55fddd4821b8, 14, 103)                                 = 0x55fddd482288
gelf_getshdr(0x55fddd482288, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482288, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482358
gelf_getshdr(0x55fddd482358, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482358, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482428
gelf_getshdr(0x55fddd482428, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482428, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4824f8
gelf_getshdr(0x55fddd4824f8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4824f8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4825c8
gelf_getshdr(0x55fddd4825c8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4825c8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482698
gelf_getshdr(0x55fddd482698, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482698, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482768
gelf_getshdr(0x55fddd482768, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482768, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482838
gelf_getshdr(0x55fddd482838, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482838, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482908
gelf_getshdr(0x55fddd482908, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482908, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4829d8
gelf_getshdr(0x55fddd4829d8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4829d8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482aa8
gelf_getshdr(0x55fddd482aa8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482aa8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482b78
gelf_getshdr(0x55fddd482b78, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482b78, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482c48
gelf_getshdr(0x55fddd482c48, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482c48, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482d18
gelf_getshdr(0x55fddd482d18, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482d18, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482de8
gelf_getshdr(0x55fddd482de8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482de8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482eb8
gelf_getshdr(0x55fddd482eb8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_strptr(0x55fddd480470, 54, 513, 0x55fddd4831f8)                                  = 0x7f097092591d
strlen(".BTF")                                                                       = 4
strncmp(".BTF", ".debug_abbrev", 13)                                                 = -34
strlen(".debug_addr")                                                                = 11
strncmp(".BTF", ".debug_addr", 11)                                                   = -34
strlen(".debug_aranges")                                                             = 14
strncmp(".BTF", ".debug_aranges", 14)                                                = -34
strlen(".debug_frame")                                                               = 12
strncmp(".BTF", ".debug_frame", 12)                                                  = -34
strlen(".debug_info")                                                                = 11
strncmp(".BTF", ".debug_info", 11)                                                   = -34
strlen(".debug_types")                                                               = 12
strncmp(".BTF", ".debug_types", 12)                                                  = -34
strlen(".debug_line")                                                                = 11
strncmp(".BTF", ".debug_line", 11)                                                   = -34
strlen(".debug_loc")                                                                 = 10
strncmp(".BTF", ".debug_loc", 10)                                                    = -34
strlen(".debug_loclists")                                                            = 15
strncmp(".BTF", ".debug_loclists", 15)                                               = -34
strlen(".debug_pubnames")                                                            = 15
strncmp(".BTF", ".debug_pubnames", 15)                                               = -34
strlen(".debug_str")                                                                 = 10
strncmp(".BTF", ".debug_str", 10)                                                    = -34
strlen(".debug_line_str")                                                            = 15
strncmp(".BTF", ".debug_line_str", 15)                                               = -34
strlen(".debug_str_offsets")                                                         = 18
strncmp(".BTF", ".debug_str_offsets", 18)                                            = -34
strlen(".debug_macinfo")                                                             = 14
strncmp(".BTF", ".debug_macinfo", 14)                                                = -34
strlen(".debug_macro")                                                               = 12
strncmp(".BTF", ".debug_macro", 12)                                                  = -34
strlen(".debug_ranges")                                                              = 13
strncmp(".BTF", ".debug_ranges", 13)                                                 = -34
strlen(".debug_rnglists")                                                            = 15
strncmp(".BTF", ".debug_rnglists", 15)                                               = -34
strlen(".eh_frame")                                                                  = 9
strncmp(".BTF", ".eh_frame", 9)                                                      = -35
strlen(".eh_frame_hdr")                                                              = 13
strncmp(".BTF", ".eh_frame_hdr", 13)                                                 = -35
strlen(".gcc_except_table")                                                          = 17
strncmp(".BTF", ".gcc_except_table", 17)                                             = -37
strlen(".gdb_index")                                                                 = 10
strncmp(".BTF", ".gdb_index", 10)                                                    = -37
elf_nextscn(0x55fddd480470, 0x55fddd482eb8, 10, 103)                                 = 0x55fddd482f88
gelf_getshdr(0x55fddd482f88, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482f88, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd483058
gelf_getshdr(0x55fddd483058, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd483058, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd483128
gelf_getshdr(0x55fddd483128, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd483128, 0x55fddd480470, 0x55fddd4831f8)          = 0
dwfl_end(0, 165, 0x55fddd480538, 0x55fddd4831f8)                                     = 0
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(0x55fddd47eef0)                                                                 = <void>
<... dwfl_getmodules resumed> )                                                      = 0
dwfl_end(0x55fddd47ea90, 0x55fddd47eee0, 0x55fddd47e, 1)                             = 6
close(3)                                                                             = 0
__cxa_finalize(0x55fddbe44780, 10, 0, 0x7fff00974ee0)                                = 1
+++ exited (status 0) +++
⬢[acme@toolbox repro_die__process]$ 

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Mark Wielaard]

Hi,

On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> Couldn't find a way to ask eu-readelf for more verbose output, where we
> could perhaps get some clue as to why it produces nothing while binutils
> readelf manages to grok it, Mark, do you know some other way to ask
> eu-readelf to produce more debug output?
> 
> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> that then made eu-readelf to not be able to process it while pahole,
> that uses eltuils' libraries, was able to process the first two CUs for
> a kernel module and all the CUs for the vmlinux file :-\

I haven't looked at the vmlinux file. But for the .ko file the issue
is that the elfutils MIPS backend isn't complete. Specifically MIPS
relocations aren't recognized (and so cannot be applied). There are
some pending patches which try to fix that:

https://patchwork.sourceware.org/project/elfutils/list/?series=31601

Cheers,

Mark

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

Hi Mark,

On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > could perhaps get some clue as to why it produces nothing while binutils
> > readelf manages to grok it, Mark, do you know some other way to ask
> > eu-readelf to produce more debug output?
> > 
> > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > that then made eu-readelf to not be able to process it while pahole,
> > that uses eltuils' libraries, was able to process the first two CUs for
> > a kernel module and all the CUs for the vmlinux file :-\
> > 
> > Mark, the whole thread is available at:
> > 
> > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> 
> I haven't looked at the vmlinux file. But for the .ko file the issue
> is that the elfutils MIPS backend isn't complete. Specifically MIPS
> relocations aren't recognized (and so cannot be applied). There are
> some pending patches which try to fix that:
> 
> https://patchwork.sourceware.org/project/elfutils/list/?series=31601

Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
work for MIPS, and I locally rebuilt elfutils and then pahole from their
respective next/main branches. For elfutils, main (935ee131cf7c) includes

  e259f126 Support Mips architecture
  f2acb069 stack: Fix stack unwind failure on mips
  db33cb0c backends: Add register_info, return_value_location, core_note mips

which partially applies the patchwork series but leaves out the support for
readelf, strip, and elflint.

I believe this means the vmlinux and .ko files I shared are OK, or is there
more backend work needed for MIPS?

The bits missing in eu-readelf would explain the blank output both Arnaldo
and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
patchwork readelf patch locally but ran into merge conflicts.

CCing Ying Huang for any more insight.

Thanks,
Tony

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Ying Huang]

Hi Tony,

Yes, I also found that merge will have some conflicts now.

And you can modify the key code as follows to fix the issue about "eu-readelf -w" could not show info.

But this can not fix other display error in the result of the "eu-readelf -w" which need modify the way to get symbol index and type.

> diff --git a/libelf/libelfP.h b/libelf/libelfP.h
> index bdd2cc6a..6565ee02 100644
> --- a/libelf/libelfP.h
> +++ b/libelf/libelfP.h
> @@ -620,4 +620,5 @@ extern void __libelf_reset_rawdata (Elf_Scn *scn, void *buf, size_t size,
>  #define ELF64_MIPS_R_TYPE1(i)          ((i) & 0xff)
>  #define ELF64_MIPS_R_TYPE2(i)           (((i) >> 8) & 0xff)
>  #define ELF64_MIPS_R_TYPE3(i)           (((i) >> 16) & 0xff)
> +#define is_debug_section_type(type) (type == SHT_PROGBITS || type == SHT_MIPS_DWARF)
>  #endif  /* libelfP.h */ > diff --git a/src/readelf.c b/src/readelf.c
> index 0e931184..e88cf67c 100644
> --- a/src/readelf.c
> +++ b/src/readelf.c > @@ -12043,7 +12139,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr)
>  	  GElf_Shdr shdr_mem;
>  	  GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
>  
> -	  if (shdr != NULL && shdr->sh_type == SHT_PROGBITS)
> +	  if (shdr != NULL && is_debug_section_type(shdr->sh_type))
>  	    {
>  	      const char *name = elf_strptr (ebl->elf, shstrndx,
>  					     shdr->sh_name);
> @@ -12073,7 +12169,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr)
>        GElf_Shdr shdr_mem;
>        GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
>  
> -      if (shdr != NULL && shdr->sh_type == SHT_PROGBITS)
> +      if (shdr != NULL && is_debug_section_type(shdr->sh_type))
>  	{
>  	  static const struct
>  	  {

Thanks,

Ying


在 2024/6/4 11:47, Tony Ambardar 写道:
> Hi Mark,
>
> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
>> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
>>> Couldn't find a way to ask eu-readelf for more verbose output, where we
>>> could perhaps get some clue as to why it produces nothing while binutils
>>> readelf manages to grok it, Mark, do you know some other way to ask
>>> eu-readelf to produce more debug output?
>>>
>>> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
>>> that then made eu-readelf to not be able to process it while pahole,
>>> that uses eltuils' libraries, was able to process the first two CUs for
>>> a kernel module and all the CUs for the vmlinux file :-\
>>>
>>> Mark, the whole thread is available at:
>>>
>>> https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
>> I haven't looked at the vmlinux file. But for the .ko file the issue
>> is that the elfutils MIPS backend isn't complete. Specifically MIPS
>> relocations aren't recognized (and so cannot be applied). There are
>> some pending patches which try to fix that:
>>
>> https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> work for MIPS, and I locally rebuilt elfutils and then pahole from their
> respective next/main branches. For elfutils, main (935ee131cf7c) includes
>
>   e259f126 Support Mips architecture
>   f2acb069 stack: Fix stack unwind failure on mips
>   db33cb0c backends: Add register_info, return_value_location, core_note mips
>
> which partially applies the patchwork series but leaves out the support for
> readelf, strip, and elflint.
>
> I believe this means the vmlinux and .ko files I shared are OK, or is there
> more backend work needed for MIPS?
>
> The bits missing in eu-readelf would explain the blank output both Arnaldo
> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> patchwork readelf patch locally but ran into merge conflicts.
>
> CCing Ying Huang for any more insight.
>
> Thanks,
> Tony

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> Hi Mark,
> 
> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > could perhaps get some clue as to why it produces nothing while binutils
> > > readelf manages to grok it, Mark, do you know some other way to ask
> > > eu-readelf to produce more debug output?
> > > 
> > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > that then made eu-readelf to not be able to process it while pahole,
> > > that uses eltuils' libraries, was able to process the first two CUs for
> > > a kernel module and all the CUs for the vmlinux file :-\
> > > 
> > > Mark, the whole thread is available at:
> > > 
> > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > 
> > I haven't looked at the vmlinux file. But for the .ko file the issue
> > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > relocations aren't recognized (and so cannot be applied). There are
> > some pending patches which try to fix that:
> > 
> > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> 
> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> work for MIPS, and I locally rebuilt elfutils and then pahole from their
> respective next/main branches. For elfutils, main (935ee131cf7c) includes
> 
>   e259f126 Support Mips architecture
>   f2acb069 stack: Fix stack unwind failure on mips
>   db33cb0c backends: Add register_info, return_value_location, core_note mips
> 
> which partially applies the patchwork series but leaves out the support for
> readelf, strip, and elflint.
> 
> I believe this means the vmlinux and .ko files I shared are OK, or is there
> more backend work needed for MIPS?
> 
> The bits missing in eu-readelf would explain the blank output both Arnaldo
> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> patchwork readelf patch locally but ran into merge conflicts.
> 
> CCing Ying Huang for any more insight.
> 
> Thanks,
> Tony

Hello all,

A short update, starting with answering my own question.

No, apparently the above commits *do not* complete the backend work. Ying
Huang submitted additional related patches since March 5: [1][2]

    strip: Adapt src/strip -o -f on mips
    readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
    elflint: adapt src/elflint --gnu src/nm on mips
    test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh

Despite the titles, these patches do include core backend changes for MIPS.
I resolved the various merge conflicts [3], rebuilt elfutils, and retested
kernel builds to now find:

  - pahole is able to read DWARF[45] info and create .BTF for modules
  - resolve_btfids can successfully patch .BTF_ids in modules
  - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)

Huzzah!


Ying:

Thank you for developing these MIPS patches. In your view, are the MIPS
changes now complete, or do you plan further updates that might improve or
impact parsing DWARF debug/reloc info in apps like pahole?


Mark:

Given that BTF usage on Linux/MIPS is basically broken without these
patches, could I request some of your review time for them to be merged? If
it's helpful, my branch [3] includes all patches with conflicts fixed, and
I also successfully ran the elfutils self-tests (including MIPS from Ying).
Please feel free to add for these patches:

    Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>


Arnaldo:

Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
I reran your tests with the updated elfutils and latest pahole (pre-1.27),
and then found:

  - everything that worked before, still works
  - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
  - we now see expected output from "eu-readelf -winfo netdevsim.ko"

Regarding pahole, DWARF parsing and BTF generation now works:
(with no more die__process: error messages seen)

    kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
    14504

but strangely pahole still doesn't read its own generated BTF:

    kodidev:~/linux$ pahole -F btf netdevsim.ko
    libbpf: Invalid BTF string section
    pahole: file 'netdevsim.ko' has no btf type information.

Poking inside a little further:
    
    kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
    [...]
    argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
    SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
    SYS_brk(0)                                       = 0x563d47e37000
    SYS_brk(0x563d47e58000)                          = 0x563d47e58000
    <... argp_parse resumed> )                       = 0
    dwarves__init(0x20000, 0, -4096, 213)            = 0
    dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
    cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
    memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
    strlen("/sys/kernel/btf/")                       = 16
    strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
    cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
    0x563d47da40c0 <unfinished ...>
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
    SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
    SYS_close(3)                                     = 0
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
    SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
    SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
    SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
    SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
    SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
    string section
    ) = 35
    SYS_close(3)                                     = 0
    <... cus__load_files resumed> )                  = 0xffffffff
    access("netdevsim.ko", 4 <unfinished ...>
    SYS_access("netdevsim.ko", 04)                   = 0
    <... access resumed> )                           = 0
    fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
    "netdevsim.ko", "btf" <unfinished ...>
    SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
    'netdevsim.ko' has no btf type information.
    ) = 57
    <... fprintf resumed> )                          = 57
    SYS_exit_group(1 <no return ...>
    +++ exited (status 1) +++

Could you help investigate this further? Maybe a libbpf issue? For the
record, I also tried building pahole with embedded libbpf 1.4.3 without
any change. (side note: please make pahole --version also cover libbpf)


Many thanks everyone for your help,
Tony

[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Mon, Jun 10, 2024 at 11:36:54PM -0700, Tony Ambardar wrote:
> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> 
> [snip]
> 
> Arnaldo:
> 
> Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
> I reran your tests with the updated elfutils and latest pahole (pre-1.27),
> and then found:
> 
>   - everything that worked before, still works
>   - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
>   - we now see expected output from "eu-readelf -winfo netdevsim.ko"
> 
> Regarding pahole, DWARF parsing and BTF generation now works:
> (with no more die__process: error messages seen)
> 
>     kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
>     14504
> 
> but strangely pahole still doesn't read its own generated BTF:
> 
>     kodidev:~/linux$ pahole -F btf netdevsim.ko
>     libbpf: Invalid BTF string section
>     pahole: file 'netdevsim.ko' has no btf type information.

After staring at this I realized that this is split BTF and we needed to
issue "pahole -F btf --btf_base vmlinux netdevsim.ko", which does work.
Unfortunately, the libbpf error message is a bit cryptic; perhaps a future
update could mention "--btf_base"?

>     
> [snip]
> 
> Many thanks everyone for your help,
> Tony
> 
> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> [3]:
> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Mark Wielaard]

Hi,

Adding elfutils-devel to CC to keep everyone up to date on the state of
the patches.

On Mon, 2024-06-10 at 23:36 -0700, Tony Ambardar wrote:
> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> > On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > > could perhaps get some clue as to why it produces nothing while binutils
> > > > readelf manages to grok it, Mark, do you know some other way to ask
> > > > eu-readelf to produce more debug output?
> > > > 
> > > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > > that then made eu-readelf to not be able to process it while pahole,
> > > > that uses eltuils' libraries, was able to process the first two CUs for
> > > > a kernel module and all the CUs for the vmlinux file :-\
> > > > 
> > > > Mark, the whole thread is available at:
> > > > 
> > > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > > 
> > > I haven't looked at the vmlinux file. But for the .ko file the issue
> > > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > > relocations aren't recognized (and so cannot be applied). There are
> > > some pending patches which try to fix that:
> > > 
> > > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> > 
> > Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> > work for MIPS, and I locally rebuilt elfutils and then pahole from their
> > respective next/main branches. For elfutils, main (935ee131cf7c) includes
> > 
> >   e259f126 Support Mips architecture
> >   f2acb069 stack: Fix stack unwind failure on mips
> >   db33cb0c backends: Add register_info, return_value_location, core_note mips
> > 
> > which partially applies the patchwork series but leaves out the support for
> > readelf, strip, and elflint.
> > 
> > I believe this means the vmlinux and .ko files I shared are OK, or is there
> > more backend work needed for MIPS?
> > 
> > The bits missing in eu-readelf would explain the blank output both Arnaldo
> > and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> > patchwork readelf patch locally but ran into merge conflicts.
> 
> A short update, starting with answering my own question.
> 
> No, apparently the above commits *do not* complete the backend work. Ying
> Huang submitted additional related patches since March 5: [1][2]
> 
>     strip: Adapt src/strip -o -f on mips
>     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
>     elflint: adapt src/elflint --gnu src/nm on mips
>     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
> 
> Despite the titles, these patches do include core backend changes for MIPS.
> I resolved the various merge conflicts [3], rebuilt elfutils, and retested
> kernel builds to now find:
> 
>   - pahole is able to read DWARF[45] info and create .BTF for modules
>   - resolve_btfids can successfully patch .BTF_ids in modules
>   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
> 
> Huzzah!
> 
> 
> Ying:
> 
> Thank you for developing these MIPS patches. In your view, are the MIPS
> changes now complete, or do you plan further updates that might improve or
> impact parsing DWARF debug/reloc info in apps like pahole?
> 
> 
> Mark:
> 
> Given that BTF usage on Linux/MIPS is basically broken without these
> patches, could I request some of your review time for them to be merged? If
> it's helpful, my branch [3] includes all patches with conflicts fixed, and
> I also successfully ran the elfutils self-tests (including MIPS from Ying).
> Please feel free to add for these patches:
> 
>     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>

Yes, I would very much like to integrate the rest of these patches. But
I keep running out of time. The main issues were that, as you noticed,
the patches mix backend and frontend tool changes a bit. I don't have
access to a MIPS system to test them on. There are a couple of
different MIPS abis (I believe all combinations of 32/64 bit and
big/little endianness), but people have only tested on mips64le (maybe
that is the only relevant one these days?) And finally the way MIPS
represents relocations is slightly different than any other ELF
architecture does. So we have to translate that somewhere to make the
standards functions work. I have to convince myself that doing that in
elf_getdata as the patches do is the right place.

> Many thanks everyone for your help,
> Tony
> 
> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> [3]:
> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Tue, Jun 11, 2024 at 03:07:29PM +0200, Mark Wielaard wrote:
> Hi,
> 
> Adding elfutils-devel to CC to keep everyone up to date on the state of
> the patches.
> 
> On Mon, 2024-06-10 at 23:36 -0700, Tony Ambardar wrote:
> > On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> > > On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > > > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > > > could perhaps get some clue as to why it produces nothing while binutils
> > > > > readelf manages to grok it, Mark, do you know some other way to ask
> > > > > eu-readelf to produce more debug output?
> > > > > 
> > > > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > > > that then made eu-readelf to not be able to process it while pahole,
> > > > > that uses eltuils' libraries, was able to process the first two CUs for
> > > > > a kernel module and all the CUs for the vmlinux file :-\
> > > > > 
> > > > > Mark, the whole thread is available at:
> > > > > 
> > > > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > > > 
> > > > I haven't looked at the vmlinux file. But for the .ko file the issue
> > > > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > > > relocations aren't recognized (and so cannot be applied). There are
> > > > some pending patches which try to fix that:
> > > > 
> > > > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> > > 
> > > Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> > > work for MIPS, and I locally rebuilt elfutils and then pahole from their
> > > respective next/main branches. For elfutils, main (935ee131cf7c) includes
> > > 
> > >   e259f126 Support Mips architecture
> > >   f2acb069 stack: Fix stack unwind failure on mips
> > >   db33cb0c backends: Add register_info, return_value_location, core_note mips
> > > 
> > > which partially applies the patchwork series but leaves out the support for
> > > readelf, strip, and elflint.
> > > 
> > > I believe this means the vmlinux and .ko files I shared are OK, or is there
> > > more backend work needed for MIPS?
> > > 
> > > The bits missing in eu-readelf would explain the blank output both Arnaldo
> > > and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> > > patchwork readelf patch locally but ran into merge conflicts.
> > 
> > A short update, starting with answering my own question.
> > 
> > No, apparently the above commits *do not* complete the backend work. Ying
> > Huang submitted additional related patches since March 5: [1][2]
> > 
> >     strip: Adapt src/strip -o -f on mips
> >     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
> >     elflint: adapt src/elflint --gnu src/nm on mips
> >     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
> > 
> > Despite the titles, these patches do include core backend changes for MIPS.
> > I resolved the various merge conflicts [3], rebuilt elfutils, and retested
> > kernel builds to now find:
> > 
> >   - pahole is able to read DWARF[45] info and create .BTF for modules
> >   - resolve_btfids can successfully patch .BTF_ids in modules
> >   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
> > 
> > Huzzah!
> > 
> > 
> > Ying:
> > 
> > Thank you for developing these MIPS patches. In your view, are the MIPS
> > changes now complete, or do you plan further updates that might improve or
> > impact parsing DWARF debug/reloc info in apps like pahole?
> > 
> > 
> > Mark:
> > 
> > Given that BTF usage on Linux/MIPS is basically broken without these
> > patches, could I request some of your review time for them to be merged? If
> > it's helpful, my branch [3] includes all patches with conflicts fixed, and
> > I also successfully ran the elfutils self-tests (including MIPS from Ying).
> > Please feel free to add for these patches:
> > 
> >     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> 
> Yes, I would very much like to integrate the rest of these patches. But
> I keep running out of time. The main issues were that, as you noticed,
> the patches mix backend and frontend tool changes a bit. I don't have
> access to a MIPS system to test them on. There are a couple of
> different MIPS abis (I believe all combinations of 32/64 bit and
> big/little endianness), but people have only tested on mips64le (maybe
> that is the only relevant one these days?) And finally the way MIPS
> represents relocations is slightly different than any other ELF
> architecture does. So we have to translate that somewhere to make the
> standards functions work. I have to convince myself that doing that in
> elf_getdata as the patches do is the right place.
> 

Glad to hear there's strong interest. Not much I can add to the code
structure discussion but I *can* confirm my testing included both
mips64el and mips32be, specifically to improve word-size/endianness
coverage. The former is supported by Debian, while the latter can still
be found on many embedded devices like consumer routers. If you need
additional testing for review/merge I can help with that (using
OpenWrt/QEMU primarily).

> > Many thanks everyone for your help,
> > Tony
> > 
> > [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> > [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> > [3]:
> > https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/
> 

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Ying Huang]

Hi Mark,

Regarding the current questions, I have a few points that needed to be explained.


在 2024/6/11 21:07, Mark Wielaard 写道:
> Hi,
>
> Adding elfutils-devel to CC to keep everyone up to date on the state of
> the patches.
>
> On Mon, 2024-06-10 at 23:36 -0700, Tony Ambardar wrote:
>> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
>>> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
>>>> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
>>>>> Couldn't find a way to ask eu-readelf for more verbose output, where we
>>>>> could perhaps get some clue as to why it produces nothing while binutils
>>>>> readelf manages to grok it, Mark, do you know some other way to ask
>>>>> eu-readelf to produce more debug output?
>>>>>
>>>>> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
>>>>> that then made eu-readelf to not be able to process it while pahole,
>>>>> that uses eltuils' libraries, was able to process the first two CUs for
>>>>> a kernel module and all the CUs for the vmlinux file :-\
>>>>>
>>>>> Mark, the whole thread is available at:
>>>>>
>>>>> https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
>>>> I haven't looked at the vmlinux file. But for the .ko file the issue
>>>> is that the elfutils MIPS backend isn't complete. Specifically MIPS
>>>> relocations aren't recognized (and so cannot be applied). There are
>>>> some pending patches which try to fix that:
>>>>
>>>> https://patchwork.sourceware.org/project/elfutils/list/?series=31601
>>> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
>>> work for MIPS, and I locally rebuilt elfutils and then pahole from their
>>> respective next/main branches. For elfutils, main (935ee131cf7c) includes
>>>
>>>   e259f126 Support Mips architecture
>>>   f2acb069 stack: Fix stack unwind failure on mips
>>>   db33cb0c backends: Add register_info, return_value_location, core_note mips
>>>
>>> which partially applies the patchwork series but leaves out the support for
>>> readelf, strip, and elflint.
>>>
>>> I believe this means the vmlinux and .ko files I shared are OK, or is there
>>> more backend work needed for MIPS?
>>>
>>> The bits missing in eu-readelf would explain the blank output both Arnaldo
>>> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
>>> patchwork readelf patch locally but ran into merge conflicts.
>> A short update, starting with answering my own question.
>>
>> No, apparently the above commits *do not* complete the backend work. Ying
>> Huang submitted additional related patches since March 5: [1][2]
>>
>>     strip: Adapt src/strip -o -f on mips
>>     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
>>     elflint: adapt src/elflint --gnu src/nm on mips
>>     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
>>
>> Despite the titles, these patches do include core backend changes for MIPS.
>> I resolved the various merge conflicts [3], rebuilt elfutils, and retested
>> kernel builds to now find:
>>
>>   - pahole is able to read DWARF[45] info and create .BTF for modules
>>   - resolve_btfids can successfully patch .BTF_ids in modules
>>   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
>>
>> Huzzah!
>>
>>
>> Ying:
>>
>> Thank you for developing these MIPS patches. In your view, are the MIPS
>> changes now complete, or do you plan further updates that might improve or
>> impact parsing DWARF debug/reloc info in apps like pahole?
>>
>>
>> Mark:
>>
>> Given that BTF usage on Linux/MIPS is basically broken without these
>> patches, could I request some of your review time for them to be merged? If
>> it's helpful, my branch [3] includes all patches with conflicts fixed, and
>> I also successfully ran the elfutils self-tests (including MIPS from Ying).
>> Please feel free to add for these patches:
>>
>>     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> Yes, I would very much like to integrate the rest of these patches. But
> I keep running out of time. The main issues were that, as you noticed,
> the patches mix backend and frontend tool changes a bit. 

The reason about the mixture and title is that only by fixing the acquisition of relocation information can 'strip' and 'readelf -w' work normally.

Now it seems really confusing, so I want to do some changes based on your opinions, change the titles or reorganize these patches to make them look more logical.

> I don't have
> access to a MIPS system to test them on. There are a couple of
> different MIPS abis (I believe all combinations of 32/64 bit and
> big/little endianness), but people have only tested on mips64le (maybe
> that is the only relevant one these days?) 

I have tested other mips abi before, I will test it agagin and attach the results of 'make check' with and without patch.

And add a description of mips abi support in the commit message.

> And finally the way MIPS
> represents relocations is slightly different than any other ELF
> architecture does. So we have to translate that somewhere to make the
> standards functions work. I have to convince myself that doing that in
> elf_getdata as the patches do is the right place.


The controversial problem was the location of the code, the code was to change the original relocation info to ensure mips can obtain the correct index value and symble index.

Or we did not modify the original data, we modify the way to obtain the index value and symble index at funtion 'gelf_getrela' in file 'libelf/gelf_getrela.c','libelf/gelf_getrel.c','libelf/gelf_update_rela.c','libelf/gelf_update_rel.c'.

Where the function 'gelf_getrela' is called，modify the relocation info that has been obtained .

What do you think of this?


Thanks,

Ying

>
>> Many thanks everyone for your help,
>> Tony
>>
>> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
>> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
>> [3]:
>> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Ying Huang]

Hi Tony,

Thanks for your fix about the conflict and test about the whole patch.

Now, there is currently one test case that has not been added, which is

tests/run-backtrace-core-mips.sh.
Thanks,
Ying
在 2024/6/11 14:36, Tony Ambardar 写道:
> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
>> Hi Mark,
>>
>> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
>>> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
>>>> Couldn't find a way to ask eu-readelf for more verbose output, where we
>>>> could perhaps get some clue as to why it produces nothing while binutils
>>>> readelf manages to grok it, Mark, do you know some other way to ask
>>>> eu-readelf to produce more debug output?
>>>>
>>>> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
>>>> that then made eu-readelf to not be able to process it while pahole,
>>>> that uses eltuils' libraries, was able to process the first two CUs for
>>>> a kernel module and all the CUs for the vmlinux file :-\
>>>>
>>>> Mark, the whole thread is available at:
>>>>
>>>> https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
>>> I haven't looked at the vmlinux file. But for the .ko file the issue
>>> is that the elfutils MIPS backend isn't complete. Specifically MIPS
>>> relocations aren't recognized (and so cannot be applied). There are
>>> some pending patches which try to fix that:
>>>
>>> https://patchwork.sourceware.org/project/elfutils/list/?series=31601
>> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
>> work for MIPS, and I locally rebuilt elfutils and then pahole from their
>> respective next/main branches. For elfutils, main (935ee131cf7c) includes
>>
>>   e259f126 Support Mips architecture
>>   f2acb069 stack: Fix stack unwind failure on mips
>>   db33cb0c backends: Add register_info, return_value_location, core_note mips
>>
>> which partially applies the patchwork series but leaves out the support for
>> readelf, strip, and elflint.
>>
>> I believe this means the vmlinux and .ko files I shared are OK, or is there
>> more backend work needed for MIPS?
>>
>> The bits missing in eu-readelf would explain the blank output both Arnaldo
>> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
>> patchwork readelf patch locally but ran into merge conflicts.
>>
>> CCing Ying Huang for any more insight.
>>
>> Thanks,
>> Tony
> Hello all,
>
> A short update, starting with answering my own question.
>
> No, apparently the above commits *do not* complete the backend work. Ying
> Huang submitted additional related patches since March 5: [1][2]
>
>     strip: Adapt src/strip -o -f on mips
>     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
>     elflint: adapt src/elflint --gnu src/nm on mips
>     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
>
> Despite the titles, these patches do include core backend changes for MIPS.
> I resolved the various merge conflicts [3], rebuilt elfutils, and retested
> kernel builds to now find:
>
>   - pahole is able to read DWARF[45] info and create .BTF for modules
>   - resolve_btfids can successfully patch .BTF_ids in modules
>   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
>
> Huzzah!
>
>
> Ying:
>
> Thank you for developing these MIPS patches. In your view, are the MIPS
> changes now complete, or do you plan further updates that might improve or
> impact parsing DWARF debug/reloc info in apps like pahole?
>
>
> Mark:
>
> Given that BTF usage on Linux/MIPS is basically broken without these
> patches, could I request some of your review time for them to be merged? If
> it's helpful, my branch [3] includes all patches with conflicts fixed, and
> I also successfully ran the elfutils self-tests (including MIPS from Ying).
> Please feel free to add for these patches:
>
>     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>
>
>
> Arnaldo:
>
> Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
> I reran your tests with the updated elfutils and latest pahole (pre-1.27),
> and then found:
>
>   - everything that worked before, still works
>   - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
>   - we now see expected output from "eu-readelf -winfo netdevsim.ko"
>
> Regarding pahole, DWARF parsing and BTF generation now works:
> (with no more die__process: error messages seen)
>
>     kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
>     14504
>
> but strangely pahole still doesn't read its own generated BTF:
>
>     kodidev:~/linux$ pahole -F btf netdevsim.ko
>     libbpf: Invalid BTF string section
>     pahole: file 'netdevsim.ko' has no btf type information.
>
> Poking inside a little further:
>     
>     kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
>     [...]
>     argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
>     SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
>     SYS_brk(0)                                       = 0x563d47e37000
>     SYS_brk(0x563d47e58000)                          = 0x563d47e58000
>     <... argp_parse resumed> )                       = 0
>     dwarves__init(0x20000, 0, -4096, 213)            = 0
>     dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
>     cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
>     memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
>     strlen("/sys/kernel/btf/")                       = 16
>     strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
>     cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
>     0x563d47da40c0 <unfinished ...>
>     SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
>     SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
>     SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
>     SYS_close(3)                                     = 0
>     SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
>     SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
>     SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
>     SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
>     SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
>     SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
>     SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
>     SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
>     string section
>     ) = 35
>     SYS_close(3)                                     = 0
>     <... cus__load_files resumed> )                  = 0xffffffff
>     access("netdevsim.ko", 4 <unfinished ...>
>     SYS_access("netdevsim.ko", 04)                   = 0
>     <... access resumed> )                           = 0
>     fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
>     "netdevsim.ko", "btf" <unfinished ...>
>     SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
>     'netdevsim.ko' has no btf type information.
>     ) = 57
>     <... fprintf resumed> )                          = 57
>     SYS_exit_group(1 <no return ...>
>     +++ exited (status 1) +++
>
> Could you help investigate this further? Maybe a libbpf issue? For the
> record, I also tried building pahole with embedded libbpf 1.4.3 without
> any change. (side note: please make pahole --version also cover libbpf)
>
>
> Many thanks everyone for your help,
> Tony
>
> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> [3]:
> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/