Problem with BTF generation on mips64el

Problem with BTF generation on mips64el

[Tony Ambardar]

[-- Attachment #1: Type: text/plain, Size: 5491 bytes --]

Hello,

For some time now I'm seeing multiple issues during BTF generation while
building recent kernels targeting mips64el, and would appreciate some help
to understand and fix the problems.

Some relate to resolve_btfids:

>   LD      vmlinux
>   BTFIDS  vmlinux
> WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_key_put
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
>   NM      System.map
>   SORTTAB vmlinux
>   OBJCOPY vmlinux.32

These do not appear to be #ifdef-related and have similar past reports [1].

I also see many pahole failures during BTF encoding of modules, such as:

>   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
>   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
>   CC [M]  net/psample/psample.mod.o
>   LD [M]  crypto/cmac.ko
>   BTF [M] crypto/cmac.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
>   LD [M]  lib/test_bpf.ko
>   BTF [M] lib/test_bpf.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
>   LD [M]  lib/crc-ccitt.ko
>   BTF [M] lib/crc-ccitt.ko
> die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> die__process_unit: tag not supported 0x11 (compile_unit)!
> die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
>   LD [M]  lib/libcrc32c.ko
>   BTF [M] lib/libcrc32c.ko
> die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> die__process_unit: tag not supported 0x11 (compile_unit)!
> die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
>   LD [M]  lib/ts_kmp.ko
>   BTF [M] lib/ts_kmp.ko
>   LD [M]  lib/ts_bm.ko
>   BTF [M] lib/ts_bm.ko
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!
> die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> or DW_TAG_skeleton_unit expected got member (0xd)!

I have seen reports of various similar "die__" messages on the dwarves
list and repo, with the hint of an elfutils connection [2] but nothing
conclusive.

Details of the git commit and build environment are as follows:

> $ git log -1 --oneline  bpf/master
> 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> test_sk_storage_tracing.c
> 
> $ lsb_release -a
> Description:    Ubuntu 22.04.4 LTS
> 
> $ cat gcc-compile.txt
> ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> 
> $ mips64el-linux-gnuabi64-gcc --version
> mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> 
> $ mips64el-linux-gnuabi64-ld --version
> GNU ld (GNU Binutils for Ubuntu) 2.38
> 
> $ pahole --version
> v1.26
> 
> $ ldd $(which pahole)
>         linux-vdso.so.1 (0x00007fff16f3f000)
>         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
>         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
>         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
>         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
>         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
>         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
>         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> 
> $ dpkg -s elfutils
> Package: elfutils
> ...
> Version: 0.186-1build1
> Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)

For reference, I also attached the full .config and build log from the
above.

I should add this is not only a problem with the latest bpf/master but
also appears to affect the 6.6.x LTS kernel, which I tested while building
a mips64el OpenWrt distro image. That build environment employs the latest
gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.

Not only do I see similar warnings from resolve_btfids and pahole, but
while running the distro image I encounter module loading failures that
suggest ELF corruption in some module .ko files, based on the following:

> root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> ...
> init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> ...

> $ man init_module
> ...
> The following errors may additionally occur for init_module():
> ...
>      EINVAL param_values is invalid, or some part of the ELF image in
>      module_image contains inconsistencies.
> ...

I'd be grateful if some of the BTF/pahole experts could please review this
issue and share next steps or other details I might provide.

Thanks,
Tony Ambardar

Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
Link: https://github.com/acmel/dwarves/issues/45 [2]

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 22159 bytes --]

[-- Attachment #3: mips64el-linux-gnuabi64.log.gz --]
[-- Type: application/gzip, Size: 20880 bytes --]

Re: Problem with BTF generation on mips64el

[Hengqi Chen]

Hi Tony,

On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
>
> Hello,
>
> For some time now I'm seeing multiple issues during BTF generation while
> building recent kernels targeting mips64el, and would appreciate some help
> to understand and fix the problems.
>
> Some relate to resolve_btfids:
>
> >   LD      vmlinux
> >   BTFIDS  vmlinux
> > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> >   NM      System.map
> >   SORTTAB vmlinux
> >   OBJCOPY vmlinux.32
>
> These do not appear to be #ifdef-related and have similar past reports [1].
>
> I also see many pahole failures during BTF encoding of modules, such as:
>
> >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> >   CC [M]  net/psample/psample.mod.o
> >   LD [M]  crypto/cmac.ko
> >   BTF [M] crypto/cmac.ko
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!

The issue seems to be related to elfutils. Have you tried build from
the latest elfutils source ?
I saw the latest MIPS backend in elfutils already implemented the
reloc_simple_type hook.

> >   LD [M]  lib/test_bpf.ko
> >   BTF [M] lib/test_bpf.ko
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!
> >   LD [M]  lib/crc-ccitt.ko
> >   BTF [M] lib/crc-ccitt.ko
> > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> > die__process_unit: tag not supported 0x11 (compile_unit)!
> > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> >   LD [M]  lib/libcrc32c.ko
> >   BTF [M] lib/libcrc32c.ko
> > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> > die__process_unit: tag not supported 0x11 (compile_unit)!
> > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> >   LD [M]  lib/ts_kmp.ko
> >   BTF [M] lib/ts_kmp.ko
> >   LD [M]  lib/ts_bm.ko
> >   BTF [M] lib/ts_bm.ko
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!
> > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > or DW_TAG_skeleton_unit expected got member (0xd)!
>
> I have seen reports of various similar "die__" messages on the dwarves
> list and repo, with the hint of an elfutils connection [2] but nothing
> conclusive.
>
> Details of the git commit and build environment are as follows:
>
> > $ git log -1 --oneline  bpf/master
> > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > test_sk_storage_tracing.c
> >
> > $ lsb_release -a
> > Description:    Ubuntu 22.04.4 LTS
> >
> > $ cat gcc-compile.txt
> > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> >
> > $ mips64el-linux-gnuabi64-gcc --version
> > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> >
> > $ mips64el-linux-gnuabi64-ld --version
> > GNU ld (GNU Binutils for Ubuntu) 2.38
> >
> > $ pahole --version
> > v1.26
> >
> > $ ldd $(which pahole)
> >         linux-vdso.so.1 (0x00007fff16f3f000)
> >         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
> >         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
> >         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
> >         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
> >         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
> >         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
> >         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> >
> > $ dpkg -s elfutils
> > Package: elfutils
> > ...
> > Version: 0.186-1build1
> > Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> > libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
>
> For reference, I also attached the full .config and build log from the
> above.
>
> I should add this is not only a problem with the latest bpf/master but
> also appears to affect the 6.6.x LTS kernel, which I tested while building
> a mips64el OpenWrt distro image. That build environment employs the latest
> gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
>
> Not only do I see similar warnings from resolve_btfids and pahole, but
> while running the distro image I encounter module loading failures that
> suggest ELF corruption in some module .ko files, based on the following:
>
> > root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> > ...
> > init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> > ...
>
> > $ man init_module
> > ...
> > The following errors may additionally occur for init_module():
> > ...
> >      EINVAL param_values is invalid, or some part of the ELF image in
> >      module_image contains inconsistencies.
> > ...
>
> I'd be grateful if some of the BTF/pahole experts could please review this
> issue and share next steps or other details I might provide.
>
> Thanks,
> Tony Ambardar
>
> Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> Link: https://github.com/acmel/dwarves/issues/45 [2]

Cheers,
Hengqi

Re: Problem with BTF generation on mips64el

[Jiri Olsa]

On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> Hi Tony,
> 
> On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > Hello,
> >
> > For some time now I'm seeing multiple issues during BTF generation while
> > building recent kernels targeting mips64el, and would appreciate some help
> > to understand and fix the problems.
> >
> > Some relate to resolve_btfids:
> >
> > >   LD      vmlinux
> > >   BTFIDS  vmlinux
> > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > >   NM      System.map
> > >   SORTTAB vmlinux
> > >   OBJCOPY vmlinux.32
> >
> > These do not appear to be #ifdef-related and have similar past reports [1].

I can reproduce the warning just for bpf_session_cookie,
which has fix in progress:
  https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t

> >
> > I also see many pahole failures during BTF encoding of modules, such as:
> >
> > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > >   CC [M]  net/psample/psample.mod.o
> > >   LD [M]  crypto/cmac.ko
> > >   BTF [M] crypto/cmac.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> The issue seems to be related to elfutils. Have you tried build from
> the latest elfutils source ?
> I saw the latest MIPS backend in elfutils already implemented the
> reloc_simple_type hook.

hi,
+1, could you also check the pahole version you used?

jirka

> 
> > >   LD [M]  lib/test_bpf.ko
> > >   BTF [M] lib/test_bpf.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > >   LD [M]  lib/crc-ccitt.ko
> > >   BTF [M] lib/crc-ccitt.ko
> > > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> > > die__process_unit: tag not supported 0x11 (compile_unit)!
> > > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> > >   LD [M]  lib/libcrc32c.ko
> > >   BTF [M] lib/libcrc32c.ko
> > > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> > > die__process_unit: tag not supported 0x11 (compile_unit)!
> > > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> > >   LD [M]  lib/ts_kmp.ko
> > >   BTF [M] lib/ts_kmp.ko
> > >   LD [M]  lib/ts_bm.ko
> > >   BTF [M] lib/ts_bm.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> >
> > I have seen reports of various similar "die__" messages on the dwarves
> > list and repo, with the hint of an elfutils connection [2] but nothing
> > conclusive.
> >
> > Details of the git commit and build environment are as follows:
> >
> > > $ git log -1 --oneline  bpf/master
> > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > test_sk_storage_tracing.c
> > >
> > > $ lsb_release -a
> > > Description:    Ubuntu 22.04.4 LTS
> > >
> > > $ cat gcc-compile.txt
> > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > >
> > > $ mips64el-linux-gnuabi64-gcc --version
> > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > >
> > > $ mips64el-linux-gnuabi64-ld --version
> > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > >
> > > $ pahole --version
> > > v1.26
> > >
> > > $ ldd $(which pahole)
> > >         linux-vdso.so.1 (0x00007fff16f3f000)
> > >         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
> > >         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
> > >         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
> > >         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
> > >         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
> > >         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
> > >         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> > >
> > > $ dpkg -s elfutils
> > > Package: elfutils
> > > ...
> > > Version: 0.186-1build1
> > > Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> > > libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
> >
> > For reference, I also attached the full .config and build log from the
> > above.
> >
> > I should add this is not only a problem with the latest bpf/master but
> > also appears to affect the 6.6.x LTS kernel, which I tested while building
> > a mips64el OpenWrt distro image. That build environment employs the latest
> > gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
> >
> > Not only do I see similar warnings from resolve_btfids and pahole, but
> > while running the distro image I encounter module loading failures that
> > suggest ELF corruption in some module .ko files, based on the following:
> >
> > > root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> > > ...
> > > init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> > > ...
> >
> > > $ man init_module
> > > ...
> > > The following errors may additionally occur for init_module():
> > > ...
> > >      EINVAL param_values is invalid, or some part of the ELF image in
> > >      module_image contains inconsistencies.
> > > ...
> >
> > I'd be grateful if some of the BTF/pahole experts could please review this
> > issue and share next steps or other details I might provide.
> >
> > Thanks,
> > Tony Ambardar
> >
> > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > Link: https://github.com/acmel/dwarves/issues/45 [2]
> 
> Cheers,
> Hengqi
> 

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

Hello Hengqi,

On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> Hi Tony,
> 
> On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > Hello,
> >
> > For some time now I'm seeing multiple issues during BTF generation while
> > building recent kernels targeting mips64el, and would appreciate some help
> > to understand and fix the problems.
> >
> SNIP
> >
> > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > >   CC [M]  net/psample/psample.mod.o
> > >   LD [M]  crypto/cmac.ko
> > >   BTF [M] crypto/cmac.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> The issue seems to be related to elfutils. Have you tried build from
> the latest elfutils source ?
> I saw the latest MIPS backend in elfutils already implemented the
> reloc_simple_type hook.

Good idea. I tried rebuilding elfutils from the latest upstream commit:
https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508

I then linked this elfutils with pahole built from the latest pahole/next:
https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993

I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
the kernel with the same config. Unfortunately, the warnings/errors from
resolve_btfids and pahole still occur.

> SNIP
> >
> > I'd be grateful if some of the BTF/pahole experts could please review this
> > issue and share next steps or other details I might provide.
> >
> > Thanks,
> > Tony Ambardar
> >
> > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > Link: https://github.com/acmel/dwarves/issues/45 [2]
> 
> Cheers,
> Hengqi

Thanks for the suggestion,
Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

Hi Jiri,

On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > Hi Tony,
> > 
> > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > For some time now I'm seeing multiple issues during BTF generation while
> > > building recent kernels targeting mips64el, and would appreciate some help
> > > to understand and fix the problems.
> > >
> > > Some relate to resolve_btfids:
> > >
> > > >   LD      vmlinux
> > > >   BTFIDS  vmlinux
> > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > >   NM      System.map
> > > >   SORTTAB vmlinux
> > > >   OBJCOPY vmlinux.32
> > >
> > > These do not appear to be #ifdef-related and have similar past reports [1].
> 
> I can reproduce the warning just for bpf_session_cookie,
> which has fix in progress:
>   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t

I gather there are different root causes for these warnings. From the link,
the issue with bpf_session_cookie seems related to conditional compilation,
which have come up before on the mailing list IIRC.

In comparison, consider my above warning for bpf_key_put, which is defined
in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
enabled in my config and so not the issue. I can in fact see the global
text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.

So I suspect the warnings might come from linker or optimization problems,
or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?

> 
> > >
> > > I also see many pahole failures during BTF encoding of modules, such as:
> > >
> > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > >   CC [M]  net/psample/psample.mod.o
> > > >   LD [M]  crypto/cmac.ko
> > > >   BTF [M] crypto/cmac.ko
> > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > 
> > The issue seems to be related to elfutils. Have you tried build from
> > the latest elfutils source ?
> > I saw the latest MIPS backend in elfutils already implemented the
> > reloc_simple_type hook.
> 
> hi,
> +1, could you also check the pahole version you used?

No luck I'm afraid with using the latest elfutils as suggested.

I used pahole v1.26 in my original testing for this bug report, as noted
below (might have been buried a bit).

> 
> jirka
> 
> SNIP
>
> > > Details of the git commit and build environment are as follows:
> > >
> > > > $ git log -1 --oneline  bpf/master
> > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > test_sk_storage_tracing.c
> > > >
> > > > $ lsb_release -a
> > > > Description:    Ubuntu 22.04.4 LTS
> > > >
> > > > $ cat gcc-compile.txt
> > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > >
> > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > >
> > > > $ mips64el-linux-gnuabi64-ld --version
> > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > >
> > > > $ pahole --version
> > > > v1.26
> 
> SNIP
>
> > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > issue and share next steps or other details I might provide.
> > >
> > > Thanks,
> > > Tony Ambardar
> > >
> > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > 
> > Cheers,
> > Hengqi

Thanks,
Tony

Re: Problem with BTF generation on mips64el

[Jiri Olsa]

On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> Hi Jiri,
> 
> On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > > Some relate to resolve_btfids:
> > > >
> > > > >   LD      vmlinux
> > > > >   BTFIDS  vmlinux
> > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > >   NM      System.map
> > > > >   SORTTAB vmlinux
> > > > >   OBJCOPY vmlinux.32
> > > >
> > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > 
> > I can reproduce the warning just for bpf_session_cookie,
> > which has fix in progress:
> >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> 
> I gather there are different root causes for these warnings. From the link,
> the issue with bpf_session_cookie seems related to conditional compilation,
> which have come up before on the mailing list IIRC.
> 
> In comparison, consider my above warning for bpf_key_put, which is defined
> in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> enabled in my config and so not the issue. I can in fact see the global
> text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.

yes, that would be a problem, the resolve_btfids goes through the BTF
lists and needs to resolve the function in vmlinux, when it's not there,
it will output the warning above

> 
> So I suspect the warnings might come from linker or optimization problems,
> or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?

I can reproduce now resolve_btfids warnings:

  LD      vmlinux
  BTFIDS  vmlinux
WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
WARN: resolve_btfids: unresolved symbol bpf_session_is_return
WARN: resolve_btfids: unresolved symbol bpf_session_cookie
WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
WARN: resolve_btfids: unresolved symbol bpf_key_put
WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages

when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
the __bpf_kfunc flags do not work properly on this cross compile chain..
and linker just optimize it away?

#define __bpf_kfunc __used noinline

thanks,
jirka

> 
> > 
> > > >
> > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > 
> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > hi,
> > +1, could you also check the pahole version you used?
> 
> No luck I'm afraid with using the latest elfutils as suggested.
> 
> I used pahole v1.26 in my original testing for this bug report, as noted
> below (might have been buried a bit).
> 
> > 
> > jirka
> > 
> > SNIP
> >
> > > > Details of the git commit and build environment are as follows:
> > > >
> > > > > $ git log -1 --oneline  bpf/master
> > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > test_sk_storage_tracing.c
> > > > >
> > > > > $ lsb_release -a
> > > > > Description:    Ubuntu 22.04.4 LTS
> > > > >
> > > > > $ cat gcc-compile.txt
> > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > >
> > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > >
> > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > >
> > > > > $ pahole --version
> > > > > v1.26
> > 
> > SNIP
> >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> 
> Thanks,
> Tony

Re: Problem with BTF generation on mips64el

[Arnaldo Carvalho de Melo]

On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> Hello Hengqi,
> 
> On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > Hi Tony,
> > 
> > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > For some time now I'm seeing multiple issues during BTF generation while
> > > building recent kernels targeting mips64el, and would appreciate some help
> > > to understand and fix the problems.
> > >
> > SNIP
> > >
> > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > >   CC [M]  net/psample/psample.mod.o
> > > >   LD [M]  crypto/cmac.ko
> > > >   BTF [M] crypto/cmac.ko
> > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > or DW_TAG_skeleton_unit expected got member (0xd)!

Can you check the kernel CONFIG_ variables related to DEBUG information
and post them here? I have this on fedora:

[acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_DWARF5 is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
[acme@nine linux]$

If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
fedora config.

- Arnaldo

> > The issue seems to be related to elfutils. Have you tried build from
> > the latest elfutils source ?
> > I saw the latest MIPS backend in elfutils already implemented the
> > reloc_simple_type hook.
> 
> Good idea. I tried rebuilding elfutils from the latest upstream commit:
> https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> 
> I then linked this elfutils with pahole built from the latest pahole/next:
> https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> 
> I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> the kernel with the same config. Unfortunately, the warnings/errors from
> resolve_btfids and pahole still occur.
> 
> > SNIP
> > >
> > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > issue and share next steps or other details I might provide.
> > >
> > > Thanks,
> > > Tony Ambardar
> > >
> > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > 
> > Cheers,
> > Hengqi
> 
> Thanks for the suggestion,
> Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > Hello Hengqi,
> > 
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > SNIP
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> Can you check the kernel CONFIG_ variables related to DEBUG information
> and post them here? I have this on fedora:
> 
> [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> [acme@nine linux]$
> 
> If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> fedora config.
> 
> - Arnaldo
> 

OK, I have those 3 settings as expected on bpf/master branch:

kodidev:~/linux$ grep CONFIG_DEBUG_INFO .config
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_NONE is not set
# CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT is not set
CONFIG_DEBUG_INFO_DWARF4=y
# CONFIG_DEBUG_INFO_DWARF5 is not set
# CONFIG_DEBUG_INFO_REDUCED is not set
CONFIG_DEBUG_INFO_COMPRESSED_NONE=y
# CONFIG_DEBUG_INFO_COMPRESSED_ZLIB is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y

I also attached the full .config in my original mail if you think other
config options might be suspect.

Is there anything else I can check? What do you make of the many pahole
"die__process:" warnings? Could it be related to your comment here:
https://github.com/acmel/dwarves/issues/45#issuecomment-1974004606

Thanks,
Tony

> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > Good idea. I tried rebuilding elfutils from the latest upstream commit:
> > https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> > 
> > I then linked this elfutils with pahole built from the latest pahole/next:
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> > 
> > I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> > the kernel with the same config. Unfortunately, the warnings/errors from
> > resolve_btfids and pahole still occur.
> > 
> > > SNIP
> > > >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> > 
> > Thanks for the suggestion,
> > Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Fri, May 31, 2024 at 04:21:48PM +0200, Jiri Olsa wrote:
> On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> > Hi Jiri,
> > 
> > On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > Hi Tony,
> > > > 
> > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > to understand and fix the problems.
> > > > >
> > > > > Some relate to resolve_btfids:
> > > > >
> > > > > >   LD      vmlinux
> > > > > >   BTFIDS  vmlinux
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > > >   NM      System.map
> > > > > >   SORTTAB vmlinux
> > > > > >   OBJCOPY vmlinux.32
> > > > >
> > > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > > 
> > > I can reproduce the warning just for bpf_session_cookie,
> > > which has fix in progress:
> > >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> > 
> > I gather there are different root causes for these warnings. From the link,
> > the issue with bpf_session_cookie seems related to conditional compilation,
> > which have come up before on the mailing list IIRC.
> > 
> > In comparison, consider my above warning for bpf_key_put, which is defined
> > in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> > enabled in my config and so not the issue. I can in fact see the global
> > text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
> 
> yes, that would be a problem, the resolve_btfids goes through the BTF
> lists and needs to resolve the function in vmlinux, when it's not there,
> it will output the warning above
> 
> > 
> > So I suspect the warnings might come from linker or optimization problems,
> > or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
> 
> I can reproduce now resolve_btfids warnings:
> 

That's great to hear! Did you also use mips64el? And were you able to see
the various "die__proc:" errors from pahole during BTF generation?

>   LD      vmlinux
>   BTFIDS  vmlinux
> WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> WARN: resolve_btfids: unresolved symbol bpf_session_is_return
> WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_key_put
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> 
> when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
> the __bpf_kfunc flags do not work properly on this cross compile chain..
> and linker just optimize it away?

Yes, I verified the problem occurs during link-time garbage cleaning, when
LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.

> 
> #define __bpf_kfunc __used noinline
> 

After researching, I believe this isn't a mips problem specifically, rather
the __bpf_kfunc tag only partly works. It appears the __used macro works at
the compiler and IR-level but is ignored by the linker. I have a working
fix that also handles the linker level, so I'll post the patches shortly.

Unfortunately, that still leaves the pahole errors generating BTF, which
are serious as they seem to break module loading.

> thanks,
> jirka

Cheers,
Tony
> 
> > 
> > > 
> > > > >
> > > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > > >
> > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > >   LD [M]  crypto/cmac.ko
> > > > > >   BTF [M] crypto/cmac.ko
> > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > > 
> > > > The issue seems to be related to elfutils. Have you tried build from
> > > > the latest elfutils source ?
> > > > I saw the latest MIPS backend in elfutils already implemented the
> > > > reloc_simple_type hook.
> > > 
> > > hi,
> > > +1, could you also check the pahole version you used?
> > 
> > No luck I'm afraid with using the latest elfutils as suggested.
> > 
> > I used pahole v1.26 in my original testing for this bug report, as noted
> > below (might have been buried a bit).
> > 
> > > 
> > > jirka
> > > 
> > > SNIP
> > >
> > > > > Details of the git commit and build environment are as follows:
> > > > >
> > > > > > $ git log -1 --oneline  bpf/master
> > > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > > test_sk_storage_tracing.c
> > > > > >
> > > > > > $ lsb_release -a
> > > > > > Description:    Ubuntu 22.04.4 LTS
> > > > > >
> > > > > > $ cat gcc-compile.txt
> > > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > > >
> > > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > > >
> > > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > > >
> > > > > > $ pahole --version
> > > > > > v1.26
> > > 
> > > SNIP
> > >
> > > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > > issue and share next steps or other details I might provide.
> > > > >
> > > > > Thanks,
> > > > > Tony Ambardar
> > > > >
> > > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > > 
> > > > Cheers,
> > > > Hengqi
> > 
> > Thanks,
> > Tony

Re: Problem with BTF generation on mips64el

[Jiri Olsa]

On Mon, Jun 03, 2024 at 02:02:45AM -0700, Tony Ambardar wrote:
> On Fri, May 31, 2024 at 04:21:48PM +0200, Jiri Olsa wrote:
> > On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> > > Hi Jiri,
> > > 
> > > On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > > Hi Tony,
> > > > > 
> > > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > > to understand and fix the problems.
> > > > > >
> > > > > > Some relate to resolve_btfids:
> > > > > >
> > > > > > >   LD      vmlinux
> > > > > > >   BTFIDS  vmlinux
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > > > >   NM      System.map
> > > > > > >   SORTTAB vmlinux
> > > > > > >   OBJCOPY vmlinux.32
> > > > > >
> > > > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > > > 
> > > > I can reproduce the warning just for bpf_session_cookie,
> > > > which has fix in progress:
> > > >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> > > 
> > > I gather there are different root causes for these warnings. From the link,
> > > the issue with bpf_session_cookie seems related to conditional compilation,
> > > which have come up before on the mailing list IIRC.
> > > 
> > > In comparison, consider my above warning for bpf_key_put, which is defined
> > > in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> > > enabled in my config and so not the issue. I can in fact see the global
> > > text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
> > 
> > yes, that would be a problem, the resolve_btfids goes through the BTF
> > lists and needs to resolve the function in vmlinux, when it's not there,
> > it will output the warning above
> > 
> > > 
> > > So I suspect the warnings might come from linker or optimization problems,
> > > or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
> > 
> > I can reproduce now resolve_btfids warnings:
> > 
> 
> That's great to hear! Did you also use mips64el? And were you able to see

nope, just mips64, couldn't find mips64el on fedora

> the various "die__proc:" errors from pahole during BTF generation?

yep, I can see them as well

> 
> >   LD      vmlinux
> >   BTFIDS  vmlinux
> > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > WARN: resolve_btfids: unresolved symbol bpf_session_is_return
> > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > 
> > when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
> > the __bpf_kfunc flags do not work properly on this cross compile chain..
> > and linker just optimize it away?
> 
> Yes, I verified the problem occurs during link-time garbage cleaning, when
> LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.
> 
> > 
> > #define __bpf_kfunc __used noinline
> > 
> 
> After researching, I believe this isn't a mips problem specifically, rather
> the __bpf_kfunc tag only partly works. It appears the __used macro works at
> the compiler and IR-level but is ignored by the linker. I have a working
> fix that also handles the linker level, so I'll post the patches shortly.

great, I was wondering how to fix that

thanks,
jirka

> 
> Unfortunately, that still leaves the pahole errors generating BTF, which
> are serious as they seem to break module loading.
> 
> > thanks,
> > jirka
> 
> Cheers,
> Tony
> > 
> > > 
> > > > 
> > > > > >
> > > > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > > > >
> > > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > > >   LD [M]  crypto/cmac.ko
> > > > > > >   BTF [M] crypto/cmac.ko
> > > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > > > 
> > > > > The issue seems to be related to elfutils. Have you tried build from
> > > > > the latest elfutils source ?
> > > > > I saw the latest MIPS backend in elfutils already implemented the
> > > > > reloc_simple_type hook.
> > > > 
> > > > hi,
> > > > +1, could you also check the pahole version you used?
> > > 
> > > No luck I'm afraid with using the latest elfutils as suggested.
> > > 
> > > I used pahole v1.26 in my original testing for this bug report, as noted
> > > below (might have been buried a bit).
> > > 
> > > > 
> > > > jirka
> > > > 
> > > > SNIP
> > > >
> > > > > > Details of the git commit and build environment are as follows:
> > > > > >
> > > > > > > $ git log -1 --oneline  bpf/master
> > > > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > > > test_sk_storage_tracing.c
> > > > > > >
> > > > > > > $ lsb_release -a
> > > > > > > Description:    Ubuntu 22.04.4 LTS
> > > > > > >
> > > > > > > $ cat gcc-compile.txt
> > > > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > > > >
> > > > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > > > >
> > > > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > > > >
> > > > > > > $ pahole --version
> > > > > > > v1.26
> > > > 
> > > > SNIP
> > > >
> > > > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > > > issue and share next steps or other details I might provide.
> > > > > >
> > > > > > Thanks,
> > > > > > Tony Ambardar
> > > > > >
> > > > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > > > 
> > > > > Cheers,
> > > > > Hengqi
> > > 
> > > Thanks,
> > > Tony

Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > Hello Hengqi,
> > 
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > SNIP
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> Can you check the kernel CONFIG_ variables related to DEBUG information
> and post them here? I have this on fedora:
> 
> [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> [acme@nine linux]$
> 
> If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> fedora config.

One more useful observation: the pahole BTF generation problems appear only
for mips64. If I build the same config for mips32be I see none of those
"die__process: errors" from pahole and the processed modules load properly.

I also tried running pahole under gdb to look further, but lack knowledge
of pahole and libdw internals, so couldn't narrow things down to a pahole,
elfutils or compiler-output problem.

To help troubleshoot further, I packaged my base vmlinux, some module .ko
files showing problems, and a reproducer script. I've uploaded the tarball
here:

https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

Thanks,
Tony

> 
> - Arnaldo
> 
> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > Good idea. I tried rebuilding elfutils from the latest upstream commit:
> > https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> > 
> > I then linked this elfutils with pahole built from the latest pahole/next:
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> > 
> > I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> > the kernel with the same config. Unfortunately, the warnings/errors from
> > resolve_btfids and pahole still occur.
> > 
> > > SNIP
> > > >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> > 
> > Thanks for the suggestion,
> > Tony

Re: Problem with BTF generation on mips64el

[Arnaldo Carvalho de Melo]

On Mon, Jun 03, 2024 at 04:20:01AM -0700, Tony Ambardar wrote:
> On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> > On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > to understand and fix the problems.

> > > > SNIP

> > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > >   LD [M]  crypto/cmac.ko
> > > > > >   BTF [M] crypto/cmac.ko
> > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!

> > Can you check the kernel CONFIG_ variables related to DEBUG information
> > and post them here? I have this on fedora:

> > [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> > CONFIG_DEBUG_INFO=y
> > # CONFIG_DEBUG_INFO_REDUCED is not set
> > # CONFIG_DEBUG_INFO_COMPRESSED is not set
> > # CONFIG_DEBUG_INFO_SPLIT is not set
> > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> > # CONFIG_DEBUG_INFO_DWARF4 is not set
> > # CONFIG_DEBUG_INFO_DWARF5 is not set
> > CONFIG_DEBUG_INFO_BTF=y
> > CONFIG_DEBUG_INFO_BTF_MODULES=y
> > [acme@nine linux]$

> > If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> > CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> > fedora config.

> One more useful observation: the pahole BTF generation problems appear only
> for mips64. If I build the same config for mips32be I see none of those
> "die__process: errors" from pahole and the processed modules load properly.

> I also tried running pahole under gdb to look further, but lack knowledge
> of pahole and libdw internals, so couldn't narrow things down to a pahole,
> elfutils or compiler-output problem.
> 
> To help troubleshoot further, I packaged my base vmlinux, some module .ko
> files showing problems, and a reproducer script. I've uploaded the tarball
> here:
> 
> https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

So, with that vmlinux, on a Fedora 39 x86_64 notebook:

⬢[acme@toolbox repro_die__process]$ readelf -wi vmlinux | head -45
Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x3f (32-bit)
   Version:       4
   Abbrev Offset: 0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_stmt_list   : 0
    <10>   DW_AT_ranges      : 0
    <14>   DW_AT_name        : (indirect string, offset: 0): arch/mips/kernel/head.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2c): GNU AS 2.42
    <20>   DW_AT_language    : 32769	(MIPS assembler)
 <1><22>: Abbrev Number: 2 (DW_TAG_subprogram)
    <23>   DW_AT_name        : (indirect string, offset: 0x38): kernel_entry
    <27>   DW_AT_external    : 1
    <27>   DW_AT_type        : <0x41>
    <28>   DW_AT_low_pc      : 0xffffffff80b0ce68
    <30>   DW_AT_high_pc     : 172
 <1><32>: Abbrev Number: 2 (DW_TAG_subprogram)
    <33>   DW_AT_name        : (indirect string, offset: 0x45): smp_bootstrap
    <37>   DW_AT_external    : 1
    <37>   DW_AT_type        : <0x41>
    <38>   DW_AT_low_pc      : 0xffffffff80b0cf14
    <40>   DW_AT_high_pc     : 44
 <1><41>: Abbrev Number: 3 (DW_TAG_unspecified_type)
 <1><42>: Abbrev Number: 0
  Compilation Unit @ offset 0x43:
   Length:        0x22ed9 (32-bit)
   Version:       4
   Abbrev Offset: 0x26
   Pointer Size:  8
 <0><4e>: Abbrev Number: 192 (DW_TAG_compile_unit)
    <50>   DW_AT_producer    : (indirect string, offset: 0x8147): GNU C11 13.3.0 -G 0 -mel -mno-check-zero-division -mabi=64 -mno-abicalls -msoft-float -march=mips64r2 -msym32 -mllsc -mips64r2 -mno-shared -g -gdwarf-4 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-pic -ffreestanding -fstack-check=no -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector -fno-stack-clash-protection -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -ffunction-sections -fdata-sections
    <54>   DW_AT_language    : 12	(ANSI C99)
    <55>   DW_AT_name        : (indirect string, offset: 0xce66): init/main.c
    <59>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <5d>   DW_AT_ranges      : 0x1350
    <61>   DW_AT_low_pc      : 0
    <69>   DW_AT_stmt_list   : 0x80
 <1><6d>: Abbrev Number: 91 (DW_TAG_base_type)
    <6e>   DW_AT_byte_size   : 8
    <6f>   DW_AT_encoding    : 7	(unsigned)
    <70>   DW_AT_name        : (indirect string, offset: 0xb1c7): long unsigned int
⬢[acme@toolbox repro_die__process]$

⬢[acme@toolbox repro_die__process]$ time pahole -F dwarf vmlinux | wc -l
84377

real	0m10.992s
user	0m9.526s
sys	0m1.445s
⬢[acme@toolbox repro_die__process]$

No warnings, 84377 lines of types printed from DWARF that file, which is
comparable to the BTF in it:

⬢[acme@toolbox repro_die__process]$ time pahole -F btf vmlinux | wc -l
83994

real	0m0.121s
user	0m0.095s
sys	0m0.031s
⬢[acme@toolbox repro_die__process]$

There are a few diffs from types generated from DWARF to those generated
from BTF (unexpected ones, that is) as noticed by btfdiff (found in the
pahole git repo), that its just some print ordering issue:

⬢[acme@toolbox repro_die__process]$ btfdiff vmlinux 
--- /tmp/btfdiff.dwarf.oX71hp	2024-06-03 11:42:32.228618100 -0300
+++ /tmp/btfdiff.btf.u762WM	2024-06-03 11:42:18.042748246 -0300
@@ -10027,6 +10027,24 @@ struct bio_map_data {
 };
 struct bio_post_read_ctx {
 	struct bio *               bio;                                                  /*     0     8 */
+	struct work_struct {
+		/* typedef atomic_long_t -> atomic64_t */ struct {
+			/* typedef s64 -> __s64 */ long long int counter;                /*     8     8 */
+		} data; /*     8     8 */
+		struct list_head {
+			struct list_head * next;                                         /*    16     8 */
+			struct list_head * prev;                                         /*    24     8 */
+		} entry; /*    16    16 */
+		/* typedef work_func_t */ void               (*func)(struct work_struct *); /*    32     8 */
+	} work; /*     8    32 */
+	unsigned int               cur_step;                                             /*    40     4 */
+	unsigned int               enabled_steps;                                        /*    44     4 */
+
+	/* size: 48, cachelines: 1, members: 4 */
+	/* last cacheline: 48 bytes */
+};
+struct bio_post_read_ctx {
+	struct bio *               bio;                                                  /*     0     8 */
 	struct f2fs_sb_info *      sbi;                                                  /*     8     8 */
 	struct work_struct {
 		/* typedef atomic_long_t -> atomic64_t */ struct {
@@ -10049,24 +10067,6 @@ struct bio_post_read_ctx {
 	/* sum members: 57, holes: 1, sum holes: 3 */
 	/* padding: 4 */
 };
-struct bio_post_read_ctx {
-	struct bio *               bio;                                                  /*     0     8 */
-	struct work_struct {
-		/* typedef atomic_long_t -> atomic64_t */ struct {
-			/* typedef s64 -> __s64 */ long long int counter;                /*     8     8 */
-		} data; /*     8     8 */
-		struct list_head {
-			struct list_head * next;                                         /*    16     8 */
-			struct list_head * prev;                                         /*    24     8 */
-		} entry; /*    16    16 */
-		/* typedef work_func_t */ void               (*func)(struct work_struct *); /*    32     8 */
-	} work; /*     8    32 */
-	unsigned int               cur_step;                                             /*    40     4 */
-	unsigned int               enabled_steps;                                        /*    44     4 */
-
-	/* size: 48, cachelines: 1, members: 4 */
-	/* last cacheline: 48 bytes */
-};
 struct bio_set {
 	struct kmem_cache *        bio_slab;                                             /*     0     8 */
 	unsigned int               front_pad;                                            /*     8     4 */
@@ -43044,6 +43044,17 @@ struct dma_block {
 	/* last cacheline: 16 bytes */
 };
 struct dma_chan {
+	int                        lock;                                                 /*     0     4 */
+
+	/* XXX 4 bytes hole, try to pack */
+
+	const char  *              device_id;                                            /*     8     8 */
+
+	/* size: 16, cachelines: 1, members: 2 */
+	/* sum members: 12, holes: 1, sum holes: 4 */
+	/* last cacheline: 16 bytes */
+};
+struct dma_chan {
 	struct dma_device *        device;                                               /*     0     8 */
 	struct device *            slave;                                                /*     8     8 */
 	/* typedef dma_cookie_t -> s32 -> __s32 */ int                        cookie;    /*    16     4 */
@@ -43071,17 +43082,6 @@ struct dma_chan {
 	/* sum members: 108, holes: 1, sum holes: 4 */
 	/* last cacheline: 48 bytes */
 };
-struct dma_chan {
-	int                        lock;                                                 /*     0     4 */
-
-	/* XXX 4 bytes hole, try to pack */
-
-	const char  *              device_id;                                            /*     8     8 */
-
-	/* size: 16, cachelines: 1, members: 2 */
-	/* sum members: 12, holes: 1, sum holes: 4 */
-	/* last cacheline: 16 bytes */
-};
 struct dma_chan_dev {
 	struct dma_chan *          chan;                                                 /*     0     8 */
 	struct device {
@@ -125786,12 +125786,11 @@ struct perf_aux_event {
 		/* typedef __u16 */ short unsigned int misc;                             /*     4     2 */
 		/* typedef __u16 */ short unsigned int size;                             /*     6     2 */
 	} header; /*     0     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     offset;                    /*     8     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     size;                      /*    16     8 */
-	/* typedef u64 -> __u64 */ long long unsigned int     flags;                     /*    24     8 */
+	/* typedef u32 -> __u32 */ unsigned int               pid;                       /*     8     4 */
+	/* typedef u32 -> __u32 */ unsigned int               tid;                       /*    12     4 */
 
-	/* size: 32, cachelines: 1, members: 4 */
-	/* last cacheline: 32 bytes */
+	/* size: 16, cachelines: 1, members: 3 */
+	/* last cacheline: 16 bytes */
 };
 struct perf_aux_event {
 	struct perf_event_header {
@@ -125799,11 +125798,12 @@ struct perf_aux_event {
 		/* typedef __u16 */ short unsigned int misc;                             /*     4     2 */
 		/* typedef __u16 */ short unsigned int size;                             /*     6     2 */
 	} header; /*     0     8 */
-	/* typedef u32 -> __u32 */ unsigned int               pid;                       /*     8     4 */
-	/* typedef u32 -> __u32 */ unsigned int               tid;                       /*    12     4 */
+	/* typedef u64 -> __u64 */ long long unsigned int     offset;                    /*     8     8 */
+	/* typedef u64 -> __u64 */ long long unsigned int     size;                      /*    16     8 */
+	/* typedef u64 -> __u64 */ long long unsigned int     flags;                     /*    24     8 */
 
-	/* size: 16, cachelines: 1, members: 3 */
-	/* last cacheline: 16 bytes */
+	/* size: 32, cachelines: 1, members: 4 */
+	/* last cacheline: 32 bytes */
 };
 struct perf_bpf_event {
 	struct bpf_prog *          prog;                                                 /*     0     8 */
@@ -163037,10 +163037,11 @@ struct syscall_tp_t {
 
 	/* XXX 4 bytes hole, try to pack */
 
-	long unsigned int          args[6];                                              /*    16    48 */
+	long unsigned int          ret;                                                  /*    16     8 */
 
-	/* size: 64, cachelines: 1, members: 3 */
-	/* sum members: 60, holes: 1, sum holes: 4 */
+	/* size: 24, cachelines: 1, members: 3 */
+	/* sum members: 20, holes: 1, sum holes: 4 */
+	/* last cacheline: 24 bytes */
 };
 struct syscall_tp_t {
 	struct trace_entry {
@@ -163053,11 +163054,10 @@ struct syscall_tp_t {
 
 	/* XXX 4 bytes hole, try to pack */
 
-	long unsigned int          ret;                                                  /*    16     8 */
+	long unsigned int          args[6];                                              /*    16    48 */
 
-	/* size: 24, cachelines: 1, members: 3 */
-	/* sum members: 20, holes: 1, sum holes: 4 */
-	/* last cacheline: 24 bytes */
+	/* size: 64, cachelines: 1, members: 3 */
+	/* sum members: 60, holes: 1, sum holes: 4 */
 };
 struct syscall_trace_enter {
 	struct trace_entry {
⬢[acme@toolbox repro_die__process]$ 

But if we do it manually and without expanding types as btfdiff does:

⬢[acme@toolbox repro_die__process]$ pahole -F btf -C dma_chan vmlinux 
struct dma_chan {
	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */

	/* XXX 4 bytes hole, try to pack */

	struct dma_chan_dev *      dev;                  /*    32     8 */
	const char  *              name;                 /*    40     8 */
	char *                     dbg_client_name;      /*    48     8 */
	struct list_head           device_node;          /*    56    16 */
	/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
	struct dma_chan_percpu *   local;                /*    72     8 */
	int                        client_count;         /*    80     4 */
	int                        table_count;          /*    84     4 */
	struct dma_router *        router;               /*    88     8 */
	void *                     route_data;           /*    96     8 */
	void *                     private;              /*   104     8 */

	/* size: 112, cachelines: 2, members: 15 */
	/* sum members: 108, holes: 1, sum holes: 4 */
	/* last cacheline: 48 bytes */
};

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf -C dma_chan vmlinux 
struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */

	/* size: 16, cachelines: 1, members: 2 */
	/* sum members: 12, holes: 1, sum holes: 4 */
	/* last cacheline: 16 bytes */
};

⬢[acme@toolbox repro_die__process]$

⬢[acme@toolbox perf-tools]$ git grep 'struct dma_chan {'
arch/mips/include/asm/mach-au1x00/au1000_dma.h:struct dma_chan {
include/linux/dmaengine.h:struct dma_chan {
kernel/dma.c:struct dma_chan {
⬢[acme@toolbox perf-tools]$

So we have this 'struct dma_chan' in the common kernel in include/linux/dmaengine.h

struct dma_chan {
        struct dma_device *device;
        struct device *slave;
        dma_cookie_t cookie;
        dma_cookie_t completed_cookie;

        /* sysfs */
        int chan_id;
        struct dma_chan_dev *dev;
        const char *name;       
#ifdef CONFIG_DEBUG_FS
        char *dbg_client_name;
#endif

        struct list_head device_node;
        struct dma_chan_percpu __percpu *local;
        int client_count;       
        int table_count;

        /* DMA router */
        struct dma_router *router;
        void *route_data;

        void *private;
};


And also this one in kernel/dma.c

struct dma_chan {
        int  lock;
        const char *device_id;
};

If we don't specify the name but grep for it:

⬢[acme@toolbox repro_die__process]$ pahole -F btf vmlinux | grep 'struct dma_chan {' -A5
struct dma_chan {
	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */
--
struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */
⬢[acme@toolbox repro_die__process]$ pahole -F dwarf vmlinux | grep 'struct dma_chan {' -A5
struct dma_chan {
	int                        lock;                 /*     0     4 */

	/* XXX 4 bytes hole, try to pack */

	const char  *              device_id;            /*     8     8 */

--
struct dma_chan {
	struct dma_device *        device;               /*     0     8 */
	struct device *            slave;                /*     8     8 */
	dma_cookie_t               cookie;               /*    16     4 */
	dma_cookie_t               completed_cookie;     /*    20     4 */
	int                        chan_id;              /*    24     4 */
⬢[acme@toolbox repro_die__process]$

So the vmlinux seems to have its BTF section successfully generated,
lemme check the .ko files.

- Arnaldo

elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Arnaldo Carvalho de Melo]

On Mon, Jun 03, 2024 at 11:56:43AM -0300, Arnaldo Carvalho de Melo wrote:
> So the vmlinux seems to have its BTF section successfully generated,
> lemme check the .ko files.

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf netdevsim.ko  | wc -l
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd)!
11448
⬢[acme@toolbox repro_die__process]$ pahole -F btf netdevsim.ko  | wc -l
libbpf: Invalid BTF string section
pahole: file 'netdevsim.ko' has no btf type information.
0
⬢[acme@toolbox repro_die__process]$

We manage to load a lot of DWARF types but then we hit that warning, no
BTF encoded.

Humm,

⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo netdevsim.ko | wc -l
0
⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | wc -l
779837
⬢[acme@toolbox repro_die__process]$

It seems this is some problem with elfutils...

After I added:

⬢[acme@toolbox pahole]$ git diff
diff --git a/dwarf_loader.c b/dwarf_loader.c
index f59477b44dfea026..b832c93cc2194eaf 100644
--- a/dwarf_loader.c
+++ b/dwarf_loader.c
@@ -2847,8 +2847,8 @@ static int die__process(Dwarf_Die *die, struct cu *cu, struct conf_load *conf)
        }
 
        if (tag != DW_TAG_compile_unit && tag != DW_TAG_type_unit) {
-               fprintf(stderr, "%s: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got %s (0x%x)!\n",
-                       __FUNCTION__, dwarf_tag_name(tag), tag);
+               fprintf(stderr, "%s: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got %s (0x%x) @ %llx!\n",
+                       __FUNCTION__, dwarf_tag_name(tag), tag, (unsigned long long)dwarf_dieoffset(die));
                return -EINVAL;
        }
 
⬢[acme@toolbox pahole]$

We get:

⬢[acme@toolbox repro_die__process]$ pahole -F dwarf netdevsim.ko > /dev/null
die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit or DW_TAG_skeleton_unit expected got member (0xd) @ 529de!
⬢[acme@toolbox repro_die__process]$

And if we look at binutils' readeld DWARF dump we get:

⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep '<529de>' -B5 -A8
  Compilation Unit @ offset 0x529d3:
   Length:        0x2132e (32-bit)
   Version:       4
   Abbrev Offset: 0x1d0e
   Pointer Size:  8
 <0><529de>: Abbrev Number: 108 (DW_TAG_compile_unit)
    <529df>   DW_AT_producer    : (indirect string, offset: 0x41389): GNU C11 13.3.0 -G 0 -mel -mno-check-zero-division -mabi=64 -mno-abicalls -msoft-float -march=mips64r2 -msym32 -mlong-calls -mllsc -mips64r2 -mno-shared -g -gdwarf-4 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-pic -ffreestanding -fstack-check=no -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fstack-protector -fno-stack-clash-protection -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack
    <529e3>   DW_AT_language    : 12	(ANSI C99)
    <529e4>   DW_AT_name        : (indirect string, offset: 0x44ce0): drivers/net/netdevsim/ethtool.c
    <529e8>   DW_AT_comp_dir    : (indirect string, offset: 0x3f407): /home/kodidev/linux
    <529ec>   DW_AT_low_pc      : 0x5270
    <529f4>   DW_AT_high_pc     : 0x65c
    <529fc>   DW_AT_stmt_list   : 0x709d
 <1><52a00>: Abbrev Number: 57 (DW_TAG_base_type)
⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep '<529de>' -B5 -A8

And at that <529de> die it _is_ a DW_TAG_compile_unit, and we managed to
process two other DW_TAG_compile_units up to that point:

⬢[acme@toolbox repro_die__process]$ readelf -wi netdevsim.ko | grep -w DW_TAG_compile_unit
 <0><b>: Abbrev Number: 188 (DW_TAG_compile_unit)
 <0><26c16>: Abbrev Number: 188 (DW_TAG_compile_unit)
 <0><529de>: Abbrev Number: 108 (DW_TAG_compile_unit)
 <0><73d10>: Abbrev Number: 204 (DW_TAG_compile_unit)
 <0><a43a8>: Abbrev Number: 160 (DW_TAG_compile_unit)
 <0><c7e66>: Abbrev Number: 110 (DW_TAG_compile_unit)
 <0><e8f21>: Abbrev Number: 158 (DW_TAG_compile_unit)
 <0><10c525>: Abbrev Number: 115 (DW_TAG_compile_unit)
 <0><12da49>: Abbrev Number: 167 (DW_TAG_compile_unit)
 <0><1546b9>: Abbrev Number: 140 (DW_TAG_compile_unit)
 <0><17656a>: Abbrev Number: 1 (DW_TAG_compile_unit)
⬢[acme@toolbox repro_die__process]$


And eu-readelf also doesn't work for that mips vmlinux:

⬢[acme@toolbox repro_die__process]$ file vmlinux 
vmlinux: ELF 64-bit LSB executable, MIPS, MIPS64 rel2 version 1 (SYSV), statically linked, BuildID[sha1]=19c8dafc6aa33a2e63b87487dfb10bf215bdf3a3, with debug_info, not stripped
⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo vmlinux 
⬢[acme@toolbox repro_die__process]$ eu-readelf -winfo+ vmlinux 
⬢[acme@toolbox repro_die__process]$ readelf -wi vmlinux | head -30
Contents of the .debug_info section:

  Compilation Unit @ offset 0:
   Length:        0x3f (32-bit)
   Version:       4
   Abbrev Offset: 0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_stmt_list   : 0
    <10>   DW_AT_ranges      : 0
    <14>   DW_AT_name        : (indirect string, offset: 0): arch/mips/kernel/head.S
    <18>   DW_AT_comp_dir    : (indirect string, offset: 0x18): /home/kodidev/linux
    <1c>   DW_AT_producer    : (indirect string, offset: 0x2c): GNU AS 2.42
    <20>   DW_AT_language    : 32769	(MIPS assembler)
 <1><22>: Abbrev Number: 2 (DW_TAG_subprogram)
    <23>   DW_AT_name        : (indirect string, offset: 0x38): kernel_entry
    <27>   DW_AT_external    : 1
    <27>   DW_AT_type        : <0x41>
    <28>   DW_AT_low_pc      : 0xffffffff80b0ce68
    <30>   DW_AT_high_pc     : 172
 <1><32>: Abbrev Number: 2 (DW_TAG_subprogram)
    <33>   DW_AT_name        : (indirect string, offset: 0x45): smp_bootstrap
    <37>   DW_AT_external    : 1
    <37>   DW_AT_type        : <0x41>
    <38>   DW_AT_low_pc      : 0xffffffff80b0cf14
    <40>   DW_AT_high_pc     : 44
 <1><41>: Abbrev Number: 3 (DW_TAG_unspecified_type)
 <1><42>: Abbrev Number: 0
  Compilation Unit @ offset 0x43:
   Length:        0x22ed9 (32-bit)
⬢[acme@toolbox repro_die__process]$

Couldn't find a way to ask eu-readelf for more verbose output, where we
could perhaps get some clue as to why it produces nothing while binutils
readelf manages to grok it, Mark, do you know some other way to ask
eu-readelf to produce more debug output?

I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
that then made eu-readelf to not be able to process it while pahole,
that uses eltuils' libraries, was able to process the first two CUs for
a kernel module and all the CUs for the vmlinux file :-\

Mark, the whole thread is available at:

https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u

Now tryint to look at elfutils source to see if I can figure out
something there..

- Arnaldo

⬢[acme@toolbox repro_die__process]$ ltrace eu-readelf -winfo netdevsim.ko
<SNIP>
strlen(".gdb_index")                                                                 = 10
strncmp(".mdebug.abi64", ".gdb_index", 10)                                           = 6
elf_nextscn(0x55fddd480470, 0x55fddd4820e8, 10, 103)                                 = 0x55fddd4821b8
gelf_getshdr(0x55fddd4821b8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_strptr(0x55fddd480470, 54, 349, 0x55fddd4831f8)                                  = 0x7f0970925879
strlen(".note.GNU-stack")                                                            = 15
strncmp(".note.GNU-stack", ".debug_abbrev", 13)                                      = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_addr")                                                                = 11
strncmp(".note.GNU-stack", ".debug_addr", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_aranges")                                                             = 14
strncmp(".note.GNU-stack", ".debug_aranges", 14)                                     = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_frame")                                                               = 12
strncmp(".note.GNU-stack", ".debug_frame", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_info")                                                                = 11
strncmp(".note.GNU-stack", ".debug_info", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_types")                                                               = 12
strncmp(".note.GNU-stack", ".debug_types", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_line")                                                                = 11
strncmp(".note.GNU-stack", ".debug_line", 11)                                        = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_loc")                                                                 = 10
strncmp(".note.GNU-stack", ".debug_loc", 10)                                         = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_loclists")                                                            = 15
strncmp(".note.GNU-stack", ".debug_loclists", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_pubnames")                                                            = 15
strncmp(".note.GNU-stack", ".debug_pubnames", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_str")                                                                 = 10
strncmp(".note.GNU-stack", ".debug_str", 10)                                         = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_line_str")                                                            = 15
strncmp(".note.GNU-stack", ".debug_line_str", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_str_offsets")                                                         = 18
strncmp(".note.GNU-stack", ".debug_str_offsets", 18)                                 = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_macinfo")                                                             = 14
strncmp(".note.GNU-stack", ".debug_macinfo", 14)                                     = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_macro")                                                               = 12
strncmp(".note.GNU-stack", ".debug_macro", 12)                                       = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_ranges")                                                              = 13
strncmp(".note.GNU-stack", ".debug_ranges", 13)                                      = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".debug_rnglists")                                                            = 15
strncmp(".note.GNU-stack", ".debug_rnglists", 15)                                    = 10
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".eh_frame")                                                                  = 9
strncmp(".note.GNU-stack", ".eh_frame", 9)                                           = 9
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".eh_frame_hdr")                                                              = 13
strncmp(".note.GNU-stack", ".eh_frame_hdr", 13)                                      = 9
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".gcc_except_table")                                                          = 17
strncmp(".note.GNU-stack", ".gcc_except_table", 17)                                  = 7
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
strlen(".gdb_index")                                                                 = 10
strncmp(".note.GNU-stack", ".gdb_index", 10)                                         = 7
strncmp(".note.GNU-stack", ".gnu.debuglto_", 14)                                     = 7
elf_nextscn(0x55fddd480470, 0x55fddd4821b8, 14, 103)                                 = 0x55fddd482288
gelf_getshdr(0x55fddd482288, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482288, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482358
gelf_getshdr(0x55fddd482358, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482358, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482428
gelf_getshdr(0x55fddd482428, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482428, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4824f8
gelf_getshdr(0x55fddd4824f8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4824f8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4825c8
gelf_getshdr(0x55fddd4825c8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4825c8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482698
gelf_getshdr(0x55fddd482698, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482698, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482768
gelf_getshdr(0x55fddd482768, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482768, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482838
gelf_getshdr(0x55fddd482838, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482838, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482908
gelf_getshdr(0x55fddd482908, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482908, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd4829d8
gelf_getshdr(0x55fddd4829d8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd4829d8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482aa8
gelf_getshdr(0x55fddd482aa8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482aa8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482b78
gelf_getshdr(0x55fddd482b78, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482b78, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482c48
gelf_getshdr(0x55fddd482c48, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482c48, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482d18
gelf_getshdr(0x55fddd482d18, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482d18, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482de8
gelf_getshdr(0x55fddd482de8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482de8, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd482eb8
gelf_getshdr(0x55fddd482eb8, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_strptr(0x55fddd480470, 54, 513, 0x55fddd4831f8)                                  = 0x7f097092591d
strlen(".BTF")                                                                       = 4
strncmp(".BTF", ".debug_abbrev", 13)                                                 = -34
strlen(".debug_addr")                                                                = 11
strncmp(".BTF", ".debug_addr", 11)                                                   = -34
strlen(".debug_aranges")                                                             = 14
strncmp(".BTF", ".debug_aranges", 14)                                                = -34
strlen(".debug_frame")                                                               = 12
strncmp(".BTF", ".debug_frame", 12)                                                  = -34
strlen(".debug_info")                                                                = 11
strncmp(".BTF", ".debug_info", 11)                                                   = -34
strlen(".debug_types")                                                               = 12
strncmp(".BTF", ".debug_types", 12)                                                  = -34
strlen(".debug_line")                                                                = 11
strncmp(".BTF", ".debug_line", 11)                                                   = -34
strlen(".debug_loc")                                                                 = 10
strncmp(".BTF", ".debug_loc", 10)                                                    = -34
strlen(".debug_loclists")                                                            = 15
strncmp(".BTF", ".debug_loclists", 15)                                               = -34
strlen(".debug_pubnames")                                                            = 15
strncmp(".BTF", ".debug_pubnames", 15)                                               = -34
strlen(".debug_str")                                                                 = 10
strncmp(".BTF", ".debug_str", 10)                                                    = -34
strlen(".debug_line_str")                                                            = 15
strncmp(".BTF", ".debug_line_str", 15)                                               = -34
strlen(".debug_str_offsets")                                                         = 18
strncmp(".BTF", ".debug_str_offsets", 18)                                            = -34
strlen(".debug_macinfo")                                                             = 14
strncmp(".BTF", ".debug_macinfo", 14)                                                = -34
strlen(".debug_macro")                                                               = 12
strncmp(".BTF", ".debug_macro", 12)                                                  = -34
strlen(".debug_ranges")                                                              = 13
strncmp(".BTF", ".debug_ranges", 13)                                                 = -34
strlen(".debug_rnglists")                                                            = 15
strncmp(".BTF", ".debug_rnglists", 15)                                               = -34
strlen(".eh_frame")                                                                  = 9
strncmp(".BTF", ".eh_frame", 9)                                                      = -35
strlen(".eh_frame_hdr")                                                              = 13
strncmp(".BTF", ".eh_frame_hdr", 13)                                                 = -35
strlen(".gcc_except_table")                                                          = 17
strncmp(".BTF", ".gcc_except_table", 17)                                             = -37
strlen(".gdb_index")                                                                 = 10
strncmp(".BTF", ".gdb_index", 10)                                                    = -37
elf_nextscn(0x55fddd480470, 0x55fddd482eb8, 10, 103)                                 = 0x55fddd482f88
gelf_getshdr(0x55fddd482f88, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd482f88, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd483058
gelf_getshdr(0x55fddd483058, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd483058, 0x55fddd480470, 0x55fddd4831f8)          = 0x55fddd483128
gelf_getshdr(0x55fddd483128, 0x7fff00974910, 0x55fddd480538, 0x55fddd4831f8)         = 0x7fff00974910
elf_nextscn(0x55fddd480470, 0x55fddd483128, 0x55fddd480470, 0x55fddd4831f8)          = 0
dwfl_end(0, 165, 0x55fddd480538, 0x55fddd4831f8)                                     = 0
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(nil)                                                                            = <void>
free(0x55fddd47eef0)                                                                 = <void>
<... dwfl_getmodules resumed> )                                                      = 0
dwfl_end(0x55fddd47ea90, 0x55fddd47eee0, 0x55fddd47e, 1)                             = 6
close(3)                                                                             = 0
__cxa_finalize(0x55fddbe44780, 10, 0, 0x7fff00974ee0)                                = 1
+++ exited (status 0) +++
⬢[acme@toolbox repro_die__process]$ 

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Mark Wielaard]

Hi,

On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> Couldn't find a way to ask eu-readelf for more verbose output, where we
> could perhaps get some clue as to why it produces nothing while binutils
> readelf manages to grok it, Mark, do you know some other way to ask
> eu-readelf to produce more debug output?
> 
> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> that then made eu-readelf to not be able to process it while pahole,
> that uses eltuils' libraries, was able to process the first two CUs for
> a kernel module and all the CUs for the vmlinux file :-\

I haven't looked at the vmlinux file. But for the .ko file the issue
is that the elfutils MIPS backend isn't complete. Specifically MIPS
relocations aren't recognized (and so cannot be applied). There are
some pending patches which try to fix that:

https://patchwork.sourceware.org/project/elfutils/list/?series=31601

Cheers,

Mark

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

Hi Mark,

On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > could perhaps get some clue as to why it produces nothing while binutils
> > readelf manages to grok it, Mark, do you know some other way to ask
> > eu-readelf to produce more debug output?
> > 
> > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > that then made eu-readelf to not be able to process it while pahole,
> > that uses eltuils' libraries, was able to process the first two CUs for
> > a kernel module and all the CUs for the vmlinux file :-\
> > 
> > Mark, the whole thread is available at:
> > 
> > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> 
> I haven't looked at the vmlinux file. But for the .ko file the issue
> is that the elfutils MIPS backend isn't complete. Specifically MIPS
> relocations aren't recognized (and so cannot be applied). There are
> some pending patches which try to fix that:
> 
> https://patchwork.sourceware.org/project/elfutils/list/?series=31601

Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
work for MIPS, and I locally rebuilt elfutils and then pahole from their
respective next/main branches. For elfutils, main (935ee131cf7c) includes

  e259f126 Support Mips architecture
  f2acb069 stack: Fix stack unwind failure on mips
  db33cb0c backends: Add register_info, return_value_location, core_note mips

which partially applies the patchwork series but leaves out the support for
readelf, strip, and elflint.

I believe this means the vmlinux and .ko files I shared are OK, or is there
more backend work needed for MIPS?

The bits missing in eu-readelf would explain the blank output both Arnaldo
and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
patchwork readelf patch locally but ran into merge conflicts.

CCing Ying Huang for any more insight.

Thanks,
Tony

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Ying Huang]

Hi Tony,

Yes, I also found that merge will have some conflicts now.

And you can modify the key code as follows to fix the issue about "eu-readelf -w" could not show info.

But this can not fix other display error in the result of the "eu-readelf -w" which need modify the way to get symbol index and type.

> diff --git a/libelf/libelfP.h b/libelf/libelfP.h
> index bdd2cc6a..6565ee02 100644
> --- a/libelf/libelfP.h
> +++ b/libelf/libelfP.h
> @@ -620,4 +620,5 @@ extern void __libelf_reset_rawdata (Elf_Scn *scn, void *buf, size_t size,
>  #define ELF64_MIPS_R_TYPE1(i)          ((i) & 0xff)
>  #define ELF64_MIPS_R_TYPE2(i)           (((i) >> 8) & 0xff)
>  #define ELF64_MIPS_R_TYPE3(i)           (((i) >> 16) & 0xff)
> +#define is_debug_section_type(type) (type == SHT_PROGBITS || type == SHT_MIPS_DWARF)
>  #endif  /* libelfP.h */ > diff --git a/src/readelf.c b/src/readelf.c
> index 0e931184..e88cf67c 100644
> --- a/src/readelf.c
> +++ b/src/readelf.c > @@ -12043,7 +12139,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr)
>  	  GElf_Shdr shdr_mem;
>  	  GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
>  
> -	  if (shdr != NULL && shdr->sh_type == SHT_PROGBITS)
> +	  if (shdr != NULL && is_debug_section_type(shdr->sh_type))
>  	    {
>  	      const char *name = elf_strptr (ebl->elf, shstrndx,
>  					     shdr->sh_name);
> @@ -12073,7 +12169,7 @@ print_debug (Dwfl_Module *dwflmod, Ebl *ebl, GElf_Ehdr *ehdr)
>        GElf_Shdr shdr_mem;
>        GElf_Shdr *shdr = gelf_getshdr (scn, &shdr_mem);
>  
> -      if (shdr != NULL && shdr->sh_type == SHT_PROGBITS)
> +      if (shdr != NULL && is_debug_section_type(shdr->sh_type))
>  	{
>  	  static const struct
>  	  {

Thanks,

Ying


在 2024/6/4 11:47, Tony Ambardar 写道:
> Hi Mark,
>
> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
>> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
>>> Couldn't find a way to ask eu-readelf for more verbose output, where we
>>> could perhaps get some clue as to why it produces nothing while binutils
>>> readelf manages to grok it, Mark, do you know some other way to ask
>>> eu-readelf to produce more debug output?
>>>
>>> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
>>> that then made eu-readelf to not be able to process it while pahole,
>>> that uses eltuils' libraries, was able to process the first two CUs for
>>> a kernel module and all the CUs for the vmlinux file :-\
>>>
>>> Mark, the whole thread is available at:
>>>
>>> https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
>> I haven't looked at the vmlinux file. But for the .ko file the issue
>> is that the elfutils MIPS backend isn't complete. Specifically MIPS
>> relocations aren't recognized (and so cannot be applied). There are
>> some pending patches which try to fix that:
>>
>> https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> work for MIPS, and I locally rebuilt elfutils and then pahole from their
> respective next/main branches. For elfutils, main (935ee131cf7c) includes
>
>   e259f126 Support Mips architecture
>   f2acb069 stack: Fix stack unwind failure on mips
>   db33cb0c backends: Add register_info, return_value_location, core_note mips
>
> which partially applies the patchwork series but leaves out the support for
> readelf, strip, and elflint.
>
> I believe this means the vmlinux and .ko files I shared are OK, or is there
> more backend work needed for MIPS?
>
> The bits missing in eu-readelf would explain the blank output both Arnaldo
> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> patchwork readelf patch locally but ran into merge conflicts.
>
> CCing Ying Huang for any more insight.
>
> Thanks,
> Tony

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> Hi Mark,
> 
> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > could perhaps get some clue as to why it produces nothing while binutils
> > > readelf manages to grok it, Mark, do you know some other way to ask
> > > eu-readelf to produce more debug output?
> > > 
> > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > that then made eu-readelf to not be able to process it while pahole,
> > > that uses eltuils' libraries, was able to process the first two CUs for
> > > a kernel module and all the CUs for the vmlinux file :-\
> > > 
> > > Mark, the whole thread is available at:
> > > 
> > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > 
> > I haven't looked at the vmlinux file. But for the .ko file the issue
> > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > relocations aren't recognized (and so cannot be applied). There are
> > some pending patches which try to fix that:
> > 
> > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> 
> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> work for MIPS, and I locally rebuilt elfutils and then pahole from their
> respective next/main branches. For elfutils, main (935ee131cf7c) includes
> 
>   e259f126 Support Mips architecture
>   f2acb069 stack: Fix stack unwind failure on mips
>   db33cb0c backends: Add register_info, return_value_location, core_note mips
> 
> which partially applies the patchwork series but leaves out the support for
> readelf, strip, and elflint.
> 
> I believe this means the vmlinux and .ko files I shared are OK, or is there
> more backend work needed for MIPS?
> 
> The bits missing in eu-readelf would explain the blank output both Arnaldo
> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> patchwork readelf patch locally but ran into merge conflicts.
> 
> CCing Ying Huang for any more insight.
> 
> Thanks,
> Tony

Hello all,

A short update, starting with answering my own question.

No, apparently the above commits *do not* complete the backend work. Ying
Huang submitted additional related patches since March 5: [1][2]

    strip: Adapt src/strip -o -f on mips
    readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
    elflint: adapt src/elflint --gnu src/nm on mips
    test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh

Despite the titles, these patches do include core backend changes for MIPS.
I resolved the various merge conflicts [3], rebuilt elfutils, and retested
kernel builds to now find:

  - pahole is able to read DWARF[45] info and create .BTF for modules
  - resolve_btfids can successfully patch .BTF_ids in modules
  - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)

Huzzah!


Ying:

Thank you for developing these MIPS patches. In your view, are the MIPS
changes now complete, or do you plan further updates that might improve or
impact parsing DWARF debug/reloc info in apps like pahole?


Mark:

Given that BTF usage on Linux/MIPS is basically broken without these
patches, could I request some of your review time for them to be merged? If
it's helpful, my branch [3] includes all patches with conflicts fixed, and
I also successfully ran the elfutils self-tests (including MIPS from Ying).
Please feel free to add for these patches:

    Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>


Arnaldo:

Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
I reran your tests with the updated elfutils and latest pahole (pre-1.27),
and then found:

  - everything that worked before, still works
  - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
  - we now see expected output from "eu-readelf -winfo netdevsim.ko"

Regarding pahole, DWARF parsing and BTF generation now works:
(with no more die__process: error messages seen)

    kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
    14504

but strangely pahole still doesn't read its own generated BTF:

    kodidev:~/linux$ pahole -F btf netdevsim.ko
    libbpf: Invalid BTF string section
    pahole: file 'netdevsim.ko' has no btf type information.

Poking inside a little further:
    
    kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
    [...]
    argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
    SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
    SYS_brk(0)                                       = 0x563d47e37000
    SYS_brk(0x563d47e58000)                          = 0x563d47e58000
    <... argp_parse resumed> )                       = 0
    dwarves__init(0x20000, 0, -4096, 213)            = 0
    dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
    cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
    memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
    strlen("/sys/kernel/btf/")                       = 16
    strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
    cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
    0x563d47da40c0 <unfinished ...>
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
    SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
    SYS_close(3)                                     = 0
    SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
    SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
    SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
    SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
    SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
    SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
    SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
    SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
    string section
    ) = 35
    SYS_close(3)                                     = 0
    <... cus__load_files resumed> )                  = 0xffffffff
    access("netdevsim.ko", 4 <unfinished ...>
    SYS_access("netdevsim.ko", 04)                   = 0
    <... access resumed> )                           = 0
    fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
    "netdevsim.ko", "btf" <unfinished ...>
    SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
    'netdevsim.ko' has no btf type information.
    ) = 57
    <... fprintf resumed> )                          = 57
    SYS_exit_group(1 <no return ...>
    +++ exited (status 1) +++

Could you help investigate this further? Maybe a libbpf issue? For the
record, I also tried building pahole with embedded libbpf 1.4.3 without
any change. (side note: please make pahole --version also cover libbpf)


Many thanks everyone for your help,
Tony

[1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
[2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
[3]:
https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Mon, Jun 10, 2024 at 11:36:54PM -0700, Tony Ambardar wrote:
> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> 
> [snip]
> 
> Arnaldo:
> 
> Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
> I reran your tests with the updated elfutils and latest pahole (pre-1.27),
> and then found:
> 
>   - everything that worked before, still works
>   - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
>   - we now see expected output from "eu-readelf -winfo netdevsim.ko"
> 
> Regarding pahole, DWARF parsing and BTF generation now works:
> (with no more die__process: error messages seen)
> 
>     kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
>     14504
> 
> but strangely pahole still doesn't read its own generated BTF:
> 
>     kodidev:~/linux$ pahole -F btf netdevsim.ko
>     libbpf: Invalid BTF string section
>     pahole: file 'netdevsim.ko' has no btf type information.

After staring at this I realized that this is split BTF and we needed to
issue "pahole -F btf --btf_base vmlinux netdevsim.ko", which does work.
Unfortunately, the libbpf error message is a bit cryptic; perhaps a future
update could mention "--btf_base"?

>     
> [snip]
> 
> Many thanks everyone for your help,
> Tony
> 
> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> [3]:
> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Mark Wielaard]

Hi,

Adding elfutils-devel to CC to keep everyone up to date on the state of
the patches.

On Mon, 2024-06-10 at 23:36 -0700, Tony Ambardar wrote:
> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> > On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > > could perhaps get some clue as to why it produces nothing while binutils
> > > > readelf manages to grok it, Mark, do you know some other way to ask
> > > > eu-readelf to produce more debug output?
> > > > 
> > > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > > that then made eu-readelf to not be able to process it while pahole,
> > > > that uses eltuils' libraries, was able to process the first two CUs for
> > > > a kernel module and all the CUs for the vmlinux file :-\
> > > > 
> > > > Mark, the whole thread is available at:
> > > > 
> > > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > > 
> > > I haven't looked at the vmlinux file. But for the .ko file the issue
> > > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > > relocations aren't recognized (and so cannot be applied). There are
> > > some pending patches which try to fix that:
> > > 
> > > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> > 
> > Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> > work for MIPS, and I locally rebuilt elfutils and then pahole from their
> > respective next/main branches. For elfutils, main (935ee131cf7c) includes
> > 
> >   e259f126 Support Mips architecture
> >   f2acb069 stack: Fix stack unwind failure on mips
> >   db33cb0c backends: Add register_info, return_value_location, core_note mips
> > 
> > which partially applies the patchwork series but leaves out the support for
> > readelf, strip, and elflint.
> > 
> > I believe this means the vmlinux and .ko files I shared are OK, or is there
> > more backend work needed for MIPS?
> > 
> > The bits missing in eu-readelf would explain the blank output both Arnaldo
> > and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> > patchwork readelf patch locally but ran into merge conflicts.
> 
> A short update, starting with answering my own question.
> 
> No, apparently the above commits *do not* complete the backend work. Ying
> Huang submitted additional related patches since March 5: [1][2]
> 
>     strip: Adapt src/strip -o -f on mips
>     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
>     elflint: adapt src/elflint --gnu src/nm on mips
>     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
> 
> Despite the titles, these patches do include core backend changes for MIPS.
> I resolved the various merge conflicts [3], rebuilt elfutils, and retested
> kernel builds to now find:
> 
>   - pahole is able to read DWARF[45] info and create .BTF for modules
>   - resolve_btfids can successfully patch .BTF_ids in modules
>   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
> 
> Huzzah!
> 
> 
> Ying:
> 
> Thank you for developing these MIPS patches. In your view, are the MIPS
> changes now complete, or do you plan further updates that might improve or
> impact parsing DWARF debug/reloc info in apps like pahole?
> 
> 
> Mark:
> 
> Given that BTF usage on Linux/MIPS is basically broken without these
> patches, could I request some of your review time for them to be merged? If
> it's helpful, my branch [3] includes all patches with conflicts fixed, and
> I also successfully ran the elfutils self-tests (including MIPS from Ying).
> Please feel free to add for these patches:
> 
>     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>

Yes, I would very much like to integrate the rest of these patches. But
I keep running out of time. The main issues were that, as you noticed,
the patches mix backend and frontend tool changes a bit. I don't have
access to a MIPS system to test them on. There are a couple of
different MIPS abis (I believe all combinations of 32/64 bit and
big/little endianness), but people have only tested on mips64le (maybe
that is the only relevant one these days?) And finally the way MIPS
represents relocations is slightly different than any other ELF
architecture does. So we have to translate that somewhere to make the
standards functions work. I have to convince myself that doing that in
elf_getdata as the patches do is the right place.

> Many thanks everyone for your help,
> Tony
> 
> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> [3]:
> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Tony Ambardar]

On Tue, Jun 11, 2024 at 03:07:29PM +0200, Mark Wielaard wrote:
> Hi,
> 
> Adding elfutils-devel to CC to keep everyone up to date on the state of
> the patches.
> 
> On Mon, 2024-06-10 at 23:36 -0700, Tony Ambardar wrote:
> > On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
> > > On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
> > > > On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > Couldn't find a way to ask eu-readelf for more verbose output, where we
> > > > > could perhaps get some clue as to why it produces nothing while binutils
> > > > > readelf manages to grok it, Mark, do you know some other way to ask
> > > > > eu-readelf to produce more debug output?
> > > > > 
> > > > > I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
> > > > > that then made eu-readelf to not be able to process it while pahole,
> > > > > that uses eltuils' libraries, was able to process the first two CUs for
> > > > > a kernel module and all the CUs for the vmlinux file :-\
> > > > > 
> > > > > Mark, the whole thread is available at:
> > > > > 
> > > > > https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
> > > > 
> > > > I haven't looked at the vmlinux file. But for the .ko file the issue
> > > > is that the elfutils MIPS backend isn't complete. Specifically MIPS
> > > > relocations aren't recognized (and so cannot be applied). There are
> > > > some pending patches which try to fix that:
> > > > 
> > > > https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> > > 
> > > Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
> > > work for MIPS, and I locally rebuilt elfutils and then pahole from their
> > > respective next/main branches. For elfutils, main (935ee131cf7c) includes
> > > 
> > >   e259f126 Support Mips architecture
> > >   f2acb069 stack: Fix stack unwind failure on mips
> > >   db33cb0c backends: Add register_info, return_value_location, core_note mips
> > > 
> > > which partially applies the patchwork series but leaves out the support for
> > > readelf, strip, and elflint.
> > > 
> > > I believe this means the vmlinux and .ko files I shared are OK, or is there
> > > more backend work needed for MIPS?
> > > 
> > > The bits missing in eu-readelf would explain the blank output both Arnaldo
> > > and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
> > > patchwork readelf patch locally but ran into merge conflicts.
> > 
> > A short update, starting with answering my own question.
> > 
> > No, apparently the above commits *do not* complete the backend work. Ying
> > Huang submitted additional related patches since March 5: [1][2]
> > 
> >     strip: Adapt src/strip -o -f on mips
> >     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
> >     elflint: adapt src/elflint --gnu src/nm on mips
> >     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
> > 
> > Despite the titles, these patches do include core backend changes for MIPS.
> > I resolved the various merge conflicts [3], rebuilt elfutils, and retested
> > kernel builds to now find:
> > 
> >   - pahole is able to read DWARF[45] info and create .BTF for modules
> >   - resolve_btfids can successfully patch .BTF_ids in modules
> >   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
> > 
> > Huzzah!
> > 
> > 
> > Ying:
> > 
> > Thank you for developing these MIPS patches. In your view, are the MIPS
> > changes now complete, or do you plan further updates that might improve or
> > impact parsing DWARF debug/reloc info in apps like pahole?
> > 
> > 
> > Mark:
> > 
> > Given that BTF usage on Linux/MIPS is basically broken without these
> > patches, could I request some of your review time for them to be merged? If
> > it's helpful, my branch [3] includes all patches with conflicts fixed, and
> > I also successfully ran the elfutils self-tests (including MIPS from Ying).
> > Please feel free to add for these patches:
> > 
> >     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> 
> Yes, I would very much like to integrate the rest of these patches. But
> I keep running out of time. The main issues were that, as you noticed,
> the patches mix backend and frontend tool changes a bit. I don't have
> access to a MIPS system to test them on. There are a couple of
> different MIPS abis (I believe all combinations of 32/64 bit and
> big/little endianness), but people have only tested on mips64le (maybe
> that is the only relevant one these days?) And finally the way MIPS
> represents relocations is slightly different than any other ELF
> architecture does. So we have to translate that somewhere to make the
> standards functions work. I have to convince myself that doing that in
> elf_getdata as the patches do is the right place.
> 

Glad to hear there's strong interest. Not much I can add to the code
structure discussion but I *can* confirm my testing included both
mips64el and mips32be, specifically to improve word-size/endianness
coverage. The former is supported by Debian, while the latter can still
be found on many embedded devices like consumer routers. If you need
additional testing for review/merge I can help with that (using
OpenWrt/QEMU primarily).

> > Many thanks everyone for your help,
> > Tony
> > 
> > [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> > [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> > [3]:
> > https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/
> 

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Ying Huang]

Hi Tony,

Thanks for your fix about the conflict and test about the whole patch.

Now, there is currently one test case that has not been added, which is

tests/run-backtrace-core-mips.sh.
Thanks,
Ying
在 2024/6/11 14:36, Tony Ambardar 写道:
> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
>> Hi Mark,
>>
>> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
>>> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
>>>> Couldn't find a way to ask eu-readelf for more verbose output, where we
>>>> could perhaps get some clue as to why it produces nothing while binutils
>>>> readelf manages to grok it, Mark, do you know some other way to ask
>>>> eu-readelf to produce more debug output?
>>>>
>>>> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
>>>> that then made eu-readelf to not be able to process it while pahole,
>>>> that uses eltuils' libraries, was able to process the first two CUs for
>>>> a kernel module and all the CUs for the vmlinux file :-\
>>>>
>>>> Mark, the whole thread is available at:
>>>>
>>>> https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
>>> I haven't looked at the vmlinux file. But for the .ko file the issue
>>> is that the elfutils MIPS backend isn't complete. Specifically MIPS
>>> relocations aren't recognized (and so cannot be applied). There are
>>> some pending patches which try to fix that:
>>>
>>> https://patchwork.sourceware.org/project/elfutils/list/?series=31601
>> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
>> work for MIPS, and I locally rebuilt elfutils and then pahole from their
>> respective next/main branches. For elfutils, main (935ee131cf7c) includes
>>
>>   e259f126 Support Mips architecture
>>   f2acb069 stack: Fix stack unwind failure on mips
>>   db33cb0c backends: Add register_info, return_value_location, core_note mips
>>
>> which partially applies the patchwork series but leaves out the support for
>> readelf, strip, and elflint.
>>
>> I believe this means the vmlinux and .ko files I shared are OK, or is there
>> more backend work needed for MIPS?
>>
>> The bits missing in eu-readelf would explain the blank output both Arnaldo
>> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
>> patchwork readelf patch locally but ran into merge conflicts.
>>
>> CCing Ying Huang for any more insight.
>>
>> Thanks,
>> Tony
> Hello all,
>
> A short update, starting with answering my own question.
>
> No, apparently the above commits *do not* complete the backend work. Ying
> Huang submitted additional related patches since March 5: [1][2]
>
>     strip: Adapt src/strip -o -f on mips
>     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
>     elflint: adapt src/elflint --gnu src/nm on mips
>     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
>
> Despite the titles, these patches do include core backend changes for MIPS.
> I resolved the various merge conflicts [3], rebuilt elfutils, and retested
> kernel builds to now find:
>
>   - pahole is able to read DWARF[45] info and create .BTF for modules
>   - resolve_btfids can successfully patch .BTF_ids in modules
>   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
>
> Huzzah!
>
>
> Ying:
>
> Thank you for developing these MIPS patches. In your view, are the MIPS
> changes now complete, or do you plan further updates that might improve or
> impact parsing DWARF debug/reloc info in apps like pahole?
>
>
> Mark:
>
> Given that BTF usage on Linux/MIPS is basically broken without these
> patches, could I request some of your review time for them to be merged? If
> it's helpful, my branch [3] includes all patches with conflicts fixed, and
> I also successfully ran the elfutils self-tests (including MIPS from Ying).
> Please feel free to add for these patches:
>
>     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>
>
>
> Arnaldo:
>
> Your stepping through DWARF/reloc diagnostics earlier was helpful. Thanks!
> I reran your tests with the updated elfutils and latest pahole (pre-1.27),
> and then found:
>
>   - everything that worked before, still works
>   - your observations from "btfdiff vmlinux" and 'struct dma_chan' persist
>   - we now see expected output from "eu-readelf -winfo netdevsim.ko"
>
> Regarding pahole, DWARF parsing and BTF generation now works:
> (with no more die__process: error messages seen)
>
>     kodidev:~/linux$ pahole -F dwarf netdevsim.ko |wc -l
>     14504
>
> but strangely pahole still doesn't read its own generated BTF:
>
>     kodidev:~/linux$ pahole -F btf netdevsim.ko
>     libbpf: Invalid BTF string section
>     pahole: file 'netdevsim.ko' has no btf type information.
>
> Poking inside a little further:
>     
>     kodidev:~/linux$ ltrace -S pahole -F btf netdevsim.ko
>     [...]
>     argp_parse(0x563d47da42a0, 4, 0x7ffd5e552698, 0 <unfinished ...>
>     SYS_318(0x7fc385bf84d8, 8, 1, 0x7fc385ce9908)    = 8
>     SYS_brk(0)                                       = 0x563d47e37000
>     SYS_brk(0x563d47e58000)                          = 0x563d47e58000
>     <... argp_parse resumed> )                       = 0
>     dwarves__init(0x20000, 0, -4096, 213)            = 0
>     dwarves__resolve_cacheline_size(0x563d47da40c0, 0, 24, 213) = 64
>     cus__new(5, 0, 64, 213)                          = 0x563d47e372a0
>     memset(0x563d47da44c0, ' ', 127)                 = 0x563d47da44c0
>     strlen("/sys/kernel/btf/")                       = 16
>     strncmp("netdevsim.ko", "/sys/kernel/btf/", 16)  = 63
>     cus__load_files(0x563d47e372a0, 0x563d47da40c0, 0x7ffd5e5526b0,
>     0x563d47da40c0 <unfinished ...>
>     SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
>     SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e552210, 4096) = 0
>     SYS_read(3, "\177ELF\002\001\001", 4096)         = 4096
>     SYS_close(3)                                     = 0
>     SYS_openat(0xffffff9c, 0x7ffd5e554603, 0x80000, 0) = 3
>     SYS_fcntl(3, 1, 0, 0x7fc3859de2f0)               = 1
>     SYS_newfstatat(3, 0x7fc385baf44f, 0x7ffd5e5521f0, 4096) = 0
>     SYS_pread(3, 0x7ffd5e5521f0, 64, 0)              = 64
>     SYS_pread(3, 0x563d47e3e470, 3520, 0x4f60c8)     = 3520
>     SYS_pread(3, 0x563d47e3f240, 525, 0x4f5eb8)      = 525
>     SYS_pread(3, 0x563d47e3f460, 0x45dd, 0x2b1fb6)   = 0x45dd
>     SYS_write(2, "libbpf: Invalid BTF string secti"..., 35libbpf: Invalid BTF
>     string section
>     ) = 35
>     SYS_close(3)                                     = 0
>     <... cus__load_files resumed> )                  = 0xffffffff
>     access("netdevsim.ko", 4 <unfinished ...>
>     SYS_access("netdevsim.ko", 04)                   = 0
>     <... access resumed> )                           = 0
>     fprintf(0x7fc385bf26a0, "pahole: file '%s' has no %s type"...,
>     "netdevsim.ko", "btf" <unfinished ...>
>     SYS_write(2, "pahole: file 'netdevsim.ko' has "..., 57pahole: file
>     'netdevsim.ko' has no btf type information.
>     ) = 57
>     <... fprintf resumed> )                          = 57
>     SYS_exit_group(1 <no return ...>
>     +++ exited (status 1) +++
>
> Could you help investigate this further? Maybe a libbpf issue? For the
> record, I also tried building pahole with embedded libbpf 1.4.3 without
> any change. (side note: please make pahole --version also cover libbpf)
>
>
> Many thanks everyone for your help,
> Tony
>
> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
> [3]:
> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/

Re: elfutils DWARF problem was: Re: Problem with BTF generation on mips64el

[Ying Huang]

Hi Mark,

Regarding the current questions, I have a few points that needed to be explained.


在 2024/6/11 21:07, Mark Wielaard 写道:
> Hi,
>
> Adding elfutils-devel to CC to keep everyone up to date on the state of
> the patches.
>
> On Mon, 2024-06-10 at 23:36 -0700, Tony Ambardar wrote:
>> On Mon, Jun 03, 2024 at 08:47:24PM -0700, Tony Ambardar wrote:
>>> On Mon, Jun 03, 2024 at 09:18:33PM +0200, Mark Wielaard wrote:
>>>> On Mon, Jun 03, 2024 at 02:40:45PM -0300, Arnaldo Carvalho de Melo wrote:
>>>>> Couldn't find a way to ask eu-readelf for more verbose output, where we
>>>>> could perhaps get some clue as to why it produces nothing while binutils
>>>>> readelf manages to grok it, Mark, do you know some other way to ask
>>>>> eu-readelf to produce more debug output?
>>>>>
>>>>> I'm unsure if the netdevsim.ko file was left in a semi encoded BTF state
>>>>> that then made eu-readelf to not be able to process it while pahole,
>>>>> that uses eltuils' libraries, was able to process the first two CUs for
>>>>> a kernel module and all the CUs for the vmlinux file :-\
>>>>>
>>>>> Mark, the whole thread is available at:
>>>>>
>>>>> https://lore.kernel.org/all/Zl3Zp5r9m6X_i_J4@x1/T/#u
>>>> I haven't looked at the vmlinux file. But for the .ko file the issue
>>>> is that the elfutils MIPS backend isn't complete. Specifically MIPS
>>>> relocations aren't recognized (and so cannot be applied). There are
>>>> some pending patches which try to fix that:
>>>>
>>>> https://patchwork.sourceware.org/project/elfutils/list/?series=31601
>>> Earlier in the thread, Hengqi Chen pointed out the latest elfutils backend
>>> work for MIPS, and I locally rebuilt elfutils and then pahole from their
>>> respective next/main branches. For elfutils, main (935ee131cf7c) includes
>>>
>>>   e259f126 Support Mips architecture
>>>   f2acb069 stack: Fix stack unwind failure on mips
>>>   db33cb0c backends: Add register_info, return_value_location, core_note mips
>>>
>>> which partially applies the patchwork series but leaves out the support for
>>> readelf, strip, and elflint.
>>>
>>> I believe this means the vmlinux and .ko files I shared are OK, or is there
>>> more backend work needed for MIPS?
>>>
>>> The bits missing in eu-readelf would explain the blank output both Arnaldo
>>> and I see from "$ eu-readelf -winfo vmlinux". I tried rebuilding with the
>>> patchwork readelf patch locally but ran into merge conflicts.
>> A short update, starting with answering my own question.
>>
>> No, apparently the above commits *do not* complete the backend work. Ying
>> Huang submitted additional related patches since March 5: [1][2]
>>
>>     strip: Adapt src/strip -o -f on mips
>>     readelf: Adapt src/readelf -h/-S/-r/-w/-l/-d/-a on mips
>>     elflint: adapt src/elflint --gnu src/nm on mips
>>     test: Add mips in run-allregs.sh and run-readelf-mixed-corenote.sh
>>
>> Despite the titles, these patches do include core backend changes for MIPS.
>> I resolved the various merge conflicts [3], rebuilt elfutils, and retested
>> kernel builds to now find:
>>
>>   - pahole is able to read DWARF[45] info and create .BTF for modules
>>   - resolve_btfids can successfully patch .BTF_ids in modules
>>   - kernel successfully loads modules with BTF and kfuncs (tested 6.6 LTS)
>>
>> Huzzah!
>>
>>
>> Ying:
>>
>> Thank you for developing these MIPS patches. In your view, are the MIPS
>> changes now complete, or do you plan further updates that might improve or
>> impact parsing DWARF debug/reloc info in apps like pahole?
>>
>>
>> Mark:
>>
>> Given that BTF usage on Linux/MIPS is basically broken without these
>> patches, could I request some of your review time for them to be merged? If
>> it's helpful, my branch [3] includes all patches with conflicts fixed, and
>> I also successfully ran the elfutils self-tests (including MIPS from Ying).
>> Please feel free to add for these patches:
>>
>>     Tested-by: Tony Ambardar <Tony.Ambardar@gmail.com>
> Yes, I would very much like to integrate the rest of these patches. But
> I keep running out of time. The main issues were that, as you noticed,
> the patches mix backend and frontend tool changes a bit. 

The reason about the mixture and title is that only by fixing the acquisition of relocation information can 'strip' and 'readelf -w' work normally.

Now it seems really confusing, so I want to do some changes based on your opinions, change the titles or reorganize these patches to make them look more logical.

> I don't have
> access to a MIPS system to test them on. There are a couple of
> different MIPS abis (I believe all combinations of 32/64 bit and
> big/little endianness), but people have only tested on mips64le (maybe
> that is the only relevant one these days?) 

I have tested other mips abi before, I will test it agagin and attach the results of 'make check' with and without patch.

And add a description of mips abi support in the commit message.

> And finally the way MIPS
> represents relocations is slightly different than any other ELF
> architecture does. So we have to translate that somewhere to make the
> standards functions work. I have to convince myself that doing that in
> elf_getdata as the patches do is the right place.


The controversial problem was the location of the code, the code was to change the original relocation info to ensure mips can obtain the correct index value and symble index.

Or we did not modify the original data, we modify the way to obtain the index value and symble index at funtion 'gelf_getrela' in file 'libelf/gelf_getrela.c','libelf/gelf_getrel.c','libelf/gelf_update_rela.c','libelf/gelf_update_rel.c'.

Where the function 'gelf_getrela' is called，modify the relocation info that has been obtained .

What do you think of this?


Thanks,

Ying

>
>> Many thanks everyone for your help,
>> Tony
>>
>> [1]: https://patchwork.sourceware.org/project/elfutils/list/?series=31601
>> [2]: https://patchwork.sourceware.org/project/elfutils/list/?series=34310
>> [3]:
>> https://github.com/guidosarducci/elfutils/commits/main-fix-mips-support-reloc/