Re: Problem with BTF generation on mips64el

[Jiri Olsa <olsajiri@gmail.com>]

On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> Hi Tony,
> 
> On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> >
> > Hello,
> >
> > For some time now I'm seeing multiple issues during BTF generation while
> > building recent kernels targeting mips64el, and would appreciate some help
> > to understand and fix the problems.
> >
> > Some relate to resolve_btfids:
> >
> > >   LD      vmlinux
> > >   BTFIDS  vmlinux
> > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > >   NM      System.map
> > >   SORTTAB vmlinux
> > >   OBJCOPY vmlinux.32
> >
> > These do not appear to be #ifdef-related and have similar past reports [1].

I can reproduce the warning just for bpf_session_cookie,
which has fix in progress:
  https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t

> >
> > I also see many pahole failures during BTF encoding of modules, such as:
> >
> > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > >   CC [M]  net/psample/psample.mod.o
> > >   LD [M]  crypto/cmac.ko
> > >   BTF [M] crypto/cmac.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> The issue seems to be related to elfutils. Have you tried build from
> the latest elfutils source ?
> I saw the latest MIPS backend in elfutils already implemented the
> reloc_simple_type hook.

hi,
+1, could you also check the pahole version you used?

jirka

> 
> > >   LD [M]  lib/test_bpf.ko
> > >   BTF [M] lib/test_bpf.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > >   LD [M]  lib/crc-ccitt.ko
> > >   BTF [M] lib/crc-ccitt.ko
> > > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x9331> not handled!
> > > die__process_unit: tag not supported 0x11 (compile_unit)!
> > > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> > >   LD [M]  lib/libcrc32c.ko
> > >   BTF [M] lib/libcrc32c.ko
> > > die__process_unit: DW_TAG_compile_unit (0x11) @ <0x99a5> not handled!
> > > die__process_unit: tag not supported 0x11 (compile_unit)!
> > > die__process: got compile_unit unexpected tag after DW_TAG_compile_unit!
> > >   LD [M]  lib/ts_kmp.ko
> > >   BTF [M] lib/ts_kmp.ko
> > >   LD [M]  lib/ts_bm.ko
> > >   BTF [M] lib/ts_bm.ko
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > or DW_TAG_skeleton_unit expected got member (0xd)!
> >
> > I have seen reports of various similar "die__" messages on the dwarves
> > list and repo, with the hint of an elfutils connection [2] but nothing
> > conclusive.
> >
> > Details of the git commit and build environment are as follows:
> >
> > > $ git log -1 --oneline  bpf/master
> > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > test_sk_storage_tracing.c
> > >
> > > $ lsb_release -a
> > > Description:    Ubuntu 22.04.4 LTS
> > >
> > > $ cat gcc-compile.txt
> > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > >
> > > $ mips64el-linux-gnuabi64-gcc --version
> > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > >
> > > $ mips64el-linux-gnuabi64-ld --version
> > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > >
> > > $ pahole --version
> > > v1.26
> > >
> > > $ ldd $(which pahole)
> > >         linux-vdso.so.1 (0x00007fff16f3f000)
> > >         libdw.so.1 => /lib/x86_64-linux-gnu/libdw.so.1 (0x00007fc39d42e000)
> > >         libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007fc39d410000)
> > >         libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fc39d3f4000)
> > >         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc39d1cb000)
> > >         liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fc39d1a0000)
> > >         libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fc39d18d000)
> > >         /lib64/ld-linux-x86-64.so.2 (0x00007fc39d59d000)
> > >
> > > $ dpkg -s elfutils
> > > Package: elfutils
> > > ...
> > > Version: 0.186-1build1
> > > Depends: libasm1 (>= 0.132), libc6 (>= 2.34), libdw1 (= 0.186-1build1),
> > > libelf1 (= 0.186-1build1), libstdc++6 (>= 4.1.1)
> >
> > For reference, I also attached the full .config and build log from the
> > above.
> >
> > I should add this is not only a problem with the latest bpf/master but
> > also appears to affect the 6.6.x LTS kernel, which I tested while building
> > a mips64el OpenWrt distro image. That build environment employs the latest
> > gcc 13.3, binutils 2.42, pahole 1.26, and elfutils 0.191.
> >
> > Not only do I see similar warnings from resolve_btfids and pahole, but
> > while running the distro image I encounter module loading failures that
> > suggest ELF corruption in some module .ko files, based on the following:
> >
> > > root@OpenWrt:/# strace insmod /lib/modules/6.6.30/nf_conntrack.ko
> > > ...
> > > init_module(0xfff3e36160, 307448, "")   = -1 EINVAL (Invalid argument)
> > > ...
> >
> > > $ man init_module
> > > ...
> > > The following errors may additionally occur for init_module():
> > > ...
> > >      EINVAL param_values is invalid, or some part of the ELF image in
> > >      module_image contains inconsistencies.
> > > ...
> >
> > I'd be grateful if some of the BTF/pahole experts could please review this
> > issue and share next steps or other details I might provide.
> >
> > Thanks,
> > Tony Ambardar
> >
> > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > Link: https://github.com/acmel/dwarves/issues/45 [2]
> 
> Cheers,
> Hengqi
>