Re: Problem with BTF generation on mips64el

[Jiri Olsa <olsajiri@gmail.com>]

On Mon, Jun 03, 2024 at 02:02:45AM -0700, Tony Ambardar wrote:
> On Fri, May 31, 2024 at 04:21:48PM +0200, Jiri Olsa wrote:
> > On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> > > Hi Jiri,
> > > 
> > > On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > > Hi Tony,
> > > > > 
> > > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > > to understand and fix the problems.
> > > > > >
> > > > > > Some relate to resolve_btfids:
> > > > > >
> > > > > > >   LD      vmlinux
> > > > > > >   BTFIDS  vmlinux
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > > > >   NM      System.map
> > > > > > >   SORTTAB vmlinux
> > > > > > >   OBJCOPY vmlinux.32
> > > > > >
> > > > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > > > 
> > > > I can reproduce the warning just for bpf_session_cookie,
> > > > which has fix in progress:
> > > >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> > > 
> > > I gather there are different root causes for these warnings. From the link,
> > > the issue with bpf_session_cookie seems related to conditional compilation,
> > > which have come up before on the mailing list IIRC.
> > > 
> > > In comparison, consider my above warning for bpf_key_put, which is defined
> > > in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> > > enabled in my config and so not the issue. I can in fact see the global
> > > text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
> > 
> > yes, that would be a problem, the resolve_btfids goes through the BTF
> > lists and needs to resolve the function in vmlinux, when it's not there,
> > it will output the warning above
> > 
> > > 
> > > So I suspect the warnings might come from linker or optimization problems,
> > > or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
> > 
> > I can reproduce now resolve_btfids warnings:
> > 
> 
> That's great to hear! Did you also use mips64el? And were you able to see

nope, just mips64, couldn't find mips64el on fedora

> the various "die__proc:" errors from pahole during BTF generation?

yep, I can see them as well

> 
> >   LD      vmlinux
> >   BTFIDS  vmlinux
> > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > WARN: resolve_btfids: unresolved symbol bpf_session_is_return
> > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > 
> > when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
> > the __bpf_kfunc flags do not work properly on this cross compile chain..
> > and linker just optimize it away?
> 
> Yes, I verified the problem occurs during link-time garbage cleaning, when
> LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.
> 
> > 
> > #define __bpf_kfunc __used noinline
> > 
> 
> After researching, I believe this isn't a mips problem specifically, rather
> the __bpf_kfunc tag only partly works. It appears the __used macro works at
> the compiler and IR-level but is ignored by the linker. I have a working
> fix that also handles the linker level, so I'll post the patches shortly.

great, I was wondering how to fix that

thanks,
jirka

> 
> Unfortunately, that still leaves the pahole errors generating BTF, which
> are serious as they seem to break module loading.
> 
> > thanks,
> > jirka
> 
> Cheers,
> Tony
> > 
> > > 
> > > > 
> > > > > >
> > > > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > > > >
> > > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > > >   LD [M]  crypto/cmac.ko
> > > > > > >   BTF [M] crypto/cmac.ko
> > > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > > > 
> > > > > The issue seems to be related to elfutils. Have you tried build from
> > > > > the latest elfutils source ?
> > > > > I saw the latest MIPS backend in elfutils already implemented the
> > > > > reloc_simple_type hook.
> > > > 
> > > > hi,
> > > > +1, could you also check the pahole version you used?
> > > 
> > > No luck I'm afraid with using the latest elfutils as suggested.
> > > 
> > > I used pahole v1.26 in my original testing for this bug report, as noted
> > > below (might have been buried a bit).
> > > 
> > > > 
> > > > jirka
> > > > 
> > > > SNIP
> > > >
> > > > > > Details of the git commit and build environment are as follows:
> > > > > >
> > > > > > > $ git log -1 --oneline  bpf/master
> > > > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > > > test_sk_storage_tracing.c
> > > > > > >
> > > > > > > $ lsb_release -a
> > > > > > > Description:    Ubuntu 22.04.4 LTS
> > > > > > >
> > > > > > > $ cat gcc-compile.txt
> > > > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > > > >
> > > > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > > > >
> > > > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > > > >
> > > > > > > $ pahole --version
> > > > > > > v1.26
> > > > 
> > > > SNIP
> > > >
> > > > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > > > issue and share next steps or other details I might provide.
> > > > > >
> > > > > > Thanks,
> > > > > > Tony Ambardar
> > > > > >
> > > > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > > > 
> > > > > Cheers,
> > > > > Hengqi
> > > 
> > > Thanks,
> > > Tony