Re: Problem with BTF generation on mips64el

[Tony Ambardar <tony.ambardar@gmail.com>]

On Fri, May 31, 2024 at 04:21:48PM +0200, Jiri Olsa wrote:
> On Fri, May 31, 2024 at 04:36:48AM -0700, Tony Ambardar wrote:
> > Hi Jiri,
> > 
> > On Fri, May 31, 2024 at 10:13:21AM +0200, Jiri Olsa wrote:
> > > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > > Hi Tony,
> > > > 
> > > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > > to understand and fix the problems.
> > > > >
> > > > > Some relate to resolve_btfids:
> > > > >
> > > > > >   LD      vmlinux
> > > > > >   BTFIDS  vmlinux
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_key_put
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> > > > > > WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> > > > > >   NM      System.map
> > > > > >   SORTTAB vmlinux
> > > > > >   OBJCOPY vmlinux.32
> > > > >
> > > > > These do not appear to be #ifdef-related and have similar past reports [1].
> > > 
> > > I can reproduce the warning just for bpf_session_cookie,
> > > which has fix in progress:
> > >   https://lore.kernel.org/bpf/20240531071557.MvfIqkn7@linutronix.de/T/#t
> > 
> > I gather there are different root causes for these warnings. From the link,
> > the issue with bpf_session_cookie seems related to conditional compilation,
> > which have come up before on the mailing list IIRC.
> > 
> > In comparison, consider my above warning for bpf_key_put, which is defined
> > in kernel/trace/bpf_trace.c. This kfunc is guarded by CONFIG_KEY, which is
> > enabled in my config and so not the issue. I can in fact see the global
> > text symbol for bpf_key_put in bpf_trace.o, but not in vmlinux.
> 
> yes, that would be a problem, the resolve_btfids goes through the BTF
> lists and needs to resolve the function in vmlinux, when it's not there,
> it will output the warning above
> 
> > 
> > So I suspect the warnings might come from linker or optimization problems,
> > or perhaps even an issue related to the __bpf_kfunc annotation. WDYT?
> 
> I can reproduce now resolve_btfids warnings:
> 

That's great to hear! Did you also use mips64el? And were you able to see
the various "die__proc:" errors from pahole during BTF generation?

>   LD      vmlinux
>   BTFIDS  vmlinux
> WARN: resolve_btfids: unresolved symbol bpf_verify_pkcs7_signature
> WARN: resolve_btfids: unresolved symbol bpf_session_is_return
> WARN: resolve_btfids: unresolved symbol bpf_session_cookie
> WARN: resolve_btfids: unresolved symbol bpf_lookup_user_key
> WARN: resolve_btfids: unresolved symbol bpf_lookup_system_key
> WARN: resolve_btfids: unresolved symbol bpf_key_put
> WARN: resolve_btfids: unresolved symbol bpf_iter_task_next
> WARN: resolve_btfids: unresolved symbol bpf_iter_css_task_new
> WARN: resolve_btfids: unresolved symbol bpf_get_file_xattr
> WARN: resolve_btfids: unresolved symbol bpf_ct_insert_entry
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_release
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_from_id
> WARN: resolve_btfids: unresolved symbol bpf_cgroup_acquire
> WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages
> 
> when I call bpf_key_put to make it 'used' it stays in vmlinux, so I wonder
> the __bpf_kfunc flags do not work properly on this cross compile chain..
> and linker just optimize it away?

Yes, I verified the problem occurs during link-time garbage cleaning, when
LTO is enabled through e.g. LD_DEAD_CODE_DATA_ELIMINATION=y.

> 
> #define __bpf_kfunc __used noinline
> 

After researching, I believe this isn't a mips problem specifically, rather
the __bpf_kfunc tag only partly works. It appears the __used macro works at
the compiler and IR-level but is ignored by the linker. I have a working
fix that also handles the linker level, so I'll post the patches shortly.

Unfortunately, that still leaves the pahole errors generating BTF, which
are serious as they seem to break module loading.

> thanks,
> jirka

Cheers,
Tony
> 
> > 
> > > 
> > > > >
> > > > > I also see many pahole failures during BTF encoding of modules, such as:
> > > > >
> > > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > > >   CC [M]  net/psample/psample.mod.o
> > > > > >   LD [M]  crypto/cmac.ko
> > > > > >   BTF [M] crypto/cmac.ko
> > > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> > > > 
> > > > The issue seems to be related to elfutils. Have you tried build from
> > > > the latest elfutils source ?
> > > > I saw the latest MIPS backend in elfutils already implemented the
> > > > reloc_simple_type hook.
> > > 
> > > hi,
> > > +1, could you also check the pahole version you used?
> > 
> > No luck I'm afraid with using the latest elfutils as suggested.
> > 
> > I used pahole v1.26 in my original testing for this bug report, as noted
> > below (might have been buried a bit).
> > 
> > > 
> > > jirka
> > > 
> > > SNIP
> > >
> > > > > Details of the git commit and build environment are as follows:
> > > > >
> > > > > > $ git log -1 --oneline  bpf/master
> > > > > > 9dfdb706e164 (bpf/master) selftests/bpf: fix inet_csk_accept prototype in
> > > > > > test_sk_storage_tracing.c
> > > > > >
> > > > > > $ lsb_release -a
> > > > > > Description:    Ubuntu 22.04.4 LTS
> > > > > >
> > > > > > $ cat gcc-compile.txt
> > > > > > ARCH=mips CROSS_COMPILE=mips64el-linux-gnuabi64- CC="ccache ${CROSS_COMPILE}gcc" make -j6
> > > > > >
> > > > > > $ mips64el-linux-gnuabi64-gcc --version
> > > > > > mips64el-linux-gnuabi64-gcc (Ubuntu 10.3.0-1ubuntu1) 10.3.0
> > > > > >
> > > > > > $ mips64el-linux-gnuabi64-ld --version
> > > > > > GNU ld (GNU Binutils for Ubuntu) 2.38
> > > > > >
> > > > > > $ pahole --version
> > > > > > v1.26
> > > 
> > > SNIP
> > >
> > > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > > issue and share next steps or other details I might provide.
> > > > >
> > > > > Thanks,
> > > > > Tony Ambardar
> > > > >
> > > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > > 
> > > > Cheers,
> > > > Hengqi
> > 
> > Thanks,
> > Tony