Re: Problem with BTF generation on mips64el

[Tony Ambardar <tony.ambardar@gmail.com>]

On Fri, May 31, 2024 at 01:06:41PM -0300, Arnaldo Carvalho de Melo wrote:
> On Fri, May 31, 2024 at 03:49:38AM -0700, Tony Ambardar wrote:
> > Hello Hengqi,
> > 
> > On Fri, May 31, 2024 at 10:17:53AM +0800, Hengqi Chen wrote:
> > > Hi Tony,
> > > 
> > > On Fri, May 31, 2024 at 9:30 AM Tony Ambardar <tony.ambardar@gmail.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > For some time now I'm seeing multiple issues during BTF generation while
> > > > building recent kernels targeting mips64el, and would appreciate some help
> > > > to understand and fix the problems.
> > > >
> > > SNIP
> > > >
> > > > >   CC [M]  net/ipv6/netfilter/nft_fib_ipv6.mod.o
> > > > >   CC [M]  net/ipv6/netfilter/ip6t_REJECT.mod.o
> > > > >   CC [M]  net/psample/psample.mod.o
> > > > >   LD [M]  crypto/cmac.ko
> > > > >   BTF [M] crypto/cmac.ko
> > > > > die__process: DW_TAG_compile_unit, DW_TAG_type_unit, DW_TAG_partial_unit
> > > > > or DW_TAG_skeleton_unit expected got member (0xd)!
> 
> Can you check the kernel CONFIG_ variables related to DEBUG information
> and post them here? I have this on fedora:
> 
> [acme@nine linux]$ grep CONFIG_DEBUG_INFO /boot/config-5.14.0-362.18.1.el9_3.x86_64 
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> [acme@nine linux]$
> 
> If you have CONFIG_DEBUG_INFO_SPLIT, CONFIG_DEBUG_INFO_COMPRESSED or
> CONFIG_DEBUG_INFO_REDUCED set to 'y', please try with the values in the
> fedora config.

One more useful observation: the pahole BTF generation problems appear only
for mips64. If I build the same config for mips32be I see none of those
"die__process: errors" from pahole and the processed modules load properly.

I also tried running pahole under gdb to look further, but lack knowledge
of pahole and libdw internals, so couldn't narrow things down to a pahole,
elfutils or compiler-output problem.

To help troubleshoot further, I packaged my base vmlinux, some module .ko
files showing problems, and a reproducer script. I've uploaded the tarball
here:

https://www.dropbox.com/scl/fi/3ce22fi2q861wqvbq9mwy/repro_die__process.tar.xz?rlkey=ev494phabmfl5qe55xrn1jh3t&st=vrp5mxhh&dl=0

Thanks,
Tony

> 
> - Arnaldo
> 
> > > The issue seems to be related to elfutils. Have you tried build from
> > > the latest elfutils source ?
> > > I saw the latest MIPS backend in elfutils already implemented the
> > > reloc_simple_type hook.
> > 
> > Good idea. I tried rebuilding elfutils from the latest upstream commit:
> > https://sourceware.org/git/?p=elfutils.git;a=commit;h=935ee131cf7c87296df9412b7e3370085e7c7508
> > 
> > I then linked this elfutils with pahole built from the latest pahole/next:
> > https://git.kernel.org/pub/scm/devel/pahole/pahole.git/commit/?h=next&id=a9ae414fef421bdeb13ff7ffe13271e1e4f58993
> > 
> > I also confirmed resolve_btfids links to the new elfutils, and then rebuilt
> > the kernel with the same config. Unfortunately, the warnings/errors from
> > resolve_btfids and pahole still occur.
> > 
> > > SNIP
> > > >
> > > > I'd be grateful if some of the BTF/pahole experts could please review this
> > > > issue and share next steps or other details I might provide.
> > > >
> > > > Thanks,
> > > > Tony Ambardar
> > > >
> > > > Link: https://lore.kernel.org/all/202401211357.OCX9yllM-lkp@intel.com/ [1]
> > > > Link: https://github.com/acmel/dwarves/issues/45 [2]
> > > 
> > > Cheers,
> > > Hengqi
> > 
> > Thanks for the suggestion,
> > Tony