Re: FAILED: load BTF from vmlinux: Invalid argument

[Jiri Olsa <olsajiri@gmail.com>]

On Fri, Aug 04, 2023 at 11:03:03PM +0100, Alan Maguire wrote:
> On 04/08/2023 17:11, Nick Desaulniers wrote:
> > + Marcus (who also just reported seeing this
> > https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1664671027
> > and might be able to help reproduce).
> > + Fangrui (because seeing dd used as a result of 90ceddcb4950 makes me shudder)
> > 
> > On Thu, Aug 3, 2023 at 3:10 PM Alan Maguire <alan.maguire@oracle.com> wrote:
> >>
> >> On 03/08/2023 21:50, Nick Desaulniers wrote:
> >>> On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >>>>
> >>>> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> >>>>> Hi Martin (and BTF/BPF team),
> >>>>> I've observed 2 user reports with the error from the subject of this email.
> >>>>> https://github.com/ClangBuiltLinux/linux/issues/1825
> >>>>> https://bbs.archlinux.org/viewtopic.php?id=284177
> >>>>>
> >>>>> Any chance you could take a look at these reports and help us figure
> >>>>> out what's going wrong here?  Nathan and I haven't been able to
> >>>>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> >>>>>
> >>>>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
> >>>>
> >>>> Masami had a problem with new versions of compilers that was solved
> >>>> with:
> >>>>
> >>>> ------------------------ 8< --------------------------------------------
> >>>>> To check that please tweak:
> >>>>>
> >>>>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> >>>>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> >>>>> # CONFIG_DEBUG_INFO_DWARF4 is not set
> >>>>> # CONFIG_DEBUG_INFO_DWARF5 is not set
> >>>>> ⬢[acme@toolbox perf-tools-next]$
> >>>>>
> >>>>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> >>>>> CONFIG_DEBUG_INFO_DWARF4.
> >>>>
> >>>> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> >>>
> >>> Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> >>> is not what I'd consider a fix. Someday we can move to
> >>> DWARFv5...someday...
> >>>
> >>> What you describe sounds like build success, but reduction in debug info.
> >>>
> >>> The reports I'm referring to seem to result in a build failure.
> >>>
> >>
> >> This is a strange one. The error in question
> >>
> >> CC .vmlinux.export.o
> >> UPD include/generated/utsversion.h
> >> CC init/version-timestamp.o
> >> LD .tmp_vmlinux.btf
> >> BTF .btf.vmlinux.bin.o
> >> libbpf: BTF header not found
> >> pahole: .tmp_vmlinux.btf: Invalid argument
> > 
> > That's slightly different from Tomasz and Marcus' report (not sure if
> > that's relevant):
> > 
> > FAILED: load BTF from vmlinux: Invalid argument
> > 
> > That seems to come from
> > tools/bpf/resolve_btfids/main.c:529
> > Which seems like some failed call to btf_parse().
> > EINVAL is getting propagated up from btf_parse(), but that's not super
> > descriptive...
> > 
> Okay, that makes more sense. Basically the stage where we read vmlinux
> BTF to do BTF id resolution (BTFIDS) is finding an empty BTF section.

+1, looks like pahole failed to generate the BTF section, the BTFIDS
is just follow up error.. we might want to consider special error
output for missing BTF data ;-)

I can't reproduce this on my setup with either gcc or clang and trying
DWARF4/5 config options and latest and 1.24 pahole version

> 
> > The hard part is that I suspect OpenMandriva (Tomasz) and Marcus are
> > both setting additional flags in their toolchains, which can make
> > reproducing tricky.
> >
> 
> I tried falling back to the config referenced in the earlier bug report
> 
> https://github.com/ClangBuiltLinux/linux/files/10050200/config_bpf.txt

hum, I did not find this in the report.. are there more kernel configs
related to this issue? seems like more people hit this

thanks,
jirka

> 
> ...but still couldn't reproduce it with LLVM 17 + pahole v1.24. That
> config did specify DWARF5; if we can reproduce this, it would probably
> be good to vary between forcing DWARF4 and DWARF5 to see if that is a
> contributing factor as Arnaldo suggested.
> 
> Alan
> 
> >>
> >> ...occurs during BTF parsing when the raw size of the BTF is smaller
> >> than the BTF header size, which should never happen unless BTF
> >> is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
> >> we should be generating it from DWARF. The only time pahole parses BTF
> >> is when it's creating split BTF for modules (it parses the base BTF), or
> >> when it's reading existing BTF, neither of which it should be doing at
> >> this stage.
> >>
> >> But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
> >> Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
> >> If that went awry somehow and .tmp_vmlinux.btf wasn't created, it
> > 
> > Wouldn't we expect some kind of linker error though in that case?
> > 
> >> would explain the "Invalid argument" error:
> >>
> >> $ pahole -J nosuchfile
> >> pahole: nosuchfile: Invalid argument
> >>
> >> I see some clang specifics in vmlinux_link(), so I think a good
> >> first step would be to check if .tmp_vlinux.btf exists prior
> >> to running pahole. The submitter mentioned swapping linkers seems to
> >> help, so that seems a promising angle. If there's a kernel .config
> >> available I can try and reproduce the failure too. Thanks!
> >>
> >> Alan
> >>
> >>>>
> >>>>   LD      .tmp_vmlinux.btf
> >>>>   BTF     .btf.vmlinux.bin.o
> >>>>   LD      .tmp_vmlinux.kallsyms1
> >>>>
> >>>> And
> >>>>
> >>>> / # strings /sys/kernel/btf/vmlinux | wc -l
> >>>> 89921
> >>>> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> >>>> kfree
> >>>>
> >>>> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> >>>> are about 30000.)
> >>>
> >>>
> >>>
> > 
> > 
> > 
>