FAILED: load BTF from vmlinux: Invalid argument

FAILED: load BTF from vmlinux: Invalid argument

[Nick Desaulniers]

Hi Martin (and BTF/BPF team),
I've observed 2 user reports with the error from the subject of this email.
https://github.com/ClangBuiltLinux/linux/issues/1825
https://bbs.archlinux.org/viewtopic.php?id=284177

Any chance you could take a look at these reports and help us figure
out what's going wrong here?  Nathan and I haven't been able to
reproduce, but this seems to be affecting OpenMandriva (and Tomasz).

Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
-- 
Thanks,
~Nick Desaulniers

Re: FAILED: load BTF from vmlinux: Invalid argument

[Arnaldo Carvalho de Melo]

Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> Hi Martin (and BTF/BPF team),
> I've observed 2 user reports with the error from the subject of this email.
> https://github.com/ClangBuiltLinux/linux/issues/1825
> https://bbs.archlinux.org/viewtopic.php?id=284177
> 
> Any chance you could take a look at these reports and help us figure
> out what's going wrong here?  Nathan and I haven't been able to
> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> 
> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?

Masami had a problem with new versions of compilers that was solved
with:

------------------------ 8< --------------------------------------------
> To check that please tweak:
>
> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> # CONFIG_DEBUG_INFO_DWARF4 is not set
> # CONFIG_DEBUG_INFO_DWARF5 is not set
> ⬢[acme@toolbox perf-tools-next]$
>
> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> CONFIG_DEBUG_INFO_DWARF4.

Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.

  LD      .tmp_vmlinux.btf
  BTF     .btf.vmlinux.bin.o
  LD      .tmp_vmlinux.kallsyms1

And

/ # strings /sys/kernel/btf/vmlinux | wc -l
89921
/ # strings /sys/kernel/btf/vmlinux | grep -w kfree
kfree

It seems the BTF is correctly generated. (with DWARF5, the number of symbols
are about 30000.)

Re: FAILED: load BTF from vmlinux: Invalid argument

[Nick Desaulniers]

On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>
> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> > Hi Martin (and BTF/BPF team),
> > I've observed 2 user reports with the error from the subject of this email.
> > https://github.com/ClangBuiltLinux/linux/issues/1825
> > https://bbs.archlinux.org/viewtopic.php?id=284177
> >
> > Any chance you could take a look at these reports and help us figure
> > out what's going wrong here?  Nathan and I haven't been able to
> > reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> >
> > Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
>
> Masami had a problem with new versions of compilers that was solved
> with:
>
> ------------------------ 8< --------------------------------------------
> > To check that please tweak:
> >
> > ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> > # CONFIG_DEBUG_INFO_DWARF4 is not set
> > # CONFIG_DEBUG_INFO_DWARF5 is not set
> > ⬢[acme@toolbox perf-tools-next]$
> >
> > i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> > CONFIG_DEBUG_INFO_DWARF4.
>
> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.

Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
is not what I'd consider a fix. Someday we can move to
DWARFv5...someday...

What you describe sounds like build success, but reduction in debug info.

The reports I'm referring to seem to result in a build failure.

>
>   LD      .tmp_vmlinux.btf
>   BTF     .btf.vmlinux.bin.o
>   LD      .tmp_vmlinux.kallsyms1
>
> And
>
> / # strings /sys/kernel/btf/vmlinux | wc -l
> 89921
> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> kfree
>
> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> are about 30000.)



-- 
Thanks,
~Nick Desaulniers

Re: FAILED: load BTF from vmlinux: Invalid argument

[Alan Maguire]

On 03/08/2023 21:50, Nick Desaulniers wrote:
> On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>>
>> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
>>> Hi Martin (and BTF/BPF team),
>>> I've observed 2 user reports with the error from the subject of this email.
>>> https://github.com/ClangBuiltLinux/linux/issues/1825
>>> https://bbs.archlinux.org/viewtopic.php?id=284177
>>>
>>> Any chance you could take a look at these reports and help us figure
>>> out what's going wrong here?  Nathan and I haven't been able to
>>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
>>>
>>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
>>
>> Masami had a problem with new versions of compilers that was solved
>> with:
>>
>> ------------------------ 8< --------------------------------------------
>>> To check that please tweak:
>>>
>>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
>>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
>>> # CONFIG_DEBUG_INFO_DWARF4 is not set
>>> # CONFIG_DEBUG_INFO_DWARF5 is not set
>>> ⬢[acme@toolbox perf-tools-next]$
>>>
>>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
>>> CONFIG_DEBUG_INFO_DWARF4.
>>
>> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> 
> Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> is not what I'd consider a fix. Someday we can move to
> DWARFv5...someday...
> 
> What you describe sounds like build success, but reduction in debug info.
> 
> The reports I'm referring to seem to result in a build failure.
>

This is a strange one. The error in question

CC .vmlinux.export.o
UPD include/generated/utsversion.h
CC init/version-timestamp.o
LD .tmp_vmlinux.btf
BTF .btf.vmlinux.bin.o
libbpf: BTF header not found
pahole: .tmp_vmlinux.btf: Invalid argument

...occurs during BTF parsing when the raw size of the BTF is smaller
than the BTF header size, which should never happen unless BTF
is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
we should be generating it from DWARF. The only time pahole parses BTF
is when it's creating split BTF for modules (it parses the base BTF), or
when it's reading existing BTF, neither of which it should be doing at
this stage.

But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
If that went awry somehow and .tmp_vmlinux.btf wasn't created, it
would explain the "Invalid argument" error:

$ pahole -J nosuchfile
pahole: nosuchfile: Invalid argument

I see some clang specifics in vmlinux_link(), so I think a good
first step would be to check if .tmp_vlinux.btf exists prior
to running pahole. The submitter mentioned swapping linkers seems to
help, so that seems a promising angle. If there's a kernel .config
available I can try and reproduce the failure too. Thanks!

Alan

>>
>>   LD      .tmp_vmlinux.btf
>>   BTF     .btf.vmlinux.bin.o
>>   LD      .tmp_vmlinux.kallsyms1
>>
>> And
>>
>> / # strings /sys/kernel/btf/vmlinux | wc -l
>> 89921
>> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
>> kfree
>>
>> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
>> are about 30000.)
> 
> 
> 

Re: FAILED: load BTF from vmlinux: Invalid argument

[Nick Desaulniers]

+ Marcus (who also just reported seeing this
https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1664671027
and might be able to help reproduce).
+ Fangrui (because seeing dd used as a result of 90ceddcb4950 makes me shudder)

On Thu, Aug 3, 2023 at 3:10 PM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On 03/08/2023 21:50, Nick Desaulniers wrote:
> > On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >>
> >> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> >>> Hi Martin (and BTF/BPF team),
> >>> I've observed 2 user reports with the error from the subject of this email.
> >>> https://github.com/ClangBuiltLinux/linux/issues/1825
> >>> https://bbs.archlinux.org/viewtopic.php?id=284177
> >>>
> >>> Any chance you could take a look at these reports and help us figure
> >>> out what's going wrong here?  Nathan and I haven't been able to
> >>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> >>>
> >>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
> >>
> >> Masami had a problem with new versions of compilers that was solved
> >> with:
> >>
> >> ------------------------ 8< --------------------------------------------
> >>> To check that please tweak:
> >>>
> >>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> >>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> >>> # CONFIG_DEBUG_INFO_DWARF4 is not set
> >>> # CONFIG_DEBUG_INFO_DWARF5 is not set
> >>> ⬢[acme@toolbox perf-tools-next]$
> >>>
> >>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> >>> CONFIG_DEBUG_INFO_DWARF4.
> >>
> >> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> >
> > Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> > is not what I'd consider a fix. Someday we can move to
> > DWARFv5...someday...
> >
> > What you describe sounds like build success, but reduction in debug info.
> >
> > The reports I'm referring to seem to result in a build failure.
> >
>
> This is a strange one. The error in question
>
> CC .vmlinux.export.o
> UPD include/generated/utsversion.h
> CC init/version-timestamp.o
> LD .tmp_vmlinux.btf
> BTF .btf.vmlinux.bin.o
> libbpf: BTF header not found
> pahole: .tmp_vmlinux.btf: Invalid argument

That's slightly different from Tomasz and Marcus' report (not sure if
that's relevant):

FAILED: load BTF from vmlinux: Invalid argument

That seems to come from
tools/bpf/resolve_btfids/main.c:529
Which seems like some failed call to btf_parse().
EINVAL is getting propagated up from btf_parse(), but that's not super
descriptive...

The hard part is that I suspect OpenMandriva (Tomasz) and Marcus are
both setting additional flags in their toolchains, which can make
reproducing tricky.

>
> ...occurs during BTF parsing when the raw size of the BTF is smaller
> than the BTF header size, which should never happen unless BTF
> is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
> we should be generating it from DWARF. The only time pahole parses BTF
> is when it's creating split BTF for modules (it parses the base BTF), or
> when it's reading existing BTF, neither of which it should be doing at
> this stage.
>
> But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
> Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
> If that went awry somehow and .tmp_vmlinux.btf wasn't created, it

Wouldn't we expect some kind of linker error though in that case?

> would explain the "Invalid argument" error:
>
> $ pahole -J nosuchfile
> pahole: nosuchfile: Invalid argument
>
> I see some clang specifics in vmlinux_link(), so I think a good
> first step would be to check if .tmp_vlinux.btf exists prior
> to running pahole. The submitter mentioned swapping linkers seems to
> help, so that seems a promising angle. If there's a kernel .config
> available I can try and reproduce the failure too. Thanks!
>
> Alan
>
> >>
> >>   LD      .tmp_vmlinux.btf
> >>   BTF     .btf.vmlinux.bin.o
> >>   LD      .tmp_vmlinux.kallsyms1
> >>
> >> And
> >>
> >> / # strings /sys/kernel/btf/vmlinux | wc -l
> >> 89921
> >> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> >> kfree
> >>
> >> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> >> are about 30000.)
> >
> >
> >



-- 
Thanks,
~Nick Desaulniers

Re: FAILED: load BTF from vmlinux: Invalid argument

[Arnaldo Carvalho de Melo]

Em Fri, Aug 04, 2023 at 09:11:09AM -0700, Nick Desaulniers escreveu:
> + Marcus (who also just reported seeing this
> https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1664671027
> and might be able to help reproduce).
> + Fangrui (because seeing dd used as a result of 90ceddcb4950 makes me shudder)
> 
> On Thu, Aug 3, 2023 at 3:10 PM Alan Maguire <alan.maguire@oracle.com> wrote:
> >
> > On 03/08/2023 21:50, Nick Desaulniers wrote:
> > > On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > >>
> > >> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> > >>> Hi Martin (and BTF/BPF team),
> > >>> I've observed 2 user reports with the error from the subject of this email.
> > >>> https://github.com/ClangBuiltLinux/linux/issues/1825
> > >>> https://bbs.archlinux.org/viewtopic.php?id=284177
> > >>>
> > >>> Any chance you could take a look at these reports and help us figure
> > >>> out what's going wrong here?  Nathan and I haven't been able to
> > >>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> > >>>
> > >>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
> > >>
> > >> Masami had a problem with new versions of compilers that was solved
> > >> with:
> > >>
> > >> ------------------------ 8< --------------------------------------------
> > >>> To check that please tweak:
> > >>>
> > >>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> > >>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> > >>> # CONFIG_DEBUG_INFO_DWARF4 is not set
> > >>> # CONFIG_DEBUG_INFO_DWARF5 is not set
> > >>> ⬢[acme@toolbox perf-tools-next]$
> > >>>
> > >>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> > >>> CONFIG_DEBUG_INFO_DWARF4.
> > >>
> > >> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> > >
> > > Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> > > is not what I'd consider a fix. Someday we can move to
> > > DWARFv5...someday...
> > >
> > > What you describe sounds like build success, but reduction in debug info.
> > >
> > > The reports I'm referring to seem to result in a build failure.
> > >
> >
> > This is a strange one. The error in question
> >
> > CC .vmlinux.export.o
> > UPD include/generated/utsversion.h
> > CC init/version-timestamp.o
> > LD .tmp_vmlinux.btf
> > BTF .btf.vmlinux.bin.o
> > libbpf: BTF header not found
> > pahole: .tmp_vmlinux.btf: Invalid argument
> 
> That's slightly different from Tomasz and Marcus' report (not sure if
> that's relevant):
> 
> FAILED: load BTF from vmlinux: Invalid argument
> 
> That seems to come from
> tools/bpf/resolve_btfids/main.c:529
> Which seems like some failed call to btf_parse().
> EINVAL is getting propagated up from btf_parse(), but that's not super
> descriptive...

Jiri, have you stumbled on this before? 

- Arnaldo
 
> The hard part is that I suspect OpenMandriva (Tomasz) and Marcus are
> both setting additional flags in their toolchains, which can make
> reproducing tricky.
> 
> >
> > ...occurs during BTF parsing when the raw size of the BTF is smaller
> > than the BTF header size, which should never happen unless BTF
> > is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
> > we should be generating it from DWARF. The only time pahole parses BTF
> > is when it's creating split BTF for modules (it parses the base BTF), or
> > when it's reading existing BTF, neither of which it should be doing at
> > this stage.
> >
> > But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
> > Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
> > If that went awry somehow and .tmp_vmlinux.btf wasn't created, it
> 
> Wouldn't we expect some kind of linker error though in that case?
> 
> > would explain the "Invalid argument" error:
> >
> > $ pahole -J nosuchfile
> > pahole: nosuchfile: Invalid argument
> >
> > I see some clang specifics in vmlinux_link(), so I think a good
> > first step would be to check if .tmp_vlinux.btf exists prior
> > to running pahole. The submitter mentioned swapping linkers seems to
> > help, so that seems a promising angle. If there's a kernel .config
> > available I can try and reproduce the failure too. Thanks!
> >
> > Alan
> >
> > >>
> > >>   LD      .tmp_vmlinux.btf
> > >>   BTF     .btf.vmlinux.bin.o
> > >>   LD      .tmp_vmlinux.kallsyms1
> > >>
> > >> And
> > >>
> > >> / # strings /sys/kernel/btf/vmlinux | wc -l
> > >> 89921
> > >> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> > >> kfree
> > >>
> > >> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> > >> are about 30000.)
> > >
> > >
> > >
> 
> 
> 
> -- 
> Thanks,
> ~Nick Desaulniers

-- 

- Arnaldo

Re: FAILED: load BTF from vmlinux: Invalid argument

[Alan Maguire]

On 04/08/2023 17:11, Nick Desaulniers wrote:
> + Marcus (who also just reported seeing this
> https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1664671027
> and might be able to help reproduce).
> + Fangrui (because seeing dd used as a result of 90ceddcb4950 makes me shudder)
> 
> On Thu, Aug 3, 2023 at 3:10 PM Alan Maguire <alan.maguire@oracle.com> wrote:
>>
>> On 03/08/2023 21:50, Nick Desaulniers wrote:
>>> On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
>>>>
>>>> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
>>>>> Hi Martin (and BTF/BPF team),
>>>>> I've observed 2 user reports with the error from the subject of this email.
>>>>> https://github.com/ClangBuiltLinux/linux/issues/1825
>>>>> https://bbs.archlinux.org/viewtopic.php?id=284177
>>>>>
>>>>> Any chance you could take a look at these reports and help us figure
>>>>> out what's going wrong here?  Nathan and I haven't been able to
>>>>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
>>>>>
>>>>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
>>>>
>>>> Masami had a problem with new versions of compilers that was solved
>>>> with:
>>>>
>>>> ------------------------ 8< --------------------------------------------
>>>>> To check that please tweak:
>>>>>
>>>>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
>>>>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
>>>>> # CONFIG_DEBUG_INFO_DWARF4 is not set
>>>>> # CONFIG_DEBUG_INFO_DWARF5 is not set
>>>>> ⬢[acme@toolbox perf-tools-next]$
>>>>>
>>>>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
>>>>> CONFIG_DEBUG_INFO_DWARF4.
>>>>
>>>> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
>>>
>>> Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
>>> is not what I'd consider a fix. Someday we can move to
>>> DWARFv5...someday...
>>>
>>> What you describe sounds like build success, but reduction in debug info.
>>>
>>> The reports I'm referring to seem to result in a build failure.
>>>
>>
>> This is a strange one. The error in question
>>
>> CC .vmlinux.export.o
>> UPD include/generated/utsversion.h
>> CC init/version-timestamp.o
>> LD .tmp_vmlinux.btf
>> BTF .btf.vmlinux.bin.o
>> libbpf: BTF header not found
>> pahole: .tmp_vmlinux.btf: Invalid argument
> 
> That's slightly different from Tomasz and Marcus' report (not sure if
> that's relevant):
> 
> FAILED: load BTF from vmlinux: Invalid argument
> 
> That seems to come from
> tools/bpf/resolve_btfids/main.c:529
> Which seems like some failed call to btf_parse().
> EINVAL is getting propagated up from btf_parse(), but that's not super
> descriptive...
> 
Okay, that makes more sense. Basically the stage where we read vmlinux
BTF to do BTF id resolution (BTFIDS) is finding an empty BTF section.

> The hard part is that I suspect OpenMandriva (Tomasz) and Marcus are
> both setting additional flags in their toolchains, which can make
> reproducing tricky.
>

I tried falling back to the config referenced in the earlier bug report

https://github.com/ClangBuiltLinux/linux/files/10050200/config_bpf.txt

...but still couldn't reproduce it with LLVM 17 + pahole v1.24. That
config did specify DWARF5; if we can reproduce this, it would probably
be good to vary between forcing DWARF4 and DWARF5 to see if that is a
contributing factor as Arnaldo suggested.

Alan

>>
>> ...occurs during BTF parsing when the raw size of the BTF is smaller
>> than the BTF header size, which should never happen unless BTF
>> is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
>> we should be generating it from DWARF. The only time pahole parses BTF
>> is when it's creating split BTF for modules (it parses the base BTF), or
>> when it's reading existing BTF, neither of which it should be doing at
>> this stage.
>>
>> But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
>> Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
>> If that went awry somehow and .tmp_vmlinux.btf wasn't created, it
> 
> Wouldn't we expect some kind of linker error though in that case?
> 
>> would explain the "Invalid argument" error:
>>
>> $ pahole -J nosuchfile
>> pahole: nosuchfile: Invalid argument
>>
>> I see some clang specifics in vmlinux_link(), so I think a good
>> first step would be to check if .tmp_vlinux.btf exists prior
>> to running pahole. The submitter mentioned swapping linkers seems to
>> help, so that seems a promising angle. If there's a kernel .config
>> available I can try and reproduce the failure too. Thanks!
>>
>> Alan
>>
>>>>
>>>>   LD      .tmp_vmlinux.btf
>>>>   BTF     .btf.vmlinux.bin.o
>>>>   LD      .tmp_vmlinux.kallsyms1
>>>>
>>>> And
>>>>
>>>> / # strings /sys/kernel/btf/vmlinux | wc -l
>>>> 89921
>>>> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
>>>> kfree
>>>>
>>>> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
>>>> are about 30000.)
>>>
>>>
>>>
> 
> 
> 

Re: FAILED: load BTF from vmlinux: Invalid argument

[Jiri Olsa]

On Fri, Aug 04, 2023 at 11:03:03PM +0100, Alan Maguire wrote:
> On 04/08/2023 17:11, Nick Desaulniers wrote:
> > + Marcus (who also just reported seeing this
> > https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1664671027
> > and might be able to help reproduce).
> > + Fangrui (because seeing dd used as a result of 90ceddcb4950 makes me shudder)
> > 
> > On Thu, Aug 3, 2023 at 3:10 PM Alan Maguire <alan.maguire@oracle.com> wrote:
> >>
> >> On 03/08/2023 21:50, Nick Desaulniers wrote:
> >>> On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >>>>
> >>>> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> >>>>> Hi Martin (and BTF/BPF team),
> >>>>> I've observed 2 user reports with the error from the subject of this email.
> >>>>> https://github.com/ClangBuiltLinux/linux/issues/1825
> >>>>> https://bbs.archlinux.org/viewtopic.php?id=284177
> >>>>>
> >>>>> Any chance you could take a look at these reports and help us figure
> >>>>> out what's going wrong here?  Nathan and I haven't been able to
> >>>>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> >>>>>
> >>>>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
> >>>>
> >>>> Masami had a problem with new versions of compilers that was solved
> >>>> with:
> >>>>
> >>>> ------------------------ 8< --------------------------------------------
> >>>>> To check that please tweak:
> >>>>>
> >>>>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> >>>>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> >>>>> # CONFIG_DEBUG_INFO_DWARF4 is not set
> >>>>> # CONFIG_DEBUG_INFO_DWARF5 is not set
> >>>>> ⬢[acme@toolbox perf-tools-next]$
> >>>>>
> >>>>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> >>>>> CONFIG_DEBUG_INFO_DWARF4.
> >>>>
> >>>> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> >>>
> >>> Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> >>> is not what I'd consider a fix. Someday we can move to
> >>> DWARFv5...someday...
> >>>
> >>> What you describe sounds like build success, but reduction in debug info.
> >>>
> >>> The reports I'm referring to seem to result in a build failure.
> >>>
> >>
> >> This is a strange one. The error in question
> >>
> >> CC .vmlinux.export.o
> >> UPD include/generated/utsversion.h
> >> CC init/version-timestamp.o
> >> LD .tmp_vmlinux.btf
> >> BTF .btf.vmlinux.bin.o
> >> libbpf: BTF header not found
> >> pahole: .tmp_vmlinux.btf: Invalid argument
> > 
> > That's slightly different from Tomasz and Marcus' report (not sure if
> > that's relevant):
> > 
> > FAILED: load BTF from vmlinux: Invalid argument
> > 
> > That seems to come from
> > tools/bpf/resolve_btfids/main.c:529
> > Which seems like some failed call to btf_parse().
> > EINVAL is getting propagated up from btf_parse(), but that's not super
> > descriptive...
> > 
> Okay, that makes more sense. Basically the stage where we read vmlinux
> BTF to do BTF id resolution (BTFIDS) is finding an empty BTF section.

+1, looks like pahole failed to generate the BTF section, the BTFIDS
is just follow up error.. we might want to consider special error
output for missing BTF data ;-)

I can't reproduce this on my setup with either gcc or clang and trying
DWARF4/5 config options and latest and 1.24 pahole version

> 
> > The hard part is that I suspect OpenMandriva (Tomasz) and Marcus are
> > both setting additional flags in their toolchains, which can make
> > reproducing tricky.
> >
> 
> I tried falling back to the config referenced in the earlier bug report
> 
> https://github.com/ClangBuiltLinux/linux/files/10050200/config_bpf.txt

hum, I did not find this in the report.. are there more kernel configs
related to this issue? seems like more people hit this

thanks,
jirka

> 
> ...but still couldn't reproduce it with LLVM 17 + pahole v1.24. That
> config did specify DWARF5; if we can reproduce this, it would probably
> be good to vary between forcing DWARF4 and DWARF5 to see if that is a
> contributing factor as Arnaldo suggested.
> 
> Alan
> 
> >>
> >> ...occurs during BTF parsing when the raw size of the BTF is smaller
> >> than the BTF header size, which should never happen unless BTF
> >> is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
> >> we should be generating it from DWARF. The only time pahole parses BTF
> >> is when it's creating split BTF for modules (it parses the base BTF), or
> >> when it's reading existing BTF, neither of which it should be doing at
> >> this stage.
> >>
> >> But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
> >> Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
> >> If that went awry somehow and .tmp_vmlinux.btf wasn't created, it
> > 
> > Wouldn't we expect some kind of linker error though in that case?
> > 
> >> would explain the "Invalid argument" error:
> >>
> >> $ pahole -J nosuchfile
> >> pahole: nosuchfile: Invalid argument
> >>
> >> I see some clang specifics in vmlinux_link(), so I think a good
> >> first step would be to check if .tmp_vlinux.btf exists prior
> >> to running pahole. The submitter mentioned swapping linkers seems to
> >> help, so that seems a promising angle. If there's a kernel .config
> >> available I can try and reproduce the failure too. Thanks!
> >>
> >> Alan
> >>
> >>>>
> >>>>   LD      .tmp_vmlinux.btf
> >>>>   BTF     .btf.vmlinux.bin.o
> >>>>   LD      .tmp_vmlinux.kallsyms1
> >>>>
> >>>> And
> >>>>
> >>>> / # strings /sys/kernel/btf/vmlinux | wc -l
> >>>> 89921
> >>>> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> >>>> kfree
> >>>>
> >>>> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> >>>> are about 30000.)
> >>>
> >>>
> >>>
> > 
> > 
> > 
> 

Re: FAILED: load BTF from vmlinux: Invalid argument

[Nick Desaulniers]

On Sat, Aug 5, 2023 at 8:42 AM Jiri Olsa <olsajiri@gmail.com> wrote:
>
> On Fri, Aug 04, 2023 at 11:03:03PM +0100, Alan Maguire wrote:
> > On 04/08/2023 17:11, Nick Desaulniers wrote:
> > > + Marcus (who also just reported seeing this
> > > https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1664671027
> > > and might be able to help reproduce).
> > > + Fangrui (because seeing dd used as a result of 90ceddcb4950 makes me shudder)
> > >
> > > On Thu, Aug 3, 2023 at 3:10 PM Alan Maguire <alan.maguire@oracle.com> wrote:
> > >>
> > >> On 03/08/2023 21:50, Nick Desaulniers wrote:
> > >>> On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > >>>>
> > >>>> Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> > >>>>> Hi Martin (and BTF/BPF team),
> > >>>>> I've observed 2 user reports with the error from the subject of this email.
> > >>>>> https://github.com/ClangBuiltLinux/linux/issues/1825
> > >>>>> https://bbs.archlinux.org/viewtopic.php?id=284177
> > >>>>>
> > >>>>> Any chance you could take a look at these reports and help us figure
> > >>>>> out what's going wrong here?  Nathan and I haven't been able to
> > >>>>> reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> > >>>>>
> > >>>>> Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
> > >>>>
> > >>>> Masami had a problem with new versions of compilers that was solved
> > >>>> with:
> > >>>>
> > >>>> ------------------------ 8< --------------------------------------------
> > >>>>> To check that please tweak:
> > >>>>>
> > >>>>> ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> > >>>>> CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> > >>>>> # CONFIG_DEBUG_INFO_DWARF4 is not set
> > >>>>> # CONFIG_DEBUG_INFO_DWARF5 is not set
> > >>>>> ⬢[acme@toolbox perf-tools-next]$
> > >>>>>
> > >>>>> i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> > >>>>> CONFIG_DEBUG_INFO_DWARF4.
> > >>>>
> > >>>> Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> > >>>
> > >>> Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> > >>> is not what I'd consider a fix. Someday we can move to
> > >>> DWARFv5...someday...
> > >>>
> > >>> What you describe sounds like build success, but reduction in debug info.
> > >>>
> > >>> The reports I'm referring to seem to result in a build failure.
> > >>>
> > >>
> > >> This is a strange one. The error in question
> > >>
> > >> CC .vmlinux.export.o
> > >> UPD include/generated/utsversion.h
> > >> CC init/version-timestamp.o
> > >> LD .tmp_vmlinux.btf
> > >> BTF .btf.vmlinux.bin.o
> > >> libbpf: BTF header not found
> > >> pahole: .tmp_vmlinux.btf: Invalid argument
> > >
> > > That's slightly different from Tomasz and Marcus' report (not sure if
> > > that's relevant):
> > >
> > > FAILED: load BTF from vmlinux: Invalid argument
> > >
> > > That seems to come from
> > > tools/bpf/resolve_btfids/main.c:529
> > > Which seems like some failed call to btf_parse().
> > > EINVAL is getting propagated up from btf_parse(), but that's not super
> > > descriptive...
> > >
> > Okay, that makes more sense. Basically the stage where we read vmlinux
> > BTF to do BTF id resolution (BTFIDS) is finding an empty BTF section.
>
> +1, looks like pahole failed to generate the BTF section, the BTFIDS
> is just follow up error.. we might want to consider special error
> output for missing BTF data ;-)
>
> I can't reproduce this on my setup with either gcc or clang and trying
> DWARF4/5 config options and latest and 1.24 pahole version

Is it possible to link pahole against an older version of libbpf and
perhaps encounter some of the observed errors?
https://github.com/ClangBuiltLinux/linux/issues/1825#issuecomment-1491629904
mentions a specific revision of libbpf.

Tomasz and Marcus,
If you can still reproduce, please provide the versions of pahole AND
libbpf you're using. If you can no longer reproduce, let's close out
the github issue as obsolete.

>
> >
> > > The hard part is that I suspect OpenMandriva (Tomasz) and Marcus are
> > > both setting additional flags in their toolchains, which can make
> > > reproducing tricky.
> > >
> >
> > I tried falling back to the config referenced in the earlier bug report
> >
> > https://github.com/ClangBuiltLinux/linux/files/10050200/config_bpf.txt
>
> hum, I did not find this in the report.. are there more kernel configs
> related to this issue? seems like more people hit this
>
> thanks,
> jirka
>
> >
> > ...but still couldn't reproduce it with LLVM 17 + pahole v1.24. That
> > config did specify DWARF5; if we can reproduce this, it would probably
> > be good to vary between forcing DWARF4 and DWARF5 to see if that is a
> > contributing factor as Arnaldo suggested.
> >
> > Alan
> >
> > >>
> > >> ...occurs during BTF parsing when the raw size of the BTF is smaller
> > >> than the BTF header size, which should never happen unless BTF
> > >> is corrupted. Thing is, at that stage we shouldn't be parsing BTF,
> > >> we should be generating it from DWARF. The only time pahole parses BTF
> > >> is when it's creating split BTF for modules (it parses the base BTF), or
> > >> when it's reading existing BTF, neither of which it should be doing at
> > >> this stage.
> > >>
> > >> But I suspect the issue is in gen_btf() in scripts/link-vmlinux.sh.
> > >> Prior to running pahole, we call "vmlinux_link .tmp_vmlinux.btf".
> > >> If that went awry somehow and .tmp_vmlinux.btf wasn't created, it
> > >
> > > Wouldn't we expect some kind of linker error though in that case?
> > >
> > >> would explain the "Invalid argument" error:
> > >>
> > >> $ pahole -J nosuchfile
> > >> pahole: nosuchfile: Invalid argument
> > >>
> > >> I see some clang specifics in vmlinux_link(), so I think a good
> > >> first step would be to check if .tmp_vlinux.btf exists prior
> > >> to running pahole. The submitter mentioned swapping linkers seems to
> > >> help, so that seems a promising angle. If there's a kernel .config
> > >> available I can try and reproduce the failure too. Thanks!
> > >>
> > >> Alan
> > >>
> > >>>>
> > >>>>   LD      .tmp_vmlinux.btf
> > >>>>   BTF     .btf.vmlinux.bin.o
> > >>>>   LD      .tmp_vmlinux.kallsyms1
> > >>>>
> > >>>> And
> > >>>>
> > >>>> / # strings /sys/kernel/btf/vmlinux | wc -l
> > >>>> 89921
> > >>>> / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> > >>>> kfree
> > >>>>
> > >>>> It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> > >>>> are about 30000.)
> > >>>
> > >>>
> > >>>
> > >
> > >
> > >
> >



-- 
Thanks,
~Nick Desaulniers

Re: FAILED: load BTF from vmlinux: Invalid argument

[Masami Hiramatsu]

Hi Nick,

On Thu, 3 Aug 2023 13:50:02 -0700
Nick Desaulniers <ndesaulniers@google.com> wrote:

> On Thu, Aug 3, 2023 at 1:39 PM Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> >
> > Em Thu, Aug 03, 2023 at 11:02:46AM -0700, Nick Desaulniers escreveu:
> > > Hi Martin (and BTF/BPF team),
> > > I've observed 2 user reports with the error from the subject of this email.
> > > https://github.com/ClangBuiltLinux/linux/issues/1825
> > > https://bbs.archlinux.org/viewtopic.php?id=284177
> > >
> > > Any chance you could take a look at these reports and help us figure
> > > out what's going wrong here?  Nathan and I haven't been able to
> > > reproduce, but this seems to be affecting OpenMandriva (and Tomasz).
> > >
> > > Sounds like perhaps llvm-objcopy vs gnu objcopy might be a relevant detail?
> >
> > Masami had a problem with new versions of compilers that was solved
> > with:
> >
> > ------------------------ 8< --------------------------------------------
> > > To check that please tweak:
> > >
> > > ⬢[acme@toolbox perf-tools-next]$ grep DWARF ../build/v6.2-rc5+/.config
> > > CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
> > > # CONFIG_DEBUG_INFO_DWARF4 is not set
> > > # CONFIG_DEBUG_INFO_DWARF5 is not set
> > > ⬢[acme@toolbox perf-tools-next]$
> > >
> > > i.e. disable CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT and enable
> > > CONFIG_DEBUG_INFO_DWARF4.
> >
> > Hm, with CONFIG_DEBUG_INFO_DWARF4, no warning were shown.
> 
> Downgrading from the now-6-year-old DWARFv5 to now-13-year-old DWARFv4
> is not what I'd consider a fix. Someday we can move to
> DWARFv5...someday...

Actually, even with DWARFv5, if I use newer clang, pahole succeded to make
BTF info. So it may depend on compiler version too.

> 
> What you describe sounds like build success, but reduction in debug info.
> 
> The reports I'm referring to seem to result in a build failure.

Yeah, it seems a different issue. I could build it at least.

Thank you,

> 
> >
> >   LD      .tmp_vmlinux.btf
> >   BTF     .btf.vmlinux.bin.o
> >   LD      .tmp_vmlinux.kallsyms1
> >
> > And
> >
> > / # strings /sys/kernel/btf/vmlinux | wc -l
> > 89921
> > / # strings /sys/kernel/btf/vmlinux | grep -w kfree
> > kfree
> >
> > It seems the BTF is correctly generated. (with DWARF5, the number of symbols
> > are about 30000.)
> 
> 
> 
> -- 
> Thanks,
> ~Nick Desaulniers


-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>