Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov <vt@altlinux.org>]

Jiri,

On Fri, Aug 26, 2022 at 10:19:55PM +0200, Jiri Olsa wrote:
> On Fri, Aug 26, 2022 at 11:53:34AM -0700, Yonghong Song wrote:
> > On 8/26/22 10:01 AM, Jiri Olsa wrote:
> > > On Fri, Aug 26, 2022 at 09:51:54AM -0700, Yonghong Song wrote:
> > > > On 8/26/22 6:52 AM, Jiri Olsa wrote:
> > > > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > > > On Fri, Aug 26, 2022 at 05:52:20AM +0300, Vitaly Chikunov wrote:
> > > > > > > Arnaldo,
> > > > > > > 
> > > > > > > On Thu, Aug 25, 2022 at 08:16:20PM +0300, Vitaly Chikunov wrote:
> > > > > > > > On Thu, Aug 25, 2022 at 01:47:59PM -0300, Arnaldo Carvalho de Melo wrote:
> > > > > > > > > On Thu, Aug 25, 2022, 1:35 PM Vitaly Chikunov <vt@altlinux.org> wrote:
> > > > > > > > > > 
> > > > > > > > > > I also noticed that after upgrading pahole to v1.24 kernel build (tested on
> > > > > > > > > > v5.18.19, v5.15.63, sorry for not testing on mainline) fails with:
> > > > > > > > > > 
> > > > > > > > > >       BTFIDS  vmlinux
> > > > > > > > > >     + ./tools/bpf/resolve_btfids/resolve_btfids vmlinux
> > > > > > > > > >     FAILED: load BTF from vmlinux: Invalid argument
> > > > > > > > > > 
> > > > > > > > > > Perhaps, .tmp_vmlinux.btf is generated incorrectly? Downgrading dwarves to
> > > > > > > > > > v1.23 resolves the issue.
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Can you try this, from Martin Reboredo (Archlinux):
> > > > > > > > > 
> > > > > > > > > Can you try a build of the kernel or the by passing the
> > > > > > > > > --skip_encoding_btf_enum64 to scripts/pahole-flags.sh?
> > > > > > > > > 
> > > > > > > > > Here's a patch for either in tree scripts/pahole-flags.sh or
> > > > > > > > > /usr/lib/modules/5.19.3-arch1-1/build/scripts/pahole-flags.sh
> > > > > > > > 
> > > > > > > > This patch helped and kernel builds successfully after applying it.
> > > > > > > > (Didn't notice this suggestion in release discussion thread.)
> > > > > > > 
> > > > > > > Even thought it now compiles with this patch, it does not boot
> > > > > > > afterwards (in virtme-like env), witch such console messages:
> > > > > > 
> > > > > > I'm talking here about 5.15.62. Yes, proposed patch does not apply there
> > > > > > (since there is no `scripts/pahole-flags.sh`), but I updated
> > > > > > `scripts/link-vmlinux.sh` with the similar `if` to append
> > > > > > `--skip_encoding_btf_enum64` which lets then compilation pass.
> > > > > > 
> > > > > > Thanks,
> > > > > > 
> > > > > > > 
> > > > > > >     [    0.767649] Run /init as init process
> > > > > > >     [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > > > >     [    0.771262] BPF:size=4 vlen=4
> > > > > > >     [    0.771511] BPF:
> > > > > > >     [    0.771680] BPF:Invalid btf_info kind_flag
> > > > > > >     [    0.772016] BPF:
> > > > > 
> > > > > I can see the same on 5.15, it looks like the libbpf change that
> > > > > pahole is compiled with is setting the type's kflag for values < 0:
> > > > > (which is the case for perf_event_task_context enum first value)
> > > > > 
> > > > >     dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> > > > > 
> > > > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > > > verifier screams
> > > > 
> > > > This is deliberate so we can have sign bit set properly for 32bit enum.
> > > > To avoid this behavior, the correct way is to turn off enum64 support
> > > > in pahole with flag --skip_encoding_btf_enum64.
> > > 
> > > I used that as well, it wouldn't compile without
> > > 
> > > the error is during the boot where the standard enum has kflag set
> > 
> > I just tried latest bpf-next, using pahole 1.24, with and without
> > --skip_encoding_btf_enum64. The following are BTF encoding
> > for enum perf_event_task_context.
> > 
> > enum perf_event_task_context {
> >         perf_invalid_context = -1,
> >         perf_hw_context = 0,
> >         perf_sw_context,
> >         perf_nr_task_contexts,
> > };
> > 
> > With --skip_encoding_btf_enum64:
> > [2285] ENUM 'perf_event_task_context' encoding=UNSIGNED size=4 vlen=4
> >         'perf_invalid_context' val=4294967295
> >         'perf_hw_context' val=0
> >         'perf_sw_context' val=1
> >         'perf_nr_task_contexts' val=2
> > 
> > Without --skip_encoding_btf_enum64:
> > [3786] ENUM 'perf_event_task_context' encoding=SIGNED size=4 vlen=4
> >         'perf_invalid_context' val=-1
> >         'perf_hw_context' val=0
> >         'perf_sw_context' val=1
> >         'perf_nr_task_contexts' val=2
> > 
> > encoding SIGNED means kflag = 1 and UNSIGNED is the default meaning
> > kflag = 0. So it looks okay to me. Could you try to use latest
> > bpftool to dump vmlinux BTF for your vmlinux binary?
> > 
> > Regarding the corresponding pahole enum64 support, we have
> > the following code,
> > 
> >         if (conf_load->skip_encoding_btf_enum64)
> >                 err = btf__add_enum_value(encoder->btf, name,
> > (uint32_t)value);
> >         else if (etype->size > 32)
> >                 err = btf__add_enum64_value(encoder->btf, name, value);
> >         else
> >                 err = btf__add_enum_value(encoder->btf, name, value);
> > 
> > If skip_encoding_btf_enum64 is enabled, the value will be passed
> > as '(uint32_t)value', so '__s64 value' in the parameter should be
> > a unsigned value and 'if (value < 0) ...' should not be
> > triggered if skip_encoding_btf_enum64 is enabled.
> > 
> > Jiri, could you double check your build environment?
> 
> ah right.. it's the build problem, 5.15 is missing backport of
>   9741e07ece7c kbuild: Unify options for BTF generation for vmlinux and modules
> 
> so modules BTF build does not pick up the --skip_encoding_btf_enum64,
> with the 5.15.61 extra change below I can boot the kernel properly
> 
> sorry for the noise
> 
> Vitaly, could you please try with the change below?

I just tested and after this additional change v5.15.63 kernel
started to boot successfully.

Thanks!

> 
> thanks,
> jirka
> 
> 
> ---
> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> index ff805777431c..47717e09000b 100644
> --- a/scripts/Makefile.modfinal
> +++ b/scripts/Makefile.modfinal
> @@ -40,7 +40,7 @@ quiet_cmd_ld_ko_o = LD [M]  $@
>  quiet_cmd_btf_ko = BTF [M] $@
>        cmd_btf_ko = 							\
>  	if [ -f vmlinux ]; then						\
> -		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --btf_base vmlinux $@; \
> +		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J --skip_encoding_btf_enum64 --btf_base vmlinux $@; \
>  	else								\
>  		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
>  	fi;