Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Arnaldo Carvalho de Melo]

Em Sat, Aug 27, 2022 at 08:33:10PM +0300, Vitaly Chikunov escreveu:
> On Fri, Aug 26, 2022 at 01:37:24PM -0300, Arnaldo Carvalho de Melo wrote:
> > Em Fri, Aug 26, 2022 at 03:52:12PM +0200, Jiri Olsa escreveu:
> > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > >   [    0.767649] Run /init as init process
> > > > >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > >   [    0.771262] BPF:size=4 vlen=4
> > > > >   [    0.771511] BPF:
> > > > >   [    0.771680] BPF:Invalid btf_info kind_flag
> > > > >   [    0.772016] BPF:

> > > I can see the same on 5.15, it looks like the libbpf change that
> > > pahole is compiled with is setting the type's kflag for values < 0:
> > > (which is the case for perf_event_task_context enum first value)

> > >   dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API

> > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > verifier screams

> > > if I compile pahole with the libbpf change below I can boot 5.15 kernel
> > > normally

> > > Yonghong, any idea?

> > This made me try to build pahole with the system libbpf instead of with
> > the one that goes with it, here, testing with libbpf 0.7.0 it wasn't
> > building as BTF_KIND_ENUM64 came with libbpf 1.0 so I added the
> > following patch to again allow with the system libbpf, i.e. using:

> >   $ cmake -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=Off

> > This will return errors if trying to encode or load enum64 tags, but
> > disabling it, as done with kernels not supporting BTF_KIND_ENUM64 should
> > now work, can you please test and report results?

> > Vitaly I checked and alt:p9 has libbpf 0.2, which is really old, unsure
> > if it would build there, but alt:sisyphus has 0.8.0, so should work
> > there, please try.

> Perhaps this does not need to be tested since Jiri found the solution?

Right, what I suggested was just an alternative, to use the system
libbpf package instead of the one that ships with pahole, as the older
libbpf woudln't have the problem hypothesized by Jiri.

But it has already been determined that the problem is elsewhere, so
this just was useful to make me realize building with libbpf-devel was
broken due to BTF_KIND_ENUM64, which should be fixed with this patch.
 
> BTW, alt:p9 is older ALT branch only for security updates, current
> branch is alt:p10, so it's relevant to replace p9 testing with p10, for
> new features.

Sure, I go on adding containers and drop them when it is not possible to
build with them, so far this is what I have for alt:

   9    25.26 alt:p8       : Ok  x86_64-alt-linux-gcc (GCC) 5.3.1 20151207 (ALT p8 5.3.1-alt3.M80P.1)
  10    81.96 alt:p9       : Ok  x86_64-alt-linux-gcc (GCC) 8.4.1 20200305 (ALT p9 8.4.1-alt0.p9.1) , clang version 10.0.0
  11    94.60 alt:p10      : Ok  x86_64-alt-linux-gcc (GCC) 10.3.1 20210703 (ALT Sisyphus 10.3.1-alt2) , clang version 11.0.1
  12    77.53 alt:sisyphus : Ok  x86_64-alt-linux-gcc (GCC) 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt1) , ALT Linux Team clang version 13.0.1

p8's clang is too old, disabled, p9 also has:

ENV NO_BUILD_BPF_SKEL=1

clang also too old to build BPF skels, but still good to build perf
proper.

> I was building dwarves with -DLIBBPF_EMBEDDED=ON, thinking it would be
> more stable (since they updated not independently from each other), is
> it recommended to turn in OFF?

What most distros are doing is to ship a libbpf and they expect and work
to make tools use it, which isn't always possible, like the patch I sent
shows.

Its a balance on having the latest and greatest libbpf that is needed
for some feature pahole uses and the desire to not link anything
statically due to security concerns (having to fix all things linking
statically with a faulty library).
 
> ps. I think something needs to be changed somewhere (in kernel src?) or
> more users will report build failures when switching to the new pahole.

Right, I voiced concerns already about having enum64 encoded by default,
but its not so simple to solve.

People living on the bleeding edge should not have problems, as recent
kernels support the new BTF encodings, its just people building older
kernels with new pahole that will have problems, right?

By test building against alt:p8 you see that I care about such version
mismatches, but its a burden, really, to try and cover all cases.

- Arnaldo

Re: pahole v1.24: FAILED: load BTF from vmlinux: Invalid argument

[Vitaly Chikunov]

On Sat, Aug 27, 2022 at 03:33:53PM -0300, Arnaldo Carvalho de Melo wrote:
> Em Sat, Aug 27, 2022 at 08:33:10PM +0300, Vitaly Chikunov escreveu:
> > On Fri, Aug 26, 2022 at 01:37:24PM -0300, Arnaldo Carvalho de Melo wrote:
> > > Em Fri, Aug 26, 2022 at 03:52:12PM +0200, Jiri Olsa escreveu:
> > > > On Fri, Aug 26, 2022 at 05:59:44AM +0300, Vitaly Chikunov wrote:
> > > > > >   [    0.767649] Run /init as init process
> > > > > >   [    0.770858] BPF:[593] ENUM perf_event_task_context
> > > > > >   [    0.771262] BPF:size=4 vlen=4
> > > > > >   [    0.771511] BPF:
> > > > > >   [    0.771680] BPF:Invalid btf_info kind_flag
> > > > > >   [    0.772016] BPF:
> 
> > > > I can see the same on 5.15, it looks like the libbpf change that
> > > > pahole is compiled with is setting the type's kflag for values < 0:
> > > > (which is the case for perf_event_task_context enum first value)
> 
> > > >   dffbbdc2d988 libbpf: Add enum64 parsing and new enum64 public API
> 
> > > > but IIUC kflag should stay zero for normal enum otherwise the btf meta
> > > > verifier screams
> 
> > > > if I compile pahole with the libbpf change below I can boot 5.15 kernel
> > > > normally
> 
> > > > Yonghong, any idea?
> 
> > > This made me try to build pahole with the system libbpf instead of with
> > > the one that goes with it, here, testing with libbpf 0.7.0 it wasn't
> > > building as BTF_KIND_ENUM64 came with libbpf 1.0 so I added the
> > > following patch to again allow with the system libbpf, i.e. using:
> 
> > >   $ cmake -DCMAKE_BUILD_TYPE=Release -DLIBBPF_EMBEDDED=Off
> 
> > > This will return errors if trying to encode or load enum64 tags, but
> > > disabling it, as done with kernels not supporting BTF_KIND_ENUM64 should
> > > now work, can you please test and report results?
> 
> > > Vitaly I checked and alt:p9 has libbpf 0.2, which is really old, unsure
> > > if it would build there, but alt:sisyphus has 0.8.0, so should work
> > > there, please try.
> 
> > Perhaps this does not need to be tested since Jiri found the solution?
> 
> Right, what I suggested was just an alternative, to use the system
> libbpf package instead of the one that ships with pahole, as the older
> libbpf woudln't have the problem hypothesized by Jiri.
> 
> But it has already been determined that the problem is elsewhere, so
> this just was useful to make me realize building with libbpf-devel was
> broken due to BTF_KIND_ENUM64, which should be fixed with this patch.
>  
> > BTW, alt:p9 is older ALT branch only for security updates, current
> > branch is alt:p10, so it's relevant to replace p9 testing with p10, for
> > new features.
> 
> Sure, I go on adding containers and drop them when it is not possible to
> build with them, so far this is what I have for alt:
> 
>    9    25.26 alt:p8       : Ok  x86_64-alt-linux-gcc (GCC) 5.3.1 20151207 (ALT p8 5.3.1-alt3.M80P.1)
>   10    81.96 alt:p9       : Ok  x86_64-alt-linux-gcc (GCC) 8.4.1 20200305 (ALT p9 8.4.1-alt0.p9.1) , clang version 10.0.0
>   11    94.60 alt:p10      : Ok  x86_64-alt-linux-gcc (GCC) 10.3.1 20210703 (ALT Sisyphus 10.3.1-alt2) , clang version 11.0.1
>   12    77.53 alt:sisyphus : Ok  x86_64-alt-linux-gcc (GCC) 12.1.1 20220518 (ALT Sisyphus 12.1.1-alt1) , ALT Linux Team clang version 13.0.1

Btw, we also have clang14.0 in sisyphus, it's just non default yet.

> 
> p8's clang is too old, disabled, p9 also has:
> 
> ENV NO_BUILD_BPF_SKEL=1
> 
> clang also too old to build BPF skels, but still good to build perf
> proper.
> 
> > I was building dwarves with -DLIBBPF_EMBEDDED=ON, thinking it would be
> > more stable (since they updated not independently from each other), is
> > it recommended to turn in OFF?
> 
> What most distros are doing is to ship a libbpf and they expect and work
> to make tools use it, which isn't always possible, like the patch I sent
> shows.
> 
> Its a balance on having the latest and greatest libbpf that is needed
> for some feature pahole uses and the desire to not link anything
> statically due to security concerns (having to fix all things linking
> statically with a faulty library).
>  
> > ps. I think something needs to be changed somewhere (in kernel src?) or
> > more users will report build failures when switching to the new pahole.
> 
> Right, I voiced concerns already about having enum64 encoded by default,
> but its not so simple to solve.
> 
> People living on the bleeding edge should not have problems, as recent
> kernels support the new BTF encodings, its just people building older
> kernels with new pahole that will have problems, right?

Yes, but even in Sisyphus (where we try to build bleeding edge stuff) we
build latest stable and longterm kernels, that's why we happen to build
v5.18.19 and v5.15.63 with dwarves 1.24. And 5.15.63 is released just two
days ago so this is not that old.

> By test building against alt:p8 you see that I care about such version
> mismatches, but its a burden, really, to try and cover all cases.

I very much appreciate your work.
Thanks!

> 
> - Arnaldo