Re: kernelci failures due to "pahole" missing

["Denys Fedoryshchenko" <denys.f@collabora.com>]

On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io wrote:
> On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> denys.f@collabora.com>
> wrote:
> 
> > Update
> > 
> > We identified all packages required for kernel build:
> > dwarves, lz4, python3/python.
> > 
> > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our
> > staging
> > pipeline, but with error:
> >  LD      vmlinux
> >   BTFIDS  vmlinux
> > FAILED unresolved symbol xdp_buff
> > 
> 
> Which config and arch was this?
gki_defconfig
arm64
But it is a bit old snapshot i'm using as reference.
Tomorrow i will try latest 5.10 if anything changed.

Container Log:
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux
FAILED unresolved symbol xdp_buff
make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
make[1]: Leaving directory '/tmp/kci/linux/build'
make: *** [Makefile:185: __sub-make] Error 2
make: Leaving directory '/tmp/kci/linux'

"branch": "staging-android",
"commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
"describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
"describe_verbose": "v5.10-19504-g4dce9d7a65b1",
"tree": "kernelci",
"url": "https://github.com/kernelci/linux.git"

> 
> 
> > 
> > I will create PR now to allocate larger builders (to address LTO
> > memory
> > overconsumption) and we will do test build on android pipeline.
> > 
> 
> Thanks for helping with this!
> 
> 
> > 
> > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> > > Short update.
> > > Some of builders crashed due OOM, but not because of logs,
> > > but because LTO during linking phase consume more than 21Gb of
> > > RAM,
> > > and k8s builders allocated for project have only 4Gb allocated.
> > > It is not easy to solve, as it is either require to change LTO to
> > > "Thin
> > > LTO" (or disable it totally) to reduce memory consumption, which
> > > is
> > > not
> > > great, or increase builders size (and that means additional costs
> > > and
> > > approvals related to that).
> > > We are discussing on IRC possible solutions to this problem.
> > > 
> > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko wrote:
> > > > At the moment I have identified a problem with more packages
> > > > that
> > > > are
> > > > missing, now it is python, and i'm adding it as well, but need
> > > > to
> > > > test
> > > > everything manually, and then on staging.
> > > > 
> > > > Here is example of build that needs python:
> > > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > 
> > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > 
> > > > Also, another problem some of the builders crashed, probably
> > > > due to
> > > > OOM
> > > > or insufficient resources.
> > > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > 
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > 
> > > > On manual tests, builds with pahole create about 819Mbyte of
> > > > warnings,
> > > > such as:
> > > > "namespace__recode_dwarf_types: couldn't find 0x13423 type for
> > > > 0x3d6a5
> > > > (member)!"
> > > > I am currently investigating whether these issues are related.
> > > > Therefore, it takes a bit more time than i expected, than just
> > > > adding
> > > > a
> > > > pahole/dwarves package.
> > > > 
> > > > I will update as soon as more information available.
> > > > 
> > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > Denys, Thanks for the update. Do you have a prediction for
> > > > > when
> > > > > it
> > > > > will land?
> > > > > 
> > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > <denys.f@collabora.com> wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > We are testing in staging fix for that:
> > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > 
> > > > > > Also i noticed in some cases lz4 package required for this
> > > > > > test,
> > > > > > so added it too.
> > > > > > 
> > > > > > Best regards,
> > > > > > Denys
> > > > > > 
> > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io
> > > > > > wrote:
> > > > > > > We have a bunch of Android kernel test failures due to
> > > > > > > missing
> > > > > > > "pahole" in
> > > > > > > the build environment (example android13-5.15
> > > > > > > gki_defconfig
> > > > > > > 
> > > > > > <
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > 
> > > > > > > ):
> > > > > > > 
> > > > > > > 
> > > > > > >   LD      vmlinux.o
> > > > > > >   MODPOST vmlinux.symvers
> > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > >   GEN     modules.builtin
> > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > > > > > Failed to generate BTF for vmlinux
> > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux]
> > > > > > > Error 1
> > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > 
> > > > > > > Would it be possible to get pahole added to the build
> > > > > > environments
> > > > > > > used to
> > > > > > > build android kernels?
> > > > > > > 
> > > > > > > -Todd
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > 
> > 
> > 
> 
> 
> 
> 
>