kernelci failures due to "pahole" missing

kernelci failures due to "pahole" missing

[Todd Kjos]

We have a bunch of Android kernel test failures due to missing "pahole" in
the build environment (example android13-5.15 gki_defconfig
<https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log>
):


  LD      vmlinux.o
  MODPOST vmlinux.symvers
  MODINFO modules.builtin.modinfo
  GEN     modules.builtin
BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF
make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
make[1]: Leaving directory '/tmp/kci/linux/build'
make: *** [Makefile:237: __sub-make] Error 2
make: Leaving directory '/tmp/kci/linux'

Would it be possible to get pahole added to the build environments used to
build android kernels?

-Todd

Re: kernelci failures due to "pahole" missing

[denys.f]

Hi, 

We are testing in staging fix for that:
https://github.com/kernelci/kernelci-core/pull/1158

Also i noticed in some cases lz4 package required for this test,
so added it too.

Best regards,
Denys

On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io wrote:
> We have a bunch of Android kernel test failures due to missing
> "pahole" in
> the build environment (example android13-5.15 gki_defconfig
> <https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> >
> ):
> 
> 
>   LD      vmlinux.o
>   MODPOST vmlinux.symvers
>   MODINFO modules.builtin.modinfo
>   GEN     modules.builtin
> BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> Failed to generate BTF for vmlinux
> Try to disable CONFIG_DEBUG_INFO_BTF
> make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
> make[1]: Leaving directory '/tmp/kci/linux/build'
> make: *** [Makefile:237: __sub-make] Error 2
> make: Leaving directory '/tmp/kci/linux'
> 
> Would it be possible to get pahole added to the build environments
> used to
> build android kernels?
> 
> -Todd
> 
> 
> 
> 
> 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

Denys, Thanks for the update. Do you have a prediction for when it will
land?

On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> Hi,
>
> We are testing in staging fix for that:
> https://github.com/kernelci/kernelci-core/pull/1158
>
> Also i noticed in some cases lz4 package required for this test,
> so added it too.
>
> Best regards,
> Denys
>
> On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io wrote:
> > We have a bunch of Android kernel test failures due to missing
> > "pahole" in
> > the build environment (example android13-5.15 gki_defconfig
> > <
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > >
> > ):
> >
> >
> >   LD      vmlinux.o
> >   MODPOST vmlinux.symvers
> >   MODINFO modules.builtin.modinfo
> >   GEN     modules.builtin
> > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > Failed to generate BTF for vmlinux
> > Try to disable CONFIG_DEBUG_INFO_BTF
> > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
> > make[1]: Leaving directory '/tmp/kci/linux/build'
> > make: *** [Makefile:237: __sub-make] Error 2
> > make: Leaving directory '/tmp/kci/linux'
> >
> > Would it be possible to get pahole added to the build environments
> > used to
> > build android kernels?
> >
> > -Todd
> >
> >
> > 
> >
> >
>
>
>

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

At the moment I have identified a problem with more packages that are
missing, now it is python, and i'm adding it as well, but need to test
everything manually, and then on staging.

Here is example of build that needs python:
https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log

Also, another problem some of the builders crashed, probably due to OOM
or insufficient resources. 
https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log

On manual tests, builds with pahole create about 819Mbyte of warnings,
such as:
"namespace__recode_dwarf_types: couldn't find 0x13423 type for 0x3d6a5
(member)!"
I am currently investigating whether these issues are related. 
Therefore, it takes a bit more time than i expected, than just adding a
pahole/dwarves package.

I will update as soon as more information available.

On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> Denys, Thanks for the update. Do you have a prediction for when it
> will land?
> 
> On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> <denys.f@collabora.com> wrote:
> > Hi, 
> > 
> > We are testing in staging fix for that:
> > https://github.com/kernelci/kernelci-core/pull/1158
> > 
> > Also i noticed in some cases lz4 package required for this test,
> > so added it too.
> > 
> > Best regards,
> > Denys
> > 
> > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io wrote:
> > > We have a bunch of Android kernel test failures due to missing
> > > "pahole" in
> > > the build environment (example android13-5.15 gki_defconfig
> > > 
> > <https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > 
> > > ):
> > > 
> > > 
> > >   LD      vmlinux.o
> > >   MODPOST vmlinux.symvers
> > >   MODINFO modules.builtin.modinfo
> > >   GEN     modules.builtin
> > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > Failed to generate BTF for vmlinux
> > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
> > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > make: *** [Makefile:237: __sub-make] Error 2
> > > make: Leaving directory '/tmp/kci/linux'
> > > 
> > > Would it be possible to get pahole added to the build
> > environments
> > > used to
> > > build android kernels?
> > > 
> > > -Todd
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

Short update.
Some of builders crashed due OOM, but not because of logs,
but because LTO during linking phase consume more than 21Gb of RAM,
and k8s builders allocated for project have only 4Gb allocated.
It is not easy to solve, as it is either require to change LTO to "Thin
LTO" (or disable it totally) to reduce memory consumption, which is not
great, or increase builders size (and that means additional costs and
approvals related to that).
We are discussing on IRC possible solutions to this problem.

On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko wrote:
> At the moment I have identified a problem with more packages that are
> missing, now it is python, and i'm adding it as well, but need to
> test
> everything manually, and then on staging.
> 
> Here is example of build that needs python:
> https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> 
> Also, another problem some of the builders crashed, probably due to
> OOM
> or insufficient resources. 
> https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> 
> On manual tests, builds with pahole create about 819Mbyte of
> warnings,
> such as:
> "namespace__recode_dwarf_types: couldn't find 0x13423 type for
> 0x3d6a5
> (member)!"
> I am currently investigating whether these issues are related. 
> Therefore, it takes a bit more time than i expected, than just adding
> a
> pahole/dwarves package.
> 
> I will update as soon as more information available.
> 
> On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > Denys, Thanks for the update. Do you have a prediction for when it
> > will land?
> > 
> > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > <denys.f@collabora.com> wrote:
> > > Hi, 
> > > 
> > > We are testing in staging fix for that:
> > > https://github.com/kernelci/kernelci-core/pull/1158
> > > 
> > > Also i noticed in some cases lz4 package required for this test,
> > > so added it too.
> > > 
> > > Best regards,
> > > Denys
> > > 
> > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io wrote:
> > > > We have a bunch of Android kernel test failures due to missing
> > > > "pahole" in
> > > > the build environment (example android13-5.15 gki_defconfig
> > > > 
> > > <https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > 
> > > > ):
> > > > 
> > > > 
> > > >   LD      vmlinux.o
> > > >   MODPOST vmlinux.symvers
> > > >   MODINFO modules.builtin.modinfo
> > > >   GEN     modules.builtin
> > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > > Failed to generate BTF for vmlinux
> > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
> > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > make: Leaving directory '/tmp/kci/linux'
> > > > 
> > > > Would it be possible to get pahole added to the build
> > > environments
> > > > used to
> > > > build android kernels?
> > > > 
> > > > -Todd
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> 
> 
> 
> 
> 
> 
> 

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

Update

We identified all packages required for kernel build:
dwarves, lz4, python3/python.

We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our staging
pipeline, but with error:
 LD      vmlinux
  BTFIDS  vmlinux
FAILED unresolved symbol xdp_buff

I will create PR now to allocate larger builders (to address LTO memory
overconsumption) and we will do test build on android pipeline.

On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> Short update.
> Some of builders crashed due OOM, but not because of logs,
> but because LTO during linking phase consume more than 21Gb of RAM,
> and k8s builders allocated for project have only 4Gb allocated.
> It is not easy to solve, as it is either require to change LTO to
> "Thin
> LTO" (or disable it totally) to reduce memory consumption, which is
> not
> great, or increase builders size (and that means additional costs and
> approvals related to that).
> We are discussing on IRC possible solutions to this problem.
> 
> On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko wrote:
> > At the moment I have identified a problem with more packages that
> > are
> > missing, now it is python, and i'm adding it as well, but need to
> > test
> > everything manually, and then on staging.
> > 
> > Here is example of build that needs python:
> > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > 
> > Also, another problem some of the builders crashed, probably due to
> > OOM
> > or insufficient resources. 
> > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > 
> > On manual tests, builds with pahole create about 819Mbyte of
> > warnings,
> > such as:
> > "namespace__recode_dwarf_types: couldn't find 0x13423 type for
> > 0x3d6a5
> > (member)!"
> > I am currently investigating whether these issues are related. 
> > Therefore, it takes a bit more time than i expected, than just
> > adding
> > a
> > pahole/dwarves package.
> > 
> > I will update as soon as more information available.
> > 
> > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > Denys, Thanks for the update. Do you have a prediction for when
> > > it
> > > will land?
> > > 
> > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > <denys.f@collabora.com> wrote:
> > > > Hi, 
> > > > 
> > > > We are testing in staging fix for that:
> > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > 
> > > > Also i noticed in some cases lz4 package required for this
> > > > test,
> > > > so added it too.
> > > > 
> > > > Best regards,
> > > > Denys
> > > > 
> > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io
> > > > wrote:
> > > > > We have a bunch of Android kernel test failures due to
> > > > > missing
> > > > > "pahole" in
> > > > > the build environment (example android13-5.15 gki_defconfig
> > > > > 
> > > > <https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > 
> > > > > ):
> > > > > 
> > > > > 
> > > > >   LD      vmlinux.o
> > > > >   MODPOST vmlinux.symvers
> > > > >   MODINFO modules.builtin.modinfo
> > > > >   GEN     modules.builtin
> > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > > > Failed to generate BTF for vmlinux
> > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
> > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > 
> > > > > Would it be possible to get pahole added to the build
> > > > environments
> > > > > used to
> > > > > build android kernels?
> > > > > 
> > > > > -Todd
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> Update
>
> We identified all packages required for kernel build:
> dwarves, lz4, python3/python.
>
> We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our staging
> pipeline, but with error:
>  LD      vmlinux
>   BTFIDS  vmlinux
> FAILED unresolved symbol xdp_buff
>

Which config and arch was this?


>
> I will create PR now to allocate larger builders (to address LTO memory
> overconsumption) and we will do test build on android pipeline.
>

Thanks for helping with this!


>
> On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> > Short update.
> > Some of builders crashed due OOM, but not because of logs,
> > but because LTO during linking phase consume more than 21Gb of RAM,
> > and k8s builders allocated for project have only 4Gb allocated.
> > It is not easy to solve, as it is either require to change LTO to
> > "Thin
> > LTO" (or disable it totally) to reduce memory consumption, which is
> > not
> > great, or increase builders size (and that means additional costs and
> > approvals related to that).
> > We are discussing on IRC possible solutions to this problem.
> >
> > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko wrote:
> > > At the moment I have identified a problem with more packages that
> > > are
> > > missing, now it is python, and i'm adding it as well, but need to
> > > test
> > > everything manually, and then on staging.
> > >
> > > Here is example of build that needs python:
> > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > >
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > >
> > > Also, another problem some of the builders crashed, probably due to
> > > OOM
> > > or insufficient resources.
> > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > >
> > > On manual tests, builds with pahole create about 819Mbyte of
> > > warnings,
> > > such as:
> > > "namespace__recode_dwarf_types: couldn't find 0x13423 type for
> > > 0x3d6a5
> > > (member)!"
> > > I am currently investigating whether these issues are related.
> > > Therefore, it takes a bit more time than i expected, than just
> > > adding
> > > a
> > > pahole/dwarves package.
> > >
> > > I will update as soon as more information available.
> > >
> > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > Denys, Thanks for the update. Do you have a prediction for when
> > > > it
> > > > will land?
> > > >
> > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > <denys.f@collabora.com> wrote:
> > > > > Hi,
> > > > >
> > > > > We are testing in staging fix for that:
> > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > >
> > > > > Also i noticed in some cases lz4 package required for this
> > > > > test,
> > > > > so added it too.
> > > > >
> > > > > Best regards,
> > > > > Denys
> > > > >
> > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io
> > > > > wrote:
> > > > > > We have a bunch of Android kernel test failures due to
> > > > > > missing
> > > > > > "pahole" in
> > > > > > the build environment (example android13-5.15 gki_defconfig
> > > > > >
> > > > > <
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > >
> > > > > > ):
> > > > > >
> > > > > >
> > > > > >   LD      vmlinux.o
> > > > > >   MODPOST vmlinux.symvers
> > > > > >   MODINFO modules.builtin.modinfo
> > > > > >   GEN     modules.builtin
> > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > > > > Failed to generate BTF for vmlinux
> > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux] Error 1
> > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > >
> > > > > > Would it be possible to get pahole added to the build
> > > > > environments
> > > > > > used to
> > > > > > build android kernels?
> > > > > >
> > > > > > -Todd
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
> > > 
> > >
> > >
> >
>
>
>

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io wrote:
> On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> denys.f@collabora.com>
> wrote:
> 
> > Update
> > 
> > We identified all packages required for kernel build:
> > dwarves, lz4, python3/python.
> > 
> > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our
> > staging
> > pipeline, but with error:
> >  LD      vmlinux
> >   BTFIDS  vmlinux
> > FAILED unresolved symbol xdp_buff
> > 
> 
> Which config and arch was this?
gki_defconfig
arm64
But it is a bit old snapshot i'm using as reference.
Tomorrow i will try latest 5.10 if anything changed.

Container Log:
  LD      .tmp_vmlinux.kallsyms1
  KSYMS   .tmp_vmlinux.kallsyms1.S
  AS      .tmp_vmlinux.kallsyms1.S
  LD      .tmp_vmlinux.kallsyms2
  KSYMS   .tmp_vmlinux.kallsyms2.S
  AS      .tmp_vmlinux.kallsyms2.S
  LD      vmlinux
  BTFIDS  vmlinux
FAILED unresolved symbol xdp_buff
make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
make[1]: Leaving directory '/tmp/kci/linux/build'
make: *** [Makefile:185: __sub-make] Error 2
make: Leaving directory '/tmp/kci/linux'

"branch": "staging-android",
"commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
"describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
"describe_verbose": "v5.10-19504-g4dce9d7a65b1",
"tree": "kernelci",
"url": "https://github.com/kernelci/linux.git"

> 
> 
> > 
> > I will create PR now to allocate larger builders (to address LTO
> > memory
> > overconsumption) and we will do test build on android pipeline.
> > 
> 
> Thanks for helping with this!
> 
> 
> > 
> > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> > > Short update.
> > > Some of builders crashed due OOM, but not because of logs,
> > > but because LTO during linking phase consume more than 21Gb of
> > > RAM,
> > > and k8s builders allocated for project have only 4Gb allocated.
> > > It is not easy to solve, as it is either require to change LTO to
> > > "Thin
> > > LTO" (or disable it totally) to reduce memory consumption, which
> > > is
> > > not
> > > great, or increase builders size (and that means additional costs
> > > and
> > > approvals related to that).
> > > We are discussing on IRC possible solutions to this problem.
> > > 
> > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko wrote:
> > > > At the moment I have identified a problem with more packages
> > > > that
> > > > are
> > > > missing, now it is python, and i'm adding it as well, but need
> > > > to
> > > > test
> > > > everything manually, and then on staging.
> > > > 
> > > > Here is example of build that needs python:
> > > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > 
> > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > 
> > > > Also, another problem some of the builders crashed, probably
> > > > due to
> > > > OOM
> > > > or insufficient resources.
> > > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > 
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > 
> > > > On manual tests, builds with pahole create about 819Mbyte of
> > > > warnings,
> > > > such as:
> > > > "namespace__recode_dwarf_types: couldn't find 0x13423 type for
> > > > 0x3d6a5
> > > > (member)!"
> > > > I am currently investigating whether these issues are related.
> > > > Therefore, it takes a bit more time than i expected, than just
> > > > adding
> > > > a
> > > > pahole/dwarves package.
> > > > 
> > > > I will update as soon as more information available.
> > > > 
> > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > Denys, Thanks for the update. Do you have a prediction for
> > > > > when
> > > > > it
> > > > > will land?
> > > > > 
> > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > <denys.f@collabora.com> wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > We are testing in staging fix for that:
> > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > 
> > > > > > Also i noticed in some cases lz4 package required for this
> > > > > > test,
> > > > > > so added it too.
> > > > > > 
> > > > > > Best regards,
> > > > > > Denys
> > > > > > 
> > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io
> > > > > > wrote:
> > > > > > > We have a bunch of Android kernel test failures due to
> > > > > > > missing
> > > > > > > "pahole" in
> > > > > > > the build environment (example android13-5.15
> > > > > > > gki_defconfig
> > > > > > > 
> > > > > > <
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > 
> > > > > > > ):
> > > > > > > 
> > > > > > > 
> > > > > > >   LD      vmlinux.o
> > > > > > >   MODPOST vmlinux.symvers
> > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > >   GEN     modules.builtin
> > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > > > > > Failed to generate BTF for vmlinux
> > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux]
> > > > > > > Error 1
> > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > 
> > > > > > > Would it be possible to get pahole added to the build
> > > > > > environments
> > > > > > > used to
> > > > > > > build android kernels?
> > > > > > > 
> > > > > > > -Todd
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > 
> > 
> > 
> 
> 
> 
> 
> 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

Any update on this?


On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
<denys.f@collabora.com> wrote:
>
> On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io wrote:
> > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > denys.f@collabora.com>
> > wrote:
> >
> > > Update
> > >
> > > We identified all packages required for kernel build:
> > > dwarves, lz4, python3/python.
> > >
> > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our
> > > staging
> > > pipeline, but with error:
> > >  LD      vmlinux
> > >   BTFIDS  vmlinux
> > > FAILED unresolved symbol xdp_buff
> > >
> >
> > Which config and arch was this?
> gki_defconfig
> arm64
> But it is a bit old snapshot i'm using as reference.
> Tomorrow i will try latest 5.10 if anything changed.
>
> Container Log:
>   LD      .tmp_vmlinux.kallsyms1
>   KSYMS   .tmp_vmlinux.kallsyms1.S
>   AS      .tmp_vmlinux.kallsyms1.S
>   LD      .tmp_vmlinux.kallsyms2
>   KSYMS   .tmp_vmlinux.kallsyms2.S
>   AS      .tmp_vmlinux.kallsyms2.S
>   LD      vmlinux
>   BTFIDS  vmlinux
> FAILED unresolved symbol xdp_buff
> make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
> make[1]: Leaving directory '/tmp/kci/linux/build'
> make: *** [Makefile:185: __sub-make] Error 2
> make: Leaving directory '/tmp/kci/linux'
>
> "branch": "staging-android",
> "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> "tree": "kernelci",
> "url": "https://github.com/kernelci/linux.git"
>
> >
> >
> > >
> > > I will create PR now to allocate larger builders (to address LTO
> > > memory
> > > overconsumption) and we will do test build on android pipeline.
> > >
> >
> > Thanks for helping with this!
> >
> >
> > >
> > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> > > > Short update.
> > > > Some of builders crashed due OOM, but not because of logs,
> > > > but because LTO during linking phase consume more than 21Gb of
> > > > RAM,
> > > > and k8s builders allocated for project have only 4Gb allocated.
> > > > It is not easy to solve, as it is either require to change LTO to
> > > > "Thin
> > > > LTO" (or disable it totally) to reduce memory consumption, which
> > > > is
> > > > not
> > > > great, or increase builders size (and that means additional costs
> > > > and
> > > > approvals related to that).
> > > > We are discussing on IRC possible solutions to this problem.
> > > >
> > > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko wrote:
> > > > > At the moment I have identified a problem with more packages
> > > > > that
> > > > > are
> > > > > missing, now it is python, and i'm adding it as well, but need
> > > > > to
> > > > > test
> > > > > everything manually, and then on staging.
> > > > >
> > > > > Here is example of build that needs python:
> > > > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > >
> > > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > >
> > > > > Also, another problem some of the builders crashed, probably
> > > > > due to
> > > > > OOM
> > > > > or insufficient resources.
> > > > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > >
> > > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > >
> > > > > On manual tests, builds with pahole create about 819Mbyte of
> > > > > warnings,
> > > > > such as:
> > > > > "namespace__recode_dwarf_types: couldn't find 0x13423 type for
> > > > > 0x3d6a5
> > > > > (member)!"
> > > > > I am currently investigating whether these issues are related.
> > > > > Therefore, it takes a bit more time than i expected, than just
> > > > > adding
> > > > > a
> > > > > pahole/dwarves package.
> > > > >
> > > > > I will update as soon as more information available.
> > > > >
> > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > Denys, Thanks for the update. Do you have a prediction for
> > > > > > when
> > > > > > it
> > > > > > will land?
> > > > > >
> > > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > > <denys.f@collabora.com> wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > We are testing in staging fix for that:
> > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > >
> > > > > > > Also i noticed in some cases lz4 package required for this
> > > > > > > test,
> > > > > > > so added it too.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Denys
> > > > > > >
> > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via groups.io
> > > > > > > wrote:
> > > > > > > > We have a bunch of Android kernel test failures due to
> > > > > > > > missing
> > > > > > > > "pahole" in
> > > > > > > > the build environment (example android13-5.15
> > > > > > > > gki_defconfig
> > > > > > > >
> > > > > > > <
> > > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > >
> > > > > > > > ):
> > > > > > > >
> > > > > > > >
> > > > > > > >   LD      vmlinux.o
> > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > >   GEN     modules.builtin
> > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
> > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux]
> > > > > > > > Error 1
> > > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > >
> > > > > > > > Would it be possible to get pahole added to the build
> > > > > > > environments
> > > > > > > > used to
> > > > > > > > build android kernels?
> > > > > > > >
> > > > > > > > -Todd
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
> >
> > 
> >
> >
>
>

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

We faced multiple issues.
1) Most builders instance not suitable for this job, as it require
large amount of RAM. Found workaround, tested in staging.
2) The problem with the build time, it significantly exceeds the time
of usual builds. Theoretically, the problem can be solved by increasing
the limits.
3) Some tools was missing, thats solved.
4) Some of the created objects during build exceed the size that the
library that we use to upload objects to storage can handle. This will
require significant changes in some parts of project to solve and major
blocker for me.

I'm still not sure what to do next with that, and I'll discuss this at
the next weekly meeting (this Thursday).

On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io wrote:
> Any update on this?
> 
> 
> On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> <denys.f@collabora.com> wrote:
> > 
> > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io wrote:
> > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > denys.f@collabora.com>
> > > wrote:
> > > 
> > > > Update
> > > > 
> > > > We identified all packages required for kernel build:
> > > > dwarves, lz4, python3/python.
> > > > 
> > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our
> > > > staging
> > > > pipeline, but with error:
> > > >  LD      vmlinux
> > > >   BTFIDS  vmlinux
> > > > FAILED unresolved symbol xdp_buff
> > > > 
> > > 
> > > Which config and arch was this?
> > gki_defconfig
> > arm64
> > But it is a bit old snapshot i'm using as reference.
> > Tomorrow i will try latest 5.10 if anything changed.
> > 
> > Container Log:
> >   LD      .tmp_vmlinux.kallsyms1
> >   KSYMS   .tmp_vmlinux.kallsyms1.S
> >   AS      .tmp_vmlinux.kallsyms1.S
> >   LD      .tmp_vmlinux.kallsyms2
> >   KSYMS   .tmp_vmlinux.kallsyms2.S
> >   AS      .tmp_vmlinux.kallsyms2.S
> >   LD      vmlinux
> >   BTFIDS  vmlinux
> > FAILED unresolved symbol xdp_buff
> > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
> > make[1]: Leaving directory '/tmp/kci/linux/build'
> > make: *** [Makefile:185: __sub-make] Error 2
> > make: Leaving directory '/tmp/kci/linux'
> > 
> > "branch": "staging-android",
> > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > "tree": "kernelci",
> > "url": "https://github.com/kernelci/linux.git"
> > 
> > > 
> > > 
> > > > 
> > > > I will create PR now to allocate larger builders (to address
> > > > LTO
> > > > memory
> > > > overconsumption) and we will do test build on android pipeline.
> > > > 
> > > 
> > > Thanks for helping with this!
> > > 
> > > 
> > > > 
> > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> > > > > Short update.
> > > > > Some of builders crashed due OOM, but not because of logs,
> > > > > but because LTO during linking phase consume more than 21Gb
> > > > > of
> > > > > RAM,
> > > > > and k8s builders allocated for project have only 4Gb
> > > > > allocated.
> > > > > It is not easy to solve, as it is either require to change
> > > > > LTO to
> > > > > "Thin
> > > > > LTO" (or disable it totally) to reduce memory consumption,
> > > > > which
> > > > > is
> > > > > not
> > > > > great, or increase builders size (and that means additional
> > > > > costs
> > > > > and
> > > > > approvals related to that).
> > > > > We are discussing on IRC possible solutions to this problem.
> > > > > 
> > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko
> > > > > wrote:
> > > > > > At the moment I have identified a problem with more
> > > > > > packages
> > > > > > that
> > > > > > are
> > > > > > missing, now it is python, and i'm adding it as well, but
> > > > > > need
> > > > > > to
> > > > > > test
> > > > > > everything manually, and then on staging.
> > > > > > 
> > > > > > Here is example of build that needs python:
> > > > > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > 
> > > > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > 
> > > > > > Also, another problem some of the builders crashed,
> > > > > > probably
> > > > > > due to
> > > > > > OOM
> > > > > > or insufficient resources.
> > > > > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > 
> > > > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > 
> > > > > > On manual tests, builds with pahole create about 819Mbyte
> > > > > > of
> > > > > > warnings,
> > > > > > such as:
> > > > > > "namespace__recode_dwarf_types: couldn't find 0x13423 type
> > > > > > for
> > > > > > 0x3d6a5
> > > > > > (member)!"
> > > > > > I am currently investigating whether these issues are
> > > > > > related.
> > > > > > Therefore, it takes a bit more time than i expected, than
> > > > > > just
> > > > > > adding
> > > > > > a
> > > > > > pahole/dwarves package.
> > > > > > 
> > > > > > I will update as soon as more information available.
> > > > > > 
> > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > Denys, Thanks for the update. Do you have a prediction
> > > > > > > for
> > > > > > > when
> > > > > > > it
> > > > > > > will land?
> > > > > > > 
> > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > We are testing in staging fix for that:
> > > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > 
> > > > > > > > Also i noticed in some cases lz4 package required for
> > > > > > > > this
> > > > > > > > test,
> > > > > > > > so added it too.
> > > > > > > > 
> > > > > > > > Best regards,
> > > > > > > > Denys
> > > > > > > > 
> > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via
> > > > > > > > groups.io
> > > > > > > > wrote:
> > > > > > > > > We have a bunch of Android kernel test failures due
> > > > > > > > > to
> > > > > > > > > missing
> > > > > > > > > "pahole" in
> > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > gki_defconfig
> > > > > > > > > 
> > > > > > > > <
> > > > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > 
> > > > > > > > > ):
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > >   LD      vmlinux.o
> > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > >   GEN     modules.builtin
> > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > available
> > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux]
> > > > > > > > > Error 1
> > > > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > 
> > > > > > > > > Would it be possible to get pahole added to the build
> > > > > > > > environments
> > > > > > > > > used to
> > > > > > > > > build android kernels?
> > > > > > > > > 
> > > > > > > > > -Todd
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> 
> 
> 
> 
> 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

Any update from your Thursday meeting?

On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> We faced multiple issues.
> 1) Most builders instance not suitable for this job, as it require
> large amount of RAM. Found workaround, tested in staging.
> 2) The problem with the build time, it significantly exceeds the time
> of usual builds. Theoretically, the problem can be solved by increasing
> the limits.
> 3) Some tools was missing, thats solved.
> 4) Some of the created objects during build exceed the size that the
> library that we use to upload objects to storage can handle. This will
> require significant changes in some parts of project to solve and major
> blocker for me.
>
> I'm still not sure what to do next with that, and I'll discuss this at
> the next weekly meeting (this Thursday).
>
> On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io wrote:
> > Any update on this?
> >
> >
> > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > <denys.f@collabora.com> wrote:
> > >
> > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io wrote:
> > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > denys.f@collabora.com>
> > > > wrote:
> > > >
> > > > > Update
> > > > >
> > > > > We identified all packages required for kernel build:
> > > > > dwarves, lz4, python3/python.
> > > > >
> > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on our
> > > > > staging
> > > > > pipeline, but with error:
> > > > >  LD      vmlinux
> > > > >   BTFIDS  vmlinux
> > > > > FAILED unresolved symbol xdp_buff
> > > > >
> > > >
> > > > Which config and arch was this?
> > > gki_defconfig
> > > arm64
> > > But it is a bit old snapshot i'm using as reference.
> > > Tomorrow i will try latest 5.10 if anything changed.
> > >
> > > Container Log:
> > >   LD      .tmp_vmlinux.kallsyms1
> > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > >   AS      .tmp_vmlinux.kallsyms1.S
> > >   LD      .tmp_vmlinux.kallsyms2
> > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > >   AS      .tmp_vmlinux.kallsyms2.S
> > >   LD      vmlinux
> > >   BTFIDS  vmlinux
> > > FAILED unresolved symbol xdp_buff
> > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
> > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > make: *** [Makefile:185: __sub-make] Error 2
> > > make: Leaving directory '/tmp/kci/linux'
> > >
> > > "branch": "staging-android",
> > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > "tree": "kernelci",
> > > "url": "https://github.com/kernelci/linux.git"
> > >
> > > >
> > > >
> > > > >
> > > > > I will create PR now to allocate larger builders (to address
> > > > > LTO
> > > > > memory
> > > > > overconsumption) and we will do test build on android pipeline.
> > > > >
> > > >
> > > > Thanks for helping with this!
> > > >
> > > >
> > > > >
> > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko wrote:
> > > > > > Short update.
> > > > > > Some of builders crashed due OOM, but not because of logs,
> > > > > > but because LTO during linking phase consume more than 21Gb
> > > > > > of
> > > > > > RAM,
> > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > allocated.
> > > > > > It is not easy to solve, as it is either require to change
> > > > > > LTO to
> > > > > > "Thin
> > > > > > LTO" (or disable it totally) to reduce memory consumption,
> > > > > > which
> > > > > > is
> > > > > > not
> > > > > > great, or increase builders size (and that means additional
> > > > > > costs
> > > > > > and
> > > > > > approvals related to that).
> > > > > > We are discussing on IRC possible solutions to this problem.
> > > > > >
> > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko
> > > > > > wrote:
> > > > > > > At the moment I have identified a problem with more
> > > > > > > packages
> > > > > > > that
> > > > > > > are
> > > > > > > missing, now it is python, and i'm adding it as well, but
> > > > > > > need
> > > > > > > to
> > > > > > > test
> > > > > > > everything manually, and then on staging.
> > > > > > >
> > > > > > > Here is example of build that needs python:
> > > > > > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > >
> > > > >
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > >
> > > > > > > Also, another problem some of the builders crashed,
> > > > > > > probably
> > > > > > > due to
> > > > > > > OOM
> > > > > > > or insufficient resources.
> > > > > > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > >
> > > > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > >
> > > > > > > On manual tests, builds with pahole create about 819Mbyte
> > > > > > > of
> > > > > > > warnings,
> > > > > > > such as:
> > > > > > > "namespace__recode_dwarf_types: couldn't find 0x13423 type
> > > > > > > for
> > > > > > > 0x3d6a5
> > > > > > > (member)!"
> > > > > > > I am currently investigating whether these issues are
> > > > > > > related.
> > > > > > > Therefore, it takes a bit more time than i expected, than
> > > > > > > just
> > > > > > > adding
> > > > > > > a
> > > > > > > pahole/dwarves package.
> > > > > > >
> > > > > > > I will update as soon as more information available.
> > > > > > >
> > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > Denys, Thanks for the update. Do you have a prediction
> > > > > > > > for
> > > > > > > > when
> > > > > > > > it
> > > > > > > > will land?
> > > > > > > >
> > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > >
> > > > > > > > > Also i noticed in some cases lz4 package required for
> > > > > > > > > this
> > > > > > > > > test,
> > > > > > > > > so added it too.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Denys
> > > > > > > > >
> > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via
> > > > > > > > > groups.io
> > > > > > > > > wrote:
> > > > > > > > > > We have a bunch of Android kernel test failures due
> > > > > > > > > > to
> > > > > > > > > > missing
> > > > > > > > > > "pahole" in
> > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > gki_defconfig
> > > > > > > > > >
> > > > > > > > > <
> > > > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > >
> > > > > > > > > > ):
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > available
> > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247: vmlinux]
> > > > > > > > > > Error 1
> > > > > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > >
> > > > > > > > > > Would it be possible to get pahole added to the build
> > > > > > > > > environments
> > > > > > > > > > used to
> > > > > > > > > > build android kernels?
> > > > > > > > > >
> > > > > > > > > > -Todd
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> > 
> >
> >
>
>
>

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

Hi,

Sorry for the delay, it took longer than this meeting as we faced major
issue - current system design cannot handle large objects upload
(larger than ~200Mbyte) and there is no easy workaround for that.

Theoretically, the new system architecture (kernelci-api) solves this
problem, but it will take several months to finalize it. 
I'll try to develop a partial upload method (so it will upload file by
chunks), but that too will take some time as i have only few hours per
week aside from primary project and other developers are also
overloaded.
Also during discussion it was suggested alternative solution: to run
dedicated jenkins job running directly on one of builders, instead of
kubernetes, as this will allow us to use scp upload method similar as
we do with rootfs builds.
But this will also require creation of a completely new job
configuration for the Jenkins.

On Fri, 2022-07-08 at 10:02 -0700, Todd Kjos via groups.io wrote:
> Any update from your Thursday meeting?
> 
> On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko
> <denys.f@collabora.com>
> wrote:
> 
> > We faced multiple issues.
> > 1) Most builders instance not suitable for this job, as it require
> > large amount of RAM. Found workaround, tested in staging.
> > 2) The problem with the build time, it significantly exceeds the
> > time
> > of usual builds. Theoretically, the problem can be solved by
> > increasing
> > the limits.
> > 3) Some tools was missing, thats solved.
> > 4) Some of the created objects during build exceed the size that
> > the
> > library that we use to upload objects to storage can handle. This
> > will
> > require significant changes in some parts of project to solve and
> > major
> > blocker for me.
> > 
> > I'm still not sure what to do next with that, and I'll discuss this
> > at
> > the next weekly meeting (this Thursday).
> > 
> > On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io wrote:
> > > Any update on this?
> > > 
> > > 
> > > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > > <denys.f@collabora.com> wrote:
> > > > 
> > > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io
> > > > wrote:
> > > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > > denys.f@collabora.com>
> > > > > wrote:
> > > > > 
> > > > > > Update
> > > > > > 
> > > > > > We identified all packages required for kernel build:
> > > > > > dwarves, lz4, python3/python.
> > > > > > 
> > > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on
> > > > > > our
> > > > > > staging
> > > > > > pipeline, but with error:
> > > > > >  LD      vmlinux
> > > > > >   BTFIDS  vmlinux
> > > > > > FAILED unresolved symbol xdp_buff
> > > > > > 
> > > > > 
> > > > > Which config and arch was this?
> > > > gki_defconfig
> > > > arm64
> > > > But it is a bit old snapshot i'm using as reference.
> > > > Tomorrow i will try latest 5.10 if anything changed.
> > > > 
> > > > Container Log:
> > > >   LD      .tmp_vmlinux.kallsyms1
> > > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > > >   AS      .tmp_vmlinux.kallsyms1.S
> > > >   LD      .tmp_vmlinux.kallsyms2
> > > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > > >   AS      .tmp_vmlinux.kallsyms2.S
> > > >   LD      vmlinux
> > > >   BTFIDS  vmlinux
> > > > FAILED unresolved symbol xdp_buff
> > > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
> > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > make: *** [Makefile:185: __sub-make] Error 2
> > > > make: Leaving directory '/tmp/kci/linux'
> > > > 
> > > > "branch": "staging-android",
> > > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > > "tree": "kernelci",
> > > > "url": "https://github.com/kernelci/linux.git"
> > > > 
> > > > > 
> > > > > 
> > > > > > 
> > > > > > I will create PR now to allocate larger builders (to
> > > > > > address
> > > > > > LTO
> > > > > > memory
> > > > > > overconsumption) and we will do test build on android
> > > > > > pipeline.
> > > > > > 
> > > > > 
> > > > > Thanks for helping with this!
> > > > > 
> > > > > 
> > > > > > 
> > > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko
> > > > > > wrote:
> > > > > > > Short update.
> > > > > > > Some of builders crashed due OOM, but not because of
> > > > > > > logs,
> > > > > > > but because LTO during linking phase consume more than
> > > > > > > 21Gb
> > > > > > > of
> > > > > > > RAM,
> > > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > > allocated.
> > > > > > > It is not easy to solve, as it is either require to
> > > > > > > change
> > > > > > > LTO to
> > > > > > > "Thin
> > > > > > > LTO" (or disable it totally) to reduce memory
> > > > > > > consumption,
> > > > > > > which
> > > > > > > is
> > > > > > > not
> > > > > > > great, or increase builders size (and that means
> > > > > > > additional
> > > > > > > costs
> > > > > > > and
> > > > > > > approvals related to that).
> > > > > > > We are discussing on IRC possible solutions to this
> > > > > > > problem.
> > > > > > > 
> > > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko
> > > > > > > wrote:
> > > > > > > > At the moment I have identified a problem with more
> > > > > > > > packages
> > > > > > > > that
> > > > > > > > are
> > > > > > > > missing, now it is python, and i'm adding it as well,
> > > > > > > > but
> > > > > > > > need
> > > > > > > > to
> > > > > > > > test
> > > > > > > > everything manually, and then on staging.
> > > > > > > > 
> > > > > > > > Here is example of build that needs python:
> > > > > > > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > > > 
> > > > > > 
> > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > 
> > > > > > > > Also, another problem some of the builders crashed,
> > > > > > > > probably
> > > > > > > > due to
> > > > > > > > OOM
> > > > > > > > or insufficient resources.
> > > > > > > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > > > 
> > > > > > 
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > 
> > > > > > > > On manual tests, builds with pahole create about
> > > > > > > > 819Mbyte
> > > > > > > > of
> > > > > > > > warnings,
> > > > > > > > such as:
> > > > > > > > "namespace__recode_dwarf_types: couldn't find 0x13423
> > > > > > > > type
> > > > > > > > for
> > > > > > > > 0x3d6a5
> > > > > > > > (member)!"
> > > > > > > > I am currently investigating whether these issues are
> > > > > > > > related.
> > > > > > > > Therefore, it takes a bit more time than i expected,
> > > > > > > > than
> > > > > > > > just
> > > > > > > > adding
> > > > > > > > a
> > > > > > > > pahole/dwarves package.
> > > > > > > > 
> > > > > > > > I will update as soon as more information available.
> > > > > > > > 
> > > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > > Denys, Thanks for the update. Do you have a
> > > > > > > > > prediction
> > > > > > > > > for
> > > > > > > > > when
> > > > > > > > > it
> > > > > > > > > will land?
> > > > > > > > > 
> > > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > > > 
> > > > > > > > > > Also i noticed in some cases lz4 package required
> > > > > > > > > > for
> > > > > > > > > > this
> > > > > > > > > > test,
> > > > > > > > > > so added it too.
> > > > > > > > > > 
> > > > > > > > > > Best regards,
> > > > > > > > > > Denys
> > > > > > > > > > 
> > > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via
> > > > > > > > > > groups.io
> > > > > > > > > > wrote:
> > > > > > > > > > > We have a bunch of Android kernel test failures
> > > > > > > > > > > due
> > > > > > > > > > > to
> > > > > > > > > > > missing
> > > > > > > > > > > "pahole" in
> > > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > > gki_defconfig
> > > > > > > > > > > 
> > > > > > > > > > <
> > > > > > 
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > > > 
> > > > > > > > > > > ):
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > > available
> > > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247:
> > > > > > > > > > > vmlinux]
> > > > > > > > > > > Error 1
> > > > > > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > > > 
> > > > > > > > > > > Would it be possible to get pahole added to the
> > > > > > > > > > > build
> > > > > > > > > > environments
> > > > > > > > > > > used to
> > > > > > > > > > > build android kernels?
> > > > > > > > > > > 
> > > > > > > > > > > -Todd
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> 
> 
> 
> 
> 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

Denys, thanks for the update. I don't consider this urgent, but I would
eventually like to see our "gki_defconfig" builds be green again.

On Tue, Jul 12, 2022 at 2:49 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> Hi,
>
> Sorry for the delay, it took longer than this meeting as we faced major
> issue - current system design cannot handle large objects upload
> (larger than ~200Mbyte) and there is no easy workaround for that.
>
> Theoretically, the new system architecture (kernelci-api) solves this
> problem, but it will take several months to finalize it.
> I'll try to develop a partial upload method (so it will upload file by
> chunks), but that too will take some time as i have only few hours per
> week aside from primary project and other developers are also
> overloaded.
> Also during discussion it was suggested alternative solution: to run
> dedicated jenkins job running directly on one of builders, instead of
> kubernetes, as this will allow us to use scp upload method similar as
> we do with rootfs builds.
> But this will also require creation of a completely new job
> configuration for the Jenkins.
>
> On Fri, 2022-07-08 at 10:02 -0700, Todd Kjos via groups.io wrote:
> > Any update from your Thursday meeting?
> >
> > On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko
> > <denys.f@collabora.com>
> > wrote:
> >
> > > We faced multiple issues.
> > > 1) Most builders instance not suitable for this job, as it require
> > > large amount of RAM. Found workaround, tested in staging.
> > > 2) The problem with the build time, it significantly exceeds the
> > > time
> > > of usual builds. Theoretically, the problem can be solved by
> > > increasing
> > > the limits.
> > > 3) Some tools was missing, thats solved.
> > > 4) Some of the created objects during build exceed the size that
> > > the
> > > library that we use to upload objects to storage can handle. This
> > > will
> > > require significant changes in some parts of project to solve and
> > > major
> > > blocker for me.
> > >
> > > I'm still not sure what to do next with that, and I'll discuss this
> > > at
> > > the next weekly meeting (this Thursday).
> > >
> > > On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io wrote:
> > > > Any update on this?
> > > >
> > > >
> > > > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > > > <denys.f@collabora.com> wrote:
> > > > >
> > > > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io
> > > > > wrote:
> > > > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > > > denys.f@collabora.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Update
> > > > > > >
> > > > > > > We identified all packages required for kernel build:
> > > > > > > dwarves, lz4, python3/python.
> > > > > > >
> > > > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on
> > > > > > > our
> > > > > > > staging
> > > > > > > pipeline, but with error:
> > > > > > >  LD      vmlinux
> > > > > > >   BTFIDS  vmlinux
> > > > > > > FAILED unresolved symbol xdp_buff
> > > > > > >
> > > > > >
> > > > > > Which config and arch was this?
> > > > > gki_defconfig
> > > > > arm64
> > > > > But it is a bit old snapshot i'm using as reference.
> > > > > Tomorrow i will try latest 5.10 if anything changed.
> > > > >
> > > > > Container Log:
> > > > >   LD      .tmp_vmlinux.kallsyms1
> > > > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > > > >   AS      .tmp_vmlinux.kallsyms1.S
> > > > >   LD      .tmp_vmlinux.kallsyms2
> > > > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > > > >   AS      .tmp_vmlinux.kallsyms2.S
> > > > >   LD      vmlinux
> > > > >   BTFIDS  vmlinux
> > > > > FAILED unresolved symbol xdp_buff
> > > > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
> > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > make: *** [Makefile:185: __sub-make] Error 2
> > > > > make: Leaving directory '/tmp/kci/linux'
> > > > >
> > > > > "branch": "staging-android",
> > > > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > > > "tree": "kernelci",
> > > > > "url": "https://github.com/kernelci/linux.git"
> > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > I will create PR now to allocate larger builders (to
> > > > > > > address
> > > > > > > LTO
> > > > > > > memory
> > > > > > > overconsumption) and we will do test build on android
> > > > > > > pipeline.
> > > > > > >
> > > > > >
> > > > > > Thanks for helping with this!
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko
> > > > > > > wrote:
> > > > > > > > Short update.
> > > > > > > > Some of builders crashed due OOM, but not because of
> > > > > > > > logs,
> > > > > > > > but because LTO during linking phase consume more than
> > > > > > > > 21Gb
> > > > > > > > of
> > > > > > > > RAM,
> > > > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > > > allocated.
> > > > > > > > It is not easy to solve, as it is either require to
> > > > > > > > change
> > > > > > > > LTO to
> > > > > > > > "Thin
> > > > > > > > LTO" (or disable it totally) to reduce memory
> > > > > > > > consumption,
> > > > > > > > which
> > > > > > > > is
> > > > > > > > not
> > > > > > > > great, or increase builders size (and that means
> > > > > > > > additional
> > > > > > > > costs
> > > > > > > > and
> > > > > > > > approvals related to that).
> > > > > > > > We are discussing on IRC possible solutions to this
> > > > > > > > problem.
> > > > > > > >
> > > > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko
> > > > > > > > wrote:
> > > > > > > > > At the moment I have identified a problem with more
> > > > > > > > > packages
> > > > > > > > > that
> > > > > > > > > are
> > > > > > > > > missing, now it is python, and i'm adding it as well,
> > > > > > > > > but
> > > > > > > > > need
> > > > > > > > > to
> > > > > > > > > test
> > > > > > > > > everything manually, and then on staging.
> > > > > > > > >
> > > > > > > > > Here is example of build that needs python:
> > > > > > > > >
> https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > > > >
> > > > > > >
> > >
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > >
> > > > > > > > > Also, another problem some of the builders crashed,
> > > > > > > > > probably
> > > > > > > > > due to
> > > > > > > > > OOM
> > > > > > > > > or insufficient resources.
> > > > > > > > >
> https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > > > >
> > > > > > >
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > >
> > > > > > > > > On manual tests, builds with pahole create about
> > > > > > > > > 819Mbyte
> > > > > > > > > of
> > > > > > > > > warnings,
> > > > > > > > > such as:
> > > > > > > > > "namespace__recode_dwarf_types: couldn't find 0x13423
> > > > > > > > > type
> > > > > > > > > for
> > > > > > > > > 0x3d6a5
> > > > > > > > > (member)!"
> > > > > > > > > I am currently investigating whether these issues are
> > > > > > > > > related.
> > > > > > > > > Therefore, it takes a bit more time than i expected,
> > > > > > > > > than
> > > > > > > > > just
> > > > > > > > > adding
> > > > > > > > > a
> > > > > > > > > pahole/dwarves package.
> > > > > > > > >
> > > > > > > > > I will update as soon as more information available.
> > > > > > > > >
> > > > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > > > Denys, Thanks for the update. Do you have a
> > > > > > > > > > prediction
> > > > > > > > > > for
> > > > > > > > > > when
> > > > > > > > > > it
> > > > > > > > > > will land?
> > > > > > > > > >
> > > > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > > > >
> > > > > > > > > > > Also i noticed in some cases lz4 package required
> > > > > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > test,
> > > > > > > > > > > so added it too.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Denys
> > > > > > > > > > >
> > > > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via
> > > > > > > > > > > groups.io
> > > > > > > > > > > wrote:
> > > > > > > > > > > > We have a bunch of Android kernel test failures
> > > > > > > > > > > > due
> > > > > > > > > > > > to
> > > > > > > > > > > > missing
> > > > > > > > > > > > "pahole" in
> > > > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > > > gki_defconfig
> > > > > > > > > > > >
> > > > > > > > > > > <
> > > > > > >
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > > > >
> > > > > > > > > > > > ):
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > > > available
> > > > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247:
> > > > > > > > > > > > vmlinux]
> > > > > > > > > > > > Error 1
> > > > > > > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > > > >
> > > > > > > > > > > > Would it be possible to get pahole added to the
> > > > > > > > > > > > build
> > > > > > > > > > > environments
> > > > > > > > > > > > used to
> > > > > > > > > > > > build android kernels?
> > > > > > > > > > > >
> > > > > > > > > > > > -Todd
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> > 
> >
> >
>
>
>

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

I apologize for the delay, as what was supposed to be a simple task
with the addition of a pahole tool turned into a fundamental problem. 
I hope to be able to solve this problem soon.
One of the options I'm considering is compressing Image and kernel.log
through xz -9.
This whille make build a bit different from standard builds. 
It is a bit hack, but might be a bit faster than implementing other
solutions.
Is it acceptable?

Here i pushed in staging almost manually
results: https://storage.staging.kernelci.org/kernelci/staging-android/ASB-2022-06-05_13-5.15-46-g15a32b46ee0f4/arm64/gki_defconfig/clang-15/logs/
As you can see kernel.log is 813Mbyte and Image is 193Mbyte, which is a
bit too much.

On Wed, 2022-07-13 at 10:32 -0700, Todd Kjos wrote:
> Denys, thanks for the update. I don't consider this urgent, but I
> would eventually like to see our "gki_defconfig" builds be green
> again.
> 
> On Tue, Jul 12, 2022 at 2:49 AM Denys Fedoryshchenko
> <denys.f@collabora.com> wrote:
> > Hi,
> > 
> > Sorry for the delay, it took longer than this meeting as we faced
> > major
> > issue - current system design cannot handle large objects upload
> > (larger than ~200Mbyte) and there is no easy workaround for that.
> > 
> > Theoretically, the new system architecture (kernelci-api) solves
> > this
> > problem, but it will take several months to finalize it. 
> > I'll try to develop a partial upload method (so it will upload file
> > by
> > chunks), but that too will take some time as i have only few hours
> > per
> > week aside from primary project and other developers are also
> > overloaded.
> > Also during discussion it was suggested alternative solution: to
> > run
> > dedicated jenkins job running directly on one of builders, instead
> > of
> > kubernetes, as this will allow us to use scp upload method similar
> > as
> > we do with rootfs builds.
> > But this will also require creation of a completely new job
> > configuration for the Jenkins.
> > 
> > On Fri, 2022-07-08 at 10:02 -0700, Todd Kjos via groups.io wrote:
> > > Any update from your Thursday meeting?
> > > 
> > > On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko
> > > <denys.f@collabora.com>
> > > wrote:
> > > 
> > > > We faced multiple issues.
> > > > 1) Most builders instance not suitable for this job, as it
> > require
> > > > large amount of RAM. Found workaround, tested in staging.
> > > > 2) The problem with the build time, it significantly exceeds
> > the
> > > > time
> > > > of usual builds. Theoretically, the problem can be solved by
> > > > increasing
> > > > the limits.
> > > > 3) Some tools was missing, thats solved.
> > > > 4) Some of the created objects during build exceed the size
> > that
> > > > the
> > > > library that we use to upload objects to storage can handle.
> > This
> > > > will
> > > > require significant changes in some parts of project to solve
> > and
> > > > major
> > > > blocker for me.
> > > > 
> > > > I'm still not sure what to do next with that, and I'll discuss
> > this
> > > > at
> > > > the next weekly meeting (this Thursday).
> > > > 
> > > > On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io
> > wrote:
> > > > > Any update on this?
> > > > > 
> > > > > 
> > > > > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > > > > <denys.f@collabora.com> wrote:
> > > > > > 
> > > > > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io
> > > > > > wrote:
> > > > > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > > > > denys.f@collabora.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Update
> > > > > > > > 
> > > > > > > > We identified all packages required for kernel build:
> > > > > > > > dwarves, lz4, python3/python.
> > > > > > > > 
> > > > > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built
> > on
> > > > > > > > our
> > > > > > > > staging
> > > > > > > > pipeline, but with error:
> > > > > > > >  LD      vmlinux
> > > > > > > >   BTFIDS  vmlinux
> > > > > > > > FAILED unresolved symbol xdp_buff
> > > > > > > > 
> > > > > > > 
> > > > > > > Which config and arch was this?
> > > > > > gki_defconfig
> > > > > > arm64
> > > > > > But it is a bit old snapshot i'm using as reference.
> > > > > > Tomorrow i will try latest 5.10 if anything changed.
> > > > > > 
> > > > > > Container Log:
> > > > > >   LD      .tmp_vmlinux.kallsyms1
> > > > > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > > > > >   AS      .tmp_vmlinux.kallsyms1.S
> > > > > >   LD      .tmp_vmlinux.kallsyms2
> > > > > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > > > > >   AS      .tmp_vmlinux.kallsyms2.S
> > > > > >   LD      vmlinux
> > > > > >   BTFIDS  vmlinux
> > > > > > FAILED unresolved symbol xdp_buff
> > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error
> > 255
> > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > make: *** [Makefile:185: __sub-make] Error 2
> > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > 
> > > > > > "branch": "staging-android",
> > > > > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > > > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > > > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > > > > "tree": "kernelci",
> > > > > > "url": "https://github.com/kernelci/linux.git"
> > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > I will create PR now to allocate larger builders (to
> > > > > > > > address
> > > > > > > > LTO
> > > > > > > > memory
> > > > > > > > overconsumption) and we will do test build on android
> > > > > > > > pipeline.
> > > > > > > > 
> > > > > > > 
> > > > > > > Thanks for helping with this!
> > > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko
> > > > > > > > wrote:
> > > > > > > > > Short update.
> > > > > > > > > Some of builders crashed due OOM, but not because of
> > > > > > > > > logs,
> > > > > > > > > but because LTO during linking phase consume more
> > than
> > > > > > > > > 21Gb
> > > > > > > > > of
> > > > > > > > > RAM,
> > > > > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > > > > allocated.
> > > > > > > > > It is not easy to solve, as it is either require to
> > > > > > > > > change
> > > > > > > > > LTO to
> > > > > > > > > "Thin
> > > > > > > > > LTO" (or disable it totally) to reduce memory
> > > > > > > > > consumption,
> > > > > > > > > which
> > > > > > > > > is
> > > > > > > > > not
> > > > > > > > > great, or increase builders size (and that means
> > > > > > > > > additional
> > > > > > > > > costs
> > > > > > > > > and
> > > > > > > > > approvals related to that).
> > > > > > > > > We are discussing on IRC possible solutions to this
> > > > > > > > > problem.
> > > > > > > > > 
> > > > > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys
> > Fedoryshchenko
> > > > > > > > > wrote:
> > > > > > > > > > At the moment I have identified a problem with more
> > > > > > > > > > packages
> > > > > > > > > > that
> > > > > > > > > > are
> > > > > > > > > > missing, now it is python, and i'm adding it as
> > well,
> > > > > > > > > > but
> > > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > test
> > > > > > > > > > everything manually, and then on staging.
> > > > > > > > > > 
> > > > > > > > > > Here is example of build that needs python:
> > > > > > > > > >
> > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > > > > > 
> > > > > > > > 
> > > >
> >
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > > > 
> > > > > > > > > > Also, another problem some of the builders crashed,
> > > > > > > > > > probably
> > > > > > > > > > due to
> > > > > > > > > > OOM
> > > > > > > > > > or insufficient resources.
> > > > > > > > > >
> > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > > > > > 
> > > > > > > > 
> > > >
> >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > > > 
> > > > > > > > > > On manual tests, builds with pahole create about
> > > > > > > > > > 819Mbyte
> > > > > > > > > > of
> > > > > > > > > > warnings,
> > > > > > > > > > such as:
> > > > > > > > > > "namespace__recode_dwarf_types: couldn't find
> > 0x13423
> > > > > > > > > > type
> > > > > > > > > > for
> > > > > > > > > > 0x3d6a5
> > > > > > > > > > (member)!"
> > > > > > > > > > I am currently investigating whether these issues
> > are
> > > > > > > > > > related.
> > > > > > > > > > Therefore, it takes a bit more time than i
> > expected,
> > > > > > > > > > than
> > > > > > > > > > just
> > > > > > > > > > adding
> > > > > > > > > > a
> > > > > > > > > > pahole/dwarves package.
> > > > > > > > > > 
> > > > > > > > > > I will update as soon as more information
> > available.
> > > > > > > > > > 
> > > > > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > > > > Denys, Thanks for the update. Do you have a
> > > > > > > > > > > prediction
> > > > > > > > > > > for
> > > > > > > > > > > when
> > > > > > > > > > > it
> > > > > > > > > > > will land?
> > > > > > > > > > > 
> > > > > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys
> > Fedoryshchenko
> > > > > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > 
> > > > > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > > > >
> > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > > > > > 
> > > > > > > > > > > > Also i noticed in some cases lz4 package
> > required
> > > > > > > > > > > > for
> > > > > > > > > > > > this
> > > > > > > > > > > > test,
> > > > > > > > > > > > so added it too.
> > > > > > > > > > > > 
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Denys
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos
> > via
> > > > > > > > > > > > groups.io
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > We have a bunch of Android kernel test
> > failures
> > > > > > > > > > > > > due
> > > > > > > > > > > > > to
> > > > > > > > > > > > > missing
> > > > > > > > > > > > > "pahole" in
> > > > > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > > > > gki_defconfig
> > > > > > > > > > > > > 
> > > > > > > > > > > > <
> > > > > > > > 
> > > >
> >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > ):
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > > > > available
> > > > > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247:
> > > > > > > > > > > > > vmlinux]
> > > > > > > > > > > > > Error 1
> > > > > > > > > > > > > make[1]: Leaving directory
> > '/tmp/kci/linux/build'
> > > > > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Would it be possible to get pahole added to
> > the
> > > > > > > > > > > > > build
> > > > > > > > > > > > environments
> > > > > > > > > > > > > used to
> > > > > > > > > > > > > build android kernels?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > -Todd
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

As long as it is easy to access/decompress the logs that's fine.

On Wed, Jul 13, 2022 at 10:40 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> I apologize for the delay, as what was supposed to be a simple task with
> the addition of a pahole tool turned into a fundamental problem.
> I hope to be able to solve this problem soon.
> One of the options I'm considering is compressing Image and kernel.log
> through xz -9.
> This whille make build a bit different from standard builds.
> It is a bit hack, but might be a bit faster than implementing other
> solutions.
> Is it acceptable?
>
> Here i pushed in staging almost manually results:
> https://storage.staging.kernelci.org/kernelci/staging-android/ASB-2022-06-05_13-5.15-46-g15a32b46ee0f4/arm64/gki_defconfig/clang-15/logs/
> As you can see kernel.log is 813Mbyte and Image is 193Mbyte, which is a
> bit too much.
>
> On Wed, 2022-07-13 at 10:32 -0700, Todd Kjos wrote:
>
> Denys, thanks for the update. I don't consider this urgent, but I would
> eventually like to see our "gki_defconfig" builds be green again.
>
> On Tue, Jul 12, 2022 at 2:49 AM Denys Fedoryshchenko <
> denys.f@collabora.com> wrote:
>
> Hi,
>
> Sorry for the delay, it took longer than this meeting as we faced major
> issue - current system design cannot handle large objects upload
> (larger than ~200Mbyte) and there is no easy workaround for that.
>
> Theoretically, the new system architecture (kernelci-api) solves this
> problem, but it will take several months to finalize it.
> I'll try to develop a partial upload method (so it will upload file by
> chunks), but that too will take some time as i have only few hours per
> week aside from primary project and other developers are also
> overloaded.
> Also during discussion it was suggested alternative solution: to run
> dedicated jenkins job running directly on one of builders, instead of
> kubernetes, as this will allow us to use scp upload method similar as
> we do with rootfs builds.
> But this will also require creation of a completely new job
> configuration for the Jenkins.
>
> On Fri, 2022-07-08 at 10:02 -0700, Todd Kjos via groups.io wrote:
> > Any update from your Thursday meeting?
> >
> > On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko
> > <denys.f@collabora.com>
> > wrote:
> >
> > > We faced multiple issues.
> > > 1) Most builders instance not suitable for this job, as it require
> > > large amount of RAM. Found workaround, tested in staging.
> > > 2) The problem with the build time, it significantly exceeds the
> > > time
> > > of usual builds. Theoretically, the problem can be solved by
> > > increasing
> > > the limits.
> > > 3) Some tools was missing, thats solved.
> > > 4) Some of the created objects during build exceed the size that
> > > the
> > > library that we use to upload objects to storage can handle. This
> > > will
> > > require significant changes in some parts of project to solve and
> > > major
> > > blocker for me.
> > >
> > > I'm still not sure what to do next with that, and I'll discuss this
> > > at
> > > the next weekly meeting (this Thursday).
> > >
> > > On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io wrote:
> > > > Any update on this?
> > > >
> > > >
> > > > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > > > <denys.f@collabora.com> wrote:
> > > > >
> > > > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io
> > > > > wrote:
> > > > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > > > denys.f@collabora.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Update
> > > > > > >
> > > > > > > We identified all packages required for kernel build:
> > > > > > > dwarves, lz4, python3/python.
> > > > > > >
> > > > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built on
> > > > > > > our
> > > > > > > staging
> > > > > > > pipeline, but with error:
> > > > > > >  LD      vmlinux
> > > > > > >   BTFIDS  vmlinux
> > > > > > > FAILED unresolved symbol xdp_buff
> > > > > > >
> > > > > >
> > > > > > Which config and arch was this?
> > > > > gki_defconfig
> > > > > arm64
> > > > > But it is a bit old snapshot i'm using as reference.
> > > > > Tomorrow i will try latest 5.10 if anything changed.
> > > > >
> > > > > Container Log:
> > > > >   LD      .tmp_vmlinux.kallsyms1
> > > > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > > > >   AS      .tmp_vmlinux.kallsyms1.S
> > > > >   LD      .tmp_vmlinux.kallsyms2
> > > > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > > > >   AS      .tmp_vmlinux.kallsyms2.S
> > > > >   LD      vmlinux
> > > > >   BTFIDS  vmlinux
> > > > > FAILED unresolved symbol xdp_buff
> > > > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error 255
> > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > make: *** [Makefile:185: __sub-make] Error 2
> > > > > make: Leaving directory '/tmp/kci/linux'
> > > > >
> > > > > "branch": "staging-android",
> > > > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > > > "tree": "kernelci",
> > > > > "url": "https://github.com/kernelci/linux.git"
> > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > I will create PR now to allocate larger builders (to
> > > > > > > address
> > > > > > > LTO
> > > > > > > memory
> > > > > > > overconsumption) and we will do test build on android
> > > > > > > pipeline.
> > > > > > >
> > > > > >
> > > > > > Thanks for helping with this!
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko
> > > > > > > wrote:
> > > > > > > > Short update.
> > > > > > > > Some of builders crashed due OOM, but not because of
> > > > > > > > logs,
> > > > > > > > but because LTO during linking phase consume more than
> > > > > > > > 21Gb
> > > > > > > > of
> > > > > > > > RAM,
> > > > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > > > allocated.
> > > > > > > > It is not easy to solve, as it is either require to
> > > > > > > > change
> > > > > > > > LTO to
> > > > > > > > "Thin
> > > > > > > > LTO" (or disable it totally) to reduce memory
> > > > > > > > consumption,
> > > > > > > > which
> > > > > > > > is
> > > > > > > > not
> > > > > > > > great, or increase builders size (and that means
> > > > > > > > additional
> > > > > > > > costs
> > > > > > > > and
> > > > > > > > approvals related to that).
> > > > > > > > We are discussing on IRC possible solutions to this
> > > > > > > > problem.
> > > > > > > >
> > > > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys Fedoryshchenko
> > > > > > > > wrote:
> > > > > > > > > At the moment I have identified a problem with more
> > > > > > > > > packages
> > > > > > > > > that
> > > > > > > > > are
> > > > > > > > > missing, now it is python, and i'm adding it as well,
> > > > > > > > > but
> > > > > > > > > need
> > > > > > > > > to
> > > > > > > > > test
> > > > > > > > > everything manually, and then on staging.
> > > > > > > > >
> > > > > > > > > Here is example of build that needs python:
> > > > > > > > >
> https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > > > >
> > > > > > >
> > >
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > >
> > > > > > > > > Also, another problem some of the builders crashed,
> > > > > > > > > probably
> > > > > > > > > due to
> > > > > > > > > OOM
> > > > > > > > > or insufficient resources.
> > > > > > > > >
> https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > > > >
> > > > > > >
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > >
> > > > > > > > > On manual tests, builds with pahole create about
> > > > > > > > > 819Mbyte
> > > > > > > > > of
> > > > > > > > > warnings,
> > > > > > > > > such as:
> > > > > > > > > "namespace__recode_dwarf_types: couldn't find 0x13423
> > > > > > > > > type
> > > > > > > > > for
> > > > > > > > > 0x3d6a5
> > > > > > > > > (member)!"
> > > > > > > > > I am currently investigating whether these issues are
> > > > > > > > > related.
> > > > > > > > > Therefore, it takes a bit more time than i expected,
> > > > > > > > > than
> > > > > > > > > just
> > > > > > > > > adding
> > > > > > > > > a
> > > > > > > > > pahole/dwarves package.
> > > > > > > > >
> > > > > > > > > I will update as soon as more information available.
> > > > > > > > >
> > > > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > > > Denys, Thanks for the update. Do you have a
> > > > > > > > > > prediction
> > > > > > > > > > for
> > > > > > > > > > when
> > > > > > > > > > it
> > > > > > > > > > will land?
> > > > > > > > > >
> > > > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys Fedoryshchenko
> > > > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > > > >
> > > > > > > > > > > Also i noticed in some cases lz4 package required
> > > > > > > > > > > for
> > > > > > > > > > > this
> > > > > > > > > > > test,
> > > > > > > > > > > so added it too.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Denys
> > > > > > > > > > >
> > > > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos via
> > > > > > > > > > > groups.io
> > > > > > > > > > > wrote:
> > > > > > > > > > > > We have a bunch of Android kernel test failures
> > > > > > > > > > > > due
> > > > > > > > > > > > to
> > > > > > > > > > > > missing
> > > > > > > > > > > > "pahole" in
> > > > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > > > gki_defconfig
> > > > > > > > > > > >
> > > > > > > > > > > <
> > > > > > >
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > > > >
> > > > > > > > > > > > ):
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > > > available
> > > > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247:
> > > > > > > > > > > > vmlinux]
> > > > > > > > > > > > Error 1
> > > > > > > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > > > >
> > > > > > > > > > > > Would it be possible to get pahole added to the
> > > > > > > > > > > > build
> > > > > > > > > > > environments
> > > > > > > > > > > > used to
> > > > > > > > > > > > build android kernels?
> > > > > > > > > > > >
> > > > > > > > > > > > -Todd
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> > 
> >
> >
>
>
>
>

Re: kernelci failures due to "pahole" missing

[Denys Fedoryshchenko]

Hi,

I think i have good news regarding this issue, which initially
was just adding pahole tool, but turned in a bit long story due many
complications.

Several problems we had to solve:
1)Several packages had to be added to build docker images, we had to
make sure it doesnt break anything else
2)gki_defconfig during build require quite a lot of resources (at least
18Gb of RAM), so we had to allocate this builds to biggest k8s nodes we
have, and on currently allocated budget i had to wait, until there is 
available resources to do test builds, as this nodes often busy.
3)Generated files size exceed internal limits that our current software
frameworks can handle and log file size + kernel size will quickly fill
the storage if left uncompressed, so we had to implement compression
before uploading files to storage

So here is one of results on production, finally:

https://storage.kernelci.org/android/android13-5.15/android13-5.15-2022-08_r1-95-gca7f18ea252e/arm64/gki_defconfig/clang-14/


On Wed, 2022-07-13 at 10:45 -0700, Todd Kjos via groups.io wrote:
> As long as it is easy to access/decompress the logs that's fine.
> 
> On Wed, Jul 13, 2022 at 10:40 AM Denys Fedoryshchenko
> <denys.f@collabora.com>
> wrote:
> 
> > I apologize for the delay, as what was supposed to be a simple task
> > with
> > the addition of a pahole tool turned into a fundamental problem.
> > I hope to be able to solve this problem soon.
> > One of the options I'm considering is compressing Image and
> > kernel.log
> > through xz -9.
> > This whille make build a bit different from standard builds.
> > It is a bit hack, but might be a bit faster than implementing other
> > solutions.
> > Is it acceptable?
> > 
> > Here i pushed in staging almost manually results:
> > https://storage.staging.kernelci.org/kernelci/staging-android/ASB-2022-06-05_13-5.15-46-g15a32b46ee0f4/arm64/gki_defconfig/clang-15/logs/
> > As you can see kernel.log is 813Mbyte and Image is 193Mbyte, which
> > is a
> > bit too much.
> > 
> > On Wed, 2022-07-13 at 10:32 -0700, Todd Kjos wrote:
> > 
> > Denys, thanks for the update. I don't consider this urgent, but I
> > would
> > eventually like to see our "gki_defconfig" builds be green again.
> > 
> > On Tue, Jul 12, 2022 at 2:49 AM Denys Fedoryshchenko <
> > denys.f@collabora.com> wrote:
> > 
> > Hi,
> > 
> > Sorry for the delay, it took longer than this meeting as we faced
> > major
> > issue - current system design cannot handle large objects upload
> > (larger than ~200Mbyte) and there is no easy workaround for that.
> > 
> > Theoretically, the new system architecture (kernelci-api) solves
> > this
> > problem, but it will take several months to finalize it.
> > I'll try to develop a partial upload method (so it will upload file
> > by
> > chunks), but that too will take some time as i have only few hours
> > per
> > week aside from primary project and other developers are also
> > overloaded.
> > Also during discussion it was suggested alternative solution: to
> > run
> > dedicated jenkins job running directly on one of builders, instead
> > of
> > kubernetes, as this will allow us to use scp upload method similar
> > as
> > we do with rootfs builds.
> > But this will also require creation of a completely new job
> > configuration for the Jenkins.
> > 
> > On Fri, 2022-07-08 at 10:02 -0700, Todd Kjos via groups.io wrote:
> > > Any update from your Thursday meeting?
> > > 
> > > On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko
> > > <denys.f@collabora.com>
> > > wrote:
> > > 
> > > > We faced multiple issues.
> > > > 1) Most builders instance not suitable for this job, as it
> > > > require
> > > > large amount of RAM. Found workaround, tested in staging.
> > > > 2) The problem with the build time, it significantly exceeds
> > > > the
> > > > time
> > > > of usual builds. Theoretically, the problem can be solved by
> > > > increasing
> > > > the limits.
> > > > 3) Some tools was missing, thats solved.
> > > > 4) Some of the created objects during build exceed the size
> > > > that
> > > > the
> > > > library that we use to upload objects to storage can handle.
> > > > This
> > > > will
> > > > require significant changes in some parts of project to solve
> > > > and
> > > > major
> > > > blocker for me.
> > > > 
> > > > I'm still not sure what to do next with that, and I'll discuss
> > > > this
> > > > at
> > > > the next weekly meeting (this Thursday).
> > > > 
> > > > On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io
> > > > wrote:
> > > > > Any update on this?
> > > > > 
> > > > > 
> > > > > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > > > > <denys.f@collabora.com> wrote:
> > > > > > 
> > > > > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io
> > > > > > wrote:
> > > > > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > > > > denys.f@collabora.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Update
> > > > > > > > 
> > > > > > > > We identified all packages required for kernel build:
> > > > > > > > dwarves, lz4, python3/python.
> > > > > > > > 
> > > > > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built
> > > > > > > > on
> > > > > > > > our
> > > > > > > > staging
> > > > > > > > pipeline, but with error:
> > > > > > > >  LD      vmlinux
> > > > > > > >   BTFIDS  vmlinux
> > > > > > > > FAILED unresolved symbol xdp_buff
> > > > > > > > 
> > > > > > > 
> > > > > > > Which config and arch was this?
> > > > > > gki_defconfig
> > > > > > arm64
> > > > > > But it is a bit old snapshot i'm using as reference.
> > > > > > Tomorrow i will try latest 5.10 if anything changed.
> > > > > > 
> > > > > > Container Log:
> > > > > >   LD      .tmp_vmlinux.kallsyms1
> > > > > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > > > > >   AS      .tmp_vmlinux.kallsyms1.S
> > > > > >   LD      .tmp_vmlinux.kallsyms2
> > > > > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > > > > >   AS      .tmp_vmlinux.kallsyms2.S
> > > > > >   LD      vmlinux
> > > > > >   BTFIDS  vmlinux
> > > > > > FAILED unresolved symbol xdp_buff
> > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error
> > > > > > 255
> > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > make: *** [Makefile:185: __sub-make] Error 2
> > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > 
> > > > > > "branch": "staging-android",
> > > > > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > > > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > > > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > > > > "tree": "kernelci",
> > > > > > "url": "https://github.com/kernelci/linux.git"
> > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > I will create PR now to allocate larger builders (to
> > > > > > > > address
> > > > > > > > LTO
> > > > > > > > memory
> > > > > > > > overconsumption) and we will do test build on android
> > > > > > > > pipeline.
> > > > > > > > 
> > > > > > > 
> > > > > > > Thanks for helping with this!
> > > > > > > 
> > > > > > > 
> > > > > > > > 
> > > > > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko
> > > > > > > > wrote:
> > > > > > > > > Short update.
> > > > > > > > > Some of builders crashed due OOM, but not because of
> > > > > > > > > logs,
> > > > > > > > > but because LTO during linking phase consume more
> > > > > > > > > than
> > > > > > > > > 21Gb
> > > > > > > > > of
> > > > > > > > > RAM,
> > > > > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > > > > allocated.
> > > > > > > > > It is not easy to solve, as it is either require to
> > > > > > > > > change
> > > > > > > > > LTO to
> > > > > > > > > "Thin
> > > > > > > > > LTO" (or disable it totally) to reduce memory
> > > > > > > > > consumption,
> > > > > > > > > which
> > > > > > > > > is
> > > > > > > > > not
> > > > > > > > > great, or increase builders size (and that means
> > > > > > > > > additional
> > > > > > > > > costs
> > > > > > > > > and
> > > > > > > > > approvals related to that).
> > > > > > > > > We are discussing on IRC possible solutions to this
> > > > > > > > > problem.
> > > > > > > > > 
> > > > > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys
> > > > > > > > > Fedoryshchenko
> > > > > > > > > wrote:
> > > > > > > > > > At the moment I have identified a problem with more
> > > > > > > > > > packages
> > > > > > > > > > that
> > > > > > > > > > are
> > > > > > > > > > missing, now it is python, and i'm adding it as
> > > > > > > > > > well,
> > > > > > > > > > but
> > > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > test
> > > > > > > > > > everything manually, and then on staging.
> > > > > > > > > > 
> > > > > > > > > > Here is example of build that needs python:
> > > > > > > > > > 
> > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > > > > > 
> > > > > > > > 
> > > > 
> > https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > > > 
> > > > > > > > > > Also, another problem some of the builders crashed,
> > > > > > > > > > probably
> > > > > > > > > > due to
> > > > > > > > > > OOM
> > > > > > > > > > or insufficient resources.
> > > > > > > > > > 
> > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > > > > > 
> > > > > > > > 
> > > > 
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > > > 
> > > > > > > > > > On manual tests, builds with pahole create about
> > > > > > > > > > 819Mbyte
> > > > > > > > > > of
> > > > > > > > > > warnings,
> > > > > > > > > > such as:
> > > > > > > > > > "namespace__recode_dwarf_types: couldn't find
> > > > > > > > > > 0x13423
> > > > > > > > > > type
> > > > > > > > > > for
> > > > > > > > > > 0x3d6a5
> > > > > > > > > > (member)!"
> > > > > > > > > > I am currently investigating whether these issues
> > > > > > > > > > are
> > > > > > > > > > related.
> > > > > > > > > > Therefore, it takes a bit more time than i
> > > > > > > > > > expected,
> > > > > > > > > > than
> > > > > > > > > > just
> > > > > > > > > > adding
> > > > > > > > > > a
> > > > > > > > > > pahole/dwarves package.
> > > > > > > > > > 
> > > > > > > > > > I will update as soon as more information
> > > > > > > > > > available.
> > > > > > > > > > 
> > > > > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > > > > Denys, Thanks for the update. Do you have a
> > > > > > > > > > > prediction
> > > > > > > > > > > for
> > > > > > > > > > > when
> > > > > > > > > > > it
> > > > > > > > > > > will land?
> > > > > > > > > > > 
> > > > > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys
> > > > > > > > > > > Fedoryshchenko
> > > > > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > 
> > > > > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > > > > https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > > > > > 
> > > > > > > > > > > > Also i noticed in some cases lz4 package
> > > > > > > > > > > > required
> > > > > > > > > > > > for
> > > > > > > > > > > > this
> > > > > > > > > > > > test,
> > > > > > > > > > > > so added it too.
> > > > > > > > > > > > 
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Denys
> > > > > > > > > > > > 
> > > > > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos
> > > > > > > > > > > > via
> > > > > > > > > > > > groups.io
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > We have a bunch of Android kernel test
> > > > > > > > > > > > > failures
> > > > > > > > > > > > > due
> > > > > > > > > > > > > to
> > > > > > > > > > > > > missing
> > > > > > > > > > > > > "pahole" in
> > > > > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > > > > gki_defconfig
> > > > > > > > > > > > > 
> > > > > > > > > > > > <
> > > > > > > > 
> > > > 
> > https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > ):
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > > > > available
> > > > > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247:
> > > > > > > > > > > > > vmlinux]
> > > > > > > > > > > > > Error 1
> > > > > > > > > > > > > make[1]: Leaving directory
> > > > > > > > > > > > > '/tmp/kci/linux/build'
> > > > > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Would it be possible to get pahole added to
> > > > > > > > > > > > > the
> > > > > > > > > > > > > build
> > > > > > > > > > > > environments
> > > > > > > > > > > > > used to
> > > > > > > > > > > > > build android kernels?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > -Todd
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > 
> > 
> > 
> > 
> 
> 
> 
> 
> 

Re: kernelci failures due to "pahole" missing

[Todd Kjos]

That's great news! It's really nice to see our gki_defconfig builds
succeeding now. Thanks.

On Tue, Sep 6, 2022 at 5:53 AM Denys Fedoryshchenko <denys.f@collabora.com>
wrote:

> Hi,
>
> I think i have good news regarding this issue, which initially
> was just adding pahole tool, but turned in a bit long story due many
> complications.
>
> Several problems we had to solve:
> 1)Several packages had to be added to build docker images, we had to
> make sure it doesnt break anything else
> 2)gki_defconfig during build require quite a lot of resources (at least
> 18Gb of RAM), so we had to allocate this builds to biggest k8s nodes we
> have, and on currently allocated budget i had to wait, until there is
> available resources to do test builds, as this nodes often busy.
> 3)Generated files size exceed internal limits that our current software
> frameworks can handle and log file size + kernel size will quickly fill
> the storage if left uncompressed, so we had to implement compression
> before uploading files to storage
>
> So here is one of results on production, finally:
>
>
> https://storage.kernelci.org/android/android13-5.15/android13-5.15-2022-08_r1-95-gca7f18ea252e/arm64/gki_defconfig/clang-14/
>
>
> On Wed, 2022-07-13 at 10:45 -0700, Todd Kjos via groups.io wrote:
> > As long as it is easy to access/decompress the logs that's fine.
> >
> > On Wed, Jul 13, 2022 at 10:40 AM Denys Fedoryshchenko
> > <denys.f@collabora.com>
> > wrote:
> >
> > > I apologize for the delay, as what was supposed to be a simple task
> > > with
> > > the addition of a pahole tool turned into a fundamental problem.
> > > I hope to be able to solve this problem soon.
> > > One of the options I'm considering is compressing Image and
> > > kernel.log
> > > through xz -9.
> > > This whille make build a bit different from standard builds.
> > > It is a bit hack, but might be a bit faster than implementing other
> > > solutions.
> > > Is it acceptable?
> > >
> > > Here i pushed in staging almost manually results:
> > >
> https://storage.staging.kernelci.org/kernelci/staging-android/ASB-2022-06-05_13-5.15-46-g15a32b46ee0f4/arm64/gki_defconfig/clang-15/logs/
> > > As you can see kernel.log is 813Mbyte and Image is 193Mbyte, which
> > > is a
> > > bit too much.
> > >
> > > On Wed, 2022-07-13 at 10:32 -0700, Todd Kjos wrote:
> > >
> > > Denys, thanks for the update. I don't consider this urgent, but I
> > > would
> > > eventually like to see our "gki_defconfig" builds be green again.
> > >
> > > On Tue, Jul 12, 2022 at 2:49 AM Denys Fedoryshchenko <
> > > denys.f@collabora.com> wrote:
> > >
> > > Hi,
> > >
> > > Sorry for the delay, it took longer than this meeting as we faced
> > > major
> > > issue - current system design cannot handle large objects upload
> > > (larger than ~200Mbyte) and there is no easy workaround for that.
> > >
> > > Theoretically, the new system architecture (kernelci-api) solves
> > > this
> > > problem, but it will take several months to finalize it.
> > > I'll try to develop a partial upload method (so it will upload file
> > > by
> > > chunks), but that too will take some time as i have only few hours
> > > per
> > > week aside from primary project and other developers are also
> > > overloaded.
> > > Also during discussion it was suggested alternative solution: to
> > > run
> > > dedicated jenkins job running directly on one of builders, instead
> > > of
> > > kubernetes, as this will allow us to use scp upload method similar
> > > as
> > > we do with rootfs builds.
> > > But this will also require creation of a completely new job
> > > configuration for the Jenkins.
> > >
> > > On Fri, 2022-07-08 at 10:02 -0700, Todd Kjos via groups.io wrote:
> > > > Any update from your Thursday meeting?
> > > >
> > > > On Wed, Jun 29, 2022 at 4:27 AM Denys Fedoryshchenko
> > > > <denys.f@collabora.com>
> > > > wrote:
> > > >
> > > > > We faced multiple issues.
> > > > > 1) Most builders instance not suitable for this job, as it
> > > > > require
> > > > > large amount of RAM. Found workaround, tested in staging.
> > > > > 2) The problem with the build time, it significantly exceeds
> > > > > the
> > > > > time
> > > > > of usual builds. Theoretically, the problem can be solved by
> > > > > increasing
> > > > > the limits.
> > > > > 3) Some tools was missing, thats solved.
> > > > > 4) Some of the created objects during build exceed the size
> > > > > that
> > > > > the
> > > > > library that we use to upload objects to storage can handle.
> > > > > This
> > > > > will
> > > > > require significant changes in some parts of project to solve
> > > > > and
> > > > > major
> > > > > blocker for me.
> > > > >
> > > > > I'm still not sure what to do next with that, and I'll discuss
> > > > > this
> > > > > at
> > > > > the next weekly meeting (this Thursday).
> > > > >
> > > > > On Tue, 2022-06-28 at 12:16 -0700, Todd Kjos via groups.io
> > > > > wrote:
> > > > > > Any update on this?
> > > > > >
> > > > > >
> > > > > > On Tue, May 10, 2022 at 12:50 PM Denys Fedoryshchenko
> > > > > > <denys.f@collabora.com> wrote:
> > > > > > >
> > > > > > > On Tue, 2022-05-10 at 10:53 -0700, Todd Kjos via groups.io
> > > > > > > wrote:
> > > > > > > > On Tue, May 10, 2022 at 12:20 AM Denys Fedoryshchenko <
> > > > > > > > denys.f@collabora.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Update
> > > > > > > > >
> > > > > > > > > We identified all packages required for kernel build:
> > > > > > > > > dwarves, lz4, python3/python.
> > > > > > > > >
> > > > > > > > > We got ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1 built
> > > > > > > > > on
> > > > > > > > > our
> > > > > > > > > staging
> > > > > > > > > pipeline, but with error:
> > > > > > > > >  LD      vmlinux
> > > > > > > > >   BTFIDS  vmlinux
> > > > > > > > > FAILED unresolved symbol xdp_buff
> > > > > > > > >
> > > > > > > >
> > > > > > > > Which config and arch was this?
> > > > > > > gki_defconfig
> > > > > > > arm64
> > > > > > > But it is a bit old snapshot i'm using as reference.
> > > > > > > Tomorrow i will try latest 5.10 if anything changed.
> > > > > > >
> > > > > > > Container Log:
> > > > > > >   LD      .tmp_vmlinux.kallsyms1
> > > > > > >   KSYMS   .tmp_vmlinux.kallsyms1.S
> > > > > > >   AS      .tmp_vmlinux.kallsyms1.S
> > > > > > >   LD      .tmp_vmlinux.kallsyms2
> > > > > > >   KSYMS   .tmp_vmlinux.kallsyms2.S
> > > > > > >   AS      .tmp_vmlinux.kallsyms2.S
> > > > > > >   LD      vmlinux
> > > > > > >   BTFIDS  vmlinux
> > > > > > > FAILED unresolved symbol xdp_buff
> > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1267: vmlinux] Error
> > > > > > > 255
> > > > > > > make[1]: Leaving directory '/tmp/kci/linux/build'
> > > > > > > make: *** [Makefile:185: __sub-make] Error 2
> > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > >
> > > > > > > "branch": "staging-android",
> > > > > > > "commit": "4dce9d7a65b14d5949b20ced210413d04d914053",
> > > > > > > "describe": "ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1",
> > > > > > > "describe_verbose": "v5.10-19504-g4dce9d7a65b1",
> > > > > > > "tree": "kernelci",
> > > > > > > "url": "https://github.com/kernelci/linux.git"
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > I will create PR now to allocate larger builders (to
> > > > > > > > > address
> > > > > > > > > LTO
> > > > > > > > > memory
> > > > > > > > > overconsumption) and we will do test build on android
> > > > > > > > > pipeline.
> > > > > > > > >
> > > > > > > >
> > > > > > > > Thanks for helping with this!
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, 2022-05-05 at 08:23 +0300, Denys Fedoryshchenko
> > > > > > > > > wrote:
> > > > > > > > > > Short update.
> > > > > > > > > > Some of builders crashed due OOM, but not because of
> > > > > > > > > > logs,
> > > > > > > > > > but because LTO during linking phase consume more
> > > > > > > > > > than
> > > > > > > > > > 21Gb
> > > > > > > > > > of
> > > > > > > > > > RAM,
> > > > > > > > > > and k8s builders allocated for project have only 4Gb
> > > > > > > > > > allocated.
> > > > > > > > > > It is not easy to solve, as it is either require to
> > > > > > > > > > change
> > > > > > > > > > LTO to
> > > > > > > > > > "Thin
> > > > > > > > > > LTO" (or disable it totally) to reduce memory
> > > > > > > > > > consumption,
> > > > > > > > > > which
> > > > > > > > > > is
> > > > > > > > > > not
> > > > > > > > > > great, or increase builders size (and that means
> > > > > > > > > > additional
> > > > > > > > > > costs
> > > > > > > > > > and
> > > > > > > > > > approvals related to that).
> > > > > > > > > > We are discussing on IRC possible solutions to this
> > > > > > > > > > problem.
> > > > > > > > > >
> > > > > > > > > > On Wed, 2022-05-04 at 12:06 +0300, Denys
> > > > > > > > > > Fedoryshchenko
> > > > > > > > > > wrote:
> > > > > > > > > > > At the moment I have identified a problem with more
> > > > > > > > > > > packages
> > > > > > > > > > > that
> > > > > > > > > > > are
> > > > > > > > > > > missing, now it is python, and i'm adding it as
> > > > > > > > > > > well,
> > > > > > > > > > > but
> > > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > test
> > > > > > > > > > > everything manually, and then on staging.
> > > > > > > > > > >
> > > > > > > > > > > Here is example of build that needs python:
> > > > > > > > > > >
> > > https://linux.kernelci.org/build/id/6271d9f8c49e05f410dc7b53/
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > >
> https://storage.kernelci.org/android/android13-5.10/ASB-2022-04-05_13-5.10-206-g4dce9d7a65b1/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > > > >
> > > > > > > > > > > Also, another problem some of the builders crashed,
> > > > > > > > > > > probably
> > > > > > > > > > > due to
> > > > > > > > > > > OOM
> > > > > > > > > > > or insufficient resources.
> > > > > > > > > > >
> > > https://linux.kernelci.org/build/id/6271db3a3c249a41a7dc7b48/
> > > > > > > > > > >
> > > > > > > > >
> > > > >
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1526-g22e9215f1bea/arm64/gki_defconfig/clang-14/logs/kernel.log
> > > > > > > > > > >
> > > > > > > > > > > On manual tests, builds with pahole create about
> > > > > > > > > > > 819Mbyte
> > > > > > > > > > > of
> > > > > > > > > > > warnings,
> > > > > > > > > > > such as:
> > > > > > > > > > > "namespace__recode_dwarf_types: couldn't find
> > > > > > > > > > > 0x13423
> > > > > > > > > > > type
> > > > > > > > > > > for
> > > > > > > > > > > 0x3d6a5
> > > > > > > > > > > (member)!"
> > > > > > > > > > > I am currently investigating whether these issues
> > > > > > > > > > > are
> > > > > > > > > > > related.
> > > > > > > > > > > Therefore, it takes a bit more time than i
> > > > > > > > > > > expected,
> > > > > > > > > > > than
> > > > > > > > > > > just
> > > > > > > > > > > adding
> > > > > > > > > > > a
> > > > > > > > > > > pahole/dwarves package.
> > > > > > > > > > >
> > > > > > > > > > > I will update as soon as more information
> > > > > > > > > > > available.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 2022-05-03 at 07:45 -0700, Todd Kjos wrote:
> > > > > > > > > > > > Denys, Thanks for the update. Do you have a
> > > > > > > > > > > > prediction
> > > > > > > > > > > > for
> > > > > > > > > > > > when
> > > > > > > > > > > > it
> > > > > > > > > > > > will land?
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, May 2, 2022 at 9:58 PM Denys
> > > > > > > > > > > > Fedoryshchenko
> > > > > > > > > > > > <denys.f@collabora.com> wrote:
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > We are testing in staging fix for that:
> > > > > > > > > > > > >
> https://github.com/kernelci/kernelci-core/pull/1158
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also i noticed in some cases lz4 package
> > > > > > > > > > > > > required
> > > > > > > > > > > > > for
> > > > > > > > > > > > > this
> > > > > > > > > > > > > test,
> > > > > > > > > > > > > so added it too.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Denys
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, 2022-04-29 at 15:22 -0700, Todd Kjos
> > > > > > > > > > > > > via
> > > > > > > > > > > > > groups.io
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > We have a bunch of Android kernel test
> > > > > > > > > > > > > > failures
> > > > > > > > > > > > > > due
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > missing
> > > > > > > > > > > > > > "pahole" in
> > > > > > > > > > > > > > the build environment (example android13-5.15
> > > > > > > > > > > > > > gki_defconfig
> > > > > > > > > > > > > >
> > > > > > > > > > > > > <
> > > > > > > > >
> > > > >
> > >
> https://storage.kernelci.org/android/android13-5.15/ASB-2022-04-05_13-5.15-1515-g7a68c2c45139/arm64/gki_defconfig/gcc-10/logs/kernel.log
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > ):
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >   LD      vmlinux.o
> > > > > > > > > > > > > >   MODPOST vmlinux.symvers
> > > > > > > > > > > > > >   MODINFO modules.builtin.modinfo
> > > > > > > > > > > > > >   GEN     modules.builtin
> > > > > > > > > > > > > > BTF: .tmp_vmlinux.btf: pahole (pahole) is not
> > > > > > > > > > > > > > available
> > > > > > > > > > > > > > Failed to generate BTF for vmlinux
> > > > > > > > > > > > > > Try to disable CONFIG_DEBUG_INFO_BTF
> > > > > > > > > > > > > > make[1]: *** [/tmp/kci/linux/Makefile:1247:
> > > > > > > > > > > > > > vmlinux]
> > > > > > > > > > > > > > Error 1
> > > > > > > > > > > > > > make[1]: Leaving directory
> > > > > > > > > > > > > > '/tmp/kci/linux/build'
> > > > > > > > > > > > > > make: *** [Makefile:237: __sub-make] Error 2
> > > > > > > > > > > > > > make: Leaving directory '/tmp/kci/linux'
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Would it be possible to get pahole added to
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > build
> > > > > > > > > > > > > environments
> > > > > > > > > > > > > > used to
> > > > > > > > > > > > > > build android kernels?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -Todd
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> > 
> >
> >
>
>
>