[PATCH] kernel.bbclass: Set pkg-config variables for building modules

[PATCH] kernel.bbclass: Set pkg-config variables for building modules

[Munehisa Kamata]

The pkg-config workaround has been applied for kernel image building, but
not for module building. On x86, this can trigger rebuilding of objtool
unnecessary at do_compile_kernelmodules task. If the kernel is built with
CONFIG_DEBUG_INFO_BTF=y, the task even rebuilds vmlinux and leads to
inconsistent build-id between bzImage and vmlinux, and also BTF mismatches
between the bzImage and modules.

To avoid the inconsistency, apply the same workaround when building
modules. For kernels 5.19+, simply set HOSTPKG_CONFIG in the make command
line.

Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
---
 meta/classes-recipe/kernel.bbclass | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index a76aaee5ba..db4461e551 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
 EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
 EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
 EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
+# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
+EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
 
 KERNEL_ALT_IMAGETYPE ??= ""
 
@@ -356,9 +358,6 @@ kernel_do_compile() {
 	export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
 	export PKG_CONFIG_SYSROOT_DIR=""
 
-	# for newer kernels (5.19+) there's a dedicated variable
-	export HOSTPKG_CONFIG="pkg-config-native"
-
 	if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
 		# kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
 		# be set....
@@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
 
 do_compile_kernelmodules() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
+
+	# setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
+	export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
+	export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
+	export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
+	export PKG_CONFIG_SYSROOT_DIR=""
+
 	if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
 		# kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
 		# be set....
-- 
2.34.1

Re: [OE-core] [PATCH] kernel.bbclass: Set pkg-config variables for building modules

[Bruce Ashfield]

On Thu, Feb 22, 2024 at 3:28 PM Munehisa Kamata via
lists.openembedded.org <kamatam=amazon.com@lists.openembedded.org>
wrote:
>
> The pkg-config workaround has been applied for kernel image building, but
> not for module building. On x86, this can trigger rebuilding of objtool
> unnecessary at do_compile_kernelmodules task. If the kernel is built with
> CONFIG_DEBUG_INFO_BTF=y, the task even rebuilds vmlinux and leads to
> inconsistent build-id between bzImage and vmlinux, and also BTF mismatches
> between the bzImage and modules.

I don't question the change, as it is what I had to do for the mail
kernel build.

What I would like to see logged is a set of steps that managed to trigger the
issue.

Since the module builds are dependent on the main kernel, everything that they
need should be available and already timestamped to not be required, so the
missing pkconfig shouldn't trigger much.

That way we'll know if there's a gap somewhere in the dependencies, or another
race condition and we can test for it.

Bruce

>
> To avoid the inconsistency, apply the same workaround when building
> modules. For kernels 5.19+, simply set HOSTPKG_CONFIG in the make command
> line.
>
> Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
> ---
>  meta/classes-recipe/kernel.bbclass | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
> index a76aaee5ba..db4461e551 100644
> --- a/meta/classes-recipe/kernel.bbclass
> +++ b/meta/classes-recipe/kernel.bbclass
> @@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
>  EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
>  EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
>  EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
> +# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
> +EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
>
>  KERNEL_ALT_IMAGETYPE ??= ""
>
> @@ -356,9 +358,6 @@ kernel_do_compile() {
>         export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
>         export PKG_CONFIG_SYSROOT_DIR=""
>
> -       # for newer kernels (5.19+) there's a dedicated variable
> -       export HOSTPKG_CONFIG="pkg-config-native"
> -
>         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
>                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
>                 # be set....
> @@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
>
>  do_compile_kernelmodules() {
>         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
> +
> +       # setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
> +       export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
> +       export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
> +       export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> +       export PKG_CONFIG_SYSROOT_DIR=""
> +
>         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
>                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
>                 # be set....
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#196041): https://lists.openembedded.org/g/openembedded-core/message/196041
> Mute This Topic: https://lists.openembedded.org/mt/104516377/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

[PATCH v2] kernel.bbclass: Set pkg-config variables for building modules

[Munehisa Kamata]

The pkg-config workaround has been applied for kernel image building, but
not for module building. So pkg-config variables are different between
do_compile and do_compile_kernelmodules tasks. It may unnecessary trigger
rebuilding of a few host tools at the later task.

Especially when CONFIG_DEBUG_INFO_BTF is enabled in the kernel, it may even
trigger rebuilding vmlinux at do_compile_kernelmodules due to the rebuilt
host tools such as certs/extract-cert or objtool (on x86). This eventually
creates an inconsistent set of kernel binaries.

Here is the repro steps:

 - Check out nanbield on x86
   - The unexpected rebuild happens on kirkstone or possibly earlier

 - Ensure that pahole is available (e.g. via meta-oe)

 - Set KERNEL_DEBUG to "True" to properly set up PAHOLE
   e.g.
   $ export KERNEL_DEBUG="True"
   $ export BB_ENV_PASSTHROUGH_ADDITIONS="${BB_ENV_PASSTHROUGH_ADDITIONS} KERNEL_DEBUG"

 - Enable CONFIG_DEBUG_INFO_BTF=y
   e.g.
   $ bitbake -c menuconfig virtual/kernel
    -> Kernel hacking
      -> Compile-time checks and compiler options
        -> Generate BTF typeinfo

 - Build the kernel
   e.g.
   $ bitbake virtual/kernel

The BTF information in the resulting bzImage and kernel modules are
inconsistent, because the module's BTF information is generated using the
"second" vmlinux that doesn't have the identical BTF to the "first" vmlinux.
These modules can't be loaded at runtime due to the BTF mismatch.

This also leads to a build-id mismatch between the installed bzImage and
vmlinux since the bzImage is created from the first vmlinux, but the
installed vmlinux is the second one.

  $ eu-readelf -n tmp/work/qemux86_64-poky-linux/linux-yocto/6.5.13+git/image/boot/{bzImage*,vmlinux*} | grep "Build ID"
   Build ID: 4a0d62ee7fef0244950f0f604253729875bea493
   Build ID: fb99b3d91399dbe42bf67ddee59e0f5a0c7f74d9

To avoid the unexpected rebuilding that results in such inconsistency, set
the same pkg-config variables when building kernel and modules. For kernel
5.19 and above, simply set the HOSTPKG_CONFIG in the make command line.

Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
---

v1 -> v2: Rewrote the commit message with the repro steps

 meta/classes-recipe/kernel.bbclass | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index a76aaee5ba..db4461e551 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
 EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
 EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
 EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
+# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
+EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
 
 KERNEL_ALT_IMAGETYPE ??= ""
 
@@ -356,9 +358,6 @@ kernel_do_compile() {
 	export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
 	export PKG_CONFIG_SYSROOT_DIR=""
 
-	# for newer kernels (5.19+) there's a dedicated variable
-	export HOSTPKG_CONFIG="pkg-config-native"
-
 	if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
 		# kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
 		# be set....
@@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
 
 do_compile_kernelmodules() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
+
+	# setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
+	export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
+	export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
+	export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
+	export PKG_CONFIG_SYSROOT_DIR=""
+
 	if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
 		# kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
 		# be set....
-- 
2.34.1

Re: [PATCH v2] kernel.bbclass: Set pkg-config variables for building modules

[Bruce Ashfield]

On Fri, Feb 23, 2024 at 3:05 AM Munehisa Kamata <kamatam@amazon.com> wrote:
>
> The pkg-config workaround has been applied for kernel image building, but
> not for module building. So pkg-config variables are different between
> do_compile and do_compile_kernelmodules tasks. It may unnecessary trigger
> rebuilding of a few host tools at the later task.
>
> Especially when CONFIG_DEBUG_INFO_BTF is enabled in the kernel, it may even
> trigger rebuilding vmlinux at do_compile_kernelmodules due to the rebuilt
> host tools such as certs/extract-cert or objtool (on x86). This eventually
> creates an inconsistent set of kernel binaries.
>
> Here is the repro steps:
>
>  - Check out nanbield on x86
>    - The unexpected rebuild happens on kirkstone or possibly earlier
>
>  - Ensure that pahole is available (e.g. via meta-oe)
>
>  - Set KERNEL_DEBUG to "True" to properly set up PAHOLE
>    e.g.
>    $ export KERNEL_DEBUG="True"
>    $ export BB_ENV_PASSTHROUGH_ADDITIONS="${BB_ENV_PASSTHROUGH_ADDITIONS} KERNEL_DEBUG"
>
>  - Enable CONFIG_DEBUG_INFO_BTF=y
>    e.g.
>    $ bitbake -c menuconfig virtual/kernel
>     -> Kernel hacking
>       -> Compile-time checks and compiler options
>         -> Generate BTF typeinfo
>
>  - Build the kernel
>    e.g.
>    $ bitbake virtual/kernel
>
> The BTF information in the resulting bzImage and kernel modules are
> inconsistent, because the module's BTF information is generated using the
> "second" vmlinux that doesn't have the identical BTF to the "first" vmlinux.
> These modules can't be loaded at runtime due to the BTF mismatch.
>
> This also leads to a build-id mismatch between the installed bzImage and
> vmlinux since the bzImage is created from the first vmlinux, but the
> installed vmlinux is the second one.
>
>   $ eu-readelf -n tmp/work/qemux86_64-poky-linux/linux-yocto/6.5.13+git/image/boot/{bzImage*,vmlinux*} | grep "Build ID"
>    Build ID: 4a0d62ee7fef0244950f0f604253729875bea493
>    Build ID: fb99b3d91399dbe42bf67ddee59e0f5a0c7f74d9
>
> To avoid the unexpected rebuilding that results in such inconsistency, set
> the same pkg-config variables when building kernel and modules. For kernel
> 5.19 and above, simply set the HOSTPKG_CONFIG in the make command line.

That is indeed not a simple workflow!

In the past, we've always had the existing packageconfig results picked up and
used by later stages of the kernel build which prevented things like this from
happening.

Have you figured out exactly which packageconfig is triggering the rebuild, and
had a look to see if something change such that the existing results
aren't used ?

All that being said, rather than repeating the exports, it would be
better to have
them in a common place, since eventually everything but the HOSTPKG_CONFIG
definition will be dropped.

Have you tried a more class-global export of the values ? We have a
few of those,
but admittedly, we have way more exports that are duplicated between do_compile
and do_compile_kernelmodules(), so the duplicated exports aren't a big issue.

Bruce

>
> Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
> ---
>
> v1 -> v2: Rewrote the commit message with the repro steps
>
>  meta/classes-recipe/kernel.bbclass | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
> index a76aaee5ba..db4461e551 100644
> --- a/meta/classes-recipe/kernel.bbclass
> +++ b/meta/classes-recipe/kernel.bbclass
> @@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
>  EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
>  EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
>  EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
> +# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
> +EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
>
>  KERNEL_ALT_IMAGETYPE ??= ""
>
> @@ -356,9 +358,6 @@ kernel_do_compile() {
>         export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
>         export PKG_CONFIG_SYSROOT_DIR=""
>
> -       # for newer kernels (5.19+) there's a dedicated variable
> -       export HOSTPKG_CONFIG="pkg-config-native"
> -
>         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
>                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
>                 # be set....
> @@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
>
>  do_compile_kernelmodules() {
>         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
> +
> +       # setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
> +       export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
> +       export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
> +       export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> +       export PKG_CONFIG_SYSROOT_DIR=""
> +
>         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
>                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
>                 # be set....
> --
> 2.34.1
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH v2] kernel.bbclass: Set pkg-config variables for building modules

[Munehisa Kamata]

Hi Bruce,

> That is indeed not a simple workflow!
> 
> In the past, we've always had the existing packageconfig results picked up and
> used by later stages of the kernel build which prevented things like this from
> happening.
> 
> Have you figured out exactly which packageconfig is triggering the rebuild, and
> had a look to see if something change such that the existing results
> aren't used ?

The missing of libcrypto.pc and libelf.pc triggers the rebuild of
certs/extract-cert and objtool respectively. When PKG_CONFIG_DIR is set to
recipe-sysroot at module build time, it points to a non-existent directory
and therefore pkg-config can't find .pc for the requested library whereas
they are present in the recipe-sysroot-native, then it causes make to
rebuild the target.

Although I admit that I don't fully understand how make particularly
decides to trigger the rebuild in this case.

> All that being said, rather than repeating the exports, it would be
> better to have
> them in a common place, since eventually everything but the HOSTPKG_CONFIG
> definition will be dropped.
> 
> Have you tried a more class-global export of the values ? We have a
> few of those,
> but admittedly, we have way more exports that are duplicated between do_compile
> and do_compile_kernelmodules(), so the duplicated exports aren't a big issue.

Yes, I actually tried it first and it worked at lesat for simple kernel
building. But I wasn't entirely sure if it was safe enough for the
ecosystem and came up with the change that is hopefully less impact. That
said, if you think it's fine, I'm happy to submit v3 with the class-global
export. It would be better indeed.


Thanks,
Munehisa

> Bruce
> 
> >
> > Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
> > ---
> >
> > v1 -> v2: Rewrote the commit message with the repro steps
> >
> >  meta/classes-recipe/kernel.bbclass | 12 +++++++++---
> >  1 file changed, 9 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
> > index a76aaee5ba..db4461e551 100644
> > --- a/meta/classes-recipe/kernel.bbclass
> > +++ b/meta/classes-recipe/kernel.bbclass
> > @@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
> >  EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
> >  EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
> >  EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
> > +# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
> > +EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
> >
> >  KERNEL_ALT_IMAGETYPE ??= ""
> >
> > @@ -356,9 +358,6 @@ kernel_do_compile() {
> >         export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> >         export PKG_CONFIG_SYSROOT_DIR=""
> >
> > -       # for newer kernels (5.19+) there's a dedicated variable
> > -       export HOSTPKG_CONFIG="pkg-config-native"
> > -
> >         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
> >                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
> >                 # be set....
> > @@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
> >
> >  do_compile_kernelmodules() {
> >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
> > +
> > +       # setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
> > +       export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
> > +       export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
> > +       export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> > +       export PKG_CONFIG_SYSROOT_DIR=""
> > +
> >         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
> >                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
> >                 # be set....
> > --
> > 2.34.1
> >
> 
> 
> -- 
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
> 
> 

Re: [PATCH v2] kernel.bbclass: Set pkg-config variables for building modules

[Bruce Ashfield]

On Sat, Feb 24, 2024 at 2:21 AM Munehisa Kamata <kamatam@amazon.com> wrote:
>
> Hi Bruce,
>
> > That is indeed not a simple workflow!
> >
> > In the past, we've always had the existing packageconfig results picked up and
> > used by later stages of the kernel build which prevented things like this from
> > happening.
> >
> > Have you figured out exactly which packageconfig is triggering the rebuild, and
> > had a look to see if something change such that the existing results
> > aren't used ?
>
> The missing of libcrypto.pc and libelf.pc triggers the rebuild of
> certs/extract-cert and objtool respectively. When PKG_CONFIG_DIR is set to
> recipe-sysroot at module build time, it points to a non-existent directory
> and therefore pkg-config can't find .pc for the requested library whereas
> they are present in the recipe-sysroot-native, then it causes make to
> rebuild the target.
>
> Although I admit that I don't fully understand how make particularly
> decides to trigger the rebuild in this case.

Fair enough. I can't say that I always understand how the kernel's
Makefiles are triggering difficult builds without significant hacking
up of the Makefiles, and it isn't worth doing for this.

At a minimum, it is good to know that the .pc files are being consulted
so we do have something reproducible between builds when everything
is full defined.

>
> > All that being said, rather than repeating the exports, it would be
> > better to have
> > them in a common place, since eventually everything but the HOSTPKG_CONFIG
> > definition will be dropped.
> >
> > Have you tried a more class-global export of the values ? We have a
> > few of those,
> > but admittedly, we have way more exports that are duplicated between do_compile
> > and do_compile_kernelmodules(), so the duplicated exports aren't a big issue.
>
> Yes, I actually tried it first and it worked at lesat for simple kernel
> building. But I wasn't entirely sure if it was safe enough for the
> ecosystem and came up with the change that is hopefully less impact. That
> said, if you think it's fine, I'm happy to submit v3 with the class-global
> export. It would be better indeed.
>

It is something that should be safe, since really, whenever we build
anything in the kernel having a consistent environment should be in
place. The kernel modules build has expanded over the years and
more of the kernel tools and common build components are coming
into play, so things we didn't previously need .. are now needed in
the modules compilation.

I had recommended that this go into test on Friday via IRC, but
there's no harm in doing a cleaned up v3 with a class global set
of exports. We can always apply it post release if it is decided there
is extra risk .. but at least it will be a reference to how we should
start consolidating some of the exports.

Bruce

>
> Thanks,
> Munehisa
>
> > Bruce
> >
> > >
> > > Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
> > > ---
> > >
> > > v1 -> v2: Rewrote the commit message with the repro steps
> > >
> > >  meta/classes-recipe/kernel.bbclass | 12 +++++++++---
> > >  1 file changed, 9 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
> > > index a76aaee5ba..db4461e551 100644
> > > --- a/meta/classes-recipe/kernel.bbclass
> > > +++ b/meta/classes-recipe/kernel.bbclass
> > > @@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
> > >  EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
> > >  EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
> > >  EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
> > > +# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
> > > +EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
> > >
> > >  KERNEL_ALT_IMAGETYPE ??= ""
> > >
> > > @@ -356,9 +358,6 @@ kernel_do_compile() {
> > >         export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> > >         export PKG_CONFIG_SYSROOT_DIR=""
> > >
> > > -       # for newer kernels (5.19+) there's a dedicated variable
> > > -       export HOSTPKG_CONFIG="pkg-config-native"
> > > -
> > >         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
> > >                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
> > >                 # be set....
> > > @@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
> > >
> > >  do_compile_kernelmodules() {
> > >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
> > > +
> > > +       # setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
> > > +       export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
> > > +       export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
> > > +       export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> > > +       export PKG_CONFIG_SYSROOT_DIR=""
> > > +
> > >         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
> > >                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
> > >                 # be set....
> > > --
> > > 2.34.1
> > >
> >
> >
> > --
> > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > thee at its end
> > - "Use the force Harry" - Gandalf, Star Trek II
> >
> >



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH v2] kernel.bbclass: Set pkg-config variables for building modules

[Munehisa Kamata]

Hi Bruce,

On Sun, 2024-02-25 06:54:58 -0800, Bruce Ashfield wrote:
>
> On Sat, Feb 24, 2024 at 2:21 AM Munehisa Kamata <kamatam@amazon.com> wrote:
> >
> > Hi Bruce,
> >
> > > That is indeed not a simple workflow!
> > >
> > > In the past, we've always had the existing packageconfig results picked up and
> > > used by later stages of the kernel build which prevented things like this from
> > > happening.
> > >
> > > Have you figured out exactly which packageconfig is triggering the rebuild, and
> > > had a look to see if something change such that the existing results
> > > aren't used ?
> >
> > The missing of libcrypto.pc and libelf.pc triggers the rebuild of
> > certs/extract-cert and objtool respectively. When PKG_CONFIG_DIR is set to
> > recipe-sysroot at module build time, it points to a non-existent directory
> > and therefore pkg-config can't find .pc for the requested library whereas
> > they are present in the recipe-sysroot-native, then it causes make to
> > rebuild the target.
> >
> > Although I admit that I don't fully understand how make particularly
> > decides to trigger the rebuild in this case.
> 
> Fair enough. I can't say that I always understand how the kernel's
> Makefiles are triggering difficult builds without significant hacking
> up of the Makefiles, and it isn't worth doing for this.
> 
> At a minimum, it is good to know that the .pc files are being consulted
> so we do have something reproducible between builds when everything
> is full defined.
> 
> >
> > > All that being said, rather than repeating the exports, it would be
> > > better to have
> > > them in a common place, since eventually everything but the HOSTPKG_CONFIG
> > > definition will be dropped.
> > >
> > > Have you tried a more class-global export of the values ? We have a
> > > few of those,
> > > but admittedly, we have way more exports that are duplicated between do_compile
> > > and do_compile_kernelmodules(), so the duplicated exports aren't a big issue.
> >
> > Yes, I actually tried it first and it worked at lesat for simple kernel
> > building. But I wasn't entirely sure if it was safe enough for the
> > ecosystem and came up with the change that is hopefully less impact. That
> > said, if you think it's fine, I'm happy to submit v3 with the class-global
> > export. It would be better indeed.
> >
> 
> It is something that should be safe, since really, whenever we build
> anything in the kernel having a consistent environment should be in
> place. The kernel modules build has expanded over the years and
> more of the kernel tools and common build components are coming
> into play, so things we didn't previously need .. are now needed in
> the modules compilation.
> 
> I had recommended that this go into test on Friday via IRC, but
> there's no harm in doing a cleaned up v3 with a class global set
> of exports. We can always apply it post release if it is decided there
> is extra risk .. but at least it will be a reference to how we should
> start consolidating some of the exports.

Since v2 has been already merged, I sent out v3 as just a new patch on the
top of the lastet master branch.


Thanks,
Munehisa
 
> Bruce
> 
> >
> > Thanks,
> > Munehisa
> >
> > > Bruce
> > >
> > > >
> > > > Signed-off-by: Munehisa Kamata <kamatam@amazon.com>
> > > > ---
> > > >
> > > > v1 -> v2: Rewrote the commit message with the repro steps
> > > >
> > > >  meta/classes-recipe/kernel.bbclass | 12 +++++++++---
> > > >  1 file changed, 9 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
> > > > index a76aaee5ba..db4461e551 100644
> > > > --- a/meta/classes-recipe/kernel.bbclass
> > > > +++ b/meta/classes-recipe/kernel.bbclass
> > > > @@ -239,6 +239,8 @@ KERNEL_EXTRA_ARGS ?= ""
> > > >  EXTRA_OEMAKE += ' CC="${KERNEL_CC}" LD="${KERNEL_LD}" OBJCOPY="${KERNEL_OBJCOPY}" STRIP="${KERNEL_STRIP}"'
> > > >  EXTRA_OEMAKE += ' HOSTCC="${BUILD_CC}" HOSTCFLAGS="${BUILD_CFLAGS}" HOSTLDFLAGS="${BUILD_LDFLAGS}" HOSTCPP="${BUILD_CPP}"'
> > > >  EXTRA_OEMAKE += ' HOSTCXX="${BUILD_CXX}" HOSTCXXFLAGS="${BUILD_CXXFLAGS}"'
> > > > +# Only for newer kernels (5.19+), native pkg-config variables are set for older kernels when building kernel and modules
> > > > +EXTRA_OEMAKE += ' HOSTPKG_CONFIG="pkg-config-native"'
> > > >
> > > >  KERNEL_ALT_IMAGETYPE ??= ""
> > > >
> > > > @@ -356,9 +358,6 @@ kernel_do_compile() {
> > > >         export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> > > >         export PKG_CONFIG_SYSROOT_DIR=""
> > > >
> > > > -       # for newer kernels (5.19+) there's a dedicated variable
> > > > -       export HOSTPKG_CONFIG="pkg-config-native"
> > > > -
> > > >         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
> > > >                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
> > > >                 # be set....
> > > > @@ -408,6 +407,13 @@ addtask transform_kernel after do_compile before do_install
> > > >
> > > >  do_compile_kernelmodules() {
> > > >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
> > > > +
> > > > +       # setup native pkg-config variables (kconfig scripts call pkg-config directly, cannot generically be overriden to pkg-config-native)
> > > > +       export PKG_CONFIG_DIR="${STAGING_DIR_NATIVE}${libdir_native}/pkgconfig"
> > > > +       export PKG_CONFIG_PATH="$PKG_CONFIG_DIR:${STAGING_DATADIR_NATIVE}/pkgconfig"
> > > > +       export PKG_CONFIG_LIBDIR="$PKG_CONFIG_DIR"
> > > > +       export PKG_CONFIG_SYSROOT_DIR=""
> > > > +
> > > >         if [ "${KERNEL_DEBUG_TIMESTAMPS}" != "1" ]; then
> > > >                 # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
> > > >                 # be set....
> > > > --
> > > > 2.34.1
> > > >
> > >
> > >
> > > --
> > > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > > thee at its end
> > > - "Use the force Harry" - Gandalf, Star Trek II
> > >
> > >
> 
> 
> 
> -- 
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
> 
>