[Buildroot] [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

[Buildroot] [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

[Michaelis, Adam J Collins via buildroot]

From 45b18a0b5af619fd90194634530cc75f995ea57a Mon Sep 17 00:00:00 2001
From: Adam Michaelis <adam.michaelis@collins.com>
Date: Wed, 16 Oct 2024 09:06:14 -0500
Subject: [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

Installing kernel build artifacts to staging so that the user could use
the exported SDK to build out-of-tree kernel modules.

Signed-off-by: Adam Michaelis <adam.michaelis@collins.com>
---
 linux/Config.in |  8 ++++++++
 linux/linux.mk  | 17 +++++++++++++++++
 2 files changed, 25 insertions(+)

diff --git a/linux/Config.in b/linux/Config.in
index e8c0722de0f9..372cd9ddd7aa 100644
--- a/linux/Config.in
+++ b/linux/Config.in
@@ -454,6 +454,14 @@ config BR2_LINUX_KERNEL_INSTALL_TARGET
          /boot if DTBs have been generated by the kernel build
          process.

+config BR2_LINUX_KERNEL_INSTALL_STAGING
+       bool "Install kmod build files to staging"
+       help
+         Select this option to have the kernel files required for
+         out-of-tree module building installed to staging. This will
+         then enable module building from an exported SDK without
+         the full Linux build and Buildroot.
+
 config BR2_LINUX_KERNEL_NEEDS_HOST_OPENSSL
        bool "Needs host OpenSSL"
        help
diff --git a/linux/linux.mk b/linux/linux.mk
index 469a80db9f55..8d46f2e4b7f2 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -585,6 +585,23 @@ define LINUX_RUN_DEPMOD
 endef
 LINUX_TARGET_FINALIZE_HOOKS += LINUX_RUN_DEPMOD

+ifeq ($(BR2_LINUX_KERNEL_INSTALL_STAGING),y)
+LINUX_INSTALL_STAGING=YES
+endif
+
+define LINUX_INSTALL_STAGING_CMDS
+       # Install artifacts needed for module builds into staging for SDK use
+       if grep -q "CONFIG_MODULES=y" $(@D)/.config; then \
+               install -d "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/arch" && \
+               rsync --exclude "*.o" -a "$(@D)/arch/$(KERNEL_ARCH)" \
+                       "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/arch/" && \
+               rsync --exclude "*.o" --exclude "*.a" -a "$(@D)/include" \
+                       "$(@D)/scripts" "$(@D)/certs" "$(@D)/Makefile" \
+                       "$(@D)/Module.symvers" "$(@D)/.config" \
+                       "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/" ;\
+       fi
+endef
+
 # Include all our extensions.
 #
 # Note: our package infrastructure uses the full-path of the last-scanned
--
2.29.0


Adam Michaelis
Prin. Eng. LLSW
Mission Systems
Collins Aerospace
Cedar Rapids, IA, USA

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

[Thomas Petazzoni via buildroot]

Hello Adam,

Thanks a lot for your patch. See below some feedback.

On Tue, 22 Oct 2024 16:06:27 +0000
"Michaelis, Adam J Collins via buildroot" <buildroot@buildroot.org> wrote:

> From 45b18a0b5af619fd90194634530cc75f995ea57a Mon Sep 17 00:00:00 2001
> From: Adam Michaelis <adam.michaelis@collins.com>
> Date: Wed, 16 Oct 2024 09:06:14 -0500
> Subject: [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

Please use git send-email to send your patches, instead of sending the
result of "git format-patch" directly in an e-mail. Indeed, otherwise
we have two times the From:, Date:, Subject:, etc. headers.

> Installing kernel build artifacts to staging so that the user could use
> the exported SDK to build out-of-tree kernel modules.

I think this could be worded a bit better. "This commit adds an option
that allows to install to the staging directory the necessary
files/artefacts that are needed to build out of tree kernel modules.
This is relevant when one wants to build out of tree kernel modules
using the Buildroot-generated SDK".

> +config BR2_LINUX_KERNEL_INSTALL_STAGING
> +       bool "Install kmod build files to staging"
> +       help
> +         Select this option to have the kernel files required for
> +         out-of-tree module building installed to staging. This will
> +         then enable module building from an exported SDK without
> +         the full Linux build and Buildroot.

I am actually wondering whether this should be an option. We don't have
any similar option for any other package.

> +define LINUX_INSTALL_STAGING_CMDS
> +       # Install artifacts needed for module builds into staging for SDK use
> +       if grep -q "CONFIG_MODULES=y" $(@D)/.config; then \
> +               install -d "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/arch" && \
> +               rsync --exclude "*.o" -a "$(@D)/arch/$(KERNEL_ARCH)" \
> +                       "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/arch/" && \
> +               rsync --exclude "*.o" --exclude "*.a" -a "$(@D)/include" \
> +                       "$(@D)/scripts" "$(@D)/certs" "$(@D)/Makefile" \
> +                       "$(@D)/Module.symvers" "$(@D)/.config" \
> +                       "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/" ;\

This feels a bit "crappy", these manual rsyncs. Doesn't the kernel has
some relevant make target that do the right thing?

Thanks a lot!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

[Brandon Maier via buildroot]

Hi Thomas,

On Tue Oct 22, 2024 at 7:24 PM UTC, Thomas Petazzoni via buildroot wrote:
> Hello Adam,
>
> Thanks a lot for your patch. See below some feedback.
>
> On Tue, 22 Oct 2024 16:06:27 +0000
> "Michaelis, Adam J Collins via buildroot" <buildroot@buildroot.org> wrote:
>
> > From 45b18a0b5af619fd90194634530cc75f995ea57a Mon Sep 17 00:00:00 2001
> > From: Adam Michaelis <adam.michaelis@collins.com>
> > Date: Wed, 16 Oct 2024 09:06:14 -0500
> > Subject: [PATCH v2 1/1] linux: Install artifacts to staging for kmod build
>
> Please use git send-email to send your patches, instead of sending the
> result of "git format-patch" directly in an e-mail. Indeed, otherwise
> we have two times the From:, Date:, Subject:, etc. headers.
>
> > Installing kernel build artifacts to staging so that the user could use
> > the exported SDK to build out-of-tree kernel modules.
>
> I think this could be worded a bit better. "This commit adds an option
> that allows to install to the staging directory the necessary
> files/artefacts that are needed to build out of tree kernel modules.
> This is relevant when one wants to build out of tree kernel modules
> using the Buildroot-generated SDK".
>
> > +config BR2_LINUX_KERNEL_INSTALL_STAGING
> > +       bool "Install kmod build files to staging"
> > +       help
> > +         Select this option to have the kernel files required for
> > +         out-of-tree module building installed to staging. This will
> > +         then enable module building from an exported SDK without
> > +         the full Linux build and Buildroot.
>
> I am actually wondering whether this should be an option. We don't have
> any similar option for any other package.

I think the main advantage to being an option is to keep the size of the
SDK down. This installs a large amount of the kernel source code which
appears to be hundreds of MBs.

>
> > +define LINUX_INSTALL_STAGING_CMDS
> > +       # Install artifacts needed for module builds into staging for SDK use
> > +       if grep -q "CONFIG_MODULES=y" $(@D)/.config; then \
> > +               install -d "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/arch" && \
> > +               rsync --exclude "*.o" -a "$(@D)/arch/$(KERNEL_ARCH)" \
> > +                       "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/arch/" && \
> > +               rsync --exclude "*.o" --exclude "*.a" -a "$(@D)/include" \
> > +                       "$(@D)/scripts" "$(@D)/certs" "$(@D)/Makefile" \
> > +                       "$(@D)/Module.symvers" "$(@D)/.config" \
> > +                       "$(STAGING_DIR)/lib/modules/$(LINUX_VERSION_PROBED)/build/" ;\
>
> This feels a bit "crappy", these manual rsyncs. Doesn't the kernel has
> some relevant make target that do the right thing?

I looked around a bit in the kernel and couldn't find anything.

I checked a few distro packages and they all seem to be variants of
this. See the following

Arch: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/blob/main/PKGBUILD?ref_type=heads#L133
Fedora: https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel.spec#_3507
Yocto: https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-kernel/linux/kernel-devsrc.bb#L37

I would guess the rsync copy is including far more stuff then is
actually necessary, as most of those are significantly more complicated.

Thanks,
Brandon

>
> Thanks a lot!
>
> Thomas
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

[Thomas Petazzoni via buildroot]

On Tue, 22 Oct 2024 20:14:54 +0000
"Brandon Maier" <brandon.maier@collins.com> wrote:

> > I am actually wondering whether this should be an option. We don't have
> > any similar option for any other package.  
> 
> I think the main advantage to being an option is to keep the size of the
> SDK down. This installs a large amount of the kernel source code which
> appears to be hundreds of MBs.

Then probably there is too much stuff getting copied. I think "all"
what's needed are the configured headers + some bits of the build
system.

> I looked around a bit in the kernel and couldn't find anything.
> 
> I checked a few distro packages and they all seem to be variants of
> this. See the following
> 
> Arch: https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/blob/main/PKGBUILD?ref_type=heads#L133
> Fedora: https://src.fedoraproject.org/rpms/kernel/blob/rawhide/f/kernel.spec#_3507
> Yocto: https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-kernel/linux/kernel-devsrc.bb#L37

Wargh, I just looked at the Yocto one, and it's awful.

What's your use case for wanting to build out of tree modules with the
SDK ? :-)

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [External] Re: [PATCH v2 1/1] linux: Install artifacts to staging for kmod build

[Michaelis, Adam J Collins via buildroot]

> -----Original Message-----
> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Sent: Tuesday, October 22, 2024 3:23 PM
> To: Maier, Brandon L Collins <Brandon.Maier@collins.com>
> Cc: Michaelis, Adam J Collins via buildroot <buildroot@buildroot.org>;
> Michaelis, Adam J Collins <adam.michaelis@collins.com>; Buchmeyer, Neal
> Collins <Neal.Buchmeyer@collins.com>; Julien Olivain <ju.o@free.fr>; Romain
> Naour <romain.naour@smile.fr>
> Subject: [External] Re: [Buildroot] [PATCH v2 1/1] linux: Install artifacts to
> staging for kmod build
>
> On Tue, 22 Oct 2024 20:14:54 +0000
> "Brandon Maier" <brandon.maier@collins.com> wrote:
>
> > > I am actually wondering whether this should be an option. We don't
> > > have any similar option for any other package.
> >
> > I think the main advantage to being an option is to keep the size of
> > the SDK down. This installs a large amount of the kernel source code
> > which appears to be hundreds of MBs.
>
> Then probably there is too much stuff getting copied. I think "all"
> what's needed are the configured headers + some bits of the build system.
>

I based the INSTALL_STAGING option on the INSTALL_TARGET option, though it is a different concept.

I grabbed those folders initially based on a StackOverflow answer (https://stackoverflow.com/questions/37581687/is-there-a-minimal-set-of-files-to-keep-from-a-linux-kernel-compile-for-compilin), but am exploring some additional pruning (without going as overboard as Yocto's script). My current method adds 123 MB to staging/ for the kernel build folder...

> > I looked around a bit in the kernel and couldn't find anything.
> >
> > I checked a few distro packages and they all seem to be variants of
> > this. See the following
> >
> > Arch:
> > https://urldefense.com/v3/__https://gitlab.archlinux.org/archlinux/pac
> > kaging/packages/linux/-
> /blob/main/PKGBUILD?ref_type=heads*L133__;Iw!!M
> >
> vWE!AYDJenGCr2mpiuIQtk9XA0ITnz2nHcWDrxsAjOBjAe7A9NM_fXovl1xdYQ
> poDGmbHn
> > Pvdqe7KDsjWat48yMmCdzCibdP0fK2$
> > Fedora:
> > https://urldefense.com/v3/__https://src.fedoraproject.org/rpms/kernel/
> >
> blob/rawhide/f/kernel.spec*_3507__;Iw!!MvWE!AYDJenGCr2mpiuIQtk9XA0I
> Tnz
> >
> 2nHcWDrxsAjOBjAe7A9NM_fXovl1xdYQpoDGmbHnPvdqe7KDsjWat48yMm
> CdzCiQhmYc71
> > $
> > Yocto:
> >
> https://urldefense.com/v3/__https://github.com/openembedded/openemb
> edd
> > ed-core/blob/master/meta/recipes-kernel/linux/kernel-devsrc.bb*L37__;I
> >
> w!!MvWE!AYDJenGCr2mpiuIQtk9XA0ITnz2nHcWDrxsAjOBjAe7A9NM_fXovl1
> xdYQpoDG
> > mbHnPvdqe7KDsjWat48yMmCdzCiX3vqMxD$
>
> Wargh, I just looked at the Yocto one, and it's awful.
>
> What's your use case for wanting to build out of tree modules with the SDK ?
> :-)

The general use case is having our BSP team build most of the system (kernel, rootfs, bootloader) then generate an SDK tar.gz that we can hand-off to application software (already works). But, we now have some frameworks that require a kernel module that varies and would like to enable the application software teams to build that using the SDK without having to do a full BSP Buildroot build.

>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel
> engineering and training
> https://urldefense.com/v3/__https://bootlin.com__;!!MvWE!AYDJenGCr2mp
> iuIQtk9XA0ITnz2nHcWDrxsAjOBjAe7A9NM_fXovl1xdYQpoDGmbHnPvdqe7K
> DsjWat48yMmCdzCiYZVErDC$
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot