Re: [OE-core] [PATCH v3 03/11] kernel.bbclass: add kernel-initrd-modules meta package

Bruce Ashfield
2025-04-08T03:42:17Z

[-- Attachment #1: Type: text/plain, Size: 10534 bytes --]

On Fri, Apr 4, 2025 at 12:29 PM Mikko Rapeli via lists.openembedded.org
<mikko.rapeli=linaro.org@lists.openembedded.org> wrote:

> At the moment linux-yocto kernels for various architectures
> are not very modular and a lot of drivers are built into the kernel
> even when they are not needed at runtime. These make the main kernel
> binary big and slow to boot. This also impacts udev in userspace
> which takes a long time processing events from all these built in drivers,
> for example when udev runs in initrd.
>
> Then constructing the initrd is very device and kernel configuration
> specific.
> initrd image needs explicitly define which binary packages to install
> to avoid pulling in complex dependencies. A full set of kernel modules
> via kernel-modules meta package is too big for initrd and most of the
> drivers are not needed for use cases like "just load modules to mount
> main rootfs". Then the initrd configuration breaks if kernel driver
> is built into the kernel since the binary package doesn't exist.
>
> Introduce kernel-initrd-modules meta package to solve these problems.
> The meta package adds dependencies to real kernel modules based on
> the kernel module file paths so that it will include several
> kernel subsystems and their drivers which are often needed to find
> main rootfs from some block device. This works when drivers are built
> as modules but does not break if drivers are built into the kernel.
>
> The resulting initrd is also smaller since only a subset of drivers
> are needed for "mount the rootfs" usecase. Tested on genericarm64
> kernel and qemu and AMD KV260 HW.
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
> .../kernel-module-split.bbclass | 48 +++++++++++++++++++
> meta/classes-recipe/kernel.bbclass | 5 +-
> meta/classes-recipe/module.bbclass | 37 ++++++++++++++
> 3 files changed, 89 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/kernel-module-split.bbclass
> b/meta/classes-recipe/kernel-module-split.bbclass
> index 9487365eb7..101c5cd959 100644
> --- a/meta/classes-recipe/kernel-module-split.bbclass
> +++ b/meta/classes-recipe/kernel-module-split.bbclass
> @@ -42,6 +42,40 @@ KERNEL_MODULE_PACKAGE_PREFIX ?= ""
> KERNEL_MODULE_PACKAGE_SUFFIX ?= "-${KERNEL_VERSION}"
> KERNEL_MODULE_PROVIDE_VIRTUAL ?= "1"
>
> +# subset of kernel modules needed in initrd, to e.g. mount rootfs from
> block device
> +KERNEL_INITRD_MODULES_META_PACKAGE ?= "${@
> d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-initrd-modules"
> +
> +# match regex to path or file name. E.g. include all drivers with files
> in path /drivers/ata/
> +KERNEL_INITRD_MODULES_REGEX ?= "(.*)(\
> +/drivers/acpi/|\
> +/drivers/ata/|\
> +/drivers/block/|\
> +/drivers/cdrom/|\
> +/drivers/char/hw_random/|\
> +/drivers/char/tpm/|\
> +/drivers/char/|\
> +/drivers/crypto/|\
> +/drivers/dax/|\
> +/drivers/firmware/arm_scmi/|\
> +/drivers/gpu/drm/|\
> +/drivers/md/|\
> +/drivers/mmc/|\
> +/drivers/mtd/|\
> +/drivers/nvdimm/|\
> +/drivers/nvme/|\
> +/drivers/pci/|\
> +/drivers/scsi/|\
> +/drivers/tee/|\
> +/drivers/tty/serial/|\
> +/drivers/virtio/|\
> +/drivers/watchdog/|\
> +/kernel/arch/|\
> +/kernel/block/|\
> +/kernel/crypto/|\
> +/kernel/fs/|\
> +/kernel/lib/\
> +)(.*)"
> +
>

Apologies for not replying to your reply to my other review, I've had some
unexpected travel
and significant time away from my desk, so I'm running way behind on
everything.

I still don't agree with these values being defined in the base classes.
There's simply too
many kernels and versions out there to make sure this works (and is sane
everywhere)
.. I've lived that for 15 years in supporting the reference kernels.

The reality is that everyone will use the defaults, and never look at them
or otherwise
make sure they are valid for their kernel. We'll end up churning the
values, when really
it is specific to a kernel, a board and even a use case. A kernel recipe is
where all of
those things are explicitly handled, so putting the regex there makes it
very clear that
you need to define something.

There's also still the issue with the repeated set of directories in the
two classes.
To address that, why can't the variable definition be moved to a .inc file ?

With the variables defined in a .inc file, it then becomes very easy for
kernel recipes
to include the .inc and get a default definition .. but we are still
putting the requirement
on them that they explicitly include the file to get the default, versus it
silently being
defined and used in the classes.

Bruce

> python split_kernel_module_packages () {
> import re
>
> @@ -183,6 +217,20 @@ python split_kernel_module_packages () {
> modules = do_split_packages(d, root='${nonarch_base_libdir}/modules',
> file_regex=module_regex, output_pattern=module_pattern, description='%s
> kernel module', postinst=postinst, postrm=postrm, recursive=True,
> hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name,
> kernel_version))
> if modules:
> d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
> +
> + initrd_metapkg = d.getVar('KERNEL_INITRD_MODULES_META_PACKAGE') or ""
> + initrd_module_regex = d.getVar('KERNEL_INITRD_MODULES_REGEX') or ""
> + if (initrd_metapkg != "") and (initrd_module_regex != ""):
> + initrd_module_regex = re.compile(initrd_module_regex)
> + initrd_modules = []
> + for module in modules:
> + files = d.getVar('FILES:' + module)
> + m = re.match(initrd_module_regex, files)
> + if m:
> + initrd_modules.append(module)
> +
> + if initrd_modules:
> + d.appendVar('RDEPENDS:' + initrd_metapkg, ' '+'
> '.join(initrd_modules))
> }
>
> do_package[vardeps] += '${@" ".join(map(lambda s: "module_conf_" + s,
> (d.getVar("KERNEL_MODULE_PROBECONF") or "").split()))}'
> diff --git a/meta/classes-recipe/kernel.bbclass
> b/meta/classes-recipe/kernel.bbclass
> index 36ce659762..3dcaebcaed 100644
> --- a/meta/classes-recipe/kernel.bbclass
> +++ b/meta/classes-recipe/kernel.bbclass
> @@ -695,13 +695,14 @@ EXPORT_FUNCTIONS do_compile do_transform_kernel
> do_transform_bundled_initramfs d
>
> # kernel-base becomes kernel-${KERNEL_VERSION}
> # kernel-image becomes kernel-image-${KERNEL_VERSION}
> -PACKAGES = "${KERNEL_PACKAGE_NAME} ${KERNEL_PACKAGE_NAME}-base
> ${KERNEL_PACKAGE_NAME}-vmlinux ${KERNEL_PACKAGE_NAME}-image
> ${KERNEL_PACKAGE_NAME}-dev ${KERNEL_PACKAGE_NAME}-modules
> ${KERNEL_PACKAGE_NAME}-dbg"
> +PACKAGES = "${KERNEL_PACKAGE_NAME} ${KERNEL_PACKAGE_NAME}-base
> ${KERNEL_PACKAGE_NAME}-vmlinux ${KERNEL_PACKAGE_NAME}-image
> ${KERNEL_PACKAGE_NAME}-dev ${KERNEL_PACKAGE_NAME}-modules
> ${KERNEL_PACKAGE_NAME}-initrd-modules ${KERNEL_PACKAGE_NAME}-dbg"
> FILES:${PN} = ""
> FILES:${KERNEL_PACKAGE_NAME}-base =
> "${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.order
> ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.builtin
> ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.builtin.modinfo"
> FILES:${KERNEL_PACKAGE_NAME}-image = ""
> FILES:${KERNEL_PACKAGE_NAME}-dev = "/${KERNEL_IMAGEDEST}/System.map*
> /${KERNEL_IMAGEDEST}/Module.symvers* /${KERNEL_IMAGEDEST}/config*
> ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
> FILES:${KERNEL_PACKAGE_NAME}-vmlinux =
> "/${KERNEL_IMAGEDEST}/vmlinux-${KERNEL_VERSION_NAME}"
> FILES:${KERNEL_PACKAGE_NAME}-modules = ""
> +FILES:${KERNEL_PACKAGE_NAME}-initrd-modules = ""
> FILES:${KERNEL_PACKAGE_NAME}-dbg = "/usr/lib/debug /usr/src/debug"
> RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (=
> ${EXTENDPKGV})"
> # Allow machines to override this dependency if kernel image files are
> @@ -716,7 +717,9 @@ ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
> ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-base = "1"
> ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-image = "1"
> ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-modules = "1"
> +ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-initrd-modules = "1"
> DESCRIPTION:${KERNEL_PACKAGE_NAME}-modules = "Kernel modules meta package"
> +DESCRIPTION:${KERNEL_PACKAGE_NAME}-initrd-modules = "Kernel initrd
> modules meta package"
>
> pkg_postinst:${KERNEL_PACKAGE_NAME}-base () {
> if [ ! -e "$D/lib/modules/${KERNEL_VERSION}" ]; then
> diff --git a/meta/classes-recipe/module.bbclass
> b/meta/classes-recipe/module.bbclass
> index f2f0b25a2d..51f864f1f9 100644
> --- a/meta/classes-recipe/module.bbclass
> +++ b/meta/classes-recipe/module.bbclass
> @@ -86,3 +86,40 @@ EXPORT_FUNCTIONS do_compile do_install
> KERNEL_MODULES_META_PACKAGE = "${PN}"
> FILES:${PN} = ""
> ALLOW_EMPTY:${PN} = "1"
> +
> +# subset of kernel modules needed in initrd, to e.g. mount rootfs from
> block device
> +KERNEL_INITRD_MODULES_META_PACKAGE ?= "${@
> d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-initrd-modules"
> +
> +# match regex to path or file name. E.g. include all drivers with files
> in path /drivers/ata/
> +KERNEL_INITRD_MODULES_REGEX ?= "(.*)(\
> +/drivers/acpi/|\
> +/drivers/ata/|\
> +/drivers/block/|\
> +/drivers/cdrom/|\
> +/drivers/char/hw_random/|\
> +/drivers/char/tpm/|\
> +/drivers/char/|\
> +/drivers/crypto/|\
> +/drivers/dax/|\
> +/drivers/firmware/arm_scmi/|\
> +/drivers/gpu/drm/|\
> +/drivers/md/|\
> +/drivers/mmc/|\
> +/drivers/mtd/|\
> +/drivers/nvdimm/|\
> +/drivers/nvme/|\
> +/drivers/pci/|\
> +/drivers/scsi/|\
> +/drivers/tee/|\
> +/drivers/tty/serial/|\
> +/drivers/virtio/|\
> +/drivers/watchdog/|\
> +/kernel/arch/|\
> +/kernel/block/|\
> +/kernel/crypto/|\
> +/kernel/fs/|\
> +/kernel/lib/\
> +)(.*)"
> +
> +FILES:${PN}-initrd = ""
> +ALLOW_EMPTY:${PN}-initrd = "1"
> --
> 2.43.0
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#214354):
> https://lists.openembedded.org/g/openembedded-core/message/214354
> Mute This Topic: https://lists.openembedded.org/mt/112087526/1050810
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

--
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 14090 bytes --]