[PATCH v4 0/9] systemd based initrd and modular kernel support

[PATCH v4 0/9] systemd based initrd and modular kernel support

[Mikko Rapeli]

systemd based initrd supports more security, encryption etc features
than custom shell script ones but oe-core lacks support for it.
Convert core-image-initramfs-boot to create systemd based initrd
if "systemd-initramfs" is set in DISTRO_FEATURES. Includes a test
for this initrd to mount rootfs via Unified Kernel Image uki.py.

Sadly the boot time is really slow, e.g. over 30 seconds with
systemd based initrd. One of the major reasons is the large
amount of kernel modules built into the linux-yocto kernel and
processing of their udev events in userspace. Thus add
kernel-initrd-modules meta package to easily install
subset of all kernel modules to initrd. The subset
supports "mount rootfs from any block device" but
doesn't include graphics, UBS etc support. Without this
generated meta package, initrd recipe needs to manually define
which exact kernel modules to include which breaks if
those modules are built into the kernel. Thus the initrd
becomes machine and kernel config specific and hard to maintain.
With this meta package a generic initrd can be created.

Separate changes to linux-yocto compile a lot more drivers
as modules but they depend on these initrd changes.

v1: https://lists.openembedded.org/g/openembedded-core/message/212832

v2: split initrd size reduction and kernel modules addition
    to separate commits, added packages exclicitly back to fix
    rootfs mount visible as wic selftest failures,
    fixed wic bugs which resulted in broken boot due to missing
    EFI bootloader binaries in some builds, moved
    "systemd: use serial-getty-generator on genericarm64" to separate
    series since it's not strictly needed and goes to a different mailing
    list

v3: enable "efi" support in systemd by default not via MACHINE_FEATURES,
    disable kernel-initrd-modules meta package generation if
    KERNEL_INITRD_MODULES_REGEX is not set at all, keeping
    default KERNEL_INITRD_MODULES_REGEX in classes to support
    external kernel recipes (which could be compatible with default
    images to e.g. run upstream tests), tested (with pmem and
    modular kernel RFC patches sent out separately) using genericarm64
    machine config on:
    qemu u-boot https://ledge.validation.linaro.org/scheduler/job/106701
    qemu edk2 https://ledge.validation.linaro.org/scheduler/job/106703
    AMD zcu102 https://ledge.validation.linaro.org/scheduler/job/106702
    AMD kv260 https://ledge.validation.linaro.org/scheduler/job/106700
    rpi4 https://ledge.validation.linaro.org/scheduler/job/106705

v4: systemd "efi" support enabled with distro feature in poky-altcfg,
    kernel-initrd-module meta package config via .inc file,
    core-image-initramfs-boot uses "systemd-initramfs" in IMAGE_FEATURES to enable
    systemd based initrd instead of distro features,
    wic patches as in v3 as I think this is the right thing to do
    but they are unrelated to modular kernel or systemd initrd and
    can be taken out of this series

Mikko Rapeli (9):
  poky-altcfg.conf: enable "efi" in DISTRO_FEATURES
  kernel.bbclass: add kernel-initrd-modules meta package
  core-image-initramfs-boot: add option to build systemd based initrd
  core-image-initramfs-boot: don't install RRECOMMENDS to reduce size
  core-image-initramfs-boot: install kernel-initrd-modules by default
  oeqa selftest uki.py: add aarch64/arm test with systemd based initrd
  image_types_wic.bbclass: capture verbose wic output by default
  wic bootimg-efi.py: fail build if no binaries installed
  image_types_wic.bbclass: depend on grub-efi and systemd-boot on
    aarch64, systemd-boot on arm

 meta-poky/conf/distro/poky-altcfg.conf        |  3 +
 meta/classes-recipe/image_types_wic.bbclass   |  4 +-
 .../kernel-module-split.bbclass               | 18 +++++
 meta/classes-recipe/kernel.bbclass            |  5 +-
 meta/classes-recipe/module.bbclass            |  3 +
 .../distro/include/kernel-initrd-modules.inc  | 37 +++++++++
 meta/lib/oeqa/selftest/cases/uki.py           | 76 +++++++++++++++++++
 .../images/core-image-initramfs-boot.bb       | 34 ++++++++-
 scripts/lib/wic/plugins/source/bootimg-efi.py |  3 +
 9 files changed, 177 insertions(+), 6 deletions(-)
 create mode 100644 meta/conf/distro/include/kernel-initrd-modules.inc

-- 
2.43.0

[PATCH v4 1/9] poky-altcfg.conf: enable "efi" in DISTRO_FEATURES

[Mikko Rapeli]

oe-core machines like genericarm64 already enable "efi" in
MACHINE_FEATURES but this does not enable systemd side
"efi" support and thus systemd does not mount ESP
partition for example.

Enabling "efi" in systemd PACKAGECONFIG got negative comments
in review. Also enabling systemd "efi" support via MACHINE_FEATURES
was rejected. Thus enabling this as distro feature for "poky-altcfg".

Many HW targets come with UEFI compatible firmware by default.
For example u-boot enables UEFI support by default on all targets.

This enables seeing UEFI firmware interfaces and data
in userspace since "efivars" is now mounted by systemd
to /sys/firmware/efi/efivars. Things like UEFI bootloader
(grub, systemd-boot etc) selections and secure boot status
can be queried from there.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta-poky/conf/distro/poky-altcfg.conf | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta-poky/conf/distro/poky-altcfg.conf b/meta-poky/conf/distro/poky-altcfg.conf
index 3d64bdd1fe..e56c4f0a37 100644
--- a/meta-poky/conf/distro/poky-altcfg.conf
+++ b/meta-poky/conf/distro/poky-altcfg.conf
@@ -14,4 +14,7 @@ POKY_INIT_MANAGER:poky-altcfg = "systemd"
 # systemd isn't suitable with musl
 POKY_INIT_MANAGER:poky-altcfg:libc-musl = "sysvinit"
 
+# for systemd etc UEFI support
+DISTRO_FEATURES:append = " efi"
+
 require conf/distro/include/poky-distro-alt-test-config.inc
-- 
2.43.0

[PATCH v4 2/9] kernel.bbclass: add kernel-initrd-modules meta package

[Mikko Rapeli]

At the moment linux-yocto kernels for various architectures
are not very modular and a lot of drivers are built into the kernel
even when they are not needed at runtime. These make the main kernel
binary big and slow to boot. This also impacts udev in userspace
which takes a long time processing events from all these built in drivers,
for example when udev runs in initrd.

Then constructing the initrd is very device and kernel configuration specific.
initrd image needs explicitly define which binary packages to install
to avoid pulling in complex dependencies. A full set of kernel modules
via kernel-modules meta package is too big for initrd and most of the
drivers are not needed for use cases like "just load modules to mount
main rootfs". Then the initrd configuration breaks if kernel driver
is built into the kernel since the binary package doesn't exist.

Introduce kernel-initrd-modules meta package to solve these problems.
The meta package adds dependencies to real kernel modules based on
the kernel module file paths so that it will include several
kernel subsystems and their drivers which are often needed to find
main rootfs from some block device. This works when drivers are built
as modules but does not break if drivers are built into the kernel.

The resulting initrd is also smaller since only a subset of drivers
are needed for "mount the rootfs" usecase. Tested on genericarm64
kernel and qemu and AMD KV260 HW.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 .../kernel-module-split.bbclass               | 18 +++++++++
 meta/classes-recipe/kernel.bbclass            |  5 ++-
 meta/classes-recipe/module.bbclass            |  3 ++
 .../distro/include/kernel-initrd-modules.inc  | 37 +++++++++++++++++++
 4 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100644 meta/conf/distro/include/kernel-initrd-modules.inc

diff --git a/meta/classes-recipe/kernel-module-split.bbclass b/meta/classes-recipe/kernel-module-split.bbclass
index 9487365eb7..8633f0f364 100644
--- a/meta/classes-recipe/kernel-module-split.bbclass
+++ b/meta/classes-recipe/kernel-module-split.bbclass
@@ -42,6 +42,10 @@ KERNEL_MODULE_PACKAGE_PREFIX ?= ""
 KERNEL_MODULE_PACKAGE_SUFFIX ?= "-${KERNEL_VERSION}"
 KERNEL_MODULE_PROVIDE_VIRTUAL ?= "1"
 
+KERNEL_INITRD_MODULES ?= "conf/distro/include/kernel-initrd-modules.inc"
+
+require ${KERNEL_INITRD_MODULES}
+
 python split_kernel_module_packages () {
     import re
 
@@ -183,6 +187,20 @@ python split_kernel_module_packages () {
     modules = do_split_packages(d, root='${nonarch_base_libdir}/modules', file_regex=module_regex, output_pattern=module_pattern, description='%s kernel module', postinst=postinst, postrm=postrm, recursive=True, hook=frob_metadata, extra_depends='%s-%s' % (kernel_package_name, kernel_version))
     if modules:
         d.appendVar('RDEPENDS:' + metapkg, ' '+' '.join(modules))
+
+    initrd_metapkg = d.getVar('KERNEL_INITRD_MODULES_META_PACKAGE') or ""
+    initrd_module_regex = d.getVar('KERNEL_INITRD_MODULES_REGEX') or ""
+    if (initrd_metapkg != "") and (initrd_module_regex != ""):
+        initrd_module_regex = re.compile(initrd_module_regex)
+        initrd_modules = []
+        for module in modules:
+            files = d.getVar('FILES:' + module)
+            m = re.match(initrd_module_regex, files)
+            if m:
+                initrd_modules.append(module)
+
+        if initrd_modules:
+            d.appendVar('RDEPENDS:' + initrd_metapkg, ' '+' '.join(initrd_modules))
 }
 
 do_package[vardeps] += '${@" ".join(map(lambda s: "module_conf_" + s, (d.getVar("KERNEL_MODULE_PROBECONF") or "").split()))}'
diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index 36ce659762..3dcaebcaed 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -695,13 +695,14 @@ EXPORT_FUNCTIONS do_compile do_transform_kernel do_transform_bundled_initramfs d
 
 # kernel-base becomes kernel-${KERNEL_VERSION}
 # kernel-image becomes kernel-image-${KERNEL_VERSION}
-PACKAGES = "${KERNEL_PACKAGE_NAME} ${KERNEL_PACKAGE_NAME}-base ${KERNEL_PACKAGE_NAME}-vmlinux ${KERNEL_PACKAGE_NAME}-image ${KERNEL_PACKAGE_NAME}-dev ${KERNEL_PACKAGE_NAME}-modules ${KERNEL_PACKAGE_NAME}-dbg"
+PACKAGES = "${KERNEL_PACKAGE_NAME} ${KERNEL_PACKAGE_NAME}-base ${KERNEL_PACKAGE_NAME}-vmlinux ${KERNEL_PACKAGE_NAME}-image ${KERNEL_PACKAGE_NAME}-dev ${KERNEL_PACKAGE_NAME}-modules ${KERNEL_PACKAGE_NAME}-initrd-modules ${KERNEL_PACKAGE_NAME}-dbg"
 FILES:${PN} = ""
 FILES:${KERNEL_PACKAGE_NAME}-base = "${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.order ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.builtin ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/modules.builtin.modinfo"
 FILES:${KERNEL_PACKAGE_NAME}-image = ""
 FILES:${KERNEL_PACKAGE_NAME}-dev = "/${KERNEL_IMAGEDEST}/System.map* /${KERNEL_IMAGEDEST}/Module.symvers* /${KERNEL_IMAGEDEST}/config* ${KERNEL_SRC_PATH} ${nonarch_base_libdir}/modules/${KERNEL_VERSION}/build"
 FILES:${KERNEL_PACKAGE_NAME}-vmlinux = "/${KERNEL_IMAGEDEST}/vmlinux-${KERNEL_VERSION_NAME}"
 FILES:${KERNEL_PACKAGE_NAME}-modules = ""
+FILES:${KERNEL_PACKAGE_NAME}-initrd-modules = ""
 FILES:${KERNEL_PACKAGE_NAME}-dbg = "/usr/lib/debug /usr/src/debug"
 RDEPENDS:${KERNEL_PACKAGE_NAME} = "${KERNEL_PACKAGE_NAME}-base (= ${EXTENDPKGV})"
 # Allow machines to override this dependency if kernel image files are
@@ -716,7 +717,9 @@ ALLOW_EMPTY:${KERNEL_PACKAGE_NAME} = "1"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-base = "1"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-image = "1"
 ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-modules = "1"
+ALLOW_EMPTY:${KERNEL_PACKAGE_NAME}-initrd-modules = "1"
 DESCRIPTION:${KERNEL_PACKAGE_NAME}-modules = "Kernel modules meta package"
+DESCRIPTION:${KERNEL_PACKAGE_NAME}-initrd-modules = "Kernel initrd modules meta package"
 
 pkg_postinst:${KERNEL_PACKAGE_NAME}-base () {
 	if [ ! -e "$D/lib/modules/${KERNEL_VERSION}" ]; then
diff --git a/meta/classes-recipe/module.bbclass b/meta/classes-recipe/module.bbclass
index f2f0b25a2d..4661501ded 100644
--- a/meta/classes-recipe/module.bbclass
+++ b/meta/classes-recipe/module.bbclass
@@ -86,3 +86,6 @@ EXPORT_FUNCTIONS do_compile do_install
 KERNEL_MODULES_META_PACKAGE = "${PN}"
 FILES:${PN} = ""
 ALLOW_EMPTY:${PN} = "1"
+
+FILES:${PN}-initrd = ""
+ALLOW_EMPTY:${PN}-initrd = "1"
diff --git a/meta/conf/distro/include/kernel-initrd-modules.inc b/meta/conf/distro/include/kernel-initrd-modules.inc
new file mode 100644
index 0000000000..f2176cb0f1
--- /dev/null
+++ b/meta/conf/distro/include/kernel-initrd-modules.inc
@@ -0,0 +1,37 @@
+# for kernel.bbclass to create "kernel-initd-modules" meta package
+# with a subset of all kernel drivers to suite usecase of mounting
+# the main rootfs from a number block devices on multiple HW devices.
+
+# subset of kernel modules needed in initrd, to e.g. mount rootfs from block device
+KERNEL_INITRD_MODULES_META_PACKAGE ?= "${@ d.getVar("KERNEL_PACKAGE_NAME") or "kernel" }-initrd-modules"
+
+# match regex to path or file name. E.g. include all drivers with files in path /drivers/ata/
+KERNEL_INITRD_MODULES_REGEX ?= "(.*)(\
+/drivers/acpi/|\
+/drivers/ata/|\
+/drivers/block/|\
+/drivers/cdrom/|\
+/drivers/char/hw_random/|\
+/drivers/char/tpm/|\
+/drivers/char/|\
+/drivers/crypto/|\
+/drivers/dax/|\
+/drivers/firmware/arm_scmi/|\
+/drivers/gpu/drm/|\
+/drivers/md/|\
+/drivers/mmc/|\
+/drivers/mtd/|\
+/drivers/nvdimm/|\
+/drivers/nvme/|\
+/drivers/pci/|\
+/drivers/scsi/|\
+/drivers/tee/|\
+/drivers/tty/serial/|\
+/drivers/virtio/|\
+/drivers/watchdog/|\
+/kernel/arch/|\
+/kernel/block/|\
+/kernel/crypto/|\
+/kernel/fs/|\
+/kernel/lib/\
+)(.*)"
-- 
2.43.0

[PATCH v4 3/9] core-image-initramfs-boot: add option to build systemd based initrd

[Mikko Rapeli]

If "systemd-initramfs" is in IMAGE_FEATURES then convert
core-image-initramfs-boot from shell scripts to systemd.
The resulting initramfs is much bigger than shell script one but
supports much more features like disk encryption and TPM devices.
Also includes udev which can load any drivers needed to mount
rootfs.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 .../images/core-image-initramfs-boot.bb       | 22 ++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/images/core-image-initramfs-boot.bb b/meta/recipes-core/images/core-image-initramfs-boot.bb
index 7258944751..2a3fa6d2e0 100644
--- a/meta/recipes-core/images/core-image-initramfs-boot.bb
+++ b/meta/recipes-core/images/core-image-initramfs-boot.bb
@@ -4,7 +4,25 @@ LICENSE = "MIT"
 
 INITRAMFS_SCRIPTS ?= "initramfs-framework-base initramfs-module-udev"
 
-PACKAGE_INSTALL = "${INITRAMFS_SCRIPTS} ${VIRTUAL-RUNTIME_base-utils} base-passwd"
+inherit image
+
+IMAGE_FEATURES[validitems] += "systemd-initramfs"
+
+PACKAGE_INSTALL = " \
+    ${VIRTUAL-RUNTIME_base-utils} \
+    ${@bb.utils.contains('IMAGE_FEATURES', 'systemd-initramfs', ' \
+        base-files \
+        coreutils \
+        e2fsprogs-mke2fs \
+        os-release-initrd \
+        util-linux-blkid \
+        util-linux-mount \
+        util-linux-umount \
+        ${VIRTUAL-RUNTIME_init_manager} \
+        ${VIRTUAL-RUNTIME_dev_manager} \
+    ', '${INITRAMFS_SCRIPTS}', d)} \
+    base-passwd \
+"
 
 # Ensure the initramfs only contains the bare minimum
 IMAGE_FEATURES = ""
@@ -18,5 +36,3 @@ IMAGE_FSTYPES = "${INITRAMFS_FSTYPES}"
 IMAGE_NAME_SUFFIX ?= ""
 IMAGE_ROOTFS_SIZE = "8192"
 IMAGE_ROOTFS_EXTRA_SPACE = "0"
-
-inherit image
-- 
2.43.0

[PATCH v4 4/9] core-image-initramfs-boot: don't install RRECOMMENDS to reduce size

[Mikko Rapeli]

Disables recommends to only install important packages.
Disable machine specific additions which are meant for
full rootfs boot. For example on genericarm64, full
set of kernel modules and WiFi related firmware is
in the machine recommends and those are not needed when
booting to rootfs from an initrd. These reduce systemd
initrd size from 200Mb to 54Mb.

Since recommends are not automatically installed anymore,
explicitly add initramfs-module-rootfs, busybox-udhcpc and libkmod
to shell script based initrd. initramfs-module-rootfs is needed to
mount the rootfs. Installing busybox-udhcpc and libkmod
for backwards compatibility.

Not installing openssl-conf, openssl-ossl-module-legacy or
ldconfig packages since they don't seem essential in initrd
to mount the rootfs.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/recipes-core/images/core-image-initramfs-boot.bb | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-core/images/core-image-initramfs-boot.bb b/meta/recipes-core/images/core-image-initramfs-boot.bb
index 2a3fa6d2e0..e72efab55d 100644
--- a/meta/recipes-core/images/core-image-initramfs-boot.bb
+++ b/meta/recipes-core/images/core-image-initramfs-boot.bb
@@ -2,7 +2,7 @@ SUMMARY = "Basic initramfs to boot a fully-featured rootfs"
 DESCRIPTION = "Small initramfs that contains just udev and init, to find the real rootfs."
 LICENSE = "MIT"
 
-INITRAMFS_SCRIPTS ?= "initramfs-framework-base initramfs-module-udev"
+INITRAMFS_SCRIPTS ?= "initramfs-framework-base initramfs-module-udev initramfs-module-rootfs"
 
 inherit image
 
@@ -22,8 +22,17 @@ PACKAGE_INSTALL = " \
         ${VIRTUAL-RUNTIME_dev_manager} \
     ', '${INITRAMFS_SCRIPTS}', d)} \
     base-passwd \
+    busybox-udhcpc \
+    libkmod \
 "
 
+# reduce size
+NO_RECOMMENDATIONS = "1"
+
+# don't install automatically, pick manually instead
+MACHINE_EXTRA_RDEPENDS = ""
+MACHINE_EXTRA_RRECOMMENDS = ""
+
 # Ensure the initramfs only contains the bare minimum
 IMAGE_FEATURES = ""
 IMAGE_LINGUAS = ""
-- 
2.43.0

[PATCH v4 5/9] core-image-initramfs-boot: install kernel-initrd-modules by default

[Mikko Rapeli]

Most real machines in core set

MACHINE_EXTRA_RRECOMMENDS = "kernel-modules"

which means that all images including initrd have all kernel
modules by default. Only default qemu machines don't do this
and instead the kernel configs there seem to include enough
drivers built-in to mount the rootfs, and this may be
reflected in qemu machine configuration e.g. which
block device emulation is used.

Since for initrd "boot to rootfs" usecase full set
of kernel modules way too much, use the subset
from kernel-initrd-modules which should work
on all machines. If not, then the KERNEL_INITRD_MODULES_REGEX
needs to be fixed for the machine or for everyone
to include any missing driver classes.

Same could be achieved by introducing a
MACHINE_EXTRA_INITRD or similar variable for machine
configs to define extra packages for initrd usecase
of mounting target rootfs. I have not done this for now
since currently only user is kernel-initrd-modules.
Installing kernel-initrd-modules by default enables
the machine specific kernel configs to move drivers
from built-in to modules. The other e.g. firmware packages
installed via MACHINE_EXTRA variables in core machines
are not needed to mount the rootfs.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/recipes-core/images/core-image-initramfs-boot.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/images/core-image-initramfs-boot.bb b/meta/recipes-core/images/core-image-initramfs-boot.bb
index e72efab55d..1690a51d42 100644
--- a/meta/recipes-core/images/core-image-initramfs-boot.bb
+++ b/meta/recipes-core/images/core-image-initramfs-boot.bb
@@ -23,6 +23,7 @@ PACKAGE_INSTALL = " \
     ', '${INITRAMFS_SCRIPTS}', d)} \
     base-passwd \
     busybox-udhcpc \
+    kernel-initrd-modules \
     libkmod \
 "
 
-- 
2.43.0

[PATCH v4 6/9] oeqa selftest uki.py: add aarch64/arm test with systemd based initrd

[Mikko Rapeli]

Use core-image-initramfs-boot with systemd to boot via UKI on
arm/aarch64. Tested on qemuarm, qemuarm64 and genericarm64
on aarch64 build host. The machines have different image
and qemuboot configs so they need to be configured in test.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/uki.py | 76 +++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/uki.py b/meta/lib/oeqa/selftest/cases/uki.py
index 9a1aa4e269..4830fb3403 100644
--- a/meta/lib/oeqa/selftest/cases/uki.py
+++ b/meta/lib/oeqa/selftest/cases/uki.py
@@ -139,3 +139,79 @@ IMAGE_CLASSES:remove = 'testimage'
             cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderEntrySelected-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep '%s'" % (uki_filename)
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+    @skipIfNotArch(['aarch64', 'arm'])
+    @OETestTag("runqemu")
+    def test_uki_boot_systemd_initrd(self):
+        """Build and boot into UEFI firmware (u-boot), systemd-boot, initrd with systemd, rootfs with systemd"""
+        image = "core-image-base"
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', image) or ""
+        cmd = "runqemu %s nographic serial wic" % (runqemu_params)
+
+        self.write_config("""
+# efi firmware must load systemd-boot, not grub
+EFI_PROVIDER = "systemd-boot"
+
+# image format must be wic, needs esp partition for firmware etc
+IMAGE_FSTYPES:pn-%s:append = " wic"
+WKS_FILE = "efi-uki-bootdisk.wks.in"
+
+# efi, uki and systemd features must be enabled
+INIT_MANAGER = "systemd"
+DISTRO_FEATURES += "systemd-initramfs"
+MACHINE_FEATURES:append = " efi"
+IMAGE_CLASSES:append:pn-core-image-base = " uki"
+
+# uki embeds also an initrd
+INITRAMFS_IMAGE = "core-image-initramfs-boot"
+
+# runqemu must not load kernel separately, it's in the uki
+QB_KERNEL_ROOT = ""
+QB_DEFAULT_KERNEL = "none"
+
+# u-boot, not all qemu* machines set this correctly
+QB_DEFAULT_BIOS = "u-boot.bin"
+# machines may not set this correctly
+QB_DEFAULT_FSTYPE = "wic"
+
+# u-boot needs to find ESP partition so use virtio block device instead of default scsi
+QB_ROOTFS_OPT = "-drive id=root,file=@ROOTFS@,if=none,format=raw -device virtio-blk-pci,drive=root"
+QB_DRIVE_TYPE = "/dev/vd"
+
+# boot command line provided via uki, not via bootloader
+UKI_CMDLINE = "rootwait root=LABEL=root"
+# enable if debug output is needed
+# UKI_CMDLINE += "systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1"
+
+# disable kvm, breaks boot
+QEMU_USE_KVM = ""
+
+IMAGE_CLASSES:remove = 'testimage'
+""" % (image))
+
+        uki_filename = get_bb_var('UKI_FILENAME', image)
+
+        bitbake(image + " u-boot")
+        with runqemu(image, ssh=False, launch_cmd=cmd) as qemu:
+            self.assertTrue(qemu.runner.logged, "Failed: %s" % cmd)
+
+            # Verify from efivars that firmware was:
+            # aarch64
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderFirmwareInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep 'Das U-Boot'"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that systemd-boot was the loader
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep systemd-boot"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that systemd-stub was used
+            cmd = "echo $( cat /sys/firmware/efi/efivars/StubInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep systemd-stub"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that the compiled uki file was booted into
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderEntrySelected-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep '%s'" % (uki_filename)
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
-- 
2.43.0

[PATCH v4 7/9] image_types_wic.bbclass: capture verbose wic output by default

[Mikko Rapeli]

Call wic with --debug to capture logs from wic internals
so that it's clear which partitions get created and which
files get copied where. wic plugins contain for example
race conditions which don't install files at all and thus
images fail to boot and it's not possible to debug these without
something in wic task logs.

For example core-image-initramfs-boot do_image_wic
log is now 576 lines which is not excessive but very
important when debugging problems, especially race
conditions which are only hit in some builds in CI.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/image_types_wic.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
index 1b422b6280..10888bc12b 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -72,7 +72,7 @@ IMAGE_CMD:wic () {
 	if [ -z "$wks" ]; then
 		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
 	fi
-	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
+	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create --debug "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
 
 	# look to see if the user specifies a custom imager
 	IMAGER=direct
-- 
2.43.0

[PATCH v4 8/9] wic bootimg-efi.py: fail build if no binaries installed

[Mikko Rapeli]

With systemd-boot, some builds included correct EFI
bootloader binaries and some not. Thus some builds
booted and some not. Check that some boot binary
was installed so that build fails if none were installed.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 scripts/lib/wic/plugins/source/bootimg-efi.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 96c710bf77..20b1c36a46 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -341,6 +341,9 @@ class BootimgEFIPlugin(SourcePlugin):
                     cp_cmd = "cp -v -p %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
                     out = exec_cmd(cp_cmd, True)
                     logger.debug("systemd-boot files:\n%s" % out)
+                # must have installed at least one EFI bootloader
+                out = exec_cmd("ls %s/EFI/BOOT/boot*.efi" % (hdddir), True)
+                logger.debug("Installed systemd-boot files:\n%s" % out)
             elif source_params['loader'] == 'uefi-kernel':
                 kernel = get_bitbake_var("KERNEL_IMAGETYPE")
                 if not kernel:
-- 
2.43.0

[PATCH v4 9/9] image_types_wic.bbclass: depend on grub-efi and systemd-boot on aarch64, systemd-boot on arm

[Mikko Rapeli]

On x86 family builds, grub and systemd-boot are always included and thus
get built and deployed before wic image builds. On aarch64 builds
that was not the case. Result is that some builds added systemd-boot binaries
to the wic image ESP partition and some not, though bugs in wic plugins
contributed here too since missing files were silently ignored.
Boot of such images fails since firmware is not able to load the
default EFI binaries.

Adding EFI_PROVIDER to WKS_FILE_DEPENDS_BOOTLOADERS is not possible
because of parsing order so adding grub-efi and systemd-boot
to aarch64 and systemd-boot to arm depends
which makes sure their do_deploy is always executed
before wic image is generated. Thus systemd-boot and grub
binaries will get copied from deploy directory to the wic
image ESP partition, and boot of the images succeeds.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/image_types_wic.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
index 10888bc12b..b2f450026d 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -113,6 +113,8 @@ WKS_FILE_DEPENDS_DEFAULT += "bmaptool-native cdrtools-native btrfs-tools-native
 # Unified kernel images need objcopy
 WKS_FILE_DEPENDS_DEFAULT += "virtual/cross-binutils"
 WKS_FILE_DEPENDS_BOOTLOADERS = ""
+WKS_FILE_DEPENDS_BOOTLOADERS:aarch64 = "grub-efi systemd-boot"
+WKS_FILE_DEPENDS_BOOTLOADERS:arm = "systemd-boot"
 WKS_FILE_DEPENDS_BOOTLOADERS:x86 = "syslinux grub-efi systemd-boot os-release"
 WKS_FILE_DEPENDS_BOOTLOADERS:x86-64 = "syslinux systemd-boot os-release"
 WKS_FILE_DEPENDS_BOOTLOADERS:x86-x32 = "syslinux grub-efi"
-- 
2.43.0

Re: [OE-core] [PATCH v4 9/9] image_types_wic.bbclass: depend on grub-efi and systemd-boot on aarch64, systemd-boot on arm

[Richard Purdie]

On Tue, 2025-04-22 at 17:35 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> On x86 family builds, grub and systemd-boot are always included and thus
> get built and deployed before wic image builds. On aarch64 builds
> that was not the case. Result is that some builds added systemd-boot binaries
> to the wic image ESP partition and some not, though bugs in wic plugins
> contributed here too since missing files were silently ignored.
> Boot of such images fails since firmware is not able to load the
> default EFI binaries.
> 
> Adding EFI_PROVIDER to WKS_FILE_DEPENDS_BOOTLOADERS is not possible
> because of parsing order so adding grub-efi and systemd-boot
> to aarch64 and systemd-boot to arm depends
> which makes sure their do_deploy is always executed
> before wic image is generated. Thus systemd-boot and grub
> binaries will get copied from deploy directory to the wic
> image ESP partition, and boot of the images succeeds.
> 
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/classes-recipe/image_types_wic.bbclass | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
> index 10888bc12b..b2f450026d 100644
> --- a/meta/classes-recipe/image_types_wic.bbclass
> +++ b/meta/classes-recipe/image_types_wic.bbclass
> @@ -113,6 +113,8 @@ WKS_FILE_DEPENDS_DEFAULT += "bmaptool-native cdrtools-native btrfs-tools-native
>  # Unified kernel images need objcopy
>  WKS_FILE_DEPENDS_DEFAULT += "virtual/cross-binutils"
>  WKS_FILE_DEPENDS_BOOTLOADERS = ""
> +WKS_FILE_DEPENDS_BOOTLOADERS:aarch64 = "grub-efi systemd-boot"
> +WKS_FILE_DEPENDS_BOOTLOADERS:arm = "systemd-boot"
>  WKS_FILE_DEPENDS_BOOTLOADERS:x86 = "syslinux grub-efi systemd-boot os-release"
>  WKS_FILE_DEPENDS_BOOTLOADERS:x86-64 = "syslinux systemd-boot os-release"
>  WKS_FILE_DEPENDS_BOOTLOADERS:x86-x32 = "syslinux grub-efi"

Several of us had questions in patch review about why some entries had
os-release and some did not. We believe other uki changes have
obsoleted the need for that but it would be good to remove it if so and
keep these entries looking consistent.

Cheers,

Richard

Re: [OE-core] [PATCH v4 8/9] wic bootimg-efi.py: fail build if no binaries installed

[Richard Purdie]

On Tue, 2025-04-22 at 17:35 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> With systemd-boot, some builds included correct EFI
> bootloader binaries and some not. Thus some builds
> booted and some not. Check that some boot binary
> was installed so that build fails if none were installed.
> 
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  scripts/lib/wic/plugins/source/bootimg-efi.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
> index 96c710bf77..20b1c36a46 100644
> --- a/scripts/lib/wic/plugins/source/bootimg-efi.py
> +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
> @@ -341,6 +341,9 @@ class BootimgEFIPlugin(SourcePlugin):
>                      cp_cmd = "cp -v -p %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
>                      out = exec_cmd(cp_cmd, True)
>                      logger.debug("systemd-boot files:\n%s" % out)
> +                # must have installed at least one EFI bootloader
> +                out = exec_cmd("ls %s/EFI/BOOT/boot*.efi" % (hdddir), True)
> +                logger.debug("Installed systemd-boot files:\n%s" % out)
>              elif source_params['loader'] == 'uefi-kernel':
>                  kernel = get_bitbake_var("KERNEL_IMAGETYPE")
>                  if not kernel:

Lets avoid the exec overhead of using "ls" here and use glob.glob() to
check for the files, which makes the code more readable/maintainable
into the future.

Thanks!

Richard

Re: [OE-core] [PATCH v4 1/9] poky-altcfg.conf: enable "efi" in DISTRO_FEATURES

[Richard Purdie]

On Tue, 2025-04-22 at 17:34 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> oe-core machines like genericarm64 already enable "efi" in
> MACHINE_FEATURES but this does not enable systemd side
> "efi" support and thus systemd does not mount ESP
> partition for example.
> 
> Enabling "efi" in systemd PACKAGECONFIG got negative comments
> in review. Also enabling systemd "efi" support via MACHINE_FEATURES
> was rejected. Thus enabling this as distro feature for "poky-altcfg".
> 
> Many HW targets come with UEFI compatible firmware by default.
> For example u-boot enables UEFI support by default on all targets.
> 
> This enables seeing UEFI firmware interfaces and data
> in userspace since "efivars" is now mounted by systemd
> to /sys/firmware/efi/efivars. Things like UEFI bootloader
> (grub, systemd-boot etc) selections and secure boot status
> can be queried from there.
> 
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta-poky/conf/distro/poky-altcfg.conf | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta-poky/conf/distro/poky-altcfg.conf b/meta-poky/conf/distro/poky-altcfg.conf
> index 3d64bdd1fe..e56c4f0a37 100644
> --- a/meta-poky/conf/distro/poky-altcfg.conf
> +++ b/meta-poky/conf/distro/poky-altcfg.conf
> @@ -14,4 +14,7 @@ POKY_INIT_MANAGER:poky-altcfg = "systemd"
>  # systemd isn't suitable with musl
>  POKY_INIT_MANAGER:poky-altcfg:libc-musl = "sysvinit"
>  
> +# for systemd etc UEFI support
> +DISTRO_FEATURES:append = " efi"
> +
>  require conf/distro/include/poky-distro-alt-test-config.inc

Having thought a bit about this, I think we should just add this by
default to nodistro. I'm trying to keep the delta between poky and
nodistro minimal if/where we can.

Cheers,

Richard

Re: [OE-core] [PATCH v4 7/9] image_types_wic.bbclass: capture verbose wic output by default

[Mathieu Dubois-Briand]

On Tue Apr 22, 2025 at 4:34 PM CEST, Mikko Rapeli via lists.openembedded.org wrote:
> Call wic with --debug to capture logs from wic internals
> so that it's clear which partitions get created and which
> files get copied where. wic plugins contain for example
> race conditions which don't install files at all and thus
> images fail to boot and it's not possible to debug these without
> something in wic task logs.
>
> For example core-image-initramfs-boot do_image_wic
> log is now 576 lines which is not excessive but very
> important when debugging problems, especially race
> conditions which are only hit in some builds in CI.
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/classes-recipe/image_types_wic.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
> index 1b422b6280..10888bc12b 100644
> --- a/meta/classes-recipe/image_types_wic.bbclass
> +++ b/meta/classes-recipe/image_types_wic.bbclass
> @@ -72,7 +72,7 @@ IMAGE_CMD:wic () {
>  	if [ -z "$wks" ]; then
>  		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
>  	fi
> -	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> +	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create --debug "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
>  
>  	# look to see if the user specifies a custom imager
>  	IMAGER=direct

Hi Mikko,

As we dropped the "oeqa wic.py: clean image build dir before rebuild in
test_permissions" patch, we again have an issue with this one.

2025-04-24 16:54:36,535 - oe-selftest - INFO - wic.Wic.test_permissions (subunit.RemotedTestCase)
2025-04-24 16:54:36,536 - oe-selftest - INFO -  ... FAIL
...
| DEBUG: Python function extend_recipe_sysroot finished
| DEBUG: Executing python function set_image_size
| DEBUG: 23394.800000 = 17996 * 1.300000
| DEBUG: 23394.800000 = max(23394.800000, 8192)[23394.800000] + 0
| DEBUG: 23395.000000 = int(23394.800000)
| DEBUG: 23395 = aligned(23395)
| DEBUG: returning 23395
| DEBUG: Python function set_image_size finished
| DEBUG: Executing shell function do_image_wic
| abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.
| Check logfile: /srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/pseudo//pseudo.log
| Aborted (core dumped)
| WARNING: exit code 134 from a shell command.
NOTE: recipe core-image-minimal-1.0-r0: task do_image_wic: Failed
ERROR: Task (/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/images/core-image-minimal.bb:do_image_wic) failed with exit code '1'
Pseudo log:
path mismatch [2 links]: ino 157047752 db '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/rootfs/var/log' req '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/tmp-wic/rootfs1/var/log'.
Setup complete, sending SIGUSR1 to pid 346075.

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1507

This can be reproduced locally:

Get https://web.git.yoctoproject.org/poky-ci-archive/tag/?h=autobuilder.yoctoproject.org/valkyrie/a-full-1456
and run 'oe-selftest -r wic.Wic.test_permissions'

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH v4 7/9] image_types_wic.bbclass: capture verbose wic output by default

[Mikko Rapeli]

Hi,

On Fri, Apr 25, 2025 at 11:03:54AM +0200, Mathieu Dubois-Briand wrote:
> On Tue Apr 22, 2025 at 4:34 PM CEST, Mikko Rapeli via lists.openembedded.org wrote:
> > Call wic with --debug to capture logs from wic internals
> > so that it's clear which partitions get created and which
> > files get copied where. wic plugins contain for example
> > race conditions which don't install files at all and thus
> > images fail to boot and it's not possible to debug these without
> > something in wic task logs.
> >
> > For example core-image-initramfs-boot do_image_wic
> > log is now 576 lines which is not excessive but very
> > important when debugging problems, especially race
> > conditions which are only hit in some builds in CI.
> >
> > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > ---
> >  meta/classes-recipe/image_types_wic.bbclass | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
> > index 1b422b6280..10888bc12b 100644
> > --- a/meta/classes-recipe/image_types_wic.bbclass
> > +++ b/meta/classes-recipe/image_types_wic.bbclass
> > @@ -72,7 +72,7 @@ IMAGE_CMD:wic () {
> >  	if [ -z "$wks" ]; then
> >  		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
> >  	fi
> > -	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> > +	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create --debug "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> >  
> >  	# look to see if the user specifies a custom imager
> >  	IMAGER=direct
> 
> Hi Mikko,
> 
> As we dropped the "oeqa wic.py: clean image build dir before rebuild in
> test_permissions" patch, we again have an issue with this one.
> 
> 2025-04-24 16:54:36,535 - oe-selftest - INFO - wic.Wic.test_permissions (subunit.RemotedTestCase)
> 2025-04-24 16:54:36,536 - oe-selftest - INFO -  ... FAIL
> ...
> | DEBUG: Python function extend_recipe_sysroot finished
> | DEBUG: Executing python function set_image_size
> | DEBUG: 23394.800000 = 17996 * 1.300000
> | DEBUG: 23394.800000 = max(23394.800000, 8192)[23394.800000] + 0
> | DEBUG: 23395.000000 = int(23394.800000)
> | DEBUG: 23395 = aligned(23395)
> | DEBUG: returning 23395
> | DEBUG: Python function set_image_size finished
> | DEBUG: Executing shell function do_image_wic
> | abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.
> | Check logfile: /srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/pseudo//pseudo.log
> | Aborted (core dumped)
> | WARNING: exit code 134 from a shell command.
> NOTE: recipe core-image-minimal-1.0-r0: task do_image_wic: Failed
> ERROR: Task (/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/images/core-image-minimal.bb:do_image_wic) failed with exit code '1'
> Pseudo log:
> path mismatch [2 links]: ino 157047752 db '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/rootfs/var/log' req '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/tmp-wic/rootfs1/var/log'.
> Setup complete, sending SIGUSR1 to pid 346075.
> 
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1507
> 
> This can be reproduced locally:
> 
> Get https://web.git.yoctoproject.org/poky-ci-archive/tag/?h=autobuilder.yoctoproject.org/valkyrie/a-full-1456
> and run 'oe-selftest -r wic.Wic.test_permissions'

Yes. This pseudo issue needs to be root caused and fixed. Will need to get
into that.

FWIW, on aarch64 build host in bitbake devshell I see vim sometimes crashing with
pseudo aborts when opening files, sometimes also when closing, and sometimes
it works. These may be related.

$ bitbake -c devshell lttng-modules
...
root@ledge:~/src/base/repo/poky/build_test/tmp/work/genericarm64-poky-linux/lttng-modules/2.13.18/lttng-modules-2.13.18# vi ../../../../../work/genericarm64-poky-linux/linux-yocto/6.12.23+git/linux-genericarm64-standard-build/.config
Vim: Caught deadly signal ABRT
Vim: Finished.
Aborted

$ uname -a
Linux ledge 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 02:32:42 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

$ head -4 /etc/os-release 
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"

Cheers,

-Mikko

pseudo aborts on aarch64 ( Re: [OE-core] [PATCH v4 7/9] image_types_wic.bbclass: capture verbose wic output by default )

[Mikko Rapeli]

Hi,

On Fri, Apr 25, 2025 at 12:34:40PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> On Fri, Apr 25, 2025 at 11:03:54AM +0200, Mathieu Dubois-Briand wrote:
> > On Tue Apr 22, 2025 at 4:34 PM CEST, Mikko Rapeli via lists.openembedded.org wrote:
> > > Call wic with --debug to capture logs from wic internals
> > > so that it's clear which partitions get created and which
> > > files get copied where. wic plugins contain for example
> > > race conditions which don't install files at all and thus
> > > images fail to boot and it's not possible to debug these without
> > > something in wic task logs.
> > >
> > > For example core-image-initramfs-boot do_image_wic
> > > log is now 576 lines which is not excessive but very
> > > important when debugging problems, especially race
> > > conditions which are only hit in some builds in CI.
> > >
> > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > > ---
> > >  meta/classes-recipe/image_types_wic.bbclass | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
> > > index 1b422b6280..10888bc12b 100644
> > > --- a/meta/classes-recipe/image_types_wic.bbclass
> > > +++ b/meta/classes-recipe/image_types_wic.bbclass
> > > @@ -72,7 +72,7 @@ IMAGE_CMD:wic () {
> > >  	if [ -z "$wks" ]; then
> > >  		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
> > >  	fi
> > > -	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> > > +	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create --debug "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> > >  
> > >  	# look to see if the user specifies a custom imager
> > >  	IMAGER=direct
> > 
> > Hi Mikko,
> > 
> > As we dropped the "oeqa wic.py: clean image build dir before rebuild in
> > test_permissions" patch, we again have an issue with this one.
> > 
> > 2025-04-24 16:54:36,535 - oe-selftest - INFO - wic.Wic.test_permissions (subunit.RemotedTestCase)
> > 2025-04-24 16:54:36,536 - oe-selftest - INFO -  ... FAIL
> > ...
> > | DEBUG: Python function extend_recipe_sysroot finished
> > | DEBUG: Executing python function set_image_size
> > | DEBUG: 23394.800000 = 17996 * 1.300000
> > | DEBUG: 23394.800000 = max(23394.800000, 8192)[23394.800000] + 0
> > | DEBUG: 23395.000000 = int(23394.800000)
> > | DEBUG: 23395 = aligned(23395)
> > | DEBUG: returning 23395
> > | DEBUG: Python function set_image_size finished
> > | DEBUG: Executing shell function do_image_wic
> > | abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.
> > | Check logfile: /srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/pseudo//pseudo.log
> > | Aborted (core dumped)
> > | WARNING: exit code 134 from a shell command.
> > NOTE: recipe core-image-minimal-1.0-r0: task do_image_wic: Failed
> > ERROR: Task (/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/images/core-image-minimal.bb:do_image_wic) failed with exit code '1'
> > Pseudo log:
> > path mismatch [2 links]: ino 157047752 db '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/rootfs/var/log' req '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/tmp-wic/rootfs1/var/log'.
> > Setup complete, sending SIGUSR1 to pid 346075.
> > 
> > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1507
> > 
> > This can be reproduced locally:
> > 
> > Get https://web.git.yoctoproject.org/poky-ci-archive/tag/?h=autobuilder.yoctoproject.org/valkyrie/a-full-1456
> > and run 'oe-selftest -r wic.Wic.test_permissions'
> 
> Yes. This pseudo issue needs to be root caused and fixed. Will need to get
> into that.
> 
> FWIW, on aarch64 build host in bitbake devshell I see vim sometimes crashing with
> pseudo aborts when opening files, sometimes also when closing, and sometimes
> it works. These may be related.
> 
> $ bitbake -c devshell lttng-modules
> ...
> root@ledge:~/src/base/repo/poky/build_test/tmp/work/genericarm64-poky-linux/lttng-modules/2.13.18/lttng-modules-2.13.18# vi ../../../../../work/genericarm64-poky-linux/linux-yocto/6.12.23+git/linux-genericarm64-standard-build/.config
> Vim: Caught deadly signal ABRT
> Vim: Finished.
> Aborted

# tail -1 ../pseudo/pseudo.log
path mismatch [1 link]: ino 36752721 db '/home/mcfrisk/src/base/repo/poky/build_test/tmp/work-shared/genericarm64/kernel-source/.Makefile.swp' req '/home/mcfrisk/src/base/repo/poky/build_test/tmp/work/genericarm64-poky-linux/linux-yocto/6.12.23+git/linux-genericarm64-standard-build/.config.swp'.

So these swap files opened and closed by vim confuse pseudo. Disabling
them with 'vi -n' fixes this.

Richard mention yesterday in the patch review call that the fast opening
and closing of files and inode reuse is triggering this. The accounting
done by pseudo breaks somehow on arm64/aarch64 but works on x86_64
build hosts.

Cheers,

-Mikko

> $ uname -a
> Linux ledge 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 02:32:42 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
> 
> $ head -4 /etc/os-release 
> PRETTY_NAME="Ubuntu 24.04 LTS"
> NAME="Ubuntu"
> VERSION_ID="24.04"
> VERSION="24.04 LTS (Noble Numbat)"
> 
> Cheers,
> 
> -Mikko

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#215449): https://lists.openembedded.org/g/openembedded-core/message/215449
> Mute This Topic: https://lists.openembedded.org/mt/112395861/7159507
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [mikko.rapeli@linaro.org]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: pseudo aborts on aarch64 ( Re: [OE-core] [PATCH v4 7/9] image_types_wic.bbclass: capture verbose wic output by default )

[Mikko Rapeli]

Hi,

On Fri, Apr 25, 2025 at 01:12:59PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> On Fri, Apr 25, 2025 at 12:34:40PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > On Fri, Apr 25, 2025 at 11:03:54AM +0200, Mathieu Dubois-Briand wrote:
> > > On Tue Apr 22, 2025 at 4:34 PM CEST, Mikko Rapeli via lists.openembedded.org wrote:
> > > > Call wic with --debug to capture logs from wic internals
> > > > so that it's clear which partitions get created and which
> > > > files get copied where. wic plugins contain for example
> > > > race conditions which don't install files at all and thus
> > > > images fail to boot and it's not possible to debug these without
> > > > something in wic task logs.
> > > >
> > > > For example core-image-initramfs-boot do_image_wic
> > > > log is now 576 lines which is not excessive but very
> > > > important when debugging problems, especially race
> > > > conditions which are only hit in some builds in CI.
> > > >
> > > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > > > ---
> > > >  meta/classes-recipe/image_types_wic.bbclass | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
> > > > index 1b422b6280..10888bc12b 100644
> > > > --- a/meta/classes-recipe/image_types_wic.bbclass
> > > > +++ b/meta/classes-recipe/image_types_wic.bbclass
> > > > @@ -72,7 +72,7 @@ IMAGE_CMD:wic () {
> > > >  	if [ -z "$wks" ]; then
> > > >  		bbfatal "No kickstart files from WKS_FILES were found: ${WKS_FILES}. Please set WKS_FILE or WKS_FILES appropriately."
> > > >  	fi
> > > > -	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> > > > +	BUILDDIR="${TOPDIR}" PSEUDO_UNLOAD=1 wic create --debug "$wks" --vars "${STAGING_DIR}/${MACHINE}/imgdata/" -e "${IMAGE_BASENAME}" -o "$build_wic/" -w "$tmp_wic" ${WIC_CREATE_EXTRA_ARGS}
> > > >  
> > > >  	# look to see if the user specifies a custom imager
> > > >  	IMAGER=direct
> > > 
> > > Hi Mikko,
> > > 
> > > As we dropped the "oeqa wic.py: clean image build dir before rebuild in
> > > test_permissions" patch, we again have an issue with this one.
> > > 
> > > 2025-04-24 16:54:36,535 - oe-selftest - INFO - wic.Wic.test_permissions (subunit.RemotedTestCase)
> > > 2025-04-24 16:54:36,536 - oe-selftest - INFO -  ... FAIL
> > > ...
> > > | DEBUG: Python function extend_recipe_sysroot finished
> > > | DEBUG: Executing python function set_image_size
> > > | DEBUG: 23394.800000 = 17996 * 1.300000
> > > | DEBUG: 23394.800000 = max(23394.800000, 8192)[23394.800000] + 0
> > > | DEBUG: 23395.000000 = int(23394.800000)
> > > | DEBUG: 23395 = aligned(23395)
> > > | DEBUG: returning 23395
> > > | DEBUG: Python function set_image_size finished
> > > | DEBUG: Executing shell function do_image_wic
> > > | abort()ing pseudo client by server request. See https://wiki.yoctoproject.org/wiki/Pseudo_Abort for more details on this.
> > > | Check logfile: /srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/pseudo//pseudo.log
> > > | Aborted (core dumped)
> > > | WARNING: exit code 134 from a shell command.
> > > NOTE: recipe core-image-minimal-1.0-r0: task do_image_wic: Failed
> > > ERROR: Task (/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/images/core-image-minimal.bb:do_image_wic) failed with exit code '1'
> > > Pseudo log:
> > > path mismatch [2 links]: ino 157047752 db '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/rootfs/var/log' req '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1239956/tmp/work/qemuarm64-poky-linux/core-image-minimal/1.0/tmp-wic/rootfs1/var/log'.
> > > Setup complete, sending SIGUSR1 to pid 346075.
> > > 
> > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1507
> > > 
> > > This can be reproduced locally:
> > > 
> > > Get https://web.git.yoctoproject.org/poky-ci-archive/tag/?h=autobuilder.yoctoproject.org/valkyrie/a-full-1456
> > > and run 'oe-selftest -r wic.Wic.test_permissions'
> > 
> > Yes. This pseudo issue needs to be root caused and fixed. Will need to get
> > into that.
> > 
> > FWIW, on aarch64 build host in bitbake devshell I see vim sometimes crashing with
> > pseudo aborts when opening files, sometimes also when closing, and sometimes
> > it works. These may be related.
> > 
> > $ bitbake -c devshell lttng-modules
> > ...
> > root@ledge:~/src/base/repo/poky/build_test/tmp/work/genericarm64-poky-linux/lttng-modules/2.13.18/lttng-modules-2.13.18# vi ../../../../../work/genericarm64-poky-linux/linux-yocto/6.12.23+git/linux-genericarm64-standard-build/.config
> > Vim: Caught deadly signal ABRT
> > Vim: Finished.
> > Aborted
> 
> # tail -1 ../pseudo/pseudo.log
> path mismatch [1 link]: ino 36752721 db '/home/mcfrisk/src/base/repo/poky/build_test/tmp/work-shared/genericarm64/kernel-source/.Makefile.swp' req '/home/mcfrisk/src/base/repo/poky/build_test/tmp/work/genericarm64-poky-linux/linux-yocto/6.12.23+git/linux-genericarm64-standard-build/.config.swp'.
> 
> So these swap files opened and closed by vim confuse pseudo. Disabling
> them with 'vi -n' fixes this.
> 
> Richard mention yesterday in the patch review call that the fast opening
> and closing of files and inode reuse is triggering this. The accounting
> done by pseudo breaks somehow on arm64/aarch64 but works on x86_64
> build hosts.

Re-reading https://wiki.yoctoproject.org/wiki/Pseudo_Abort and
I don't think this is a bug. Just a very annoying thing. User can't use
vim editor inside and outside of pseudo/"bitbake -c devshell". The process
will open temp files in various locations and possibly delete them and
pseudo will get confused and start aborting.

I don't think this can be fixed. Workarounds, well, don't edit anything
under devshell. I need to find new ways to create patches to various
recipes. I'm used to opening devshell after do_install to test applying
patches and then manually running do_configure, do_compile and do_install
tasks to test things out before doing full recipe and image builds.
Would be nice if the pseudo checks only applied to files inside
recipe workspace, but I guess that filtering is tricky.

Then this wic selftest regression, since there was opposition to enabling
more verbose logs so I will just drop this. There are real bugs
in wic which for some reason only get exposed by this verbose flag.

The bootloader config files generated by wic are done without pseudo
and thus wic and bitbake builds differ. This is true for systemd-boot,
this failing case, and also with grub when EFI_LOADER = "grub-efi"
which aborts with:

path mismatch [2 links]: ino 33909680 db '/home/mcfrisk/src/base/repo/poky/build_test-st/tmp/work/genericarm64-poky-linux/core-image-minimal/1.0/rootfs/boot/Image' req '/home/mcfrisk/src/base/repo/poky/build_test-st/tmp/work/genericarm64-poky-linux/core-image-minimal/1.0/tmp-wic/rootfs1/boot/Image'.

To me the difference is calling "wic" as normal user vs calling
"wic" under pseudo fakeroot shell as root inside bitbake env.

I don't think the output of both can ever be the same.

The failing sequence is:

 * modify wks file
 * call "wic" to build the image
 * build the same image with bitbake

The failure happens at bitbake image build. If bitbake image is built before
wic then the test passes. Same with cleaning the sysroot and pseudo databases
before building the image.

A lot of the wic plugins call "cp" and write directly to config files
without passing through pseudo. All of these break pseudo.
Fix in wic would always need to use pseudo when creating
files, directores and when copying files. This is currently not
the case and a lot of code would need to be refactored.
I'm not willing to do this now, sorry.

Cheers,

-Mikko