[PATCH v8 0/8] systemd uki support

[PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

These changes enable building systemd uki images which combine
kernel, kernel command line, initrd and possibly signatures to
a single UEFI binary. This binary can be booted with UEFI firmware
and systemd-boot. No grub is needed and UEFI firmware and/or
systemd-boot provide possibilities for boot menus.
The uki binary can also be signed for UEFI secure boot
so the secure boot extends from firmware to kernel and initrd.
Binding secure boot to full userspace is then easier since for example
kernel command line and initrd contain the support needed to mount
encrypted dm-verity etc partitions, and/or create partitions on demand
with systemd-repart using device specific TPM devices for encryption.

Tested on qemuarm64-secureboot machine from meta-arm with changes to
support secure boot. Slightly different configuration tested on
multiple arm64 System Ready boards with UEFI firmware, real and firmware
based TPM devices. Tested with ovmf firmware on x86_64 with selftests but
without secure boot which seems to be harder to setup in ovmf.

Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and
wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests
on a build machine with zfs filesystem. Will investigate this further.
The issue seems to be in mkfs.ext4 producing broken filesystem, and partially
in the tests which don't run the correct rootfs file (.ext4 vs .wic).
Will debug this further and it is IMO unrelated to these changes since
they reproduce on pure master branch without this series.

v8: fixed comments from Ross Burton: debug print from warning to debug,
    dropped duplicate DISTRO_FEATURE setting for systemd in tests,
    removed aarch64 comment from tests which are currently x86 only.
    Fixed the new aarch64 wic selftest to run on both genericarm64
    and qemuarm64 by adding bios, virtio disk driver etc settings
    for runqemu (already set in genericarm64 but missing from qemuarm64).

v7: add missing "ovmf" to runqemu argument to
    test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang

v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on
    genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added
    selftest to cover this wks usage on x86 and aarch64

v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release
    dependency for all archs" since systemd-boot does not support all
    architectures

v4: handle missing runqemu variable from build config, add
python3-pefile to fast ptest list

v3: rebased, fixed and added more sefltests, removed wic plugin side uki
support

v2: https://lists.openembedded.org/g/openembedded-core/message/204090

Michelle Lin (1):
  uki.bbclass: add class for building Unified Kernel Images (UKI)

Mikko Rapeli (7):
  wic bootimg-efi.py: keep timestamps and add debug prints
  wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass
  oeqa selftest uki.py: add tests for uki.bbclass
  oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu
  oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi
    test
  oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu
  oeqa selftest wic.py: support UKIs via uki.bbclass

 meta-selftest/wic/test_efi_plugin.wks         |   5 +-
 .../test_efi_plugin_plain_systemd-boot.wks    |   5 +
 meta/classes-recipe/image_types_wic.bbclass   |   3 +-
 meta/classes-recipe/uki.bbclass               | 195 ++++++++++++++++++
 .../oeqa/selftest/cases/efibootpartition.py   |  11 +-
 meta/lib/oeqa/selftest/cases/uki.py           | 133 ++++++++++++
 meta/lib/oeqa/selftest/cases/wic.py           | 131 ++++++++++--
 .../wic/canned-wks/efi-uki-bootdisk.wks.in    |   3 +
 scripts/lib/wic/plugins/source/bootimg-efi.py | 183 +++++-----------
 9 files changed, 511 insertions(+), 158 deletions(-)
 create mode 100644 meta-selftest/wic/test_efi_plugin_plain_systemd-boot.wks
 create mode 100644 meta/classes-recipe/uki.bbclass
 create mode 100644 meta/lib/oeqa/selftest/cases/uki.py
 create mode 100644 scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in

-- 
2.34.1

[PATCH v8 1/8] uki.bbclass: add class for building Unified Kernel Images (UKI)

[Mikko Rapeli]

From: Michelle Lin <michelle.linto91@gmail.com>

This class calls systemd ukify tool, which will combine
kernel/initrd/stub components to build the UKI. To sign the UKI
(i.e. SecureBoot), the keys/cert files can be specified
in a configuration file or UEFI binary signing can be done
via separate steps, see qemuarm64-secureboot in meta-arm.
UKIs are loaded by UEFI firmware on target which can improve
security by loading only correctly signed kernel, initrd and kernel
command line.

Using systemd-measure to pre-calculate TPM PCR values and sign them is
not supported since that requires a TPM device on the build host. Thus
"ConditionSecurity=measured-uki" default from systemd 256 does not work
but "ConditionSecurity=tpm2" in combination with secure boot will.
These can be used to boot securely into systemd-boot, kernel, kernel
command line and initrd which then securely mounts a read-only dm-verity
/usr partition and creates a TPM encrypted read-write / rootfs.

Tested via qemuarm64-secureboot in meta-arm with
https://lists.yoctoproject.org/g/meta-arm/topic/patch_v3_02_13/108031399
and a few more changes needed, will be posted separately.

Signed-off-by: Michelle Lin <michelle.linto91@gmail.com>
Acked-by: Erik Schilling <erik.schilling@linaro.org>
Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/uki.bbclass | 195 ++++++++++++++++++++++++++++++++
 1 file changed, 195 insertions(+)
 create mode 100644 meta/classes-recipe/uki.bbclass

diff --git a/meta/classes-recipe/uki.bbclass b/meta/classes-recipe/uki.bbclass
new file mode 100644
index 0000000000..d4f25c7fd2
--- /dev/null
+++ b/meta/classes-recipe/uki.bbclass
@@ -0,0 +1,195 @@
+# Unified kernel image (UKI) class
+#
+# This bbclass merges kernel, initrd etc as a UKI standard UEFI binary,
+# to be loaded with UEFI firmware and systemd-boot on target HW.
+# TPM PCR pre-calculation is not supported since systemd-measure tooling
+# is meant to run on target, not in cross compile environment.
+#
+# See:
+# https://www.freedesktop.org/software/systemd/man/latest/ukify.html
+# https://uapi-group.org/specifications/specs/unified_kernel_image/
+#
+# The UKI contains:
+#
+#   - UEFI stub
+#     The linux kernel can generate a UEFI stub, however the one from systemd-boot can fetch
+#     the command line from a separate section of the EFI application, avoiding the need to
+#     rebuild the kernel.
+#   - kernel
+#   - initramfs
+#   - kernel command line
+#   - uname -r kernel version
+#   - /etc/os-release to create a boot menu with version details
+#   - optionally secure boot signature(s)
+#   - other metadata (e.g. TPM PCR measurements)
+#
+# Usage instructions:
+#
+#   - requires UEFI compatible firmware on target, e.g. qemuarm64-secureboot u-boot based
+#     from meta-arm or qemux86 ovmf/edk2 based firmware for x86_64
+#
+#   - Distro/build config:
+#
+#     INIT_MANAGER = "systemd"
+#     MACHINE_FEATURES:append = " efi"
+#     EFI_PROVIDER = "systemd-boot"
+#     INITRAMFS_IMAGE = "core-image-minimal-initramfs"
+#
+#   - image recipe:
+#
+#     inherit uki
+#
+#   - qemuboot/runqemu changes in image recipe or build config:
+#
+#     # Kernel command line must be inside the signed uki
+#     QB_KERNEL_ROOT = ""
+#     # kernel is in the uki image, not loaded separately
+#     QB_DEFAULT_KERNEL = "none"
+#
+#   - for UEFI secure boot, systemd-boot and uki (including kernel) can
+#     be signed but require sbsign-tool-native (recipe available from meta-secure-core,
+#     see also qemuarm64-secureboot from meta-arm). Set variable
+#     UKI_SB_KEY to path of private key and UKI_SB_CERT for certificate.
+#     Note that systemd-boot also need to be signed with the same key.
+#
+#   - at runtime, UEFI firmware will load and boot systemd-boot which
+#     creates a menu from all detected uki binaries. No need to manually
+#     setup boot menu entries.
+#
+#   - see efi-uki-bootdisk.wks.in how to create ESP partition which hosts systemd-boot,
+#     config file(s) for systemd-boot and the UKI binaries.
+#
+
+DEPENDS += "\
+    os-release \
+    systemd-boot \
+    systemd-boot-native \
+    virtual/${TARGET_PREFIX}binutils \
+    virtual/kernel \
+"
+
+inherit image-artifact-names
+require ../conf/image-uefi.conf
+
+INITRAMFS_IMAGE ?= "core-image-minimal-initramfs"
+
+INITRD_ARCHIVE ?= "${INITRAMFS_IMAGE}-${MACHINE}.${INITRAMFS_FSTYPES}"
+
+do_image_complete[depends] += "${INITRAMFS_IMAGE}:do_image_complete"
+
+UKIFY_CMD ?= "ukify build"
+UKI_CONFIG_FILE ?= "${UNPACKDIR}/uki.conf"
+UKI_FILENAME ?= "uki.efi"
+UKI_KERNEL_FILENAME ?= "${KERNEL_IMAGETYPE}"
+UKI_CMDLINE ?= "rootwait root=LABEL=root console=${KERNEL_CONSOLE}"
+# secure boot keys and cert, needs sbsign-tools-native (meta-secure-core)
+#UKI_SB_KEY ?= ""
+#UKI_SB_CERT ?= ""
+
+IMAGE_EFI_BOOT_FILES ?= "${UKI_FILENAME};EFI/Linux/${UKI_FILENAME}"
+
+do_uki[depends] += " \
+                        systemd-boot:do_deploy \
+                        virtual/kernel:do_deploy \
+                     "
+do_uki[depends] += "${@ '${INITRAMFS_IMAGE}:do_image_complete' if d.getVar('INITRAMFS_IMAGE') else ''}"
+
+# ensure that the build directory is empty everytime we generate a newly-created uki
+do_uki[cleandirs] = "${B}"
+# influence the build directory at the start of the builds
+do_uki[dirs] = "${B}"
+
+# we want to allow specifying files in SRC_URI, such as for signing the UKI
+python () {
+    d.delVarFlag("do_fetch","noexec")
+    d.delVarFlag("do_unpack","noexec")
+}
+
+# main task
+python do_uki() {
+    import glob
+    import bb.process
+
+    # base ukify command, can be extended if needed
+    ukify_cmd = d.getVar('UKIFY_CMD')
+
+    deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
+
+    # architecture
+    target_arch = d.getVar('EFI_ARCH')
+    if target_arch:
+        ukify_cmd += " --efi-arch %s" % (target_arch)
+
+    # systemd stubs
+    stub = "%s/linux%s.efi.stub" % (d.getVar('DEPLOY_DIR_IMAGE'), target_arch)
+    if not os.path.exists(stub):
+        bb.fatal(f"ERROR: cannot find {stub}.")
+    ukify_cmd += " --stub %s" % (stub)
+
+    # initrd
+    initramfs_image = "%s" % (d.getVar('INITRD_ARCHIVE'))
+    ukify_cmd += " --initrd=%s" % (os.path.join(deploy_dir_image, initramfs_image))
+
+    deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
+
+    # kernel
+    kernel_filename = d.getVar('UKI_KERNEL_FILENAME') or None
+    if kernel_filename:
+        kernel = "%s/%s" % (deploy_dir_image, kernel_filename)
+        if not os.path.exists(kernel):
+            bb.fatal(f"ERROR: cannot find %s" % (kernel))
+        ukify_cmd += " --linux=%s" % (kernel)
+        # not always needed, ukify can detect version from kernel binary
+        kernel_version = d.getVar('KERNEL_VERSION')
+        if kernel_version:
+            ukify_cmd += "--uname %s" % (kernel_version)
+    else:
+        bb.fatal("ERROR - UKI_KERNEL_FILENAME not set")
+
+    # command line
+    cmdline = d.getVar('UKI_CMDLINE')
+    if cmdline:
+        ukify_cmd += " --cmdline='%s'" % (cmdline)
+
+    # dtb
+    if d.getVar('KERNEL_DEVICETREE'):
+        for dtb in d.getVar('KERNEL_DEVICETREE').split():
+            dtb_path = "%s/%s" % (deploy_dir_image, dtb)
+            if not os.path.exists(dtb_path):
+                bb.fatal(f"ERROR: cannot find {dtb_path}.")
+            ukify_cmd += " --devicetree %s" % (dtb_path)
+
+    # custom config for ukify
+    if os.path.exists(d.getVar('UKI_CONFIG_FILE')):
+        ukify_cmd += " --config=%s" % (d.getVar('UKI_CONFIG_FILE'))
+
+    # systemd tools
+    ukify_cmd += " --tools=%s%s/lib/systemd/tools" % \
+        (d.getVar("RECIPE_SYSROOT_NATIVE"), d.getVar("prefix"))
+
+    # version
+    ukify_cmd += " --os-release=@%s%s/lib/os-release" % \
+        (d.getVar("RECIPE_SYSROOT"), d.getVar("prefix"))
+
+    # TODO: tpm2 measure for secure boot, depends on systemd-native and TPM tooling
+    # needed in systemd > 254 to fulfill ConditionSecurity=measured-uki
+    # Requires TPM device on build host, thus not supported at build time.
+    #ukify_cmd += " --measure"
+
+    # securebooot signing, also for kernel
+    key = d.getVar('UKI_SB_KEY')
+    if key:
+        ukify_cmd += " --sign-kernel --secureboot-private-key='%s'" % (key)
+    cert = d.getVar('UKI_SB_CERT')
+    if cert:
+        ukify_cmd += " --secureboot-certificate='%s'" % (cert)
+
+    # custom output UKI filename
+    output = " --output=%s/%s" % (d.getVar('DEPLOY_DIR_IMAGE'), d.getVar('UKI_FILENAME'))
+    ukify_cmd += " %s" % (output)
+
+    # Run the ukify command
+    bb.debug("uki: running command: %s" % (ukify_cmd))
+    bb.process.run(ukify_cmd, shell=True)
+}
+addtask uki after do_rootfs before do_deploy do_image_complete do_image_wic
-- 
2.34.1

[PATCH v8 2/8] wic bootimg-efi.py: keep timestamps and add debug prints

[Mikko Rapeli]

Keep timestamps etc to help build reproducibility.

Add prints to see what is being copied to ESP partition.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 scripts/lib/wic/plugins/source/bootimg-efi.py | 49 ++++++++++++-------
 1 file changed, 30 insertions(+), 19 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 7cc5131541..d00f5428da 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -43,16 +43,18 @@ class BootimgEFIPlugin(SourcePlugin):
         if initrd:
             initrds = initrd.split(';')
             for rd in initrds:
-                cp_cmd = "cp %s/%s %s" % (bootimg_dir, rd, hdddir)
-                exec_cmd(cp_cmd, True)
+                cp_cmd = "cp -v -p %s/%s %s" % (bootimg_dir, rd, hdddir)
+                out = exec_cmd(cp_cmd, True)
+                logger.debug("initrd files:\n%s" % (out))
         else:
             logger.debug("Ignoring missing initrd")
 
         if dtb:
             if ';' in dtb:
                 raise WicError("Only one DTB supported, exiting")
-            cp_cmd = "cp %s/%s %s" % (bootimg_dir, dtb, hdddir)
-            exec_cmd(cp_cmd, True)
+            cp_cmd = "cp -v -p %s/%s %s" % (bootimg_dir, dtb, hdddir)
+            out = exec_cmd(cp_cmd, True)
+            logger.debug("dtb files:\n%s" % (out))
 
     @classmethod
     def do_configure_grubefi(cls, hdddir, creator, cr_workdir, source_params):
@@ -150,6 +152,7 @@ class BootimgEFIPlugin(SourcePlugin):
                      "%s/hdd/boot/loader/loader.conf", cr_workdir)
         cfg = open("%s/hdd/boot/loader/loader.conf" % cr_workdir, "w")
         cfg.write(loader_conf)
+        logger.debug("loader.conf:\n%s" % (loader_conf))
         cfg.close()
 
         configfile = creator.ks.bootloader.configfile
@@ -401,30 +404,33 @@ class BootimgEFIPlugin(SourcePlugin):
                 exec_native_cmd(objcopy_cmd, native_sysroot)
         else:
             if source_params.get('install-kernel-into-boot-dir') != 'false':
-                install_cmd = "install -m 0644 %s/%s %s/%s" % \
+                install_cmd = "install -v -p -m 0644 %s/%s %s/%s" % \
                     (staging_kernel_dir, kernel, hdddir, kernel)
-                exec_cmd(install_cmd)
+                out = exec_cmd(install_cmd)
+                logger.debug("Installed kernel files:\n%s" % out)
 
         if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
             for src_path, dst_path in cls.install_task:
-                install_cmd = "install -m 0644 -D %s %s" \
+                install_cmd = "install -v -p -m 0644 -D %s %s" \
                               % (os.path.join(kernel_dir, src_path),
                                  os.path.join(hdddir, dst_path))
-                exec_cmd(install_cmd)
+                out = exec_cmd(install_cmd)
+                logger.debug("Installed IMAGE_EFI_BOOT_FILES:\n%s" % out)
 
         try:
             if source_params['loader'] == 'grub-efi':
                 shutil.copyfile("%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir,
                                 "%s/grub.cfg" % cr_workdir)
                 for mod in [x for x in os.listdir(kernel_dir) if x.startswith("grub-efi-")]:
-                    cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[9:])
+                    cp_cmd = "cp -v -p %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[9:])
                     exec_cmd(cp_cmd, True)
                 shutil.move("%s/grub.cfg" % cr_workdir,
                             "%s/hdd/boot/EFI/BOOT/grub.cfg" % cr_workdir)
             elif source_params['loader'] == 'systemd-boot':
                 for mod in [x for x in os.listdir(kernel_dir) if x.startswith("systemd-")]:
-                    cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
-                    exec_cmd(cp_cmd, True)
+                    cp_cmd = "cp -v -p %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, mod[8:])
+                    out = exec_cmd(cp_cmd, True)
+                    logger.debug("systemd-boot files:\n%s" % out)
             elif source_params['loader'] == 'uefi-kernel':
                 kernel = get_bitbake_var("KERNEL_IMAGETYPE")
                 if not kernel:
@@ -445,8 +451,9 @@ class BootimgEFIPlugin(SourcePlugin):
                     raise WicError("UEFI stub kernel is incompatible with target %s" % target)
 
                 for mod in [x for x in os.listdir(kernel_dir) if x.startswith(kernel)]:
-                    cp_cmd = "cp %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, kernel_efi_image)
-                    exec_cmd(cp_cmd, True)
+                    cp_cmd = "cp -v -p %s/%s %s/EFI/BOOT/%s" % (kernel_dir, mod, hdddir, kernel_efi_image)
+                    out = exec_cmd(cp_cmd, True)
+                    logger.debug("uefi-kernel files:\n%s" % out)
             else:
                 raise WicError("unrecognized bootimg-efi loader: %s" %
                                source_params['loader'])
@@ -455,13 +462,15 @@ class BootimgEFIPlugin(SourcePlugin):
 
         startup = os.path.join(kernel_dir, "startup.nsh")
         if os.path.exists(startup):
-            cp_cmd = "cp %s %s/" % (startup, hdddir)
-            exec_cmd(cp_cmd, True)
+            cp_cmd = "cp -v -p %s %s/" % (startup, hdddir)
+            out = exec_cmd(cp_cmd, True)
+            logger.debug("startup files:\n%s" % out)
 
         for paths in part.include_path or []:
             for path in paths:
-                cp_cmd = "cp -r %s %s/" % (path, hdddir)
+                cp_cmd = "cp -v -p -r %s %s/" % (path, hdddir)
                 exec_cmd(cp_cmd, True)
+                logger.debug("include_path files:\n%s" % out)
 
         du_cmd = "du -bks %s" % hdddir
         out = exec_cmd(du_cmd)
@@ -489,12 +498,14 @@ class BootimgEFIPlugin(SourcePlugin):
 
         label = part.label if part.label else "ESP"
 
-        dosfs_cmd = "mkdosfs -n %s -i %s -C %s %d" % \
+        dosfs_cmd = "mkdosfs -v -n %s -i %s -C %s %d" % \
                     (label, part.fsuuid, bootimg, blocks)
         exec_native_cmd(dosfs_cmd, native_sysroot)
+        logger.debug("mkdosfs:\n%s" % (str(out)))
 
-        mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
-        exec_native_cmd(mcopy_cmd, native_sysroot)
+        mcopy_cmd = "mcopy -v -p -i %s -s %s/* ::/" % (bootimg, hdddir)
+        out = exec_native_cmd(mcopy_cmd, native_sysroot)
+        logger.debug("mcopy:\n%s" % (str(out)))
 
         chmod_cmd = "chmod 644 %s" % bootimg
         exec_cmd(chmod_cmd)
-- 
2.34.1

[PATCH v8 3/8] wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass

[Mikko Rapeli]

Remove custom wic plugin implementation and use systemd ukify reference
implementation when generating UKI images. Fail if users still have
create-unified-kernel-image in wic image config. uki.bbclass use is
detected from IMAGE_CLASSES variable ("inherit uki" in image
recipe) so export that to wic plugins.

If UKI is used, then only generate a minimal loader config for
systemd-boot which basically just sets a timeout. Also set 5 second
timeout by default instead of failing if wic bootloader config is
missing. Boot menu is generated at runtime based on UKI binaries
found from ESP partition.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/classes-recipe/image_types_wic.bbclass   |   3 +-
 scripts/lib/wic/plugins/source/bootimg-efi.py | 140 ++++--------------
 2 files changed, 27 insertions(+), 116 deletions(-)

diff --git a/meta/classes-recipe/image_types_wic.bbclass b/meta/classes-recipe/image_types_wic.bbclass
index 1fa016c16e..9a2996658a 100644
--- a/meta/classes-recipe/image_types_wic.bbclass
+++ b/meta/classes-recipe/image_types_wic.bbclass
@@ -15,6 +15,7 @@ WICVARS ?= "\
 	HOSTTOOLS_DIR \
 	IMAGE_BASENAME \
 	IMAGE_BOOT_FILES \
+	IMAGE_CLASSES \
 	IMAGE_EFI_BOOT_FILES \
 	IMAGE_LINK_NAME \
 	IMAGE_ROOTFS \
@@ -113,7 +114,7 @@ WKS_FILE_DEPENDS_DEFAULT += "bmaptool-native cdrtools-native btrfs-tools-native
 WKS_FILE_DEPENDS_DEFAULT += "virtual/${TARGET_PREFIX}binutils"
 WKS_FILE_DEPENDS_BOOTLOADERS = ""
 WKS_FILE_DEPENDS_BOOTLOADERS:x86 = "syslinux grub-efi systemd-boot os-release"
-WKS_FILE_DEPENDS_BOOTLOADERS:x86-64 = "syslinux grub-efi systemd-boot os-release"
+WKS_FILE_DEPENDS_BOOTLOADERS:x86-64 = "syslinux systemd-boot os-release"
 WKS_FILE_DEPENDS_BOOTLOADERS:x86-x32 = "syslinux grub-efi"
 
 WKS_FILE_DEPENDS ??= "${WKS_FILE_DEPENDS_DEFAULT} ${WKS_FILE_DEPENDS_BOOTLOADERS}"
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index d00f5428da..c05f38f229 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -125,8 +125,16 @@ class BootimgEFIPlugin(SourcePlugin):
     @classmethod
     def do_configure_systemdboot(cls, hdddir, creator, cr_workdir, source_params):
         """
-        Create loader-specific systemd-boot/gummiboot config
+        Create loader-specific systemd-boot/gummiboot config. Unified Kernel Image (uki)
+        support is done in image recipe with uki.bbclass and only systemd-boot loader config
+        and ESP partition structure is created here.
         """
+        # detect uki.bbclass usage
+        image_classes = get_bitbake_var("IMAGE_CLASSES").split()
+        unified_image = False
+        if "uki" in image_classes:
+            unified_image = True
+
         install_cmd = "install -d %s/loader" % hdddir
         exec_cmd(install_cmd)
 
@@ -134,19 +142,10 @@ class BootimgEFIPlugin(SourcePlugin):
         exec_cmd(install_cmd)
 
         bootloader = creator.ks.bootloader
-
-        unified_image = source_params.get('create-unified-kernel-image') == "true"
-
         loader_conf = ""
-        if not unified_image:
-            loader_conf += "default boot\n"
-        loader_conf += "timeout %d\n" % bootloader.timeout
 
-        initrd = source_params.get('initrd')
-        dtb = source_params.get('dtb')
-
-        if not unified_image:
-            cls._copy_additional_files(hdddir, initrd, dtb)
+        # 5 seconds is a sensible default timeout
+        loader_conf += "timeout %d\n" % (bootloader.timeout or 5)
 
         logger.debug("Writing systemd-boot config "
                      "%s/hdd/boot/loader/loader.conf", cr_workdir)
@@ -155,8 +154,14 @@ class BootimgEFIPlugin(SourcePlugin):
         logger.debug("loader.conf:\n%s" % (loader_conf))
         cfg.close()
 
+        initrd = source_params.get('initrd')
+        dtb = source_params.get('dtb')
+        if not unified_image:
+            cls._copy_additional_files(hdddir, initrd, dtb)
+
         configfile = creator.ks.bootloader.configfile
         custom_cfg = None
+        boot_conf = ""
         if configfile:
             custom_cfg = get_custom_config(configfile)
             if custom_cfg:
@@ -167,8 +172,7 @@ class BootimgEFIPlugin(SourcePlugin):
             else:
                 raise WicError("configfile is specified but failed to "
                                "get it from %s.", configfile)
-
-        if not custom_cfg:
+        else:
             # Create systemd-boot configuration using parameters from wks file
             kernel = get_bitbake_var("KERNEL_IMAGETYPE")
             if get_bitbake_var("INITRAMFS_IMAGE_BUNDLE") == "1":
@@ -178,7 +182,6 @@ class BootimgEFIPlugin(SourcePlugin):
 
             title = source_params.get('title')
 
-            boot_conf = ""
             boot_conf += "title %s\n" % (title if title else "boot")
             boot_conf += "linux /%s\n" % kernel
 
@@ -203,6 +206,7 @@ class BootimgEFIPlugin(SourcePlugin):
                          "%s/hdd/boot/loader/entries/boot.conf", cr_workdir)
             cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w")
             cfg.write(boot_conf)
+            logger.debug("boot.conf:\n%s" % (boot_conf))
             cfg.close()
 
 
@@ -307,107 +311,13 @@ class BootimgEFIPlugin(SourcePlugin):
                     (get_bitbake_var("KERNEL_IMAGETYPE"), get_bitbake_var("INITRAMFS_LINK_NAME"))
 
         if source_params.get('create-unified-kernel-image') == "true":
-            initrd = source_params.get('initrd')
-            if not initrd:
-                raise WicError("initrd= must be specified when create-unified-kernel-image=true, exiting")
-
-            deploy_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
-            efi_stub = glob("%s/%s" % (deploy_dir, "linux*.efi.stub"))
-            if len(efi_stub) == 0:
-                raise WicError("Unified Kernel Image EFI stub not found, exiting")
-            efi_stub = efi_stub[0]
-
-            with tempfile.TemporaryDirectory() as tmp_dir:
-                label = source_params.get('label')
-                label_conf = "root=%s" % creator.rootdev
-                if label:
-                    label_conf = "LABEL=%s" % label
-
-                bootloader = creator.ks.bootloader
-                cmdline = open("%s/cmdline" % tmp_dir, "w")
-                cmdline.write("%s %s" % (label_conf, bootloader.append))
-                cmdline.close()
+            raise WicError("create-unified-kernel-image is no longer supported. Please use uki.bbclass.")
 
-                initrds = initrd.split(';')
-                initrd = open("%s/initrd" % tmp_dir, "wb")
-                for f in initrds:
-                    with open("%s/%s" % (deploy_dir, f), 'rb') as in_file:
-                        shutil.copyfileobj(in_file, initrd)
-                initrd.close()
-
-                # Searched by systemd-boot:
-                # https://systemd.io/BOOT_LOADER_SPECIFICATION/#type-2-efi-unified-kernel-images
-                install_cmd = "install -d %s/EFI/Linux" % hdddir
-                exec_cmd(install_cmd)
-
-                staging_dir_host = get_bitbake_var("STAGING_DIR_HOST")
-                target_sys = get_bitbake_var("TARGET_SYS")
-
-                objdump_cmd = "%s-objdump" % target_sys
-                objdump_cmd += " -p %s" % efi_stub
-                objdump_cmd += " | awk '{ if ($1 == \"SectionAlignment\"){print $2} }'"
-
-                ret, align_str = exec_native_cmd(objdump_cmd, native_sysroot)
-                align = int(align_str, 16)
-
-                objdump_cmd = "%s-objdump" % target_sys
-                objdump_cmd += " -h %s | tail -2" % efi_stub
-                ret, output = exec_native_cmd(objdump_cmd, native_sysroot)
-
-                offset = int(output.split()[2], 16) + int(output.split()[3], 16)
-
-                osrel_off = offset + align - offset % align
-                osrel_path = "%s/usr/lib/os-release" % staging_dir_host
-                osrel_sz = os.stat(osrel_path).st_size
-
-                cmdline_off = osrel_off + osrel_sz
-                cmdline_off = cmdline_off + align - cmdline_off % align
-                cmdline_sz = os.stat(cmdline.name).st_size
-
-                dtb_off = cmdline_off + cmdline_sz
-                dtb_off = dtb_off + align - dtb_off % align
-
-                dtb = source_params.get('dtb')
-                if dtb:
-                    if ';' in dtb:
-                        raise WicError("Only one DTB supported, exiting")
-                    dtb_path = "%s/%s" % (deploy_dir, dtb)
-                    dtb_params = '--add-section .dtb=%s --change-section-vma .dtb=0x%x' % \
-                            (dtb_path, dtb_off)
-                    linux_off = dtb_off + os.stat(dtb_path).st_size
-                    linux_off = linux_off + align - linux_off % align
-                else:
-                    dtb_params = ''
-                    linux_off = dtb_off
-
-                linux_path = "%s/%s" % (staging_kernel_dir, kernel)
-                linux_sz = os.stat(linux_path).st_size
-
-                initrd_off = linux_off + linux_sz
-                initrd_off = initrd_off + align - initrd_off % align
-
-                # https://www.freedesktop.org/software/systemd/man/systemd-stub.html
-                objcopy_cmd = "%s-objcopy" % target_sys
-                objcopy_cmd += " --enable-deterministic-archives"
-                objcopy_cmd += " --preserve-dates"
-                objcopy_cmd += " --add-section .osrel=%s" % osrel_path
-                objcopy_cmd += " --change-section-vma .osrel=0x%x" % osrel_off
-                objcopy_cmd += " --add-section .cmdline=%s" % cmdline.name
-                objcopy_cmd += " --change-section-vma .cmdline=0x%x" % cmdline_off
-                objcopy_cmd += dtb_params
-                objcopy_cmd += " --add-section .linux=%s" % linux_path
-                objcopy_cmd += " --change-section-vma .linux=0x%x" % linux_off
-                objcopy_cmd += " --add-section .initrd=%s" % initrd.name
-                objcopy_cmd += " --change-section-vma .initrd=0x%x" % initrd_off
-                objcopy_cmd += " %s %s/EFI/Linux/linux.efi" % (efi_stub, hdddir)
-
-                exec_native_cmd(objcopy_cmd, native_sysroot)
-        else:
-            if source_params.get('install-kernel-into-boot-dir') != 'false':
-                install_cmd = "install -v -p -m 0644 %s/%s %s/%s" % \
-                    (staging_kernel_dir, kernel, hdddir, kernel)
-                out = exec_cmd(install_cmd)
-                logger.debug("Installed kernel files:\n%s" % out)
+        if source_params.get('install-kernel-into-boot-dir') != 'false':
+            install_cmd = "install -v -p -m 0644 %s/%s %s/%s" % \
+                (staging_kernel_dir, kernel, hdddir, kernel)
+            out = exec_cmd(install_cmd)
+            logger.debug("Installed kernel files:\n%s" % out)
 
         if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
             for src_path, dst_path in cls.install_task:
-- 
2.34.1

[PATCH v8 4/8] oeqa selftest uki.py: add tests for uki.bbclass

[Mikko Rapeli]

Tests builds and boots qemu into uki binary with systemd and sysvinit.
Due to depedency to x86 specific ovmf UEFI firmware, tests
are specific to x86 curently. UEFI firmware for ARM can be generated
via qemuarm64-secureboot machine in meta-arm and similar tests
on qemu will pass.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/uki.py           | 133 ++++++++++++++++++
 .../wic/canned-wks/efi-uki-bootdisk.wks.in    |   3 +
 2 files changed, 136 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/uki.py
 create mode 100644 scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in

diff --git a/meta/lib/oeqa/selftest/cases/uki.py b/meta/lib/oeqa/selftest/cases/uki.py
new file mode 100644
index 0000000000..4121a094dc
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/uki.py
@@ -0,0 +1,133 @@
+# Based on runqemu.py test file
+#
+# Copyright (c) 2017 Wind River Systems, Inc.
+#
+# SPDX-License-Identifier: MIT
+#
+
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import bitbake, runqemu, get_bb_var
+from oeqa.core.decorator.data import skipIfNotArch
+from oeqa.core.decorator import OETestTag
+import oe.types
+
+class UkiTest(OESelftestTestCase):
+    """Boot Unified Kernel Image (UKI) generated with uki.bbclass on UEFI firmware (omvf/edk2)"""
+
+    @skipIfNotArch(['i586', 'i686', 'x86_64'])
+    @OETestTag("runqemu")
+    def test_uki_boot_systemd(self):
+        """Build and boot into UEFI firmware (omvf/edk2), systemd-boot, initrd without systemd, rootfs with systemd"""
+        image = "core-image-minimal"
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', image) or ""
+        cmd = "runqemu %s nographic serial wic ovmf" % (runqemu_params)
+        if oe.types.qemu_use_kvm(self.td.get('QEMU_USE_KVM', 0), self.td["TARGET_ARCH"]):
+            cmd += " kvm"
+
+        self.write_config("""
+# efi firmware must load systemd-boot, not grub
+EFI_PROVIDER = "systemd-boot"
+
+# image format must be wic, needs esp partition for firmware etc
+IMAGE_FSTYPES:pn-%s:append = " wic"
+WKS_FILE = "efi-uki-bootdisk.wks.in"
+
+# efi, uki and systemd features must be enabled
+INIT_MANAGER = "systemd"
+MACHINE_FEATURES:append = " efi"
+IMAGE_CLASSES:append:pn-core-image-minimal = " uki"
+
+# uki embeds also an initrd
+INITRAMFS_IMAGE = "core-image-minimal-initramfs"
+
+# runqemu must not load kernel separately, it's in the uki
+QB_KERNEL_ROOT = ""
+QB_DEFAULT_KERNEL = "none"
+
+# boot command line provided via uki, not via bootloader
+UKI_CMDLINE = "rootwait root=LABEL=root console=${KERNEL_CONSOLE}"
+""" % (image))
+
+        uki_filename = get_bb_var('UKI_FILENAME', image)
+
+        bitbake(image + " ovmf")
+        with runqemu(image, ssh=False, launch_cmd=cmd) as qemu:
+            self.assertTrue(qemu.runner.logged, "Failed: %s" % cmd)
+
+            # Verify from efivars that firmware was:
+            # x86_64, qemux86_64, ovmf = edk2
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderFirmwareInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep 'EDK II'"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that systemd-boot was the loader
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep systemd-boot"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that systemd-stub was used
+            cmd = "echo $( cat /sys/firmware/efi/efivars/StubInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep systemd-stub"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that the compiled uki file was booted into
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderEntrySelected-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep '%s'" % (uki_filename)
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+    @skipIfNotArch(['i586', 'i686', 'x86_64'])
+    @OETestTag("runqemu")
+    def test_uki_sysvinit(self):
+        """Build and boot into UEFI firmware (omvf/edk2), systemd-boot, initrd with sysvinit, rootfs with sysvinit"""
+        config = """
+# efi firmware must load systemd-boot, not grub
+EFI_PROVIDER = "systemd-boot"
+
+# image format must be wic, needs esp partition for firmware etc
+IMAGE_FSTYPES:pn-core-image-base:append = " wic"
+WKS_FILE = "efi-uki-bootdisk.wks.in"
+
+# efi, uki and systemd features must be enabled
+MACHINE_FEATURES:append = " efi"
+DISTRO_FEATURES_NATIVE:append = " systemd"
+IMAGE_CLASSES:append:pn-core-image-base = " uki"
+
+# uki embeds also an initrd, no systemd or udev
+INITRAMFS_IMAGE = "core-image-initramfs-boot"
+
+# runqemu must not load kernel separately, it's in the uki
+QB_KERNEL_ROOT = ""
+QB_DEFAULT_KERNEL = "none"
+
+# boot command line provided via uki, not via bootloader
+UKI_CMDLINE = "rootwait root=LABEL=root console=${KERNEL_CONSOLE}"
+
+"""
+        self.append_config(config)
+        bitbake('core-image-base ovmf')
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-base') or ""
+        uki_filename = get_bb_var('UKI_FILENAME', 'core-image-base')
+        self.remove_config(config)
+
+        with runqemu('core-image-base', ssh=False,
+                     runqemuparams='%s slirp nographic ovmf' % (runqemu_params), image_fstype='wic') as qemu:
+            # Verify from efivars that firmware was:
+            # x86_64, qemux86_64, ovmf = edk2
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderFirmwareInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep 'EDK II'"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that systemd-boot was the loader
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep systemd-boot"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that systemd-stub was used
+            cmd = "echo $( cat /sys/firmware/efi/efivars/StubInfo-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep systemd-stub"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+            # Check that the compiled uki file was booted into
+            cmd = "echo $( cat /sys/firmware/efi/efivars/LoaderEntrySelected-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f ) | grep '%s'" % (uki_filename)
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
diff --git a/scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in b/scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in
new file mode 100644
index 0000000000..1ea9c8a845
--- /dev/null
+++ b/scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in
@@ -0,0 +1,3 @@
+bootloader --ptable gpt --timeout=5
+part /boot --source bootimg-efi --sourceparams="loader=${EFI_PROVIDER}" --label boot --active --align 1024 --use-uuid --part-name="ESP" --part-type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B --fixed-size 512M
+part / --source rootfs --fstype=ext4 --label root --align 1024 --exclude-path boot/
-- 
2.34.1

[PATCH v8 5/8] oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu

[Mikko Rapeli]

TEST_RUNQEMUPARAMS variable is used to add runqemu parameters like
"slirp" networking. Support this also in selftests so that "slirp"
networking can be used instead of the tun/tap devices setup which
is easier to work with on shared build machines.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/efibootpartition.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/efibootpartition.py b/meta/lib/oeqa/selftest/cases/efibootpartition.py
index fa74103dec..e6d6a91a22 100644
--- a/meta/lib/oeqa/selftest/cases/efibootpartition.py
+++ b/meta/lib/oeqa/selftest/cases/efibootpartition.py
@@ -6,7 +6,7 @@
 #
 
 from oeqa.selftest.case import OESelftestTestCase
-from oeqa.utils.commands import bitbake, runqemu
+from oeqa.utils.commands import bitbake, runqemu, get_bb_var
 from oeqa.core.decorator.data import skipIfNotMachine
 import oe.types
 
@@ -14,10 +14,11 @@ class GenericEFITest(OESelftestTestCase):
     """EFI booting test class"""
     @skipIfNotMachine("qemux86-64", "test is qemux86-64 specific currently")
     def test_boot_efi(self):
-        cmd = "runqemu nographic serial wic ovmf"
+        image = "core-image-minimal"
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', image) or ""
+        cmd = "runqemu %s nographic serial wic ovmf" % (runqemu_params)
         if oe.types.qemu_use_kvm(self.td.get('QEMU_USE_KVM', 0), self.td["TARGET_ARCH"]):
             cmd += " kvm"
-        image = "core-image-minimal"
 
         self.write_config("""
 EFI_PROVIDER = "systemd-boot"
-- 
2.34.1

[PATCH v8 6/8] oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi test

[Mikko Rapeli]

The test is actually using grub-efi not systemd-boot so
remove it completely. systemd-boot will be tested via uki.py
tests.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/efibootpartition.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/efibootpartition.py b/meta/lib/oeqa/selftest/cases/efibootpartition.py
index e6d6a91a22..fcfcdaf7e4 100644
--- a/meta/lib/oeqa/selftest/cases/efibootpartition.py
+++ b/meta/lib/oeqa/selftest/cases/efibootpartition.py
@@ -21,11 +21,11 @@ class GenericEFITest(OESelftestTestCase):
             cmd += " kvm"
 
         self.write_config("""
-EFI_PROVIDER = "systemd-boot"
+EFI_PROVIDER = "grub-efi"
 IMAGE_FSTYPES:pn-%s:append = " wic"
 MACHINE_FEATURES:append = " efi"
 WKS_FILE = "efi-bootdisk.wks.in"
-IMAGE_INSTALL:append = " grub-efi systemd-boot kernel-image-bzimage"
+IMAGE_INSTALL:append = " grub-efi kernel-image-bzimage"
 """
 % (image))
 
-- 
2.34.1

[PATCH v8 7/8] oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu

[Mikko Rapeli]

To support "slirp" networking on shared build machines instead
of tun/tap devices. Users can set

TEST_RUNQEMUPARAMS = "slirp"

in their build/conf/local.conf to run selftests using "slirp"
networking. The same works for testimage.bbclass and oeqa runtime
tests.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/lib/oeqa/selftest/cases/wic.py | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index b616759209..f2a46c965b 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -939,7 +939,8 @@ class Wic2(WicTestCase):
         bitbake('wic-image-minimal')
         self.remove_config(config)
 
-        with runqemu('wic-image-minimal', ssh=False, runqemuparams='nographic') as qemu:
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'wic-image-minimal') or ""
+        with runqemu('wic-image-minimal', ssh=False, runqemuparams='%s nographic' % (runqemu_params)) as qemu:
             cmd = "mount | grep '^/dev/' | cut -f1,3 -d ' ' | egrep -c -e '/dev/sda1 /boot' " \
                   "-e '/dev/root /|/dev/sda2 /' -e '/dev/sda3 /media' -e '/dev/sda4 /mnt'"
             status, output = qemu.run_serial(cmd)
@@ -959,8 +960,9 @@ class Wic2(WicTestCase):
         bitbake('core-image-minimal ovmf')
         self.remove_config(config)
 
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-minimal') or ""
         with runqemu('core-image-minimal', ssh=False,
-                     runqemuparams='nographic ovmf', image_fstype='wic') as qemu:
+                     runqemuparams='%s nographic ovmf' % (runqemu_params), image_fstype='wic') as qemu:
             cmd = "grep sda. /proc/partitions  |wc -l"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
@@ -1154,8 +1156,9 @@ class Wic2(WicTestCase):
         bitbake('core-image-minimal-mtdutils')
         self.remove_config(config)
 
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-minimal-mtdutils') or ""
         with runqemu('core-image-minimal-mtdutils', ssh=False,
-                     runqemuparams='nographic', image_fstype='wic') as qemu:
+                     runqemuparams='%s nographic' % (runqemu_params), image_fstype='wic') as qemu:
             cmd = "grep sda. /proc/partitions  |wc -l"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
@@ -1214,8 +1217,9 @@ class Wic2(WicTestCase):
         bitbake('core-image-minimal')
         self.remove_config(config)
 
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-minimal') or ""
         with runqemu('core-image-minimal', ssh=False,
-                     runqemuparams='nographic', image_fstype='wic') as qemu:
+                     runqemuparams='%s nographic' % (runqemu_params), image_fstype='wic') as qemu:
             # Check that we have ONLY two /dev/sda* partitions (/boot and /)
             cmd = "grep sda. /proc/partitions | wc -l"
             status, output = qemu.run_serial(cmd)
@@ -1446,8 +1450,8 @@ class Wic2(WicTestCase):
             os.rename(image_path, image_path + '.bak')
             os.rename(new_image_path, image_path)
 
-            # Check if it boots in qemu
-            with runqemu('core-image-minimal', ssh=False, runqemuparams='nographic') as qemu:
+            runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-minimal') or ""
+            with runqemu('core-image-minimal', ssh=False, runqemuparams='%s nographic' % (runqemu_params)) as qemu:
                 cmd = "ls /etc/"
                 status, output = qemu.run_serial('true')
                 self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
-- 
2.34.1

[PATCH v8 8/8] oeqa selftest wic.py: support UKIs via uki.bbclass

[Mikko Rapeli]

Use label to detect rootfs since UKI with kernel command
line is generated before rootfs is generated by wic.

Adapt wic tests to build and boot uki.bbclass generated
UKIs.

Keeping one UKI test in wic.py and rest of the UKI features
are tested with dedicated uki.py test. Add plain non-UKI
systemd-boot tests to wic suite for aarch64 and x86.

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta-selftest/wic/test_efi_plugin.wks         |   5 +-
 .../test_efi_plugin_plain_systemd-boot.wks    |   5 +
 meta/lib/oeqa/selftest/cases/wic.py           | 115 ++++++++++++++++--
 3 files changed, 110 insertions(+), 15 deletions(-)
 create mode 100644 meta-selftest/wic/test_efi_plugin_plain_systemd-boot.wks

diff --git a/meta-selftest/wic/test_efi_plugin.wks b/meta-selftest/wic/test_efi_plugin.wks
index 1603d6c4bb..e876a4be0e 100644
--- a/meta-selftest/wic/test_efi_plugin.wks
+++ b/meta-selftest/wic/test_efi_plugin.wks
@@ -1,6 +1,5 @@
 # short-description: This file is used in oe-selftest wic module to test efi plugin
-
-part /boot --source bootimg-efi --sourceparams="loader=systemd-boot,create-unified-kernel-image=true,initrd=${INITRAMFS_IMAGE}-${MACHINE}.${INITRAMFS_FSTYPES}" --active --align 1024 --use-uuid
-part / --source rootfs --fstype=ext4 --align 1024 --use-uuid
+part /boot --source bootimg-efi --sourceparams="loader=systemd-boot" --active --align 1024 --use-uuid
+part / --source rootfs --fstype=ext4 --align 1024 --use-uuid --label root
 
 bootloader  --timeout=0 --append="console=ttyS0,115200n8"
diff --git a/meta-selftest/wic/test_efi_plugin_plain_systemd-boot.wks b/meta-selftest/wic/test_efi_plugin_plain_systemd-boot.wks
new file mode 100644
index 0000000000..2745c19e85
--- /dev/null
+++ b/meta-selftest/wic/test_efi_plugin_plain_systemd-boot.wks
@@ -0,0 +1,5 @@
+# short-description: This file is used in oe-selftest wic module to test efi plugin
+part /boot --source bootimg-efi --sourceparams="loader=systemd-boot,initrd=${INITRAMFS_IMAGE}-${MACHINE}.${INITRAMFS_FSTYPES}" --active --align 1024 --use-uuid
+part / --source rootfs --fstype=ext4 --align 1024 --use-uuid
+
+bootloader  --timeout=0 --append="console=ttyS0,115200n8"
diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index f2a46c965b..e5caef74c4 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1292,24 +1292,46 @@ class Wic2(WicTestCase):
     @skipIfNotArch(['i586', 'i686', 'x86_64'])
     @OETestTag("runqemu")
     def test_efi_plugin_unified_kernel_image_qemu(self):
-        """Test efi plugin's Unified Kernel Image feature in qemu"""
-        config = 'IMAGE_FSTYPES = "wic"\n'\
-                 'INITRAMFS_IMAGE = "core-image-minimal-initramfs"\n'\
-                 'WKS_FILE = "test_efi_plugin.wks"\n'\
-                 'MACHINE_FEATURES:append = " efi"\n'
+        """Test Unified Kernel Image feature in qemu without systemd in initramfs or rootfs"""
+        config = """
+# efi firmware must load systemd-boot, not grub
+EFI_PROVIDER = "systemd-boot"
+
+# image format must be wic, needs esp partition for firmware etc
+IMAGE_FSTYPES:pn-core-image-base:append = " wic"
+WKS_FILE = "test_efi_plugin.wks"
+
+# efi, uki and systemd features must be enabled
+MACHINE_FEATURES:append = " efi"
+DISTRO_FEATURES_NATIVE:append = " systemd"
+IMAGE_CLASSES:append:pn-core-image-base = " uki"
+
+# uki embeds also an initrd, no systemd or udev
+INITRAMFS_IMAGE = "core-image-initramfs-boot"
+
+# runqemu must not load kernel separately, it's in the uki
+QB_KERNEL_ROOT = ""
+QB_DEFAULT_KERNEL = "none"
+
+# boot command line provided via uki, not via bootloader
+UKI_CMDLINE = "rootwait root=LABEL=root console=${KERNEL_CONSOLE}"
+
+"""
         self.append_config(config)
-        bitbake('core-image-minimal core-image-minimal-initramfs ovmf')
+        bitbake('core-image-base ovmf')
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-base') or ""
+        uki_filename = get_bb_var('UKI_FILENAME', 'core-image-base')
         self.remove_config(config)
 
-        with runqemu('core-image-minimal', ssh=False,
-                     runqemuparams='nographic ovmf', image_fstype='wic') as qemu:
-            # Check that /boot has EFI bootx64.efi (required for EFI)
-            cmd = "ls /boot/EFI/BOOT/bootx64.efi | wc -l"
+        with runqemu('core-image-base', ssh=False,
+                     runqemuparams='%s nographic ovmf' % (runqemu_params), image_fstype='wic') as qemu:
+            # Check that /boot has EFI boot*.efi (required for EFI)
+            cmd = "ls /boot/EFI/BOOT/boot*.efi | wc -l"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
             self.assertEqual(output, '1')
-            # Check that /boot has EFI/Linux/linux.efi (required for Unified Kernel Images auto detection)
-            cmd = "ls /boot/EFI/Linux/linux.efi | wc -l"
+            # Check that /boot has EFI/Linux/${UKI_FILENAME} (required for Unified Kernel Images auto detection)
+            cmd = "ls /boot/EFI/Linux/%s | wc -l" % (uki_filename)
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
             self.assertEqual(output, '1')
@@ -1319,6 +1341,75 @@ class Wic2(WicTestCase):
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
             self.assertEqual(output, '0')
 
+    @skipIfNotArch(['aarch64'])
+    @OETestTag("runqemu")
+    def test_efi_plugin_plain_systemd_boot_qemu_aarch64(self):
+        """Test plain systemd-boot in qemu with systemd"""
+        config = """
+INIT_MANAGER = "systemd"
+EFI_PROVIDER = "systemd-boot"
+
+# image format must be wic, needs esp partition for firmware etc
+IMAGE_FSTYPES:pn-core-image-base:append = " wic"
+WKS_FILE = "test_efi_plugin_plain_systemd-boot.wks"
+
+INITRAMFS_IMAGE = "core-image-initramfs-boot"
+
+# to configure runqemu
+IMAGE_CLASSES += "qemuboot"
+# u-boot efi firmware
+QB_DEFAULT_BIOS = "u-boot.bin"
+# need to use virtio, scsi not supported by u-boot by default
+QB_DRIVE_TYPE = "/dev/vd"
+"""
+        self.append_config(config)
+        bitbake('core-image-base u-boot')
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-base') or ""
+        self.remove_config(config)
+
+        with runqemu('core-image-base', ssh=False,
+                     runqemuparams='%s nographic' % (runqemu_params), image_fstype='wic') as qemu:
+            # Check that /boot has EFI boot*.efi (required for EFI)
+            cmd = "ls /boot/EFI/BOOT/boot*.efi | wc -l"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+            self.assertEqual(output, '1')
+            # Check that boot.conf exists
+            cmd = "cat /boot/loader/entries/boot.conf"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
+    @skipIfNotArch(['i586', 'i686', 'x86_64'])
+    @OETestTag("runqemu")
+    def test_efi_plugin_plain_systemd_boot_qemu_x86(self):
+        """Test plain systemd-boot to systemd in qemu"""
+        config = """
+INIT_MANAGER = "systemd"
+EFI_PROVIDER = "systemd-boot"
+
+# image format must be wic, needs esp partition for firmware etc
+IMAGE_FSTYPES:pn-core-image-base:append = " wic"
+WKS_FILE = "test_efi_plugin_plain_systemd-boot.wks"
+
+INITRAMFS_IMAGE = "core-image-initramfs-boot"
+"""
+        self.append_config(config)
+        bitbake('core-image-base ovmf')
+        runqemu_params = get_bb_var('TEST_RUNQEMUPARAMS', 'core-image-base') or ""
+        self.remove_config(config)
+
+        with runqemu('core-image-base', ssh=False,
+                     runqemuparams='%s nographic ovmf' % (runqemu_params), image_fstype='wic') as qemu:
+            # Check that /boot has EFI boot*.efi (required for EFI)
+            cmd = "ls /boot/EFI/BOOT/boot*.efi | wc -l"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+            self.assertEqual(output, '1')
+            # Check that boot.conf exists
+            cmd = "cat /boot/loader/entries/boot.conf"
+            status, output = qemu.run_serial(cmd)
+            self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
+
     def test_fs_types(self):
         """Test filesystem types for empty and not empty partitions"""
         img = 'core-image-minimal'
-- 
2.34.1

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Richard Purdie]

On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> These changes enable building systemd uki images which combine
> kernel, kernel command line, initrd and possibly signatures to
> a single UEFI binary. This binary can be booted with UEFI firmware
> and systemd-boot. No grub is needed and UEFI firmware and/or
> systemd-boot provide possibilities for boot menus.
> The uki binary can also be signed for UEFI secure boot
> so the secure boot extends from firmware to kernel and initrd.
> Binding secure boot to full userspace is then easier since for example
> kernel command line and initrd contain the support needed to mount
> encrypted dm-verity etc partitions, and/or create partitions on demand
> with systemd-repart using device specific TPM devices for encryption.
> 
> Tested on qemuarm64-secureboot machine from meta-arm with changes to
> support secure boot. Slightly different configuration tested on
> multiple arm64 System Ready boards with UEFI firmware, real and firmware
> based TPM devices. Tested with ovmf firmware on x86_64 with selftests but
> without secure boot which seems to be harder to setup in ovmf.
> 
> Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and
> wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests
> on a build machine with zfs filesystem. Will investigate this further.
> The issue seems to be in mkfs.ext4 producing broken filesystem, and partially
> in the tests which don't run the correct rootfs file (.ext4 vs .wic).
> Will debug this further and it is IMO unrelated to these changes since
> they reproduce on pure master branch without this series.
> 
> v8: fixed comments from Ross Burton: debug print from warning to debug,
>     dropped duplicate DISTRO_FEATURE setting for systemd in tests,
>     removed aarch64 comment from tests which are currently x86 only.
>     Fixed the new aarch64 wic selftest to run on both genericarm64
>     and qemuarm64 by adding bios, virtio disk driver etc settings
>     for runqemu (already set in genericarm64 but missing from qemuarm64).
> 
> v7: add missing "ovmf" to runqemu argument to
>     test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang
> 
> v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on
>     genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added
>     selftest to cover this wks usage on x86 and aarch64
> 
> v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release
>     dependency for all archs" since systemd-boot does not support all
>     architectures
> 
> v4: handle missing runqemu variable from build config, add
> python3-pefile to fast ptest list
> 
> v3: rebased, fixed and added more sefltests, removed wic plugin side uki
> support
> 
> v2: https://lists.openembedded.org/g/openembedded-core/message/204090
> 
> Michelle Lin (1):
>   uki.bbclass: add class for building Unified Kernel Images (UKI)
> 
> Mikko Rapeli (7):
>   wic bootimg-efi.py: keep timestamps and add debug prints
>   wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass
>   oeqa selftest uki.py: add tests for uki.bbclass
>   oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu
>   oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi
>     test
>   oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu
>   oeqa selftest wic.py: support UKIs via uki.bbclass
> 

I'm still seeing failures in CI:

https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio

which is despite setting:

https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6

QEMU_USE_KVM = "False"

into the test to ensure it doesn't have the issue the barebox testing
was seeing.

I've sent a patch to try and clean up the lock error.

There is also this:

https://valkyrie.yoctoproject.org//#/builders/76/builds/235
https://valkyrie.yoctoproject.org//#/builders/48/builds/181
https://valkyrie.yoctoproject.org//#/builders/54/builds/230

which is due to the binaries being run "in tree" within the edk2 build
as well as from the sysroot. This generates two sets of pyc files which
then conflict (or not) depending on which host the build ran on and
which pyc files are in sstate.

We're going to have to get this fixed before it can merge, probably by
deleting the pyc files at install unless we can find anything more
elegant.

Cheers,

Richard

[PATCH] ovmf-native: remove .pyc files from install

[Mikko Rapeli]

They break builds which share sstate files on different
machines and paths:

ERROR: ovmf-edk2-stable202408-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:extend_recipe_sysroot(d)
     0003:
File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 624, function: extend_recipe_sysroot
     0620:
     0621:    # Handle deferred binfiles
     0622:    for l in binfiles:
     0623:        (targetdir, dest) = binfiles[l]
 *** 0624:        staging_copyfile(l, targetdir, dest, postinsts, seendirs)
     0625:
     0626:    bb.note("Installed into sysroot: %s" % str(msg_adding))
     0627:    bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
     0628:
File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile
     0161:        os.symlink(linkto, dest)
     0162:        #bb.warn(c)
     0163:    else:
     0164:        try:
 *** 0165:            os.link(c, dest)
     0166:        except OSError as err:
     0167:            if err.errno == errno.EXDEV:
     0168:                bb.utils.copyfile(c, dest)
     0169:            else:
Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/sysroots-components/x86_64/ovmf-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc' -> '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/work/core2-64-poky-linux/ovmf/edk2-stable202408/recipe-sysroot-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc'

Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
---
 meta/recipes-core/ovmf/ovmf_git.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/ovmf/ovmf_git.bb b/meta/recipes-core/ovmf/ovmf_git.bb
index 202182397d..e761736453 100644
--- a/meta/recipes-core/ovmf/ovmf_git.bb
+++ b/meta/recipes-core/ovmf/ovmf_git.bb
@@ -223,6 +223,7 @@ do_compile:class-target() {
 
 do_install:class-native() {
     install -d ${D}/${bindir}/edk2_basetools
+    find ${S}/BaseTools -name \*.pyc -exec rm -rf \{\} \;
     cp -r ${S}/BaseTools ${D}/${bindir}/${EDK_TOOLS_DIR}
 }
 
-- 
2.34.1

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

Hi,

On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote:
> On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > These changes enable building systemd uki images which combine
> > kernel, kernel command line, initrd and possibly signatures to
> > a single UEFI binary. This binary can be booted with UEFI firmware
> > and systemd-boot. No grub is needed and UEFI firmware and/or
> > systemd-boot provide possibilities for boot menus.
> > The uki binary can also be signed for UEFI secure boot
> > so the secure boot extends from firmware to kernel and initrd.
> > Binding secure boot to full userspace is then easier since for example
> > kernel command line and initrd contain the support needed to mount
> > encrypted dm-verity etc partitions, and/or create partitions on demand
> > with systemd-repart using device specific TPM devices for encryption.
> > 
> > Tested on qemuarm64-secureboot machine from meta-arm with changes to
> > support secure boot. Slightly different configuration tested on
> > multiple arm64 System Ready boards with UEFI firmware, real and firmware
> > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but
> > without secure boot which seems to be harder to setup in ovmf.
> > 
> > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and
> > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests
> > on a build machine with zfs filesystem. Will investigate this further.
> > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially
> > in the tests which don't run the correct rootfs file (.ext4 vs .wic).
> > Will debug this further and it is IMO unrelated to these changes since
> > they reproduce on pure master branch without this series.
> > 
> > v8: fixed comments from Ross Burton: debug print from warning to debug,
> > ��� dropped duplicate DISTRO_FEATURE setting for systemd in tests,
> > ��� removed aarch64 comment from tests which are currently x86 only.
> > ��� Fixed the new aarch64 wic selftest to run on both genericarm64
> > ��� and qemuarm64 by adding bios, virtio disk driver etc settings
> > ��� for runqemu (already set in genericarm64 but missing from qemuarm64).
> > 
> > v7: add missing "ovmf" to runqemu argument to
> > ��� test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang
> > 
> > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on
> > ��� genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added
> > ��� selftest to cover this wks usage on x86 and aarch64
> > 
> > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release
> > ��� dependency for all archs" since systemd-boot does not support all
> > ��� architectures
> > 
> > v4: handle missing runqemu variable from build config, add
> > python3-pefile to fast ptest list
> > 
> > v3: rebased, fixed and added more sefltests, removed wic plugin side uki
> > support
> > 
> > v2: https://lists.openembedded.org/g/openembedded-core/message/204090
> > 
> > Michelle Lin (1):
> > � uki.bbclass: add class for building Unified Kernel Images (UKI)
> > 
> > Mikko Rapeli (7):
> > � wic bootimg-efi.py: keep timestamps and add debug prints
> > � wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass
> > � oeqa selftest uki.py: add tests for uki.bbclass
> > � oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu
> > � oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi
> > ��� test
> > � oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu
> > � oeqa selftest wic.py: support UKIs via uki.bbclass
> > 
> 
> I'm still seeing failures in CI:
> 
> https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio
> 
> which is despite setting:
> 
> https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6
> 
> QEMU_USE_KVM = "False"
> 
> into the test to ensure it doesn't have the issue the barebox testing
> was seeing.
> 
> I've sent a patch to try and clean up the lock error.
> 
> There is also this:
> 
> https://valkyrie.yoctoproject.org//#/builders/76/builds/235
> https://valkyrie.yoctoproject.org//#/builders/48/builds/181
> https://valkyrie.yoctoproject.org//#/builders/54/builds/230
> 
> which is due to the binaries being run "in tree" within the edk2 build
> as well as from the sysroot. This generates two sets of pyc files which
> then conflict (or not) depending on which host the build ran on and
> which pyc files are in sstate.
> 
> We're going to have to get this fixed before it can merge, probably by
> deleting the pyc files at install unless we can find anything more
> elegant.

Sent an ovmf-native patch separately for this.

Cheers,

-Mikko

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Alexander Kanavin]

This doesn't quite make sense. Why is the problem specific to .pyc
files and not other files ovmf installs?

Alex

On Mon, 14 Oct 2024 at 12:28, Mikko Rapeli via lists.openembedded.org
<mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
>
> They break builds which share sstate files on different
> machines and paths:
>
> ERROR: ovmf-edk2-stable202408-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:extend_recipe_sysroot(d)
>      0003:
> File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 624, function: extend_recipe_sysroot
>      0620:
>      0621:    # Handle deferred binfiles
>      0622:    for l in binfiles:
>      0623:        (targetdir, dest) = binfiles[l]
>  *** 0624:        staging_copyfile(l, targetdir, dest, postinsts, seendirs)
>      0625:
>      0626:    bb.note("Installed into sysroot: %s" % str(msg_adding))
>      0627:    bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists))
>      0628:
> File: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile
>      0161:        os.symlink(linkto, dest)
>      0162:        #bb.warn(c)
>      0163:    else:
>      0164:        try:
>  *** 0165:            os.link(c, dest)
>      0166:        except OSError as err:
>      0167:            if err.errno == errno.EXDEV:
>      0168:                bb.utils.copyfile(c, dest)
>      0169:            else:
> Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/sysroots-components/x86_64/ovmf-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc' -> '/srv/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-667282/tmp/work/core2-64-poky-linux/ovmf/edk2-stable202408/recipe-sysroot-native/usr/bin/edk2_basetools/BaseTools/Source/Python/AutoGen/__pycache__/WorkspaceAutoGen.cpython-312.pyc'
>
> Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> ---
>  meta/recipes-core/ovmf/ovmf_git.bb | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/recipes-core/ovmf/ovmf_git.bb b/meta/recipes-core/ovmf/ovmf_git.bb
> index 202182397d..e761736453 100644
> --- a/meta/recipes-core/ovmf/ovmf_git.bb
> +++ b/meta/recipes-core/ovmf/ovmf_git.bb
> @@ -223,6 +223,7 @@ do_compile:class-target() {
>
>  do_install:class-native() {
>      install -d ${D}/${bindir}/edk2_basetools
> +    find ${S}/BaseTools -name \*.pyc -exec rm -rf \{\} \;
>      cp -r ${S}/BaseTools ${D}/${bindir}/${EDK_TOOLS_DIR}
>  }
>
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#205740): https://lists.openembedded.org/g/openembedded-core/message/205740
> Mute This Topic: https://lists.openembedded.org/mt/108998817/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Mikko Rapeli]

Hi,

On Mon, Oct 14, 2024 at 01:05:51PM +0200, Alexander Kanavin wrote:
> This doesn't quite make sense. Why is the problem specific to .pyc
> files and not other files ovmf installs?

I don't know. At this point I'd rather remove all tests in uki
support which are causing so many issues.

I tried to dig deeper into ovmf native but it looks like a mess.
meta-arm edk2 doesn't compile anything for native but just installs
the python scripts and gets away with it. Same did not work for ovmf.

Cheers,

-Mikko

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Alexander Kanavin]

On Mon, 14 Oct 2024 at 13:21, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> I don't know. At this point I'd rather remove all tests in uki
> support which are causing so many issues.
>
> I tried to dig deeper into ovmf native but it looks like a mess.
> meta-arm edk2 doesn't compile anything for native but just installs
> the python scripts and gets away with it. Same did not work for ovmf.

The tests are important.

Were you able to reproduce the issue locally? Ovmf is an awful
component and a headache to work with, but I could take a look if I
can get the same error. I'm just worried that deleting .pyc will only
trigger the same error for the essential installed items.

Alex

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Mikko Rapeli]

Hi,

On Mon, Oct 14, 2024 at 01:28:07PM +0200, Alexander Kanavin wrote:
> On Mon, 14 Oct 2024 at 13:21, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > I don't know. At this point I'd rather remove all tests in uki
> > support which are causing so many issues.
> >
> > I tried to dig deeper into ovmf native but it looks like a mess.
> > meta-arm edk2 doesn't compile anything for native but just installs
> > the python scripts and gets away with it. Same did not work for ovmf.
> 
> The tests are important.
>
> Were you able to reproduce the issue locally? Ovmf is an awful
> component and a headache to work with, but I could take a look if I
> can get the same error. I'm just worried that deleting .pyc will only
> trigger the same error for the essential installed items.

No, I'm not able to reproduce the problem. Maybe I'd need to share the sstate
to another machine or user to trigger this. Just changing build paths
doesn't trigger the problem on my build machine.

Alternatively I can remove all x86/ovmf using tests and make things
work with u-boot firmware and aarch64, but then I need to test both
qemuarm64 and genericarm64 since details differ between them.

Cheers,

-Mikko

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Alexander Kanavin]

On Mon, 14 Oct 2024 at 13:39, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> No, I'm not able to reproduce the problem. Maybe I'd need to share the sstate
> to another machine or user to trigger this. Just changing build paths
> doesn't trigger the problem on my build machine.

Can I see the link to the autobuilder failure where this was observed?

Alex

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Richard Purdie]

On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via
lists.openembedded.org wrote:
> This doesn't quite make sense. Why is the problem specific to .pyc
> files and not other files ovmf installs?

If it helps, I do roughly understand what is happening here and I
believe this is the right fix.

Cheers,

Richard

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

Hi,

On Mon, Oct 14, 2024 at 01:42:46PM +0200, Alexander Kanavin wrote:
> On Mon, 14 Oct 2024 at 13:39, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > No, I'm not able to reproduce the problem. Maybe I'd need to share the sstate
> > to another machine or user to trigger this. Just changing build paths
> > doesn't trigger the problem on my build machine.
> 
> Can I see the link to the autobuilder failure where this was observed?

Yes:

On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote:
> I'm still seeing failures in CI:
> 
> https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio
> 
> which is despite setting:
> 
> https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6
> 
> QEMU_USE_KVM = "False"
> 
> into the test to ensure it doesn't have the issue the barebox testing
> was seeing.
> 
> I've sent a patch to try and clean up the lock error.
> 
> There is also this:
> 
> https://valkyrie.yoctoproject.org//#/builders/76/builds/235
> https://valkyrie.yoctoproject.org//#/builders/48/builds/181
> https://valkyrie.yoctoproject.org//#/builders/54/builds/230
> 
> which is due to the binaries being run "in tree" within the edk2 build
> as well as from the sysroot. This generates two sets of pyc files which
> then conflict (or not) depending on which host the build ran on and
> which pyc files are in sstate.
> 
> We're going to have to get this fixed before it can merge, probably by
> deleting the pyc files at install unless we can find anything more
> elegant.

After few experiments failed, I went with this.

Cheers,

-Mikko

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Ross Burton]

On 14 Oct 2024, at 13:01, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> 
> On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via
> lists.openembedded.org wrote:
>> This doesn't quite make sense. Why is the problem specific to .pyc
>> files and not other files ovmf installs?
> 
> If it helps, I do roughly understand what is happening here and I
> believe this is the right fix.

So my understanding is that this is a general problem: if a native recipe ships a .py but not .pyc then when that script is used python will write a .pyc _into the sysroot_. If we then upgrade that recipe and it now ships a .pyc then you get this error, as the sysroot construction call wants to create a file that already exists.

I think for scripts that are ran with python3-native then we should be shipping the .pyc files, even in native recipes, because then they’re pregenerated and will be used by python.  The fun is when the script is ran by the host python, which is of indeterminate version.

I wonder if python has a way to control where the files get written, so we could put freshly generated files into a separate directory that isn’t part of the sysroot?  We could also make the sysroot read-only outside of the sysroot manipulation, which would force python to do something different too (though I believe this is just “don’t write pyc?).

Note that this problem impacts plenty of other recipes that for historical reasons can't ship bytecode into the sysroot (see eg python3-setuptools), so I do think we need a general solution and not just a creeping and irreversible trend of deleting .pyc from the sysroot.

Ross

Re: [OE-core] [PATCH] ovmf-native: remove .pyc files from install

[Richard Purdie]

On Mon, 2024-10-14 at 12:34 +0000, Ross Burton via
lists.openembedded.org wrote:
> On 14 Oct 2024, at 13:01, Richard Purdie via lists.openembedded.org
> <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> > 
> > On Mon, 2024-10-14 at 13:05 +0200, Alexander Kanavin via
> > lists.openembedded.org wrote:
> > > This doesn't quite make sense. Why is the problem specific to
> > > .pyc
> > > files and not other files ovmf installs?
> > 
> > If it helps, I do roughly understand what is happening here and I
> > believe this is the right fix.
> 
> So my understanding is that this is a general problem: if a native
> recipe ships a .py but not .pyc then when that script is used python
> will write a .pyc _into the sysroot_. If we then upgrade that recipe
> and it now ships a .pyc then you get this error, as the sysroot
> construction call wants to create a file that already exists.
> 
> I think for scripts that are ran with python3-native then we should
> be shipping the .pyc files, even in native recipes, because then
> they’re pregenerated and will be used by python.  The fun is when the
> script is ran by the host python, which is of indeterminate version.

For fun, there are things which could be called by either the host
python or python3-native from the sysroot depending on whether that was
added to PATH for other reasons. That complicates things somewhat.

> I wonder if python has a way to control where the files get written,
> so we could put freshly generated files into a separate directory
> that isn’t part of the sysroot?  We could also make the sysroot read-
> only outside of the sysroot manipulation, which would force python to
> do something different too (though I believe this is just “don’t
> write pyc?).
> 
> Note that this problem impacts plenty of other recipes that for
> historical reasons can't ship bytecode into the sysroot (see eg
> python3-setuptools), so I do think we need a general solution and not
> just a creeping and irreversible trend of deleting .pyc from the
> sysroot.

I'm torn. It is a complex problem causing issues for patches which are
otherwise ok and requires a great deal of thought to fix differently.

The performance impact of python generating those files at first run
probably is negligible, compared even to the overhead of extracting
files from sstate.

Is this an issue we want to focus effort on? Are there gains here to be
had?

Cheers,

Richard

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

Hi,

On Mon, Oct 14, 2024 at 01:30:56PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> Hi,
> 
> On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote:
> > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > > These changes enable building systemd uki images which combine
> > > kernel, kernel command line, initrd and possibly signatures to
> > > a single UEFI binary. This binary can be booted with UEFI firmware
> > > and systemd-boot. No grub is needed and UEFI firmware and/or
> > > systemd-boot provide possibilities for boot menus.
> > > The uki binary can also be signed for UEFI secure boot
> > > so the secure boot extends from firmware to kernel and initrd.
> > > Binding secure boot to full userspace is then easier since for example
> > > kernel command line and initrd contain the support needed to mount
> > > encrypted dm-verity etc partitions, and/or create partitions on demand
> > > with systemd-repart using device specific TPM devices for encryption.
> > > 
> > > Tested on qemuarm64-secureboot machine from meta-arm with changes to
> > > support secure boot. Slightly different configuration tested on
> > > multiple arm64 System Ready boards with UEFI firmware, real and firmware
> > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but
> > > without secure boot which seems to be harder to setup in ovmf.
> > > 
> > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and
> > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests
> > > on a build machine with zfs filesystem. Will investigate this further.
> > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially
> > > in the tests which don't run the correct rootfs file (.ext4 vs .wic).
> > > Will debug this further and it is IMO unrelated to these changes since
> > > they reproduce on pure master branch without this series.
> > > 
> > > v8: fixed comments from Ross Burton: debug print from warning to debug,
> > > ��� dropped duplicate DISTRO_FEATURE setting for systemd in tests,
> > > ��� removed aarch64 comment from tests which are currently x86 only.
> > > ��� Fixed the new aarch64 wic selftest to run on both genericarm64
> > > ��� and qemuarm64 by adding bios, virtio disk driver etc settings
> > > ��� for runqemu (already set in genericarm64 but missing from qemuarm64).
> > > 
> > > v7: add missing "ovmf" to runqemu argument to
> > > ��� test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang
> > > 
> > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on
> > > ��� genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added
> > > ��� selftest to cover this wks usage on x86 and aarch64
> > > 
> > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release
> > > ��� dependency for all archs" since systemd-boot does not support all
> > > ��� architectures
> > > 
> > > v4: handle missing runqemu variable from build config, add
> > > python3-pefile to fast ptest list
> > > 
> > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki
> > > support
> > > 
> > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090
> > > 
> > > Michelle Lin (1):
> > > � uki.bbclass: add class for building Unified Kernel Images (UKI)
> > > 
> > > Mikko Rapeli (7):
> > > � wic bootimg-efi.py: keep timestamps and add debug prints
> > > � wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass
> > > � oeqa selftest uki.py: add tests for uki.bbclass
> > > � oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu
> > > � oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi
> > > ��� test
> > > � oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu
> > > � oeqa selftest wic.py: support UKIs via uki.bbclass
> > > 
> > 
> > I'm still seeing failures in CI:
> > 
> > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio
> > 
> > which is despite setting:
> > 
> > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6
> > 
> > QEMU_USE_KVM = "False"
> > 
> > into the test to ensure it doesn't have the issue the barebox testing
> > was seeing.
> > 
> > I've sent a patch to try and clean up the lock error.
> > 
> > There is also this:
> > 
> > https://valkyrie.yoctoproject.org//#/builders/76/builds/235
> > https://valkyrie.yoctoproject.org//#/builders/48/builds/181
> > https://valkyrie.yoctoproject.org//#/builders/54/builds/230
> > 
> > which is due to the binaries being run "in tree" within the edk2 build
> > as well as from the sysroot. This generates two sets of pyc files which
> > then conflict (or not) depending on which host the build ran on and
> > which pyc files are in sstate.
> > 
> > We're going to have to get this fixed before it can merge, probably by
> > deleting the pyc files at install unless we can find anything more
> > elegant.
> 
> Sent an ovmf-native patch separately for this.

With ovmf-native change applied to master, can this series be tried again?

Or are some changes needed?

Cheers,

-Mikko

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Richard Purdie]

On Tue, 2024-10-15 at 09:44 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Mon, Oct 14, 2024 at 01:30:56PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > Hi,
> > 
> > On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote:
> > > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > > > These changes enable building systemd uki images which combine
> > > > kernel, kernel command line, initrd and possibly signatures to
> > > > a single UEFI binary. This binary can be booted with UEFI firmware
> > > > and systemd-boot. No grub is needed and UEFI firmware and/or
> > > > systemd-boot provide possibilities for boot menus.
> > > > The uki binary can also be signed for UEFI secure boot
> > > > so the secure boot extends from firmware to kernel and initrd.
> > > > Binding secure boot to full userspace is then easier since for example
> > > > kernel command line and initrd contain the support needed to mount
> > > > encrypted dm-verity etc partitions, and/or create partitions on demand
> > > > with systemd-repart using device specific TPM devices for encryption.
> > > > 
> > > > Tested on qemuarm64-secureboot machine from meta-arm with changes to
> > > > support secure boot. Slightly different configuration tested on
> > > > multiple arm64 System Ready boards with UEFI firmware, real and firmware
> > > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but
> > > > without secure boot which seems to be harder to setup in ovmf.
> > > > 
> > > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and
> > > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests
> > > > on a build machine with zfs filesystem. Will investigate this further.
> > > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially
> > > > in the tests which don't run the correct rootfs file (.ext4 vs .wic).
> > > > Will debug this further and it is IMO unrelated to these changes since
> > > > they reproduce on pure master branch without this series.
> > > > 
> > > > v8: fixed comments from Ross Burton: debug print from warning to debug,
> > > >     dropped duplicate DISTRO_FEATURE setting for systemd in tests,
> > > >     removed aarch64 comment from tests which are currently x86 only.
> > > >     Fixed the new aarch64 wic selftest to run on both genericarm64
> > > >     and qemuarm64 by adding bios, virtio disk driver etc settings
> > > >     for runqemu (already set in genericarm64 but missing from qemuarm64).
> > > > 
> > > > v7: add missing "ovmf" to runqemu argument to
> > > >     test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang
> > > > 
> > > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on
> > > >     genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added
> > > >     selftest to cover this wks usage on x86 and aarch64
> > > > 
> > > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release
> > > >     dependency for all archs" since systemd-boot does not support all
> > > >     architectures
> > > > 
> > > > v4: handle missing runqemu variable from build config, add
> > > > python3-pefile to fast ptest list
> > > > 
> > > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki
> > > > support
> > > > 
> > > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090
> > > > 
> > > > Michelle Lin (1):
> > > >   uki.bbclass: add class for building Unified Kernel Images (UKI)
> > > > 
> > > > Mikko Rapeli (7):
> > > >   wic bootimg-efi.py: keep timestamps and add debug prints
> > > >   wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass
> > > >   oeqa selftest uki.py: add tests for uki.bbclass
> > > >   oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu
> > > >   oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi
> > > >     test
> > > >   oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu
> > > >   oeqa selftest wic.py: support UKIs via uki.bbclass
> > > > 
> > > 
> > > I'm still seeing failures in CI:
> > > 
> > > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio
> > > 
> > > which is despite setting:
> > > 
> > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6
> > > 
> > > QEMU_USE_KVM = "False"
> > > 
> > > into the test to ensure it doesn't have the issue the barebox testing
> > > was seeing.
> > > 
> > > I've sent a patch to try and clean up the lock error.
> > > 
> > > There is also this:
> > > 
> > > https://valkyrie.yoctoproject.org//#/builders/76/builds/235
> > > https://valkyrie.yoctoproject.org//#/builders/48/builds/181
> > > https://valkyrie.yoctoproject.org//#/builders/54/builds/230
> > > 
> > > which is due to the binaries being run "in tree" within the edk2 build
> > > as well as from the sysroot. This generates two sets of pyc files which
> > > then conflict (or not) depending on which host the build ran on and
> > > which pyc files are in sstate.
> > > 
> > > We're going to have to get this fixed before it can merge, probably by
> > > deleting the pyc files at install unless we can find anything more
> > > elegant.
> > 
> > Sent an ovmf-native patch separately for this.
> 
> With ovmf-native change applied to master, can this series be tried again?
> 
> Or are some changes needed?

This handles the pyc issue which is useful as it caused widespread
build failures. The question of the first arm test failure remains :/.

I will run it through testing again though. I'm juggling far too many
failures at the moment, I can't tell what is "good" and what isn't very
easily.

Cheers,

Richard

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Richard Purdie]

On Tue, 2024-10-15 at 09:44 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Mon, Oct 14, 2024 at 01:30:56PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > Hi,
> > 
> > On Sun, Oct 13, 2024 at 08:43:15AM +0100, Richard Purdie wrote:
> > > On Fri, 2024-10-11 at 15:20 +0300, Mikko Rapeli via lists.openembedded.org wrote:
> > > > These changes enable building systemd uki images which combine
> > > > kernel, kernel command line, initrd and possibly signatures to
> > > > a single UEFI binary. This binary can be booted with UEFI firmware
> > > > and systemd-boot. No grub is needed and UEFI firmware and/or
> > > > systemd-boot provide possibilities for boot menus.
> > > > The uki binary can also be signed for UEFI secure boot
> > > > so the secure boot extends from firmware to kernel and initrd.
> > > > Binding secure boot to full userspace is then easier since for example
> > > > kernel command line and initrd contain the support needed to mount
> > > > encrypted dm-verity etc partitions, and/or create partitions on demand
> > > > with systemd-repart using device specific TPM devices for encryption.
> > > > 
> > > > Tested on qemuarm64-secureboot machine from meta-arm with changes to
> > > > support secure boot. Slightly different configuration tested on
> > > > multiple arm64 System Ready boards with UEFI firmware, real and firmware
> > > > based TPM devices. Tested with ovmf firmware on x86_64 with selftests but
> > > > without secure boot which seems to be harder to setup in ovmf.
> > > > 
> > > > Sadly I see two wic selftests, wic.Wic2.test_rawcopy_plugin_qemu and
> > > > wic.Wic2.test_expand_mbr_image, failing when executing all wic selftests
> > > > on a build machine with zfs filesystem. Will investigate this further.
> > > > The issue seems to be in mkfs.ext4 producing broken filesystem, and partially
> > > > in the tests which don't run the correct rootfs file (.ext4 vs .wic).
> > > > Will debug this further and it is IMO unrelated to these changes since
> > > > they reproduce on pure master branch without this series.
> > > > 
> > > > v8: fixed comments from Ross Burton: debug print from warning to debug,
> > > >     dropped duplicate DISTRO_FEATURE setting for systemd in tests,
> > > >     removed aarch64 comment from tests which are currently x86 only.
> > > >     Fixed the new aarch64 wic selftest to run on both genericarm64
> > > >     and qemuarm64 by adding bios, virtio disk driver etc settings
> > > >     for runqemu (already set in genericarm64 but missing from qemuarm64).
> > > > 
> > > > v7: add missing "ovmf" to runqemu argument to
> > > >     test_efi_plugin_plain_systemd_boot_qemu_x86 to fix boot hang
> > > > 
> > > > v6: fixes wic refactoring botch which broken non-uki systemd-boot usage on
> > > >     genericarm64 reported by Ross Burton <Ross.Burton@arm.com>, added
> > > >     selftest to cover this wks usage on x86 and aarch64
> > > > 
> > > > v5: drop patch "image_types_wic.bbclass: set systemd-boot and os-release
> > > >     dependency for all archs" since systemd-boot does not support all
> > > >     architectures
> > > > 
> > > > v4: handle missing runqemu variable from build config, add
> > > > python3-pefile to fast ptest list
> > > > 
> > > > v3: rebased, fixed and added more sefltests, removed wic plugin side uki
> > > > support
> > > > 
> > > > v2: https://lists.openembedded.org/g/openembedded-core/message/204090
> > > > 
> > > > Michelle Lin (1):
> > > >   uki.bbclass: add class for building Unified Kernel Images (UKI)
> > > > 
> > > > Mikko Rapeli (7):
> > > >   wic bootimg-efi.py: keep timestamps and add debug prints
> > > >   wic bootimg-efi.py: change UKI support from wic plugin to uki.bbclass
> > > >   oeqa selftest uki.py: add tests for uki.bbclass
> > > >   oeqa selftest efibootpartition.py: add TEST_RUNQEMUPARAMS to runqemu
> > > >   oeqa selftest efibootpartition.py: remove systemd-boot from grub-efi
> > > >     test
> > > >   oeqa selftest wic.py: add TEST_RUNQEMUPARAMS to runqemu
> > > >   oeqa selftest wic.py: support UKIs via uki.bbclass
> > > > 
> > > 
> > > I'm still seeing failures in CI:
> > > 
> > > https://valkyrie.yoctoproject.org//#/builders/23/builds/249/steps/14/logs/stdio
> > > 
> > > which is despite setting:
> > > 
> > > https://git.yoctoproject.org/poky/commit/?h=master-next&id=6211ad9210e82a5a8dd157c63752ad332c2f5de6
> > > 
> > > QEMU_USE_KVM = "False"
> > > 
> > > into the test to ensure it doesn't have the issue the barebox testing
> > > was seeing.
> > > 
> > > I've sent a patch to try and clean up the lock error.
> > > 
> > > There is also this:
> > > 
> > > https://valkyrie.yoctoproject.org//#/builders/76/builds/235
> > > https://valkyrie.yoctoproject.org//#/builders/48/builds/181
> > > https://valkyrie.yoctoproject.org//#/builders/54/builds/230
> > > 
> > > which is due to the binaries being run "in tree" within the edk2 build
> > > as well as from the sysroot. This generates two sets of pyc files which
> > > then conflict (or not) depending on which host the build ran on and
> > > which pyc files are in sstate.
> > > 
> > > We're going to have to get this fixed before it can merge, probably by
> > > deleting the pyc files at install unless we can find anything more
> > > elegant.
> > 
> > Sent an ovmf-native patch separately for this.
> 
> With ovmf-native change applied to master, can this series be tried again?
> 
> Or are some changes needed?

The arm test still fails:

https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio

The traceback is at least cleaner now I fixed the lock error. I did
test with KVM disabled.

Cheers,

Richard

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

Hi,

On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote:
> The arm test still fails:
> 
> https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio
> 
> The traceback is at least cleaner now I fixed the lock error. I did
> test with KVM disabled.

Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64
build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu 20.04?

I will remove the tests as it is clearly too hard to get these running.

It is a bit hard to figure out in which environments selftests are running.

Cheers,

-Mikko

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1573 bytes --]

Don’t remove the tests please. Without them things tend to regress. I
understand it’s frustrating, but we do have arm people here that had pushed
hard for testing in arm hosts, (without really thinking through how
contributors are supposed to deal with such failures) and I think it’s now
on them to help you out.

Alex

On Tue 15. Oct 2024 at 13.23, Mikko Rapeli via lists.openembedded.org
<mikko.rapeli=linaro.org@lists.openembedded.org> wrote:

> Hi,
>
> On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote:
> > The arm test still fails:
> >
> >
> https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio
> >
> > The traceback is at least cleaner now I fixed the lock error. I did
> > test with KVM disabled.
>
> Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64
> build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu
> 20.04?
>
> I will remove the tests as it is clearly too hard to get these running.
>
> It is a bit hard to figure out in which environments selftests are running.
>
> Cheers,
>
> -Mikko
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#205806):
> https://lists.openembedded.org/g/openembedded-core/message/205806
> Mute This Topic: https://lists.openembedded.org/mt/108947467/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2659 bytes --]

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1881 bytes --]

Alternatively just mark the test as specific to x86-64 host, there are
several self tests that do this.

Alex

On Tue 15. Oct 2024 at 13.32, Alexander Kanavin via lists.openembedded.org
<alex.kanavin=gmail.com@lists.openembedded.org> wrote:

> Don’t remove the tests please. Without them things tend to regress. I
> understand it’s frustrating, but we do have arm people here that had pushed
> hard for testing in arm hosts, (without really thinking through how
> contributors are supposed to deal with such failures) and I think it’s now
> on them to help you out.
>
> Alex
>
> On Tue 15. Oct 2024 at 13.23, Mikko Rapeli via lists.openembedded.org
> <mikko.rapeli=linaro.org@lists.openembedded.org> wrote:
>
>> Hi,
>>
>> On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote:
>> > The arm test still fails:
>> >
>> >
>> https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio
>> >
>> > The traceback is at least cleaner now I fixed the lock error. I did
>> > test with KVM disabled.
>>
>> Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64
>> build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu
>> 20.04?
>>
>> I will remove the tests as it is clearly too hard to get these running.
>>
>> It is a bit hard to figure out in which environments selftests are
>> running.
>>
>> Cheers,
>>
>> -Mikko
>>
>>
>>
>>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#205807):
> https://lists.openembedded.org/g/openembedded-core/message/205807
> Mute This Topic: https://lists.openembedded.org/mt/108947467/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3387 bytes --]

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Richard Purdie]

On Tue, 2024-10-15 at 14:23 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote:
> > The arm test still fails:
> > 
> > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio
> > 
> > The traceback is at least cleaner now I fixed the lock error. I did
> > test with KVM disabled.
> 
> Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64
> build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu
> 20.04?
> 
> I will remove the tests as it is clearly too hard to get these
> running.
> 
> It is a bit hard to figure out in which environments selftests are
> running.

Please don't remove them. The automated tests are the main reason the
project actually continues to function in any sane way.

For that reason I really do appreciate having them and would like to
see more of them.

Unfortunately I don't know enough about the subject area to help on
this. I have at least tried to improve the failure messages and run out
the one area I knew was potentially an issue (KVM on arm).

Cheers,

Richard

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

Hi,

On Tue, Oct 15, 2024 at 03:13:44PM +0100, Richard Purdie wrote:
> On Tue, 2024-10-15 at 14:23 +0300, Mikko Rapeli wrote:
> > Hi,
> > 
> > On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote:
> > > The arm test still fails:
> > > 
> > > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio
> > > 
> > > The traceback is at least cleaner now I fixed the lock error. I did
> > > test with KVM disabled.
> > 
> > Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64
> > build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu
> > 20.04?
> > 
> > I will remove the tests as it is clearly too hard to get these
> > running.
> > 
> > It is a bit hard to figure out in which environments selftests are
> > running.
> 
> Please don't remove them. The automated tests are the main reason the
> project actually continues to function in any sane way.
> 
> For that reason I really do appreciate having them and would like to
> see more of them.
> 
> Unfortunately I don't know enough about the subject area to help on
> this. I have at least tried to improve the failure messages and run out
> the one area I knew was potentially an issue (KVM on arm).

Did the test pass on x86_64 build host in your CI?

How about I run the wic.Wic2.test_efi_plugin_plain_systemd_boot_qemu_aarch64
test only on x86_64 BUILD_ARCH and aarch64 TARGET_ARCH for now?

A decorator like this could help:

--- a/meta/lib/oeqa/core/decorator/data.py
+++ b/meta/lib/oeqa/core/decorator/data.py
@@ -228,3 +228,15 @@ class skipIfNotArch(OETestDecorator):
         arch = self.case.td['HOST_ARCH']
         if arch not in self.archs:
              self.case.skipTest('Test skipped on %s' % arch)
+
+@registerDecorator
+class skipIfNotBuildArch(OETestDecorator):
+    """
+    Skip test if BUILD_ARCH is not present in the tuple specified.
+    """
+
+    attrs = ('archs',)
+    def setUpDecorator(self):
+        arch = self.case.td['BUILD_ARCH']
+        if arch not in self.archs:
+             self.case.skipTest('Test skipped on %s' % arch)

Cheers,

-Mikko

Re: [OE-core] [PATCH v8 0/8] systemd uki support

[Mikko Rapeli]

Hi,

On Tue, Oct 15, 2024 at 02:23:07PM +0300, Mikko Rapeli via lists.openembedded.org wrote:
> Hi,
> 
> On Tue, Oct 15, 2024 at 11:43:33AM +0100, Richard Purdie wrote:
> > The arm test still fails:
> > 
> > https://valkyrie.yoctoproject.org/#/builders/23/builds/267/steps/14/logs/stdio
> > 
> > The traceback is at least cleaner now I fixed the lock error. I did
> > test with KVM disabled.
> 
> Bugger. I had tested this on qemuarm64 and genericarm64 on an x86_64
> build host with Ubuntu 22.04. Is this an arm64 build host with Ubuntu 20.04?
> 
> I will remove the tests as it is clearly too hard to get these running.
> 
> It is a bit hard to figure out in which environments selftests are running.

I sent out a v9 which disables the test on non-x86_64 build hosts.
I hope that was the only failure seen from this series.

Cheers,

-Mikko