[PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Andrey Popov]

Since syslinux is only compatible with platforms that use x86-based
CPUs, this change allows creation of bootable ISO images for other
EFI-compatible platforms by replacing invocation of the isohybrid
tool for those platforms with a python script that creates MBR
partition table with a single entry that points to a bootable
EFI image placed inside the ISO image.

Signed-off-by: Andrey Popov <andrey.popov@yadro.com>
---
 meta/classes-recipe/image-live.bbclass | 129 ++++++++++++++++++++++---
 1 file changed, 115 insertions(+), 14 deletions(-)

diff --git a/meta/classes-recipe/image-live.bbclass b/meta/classes-recipe/image-live.bbclass
index 95dd44a..ac7fc1b 100644
--- a/meta/classes-recipe/image-live.bbclass
+++ b/meta/classes-recipe/image-live.bbclass
@@ -2,15 +2,15 @@
 #
 # SPDX-License-Identifier: MIT
 
-# Creates a bootable image using syslinux, your kernel and an optional
+# Creates a bootable image using syslinux (for x86), your kernel and an optional
 # initrd
 
 #
 # End result is two things:
 #
-# 1. A .hddimg file which is an msdos filesystem containing syslinux, a kernel,
-# an initrd and a rootfs image. These can be written to harddisks directly and
-# also booted on USB flash disks (write them there with dd).
+# 1. A .hddimg file which is an msdos filesystem containing syslinux (for x86),
+# a kernel, an initrd and a rootfs image. These can be written to harddisks
+# directly and also booted on USB flash disks (write them there with dd).
 #
 # 2. A CD .iso image
 
@@ -29,8 +29,10 @@ do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \
                         mtools-native:do_populate_sysroot \
                         cdrtools-native:do_populate_sysroot \
                         virtual/kernel:do_deploy \
-                        ${MLPREFIX}syslinux:do_populate_sysroot \
-                        syslinux-native:do_populate_sysroot \
+                        ${@d.getVar('MLPREFIX') + 'syslinux:do_populate_sysroot \
+                           syslinux-native:do_populate_sysroot' if __import__('re').match('i.86', d.getVar('TARGET_ARCH')) or \
+                                                                   __import__('re').match('x86_64', d.getVar('TARGET_ARCH')) else \
+                           'xorriso-native:do_populate_sysroot'} \
                         ${@'%s:do_image_%s' % (d.getVar('PN'), d.getVar('LIVE_ROOTFS_TYPE').replace('-', '_')) if d.getVar('ROOTFS') else ''} \
                         "
 
@@ -71,14 +73,92 @@ MKISOFS_OPTIONS = "-no-emul-boot -boot-load-size 4 -boot-info-table"
 BOOTIMG_VOLUME_ID   ?= "boot"
 BOOTIMG_EXTRA_SPACE ?= "512"
 
+def compute_chs(sector_z):
+    C = int(sector_z / (63 * 255))
+    H = int((sector_z % (63 * 255)) / 63)
+    # convert zero-based sector to CHS format
+    S = int(sector_z % 63) + 1
+    # munge accord to partition table format
+    S = (S & 0x3f) | (((C >> 8) & 0x3) << 6)
+    C = (C & 0xFF)
+    return (C, H, S)
+
+def mk_efi_part_table(iso, start, length):
+    from struct import pack
+
+    # Compute starting and ending CHS addresses for the partition entry.
+    (s_C, s_H, s_S) = compute_chs(start)
+    (e_C, e_H, e_S) = compute_chs(start + length - 1)
+
+    # Write the 66 byte partition table to bytes 0x1BE through 0x1FF in
+    # sector 0 of the .ISO.
+    #
+    # See the partition table format here:
+    # http://en.wikipedia.org/wiki/Master_boot_record#Sector_layout
+    f = open(iso, 'r+b')
+    f.seek(0x1BE)
+    f.write(pack("<8BLL48xH", 0x80, s_H, s_S, s_C,
+                 0xEF, e_H, e_S, e_C, start, length, 0xAA55))
+    f.close()
+
+def runtool(cmdln_or_args):
+    import subprocess
+
+    if isinstance(cmdln_or_args, list):
+        cmd = cmdln_or_args[0]
+        shell = False
+    else:
+        import shlex
+        cmd = shlex.split(cmdln_or_args)[0]
+        shell = True
+
+    sout = subprocess.PIPE
+    serr = subprocess.STDOUT
+
+    try:
+        process = subprocess.Popen(cmdln_or_args, stdout=sout,
+                                   stderr=serr, shell=shell)
+        sout, serr = process.communicate()
+        # combine stdout and stderr, filter None out and decode
+        out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
+    except OSError as err:
+        bb.fatal("Cannot run command %s: %s" % (cmd, err))
+
+    return process.returncode, out
+
+def exec_cmd(cmd_and_args):
+    args = cmd_and_args.split()
+    ret, out = runtool(args)
+    out = out.strip()
+    if ret != 0:
+        bb.fatal("exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
+                 (cmd_and_args, ret, out))
+    return ret, out
+
+def install_efi_part_table(iso_img):
+    find_efi_img_cmd = "xorriso -indev %s -find /efi.img \
+                        -name efi.img -exec report_lba --" % iso_img
+    ret, out = exec_cmd(find_efi_img_cmd)
+    efi_img_start = -1
+    efi_img_length = -1
+    for line in out.split("\n"):
+        if "File data lba:" in line and "/efi.img" in line:
+            file_stat = line[14:].split(',')
+            efi_img_start = int(file_stat[1].strip()) * 4
+            efi_img_length = int(int(file_stat[3].strip()) / 512)
+            break
+    if (efi_img_start < 0) or (efi_img_length < 0):
+        bb.fatal("Failed to determine /efi.img attributes")
+    mk_efi_part_table(iso_img, efi_img_start, efi_img_length)
+
 populate_live() {
-    populate_kernel $1
+	populate_kernel $1
 	if [ -s "${ROOTFS}" ]; then
 		install -m 0644 ${ROOTFS} $1/rootfs.img
 	fi
 }
 
-build_iso() {
+build_iso_base() {
 	# Only create an ISO if we have an INITRD and the live or iso image type was selected
 	if [ -z "${INITRD}" ] || [ "${@bb.utils.contains_any('IMAGE_FSTYPES', 'live iso', '1', '0', d)}" != "1" ]; then
 		bbnote "ISO image will not be created."
@@ -104,11 +184,13 @@ build_iso() {
 	fi
 
 	# EFI only
-	if [ "${PCBIOS}" != "1" ] && [ "${EFI}" = "1" ] ; then
-		# Work around bug in isohybrid where it requires isolinux.bin
-		# In the boot catalog, even though it is not used
-		mkdir -p ${ISODIR}/${ISOLINUXDIR}
-		install -m 0644 ${STAGING_DATADIR}/syslinux/isolinux.bin ${ISODIR}${ISOLINUXDIR}
+	if [ -n "$(echo ${TARGET_ARCH} | grep -e i.86)" ] || [ "${TARGET_ARCH}" = "x86_64" ]; then
+		if [ "${PCBIOS}" != "1" ] && [ "${EFI}" = "1" ]; then
+			# Work around bug in isohybrid where it requires isolinux.bin
+			# In the boot catalog, even though it is not used
+			mkdir -p ${ISODIR}/${ISOLINUXDIR}
+			install -m 0644 ${STAGING_DATADIR}/syslinux/isolinux.bin ${ISODIR}${ISOLINUXDIR}
+		fi
 	fi
 
 	# We used to have support for zisofs; this is a relic of that
@@ -128,7 +210,15 @@ build_iso() {
 		fi
 	fi
 
-	if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ] ; then
+	if [ -z "$(echo ${TARGET_ARCH} | grep -e i.86)" ] && [ "${TARGET_ARCH}" != "x86_64" ]; then
+		mkisofs -A ${BOOTIMG_VOLUME_ID} -V ${BOOTIMG_VOLUME_ID} \
+		        -o ${IMGDEPLOYDIR}/${IMAGE_NAME}.iso \
+			$mkisofs_compress_opts $mkisofs_iso_level \
+			${ISODIR}
+		return
+	fi
+
+	if [ "${PCBIOS}" = "1" ] && [ "${EFI}" != "1" ]; then
 		# PCBIOS only media
 		mkisofs -V ${BOOTIMG_VOLUME_ID} \
 		        -o ${IMGDEPLOYDIR}/${IMAGE_NAME}.iso \
@@ -150,6 +240,17 @@ build_iso() {
 	isohybrid $isohybrid_args ${IMGDEPLOYDIR}/${IMAGE_NAME}.iso
 }
 
+python build_iso() {
+    import re
+
+    bb.build.exec_func("build_iso_base", d)
+    target_arch = d.getVar("TARGET_ARCH")
+    if not re.match("i.86", target_arch) and not re.match("x86_64", target_arch) \
+       and d.getVar("EFI") == "1":
+        install_efi_part_table(d.getVar("IMGDEPLOYDIR") + "/" + \
+                               d.getVar("IMAGE_NAME") + ".iso")
+}
+
 build_fat_img() {
 	FATSOURCEDIR=$1
 	FATIMG=$2
-- 
2.34.1

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Ross Burton]

On 7 Nov 2023, at 15:12, Andrey Popov via lists.openembedded.org <andrey.popov=yadro.com@lists.openembedded.org> wrote:
> 
> Since syslinux is only compatible with platforms that use x86-based
> CPUs, this change allows creation of bootable ISO images for other
> EFI-compatible platforms by replacing invocation of the isohybrid
> tool for those platforms with a python script that creates MBR
> partition table with a single entry that points to a bootable
> EFI image placed inside the ISO image.

I’ll reply with comments on the code shortly, but my initial thoughts are mainly does anyone actually use ISO images still, especially on non-x86 systems?  I’m curious what problem you’re trying to solve here.

Ross

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Chuck Wolber]

[-- Attachment #1: Type: text/plain, Size: 605 bytes --]

On Tue, Nov 28, 2023 at 9:55 AM Ross Burton <ross.burton@arm.com> wrote:

>
> I’ll reply with comments on the code shortly, but my initial thoughts are
> mainly does anyone actually use ISO images still, especially on non-x86
> systems?  I’m curious what problem you’re trying to solve here.
>

I can only speak for the projects I am involved in - yes, ISO usage is
still *very much* alive and well. Mostly x86 type systems for the time
being, but non-x86 is on the horizon.

..Ch:W..

-- 
*"Perfection must be reached by degrees; she requires the slow hand of
time." - Voltaire*

[-- Attachment #2: Type: text/html, Size: 1080 bytes --]

Re: [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Andrey Popov]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

Hello, Ross!

Currently, we are using a Yocto-based network equipment software distribution called TanoWRT (https://tano-systems.gitlab.io/meta-tanowrt/).
This distribution provides a way to create images that can be used to facilitate network boot process using PXE. Those images are actually ISO images, and they work well for x86-based platforms, since they use already available syslinux stubs that were used for that purpose for a very long time. Unfortunately, syslinux does not support platforms that use CPU of different architectures, which, however, still support network boot process as per the UEFI spec. For those platforms, the image that is used for network boot is still the very same ISO image, the only difference is that it has the 2nd stage bootloader in a standard EFI PE format prepended to it.
I hope that my explanation answers you question.

Best regards, Andrey.

[-- Attachment #2: Type: text/html, Size: 920 bytes --]

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Ross Burton]

On 28 Nov 2023, at 19:00, Chuck Wolber <chuckwolber@gmail.com> wrote:
> 
> On Tue, Nov 28, 2023 at 9:55 AM Ross Burton <ross.burton@arm.com> wrote:
> 
> I’ll reply with comments on the code shortly, but my initial thoughts are mainly does anyone actually use ISO images still, especially on non-x86 systems?  I’m curious what problem you’re trying to solve here.
> 
> I can only speak for the projects I am involved in - yes, ISO usage is still *very much* alive and well. Mostly x86 type systems for the time being, but non-x86 is on the horizon.

And by ISO we’re talking about proper burnt CD or DVDs?

Ross

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Chuck Wolber]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

On Tue, Nov 28, 2023 at 11:50 AM Ross Burton <Ross.Burton@arm.com> wrote:

> On 28 Nov 2023, at 19:00, Chuck Wolber <chuckwolber@gmail.com> wrote:
> >
> > On Tue, Nov 28, 2023 at 9:55 AM Ross Burton <ross.burton@arm.com> wrote:
> >
> > I’ll reply with comments on the code shortly, but my initial thoughts
> are mainly does anyone actually use ISO images still, especially on non-x86
> systems?  I’m curious what problem you’re trying to solve here.
> >
> > I can only speak for the projects I am involved in - yes, ISO usage is
> still *very much* alive and well. Mostly x86 type systems for the time
> being, but non-x86 is on the horizon.
>
> And by ISO we’re talking about proper burnt CD or DVDs?
>

Currently yes, but probably moving to dd'ing the ISO to a USB stick in the
next few years.

..Ch:W..


-- 
*"Perfection must be reached by degrees; she requires the slow hand of
time." - Voltaire*

[-- Attachment #2: Type: text/html, Size: 1597 bytes --]

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Ross Burton]

On 7 Nov 2023, at 15:12, Andrey Popov via lists.openembedded.org <andrey.popov=yadro.com@lists.openembedded.org> wrote:
> @@ -29,8 +29,10 @@ do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \
>                         mtools-native:do_populate_sysroot \
>                         cdrtools-native:do_populate_sysroot \
>                         virtual/kernel:do_deploy \
> -                        ${MLPREFIX}syslinux:do_populate_sysroot \
> -                        syslinux-native:do_populate_sysroot \
> +                        ${@d.getVar('MLPREFIX') + 'syslinux:do_populate_sysroot \
> +                           syslinux-native:do_populate_sysroot' if __import__('re').match('i.86', d.getVar('TARGET_ARCH')) or \
> +                                                                   __import__('re').match('x86_64', d.getVar('TARGET_ARCH')) else \
> +                           'xorriso-native:do_populate_sysroot'} \
>                         ${@'%s:do_image_%s' % (d.getVar('PN'), d.getVar('LIVE_ROOTFS_TYPE').replace('-', '_')) if d.getVar('ROOTFS') else ''} \
>                         "

Instead of looking at the TARGET_ARCH, please use the perfectly good machine features that already exist: pcbios and efi. If the target is EFI-only there’s no point in supporting syslinux, and it would be nice to be able to drop it entirely one day in the future.  This class shouldn’t be checking the architecture at all, simply check for the efi and pcbios features.

This also pulls in a hard-dependency on xorriso which isn’t in oe-core, so that will break the users of the class that just use core.  xorriso also doesn’t exist in any layer anymore as I removed it from meta-oe in nanbield, so I know that you’ve not tested this code with master or nanbield.

> @@ -71,14 +73,92 @@ MKISOFS_OPTIONS = "-no-emul-boot -boot-load-size 4 -boot-info-table"
> BOOTIMG_VOLUME_ID   ?= "boot"
> BOOTIMG_EXTRA_SPACE ?= "512"
> 
> +def compute_chs(sector_z):
> +    C = int(sector_z / (63 * 255))
> +    H = int((sector_z % (63 * 255)) / 63)
> +    # convert zero-based sector to CHS format
> +    S = int(sector_z % 63) + 1
> +    # munge accord to partition table format
> +    S = (S & 0x3f) | (((C >> 8) & 0x3) << 6)
> +    C = (C & 0xFF)
> +    return (C, H, S)
> +
> +def mk_efi_part_table(iso, start, length):
> +    from struct import pack
> +
> +    # Compute starting and ending CHS addresses for the partition entry.
> +    (s_C, s_H, s_S) = compute_chs(start)
> +    (e_C, e_H, e_S) = compute_chs(start + length - 1)
> +
> +    # Write the 66 byte partition table to bytes 0x1BE through 0x1FF in
> +    # sector 0 of the .ISO.
> +    #
> +    # See the partition table format here:
> +    # http://en.wikipedia.org/wiki/Master_boot_record#Sector_layout
> +    f = open(iso, 'r+b')
> +    f.seek(0x1BE)
> +    f.write(pack("<8BLL48xH", 0x80, s_H, s_S, s_C,
> +                 0xEF, e_H, e_S, e_C, start, length, 0xAA55))
> +    f.close()

Why not continue to use syslinux-native to provide isohybrid? My understanding is that isohybrid simply messes with partition tables and doesn’t depend on a target syslinux. Of course if I’m wrong then this needs to be explained in the class: we’ve a problem with complicated classes being written without any explanation of what is happening so lets try to improve the situation instead of making it worse.

> +def runtool(cmdln_or_args):
> +    import subprocess
> +
> +    if isinstance(cmdln_or_args, list):
> +        cmd = cmdln_or_args[0]
> +        shell = False
> +    else:
> +        import shlex
> +        cmd = shlex.split(cmdln_or_args)[0]
> +        shell = True
> +
> +    sout = subprocess.PIPE
> +    serr = subprocess.STDOUT
> +
> +    try:
> +        process = subprocess.Popen(cmdln_or_args, stdout=sout,
> +                                   stderr=serr, shell=shell)
> +        sout, serr = process.communicate()
> +        # combine stdout and stderr, filter None out and decode
> +        out = ''.join([out.decode('utf-8') for out in [sout, serr] if out])
> +    except OSError as err:
> +        bb.fatal("Cannot run command %s: %s" % (cmd, err))
> +
> +    return process.returncode, out
> +
> +def exec_cmd(cmd_and_args):
> +    args = cmd_and_args.split()
> +    ret, out = runtool(args)
> +    out = out.strip()
> +    if ret != 0:
> +        bb.fatal("exec_cmd: %s returned '%s' instead of 0\noutput: %s" % \
> +                 (cmd_and_args, ret, out))
> +    return ret, out

These two functions are basically a bad reimplementation of subproccess.run(). Don’t copy-paste, just use modern API.

Ross

Re: [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Andrey Popov]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

Hello, Ross!

Since it's no longer possible to use xorriso in recent versions, I would like to ask if it's possible to apply an updated version of this patch to the kirkstone branch.
If so, I will send the new version matching that branch.

Best regards, Andrey.

[-- Attachment #2: Type: text/html, Size: 289 bytes --]

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Steve Sakoman]

On Wed, Nov 29, 2023 at 7:26 AM Andrey Popov <andrey.popov@yadro.com> wrote:
>
> Hello, Ross!
>
> Since it's no longer possible to use xorriso in recent versions, I would like to ask if it's possible to apply an updated version of this patch to the kirkstone branch.
> If so, I will send the new version matching that branch.

Since this adds a new feature it isn't something that I can take for kirkstone.

Steve

>
> Best regards, Andrey.
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191459): https://lists.openembedded.org/g/openembedded-core/message/191459
> Mute This Topic: https://lists.openembedded.org/mt/102444305/3620601
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Ross Burton]

On 29 Nov 2023, at 17:26, Andrey Popov via lists.openembedded.org <andrey.popov=yadro.com@lists.openembedded.org> wrote:
> Since it's no longer possible to use xorriso in recent versions, I would like to ask if it's possible to apply an updated version of this patch to the kirkstone branch.

Xorriso has been removed but it was replaced with the separate components, so you don’t need to depend on the entire suite just for one part.

The point that these are in meta-oe still stands, though.

It’s not possible to submit new features to a stable branch without it landing in master first.  Otherwise someone (including you) would use the feature in kirkstone, upgrade, and wonder why this feature doesn’t exist. 

Ross

Re: [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Andrey Popov]

[-- Attachment #1: Type: text/plain, Size: 160 bytes --]

Hello, Ross, Steve!

Would it be possible to apply backported patch to kirkstone branch if updated patch actually lands for master?

Best regards, Andrey.

[-- Attachment #2: Type: text/html, Size: 176 bytes --]

Re: [OE-core] [PATCH v2] image-live: Add support for building EFI-bootable ISO images for non-x86-based archs

[Steve Sakoman]

On Thu, Nov 30, 2023 at 6:20 AM Andrey Popov <andrey.popov@yadro.com> wrote:
>
> Hello, Ross, Steve!
>
> Would it be possible to apply backported patch to kirkstone branch if updated patch actually lands for master?

Since it is a new feature, no it couldn't be taken for a stable branch.

Steve

> Best regards, Andrey.
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#191514): https://lists.openembedded.org/g/openembedded-core/message/191514
> Mute This Topic: https://lists.openembedded.org/mt/102444305/3620601
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [steve@sakoman.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>