[PATCH] wic/plugins/source/bootimg-efi: Configure installation of kernel image into boot partition.

[PATCH] wic/plugins/source/bootimg-efi: Configure installation of kernel image into boot partition.

[Kareem Zarka]

The issue with installing the kernel image to both rootfs
and boot partition is that some systems rely on the kernel image in
rootfs and not in the boot partition.
This leads to duplication of the kernel image, which can cause
unnecessary storage usage.
This patch provides a solution to the problem by adding a new parameter
"install-kernel-into-boot-dir" to the wic kickstart file.
If this parameter is set to 'true', the plugin will install the
kernel image to the boot partition. If the parameter is set to 'false',
the plugin will skip installing the kernel image, avoiding duplication.
Tests for this functionality will be added in the next patch.

Signed-off-by: Kareem Zarka <kareem.zarka@huawei.com>
---
 scripts/lib/wic/plugins/source/bootimg-efi.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index d6aeab2aad..09e9d6417c 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -363,9 +363,10 @@ class BootimgEFIPlugin(SourcePlugin):
                 objcopy_cmd += " %s %s/EFI/Linux/linux.efi" % (efi_stub, hdddir)
                 exec_native_cmd(objcopy_cmd, native_sysroot)
         else:
-            install_cmd = "install -m 0644 %s/%s %s/%s" % \
-                (staging_kernel_dir, kernel, hdddir, kernel)
-            exec_cmd(install_cmd)
+            if source_params.get('install-kernel-into-boot-dir') != 'false':
+                install_cmd = "install -m 0644 %s/%s %s/%s" % \
+                    (staging_kernel_dir, kernel, hdddir, kernel)
+                exec_cmd(install_cmd)
 
         if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
             for src_path, dst_path in cls.install_task:
-- 
2.25.1

[PATCH] meta/lib/oeqa/selftest/cases/wic: Add tests for configuring kernel image install into boot partition.

[Kareem Zarka]

- test_skip_kernel_install: This test verifies that the kernel is not
installed in the boot partition when the 'install-kernel-into-boot-dir'
parameter is set to false.
- test_kernel_install: This test verifies that the kernel is installed
in the boot partition when the 'install-kernel-into-boot-dir' parameter
is set to true .
Both tests use a WKS (Kickstart) file to specify the desired
configuration, build a disk image using WIC, and extract the disk image
to a temporary directory to verify the results.

Signed-off-by: Kareem Zarka <kareem.zarka@huawei.com>
---
 meta/lib/oeqa/selftest/cases/wic.py | 70 +++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index b9430cdb3b..7f5db1dc73 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -16,6 +16,7 @@ import hashlib
 from glob import glob
 from shutil import rmtree, copy
 from tempfile import NamedTemporaryFile
+from tempfile import TemporaryDirectory
 
 from oeqa.selftest.case import OESelftestTestCase
 from oeqa.core.decorator import OETestTag
@@ -146,6 +147,75 @@ class CLITests(OESelftestTestCase):
         self.assertEqual(1, runCmd('wic', ignore_status=True).status)
 
 class Wic(WicTestCase):
+    def test_skip_kernel_install(self):
+        """Test the functionality of not installing the kernel in the boot directory using the wic plugin"""
+        # Build the mtools package to support FAT filesystem handling
+        bitbake("mtools") 
+        # create a temporary file for the WKS content
+        with NamedTemporaryFile("w", suffix=".wks") as wks:
+            wks.write(
+                'part --source bootimg-efi '
+                '--sourceparams="loader=grub-efi,install-kernel-into-boot-dir=false" '
+                '--label boot --active\n'
+            )
+            wks.flush()
+            # create a temporary directory to extract the disk image to
+            with TemporaryDirectory() as tmpdir:
+                img = 'core-image-minimal'
+                # build the image using the WKS file
+                cmd = "wic create %s -e %s -o %s" % (
+                    wks.name, img, self.resultdir)
+                runCmd(cmd)
+                wksname = os.path.splitext(os.path.basename(wks.name))[0]
+                out = glob(os.path.join(
+                    self.resultdir, "%s-*.direct" % wksname))
+                self.assertEqual(1, len(out))
+                # extract the content of the disk image to the temporary directory
+                cmd = "wic cp %s:1 %s" % (out[0], tmpdir)
+                runCmd(cmd)
+                # check if the kernel is installed or not
+                kimgtype = get_bb_var('KERNEL_IMAGETYPE', img)
+                for file in os.listdir(tmpdir):
+                    if file == kimgtype:
+                        raise AssertionError(
+                            "The kernel image '{}' was found in the partition".format(kimgtype)
+                        )
+
+    def test_kernel_install(self):
+        """Test the installation of the kernel to the boot directory in the wic plugin"""
+        # Build the mtools package to support FAT filesystem handling
+        bitbake("mtools") 
+        # create a temporary file for the WKS content
+        with NamedTemporaryFile("w", suffix=".wks") as wks:
+            wks.write(
+                'part --source bootimg-efi '
+                '--sourceparams="loader=grub-efi,install-kernel-into-boot-dir=true" '
+                '--label boot --active\n'
+            )
+            wks.flush()
+            # create a temporary directory to extract the disk image to
+            with TemporaryDirectory() as tmpdir:
+                img = 'core-image-minimal'
+                # build the image using the WKS file
+                cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
+                runCmd(cmd)
+                wksname = os.path.splitext(os.path.basename(wks.name))[0]
+                out = glob(os.path.join(self.resultdir, "%s-*.direct" % wksname))
+                self.assertEqual(1, len(out))
+                # extract the content of the disk image to the temporary directory
+                cmd = "wic cp %s:1 %s" % (out[0], tmpdir)
+                runCmd(cmd)
+                # check if the kernel is installed or not
+                kimgtype = get_bb_var('KERNEL_IMAGETYPE', img)
+                found = False
+                for file in os.listdir(tmpdir):
+                    if file == kimgtype:
+                        found = True
+                        break
+                self.assertTrue(
+                    found, "The kernel image '{}' was not found in the boot partition".format(kimgtype)
+                )
+
     def test_build_image_name(self):
         """Test wic create wictestdisk --image-name=core-image-minimal"""
         cmd = "wic create wictestdisk --image-name=core-image-minimal -o %s" % self.resultdir
-- 
2.25.1

Re: [PATCH] meta/lib/oeqa/selftest/cases/wic: Add tests for configuring kernel image install into boot partition.

[Taedcke, Christian]

Hello Kareem,

are you planning to continue withis this review? If not i would try to 
get this merged.

On 08.03.2023 11:34, Kareem Zarka wrote:
> - test_skip_kernel_install: This test verifies that the kernel is not
> installed in the boot partition when the 'install-kernel-into-boot-dir'
> parameter is set to false.
> - test_kernel_install: This test verifies that the kernel is installed
> in the boot partition when the 'install-kernel-into-boot-dir' parameter
> is set to true .
> Both tests use a WKS (Kickstart) file to specify the desired
> configuration, build a disk image using WIC, and extract the disk image
> to a temporary directory to verify the results.
> 
> Signed-off-by: Kareem Zarka <kareem.zarka@huawei.com>
> ---
>   meta/lib/oeqa/selftest/cases/wic.py | 70 +++++++++++++++++++++++++++++
>   1 file changed, 70 insertions(+)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
> index b9430cdb3b..7f5db1dc73 100644
> --- a/meta/lib/oeqa/selftest/cases/wic.py
> +++ b/meta/lib/oeqa/selftest/cases/wic.py
> @@ -16,6 +16,7 @@ import hashlib
>   from glob import glob
>   from shutil import rmtree, copy
>   from tempfile import NamedTemporaryFile
> +from tempfile import TemporaryDirectory
>   
>   from oeqa.selftest.case import OESelftestTestCase
>   from oeqa.core.decorator import OETestTag
> @@ -146,6 +147,75 @@ class CLITests(OESelftestTestCase):
>           self.assertEqual(1, runCmd('wic', ignore_status=True).status)
>   
>   class Wic(WicTestCase):
> +    def test_skip_kernel_install(self):
> +        """Test the functionality of not installing the kernel in the boot directory using the wic plugin"""
> +        # Build the mtools package to support FAT filesystem handling
> +        bitbake("mtools")

I believe this should be 'bitbake("mtools-native")'. This should fix the 
error on the build server.

> +        # create a temporary file for the WKS content
> +        with NamedTemporaryFile("w", suffix=".wks") as wks:
> +            wks.write(
> +                'part --source bootimg-efi '
> +                '--sourceparams="loader=grub-efi,install-kernel-into-boot-dir=false" '
> +                '--label boot --active\n'
> +            )
> +            wks.flush()
> +            # create a temporary directory to extract the disk image to
> +            with TemporaryDirectory() as tmpdir:
> +                img = 'core-image-minimal'
> +                # build the image using the WKS file
> +                cmd = "wic create %s -e %s -o %s" % (
> +                    wks.name, img, self.resultdir)
> +                runCmd(cmd)
> +                wksname = os.path.splitext(os.path.basename(wks.name))[0]
> +                out = glob(os.path.join(
> +                    self.resultdir, "%s-*.direct" % wksname))
> +                self.assertEqual(1, len(out))
> +                # extract the content of the disk image to the temporary directory
> +                cmd = "wic cp %s:1 %s" % (out[0], tmpdir)
> +                runCmd(cmd)
> +                # check if the kernel is installed or not
> +                kimgtype = get_bb_var('KERNEL_IMAGETYPE', img)
> +                for file in os.listdir(tmpdir):
> +                    if file == kimgtype:
> +                        raise AssertionError(
> +                            "The kernel image '{}' was found in the partition".format(kimgtype)
> +                        )
> +
> +    def test_kernel_install(self):
> +        """Test the installation of the kernel to the boot directory in the wic plugin"""
> +        # Build the mtools package to support FAT filesystem handling
> +        bitbake("mtools")

Probably 'bitbake("mtools-native")'

> +        # create a temporary file for the WKS content
> +        with NamedTemporaryFile("w", suffix=".wks") as wks:
> +            wks.write(
> +                'part --source bootimg-efi '
> +                '--sourceparams="loader=grub-efi,install-kernel-into-boot-dir=true" '
> +                '--label boot --active\n'
> +            )
> +            wks.flush()
> +            # create a temporary directory to extract the disk image to
> +            with TemporaryDirectory() as tmpdir:
> +                img = 'core-image-minimal'
> +                # build the image using the WKS file
> +                cmd = "wic create %s -e %s -o %s" % (wks.name, img, self.resultdir)
> +                runCmd(cmd)
> +                wksname = os.path.splitext(os.path.basename(wks.name))[0]
> +                out = glob(os.path.join(self.resultdir, "%s-*.direct" % wksname))
> +                self.assertEqual(1, len(out))
> +                # extract the content of the disk image to the temporary directory
> +                cmd = "wic cp %s:1 %s" % (out[0], tmpdir)
> +                runCmd(cmd)
> +                # check if the kernel is installed or not
> +                kimgtype = get_bb_var('KERNEL_IMAGETYPE', img)
> +                found = False
> +                for file in os.listdir(tmpdir):
> +                    if file == kimgtype:
> +                        found = True
> +                        break
> +                self.assertTrue(
> +                    found, "The kernel image '{}' was not found in the boot partition".format(kimgtype)
> +                )
> +
>       def test_build_image_name(self):
>           """Test wic create wictestdisk --image-name=core-image-minimal"""
>           cmd = "wic create wictestdisk --image-name=core-image-minimal -o %s" % self.resultdir

Regards,
Christian

Re: [OE-core] [PATCH] wic/plugins/source/bootimg-efi: Configure installation of kernel image into boot partition.

[Alexandre Belloni]

Hello,

This still fails on the autobuilders, please fix before submitting a new
version (and please mark it as a new version):

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/4853/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4933/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/4906/steps/14/logs/stdio

2023-03-09 09:58:24,299 - oe-selftest - INFO - wic.Wic.test_kernel_install (subunit.RemotedTestCase)
2023-03-09 09:58:24,299 - oe-selftest - INFO -  ... FAIL

2023-03-09 09:58:24,299 - oe-selftest - INFO - 6: 47/60 506/526 (23.54s) (0 failed) (wic.Wic.test_kernel_install)
2023-03-09 09:58:24,300 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/wic.py", line 207, in test_kernel_install
    runCmd(cmd)
  File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/utils/commands.py", line 214, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'wic cp /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-1380163/wic-tmp/tmpw4s70lpy-202303090958-sda.direct:1 /tmp/tmpbww7gjoy' returned non-zero exit status 1:
ERROR: Can't find executable 'mcopy'



On 08/03/2023 11:34:47+0100, Kareem Zarka wrote:
> The issue with installing the kernel image to both rootfs
> and boot partition is that some systems rely on the kernel image in
> rootfs and not in the boot partition.
> This leads to duplication of the kernel image, which can cause
> unnecessary storage usage.
> This patch provides a solution to the problem by adding a new parameter
> "install-kernel-into-boot-dir" to the wic kickstart file.
> If this parameter is set to 'true', the plugin will install the
> kernel image to the boot partition. If the parameter is set to 'false',
> the plugin will skip installing the kernel image, avoiding duplication.
> Tests for this functionality will be added in the next patch.
> 
> Signed-off-by: Kareem Zarka <kareem.zarka@huawei.com>
> ---
>  scripts/lib/wic/plugins/source/bootimg-efi.py | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
> index d6aeab2aad..09e9d6417c 100644
> --- a/scripts/lib/wic/plugins/source/bootimg-efi.py
> +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
> @@ -363,9 +363,10 @@ class BootimgEFIPlugin(SourcePlugin):
>                  objcopy_cmd += " %s %s/EFI/Linux/linux.efi" % (efi_stub, hdddir)
>                  exec_native_cmd(objcopy_cmd, native_sysroot)
>          else:
> -            install_cmd = "install -m 0644 %s/%s %s/%s" % \
> -                (staging_kernel_dir, kernel, hdddir, kernel)
> -            exec_cmd(install_cmd)
> +            if source_params.get('install-kernel-into-boot-dir') != 'false':
> +                install_cmd = "install -m 0644 %s/%s %s/%s" % \
> +                    (staging_kernel_dir, kernel, hdddir, kernel)
> +                exec_cmd(install_cmd)
>  
>          if get_bitbake_var("IMAGE_EFI_BOOT_FILES"):
>              for src_path, dst_path in cls.install_task:
> -- 
> 2.25.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#178164): https://lists.openembedded.org/g/openembedded-core/message/178164
> Mute This Topic: https://lists.openembedded.org/mt/97469748/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com