[PATCH v3 1/2] wic: add --extra-partition-space option to set unused space

[PATCH v3 1/2] wic: add --extra-partition-space option to set unused space

[pierre-loup.gosse]

From: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>

By default, the content of the partition is filled by the filesystem
without leaving any unused free space. The --extra-space flag adds
extra space to the filesystem size, not to the partition.

Unused free space after the filesystem can be useful for some cases,
such as encrypting a partition at runtime.
With --extra-partition-space 32M, we ensure that the last 32M of the
partition is unused: this space does not contain filesystem data and
can store the LUKS2 header.

The implementation sets a difference between the partition and
filesystem size:
  - With --fixed-size, the extra part space is removed from the
    filesystem size.
  - Otherwise (with or without --size flag), the extra part space is
    added to the partition size.

Signed-off-by: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>

CC: Alexander Kanavin <alex.kanavin@gmail.com>
CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
changes in v2:
- renaming the option to --extra-part-space
- adding tests

changes in v3:
- renaming the option to --extra-partition-space
- fixing the tests
- adding --extra-partition-space help description
---
 meta/lib/oeqa/selftest/cases/wic.py | 58 ++++++++++++++++++++++++++++-
 scripts/lib/wic/help.py             |  8 ++++
 scripts/lib/wic/ksparser.py         |  3 ++
 scripts/lib/wic/partition.py        | 38 +++++++++++--------
 4 files changed, 90 insertions(+), 17 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 680f99d381..09257f2624 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1125,7 +1125,7 @@ run_wic_cmd() {
 
         return wkspath
 
-    def _get_wic_partitions(self, wkspath, native_sysroot=None, ignore_status=False):
+    def _get_wic(self, wkspath, ignore_status=False):
         p = runCmd("wic create %s -e core-image-minimal -o %s" % (wkspath, self.resultdir),
                    ignore_status=ignore_status)
 
@@ -1139,7 +1139,13 @@ run_wic_cmd() {
         if not wicout:
             return (p, None)
 
-        wicimg = wicout[0]
+        return (p, wicout[0])
+
+    def _get_wic_partitions(self, wkspath, native_sysroot=None, ignore_status=False):
+        p, wicimg = self._get_wic(wkspath, ignore_status)
+
+        if wicimg is None:
+            return (p, None)
 
         if not native_sysroot:
             native_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "wic-tools")
@@ -1279,6 +1285,54 @@ run_wic_cmd() {
             size = int(size[:-3])
             self.assertGreaterEqual(size, 204800)
 
+    def test_extra_partition_space(self):
+        native_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "wic-tools")
+
+        with NamedTemporaryFile("w", suffix=".wks") as tempf:
+            tempf.write("bootloader --ptable gpt\n" \
+                        "part                 --ondisk hda --size 10M        --extra-partition-space 10M --fstype=ext4\n" \
+                        "part                 --ondisk hda --fixed-size 20M  --extra-partition-space 10M --fstype=ext4\n" \
+                        "part --source rootfs --ondisk hda                   --extra-partition-space 10M --fstype=ext4\n" \
+                        "part --source rootfs --ondisk hda --fixed-size 200M --extra-partition-space 10M --fstype=ext4\n")
+            tempf.flush()
+
+            _, wicimg = self._get_wic(tempf.name)
+
+            res = runCmd("parted -m %s unit b p" % wicimg,
+                            native_sysroot=native_sysroot, stderr=subprocess.PIPE)
+
+            # parse parted output which looks like this:
+            # BYT;\n
+            # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n
+            # 1:0.00MiB:200MiB:200MiB:ext4::;\n
+            partlns = res.output.splitlines()[2:]
+
+            self.assertEqual(4, len(partlns))
+
+            # Test for each partitions that the extra part space exists
+            for part in range(0, len(partlns)):
+                part_file = os.path.join(self.resultdir, "selftest_img.part%d" % (part + 1))
+                partln = partlns[part].split(":")
+                self.assertEqual(7, len(partln))
+                self.assertRegex(partln[3], r'^[0-9]+B$')
+                part_size = int(partln[3].rstrip("B"))
+                start = int(partln[1].rstrip("B")) / 512
+                length = part_size / 512
+                runCmd("dd if=%s of=%s skip=%d count=%d" %
+                                            (wicimg, part_file, start, length))
+                res = runCmd("dumpe2fs %s -h | grep \"^Block count\"" % part_file)
+                fs_size = int(res.output.split(":")[1].strip()) * 1024
+                self.assertLessEqual(fs_size + 10485760, part_size, "part file: %s" % part_file)
+
+        with NamedTemporaryFile("w", suffix=".wks") as tempf:
+            # Test that image creation fails if space left is too small for the rootfs
+            tempf.write("bootloader --ptable gpt\n" \
+                        "part --source rootfs --ondisk hda --fixed-size 200M --extra-partition-space 150M --fstype=ext4\n")
+            tempf.flush()
+
+            p, _ = self._get_wic(tempf.name, ignore_status=True)
+            self.assertNotEqual(p.status, 0, "wic exited successfully when an error was expected:\n%s" % p.output)
+
     # TODO this test could also work on aarch64
     @skipIfNotArch(['i586', 'i686', 'x86_64'])
     @OETestTag("runqemu")
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 2e3061f343..800c0abf0f 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -1020,6 +1020,14 @@ DESCRIPTION
                         By default, 10MB. This option cannot be used
                         with --fixed-size option.
 
+         --extra-partition-space: This option is specific to wic. It adds extra
+                                  empty space after the space filled by the
+                                  filesystem. With --fixed-size, the extra
+                                  partition space is removed from the filesystem
+                                  size. Otherwise (with or without --size flag),
+                                  the extra partition space is added to the final
+                                  paritition size. The default value is 0MB.
+
          --overhead-factor: This option is specific to wic. The
                             size of the partition is multiplied by
                             this factor. It has to be greater than or
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 7ef3dc83dd..e86e1069f5 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -154,6 +154,7 @@ class KickStart():
         part.add_argument('--include-path', nargs='+', action='append')
         part.add_argument('--change-directory')
         part.add_argument("--extra-space", type=sizetype("M"))
+        part.add_argument('--extra-partition-space', type=sizetype("M"))
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fspassno', dest='fspassno')
         part.add_argument('--fstype', default='vfat',
@@ -259,6 +260,8 @@ class KickStart():
                             err = "%s:%d: Must set the label with --label" \
                                   % (confpath, lineno)
                             raise KickStartError(err)
+                        if not parsed.extra_partition_space:
+                            parsed.extra_partition_space = 0
                         # using ArgumentParser one cannot easily tell if option
                         # was passed as argument, if said option has a default
                         # value; --overhead-factor/--extra-space cannot be used
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index b34691d313..e71052fd15 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -29,6 +29,7 @@ class Partition():
         self.disk = args.disk
         self.device = None
         self.extra_space = args.extra_space
+        self.extra_partition_space = args.extra_partition_space
         self.exclude_path = args.exclude_path
         self.include_path = args.include_path
         self.change_directory = args.change_directory
@@ -91,17 +92,16 @@ class Partition():
     def get_rootfs_size(self, actual_rootfs_size=0):
         """
         Calculate the required size of rootfs taking into consideration
-        --size/--fixed-size flags as well as overhead and extra space, as
-        specified in kickstart file. Raises an error if the
-        `actual_rootfs_size` is larger than fixed-size rootfs.
-
+        --size/--fixed-size and --extra-partition-space flags as well as overhead
+        and extra space, as specified in kickstart file. Raises an error
+        if the `actual_rootfs_size` is larger than fixed-size rootfs.
         """
         if self.fixed_size:
-            rootfs_size = self.fixed_size
+            rootfs_size = self.fixed_size - self.extra_partition_space
             if actual_rootfs_size > rootfs_size:
                 raise WicError("Actual rootfs size (%d kB) is larger than "
-                               "allowed size %d kB" %
-                               (actual_rootfs_size, rootfs_size))
+                               "allowed size %d kB (including %d kB extra part space)" %
+                               (actual_rootfs_size, rootfs_size, self.extra_partition_space))
         else:
             extra_blocks = self.get_extra_block_count(actual_rootfs_size)
             if extra_blocks < self.extra_space:
@@ -119,10 +119,18 @@ class Partition():
     def disk_size(self):
         """
         Obtain on-disk size of partition taking into consideration
-        --size/--fixed-size options.
+        --size/--fixed-size and --extra-partition-space options.
+
+        """
+        return self.fixed_size if self.fixed_size else self.size + self.extra_partition_space
 
+    @property
+    def fs_size(self):
+        """
+        Obtain on-disk size of filesystem inside the partition taking into
+        consideration --size/--fixed-size and --extra-partition-space options.
         """
-        return self.fixed_size if self.fixed_size else self.size
+        return self.fixed_size - self.extra_partition_space if self.fixed_size else self.size
 
     def prepare(self, creator, cr_workdir, oe_builddir, rootfs_dir,
                 bootimg_dir, kernel_dir, native_sysroot, updated_fstab_path):
@@ -202,10 +210,10 @@ class Partition():
                            "This a bug in source plugin %s and needs to be fixed." %
                            (self.mountpoint, self.source))
 
-        if self.fixed_size and self.size > self.fixed_size:
+        if self.fixed_size and self.size + self.extra_partition_space > self.fixed_size:
             raise WicError("File system image of partition %s is "
-                           "larger (%d kB) than its allowed size %d kB" %
-                           (self.mountpoint, self.size, self.fixed_size))
+                           "larger (%d kB + %d kB extra part space) than its allowed size %d kB" %
+                           (self.mountpoint, self.size, self.extra_partition_space, self.fixed_size))
 
     def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
                        native_sysroot, real_rootfs = True, pseudo_dir = None):
@@ -440,7 +448,7 @@ class Partition():
         """
         Prepare an empty ext2/3/4 partition.
         """
-        size = self.disk_size
+        size = self.fs_size
         with open(rootfs, 'w') as sparse:
             os.ftruncate(sparse.fileno(), size * 1024)
 
@@ -464,7 +472,7 @@ class Partition():
         """
         Prepare an empty btrfs partition.
         """
-        size = self.disk_size
+        size = self.fs_size
         with open(rootfs, 'w') as sparse:
             os.ftruncate(sparse.fileno(), size * 1024)
 
@@ -482,7 +490,7 @@ class Partition():
         """
         Prepare an empty vfat partition.
         """
-        blocks = self.disk_size
+        blocks = self.fs_size
 
         label_str = "-n boot"
         if self.label:
-- 
2.34.1

[PATCH v3 2/2] wic: rename wks flag --extra-space to --extra-filesystem-space

[pierre-loup.gosse]

From: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>

Makes a clear distinction with --extra-partition-space flag.

Signed-off-by: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>

CC: Alexander Kanavin <alex.kanavin@gmail.com>
CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
---
changes in v3:
- adding this patch
---
 meta/lib/oeqa/selftest/cases/wic.py |  4 ++--
 scripts/lib/wic/help.py             | 12 ++++++------
 scripts/lib/wic/ksparser.py         | 25 +++++++++++++------------
 scripts/lib/wic/partition.py        |  6 +++---
 4 files changed, 24 insertions(+), 23 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/wic.py b/meta/lib/oeqa/selftest/cases/wic.py
index 09257f2624..643d6c7200 100644
--- a/meta/lib/oeqa/selftest/cases/wic.py
+++ b/meta/lib/oeqa/selftest/cases/wic.py
@@ -1270,12 +1270,12 @@ run_wic_cmd() {
             p, _ = self._get_wic_partitions(tempf.name, ignore_status=True)
             self.assertNotEqual(p.status, 0, "wic exited successfully when an error was expected:\n%s" % p.output)
 
-    def test_extra_space(self):
+    def test_extra_filesystem_space(self):
         native_sysroot = get_bb_var("RECIPE_SYSROOT_NATIVE", "wic-tools")
 
         with NamedTemporaryFile("w", suffix=".wks") as tempf:
             tempf.write("bootloader --ptable gpt\n" \
-                        "part /     --source rootfs --ondisk hda --extra-space 200M --fstype=ext4\n")
+                        "part /     --source rootfs --ondisk hda --extra-filesystem-space 200M --fstype=ext4\n")
             tempf.flush()
 
             _, partlns = self._get_wic_partitions(tempf.name, native_sysroot)
diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 800c0abf0f..6b49a67de9 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -1013,12 +1013,12 @@ DESCRIPTION
          --no-fstab-update: This option is specific to wic. It does not update the
                             '/etc/fstab' stock file for the given partition.
 
-         --extra-space: This option is specific to wic. It adds extra
-                        space after the space filled by the content
-                        of the partition. The final size can go
-                        beyond the size specified by --size.
-                        By default, 10MB. This option cannot be used
-                        with --fixed-size option.
+         --extra-filesystem-space: This option is specific to wic. It adds extra
+                                   space after the space filled by the content
+                                   of the partition. The final size can go
+                                   beyond the size specified by --size.
+                                   By default, 10MB. This option cannot be used
+                                   with --fixed-size option.
 
          --extra-partition-space: This option is specific to wic. It adds extra
                                   empty space after the space filled by the
diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index e86e1069f5..dbf0cf2dc3 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -132,7 +132,7 @@ def systemidtype(arg):
 class KickStart():
     """Kickstart parser implementation."""
 
-    DEFAULT_EXTRA_SPACE = 10*1024
+    DEFAULT_EXTRA_FILESYSTEM_SPACE = 10*1024
     DEFAULT_OVERHEAD_FACTOR = 1.3
 
     def __init__(self, confpath):
@@ -153,7 +153,7 @@ class KickStart():
         part.add_argument('--exclude-path', nargs='+')
         part.add_argument('--include-path', nargs='+', action='append')
         part.add_argument('--change-directory')
-        part.add_argument("--extra-space", type=sizetype("M"))
+        part.add_argument('--extra-filesystem-space', type=sizetype("M"))
         part.add_argument('--extra-partition-space', type=sizetype("M"))
         part.add_argument('--fsoptions', dest='fsopts')
         part.add_argument('--fspassno', dest='fspassno')
@@ -175,9 +175,9 @@ class KickStart():
         part.add_argument('--hidden', action='store_true')
 
         # --size and --fixed-size cannot be specified together; options
-        # ----extra-space and --overhead-factor should also raise a parser
-        # --error, but since nesting mutually exclusive groups does not work,
-        # ----extra-space/--overhead-factor are handled later
+        # ----extra-filesystem-space and --overhead-factor should also raise a
+        # parser error, but since nesting mutually exclusive groups does not work,
+        # ----extra-filesystem-space/--overhead-factor are handled later
         sizeexcl = part.add_mutually_exclusive_group()
         sizeexcl.add_argument('--size', type=sizetype("M"), default=0)
         sizeexcl.add_argument('--fixed-size', type=sizetype("M"), default=0)
@@ -264,12 +264,13 @@ class KickStart():
                             parsed.extra_partition_space = 0
                         # using ArgumentParser one cannot easily tell if option
                         # was passed as argument, if said option has a default
-                        # value; --overhead-factor/--extra-space cannot be used
-                        # with --fixed-size, so at least detect when these were
-                        # passed with non-0 values ...
+                        # value; --overhead-factor/--extra-filesystem-space
+                        # cannot be used with --fixed-size, so at least detect
+                        # when these were passed with non-0 values ...
                         if parsed.fixed_size:
-                            if parsed.overhead_factor or parsed.extra_space:
-                                err = "%s:%d: arguments --overhead-factor and --extra-space not "\
+                            if parsed.overhead_factor or parsed.extra_filesystem_space:
+                                err = "%s:%d: arguments --overhead-factor and "\
+                                      "--extra-filesystem-space not "\
                                       "allowed with argument --fixed-size" \
                                       % (confpath, lineno)
                                 raise KickStartError(err)
@@ -280,8 +281,8 @@ class KickStart():
                             # with value equal to 0)
                             if '--overhead-factor' not in line_args:
                                 parsed.overhead_factor = self.DEFAULT_OVERHEAD_FACTOR
-                            if '--extra-space' not in line_args:
-                                parsed.extra_space = self.DEFAULT_EXTRA_SPACE
+                            if '--extra-filesystem-space' not in line_args:
+                                parsed.extra_filesystem_space = self.DEFAULT_EXTRA_FILESYSTEM_SPACE
 
                         self.partnum += 1
                         self.partitions.append(Partition(parsed, self.partnum))
diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py
index e71052fd15..7f95f710ff 100644
--- a/scripts/lib/wic/partition.py
+++ b/scripts/lib/wic/partition.py
@@ -28,7 +28,7 @@ class Partition():
         self.align = args.align
         self.disk = args.disk
         self.device = None
-        self.extra_space = args.extra_space
+        self.extra_filesystem_space = args.extra_filesystem_space
         self.extra_partition_space = args.extra_partition_space
         self.exclude_path = args.exclude_path
         self.include_path = args.include_path
@@ -104,8 +104,8 @@ class Partition():
                                (actual_rootfs_size, rootfs_size, self.extra_partition_space))
         else:
             extra_blocks = self.get_extra_block_count(actual_rootfs_size)
-            if extra_blocks < self.extra_space:
-                extra_blocks = self.extra_space
+            if extra_blocks < self.extra_filesystem_space:
+                extra_blocks = self.extra_filesystem_space
 
             rootfs_size = actual_rootfs_size + extra_blocks
             rootfs_size = int(rootfs_size * self.overhead_factor)
-- 
2.34.1

Re: [PATCH v3 1/2] wic: add --extra-partition-space option to set unused space

[pierre-loup.gosse]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]

The following wic.Wics tests passed:

* test_extra_filesystem_space
* test_extra_partition_space
* test_fixed_size_error
* test_offset

[-- Attachment #2: Type: text/html, Size: 191 bytes --]

Re: [OE-core] [PATCH v3 2/2] wic: rename wks flag --extra-space to --extra-filesystem-space

[Quentin Schulz]

Hi Pierre-Loup,

On 8/11/25 3:48 PM, pierre-loup.gosse via lists.openembedded.org wrote:
> [You don't often get email from pierre-loup.gosse=smile.fr@lists.openembedded.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> From: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>
> 
> Makes a clear distinction with --extra-partition-space flag.
> 
> Signed-off-by: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>
> 
> CC: Alexander Kanavin <alex.kanavin@gmail.com>
> CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>

Would you mind sending a patch for the documentation (yocto-docs) 
if/once this gets merged to rename the occurrences of this option and 
also notify users of the change in the appropriate migration manual?

Cheers,
Quentin

Re: [PATCH v3 2/2] wic: rename wks flag --extra-space to --extra-filesystem-space

[pierre-loup.gosse]

[-- Attachment #1: Type: text/plain, Size: 80 bytes --]

Hi Quentin,

Yes of course, I planned to send a patch for the documentation.

[-- Attachment #2: Type: text/html, Size: 134 bytes --]

Re: [PATCH v3 1/2] wic: add --extra-partition-space option to set unused space

[Mathieu Dubois-Briand]

On Mon Aug 11, 2025 at 3:48 PM CEST, pierre-loup.gosse wrote:
> From: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>
>
> By default, the content of the partition is filled by the filesystem
> without leaving any unused free space. The --extra-space flag adds
> extra space to the filesystem size, not to the partition.
>
> Unused free space after the filesystem can be useful for some cases,
> such as encrypting a partition at runtime.
> With --extra-partition-space 32M, we ensure that the last 32M of the
> partition is unused: this space does not contain filesystem data and
> can store the LUKS2 header.
>
> The implementation sets a difference between the partition and
> filesystem size:
>   - With --fixed-size, the extra part space is removed from the
>     filesystem size.
>   - Otherwise (with or without --size flag), the extra part space is
>     added to the partition size.
>
> Signed-off-by: Pierre-Loup GOSSE <pierre-loup.gosse@smile.fr>
>
> CC: Alexander Kanavin <alex.kanavin@gmail.com>
> CC: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
> ---

Hi Pierre-Loup,

Thanks for the new version.

It looks like some of the tests are failing on the autobuilder:

2025-08-12 11:56:06,941 - oe-selftest - INFO - 9: 73/92 537/635 (83.03s) (0 failed) (wic.Wic2.test_extra_partition_space)
2025-08-12 11:56:06,941 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", line 1375, in test_extra_partition_space
    self.assertNotEqual(p.status, 0, "wic exited successfully when an error was expected:\n%s" % p.output)
  File "/usr/lib/python3.12/unittest/case.py", line 894, in assertNotEqual
    raise self.failureException(msg)
AssertionError: 0 == 0 : wic exited successfully when an error was expected:
...
2025-08-12 12:40:46,950 - oe-selftest - INFO - FAIL: wic.Wic2.test_extra_partition_space (subunit.RemotedTestCase)

https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2285
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2135
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2064

Can you have a look at the issue please?

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH v3 1/2] wic: add --extra-partition-space option to set unused space

[pierre-loup.gosse]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

Hi Mathieu,

Thanks for your review.

The second part of the test passed in my local self-test build environment, but not in your autobuilder. It appears that the core-image-minimal filesystem sizes differ: the test expects a WicError if the filesystem exceeds 50 MB.

That said, I think this part of the test might be irrelevant. However, I’m curious to understand why it passed in my environment and not yours. I expected the built filesystem to be the same size in both environment. Did I overlook something when running my tests?

I’ll submit a v4 with the necessary changes.

Pierre-Loup

[-- Attachment #2: Type: text/html, Size: 771 bytes --]