Re: [OE-core] [PATCH v2] wic: do not ignore ROOTFS_SIZE if the rootfs is modified

["Mathieu Dubois-Briand" <mathieu.dubois-briand@bootlin.com>]

On Thu Apr 10, 2025 at 1:36 PM CEST, Trevor Woerner via lists.openembedded.org wrote:
> If the *.wks file contains a "--source rootfs" then
> lib/wic/plugins/source/rootfs.py will be invoked to generate (what is assumed
> to be) the rootfs partition. If the rootfs partition needs to be tweaked or
> modified, the "rootfs.py" plugin will make a copy of the filesystem and then
> perform the changes on that copy. In other words, if the "--source rootfs"
> line of the *.wks file also contains any of:
>
> 	--exclude-path
> 	--include-path
> 	--change-directory
> 	--use-label (i.e. modify etc/fstab)
>
> then the rootfs will be copied first, then the copy is modified.
>
> If, for example, the unmodified IMAGE_ROOTFS is:
>
> 	.../tmp/work/qemuarm64_secureboot-oe-linux/core-image-base/1.0/rootfs
>
> then the copy would be made at:
>
> 	.../tmp/work/qemuarm64_secureboot-oe-linux/core-image-base/1.0/tmp-wic/rootfs${LINENO}
>
> where ${LINENO} is the line number where this "--source rootfs" line appears
> in the *wks file.
>
> When it comes time to make an actual partition of a specific filesystem type,
> lib/wic/partition.py::prepare_rootfs() is called. It is in this function that
> wic figures out if any extra size needs to be added. The bitbake variable used
> to specify the ultimate rootfs size is ROOTFS_SIZE, and since this variable is
> only valid for the rootfs (and not any other partitions), the code also
> verifies that the partition being created is ${IMAGE_ROOTFS}:
>
> 	rsize_bb = get_bitbake_var('ROOTFS_SIZE')
> 	rdir = get_bitbake_var('IMAGE_ROOTFS')
> 	if rsize_bb and rdir == rootfs_dir:
> 		<use rsize_bb>
> 	else:
> 		<calculate the partition size using "du -ks $p">
>
> As noted above, if lib/wic/plugins/source/rootfs.py has made a copy, then the
> "rdir == rootfs_dir" clause will fail and the code will assume this partition
> is not a rootfs since the strings do not compare equal.
>
> Therefore, in order to determine if this is a rootfs, retain the existing
> "rdir == rootfs_dir" comparison, but also add another one to check whether or
> not this is a wic-generated copy of the rootfs.
>
> STEPS TO REPRODUCE:
> 	- start with the following *wks file:
> 		bootloader --ptable gpt
> 		part /boot --size=100M --active --fstype=ext4 --label boot
> 		part /     --source rootfs      --fstype=ext4 --label root
> 	- and the following extra variable in conf/local.conf:
> 		IMAGE_ROOTFS_EXTRA_SPACE = "500000"
> 	- build an image
> 	- run it in qemu
> 		$ runqemu slirp nographic serial
> 	- verify the root partition has extra space:
> 		root@qemuarm64-secureboot:~# df -h
> 		Filesystem                Size      Used Available Use% Mounted on
> 		/dev/root               721.5M     67.4M    600.6M  10% /
> 		devtmpfs                477.7M         0    477.7M   0% /dev
> 		tmpfs                    40.0K         0     40.0K   0% /mnt
> 		tmpfs                   489.3M     92.0K    489.2M   0% /run
> 		tmpfs                   489.3M     68.0K    489.2M   0% /var/volatile
> 		/dev/vda1               120.4M     19.9M     91.4M  18% /boot
> 	- modify the "/" line of the *wks file to be:
> 		part /     --source rootfs      --fstype=ext4 --label root --exclude-path boot/
> 	- build image
>
> 	when it fails:
> 		root@qemuarm64-secureboot:~# df -h
> 		Filesystem                Size      Used Available Use% Mounted on
> 		/dev/root                73.4M     41.9M     25.8M  62% /
> 		devtmpfs                477.7M         0    477.7M   0% /dev
> 		tmpfs                    40.0K         0     40.0K   0% /mnt
> 		tmpfs                   489.3M     92.0K    489.2M   0% /run
> 		tmpfs                   489.3M     68.0K    489.2M   0% /var/volatile
> 		/dev/vda1               120.4M     19.9M     91.4M  18% /boot
>
> 	after this fix:
> 		root@qemuarm64-secureboot:~# df -h
> 		Filesystem                Size      Used Available Use% Mounted on
> 		/dev/root               721.5M     47.4M    620.6M   7% /
> 		devtmpfs                477.7M         0    477.7M   0% /dev
> 		tmpfs                    40.0K         0     40.0K   0% /mnt
> 		tmpfs                   489.3M     92.0K    489.2M   0% /run
> 		tmpfs                   489.3M     68.0K    489.2M   0% /var/volatile
> 		/dev/vda1               120.4M     19.9M     91.4M  18% /boot
>
> Doing the math we see that the /boot partition is ~20MB and in the first image
> the / partition contains this ~20MB in addition to the rest of the rootfs.
> This ~20MB is completely wasted since it is used in the / partition, but then
> the /boot partition is mounted on top of it, making the /boot directory of /
> inaccessible. After the fix the / partition has an additional ~20MB since the
> /boot portion is excluded.
>
> Fixes [YOCTO #15555]
>
> Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> ---

Hi Trevor,

Thanks for the v2, but I believe we still have an issue. Maybe a missing dependency?

2025-04-13 19:18:05,421 - oe-selftest - INFO - FAIL: wic.Wic.test_exclude_path_with_extra_space (subunit.RemotedTestCase)
2025-04-13 19:18:05,421 - oe-selftest - INFO - ----------------------------------------------------------------------
2025-04-13 19:18:05,421 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 567, in test_exclude_path_with_extra_space
    size_of_root_partition = int(runCmd("wic ls %s" % wicout).output.split('\n')[2].split()[3])
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 214, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'wic ls /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2511849/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-20250413181535.wic' returned non-zero exit status 1:
ERROR: Can't find executable parted

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/1350

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com