Re: [OE-core] [PATCH v2] wic: do not ignore ROOTFS_SIZE if the rootfs is modified

[Trevor Woerner <twoerner@gmail.com>]

On Mon 2025-04-14 @ 01:40:34 PM, Mathieu Dubois-Briand wrote:
> On Thu Apr 10, 2025 at 1:36 PM CEST, Trevor Woerner via lists.openembedded.org wrote:
> > If the *.wks file contains a "--source rootfs" then
> > lib/wic/plugins/source/rootfs.py will be invoked to generate (what is assumed
> > to be) the rootfs partition. If the rootfs partition needs to be tweaked or
> > modified, the "rootfs.py" plugin will make a copy of the filesystem and then
> > perform the changes on that copy. In other words, if the "--source rootfs"
> > line of the *.wks file also contains any of:
> >
> > 	--exclude-path
> > 	--include-path
> > 	--change-directory
> > 	--use-label (i.e. modify etc/fstab)
> >
> > then the rootfs will be copied first, then the copy is modified.
> >
> > If, for example, the unmodified IMAGE_ROOTFS is:
> >
> > 	.../tmp/work/qemuarm64_secureboot-oe-linux/core-image-base/1.0/rootfs
> >
> > then the copy would be made at:
> >
> > 	.../tmp/work/qemuarm64_secureboot-oe-linux/core-image-base/1.0/tmp-wic/rootfs${LINENO}
> >
> > where ${LINENO} is the line number where this "--source rootfs" line appears
> > in the *wks file.
> >
> > When it comes time to make an actual partition of a specific filesystem type,
> > lib/wic/partition.py::prepare_rootfs() is called. It is in this function that
> > wic figures out if any extra size needs to be added. The bitbake variable used
> > to specify the ultimate rootfs size is ROOTFS_SIZE, and since this variable is
> > only valid for the rootfs (and not any other partitions), the code also
> > verifies that the partition being created is ${IMAGE_ROOTFS}:
> >
> > 	rsize_bb = get_bitbake_var('ROOTFS_SIZE')
> > 	rdir = get_bitbake_var('IMAGE_ROOTFS')
> > 	if rsize_bb and rdir == rootfs_dir:
> > 		<use rsize_bb>
> > 	else:
> > 		<calculate the partition size using "du -ks $p">
> >
> > As noted above, if lib/wic/plugins/source/rootfs.py has made a copy, then the
> > "rdir == rootfs_dir" clause will fail and the code will assume this partition
> > is not a rootfs since the strings do not compare equal.
> >
> > Therefore, in order to determine if this is a rootfs, retain the existing
> > "rdir == rootfs_dir" comparison, but also add another one to check whether or
> > not this is a wic-generated copy of the rootfs.
> >
> > STEPS TO REPRODUCE:
> > 	- start with the following *wks file:
> > 		bootloader --ptable gpt
> > 		part /boot --size=100M --active --fstype=ext4 --label boot
> > 		part /     --source rootfs      --fstype=ext4 --label root
> > 	- and the following extra variable in conf/local.conf:
> > 		IMAGE_ROOTFS_EXTRA_SPACE = "500000"
> > 	- build an image
> > 	- run it in qemu
> > 		$ runqemu slirp nographic serial
> > 	- verify the root partition has extra space:
> > 		root@qemuarm64-secureboot:~# df -h
> > 		Filesystem                Size      Used Available Use% Mounted on
> > 		/dev/root               721.5M     67.4M    600.6M  10% /
> > 		devtmpfs                477.7M         0    477.7M   0% /dev
> > 		tmpfs                    40.0K         0     40.0K   0% /mnt
> > 		tmpfs                   489.3M     92.0K    489.2M   0% /run
> > 		tmpfs                   489.3M     68.0K    489.2M   0% /var/volatile
> > 		/dev/vda1               120.4M     19.9M     91.4M  18% /boot
> > 	- modify the "/" line of the *wks file to be:
> > 		part /     --source rootfs      --fstype=ext4 --label root --exclude-path boot/
> > 	- build image
> >
> > 	when it fails:
> > 		root@qemuarm64-secureboot:~# df -h
> > 		Filesystem                Size      Used Available Use% Mounted on
> > 		/dev/root                73.4M     41.9M     25.8M  62% /
> > 		devtmpfs                477.7M         0    477.7M   0% /dev
> > 		tmpfs                    40.0K         0     40.0K   0% /mnt
> > 		tmpfs                   489.3M     92.0K    489.2M   0% /run
> > 		tmpfs                   489.3M     68.0K    489.2M   0% /var/volatile
> > 		/dev/vda1               120.4M     19.9M     91.4M  18% /boot
> >
> > 	after this fix:
> > 		root@qemuarm64-secureboot:~# df -h
> > 		Filesystem                Size      Used Available Use% Mounted on
> > 		/dev/root               721.5M     47.4M    620.6M   7% /
> > 		devtmpfs                477.7M         0    477.7M   0% /dev
> > 		tmpfs                    40.0K         0     40.0K   0% /mnt
> > 		tmpfs                   489.3M     92.0K    489.2M   0% /run
> > 		tmpfs                   489.3M     68.0K    489.2M   0% /var/volatile
> > 		/dev/vda1               120.4M     19.9M     91.4M  18% /boot
> >
> > Doing the math we see that the /boot partition is ~20MB and in the first image
> > the / partition contains this ~20MB in addition to the rest of the rootfs.
> > This ~20MB is completely wasted since it is used in the / partition, but then
> > the /boot partition is mounted on top of it, making the /boot directory of /
> > inaccessible. After the fix the / partition has an additional ~20MB since the
> > /boot portion is excluded.
> >
> > Fixes [YOCTO #15555]
> >
> > Signed-off-by: Trevor Woerner <twoerner@gmail.com>
> > ---
> 
> Hi Trevor,
> 
> Thanks for the v2, but I believe we still have an issue. Maybe a missing dependency?
> 
> 2025-04-13 19:18:05,421 - oe-selftest - INFO - FAIL: wic.Wic.test_exclude_path_with_extra_space (subunit.RemotedTestCase)
> 2025-04-13 19:18:05,421 - oe-selftest - INFO - ----------------------------------------------------------------------
> 2025-04-13 19:18:05,421 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
>   File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 567, in test_exclude_path_with_extra_space
>     size_of_root_partition = int(runCmd("wic ls %s" % wicout).output.split('\n')[2].split()[3])
>                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 214, in runCmd
>     raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
> AssertionError: Command 'wic ls /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2511849/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.rootfs-20250413181535.wic' returned non-zero exit status 1:
> ERROR: Can't find executable parted

Ouch, this might end up taking a couple of iterations. I don't want to blindly
install every wic native dependency, but I'm would be just guessing which
native utilities the buildbots have available to run the "wic ls" command.

Thanks for the feedback!