* [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file
@ 2025-09-02 19:05 dani.barra25
2025-09-02 19:05 ` [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely dani.barra25
2025-09-04 15:58 ` [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file Mathieu Dubois-Briand
0 siblings, 2 replies; 20+ messages in thread
From: dani.barra25 @ 2025-09-02 19:05 UTC (permalink / raw)
To: openembedded-core; +Cc: hongxu.jia, trevor.woerner, Daniel Andrade
From: Daniel Andrade <dani.barra25@gmail.com>
Using `install` in the rootfs plugin forces fstab to be replaced entirely, meaning that even its Inodes will change, leading xattrs and SELinux context stored by pseudo not to be applied.
The fix just uses `cp` without preserving attributes from the temporary fstab since none of them are needed, just the content.
Same thing happens with the predefined mechanisms for ext4 and msdos. Using debugfs there is no way to replace contents while maintaining metadata, so the approach taken on the path was to remove the different fstab logic for those fstypes and also use the same modified cp command. Reviewing the builds I did it seems to work for all of the fstypes.
Another problem is that the timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`).
Since you are using that variable everywhere, it is not the same value as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` that is applied in every other file.
Daniel Andrade (1):
wic: Content of the temporary updated fstab should be copied into the
original not replacing it entirely.
meta/conf/bitbake.conf | 4 +++-
scripts/lib/wic/partition.py | 15 +--------------
scripts/lib/wic/plugins/source/rootfs.py | 4 ++--
3 files changed, 6 insertions(+), 17 deletions(-)
^ permalink raw reply [flat|nested] 20+ messages in thread* [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely. 2025-09-02 19:05 [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file dani.barra25 @ 2025-09-02 19:05 ` dani.barra25 2025-10-11 13:18 ` [OE-core] " Mathieu Dubois-Briand 2025-09-04 15:58 ` [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file Mathieu Dubois-Briand 1 sibling, 1 reply; 20+ messages in thread From: dani.barra25 @ 2025-09-02 19:05 UTC (permalink / raw) To: openembedded-core; +Cc: hongxu.jia, trevor.woerner, Daniel Andrade From: Daniel Andrade <dani.barra25@gmail.com> Fixes [15947] The current functionality to update fstab generates a new temporary fstab with the new partition configuration. However, this file does not retain any metadata being it a completely new file. Because of this, when the rootfs plugin under `poky/scripts/lib/wic/plugins/source/rootfs.py` copies the file, it overrides the original fstab metadata. The patch removes implementation of msdos/ext code for fstab since it is not possible to append content without rewriting the file to preserve metadata. The timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp SOURCE_DATE_EPOCH_FALLBACK. Since that variable is used everywhere, it is not the same value as REPRODUCIBLE_TIMESTAMP_ROOTFS under poky/meta/conf/bitbake.conf that is applied in every other file. Signed-off-by: Daniel Andrade <dani.barra25@gmail.com> --- meta/conf/bitbake.conf | 4 +++- scripts/lib/wic/partition.py | 15 +-------------- scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- 3 files changed, 6 insertions(+), 17 deletions(-) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index acf4e2d153..0581fc3564 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -684,8 +684,10 @@ export PYTHONHASHSEED = "0" export PERL_HASH_SEED = "0" export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}" # A SOURCE_DATE_EPOCH of '0' might be misinterpreted as no SDE +# If these 2 are misaligned, when generating fstab later on wic, the timestamps will never match because +# The repeatibility uses the REPRODUCIBLE_TIMESTAMP_ROOTFS while wic uses SOURCE_DATE_EPOCH (therefore its fallback) SOURCE_DATE_EPOCH_FALLBACK ??= "1302044400" -REPRODUCIBLE_TIMESTAMP_ROOTFS ??= "1520598896" +REPRODUCIBLE_TIMESTAMP_ROOTFS ??= "1302044400" ################################################################## # Settings used by bitbake-layers. diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index bf2c34d594..82d754835c 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -131,7 +131,7 @@ class Partition(): partition command parameters. """ self.updated_fstab_path = updated_fstab_path - if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"): + if self.updated_fstab_path: self.update_fstab_in_rootfs = True if not self.source: @@ -295,15 +295,6 @@ class Partition(): (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") - with open(debugfs_script_path, "w") as f: - f.write("cd etc\n") - f.write("rm fstab\n") - f.write("write %s fstab\n" % (self.updated_fstab_path)) - debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs) - exec_native_cmd(debugfs_cmd, native_sysroot) - mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) @@ -400,10 +391,6 @@ class Partition(): mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) exec_native_cmd(mcopy_cmd, native_sysroot) - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) - exec_native_cmd(mcopy_cmd, native_sysroot) - chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index e29f3a4c2f..3848af2a91 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -223,8 +223,8 @@ class RootfsPlugin(SourcePlugin): part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab")) if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update: fstab_path = os.path.join(new_rootfs, "etc/fstab") - # Assume that fstab should always be owned by root with fixed permissions - install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path) + # We dont want any metadata of the updated fstab, just the one that already existed + install_cmd = "cp --no-preserve=all %s %s" % (part.updated_fstab_path, fstab_path) if new_pseudo: pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) else: ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely. 2025-09-02 19:05 ` [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely dani.barra25 @ 2025-10-11 13:18 ` Mathieu Dubois-Briand 2025-10-17 11:04 ` dani.barra25 0 siblings, 1 reply; 20+ messages in thread From: Mathieu Dubois-Briand @ 2025-10-11 13:18 UTC (permalink / raw) To: dani.barra25, openembedded-core; +Cc: hongxu.jia, trevor.woerner On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via lists.openembedded.org wrote: > From: Daniel Andrade <dani.barra25@gmail.com> > > Fixes [15947] > The current functionality to update fstab generates a new temporary fstab with the new partition configuration. > However, this file does not retain any metadata being it a completely new file. > Because of this, when the rootfs plugin under `poky/scripts/lib/wic/plugins/source/rootfs.py` copies the file, it overrides the original fstab metadata. > The patch removes implementation of msdos/ext code for fstab since it is not possible to append content without rewriting the file to preserve metadata. > > The timestamp applied to fstab is not the same as every other file. > It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp SOURCE_DATE_EPOCH_FALLBACK. > Since that variable is used everywhere, it is not the same value as REPRODUCIBLE_TIMESTAMP_ROOTFS under poky/meta/conf/bitbake.conf that is applied in every other file. > > Signed-off-by: Daniel Andrade <dani.barra25@gmail.com> > --- Hi Daniel, I finally had a bit of time to have a deeper look at these changes and at the wic.Wic.test_no_fstab_update test. It turns out the test is correct and your patch is indeed introducing a regression. So our first assumption was the test was wrong, as changing the filesystem resulted in a test fail. But actually, if you change the filesystem in the wic command file, you also have to change the command extracting the file from the filesystem. E.g. for squashfs, you need to replace "debugfs -R cat" with "sqfscat". Now talking about your patch, my comments below. > diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py > index bf2c34d594..82d754835c 100644 > --- a/scripts/lib/wic/partition.py > +++ b/scripts/lib/wic/partition.py > @@ -131,7 +131,7 @@ class Partition(): > partition command parameters. > """ > self.updated_fstab_path = updated_fstab_path > - if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"): > + if self.updated_fstab_path: > self.update_fstab_in_rootfs = True > > if not self.source: > @@ -295,15 +295,6 @@ class Partition(): > (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) > > - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: > - debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") > - with open(debugfs_script_path, "w") as f: > - f.write("cd etc\n") > - f.write("rm fstab\n") > - f.write("write %s fstab\n" % (self.updated_fstab_path)) > - debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs) > - exec_native_cmd(debugfs_cmd, native_sysroot) > - > mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) > exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) > > @@ -400,10 +391,6 @@ class Partition(): > mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) > exec_native_cmd(mcopy_cmd, native_sysroot) > > - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: > - mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) > - exec_native_cmd(mcopy_cmd, native_sysroot) > - > chmod_cmd = "chmod 644 %s" % rootfs > exec_cmd(chmod_cmd) > So you are removing an optimisation on ext and fat filesystems. This might raise further questions from other reviewers, but if that's needed, why not. > diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py > index e29f3a4c2f..3848af2a91 100644 > --- a/scripts/lib/wic/plugins/source/rootfs.py > +++ b/scripts/lib/wic/plugins/source/rootfs.py > @@ -223,8 +223,8 @@ class RootfsPlugin(SourcePlugin): > part.has_fstab = os.path.exists(os.path.join(new_rootfs, "etc/fstab")) > if part.update_fstab_in_rootfs and part.has_fstab and not part.no_fstab_update: > fstab_path = os.path.join(new_rootfs, "etc/fstab") > - # Assume that fstab should always be owned by root with fixed permissions > - install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path) > + # We dont want any metadata of the updated fstab, just the one that already existed > + install_cmd = "cp --no-preserve=all %s %s" % (part.updated_fstab_path, fstab_path) Now this is what makes the test fail. So I understand the switch from install to cp, as it will reuse the same inode instead of creating a new one. But this is also what causes the test regression. When creating a filesystem image with a modified fstab, a copy of the filesystem content is first made, so file modifications only affect this specific image. But in order to gain time and disk space, this copy is made using hardlinks: see use of copyhardlinktree() in do_prepare_partition() from scripts/lib/wic/plugins/source/rootfs.py. So as cp will keep the same inode, it means you will modify the content on both the copy and the reference, and so all further generated image. So I'm sorry, we will have to find another way to implement this change. Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely. 2025-10-11 13:18 ` [OE-core] " Mathieu Dubois-Briand @ 2025-10-17 11:04 ` dani.barra25 2025-10-17 11:04 ` [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities dani.barra25 0 siblings, 1 reply; 20+ messages in thread From: dani.barra25 @ 2025-10-17 11:04 UTC (permalink / raw) To: openembedded-core, mathieu.dubois-briand Hi Mathieu, Thank you for the you time testing this thoroughly. After some test, I was able to provide a new alternative (attached to this message if everything goes well). For the performance part of ext* and msdos fstype I have nothing to counter argument. I did not find a way to add xattrs using those methods and, according to what seemed to be the natural way of the code design of wic, I let all the fstype flal under rootfs.py. Any other solution is welcome! For the second part, this time, instead of using cp, I let the install be like it was before. However, instead of running it under the pseudo environment, I executed it as the host machine. This way, I was able to recheck the database using pseudo, which will fail and therefore led the files.db to be rebuilt. This happens because he is able to detect the inodes of fstab entry on all the tables do not match and update them accordingly. This way we don't run the risk of modifying the original content of fstab under the normal yocto output files because the file is actually not the same. I also ran oe-selftest -r wic.Wic.test_no_fstab_update on a core-image-minimal image with the patch and it seemed to have passed. Best Regards, Daniel ^ permalink raw reply [flat|nested] 20+ messages in thread
* [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities 2025-10-17 11:04 ` dani.barra25 @ 2025-10-17 11:04 ` dani.barra25 2025-10-19 10:02 ` Mathieu Dubois-Briand 0 siblings, 1 reply; 20+ messages in thread From: dani.barra25 @ 2025-10-17 11:04 UTC (permalink / raw) To: openembedded-core, mathieu.dubois-briand; +Cc: Daniel Andrade From: Daniel Andrade <dani.barra25@gmail.com> The wic fstab generation capability was removing the xattrs of the /etc/fstab, possibly rendering the produced image unusable in some case (e.g.: SELinux enforced on a read-only rootfs). The file appears as unlabeled no matter the fstype chosen. To bypass this, some late procedures done on ext* and msdos fstype have been removed and all the fstab logic was passed to rootfs.py. On this one, the idea is to still use the install command, but in this case, executed outside the pseudo environment. After the file as sucessfully been place, the idea was to check again for inconsistencies in the database (now the fstab inode is different) and force it to rebuild with the correct inode, ensure fstab inode is correctly set on all the tables, including the xattrs one. Signed-off-by: Daniel Andrade <dani.barra25@gmail.com> --- scripts/lib/wic/partition.py | 15 +-------------- scripts/lib/wic/plugins/source/rootfs.py | 8 ++++---- 2 files changed, 5 insertions(+), 18 deletions(-) diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index bf2c34d594..82d754835c 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -131,7 +131,7 @@ class Partition(): partition command parameters. """ self.updated_fstab_path = updated_fstab_path - if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"): + if self.updated_fstab_path: self.update_fstab_in_rootfs = True if not self.source: @@ -295,15 +295,6 @@ class Partition(): (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") - with open(debugfs_script_path, "w") as f: - f.write("cd etc\n") - f.write("rm fstab\n") - f.write("write %s fstab\n" % (self.updated_fstab_path)) - debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs) - exec_native_cmd(debugfs_cmd, native_sysroot) - mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) @@ -400,10 +391,6 @@ class Partition(): mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) exec_native_cmd(mcopy_cmd, native_sysroot) - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) - exec_native_cmd(mcopy_cmd, native_sysroot) - chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index e29f3a4c2f..c0d9a28b91 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -225,11 +225,11 @@ class RootfsPlugin(SourcePlugin): fstab_path = os.path.join(new_rootfs, "etc/fstab") # Assume that fstab should always be owned by root with fixed permissions install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path) + exec_native_cmd(install_cmd, native_sysroot) if new_pseudo: - pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) - else: - pseudo = None - exec_native_cmd(install_cmd, native_sysroot, pseudo) + pseudo_cmd = "%s -B " % (cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo)) + exec_native_cmd(pseudo_cmd, native_sysroot) + part.prepare_rootfs(cr_workdir, oe_builddir, new_rootfs or part.rootfs_dir, native_sysroot, ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities 2025-10-17 11:04 ` [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities dani.barra25 @ 2025-10-19 10:02 ` Mathieu Dubois-Briand [not found] ` <20251019153922.27208-1-dani.barra25@gmail.com> 2025-10-19 15:41 ` [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities Daniel Andrade 0 siblings, 2 replies; 20+ messages in thread From: Mathieu Dubois-Briand @ 2025-10-19 10:02 UTC (permalink / raw) To: dani.barra25, openembedded-core On Fri Oct 17, 2025 at 1:04 PM CEST, dani.barra25 wrote: > From: Daniel Andrade <dani.barra25@gmail.com> > > The wic fstab generation capability was removing the xattrs of the /etc/fstab, possibly rendering the produced image unusable in some case (e.g.: SELinux enforced on a read-only rootfs). > The file appears as unlabeled no matter the fstype chosen. > To bypass this, some late procedures done on ext* and msdos fstype have been removed and all the fstab logic was passed to rootfs.py. > On this one, the idea is to still use the install command, but in this case, executed outside the pseudo environment. After the file as sucessfully been place, the idea was to check again for inconsistencies > in the database (now the fstab inode is different) and force it to rebuild with the correct inode, ensure fstab inode is correctly set on all the tables, including the xattrs one. > > Signed-off-by: Daniel Andrade <dani.barra25@gmail.com> > --- Hi Daniel, Thanks for the new version. It looks like this one is now breaking wic.Wic.test_exclude_path and wic.Wic.test_include_path tests: 2025-10-19 07:55:18,010 - oe-selftest - INFO - wic.Wic.test_exclude_path (subunit.RemotedTestCase) 2025-10-19 07:55:18,011 - oe-selftest - INFO - ... FAIL ... AssertionError: Command 'wic create temp.wks -e core-image-minimal -o /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/wic-tmp' returned non-zero exit status 1: ... ERROR: _exec_cmd: export PATH=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/sbin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/bin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/hosttools:$PATH;export PSEUDO_PREFIX=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr;export PSEUDO_LOCALSTATEDIR=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/wic-tmp/tmp.wic.dhlnwwia/pseudo1;export PSEUDO_PASSWD=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/wic-tmp/tmp.wic.dhlnwwia/rootfs1;export PSEUDO_NOSYMLINKEXP=1;/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo -B returned '1' instead of 0 output: PRAGMA journal_mode = OFF;: database is locked PRAGMA synchronous = OFF;: database is locked Failed: database is locked Error getting 'logs.db' database. PRAGMA journal_mode = OFF;: database is locked PRAGMA synchronous = OFF;: database is locked Failed: database is locked Error getting 'files.db' database. error during load from disk: database is locked pdb_files: database error. Couldn't start file list, can't scan. db cleanup for server shutdown, 07:55:17.956 memory-to-file backup complete, 07:55:17.956. db cleanup finished, 07:55:17.956 considering table files considering table xattrs considering table migrations https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2574 Can you have a look at these errors? Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 20+ messages in thread
[parent not found: <20251019153922.27208-1-dani.barra25@gmail.com>]
* [PATCH] Fstab xattrs: This update ensures that fstab xattrs are correctly updated without adding a lock to the database. [not found] ` <20251019153922.27208-1-dani.barra25@gmail.com> @ 2025-10-19 15:39 ` dani.barra25 2025-10-20 12:09 ` [OE-core] " Alexander Kanavin 0 siblings, 1 reply; 20+ messages in thread From: dani.barra25 @ 2025-10-19 15:39 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Andrade From: Daniel Andrade <dani.barra25@gmail.com> The process of repairing the database using -B flag of pseudo sometimes causes errors because the database lock file is present. Therefore, to fix it, first we ensure that any connection to files.db is closed using pseudo flag -S followed by the actual command to repair it after fstab was successfully updated. Signed-off-by: Daniel Andrade <dani.barra25@gmail.com> --- scripts/lib/wic/partition.py | 15 +-------------- scripts/lib/wic/plugins/source/rootfs.py | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 19 deletions(-) diff --git a/scripts/lib/wic/partition.py b/scripts/lib/wic/partition.py index bf2c34d594..82d754835c 100644 --- a/scripts/lib/wic/partition.py +++ b/scripts/lib/wic/partition.py @@ -131,7 +131,7 @@ class Partition(): partition command parameters. """ self.updated_fstab_path = updated_fstab_path - if self.updated_fstab_path and not (self.fstype.startswith("ext") or self.fstype == "msdos"): + if self.updated_fstab_path: self.update_fstab_in_rootfs = True if not self.source: @@ -295,15 +295,6 @@ class Partition(): (self.fstype, extraopts, rootfs, label_str, self.fsuuid, rootfs_dir) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - debugfs_script_path = os.path.join(cr_workdir, "debugfs_script") - with open(debugfs_script_path, "w") as f: - f.write("cd etc\n") - f.write("rm fstab\n") - f.write("write %s fstab\n" % (self.updated_fstab_path)) - debugfs_cmd = "debugfs -w -f %s %s" % (debugfs_script_path, rootfs) - exec_native_cmd(debugfs_cmd, native_sysroot) - mkfs_cmd = "fsck.%s -pvfD %s" % (self.fstype, rootfs) exec_native_cmd(mkfs_cmd, native_sysroot, pseudo=pseudo) @@ -400,10 +391,6 @@ class Partition(): mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (rootfs, rootfs_dir) exec_native_cmd(mcopy_cmd, native_sysroot) - if self.updated_fstab_path and self.has_fstab and not self.no_fstab_update: - mcopy_cmd = "mcopy -m -i %s %s ::/etc/fstab" % (rootfs, self.updated_fstab_path) - exec_native_cmd(mcopy_cmd, native_sysroot) - chmod_cmd = "chmod 644 %s" % rootfs exec_cmd(chmod_cmd) diff --git a/scripts/lib/wic/plugins/source/rootfs.py b/scripts/lib/wic/plugins/source/rootfs.py index e29f3a4c2f..09446baef2 100644 --- a/scripts/lib/wic/plugins/source/rootfs.py +++ b/scripts/lib/wic/plugins/source/rootfs.py @@ -225,12 +225,21 @@ class RootfsPlugin(SourcePlugin): fstab_path = os.path.join(new_rootfs, "etc/fstab") # Assume that fstab should always be owned by root with fixed permissions install_cmd = "install -m 0644 -p %s %s" % (part.updated_fstab_path, fstab_path) + exec_native_cmd(install_cmd, native_sysroot) + if new_pseudo: - pseudo = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) - else: - pseudo = None - exec_native_cmd(install_cmd, native_sysroot, pseudo) - + pseudo_prefix = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) + + # Shutdown any existing pseudo server + # This ensures no other processes are using the database + shutdown_cmd = "%s -S" % pseudo_prefix + exec_native_cmd(shutdown_cmd, native_sysroot) + + # Database is not locked anymore + # Repairs inodes related to fstab. This way xattrs and metadata is correctly applied to the new file + repair_cmd = "%s -B" % pseudo_prefix + exec_native_cmd(repair_cmd, native_sysroot) + part.prepare_rootfs(cr_workdir, oe_builddir, new_rootfs or part.rootfs_dir, native_sysroot, pseudo_dir = new_pseudo or pseudo_dir) ^ permalink raw reply related [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH] Fstab xattrs: This update ensures that fstab xattrs are correctly updated without adding a lock to the database. 2025-10-19 15:39 ` [PATCH] Fstab xattrs: This update ensures that fstab xattrs are correctly updated without adding a lock to the database dani.barra25 @ 2025-10-20 12:09 ` Alexander Kanavin 2025-10-20 13:18 ` Daniel Andrade 2025-10-23 13:04 ` Ross Burton 0 siblings, 2 replies; 20+ messages in thread From: Alexander Kanavin @ 2025-10-20 12:09 UTC (permalink / raw) To: dani.barra25; +Cc: openembedded-core On Sun, 19 Oct 2025 at 23:45, dani.barra25 via lists.openembedded.org <dani.barra25=gmail.com@lists.openembedded.org> wrote: > + pseudo_prefix = cls.__get_pseudo(native_sysroot, new_rootfs, new_pseudo) > + > + # Shutdown any existing pseudo server > + # This ensures no other processes are using the database > + shutdown_cmd = "%s -S" % pseudo_prefix > + exec_native_cmd(shutdown_cmd, native_sysroot) > + > + # Database is not locked anymore > + # Repairs inodes related to fstab. This way xattrs and metadata is correctly applied to the new file > + repair_cmd = "%s -B" % pseudo_prefix > + exec_native_cmd(repair_cmd, native_sysroot) > + Is it possible to describe how to reproduce and observe the issue that this patch aims to fix? This snippet makes me *very* nervous as pseudo is used by mulitple bitbake tasks in parallel, and doing shutdowns and repairs in the middle of it can blow up spectacularly. Alex ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH] Fstab xattrs: This update ensures that fstab xattrs are correctly updated without adding a lock to the database. 2025-10-20 12:09 ` [OE-core] " Alexander Kanavin @ 2025-10-20 13:18 ` Daniel Andrade 2025-10-23 13:04 ` Ross Burton 1 sibling, 0 replies; 20+ messages in thread From: Daniel Andrade @ 2025-10-20 13:18 UTC (permalink / raw) To: Alexander Kanavin; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 5062 bytes --] Hello Alex, Yes I can. Maybe I am doing something wrong since the patch is an alternative to past threads but somehow it gets separated without context. Starting from the beginning. My original patch aimed to fix a simple thing. No matter what fstype you were using, no xattrs from the build process were assigned to /etc/fstab if you had multiple partitions. And this was particularly problematic when you have read-only rootfs like squashfs and you need to have all the metadata available during build, like the SELinux file labels, since they cannot be set during boot time. This is because it creates a temp folder and then creates a new fstab that is later replaced in the image. The replacement is made via 2 options: 1. If it was ext* or msdos, it will fall under specific methods for those types that will open the already built image and remove the fstab and writing the new one, losing all the metadata of that file. 2. It falls under the "install" cmd of rootfs.py and this one was being executed inside the pseudo environment. This means the original entry of fstab on files db is completely replace with this new file and all the xattrs are gone To avoid this, my first patch removed the debugfs specifics of ext* and msdos, since those did not leveragy any "write only content" mechanism without replacing the actual file. Then, all the fstab modifications would lie under rootfs.py which seemed to be the common point for everything. On this I simply replaced "install" with "cp" since I only wanted to modify the contents. https://lists.openembedded.org/g/openembedded-core/topic/115043355 Later in the discussion, Mathieu correctly pointed out that it could not be used because it fails a test due to the usage of copyhardlinktree() for this temporary rootfs folder. This meant that I was not only modifying the temporary fstab, but also the original one, leading to a crash of wic.Wic.test_no_fstab_update. https://lists.openembedded.org/g/openembedded-core/topic/115043356 The second approach I took was looking at how pseudo behaved. This time, I decided to maintain the original install cmd, but instead of running it under fakeroot, I ran it on the host system. This meant that, if a later check of the db occurred, it would correctly detect an inode change on fstab and therefore it could fix it using the -B option. And that is what I did : https://lists.openembedded.org/g/openembedded-core/topic/115805391 I only neglected the fact that, in reality, other tests could have failed because of this. And, in fact, it happened. Because of this, tests regarding including and excluding paths were failing because the db was locked. That's when I reached the current patch alternative. I knew it could cause problems but honestly I never considered it would and when I run wic tests, I didn't see any failure. In reality, I was trying to replicate what was being done a few lines above with the options -B -m and -M. The difference is that pseudo doesn't have a flag to repair a single file, therefore this was the only solution found. A good alternative to this would be pseudo actually having a functionality to repair single file inodes, avoiding to completely check the whole database and therefore better management on this side. In the end this bug rendered all my images that enforced SELinux with read only rootfs (squashfs) useless and I'd assume I might not be the only one with this issue. Another thing I notice is that, for repeatability purposes, there are multiple variables being used to timestamp files. This is just a recommendation/question. Shouldn't all those variable timestamps be aligned also? In any case, if you have some other idea on how to approach this, let me know! Daniel Alexander Kanavin <alex.kanavin@gmail.com> escreveu (segunda, 20/10/2025 à(s) 13:09): > On Sun, 19 Oct 2025 at 23:45, dani.barra25 via lists.openembedded.org > <dani.barra25=gmail.com@lists.openembedded.org> wrote: > > > + pseudo_prefix = cls.__get_pseudo(native_sysroot, > new_rootfs, new_pseudo) > > + > > + # Shutdown any existing pseudo server > > + # This ensures no other processes are using the > database > > + shutdown_cmd = "%s -S" % pseudo_prefix > > + exec_native_cmd(shutdown_cmd, native_sysroot) > > + > > + # Database is not locked anymore > > + # Repairs inodes related to fstab. This way xattrs > and metadata is correctly applied to the new file > > + repair_cmd = "%s -B" % pseudo_prefix > > + exec_native_cmd(repair_cmd, native_sysroot) > > + > > Is it possible to describe how to reproduce and observe the issue that > this patch aims to fix? This snippet makes me *very* nervous as pseudo > is used by mulitple bitbake tasks in parallel, and doing shutdowns and > repairs in the middle of it can blow up spectacularly. > > Alex > [-- Attachment #2: Type: text/html, Size: 6277 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH] Fstab xattrs: This update ensures that fstab xattrs are correctly updated without adding a lock to the database. 2025-10-20 12:09 ` [OE-core] " Alexander Kanavin 2025-10-20 13:18 ` Daniel Andrade @ 2025-10-23 13:04 ` Ross Burton 1 sibling, 0 replies; 20+ messages in thread From: Ross Burton @ 2025-10-23 13:04 UTC (permalink / raw) To: dani.barra25@gmail.com Cc: openembedded-core@lists.openembedded.org, alex.kanavin@gmail.com On 20 Oct 2025, at 13:09, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote: > > Is it possible to describe how to reproduce and observe the issue that > this patch aims to fix? This snippet makes me *very* nervous as pseudo > is used by mulitple bitbake tasks in parallel, and doing shutdowns and > repairs in the middle of it can blow up spectacularly. Agreed. If you’re building a single image then yes, the wic tasks will typically happen at the end with nothing else in parallel. But this is not a guarantee and killing pseudo whilst other recipes might be mid-build is not acceptable. Ross ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities 2025-10-19 10:02 ` Mathieu Dubois-Briand [not found] ` <20251019153922.27208-1-dani.barra25@gmail.com> @ 2025-10-19 15:41 ` Daniel Andrade 1 sibling, 0 replies; 20+ messages in thread From: Daniel Andrade @ 2025-10-19 15:41 UTC (permalink / raw) To: Mathieu Dubois-Briand; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 4798 bytes --] Hi Mathieu, I did not notice that it was failing other tests. It seemed like the error was caused due to a lock file. To fix it, before issuing the repair of the database, I first shutdown any open connection using -S. Doing this I saw that the 3 tests passed and fstab was correctly labeled. The other thing I noticed was the different timestamps, but as I said a while ago, it was caused due to using SOURCE_DATE_EPOCH, REPRODUCIBLE_TIMESTAMP_ROOTFS, and SOURCE_DATE_EPOCH_FALLBACK. This should also be reviewed since it might compromise build repeatability. In any case, it was not the topic of the patch, I just wanted to state this in case someone checked the contents of the produced image. Best Regards, Daniel A domingo, 19/10/2025, 11:02, Mathieu Dubois-Briand < mathieu.dubois-briand@bootlin.com> escreveu: > On Fri Oct 17, 2025 at 1:04 PM CEST, dani.barra25 wrote: > > From: Daniel Andrade <dani.barra25@gmail.com> > > > > The wic fstab generation capability was removing the xattrs of the > /etc/fstab, possibly rendering the produced image unusable in some case > (e.g.: SELinux enforced on a read-only rootfs). > > The file appears as unlabeled no matter the fstype chosen. > > To bypass this, some late procedures done on ext* and msdos fstype have > been removed and all the fstab logic was passed to rootfs.py. > > On this one, the idea is to still use the install command, but in this > case, executed outside the pseudo environment. After the file as > sucessfully been place, the idea was to check again for inconsistencies > > in the database (now the fstab inode is different) and force it to > rebuild with the correct inode, ensure fstab inode is correctly set on all > the tables, including the xattrs one. > > > > Signed-off-by: Daniel Andrade <dani.barra25@gmail.com> > > --- > > Hi Daniel, > > Thanks for the new version. It looks like this one is now breaking > wic.Wic.test_exclude_path and wic.Wic.test_include_path tests: > > 2025-10-19 07:55:18,010 - oe-selftest - INFO - wic.Wic.test_exclude_path > (subunit.RemotedTestCase) > 2025-10-19 07:55:18,011 - oe-selftest - INFO - ... FAIL > ... > AssertionError: Command 'wic create temp.wks -e core-image-minimal -o > /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/wic-tmp' > returned non-zero exit status 1: > ... > ERROR: _exec_cmd: export > PATH=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/sbin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr/sbin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr/bin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/bin:/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/hosttools:$PATH;export > PSEUDO_PREFIX=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/work/x86-64-v3-poky-linux/wic-tools/1.0/recipe-sysroot-native/usr;export > PSEUDO_LOCALSTATEDIR=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/wic-tmp/tmp.wic.dhlnwwia/pseudo1;export > PSEUDO_PASSWD=/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/wic-tmp/tmp.wic.dhlnwwia/rootfs1;export > PSEUDO_NOSYMLINKEXP=1;/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3133304/tmp/sysroots-components/x86_64/pseudo-native/usr/bin/pseudo > -B returned '1' instead of 0 > output: PRAGMA journal_mode = OFF;: database is locked > PRAGMA synchronous = OFF;: database is locked > Failed: database is locked > Error getting 'logs.db' database. > PRAGMA journal_mode = OFF;: database is locked > PRAGMA synchronous = OFF;: database is locked > Failed: database is locked > Error getting 'files.db' database. > error during load from disk: database is locked > pdb_files: database error. > Couldn't start file list, can't scan. > db cleanup for server shutdown, 07:55:17.956 > memory-to-file backup complete, 07:55:17.956. > db cleanup finished, 07:55:17.956 > considering table files > considering table xattrs > considering table migrations > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2574 > > Can you have a look at these errors? > > Thanks, > Mathieu > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > [-- Attachment #2: Type: text/html, Size: 5861 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-02 19:05 [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file dani.barra25 2025-09-02 19:05 ` [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely dani.barra25 @ 2025-09-04 15:58 ` Mathieu Dubois-Briand 2025-09-11 16:56 ` Randy MacLeod 1 sibling, 1 reply; 20+ messages in thread From: Mathieu Dubois-Briand @ 2025-09-04 15:58 UTC (permalink / raw) To: dani.barra25, openembedded-core; +Cc: hongxu.jia, trevor.woerner On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via lists.openembedded.org wrote: > From: Daniel Andrade <dani.barra25@gmail.com> > > Using `install` in the rootfs plugin forces fstab to be replaced entirely, meaning that even its Inodes will change, leading xattrs and SELinux context stored by pseudo not to be applied. > The fix just uses `cp` without preserving attributes from the temporary fstab since none of them are needed, just the content. > Same thing happens with the predefined mechanisms for ext4 and msdos. Using debugfs there is no way to replace contents while maintaining metadata, so the approach taken on the path was to remove the different fstab logic for those fstypes and also use the same modified cp command. Reviewing the builds I did it seems to work for all of the fstypes. > > Another problem is that the timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). > Since you are using that variable everywhere, it is not the same value as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` that is applied in every other file. > > Daniel Andrade (1): > wic: Content of the temporary updated fstab should be copied into the > original not replacing it entirely. > > meta/conf/bitbake.conf | 4 +++- > scripts/lib/wic/partition.py | 15 +-------------- > scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- > 3 files changed, 6 insertions(+), 17 deletions(-) Hi Daniel, Thanks for your patch. It looks like it is breaking a test: 2025-09-04 15:28:11,112 - oe-selftest - INFO - wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL ... 2025-09-04 15:28:11,114 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", line 859, in test_no_fstab_update self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python3.12/unittest/case.py", line 1251, in assertMultiLineEqual self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.12/unittest/case.py", line 715, in fail raise self.failureException(msg) AssertionError: 'af3c087d6c9131735c8d1f270a226892' != '9edb8255abd217fdb20e118833afb856' - af3c087d6c9131735c8d1f270a226892 + 9edb8255abd217fdb20e118833afb856 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 Can you fix it please? Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-04 15:58 ` [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file Mathieu Dubois-Briand @ 2025-09-11 16:56 ` Randy MacLeod 2025-09-13 10:02 ` Daniel Andrade 0 siblings, 1 reply; 20+ messages in thread From: Randy MacLeod @ 2025-09-11 16:56 UTC (permalink / raw) To: mathieu.dubois-briand, dani.barra25, openembedded-core Cc: hongxu.jia, trevor.woerner [-- Attachment #1: Type: text/plain, Size: 3748 bytes --] On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via lists.openembedded.org wrote: > On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via lists.openembedded.org wrote: >> From: Daniel Andrade<dani.barra25@gmail.com> >> >> Using `install` in the rootfs plugin forces fstab to be replaced entirely, meaning that even its Inodes will change, leading xattrs and SELinux context stored by pseudo not to be applied. >> The fix just uses `cp` without preserving attributes from the temporary fstab since none of them are needed, just the content. >> Same thing happens with the predefined mechanisms for ext4 and msdos. Using debugfs there is no way to replace contents while maintaining metadata, so the approach taken on the path was to remove the different fstab logic for those fstypes and also use the same modified cp command. Reviewing the builds I did it seems to work for all of the fstypes. >> >> Another problem is that the timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). >> Since you are using that variable everywhere, it is not the same value as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` that is applied in every other file. >> >> Daniel Andrade (1): >> wic: Content of the temporary updated fstab should be copied into the >> original not replacing it entirely. >> >> meta/conf/bitbake.conf | 4 +++- >> scripts/lib/wic/partition.py | 15 +-------------- >> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- >> 3 files changed, 6 insertions(+), 17 deletions(-) > Hi Daniel, > > Thanks for your patch. > > It looks like it is breaking a test: > > 2025-09-04 15:28:11,112 - oe-selftest - INFO - wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) > 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL > ... > 2025-09-04 15:28:11,114 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", line 859, in test_no_fstab_update > self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) > File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual > assertion_func(first, second, msg=msg) > File "/usr/lib/python3.12/unittest/case.py", line 1251, in assertMultiLineEqual > self.fail(self._formatMessage(msg, standardMsg)) > File "/usr/lib/python3.12/unittest/case.py", line 715, in fail > raise self.failureException(msg) > AssertionError: 'af3c087d6c9131735c8d1f270a226892' != '9edb8255abd217fdb20e118833afb856' > - af3c087d6c9131735c8d1f270a226892 > + 9edb8255abd217fdb20e118833afb856 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 > > Can you fix it please? Ping? I think this is being tracked by: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 "WIC does not preserve metadata when updating fstab" Btw, we were just following the "Need Info" process during the bug review meeting so that's why I'm sending this email. ../Randy > > Thanks, > Mathieu > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#222962):https://lists.openembedded.org/g/openembedded-core/message/222962 > Mute This Topic:https://lists.openembedded.org/mt/115043355/3616765 > Group Owner:openembedded-core+owner@lists.openembedded.org > Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com] > -=-=-=-=-=-=-=-=-=-=-=- > -- # Randy MacLeod # Wind River Linux [-- Attachment #2: Type: text/html, Size: 5758 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-11 16:56 ` Randy MacLeod @ 2025-09-13 10:02 ` Daniel Andrade 2025-09-23 15:05 ` Daniel Andrade 0 siblings, 1 reply; 20+ messages in thread From: Daniel Andrade @ 2025-09-13 10:02 UTC (permalink / raw) To: Randy MacLeod, openembedded-core, mathieu.dubois-briand Cc: hongxu.jia, trevor.woerner [-- Attachment #1: Type: text/plain, Size: 3660 bytes --] Hello guys, sorry for the delay. I have been busy, but I hope to provide you with a fix in the next few days. Sorry about that, Daniel Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, 11/09/2025 à(s) 17:56): > On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via lists.openembedded.org > wrote: > > On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via lists.openembedded.org wrote: > > From: Daniel Andrade <dani.barra25@gmail.com> <dani.barra25@gmail.com> > > Using `install` in the rootfs plugin forces fstab to be replaced entirely, meaning that even its Inodes will change, leading xattrs and SELinux context stored by pseudo not to be applied. > The fix just uses `cp` without preserving attributes from the temporary fstab since none of them are needed, just the content. > Same thing happens with the predefined mechanisms for ext4 and msdos. Using debugfs there is no way to replace contents while maintaining metadata, so the approach taken on the path was to remove the different fstab logic for those fstypes and also use the same modified cp command. Reviewing the builds I did it seems to work for all of the fstypes. > > Another problem is that the timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). > Since you are using that variable everywhere, it is not the same value as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` that is applied in every other file. > > Daniel Andrade (1): > wic: Content of the temporary updated fstab should be copied into the > original not replacing it entirely. > > meta/conf/bitbake.conf | 4 +++- > scripts/lib/wic/partition.py | 15 +-------------- > scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- > 3 files changed, 6 insertions(+), 17 deletions(-) > > Hi Daniel, > > Thanks for your patch. > > It looks like it is breaking a test: > > 2025-09-04 15:28:11,112 - oe-selftest - INFO - wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) > 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL > ... > 2025-09-04 15:28:11,114 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", line 859, in test_no_fstab_update > self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) > File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual > assertion_func(first, second, msg=msg) > File "/usr/lib/python3.12/unittest/case.py", line 1251, in assertMultiLineEqual > self.fail(self._formatMessage(msg, standardMsg)) > File "/usr/lib/python3.12/unittest/case.py", line 715, in fail > raise self.failureException(msg) > AssertionError: 'af3c087d6c9131735c8d1f270a226892' != '9edb8255abd217fdb20e118833afb856' > - af3c087d6c9131735c8d1f270a226892 > + 9edb8255abd217fdb20e118833afb856 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 > > Can you fix it please? > > Ping? > > I think this is being tracked by: > > https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 > > "WIC does not preserve metadata when updating fstab" > > Btw, we were just following the "Need Info" process during the bug review > meeting > so that's why I'm sending this email. > > ../Randy > > Thanks, > Mathieu > > > > > > > > -- > # Randy MacLeod > # Wind River Linux > > [-- Attachment #2: Type: text/html, Size: 5295 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-13 10:02 ` Daniel Andrade @ 2025-09-23 15:05 ` Daniel Andrade 2025-09-23 18:44 ` Mathieu Dubois-Briand 0 siblings, 1 reply; 20+ messages in thread From: Daniel Andrade @ 2025-09-23 15:05 UTC (permalink / raw) To: Randy MacLeod, openembedded-core, mathieu.dubois-briand Cc: hongxu.jia, trevor.woerner [-- Attachment #1: Type: text/plain, Size: 4605 bytes --] Good Afternoon, I have been trying to figure it out and something seems off. The specific test you refer to may be hiding a problem or maybe I have my testbench compromised and I would like your help to test. The test `test_no_fstab_update ` verifies if the fstab was updated or not because on the wks file used to do it there is a --no-fstab-update. The thing is, I think this test is passing because what is forcing it to not update is a constraint that I removed previously on the patch (the one that checks if the system is ext* or msdos) and not because the flag is triggering its intended functionality. The crosscheck I did was modifying the fstype from ext4 to squashfs and indeed the test failed. Can someone also verify this? Daniel Daniel Andrade <dani.barra25@gmail.com> escreveu (sábado, 13/09/2025 à(s) 11:02): > Hello guys, sorry for the delay. > I have been busy, but I hope to provide you with a fix in the next few > days. > > Sorry about that, > Daniel > > Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, 11/09/2025 > à(s) 17:56): > >> On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via >> lists.openembedded.org wrote: >> >> On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via lists.openembedded.org wrote: >> >> From: Daniel Andrade <dani.barra25@gmail.com> <dani.barra25@gmail.com> >> >> Using `install` in the rootfs plugin forces fstab to be replaced entirely, meaning that even its Inodes will change, leading xattrs and SELinux context stored by pseudo not to be applied. >> The fix just uses `cp` without preserving attributes from the temporary fstab since none of them are needed, just the content. >> Same thing happens with the predefined mechanisms for ext4 and msdos. Using debugfs there is no way to replace contents while maintaining metadata, so the approach taken on the path was to remove the different fstab logic for those fstypes and also use the same modified cp command. Reviewing the builds I did it seems to work for all of the fstypes. >> >> Another problem is that the timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). >> Since you are using that variable everywhere, it is not the same value as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` that is applied in every other file. >> >> Daniel Andrade (1): >> wic: Content of the temporary updated fstab should be copied into the >> original not replacing it entirely. >> >> meta/conf/bitbake.conf | 4 +++- >> scripts/lib/wic/partition.py | 15 +-------------- >> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- >> 3 files changed, 6 insertions(+), 17 deletions(-) >> >> Hi Daniel, >> >> Thanks for your patch. >> >> It looks like it is breaking a test: >> >> 2025-09-04 15:28:11,112 - oe-selftest - INFO - wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) >> 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL >> ... >> 2025-09-04 15:28:11,114 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): >> File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", line 859, in test_no_fstab_update >> self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) >> File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual >> assertion_func(first, second, msg=msg) >> File "/usr/lib/python3.12/unittest/case.py", line 1251, in assertMultiLineEqual >> self.fail(self._formatMessage(msg, standardMsg)) >> File "/usr/lib/python3.12/unittest/case.py", line 715, in fail >> raise self.failureException(msg) >> AssertionError: 'af3c087d6c9131735c8d1f270a226892' != '9edb8255abd217fdb20e118833afb856' >> - af3c087d6c9131735c8d1f270a226892 >> + 9edb8255abd217fdb20e118833afb856 >> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 >> >> Can you fix it please? >> >> Ping? >> >> I think this is being tracked by: >> >> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 >> >> "WIC does not preserve metadata when updating fstab" >> >> Btw, we were just following the "Need Info" process during the bug review >> meeting >> so that's why I'm sending this email. >> >> ../Randy >> >> Thanks, >> Mathieu >> >> >> >> >> >> >> >> -- >> # Randy MacLeod >> # Wind River Linux >> >> [-- Attachment #2: Type: text/html, Size: 6649 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-23 15:05 ` Daniel Andrade @ 2025-09-23 18:44 ` Mathieu Dubois-Briand 2025-09-24 16:04 ` Daniel Andrade 0 siblings, 1 reply; 20+ messages in thread From: Mathieu Dubois-Briand @ 2025-09-23 18:44 UTC (permalink / raw) To: Daniel Andrade, Randy MacLeod, openembedded-core Cc: hongxu.jia, trevor.woerner Hi Daniel, Sorry, but I'm not sure to get your point. I ran the test locally, and it does fail. It also fails if I modify it to use squashfs instead of ext4. My reproduction procedure: git clone https://git.yoctoproject.org/poky-ci-archive -b autobuilder.yoctoproject.org/valkyrie/a-full-2320 . oe-init-build-env echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf oe-selftest -r wic.Wic.test_no_fstab_update Thanks, Mathieu On Tue Sep 23, 2025 at 5:05 PM CEST, Daniel Andrade wrote: > Good Afternoon, > > I have been trying to figure it out and something seems off. > The specific test you refer to may be hiding a problem or maybe I have my > testbench compromised and I would like your help to test. > The test `test_no_fstab_update ` verifies if the fstab was updated or not > because on the wks file used to do it there is a --no-fstab-update. The > thing is, I think this test is passing because what is forcing it to not > update is a constraint that I removed previously on the patch (the one that > checks if the system is ext* or msdos) and not because the flag is > triggering its intended functionality. > The crosscheck I did was modifying the fstype from ext4 to squashfs and > indeed the test failed. > Can someone also verify this? > > Daniel > > Daniel Andrade <dani.barra25@gmail.com> escreveu (sábado, 13/09/2025 à(s) > 11:02): > >> Hello guys, sorry for the delay. >> I have been busy, but I hope to provide you with a fix in the next few >> days. >> >> Sorry about that, >> Daniel >> >> Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, 11/09/2025 >> à(s) 17:56): >> >>> On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via >>> lists.openembedded.org wrote: >>> >>> On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via lists.openembedded.org wrote: >>> >>> From: Daniel Andrade <dani.barra25@gmail.com> <dani.barra25@gmail.com> >>> >>> Using `install` in the rootfs plugin forces fstab to be replaced entirely, meaning that even its Inodes will change, leading xattrs and SELinux context stored by pseudo not to be applied. >>> The fix just uses `cp` without preserving attributes from the temporary fstab since none of them are needed, just the content. >>> Same thing happens with the predefined mechanisms for ext4 and msdos. Using debugfs there is no way to replace contents while maintaining metadata, so the approach taken on the path was to remove the different fstab logic for those fstypes and also use the same modified cp command. Reviewing the builds I did it seems to work for all of the fstypes. >>> >>> Another problem is that the timestamp applied to fstab is not the same as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). >>> Since you are using that variable everywhere, it is not the same value as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` that is applied in every other file. >>> >>> Daniel Andrade (1): >>> wic: Content of the temporary updated fstab should be copied into the >>> original not replacing it entirely. >>> >>> meta/conf/bitbake.conf | 4 +++- >>> scripts/lib/wic/partition.py | 15 +-------------- >>> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- >>> 3 files changed, 6 insertions(+), 17 deletions(-) >>> >>> Hi Daniel, >>> >>> Thanks for your patch. >>> >>> It looks like it is breaking a test: >>> >>> 2025-09-04 15:28:11,112 - oe-selftest - INFO - wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) >>> 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL >>> ... >>> 2025-09-04 15:28:11,114 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): >>> File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", line 859, in test_no_fstab_update >>> self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) >>> File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual >>> assertion_func(first, second, msg=msg) >>> File "/usr/lib/python3.12/unittest/case.py", line 1251, in assertMultiLineEqual >>> self.fail(self._formatMessage(msg, standardMsg)) >>> File "/usr/lib/python3.12/unittest/case.py", line 715, in fail >>> raise self.failureException(msg) >>> AssertionError: 'af3c087d6c9131735c8d1f270a226892' != '9edb8255abd217fdb20e118833afb856' >>> - af3c087d6c9131735c8d1f270a226892 >>> + 9edb8255abd217fdb20e118833afb856 >>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 >>> >>> Can you fix it please? >>> >>> Ping? >>> >>> I think this is being tracked by: >>> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 >>> >>> "WIC does not preserve metadata when updating fstab" >>> >>> Btw, we were just following the "Need Info" process during the bug review >>> meeting >>> so that's why I'm sending this email. >>> >>> ../Randy >>> >>> Thanks, >>> Mathieu >>> >>> >>> >>> -=-=-=-=-=-=-=-=-=-=-=- >>> Links: You receive all messages sent to this group. >>> View/Reply Online (#222962): https://lists.openembedded.org/g/openembedded-core/message/222962 >>> Mute This Topic: https://lists.openembedded.org/mt/115043355/3616765 >>> Group Owner: openembedded-core+owner@lists.openembedded.org >>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com] >>> -=-=-=-=-=-=-=-=-=-=-=- >>> >>> >>> >>> -- >>> # Randy MacLeod >>> # Wind River Linux >>> >>> -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-23 18:44 ` Mathieu Dubois-Briand @ 2025-09-24 16:04 ` Daniel Andrade 2025-09-25 8:57 ` Mathieu Dubois-Briand 0 siblings, 1 reply; 20+ messages in thread From: Daniel Andrade @ 2025-09-24 16:04 UTC (permalink / raw) To: Mathieu Dubois-Briand Cc: Randy MacLeod, openembedded-core, hongxu.jia, trevor.woerner [-- Attachment #1: Type: text/plain, Size: 7509 bytes --] Hello Mathieu, Thank you for the quick answer. Ok, I will start from the beginning. The failing test is supposed to test `--no-fstab-update` flags on wks file. According to the understanding of the code, the flag is being passed across python scripts either as `self.no_fstab_update`, like in scripts/lib/wic/partition.py, or `part.no_fstab_update`, on scripts/lib/wic/plugins/source/rootfs.py. The issue that is arising from the `oe-selftest -r wic.Wic.test_no_fstab_update` is that it is not actually testing it, because somehow the assigning of the previous variables is not correct, which leads to a later assumption. The thing that is really doing the "no update" is the line 134 of `scripts/lib/wic/partition.py` where it excludes ext* and msdos partitions from falling in the common plugin for Rootfs. Because of this, I suggested someone to, in a clean and a different environment compared to mine, change the wks file used to `squashfs` fstype (or any other supported wic fstype) instead of `ext4` which indeed also made the test fail. In summary, the test is failing not because of the patch (even though maybe something can be improved or fixed later), but because it is just checking for ext4 partitions that fall in a different way of approaching fstab while the real mechanism is not behaving as desired. TLDR: The test is getting passed not because it is enforcing `--no-fstab-update` but because the partition is ext* or msdos. Changing it to squashfs, btrfs or any other wic supported partition, with or without the patch, will cause this to fail. This is an extra thing non-related with the proposed patch but that is preventing it to behave correctly as intended by the test. Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> escreveu (terça, 23/09/2025 à(s) 19:44): > Hi Daniel, > > Sorry, but I'm not sure to get your point. > > I ran the test locally, and it does fail. It also fails if I modify it > to use squashfs instead of ext4. > > My reproduction procedure: > git clone https://git.yoctoproject.org/poky-ci-archive -b > autobuilder.yoctoproject.org/valkyrie/a-full-2320 > . oe-init-build-env > echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf > oe-selftest -r wic.Wic.test_no_fstab_update > > Thanks, > Mathieu > > On Tue Sep 23, 2025 at 5:05 PM CEST, Daniel Andrade wrote: > > Good Afternoon, > > > > I have been trying to figure it out and something seems off. > > The specific test you refer to may be hiding a problem or maybe I have my > > testbench compromised and I would like your help to test. > > The test `test_no_fstab_update ` verifies if the fstab was updated or not > > because on the wks file used to do it there is a --no-fstab-update. The > > thing is, I think this test is passing because what is forcing it to not > > update is a constraint that I removed previously on the patch (the one > that > > checks if the system is ext* or msdos) and not because the flag is > > triggering its intended functionality. > > The crosscheck I did was modifying the fstype from ext4 to squashfs and > > indeed the test failed. > > Can someone also verify this? > > > > Daniel > > > > Daniel Andrade <dani.barra25@gmail.com> escreveu (sábado, 13/09/2025 > à(s) > > 11:02): > > > >> Hello guys, sorry for the delay. > >> I have been busy, but I hope to provide you with a fix in the next few > >> days. > >> > >> Sorry about that, > >> Daniel > >> > >> Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, > 11/09/2025 > >> à(s) 17:56): > >> > >>> On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via > >>> lists.openembedded.org wrote: > >>> > >>> On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via > lists.openembedded.org wrote: > >>> > >>> From: Daniel Andrade <dani.barra25@gmail.com> <dani.barra25@gmail.com> > >>> > >>> Using `install` in the rootfs plugin forces fstab to be replaced > entirely, meaning that even its Inodes will change, leading xattrs and > SELinux context stored by pseudo not to be applied. > >>> The fix just uses `cp` without preserving attributes from the > temporary fstab since none of them are needed, just the content. > >>> Same thing happens with the predefined mechanisms for ext4 and msdos. > Using debugfs there is no way to replace contents while maintaining > metadata, so the approach taken on the path was to remove the different > fstab logic for those fstypes and also use the same modified cp command. > Reviewing the builds I did it seems to work for all of the fstypes. > >>> > >>> Another problem is that the timestamp applied to fstab is not the same > as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to > the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). > >>> Since you are using that variable everywhere, it is not the same value > as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` > that is applied in every other file. > >>> > >>> Daniel Andrade (1): > >>> wic: Content of the temporary updated fstab should be copied into the > >>> original not replacing it entirely. > >>> > >>> meta/conf/bitbake.conf | 4 +++- > >>> scripts/lib/wic/partition.py | 15 +-------------- > >>> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- > >>> 3 files changed, 6 insertions(+), 17 deletions(-) > >>> > >>> Hi Daniel, > >>> > >>> Thanks for your patch. > >>> > >>> It looks like it is breaking a test: > >>> > >>> 2025-09-04 15:28:11,112 - oe-selftest - INFO - > wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) > >>> 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL > >>> ... > >>> 2025-09-04 15:28:11,114 - oe-selftest - INFO - > testtools.testresult.real._StringException: Traceback (most recent call > last): > >>> File > "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", > line 859, in test_no_fstab_update > >>> self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) > >>> File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual > >>> assertion_func(first, second, msg=msg) > >>> File "/usr/lib/python3.12/unittest/case.py", line 1251, in > assertMultiLineEqual > >>> self.fail(self._formatMessage(msg, standardMsg)) > >>> File "/usr/lib/python3.12/unittest/case.py", line 715, in fail > >>> raise self.failureException(msg) > >>> AssertionError: 'af3c087d6c9131735c8d1f270a226892' != > '9edb8255abd217fdb20e118833afb856' > >>> - af3c087d6c9131735c8d1f270a226892 > >>> + 9edb8255abd217fdb20e118833afb856 > >>> > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 > >>> > >>> Can you fix it please? > >>> > >>> Ping? > >>> > >>> I think this is being tracked by: > >>> > >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 > >>> > >>> "WIC does not preserve metadata when updating fstab" > >>> > >>> Btw, we were just following the "Need Info" process during the bug > review > >>> meeting > >>> so that's why I'm sending this email. > >>> > >>> ../Randy > >>> > >>> Thanks, > >>> Mathieu > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> # Randy MacLeod > >>> # Wind River Linux > >>> > >>> > > > > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > [-- Attachment #2: Type: text/html, Size: 9977 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-24 16:04 ` Daniel Andrade @ 2025-09-25 8:57 ` Mathieu Dubois-Briand 2025-09-25 11:11 ` Mathieu Dubois-Briand 0 siblings, 1 reply; 20+ messages in thread From: Mathieu Dubois-Briand @ 2025-09-25 8:57 UTC (permalink / raw) To: Daniel Andrade Cc: Randy MacLeod, openembedded-core, hongxu.jia, trevor.woerner OK! So I misunderstood your previous mail. I confirm that on master branch the test succeed, but fails if I change the filesystem to squashfs or btrfs. On a first glance, it looks like you are right, but I can't say I am experienced with wic internals. We might want a second opinion here. If the test is wrong, it has to be fixed. And so probably the tested code also have to be fixed. I this something you can do? Thanks, Mathieu On Wed Sep 24, 2025 at 6:04 PM CEST, Daniel Andrade wrote: > Hello Mathieu, > > Thank you for the quick answer. > Ok, I will start from the beginning. > > The failing test is supposed to test `--no-fstab-update` flags on wks file. > According to the understanding of the code, the flag is being passed across > python scripts either as `self.no_fstab_update`, like in > scripts/lib/wic/partition.py, or `part.no_fstab_update`, on > scripts/lib/wic/plugins/source/rootfs.py. > The issue that is arising from the `oe-selftest -r > wic.Wic.test_no_fstab_update` is that it is not actually testing it, > because somehow the assigning of the previous variables is not correct, > which leads to a later assumption. > The thing that is really doing the "no update" is the line 134 of > `scripts/lib/wic/partition.py` where it excludes ext* and msdos partitions > from falling in the common plugin for Rootfs. > Because of this, I suggested someone to, in a clean and a different > environment compared to mine, change the wks file used to `squashfs` fstype > (or any other supported wic fstype) instead of `ext4` which indeed also > made the test fail. > In summary, the test is failing not because of the patch (even though maybe > something can be improved or fixed later), but because it is just checking > for ext4 partitions that fall in a different way of approaching fstab while > the real mechanism is not behaving as desired. > > TLDR: The test is getting passed not because it is enforcing > `--no-fstab-update` but because the partition is ext* or msdos. Changing it > to squashfs, btrfs or any other wic supported partition, with or without > the patch, will cause this to fail. This is an extra thing non-related with > the proposed patch but that is preventing it to behave correctly as > intended by the test. > > Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> escreveu (terça, > 23/09/2025 à(s) 19:44): > >> Hi Daniel, >> >> Sorry, but I'm not sure to get your point. >> >> I ran the test locally, and it does fail. It also fails if I modify it >> to use squashfs instead of ext4. >> >> My reproduction procedure: >> git clone https://git.yoctoproject.org/poky-ci-archive -b >> autobuilder.yoctoproject.org/valkyrie/a-full-2320 >> . oe-init-build-env >> echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf >> oe-selftest -r wic.Wic.test_no_fstab_update >> >> Thanks, >> Mathieu >> >> On Tue Sep 23, 2025 at 5:05 PM CEST, Daniel Andrade wrote: >> > Good Afternoon, >> > >> > I have been trying to figure it out and something seems off. >> > The specific test you refer to may be hiding a problem or maybe I have my >> > testbench compromised and I would like your help to test. >> > The test `test_no_fstab_update ` verifies if the fstab was updated or not >> > because on the wks file used to do it there is a --no-fstab-update. The >> > thing is, I think this test is passing because what is forcing it to not >> > update is a constraint that I removed previously on the patch (the one >> that >> > checks if the system is ext* or msdos) and not because the flag is >> > triggering its intended functionality. >> > The crosscheck I did was modifying the fstype from ext4 to squashfs and >> > indeed the test failed. >> > Can someone also verify this? >> > >> > Daniel >> > >> > Daniel Andrade <dani.barra25@gmail.com> escreveu (sábado, 13/09/2025 >> à(s) >> > 11:02): >> > >> >> Hello guys, sorry for the delay. >> >> I have been busy, but I hope to provide you with a fix in the next few >> >> days. >> >> >> >> Sorry about that, >> >> Daniel >> >> >> >> Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, >> 11/09/2025 >> >> à(s) 17:56): >> >> >> >>> On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via >> >>> lists.openembedded.org wrote: >> >>> >> >>> On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via >> lists.openembedded.org wrote: >> >>> >> >>> From: Daniel Andrade <dani.barra25@gmail.com> <dani.barra25@gmail.com> >> >>> >> >>> Using `install` in the rootfs plugin forces fstab to be replaced >> entirely, meaning that even its Inodes will change, leading xattrs and >> SELinux context stored by pseudo not to be applied. >> >>> The fix just uses `cp` without preserving attributes from the >> temporary fstab since none of them are needed, just the content. >> >>> Same thing happens with the predefined mechanisms for ext4 and msdos. >> Using debugfs there is no way to replace contents while maintaining >> metadata, so the approach taken on the path was to remove the different >> fstab logic for those fstypes and also use the same modified cp command. >> Reviewing the builds I did it seems to work for all of the fstypes. >> >>> >> >>> Another problem is that the timestamp applied to fstab is not the same >> as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to >> the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). >> >>> Since you are using that variable everywhere, it is not the same value >> as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` >> that is applied in every other file. >> >>> >> >>> Daniel Andrade (1): >> >>> wic: Content of the temporary updated fstab should be copied into the >> >>> original not replacing it entirely. >> >>> >> >>> meta/conf/bitbake.conf | 4 +++- >> >>> scripts/lib/wic/partition.py | 15 +-------------- >> >>> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- >> >>> 3 files changed, 6 insertions(+), 17 deletions(-) >> >>> >> >>> Hi Daniel, >> >>> >> >>> Thanks for your patch. >> >>> >> >>> It looks like it is breaking a test: >> >>> >> >>> 2025-09-04 15:28:11,112 - oe-selftest - INFO - >> wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) >> >>> 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL >> >>> ... >> >>> 2025-09-04 15:28:11,114 - oe-selftest - INFO - >> testtools.testresult.real._StringException: Traceback (most recent call >> last): >> >>> File >> "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", >> line 859, in test_no_fstab_update >> >>> self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) >> >>> File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual >> >>> assertion_func(first, second, msg=msg) >> >>> File "/usr/lib/python3.12/unittest/case.py", line 1251, in >> assertMultiLineEqual >> >>> self.fail(self._formatMessage(msg, standardMsg)) >> >>> File "/usr/lib/python3.12/unittest/case.py", line 715, in fail >> >>> raise self.failureException(msg) >> >>> AssertionError: 'af3c087d6c9131735c8d1f270a226892' != >> '9edb8255abd217fdb20e118833afb856' >> >>> - af3c087d6c9131735c8d1f270a226892 >> >>> + 9edb8255abd217fdb20e118833afb856 >> >>> >> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 >> >>> >> >>> Can you fix it please? >> >>> >> >>> Ping? >> >>> >> >>> I think this is being tracked by: >> >>> >> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 >> >>> >> >>> "WIC does not preserve metadata when updating fstab" >> >>> >> >>> Btw, we were just following the "Need Info" process during the bug >> review >> >>> meeting >> >>> so that's why I'm sending this email. >> >>> >> >>> ../Randy >> >>> >> >>> Thanks, >> >>> Mathieu >> >>> >> >>> >> >>> >> >>> -=-=-=-=-=-=-=-=-=-=-=- >> >>> Links: You receive all messages sent to this group. >> >>> View/Reply Online (#222962): >> https://lists.openembedded.org/g/openembedded-core/message/222962 >> >>> Mute This Topic: https://lists.openembedded.org/mt/115043355/3616765 >> >>> Group Owner: openembedded-core+owner@lists.openembedded.org >> >>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub >> [randy.macleod@windriver.com] >> >>> -=-=-=-=-=-=-=-=-=-=-=- >> >>> >> >>> >> >>> >> >>> -- >> >>> # Randy MacLeod >> >>> # Wind River Linux >> >>> >> >>> >> >> >> >> >> -- >> Mathieu Dubois-Briand, Bootlin >> Embedded Linux and Kernel engineering >> https://bootlin.com >> >> -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-25 8:57 ` Mathieu Dubois-Briand @ 2025-09-25 11:11 ` Mathieu Dubois-Briand 2025-09-25 11:40 ` Daniel Andrade 0 siblings, 1 reply; 20+ messages in thread From: Mathieu Dubois-Briand @ 2025-09-25 11:11 UTC (permalink / raw) To: Daniel Andrade, Trevor Woerner Cc: Randy MacLeod, openembedded-core, hongxu.jia, trevor.woerner Hi Trevor, I'm not sure you have seen the following mail thread. It was suggested we seek for your advices during today patch review call. Do you have any opinion about this test, and if the behaviour we are seeing is expected? Thanks, Mathieu On Thu Sep 25, 2025 at 10:57 AM CEST, Mathieu Dubois-Briand wrote: > OK! So I misunderstood your previous mail. > > I confirm that on master branch the test succeed, but fails if I change > the filesystem to squashfs or btrfs. > > On a first glance, it looks like you are right, but I can't say I am > experienced with wic internals. We might want a second opinion here. > > If the test is wrong, it has to be fixed. And so probably the tested > code also have to be fixed. I this something you can do? > > Thanks, > Mathieu > > On Wed Sep 24, 2025 at 6:04 PM CEST, Daniel Andrade wrote: >> Hello Mathieu, >> >> Thank you for the quick answer. >> Ok, I will start from the beginning. >> >> The failing test is supposed to test `--no-fstab-update` flags on wks file. >> According to the understanding of the code, the flag is being passed across >> python scripts either as `self.no_fstab_update`, like in >> scripts/lib/wic/partition.py, or `part.no_fstab_update`, on >> scripts/lib/wic/plugins/source/rootfs.py. >> The issue that is arising from the `oe-selftest -r >> wic.Wic.test_no_fstab_update` is that it is not actually testing it, >> because somehow the assigning of the previous variables is not correct, >> which leads to a later assumption. >> The thing that is really doing the "no update" is the line 134 of >> `scripts/lib/wic/partition.py` where it excludes ext* and msdos partitions >> from falling in the common plugin for Rootfs. >> Because of this, I suggested someone to, in a clean and a different >> environment compared to mine, change the wks file used to `squashfs` fstype >> (or any other supported wic fstype) instead of `ext4` which indeed also >> made the test fail. >> In summary, the test is failing not because of the patch (even though maybe >> something can be improved or fixed later), but because it is just checking >> for ext4 partitions that fall in a different way of approaching fstab while >> the real mechanism is not behaving as desired. >> >> TLDR: The test is getting passed not because it is enforcing >> `--no-fstab-update` but because the partition is ext* or msdos. Changing it >> to squashfs, btrfs or any other wic supported partition, with or without >> the patch, will cause this to fail. This is an extra thing non-related with >> the proposed patch but that is preventing it to behave correctly as >> intended by the test. >> >> Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> escreveu (terça, >> 23/09/2025 à(s) 19:44): >> >>> Hi Daniel, >>> >>> Sorry, but I'm not sure to get your point. >>> >>> I ran the test locally, and it does fail. It also fails if I modify it >>> to use squashfs instead of ext4. >>> >>> My reproduction procedure: >>> git clone https://git.yoctoproject.org/poky-ci-archive -b >>> autobuilder.yoctoproject.org/valkyrie/a-full-2320 >>> . oe-init-build-env >>> echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf >>> oe-selftest -r wic.Wic.test_no_fstab_update >>> >>> Thanks, >>> Mathieu >>> >>> On Tue Sep 23, 2025 at 5:05 PM CEST, Daniel Andrade wrote: >>> > Good Afternoon, >>> > >>> > I have been trying to figure it out and something seems off. >>> > The specific test you refer to may be hiding a problem or maybe I have my >>> > testbench compromised and I would like your help to test. >>> > The test `test_no_fstab_update ` verifies if the fstab was updated or not >>> > because on the wks file used to do it there is a --no-fstab-update. The >>> > thing is, I think this test is passing because what is forcing it to not >>> > update is a constraint that I removed previously on the patch (the one >>> that >>> > checks if the system is ext* or msdos) and not because the flag is >>> > triggering its intended functionality. >>> > The crosscheck I did was modifying the fstype from ext4 to squashfs and >>> > indeed the test failed. >>> > Can someone also verify this? >>> > >>> > Daniel >>> > >>> > Daniel Andrade <dani.barra25@gmail.com> escreveu (sábado, 13/09/2025 >>> à(s) >>> > 11:02): >>> > >>> >> Hello guys, sorry for the delay. >>> >> I have been busy, but I hope to provide you with a fix in the next few >>> >> days. >>> >> >>> >> Sorry about that, >>> >> Daniel >>> >> >>> >> Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, >>> 11/09/2025 >>> >> à(s) 17:56): >>> >> >>> >>> On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via >>> >>> lists.openembedded.org wrote: >>> >>> >>> >>> On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via >>> lists.openembedded.org wrote: >>> >>> >>> >>> From: Daniel Andrade <dani.barra25@gmail.com> <dani.barra25@gmail.com> >>> >>> >>> >>> Using `install` in the rootfs plugin forces fstab to be replaced >>> entirely, meaning that even its Inodes will change, leading xattrs and >>> SELinux context stored by pseudo not to be applied. >>> >>> The fix just uses `cp` without preserving attributes from the >>> temporary fstab since none of them are needed, just the content. >>> >>> Same thing happens with the predefined mechanisms for ext4 and msdos. >>> Using debugfs there is no way to replace contents while maintaining >>> metadata, so the approach taken on the path was to remove the different >>> fstab logic for those fstypes and also use the same modified cp command. >>> Reviewing the builds I did it seems to work for all of the fstypes. >>> >>> >>> >>> Another problem is that the timestamp applied to fstab is not the same >>> as every other file. It seems like the `SOURCE_DATE_EPOCH` variable goes to >>> the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). >>> >>> Since you are using that variable everywhere, it is not the same value >>> as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` >>> that is applied in every other file. >>> >>> >>> >>> Daniel Andrade (1): >>> >>> wic: Content of the temporary updated fstab should be copied into the >>> >>> original not replacing it entirely. >>> >>> >>> >>> meta/conf/bitbake.conf | 4 +++- >>> >>> scripts/lib/wic/partition.py | 15 +-------------- >>> >>> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- >>> >>> 3 files changed, 6 insertions(+), 17 deletions(-) >>> >>> >>> >>> Hi Daniel, >>> >>> >>> >>> Thanks for your patch. >>> >>> >>> >>> It looks like it is breaking a test: >>> >>> >>> >>> 2025-09-04 15:28:11,112 - oe-selftest - INFO - >>> wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) >>> >>> 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL >>> >>> ... >>> >>> 2025-09-04 15:28:11,114 - oe-selftest - INFO - >>> testtools.testresult.real._StringException: Traceback (most recent call >>> last): >>> >>> File >>> "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", >>> line 859, in test_no_fstab_update >>> >>> self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) >>> >>> File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual >>> >>> assertion_func(first, second, msg=msg) >>> >>> File "/usr/lib/python3.12/unittest/case.py", line 1251, in >>> assertMultiLineEqual >>> >>> self.fail(self._formatMessage(msg, standardMsg)) >>> >>> File "/usr/lib/python3.12/unittest/case.py", line 715, in fail >>> >>> raise self.failureException(msg) >>> >>> AssertionError: 'af3c087d6c9131735c8d1f270a226892' != >>> '9edb8255abd217fdb20e118833afb856' >>> >>> - af3c087d6c9131735c8d1f270a226892 >>> >>> + 9edb8255abd217fdb20e118833afb856 >>> >>> >>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 >>> >>> >>> >>> Can you fix it please? >>> >>> >>> >>> Ping? >>> >>> >>> >>> I think this is being tracked by: >>> >>> >>> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 >>> >>> >>> >>> "WIC does not preserve metadata when updating fstab" >>> >>> >>> >>> Btw, we were just following the "Need Info" process during the bug >>> review >>> >>> meeting >>> >>> so that's why I'm sending this email. >>> >>> >>> >>> ../Randy >>> >>> >>> >>> Thanks, >>> >>> Mathieu >>> >>> >>> >>> >>> >>> >>> >>> -=-=-=-=-=-=-=-=-=-=-=- >>> >>> Links: You receive all messages sent to this group. >>> >>> View/Reply Online (#222962): >>> https://lists.openembedded.org/g/openembedded-core/message/222962 >>> >>> Mute This Topic: https://lists.openembedded.org/mt/115043355/3616765 >>> >>> Group Owner: openembedded-core+owner@lists.openembedded.org >>> >>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub >>> [randy.macleod@windriver.com] >>> >>> -=-=-=-=-=-=-=-=-=-=-=- >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> # Randy MacLeod >>> >>> # Wind River Linux >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> Mathieu Dubois-Briand, Bootlin >>> Embedded Linux and Kernel engineering >>> https://bootlin.com >>> >>> -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 20+ messages in thread
* Re: [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file 2025-09-25 11:11 ` Mathieu Dubois-Briand @ 2025-09-25 11:40 ` Daniel Andrade 0 siblings, 0 replies; 20+ messages in thread From: Daniel Andrade @ 2025-09-25 11:40 UTC (permalink / raw) To: Mathieu Dubois-Briand, Trevor Woerner Cc: Randy MacLeod, openembedded-core, hongxu.jia, trevor.woerner [-- Attachment #1: Type: text/plain, Size: 10009 bytes --] Hi Trevor and Mathieu, If, by any chance, you need more information, goal or objective with what I am trying to accomplish, or any other thing you might find useful, please let me know. I am available to provide more explanation, either via this thread or via a separate email or meeting. For the moment I will halt the research/fixing I was looking/doing until you provide further confirmation on the behaviour of that mechanism. Have a nice day, Daniel Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> escreveu (quinta, 25/09/2025 à(s) 12:11): > Hi Trevor, > > I'm not sure you have seen the following mail thread. It was suggested > we seek for your advices during today patch review call. > > Do you have any opinion about this test, and if the behaviour we are > seeing is expected? > > Thanks, > Mathieu > > On Thu Sep 25, 2025 at 10:57 AM CEST, Mathieu Dubois-Briand wrote: > > OK! So I misunderstood your previous mail. > > > > I confirm that on master branch the test succeed, but fails if I change > > the filesystem to squashfs or btrfs. > > > > On a first glance, it looks like you are right, but I can't say I am > > experienced with wic internals. We might want a second opinion here. > > > > If the test is wrong, it has to be fixed. And so probably the tested > > code also have to be fixed. I this something you can do? > > > > Thanks, > > Mathieu > > > > On Wed Sep 24, 2025 at 6:04 PM CEST, Daniel Andrade wrote: > >> Hello Mathieu, > >> > >> Thank you for the quick answer. > >> Ok, I will start from the beginning. > >> > >> The failing test is supposed to test `--no-fstab-update` flags on wks > file. > >> According to the understanding of the code, the flag is being passed > across > >> python scripts either as `self.no_fstab_update`, like in > >> scripts/lib/wic/partition.py, or `part.no_fstab_update`, on > >> scripts/lib/wic/plugins/source/rootfs.py. > >> The issue that is arising from the `oe-selftest -r > >> wic.Wic.test_no_fstab_update` is that it is not actually testing it, > >> because somehow the assigning of the previous variables is not correct, > >> which leads to a later assumption. > >> The thing that is really doing the "no update" is the line 134 of > >> `scripts/lib/wic/partition.py` where it excludes ext* and msdos > partitions > >> from falling in the common plugin for Rootfs. > >> Because of this, I suggested someone to, in a clean and a different > >> environment compared to mine, change the wks file used to `squashfs` > fstype > >> (or any other supported wic fstype) instead of `ext4` which indeed also > >> made the test fail. > >> In summary, the test is failing not because of the patch (even though > maybe > >> something can be improved or fixed later), but because it is just > checking > >> for ext4 partitions that fall in a different way of approaching fstab > while > >> the real mechanism is not behaving as desired. > >> > >> TLDR: The test is getting passed not because it is enforcing > >> `--no-fstab-update` but because the partition is ext* or msdos. > Changing it > >> to squashfs, btrfs or any other wic supported partition, with or without > >> the patch, will cause this to fail. This is an extra thing non-related > with > >> the proposed patch but that is preventing it to behave correctly as > >> intended by the test. > >> > >> Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> escreveu > (terça, > >> 23/09/2025 à(s) 19:44): > >> > >>> Hi Daniel, > >>> > >>> Sorry, but I'm not sure to get your point. > >>> > >>> I ran the test locally, and it does fail. It also fails if I modify it > >>> to use squashfs instead of ext4. > >>> > >>> My reproduction procedure: > >>> git clone https://git.yoctoproject.org/poky-ci-archive -b > >>> autobuilder.yoctoproject.org/valkyrie/a-full-2320 > >>> . oe-init-build-env > >>> echo 'SANITY_TESTED_DISTROS = ""' >> conf/local.conf > >>> oe-selftest -r wic.Wic.test_no_fstab_update > >>> > >>> Thanks, > >>> Mathieu > >>> > >>> On Tue Sep 23, 2025 at 5:05 PM CEST, Daniel Andrade wrote: > >>> > Good Afternoon, > >>> > > >>> > I have been trying to figure it out and something seems off. > >>> > The specific test you refer to may be hiding a problem or maybe I > have my > >>> > testbench compromised and I would like your help to test. > >>> > The test `test_no_fstab_update ` verifies if the fstab was updated > or not > >>> > because on the wks file used to do it there is a --no-fstab-update. > The > >>> > thing is, I think this test is passing because what is forcing it to > not > >>> > update is a constraint that I removed previously on the patch (the > one > >>> that > >>> > checks if the system is ext* or msdos) and not because the flag is > >>> > triggering its intended functionality. > >>> > The crosscheck I did was modifying the fstype from ext4 to squashfs > and > >>> > indeed the test failed. > >>> > Can someone also verify this? > >>> > > >>> > Daniel > >>> > > >>> > Daniel Andrade <dani.barra25@gmail.com> escreveu (sábado, 13/09/2025 > >>> à(s) > >>> > 11:02): > >>> > > >>> >> Hello guys, sorry for the delay. > >>> >> I have been busy, but I hope to provide you with a fix in the next > few > >>> >> days. > >>> >> > >>> >> Sorry about that, > >>> >> Daniel > >>> >> > >>> >> Randy MacLeod <randy.macleod@windriver.com> escreveu (quinta, > >>> 11/09/2025 > >>> >> à(s) 17:56): > >>> >> > >>> >>> On 2025-09-04 11:58 a.m., Mathieu Dubois-Briand via > >>> >>> lists.openembedded.org wrote: > >>> >>> > >>> >>> On Tue Sep 2, 2025 at 9:05 PM CEST, dani.barra25 via > >>> lists.openembedded.org wrote: > >>> >>> > >>> >>> From: Daniel Andrade <dani.barra25@gmail.com> < > dani.barra25@gmail.com> > >>> >>> > >>> >>> Using `install` in the rootfs plugin forces fstab to be replaced > >>> entirely, meaning that even its Inodes will change, leading xattrs and > >>> SELinux context stored by pseudo not to be applied. > >>> >>> The fix just uses `cp` without preserving attributes from the > >>> temporary fstab since none of them are needed, just the content. > >>> >>> Same thing happens with the predefined mechanisms for ext4 and > msdos. > >>> Using debugfs there is no way to replace contents while maintaining > >>> metadata, so the approach taken on the path was to remove the different > >>> fstab logic for those fstypes and also use the same modified cp > command. > >>> Reviewing the builds I did it seems to work for all of the fstypes. > >>> >>> > >>> >>> Another problem is that the timestamp applied to fstab is not the > same > >>> as every other file. It seems like the `SOURCE_DATE_EPOCH` variable > goes to > >>> the fallback timestamp (`SOURCE_DATE_EPOCH_FALLBACK`). > >>> >>> Since you are using that variable everywhere, it is not the same > value > >>> as ` REPRODUCIBLE_TIMESTAMP_ROOTFS` under `poky/meta/conf/bitbake.conf` > >>> that is applied in every other file. > >>> >>> > >>> >>> Daniel Andrade (1): > >>> >>> wic: Content of the temporary updated fstab should be copied > into the > >>> >>> original not replacing it entirely. > >>> >>> > >>> >>> meta/conf/bitbake.conf | 4 +++- > >>> >>> scripts/lib/wic/partition.py | 15 +-------------- > >>> >>> scripts/lib/wic/plugins/source/rootfs.py | 4 ++-- > >>> >>> 3 files changed, 6 insertions(+), 17 deletions(-) > >>> >>> > >>> >>> Hi Daniel, > >>> >>> > >>> >>> Thanks for your patch. > >>> >>> > >>> >>> It looks like it is breaking a test: > >>> >>> > >>> >>> 2025-09-04 15:28:11,112 - oe-selftest - INFO - > >>> wic.Wic.test_no_fstab_update (subunit.RemotedTestCase) > >>> >>> 2025-09-04 15:28:11,113 - oe-selftest - INFO - ... FAIL > >>> >>> ... > >>> >>> 2025-09-04 15:28:11,114 - oe-selftest - INFO - > >>> testtools.testresult.real._StringException: Traceback (most recent call > >>> last): > >>> >>> File > >>> > "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/lib/oeqa/selftest/cases/wic.py", > >>> line 859, in test_no_fstab_update > >>> >>> self.assertEqual(bf_fstab_md5sum, part_fstab_md5sum[1]) > >>> >>> File "/usr/lib/python3.12/unittest/case.py", line 885, in > assertEqual > >>> >>> assertion_func(first, second, msg=msg) > >>> >>> File "/usr/lib/python3.12/unittest/case.py", line 1251, in > >>> assertMultiLineEqual > >>> >>> self.fail(self._formatMessage(msg, standardMsg)) > >>> >>> File "/usr/lib/python3.12/unittest/case.py", line 715, in fail > >>> >>> raise self.failureException(msg) > >>> >>> AssertionError: 'af3c087d6c9131735c8d1f270a226892' != > >>> '9edb8255abd217fdb20e118833afb856' > >>> >>> - af3c087d6c9131735c8d1f270a226892 > >>> >>> + 9edb8255abd217fdb20e118833afb856 > >>> >>> > >>> > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/2412https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2268 > >>> >>> > >>> >>> Can you fix it please? > >>> >>> > >>> >>> Ping? > >>> >>> > >>> >>> I think this is being tracked by: > >>> >>> > >>> >>> https://bugzilla.yoctoproject.org/show_bug.cgi?id=15947 > >>> >>> > >>> >>> "WIC does not preserve metadata when updating fstab" > >>> >>> > >>> >>> Btw, we were just following the "Need Info" process during the bug > >>> review > >>> >>> meeting > >>> >>> so that's why I'm sending this email. > >>> >>> > >>> >>> ../Randy > >>> >>> > >>> >>> Thanks, > >>> >>> Mathieu > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> -- > >>> >>> # Randy MacLeod > >>> >>> # Wind River Linux > >>> >>> > >>> >>> > >>> > >>> > >>> > >>> > >>> -- > >>> Mathieu Dubois-Briand, Bootlin > >>> Embedded Linux and Kernel engineering > >>> https://bootlin.com > >>> > >>> > > > > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > [-- Attachment #2: Type: text/html, Size: 14562 bytes --] ^ permalink raw reply [flat|nested] 20+ messages in thread
end of thread, other threads:[~2025-10-23 13:05 UTC | newest]
Thread overview: 20+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-09-02 19:05 [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file dani.barra25
2025-09-02 19:05 ` [PATCH 1/1] wic: Content of the temporary updated fstab should be copied into the original not replacing it entirely dani.barra25
2025-10-11 13:18 ` [OE-core] " Mathieu Dubois-Briand
2025-10-17 11:04 ` dani.barra25
2025-10-17 11:04 ` [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities dani.barra25
2025-10-19 10:02 ` Mathieu Dubois-Briand
[not found] ` <20251019153922.27208-1-dani.barra25@gmail.com>
2025-10-19 15:39 ` [PATCH] Fstab xattrs: This update ensures that fstab xattrs are correctly updated without adding a lock to the database dani.barra25
2025-10-20 12:09 ` [OE-core] " Alexander Kanavin
2025-10-20 13:18 ` Daniel Andrade
2025-10-23 13:04 ` Ross Burton
2025-10-19 15:41 ` [PATCH 1/1] Fstab: Fix xattrs not being maintained on fstab file when using wic fstab update funtionalities Daniel Andrade
2025-09-04 15:58 ` [OE-core] [PATCH 0/1] wic: updated fstab does not preserve metadata of the original file Mathieu Dubois-Briand
2025-09-11 16:56 ` Randy MacLeod
2025-09-13 10:02 ` Daniel Andrade
2025-09-23 15:05 ` Daniel Andrade
2025-09-23 18:44 ` Mathieu Dubois-Briand
2025-09-24 16:04 ` Daniel Andrade
2025-09-25 8:57 ` Mathieu Dubois-Briand
2025-09-25 11:11 ` Mathieu Dubois-Briand
2025-09-25 11:40 ` Daniel Andrade
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox