[isar-cip-core][PATCH 0/3] Added support for rootfs-overlay (for development).

[isar-cip-core][PATCH 0/3] Added support for rootfs-overlay (for development).

[alexander.heinisch]

From: Alexander Heinisch <alexander.heinisch@siemens.com>

Updateable images based on isar-cip-core come with immutable rootfs
out of the box. During development this oftentimes comes with the penalty
of having to either rebuild images with additional files, packages aso.
to accomplish development and debugging tasks or derive from the production
image in a way that some fundamental features like verity, squashfs, swupdate, aso.
are not part of such.

Both cases are far from optimal. With this patch series we want to share
our attempt to streamline development by staying as close as possible to
our production image by overlaying the squashfs or erofs based ro filesystem
with an overlay. To make changes redundant accross reboots, we decided
to use a persistent storage option in favour of a tmpfs based approach.
(Although, technically there is no limitation on doing so)

Keep in mind, support for the root overlay is thought as a development
feature! Thus, it is not thought for production, as it invalidates many
of the properties we have from ro-filesystems. Also, keep in mind that
changes on the overlay may partially hide updates or lead to "unexpected"
results after applying updates.

Alexander Heinisch (3):
  Removed "ro" option from read-only-rootfs's fstab.
  Fix return value handling on filesystem check
  Added support for rootfs-overlay.

 classes/read-only-rootfs.bbclass              |  2 +-
 .../files/local-bottom.tmpl                   | 97 ++++++++++++++++---
 .../initramfs-overlay-hook_0.3.bb             |  2 +-
 3 files changed, 87 insertions(+), 14 deletions(-)

-- 
2.39.5

[isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[alexander.heinisch]

From: Alexander Heinisch <alexander.heinisch@siemens.com>

Since we only support erofs and squashfs this option is not needed
any more. Further, it causes potential overlay rootfs variants
to be remounted read-only.

Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
---
 classes/read-only-rootfs.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only-rootfs.bbclass
index 35a3ab3..83ddc33 100644
--- a/classes/read-only-rootfs.bbclass
+++ b/classes/read-only-rootfs.bbclass
@@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
 image_configure_fstab() {
     sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF
 # Begin /etc/fstab
-/dev/root	/		auto		defaults,ro			0	0
+/dev/root	/		auto		defaults			0	0
 LABEL=var	/var		auto		defaults			0	0
 proc		/proc		proc		nosuid,noexec,nodev		0	0
 sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
-- 
2.39.5

[isar-cip-core][PATCH 2/3] Fix return value handling on filesystem check

[alexander.heinisch]

From: Alexander Heinisch <alexander.heinisch@siemens.com>

Detected errors during the filesystem check caused the script to
abort. This is caused by the `set -e` in the script-header.tmpl
in isar/meta/recipes-initramfs/initramfs-hook/files/script-header.tmpl

Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
---
 .../initramfs-overlay-hook/files/local-bottom.tmpl           | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
index 1087c3f..71cc63c 100644
--- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
+++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
@@ -23,8 +23,9 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 
 	case $partition_fstype in
 	ext*)
-		e2fsck -p -f "$ovl_partition_device"
-		fsck_ret="$?"
+		fsck_ret=0
+		e2fsck -p -f "$ovl_partition_device" || fsck_ret="$?"
+
 		# e2fsck returns a 1 in case of repairing the file system
 		# https://man7.org/linux/man-pages/man8/e2fsck.8.html#EXIT_CODE
 		if [ "$fsck_ret" -gt "1" ]  && [ -x "$ovl_recovery_script" ]; then
-- 
2.39.5

[isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[alexander.heinisch]

From: Alexander Heinisch <alexander.heinisch@siemens.com>

In addition to the overlays for e.g. /etc we now support persistent
overlays for rootfs as well.
To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"

This helps when developing or debugging an existing image, while
still maintaining being as close to the prod image as possible.
That way all image features like swupdate / verity / aso. remain
intact with the sole exeception of having the possibility to
work on a rw rootfs.
Bare in mind while this comes in handy for debugging or developing
new features, such feature has to be used with care. e.g. when
updating the rootfs, overwritten files from the overlay could
potentially cause unexpected results!

Compared to typical overlays we have to move the storage device used
for persistence off the previous rootfs. Otherwise, the recursion in
the overlay setup would cause panics when finally switching root.
In addition to that we move all mountpoints setup prior to overlaying
to the new overlayed root.

Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
---
 .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
 .../initramfs-overlay-hook_0.3.bb             |  2 +-
 2 files changed, 83 insertions(+), 11 deletions(-)

diff --git a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
index 71cc63c..a17e4a4 100644
--- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
+++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
@@ -6,14 +6,19 @@
 # Authors:
 #  Jan Kiszka <jan.kiszka@siemens.com>
 #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
+#  Alexander Heinisch <alexander.heinisch@siemens.com>
 #
 
 ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
 ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
 
-root_mount_storage=${rootmnt}${ovl_storage_path}
+root_mount_storage=${ovl_storage_path}
 storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
 
+# Setup mountpoint for persistent storage
+# Either, it already got setup by a previous initramfs script,
+# then we have to move it from $rootmnt to $storage_mount_point or it has not
+# been setup yet, then check it and mount it under $storage_mount_point
 if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
 	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
@@ -21,6 +26,8 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 
 	partition_fstype=$(get_fstype "${ovl_partition_device}")
 
+	# FS checks should be moved to a separate hook / or at least initramfs lib
+	# so generic functionality can be reused here!
 	case $partition_fstype in
 	ext*)
 		fsck_ret=0
@@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
 		fi
 		;;
 	esac
+
+	# Mount overlay device to initramfs - will be moved to rootfs at the end
+	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
 	if ! mount -t ${partition_fstype} \
 		 -o ${ovl_mount_option} \
 		 ${ovl_partition_device} \
-		 ${rootmnt}${storage_mount_point}; then
+		 ${storage_mount_point}; then
 		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
 	fi
+else
+	# If the storage device is already mounted on the final rootfs
+	# move it to the initramfs. Otherwise, we cannot move back the
+	# storage mount to the resulting rootfs in the end.
+
+	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
+	mount -n -o move ${rootmnt}${storage_mount_point} ${storage_mount_point}
 fi
 
+[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
+
 for ovl_lower_dir in ${ovl_lower_dirs}; do
-	# remove the first '/' and replace all '/' with '_'
-	# on variable $ovl_lower_dir
-	work_dir_name=$(awk -v var=$ovl_lower_dir \
-		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
+	root_moved=false
+
+	if [ $ovl_lower_dir != "/" ]; then
+		log_begin_msg "Mounting overlay $ovl_lower_dir"
+
+		# remove the first '/' and replace all '/' with '_'
+		# on variable $ovl_lower_dir
+		work_dir_name=$(awk -v var=$ovl_lower_dir \
+			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
+
+		lower_dir=${rootmnt}${ovl_lower_dir}
+		upper_dir=${root_mount_storage}${ovl_lower_dir}
+		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
+		mount_dir=${lower_dir}
+	else
+		log_begin_msg "Mounting root overlay"
+
+		# When overlaying root we need to move the mountpoint
+		# from ${rootmnt} to a dedicated mountpoint inside
+		# the storage dir to prevent a "recursion" within
+		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
+		# While this is not a limitation of overlay itself
+		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
+		# outside the tree of new root which panics when switching root
+
+		# create lower dir to move ro-rootfs to
+		lower_dir=${root_mount_storage}/lower-root
+		mkdir -p ${lower_dir}
+
+		# make the readonly root available
+		mount -n -o move ${rootmnt} ${lower_dir}
+		root_moved=true
 
-	lower_dir=${rootmnt}${ovl_lower_dir}
-	upper_dir=${root_mount_storage}${ovl_lower_dir}
-	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
+		upper_dir=${root_mount_storage}/root
+		work_dir=${root_mount_storage}/.root-atomic
+
+		mount_dir=${rootmnt}
+	fi
 
 	mkdir -p ${upper_dir}
 	mkdir -p ${work_dir}
 
+	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
 	if ! mount -t overlay \
 		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
-		   overlay ${lower_dir}; then
+		   overlay ${mount_dir}; then
 		panic "Can't mount overlay for '$ovl_lower_dir' !"
 	fi
+
+	if ${root_moved}; then
+		# Loop over all mountpoints already setup before we established the root overlay
+		# and move them to the resulting overlayed rootfs
+		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
+		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
+			while read -r old_mountpoint
+			do
+				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
+
+				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
+				mount -n -o move ${old_mountpoint} ${new_mountpoint}
+			done
+	fi
+
+	log_end_msg
 done
 
+# Finally, move the overlay workdir from initramfs to the final rootfs
+[ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
+mount -n -o move ${storage_mount_point} ${rootmnt}${storage_mount_point}
+
+log_success_msg "Overlays setup successfully"
+
 exit 0
diff --git a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
index ec7f85b..af610e5 100644
--- a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
+++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
@@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
 DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
 
 HOOK_ADD_MODULES = "overlay"
-HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
+HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
 
 SCRIPT_PREREQ="crypt"
 
-- 
2.39.5

Re: [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Quirin Gylstorff]

On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
> From: Alexander Heinisch <alexander.heinisch@siemens.com>
> 
> In addition to the overlays for e.g. /etc we now support persistent
> overlays for rootfs as well.
> To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
> or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> 

I would like to avoid that anti pattern of creating an overlay over the 
complete rootfs. What are the debug case you need a full overlay?

Quirin

> This helps when developing or debugging an existing image, while
> still maintaining being as close to the prod image as possible.
> That way all image features like swupdate / verity / aso. remain
> intact with the sole exeception of having the possibility to
> work on a rw rootfs.
> Bare in mind while this comes in handy for debugging or developing
> new features, such feature has to be used with care. e.g. when
> updating the rootfs, overwritten files from the overlay could
> potentially cause unexpected results!
> 
> Compared to typical overlays we have to move the storage device used
> for persistence off the previous rootfs. Otherwise, the recursion in
> the overlay setup would cause panics when finally switching root.
> In addition to that we move all mountpoints setup prior to overlaying
> to the new overlayed root.
> 
> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
> ---
>   .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
>   .../initramfs-overlay-hook_0.3.bb             |  2 +-
>   2 files changed, 83 insertions(+), 11 deletions(-)
> 
> diff --git a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> index 71cc63c..a17e4a4 100644
> --- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> +++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> @@ -6,14 +6,19 @@
>   # Authors:
>   #  Jan Kiszka <jan.kiszka@siemens.com>
>   #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
> +#  Alexander Heinisch <alexander.heinisch@siemens.com>
>   #
>   
>   ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
>   ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
>   
> -root_mount_storage=${rootmnt}${ovl_storage_path}
> +root_mount_storage=${ovl_storage_path}
>   storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
>   
> +# Setup mountpoint for persistent storage
> +# Either, it already got setup by a previous initramfs script,
> +# then we have to move it from $rootmnt to $storage_mount_point or it has not
> +# been setup yet, then check it and mount it under $storage_mount_point
>   if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>   	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
>   	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
> @@ -21,6 +26,8 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>   
>   	partition_fstype=$(get_fstype "${ovl_partition_device}")
>   
> +	# FS checks should be moved to a separate hook / or at least initramfs lib
> +	# so generic functionality can be reused here!
>   	case $partition_fstype in
>   	ext*)
>   		fsck_ret=0
> @@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>   		fi
>   		;;
>   	esac
> +
> +	# Mount overlay device to initramfs - will be moved to rootfs at the end
> +	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
>   	if ! mount -t ${partition_fstype} \
>   		 -o ${ovl_mount_option} \
>   		 ${ovl_partition_device} \
> -		 ${rootmnt}${storage_mount_point}; then
> +		 ${storage_mount_point}; then
>   		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
>   	fi
> +else
> +	# If the storage device is already mounted on the final rootfs
> +	# move it to the initramfs. Otherwise, we cannot move back the
> +	# storage mount to the resulting rootfs in the end.
> +
> +	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
> +	mount -n -o move ${rootmnt}${storage_mount_point} ${storage_mount_point}
>   fi
>   
> +[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
> +
>   for ovl_lower_dir in ${ovl_lower_dirs}; do
> -	# remove the first '/' and replace all '/' with '_'
> -	# on variable $ovl_lower_dir
> -	work_dir_name=$(awk -v var=$ovl_lower_dir \
> -		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> +	root_moved=false
> +
> +	if [ $ovl_lower_dir != "/" ]; then
> +		log_begin_msg "Mounting overlay $ovl_lower_dir"
> +
> +		# remove the first '/' and replace all '/' with '_'
> +		# on variable $ovl_lower_dir
> +		work_dir_name=$(awk -v var=$ovl_lower_dir \
> +			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> +
> +		lower_dir=${rootmnt}${ovl_lower_dir}
> +		upper_dir=${root_mount_storage}${ovl_lower_dir}
> +		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> +		mount_dir=${lower_dir}
> +	else
> +		log_begin_msg "Mounting root overlay"
> +
> +		# When overlaying root we need to move the mountpoint
> +		# from ${rootmnt} to a dedicated mountpoint inside
> +		# the storage dir to prevent a "recursion" within
> +		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
> +		# While this is not a limitation of overlay itself
> +		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
> +		# outside the tree of new root which panics when switching root
> +
> +		# create lower dir to move ro-rootfs to
> +		lower_dir=${root_mount_storage}/lower-root
> +		mkdir -p ${lower_dir}
> +
> +		# make the readonly root available
> +		mount -n -o move ${rootmnt} ${lower_dir}
> +		root_moved=true
>   
> -	lower_dir=${rootmnt}${ovl_lower_dir}
> -	upper_dir=${root_mount_storage}${ovl_lower_dir}
> -	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> +		upper_dir=${root_mount_storage}/root
> +		work_dir=${root_mount_storage}/.root-atomic
> +
> +		mount_dir=${rootmnt}
> +	fi
>   
>   	mkdir -p ${upper_dir}
>   	mkdir -p ${work_dir}
>   
> +	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
>   	if ! mount -t overlay \
>   		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
> -		   overlay ${lower_dir}; then
> +		   overlay ${mount_dir}; then
>   		panic "Can't mount overlay for '$ovl_lower_dir' !"
>   	fi
> +
> +	if ${root_moved}; then
> +		# Loop over all mountpoints already setup before we established the root overlay
> +		# and move them to the resulting overlayed rootfs
> +		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
> +		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
> +			while read -r old_mountpoint
> +			do
> +				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
> +
> +				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
> +				mount -n -o move ${old_mountpoint} ${new_mountpoint}
> +			done
> +	fi
> +
> +	log_end_msg
>   done
>   
> +# Finally, move the overlay workdir from initramfs to the final rootfs
> +[ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
> +mount -n -o move ${storage_mount_point} ${rootmnt}${storage_mount_point}
> +
> +log_success_msg "Overlays setup successfully"
> +
>   exit 0
> diff --git a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
> index ec7f85b..af610e5 100644
> --- a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
> +++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.bb
> @@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
>   DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
>   
>   HOOK_ADD_MODULES = "overlay"
> -HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
> +HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
>   
>   SCRIPT_PREREQ="crypt"
>   

Re: [isar-cip-core][PATCH 0/3] Added support for rootfs-overlay (for development).

[Jan Kiszka]

On 17.02.25 11:00, alexander.heinisch@siemens.com wrote:
> From: Alexander Heinisch <alexander.heinisch@siemens.com>
> 
> Updateable images based on isar-cip-core come with immutable rootfs
> out of the box. During development this oftentimes comes with the penalty
> of having to either rebuild images with additional files, packages aso.
> to accomplish development and debugging tasks or derive from the production
> image in a way that some fundamental features like verity, squashfs, swupdate, aso.
> are not part of such.
> 
> Both cases are far from optimal. With this patch series we want to share
> our attempt to streamline development by staying as close as possible to
> our production image by overlaying the squashfs or erofs based ro filesystem
> with an overlay. To make changes redundant accross reboots, we decided
> to use a persistent storage option in favour of a tmpfs based approach.
> (Although, technically there is no limitation on doing so)
> 
> Keep in mind, support for the root overlay is thought as a development
> feature! Thus, it is not thought for production, as it invalidates many
> of the properties we have from ro-filesystems. Also, keep in mind that
> changes on the overlay may partially hide updates or lead to "unexpected"
> results after applying updates.
> 
> Alexander Heinisch (3):
>   Removed "ro" option from read-only-rootfs's fstab.
>   Fix return value handling on filesystem check
>   Added support for rootfs-overlay.
> 
>  classes/read-only-rootfs.bbclass              |  2 +-
>  .../files/local-bottom.tmpl                   | 97 ++++++++++++++++---
>  .../initramfs-overlay-hook_0.3.bb             |  2 +-
>  3 files changed, 87 insertions(+), 14 deletions(-)
> 

Thanks, already applied patches 1 and 2. 3 is still under review and
will need at least some rewording.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Heinisch, Alexander]

> On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
> > From: Alexander Heinisch <alexander.heinisch@siemens.com>
> > 
> > In addition to the overlays for e.g. /etc we now support persistent 
> > overlays for rootfs as well.
> > To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
> > or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> > 
> 
> I would like to avoid that anti pattern of creating an overlay over the complete rootfs. What are the debug case you need a full overlay?
>
To be very clear: This is not meant for debugging in the field! Never, ever.

The intention of having a root-overlay for us was to have an easy way
to develop and extend recipes based on our existing images. That means,
we wanted to have the possibility to install packages with apt, or copy 
additional binaries, services, configs on the device without having to 
rebuild our image all the time, still maintaining to have a working environment
as close as possible to the production image. 
That enables us to develop features in a debian native environment, still being
close to the features present in the final target image. 
After being confident, the steps done and changes reflected on the overlay can 
be transferred fairly easy to a recipe and end up in a production image (image without the overlay).

BR Alexander
 
> Quirin
> 
> > This helps when developing or debugging an existing image, while still 
> > maintaining being as close to the prod image as possible.
> > That way all image features like swupdate / verity / aso. remain 
> > intact with the sole exeception of having the possibility to work on a 
> > rw rootfs.
> > Bare in mind while this comes in handy for debugging or developing new 
> > features, such feature has to be used with care. e.g. when updating 
> > the rootfs, overwritten files from the overlay could potentially cause 
> > unexpected results!
> > 
> > Compared to typical overlays we have to move the storage device used 
> > for persistence off the previous rootfs. Otherwise, the recursion in 
> > the overlay setup would cause panics when finally switching root.
> > In addition to that we move all mountpoints setup prior to overlaying 
> > to the new overlayed root.
> > 
> > Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
> > ---
> >   .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
> >   .../initramfs-overlay-hook_0.3.bb             |  2 +-
> >   2 files changed, 83 insertions(+), 11 deletions(-)
> > 
> > diff --git 
> > a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl 
> > b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> > index 71cc63c..a17e4a4 100644
> > --- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> > +++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
> > @@ -6,14 +6,19 @@
> >   # Authors:
> >   #  Jan Kiszka <jan.kiszka@siemens.com>
> >   #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
> > +#  Alexander Heinisch <alexander.heinisch@siemens.com>
> >   #
> >   
> >   ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
> >   ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
> >   
> > -root_mount_storage=${rootmnt}${ovl_storage_path}
> > +root_mount_storage=${ovl_storage_path}
> >   storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
> >   
> > +# Setup mountpoint for persistent storage # Either, it already got 
> > +setup by a previous initramfs script, # then we have to move it from 
> > +$rootmnt to $storage_mount_point or it has not # been setup yet, then 
> > +check it and mount it under $storage_mount_point
> >   if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
> >   	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
> >   	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
> > @@ -21,6 +26,8 @@ if ! mountpoint -q 
> > "${rootmnt}${storage_mount_point}"; then
> >   
> >   	partition_fstype=$(get_fstype "${ovl_partition_device}")
> >   
> > +	# FS checks should be moved to a separate hook / or at least initramfs lib
> > +	# so generic functionality can be reused here!
> >   	case $partition_fstype in
> >   	ext*)
> >   		fsck_ret=0
> > @@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
> >   		fi
> >   		;;
> >   	esac
> > +
> > +	# Mount overlay device to initramfs - will be moved to rootfs at the end
> > +	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
> >   	if ! mount -t ${partition_fstype} \
> >   		 -o ${ovl_mount_option} \
> >   		 ${ovl_partition_device} \
> > -		 ${rootmnt}${storage_mount_point}; then
> > +		 ${storage_mount_point}; then
> >   		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
> >   	fi
> > +else
> > +	# If the storage device is already mounted on the final rootfs
> > +	# move it to the initramfs. Otherwise, we cannot move back the
> > +	# storage mount to the resulting rootfs in the end.
> > +
> > +	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
> > +	mount -n -o move ${rootmnt}${storage_mount_point} 
> > +${storage_mount_point}
> >   fi
> >   
> > +[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
> > +
> >   for ovl_lower_dir in ${ovl_lower_dirs}; do
> > -	# remove the first '/' and replace all '/' with '_'
> > -	# on variable $ovl_lower_dir
> > -	work_dir_name=$(awk -v var=$ovl_lower_dir \
> > -		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> > +	root_moved=false
> > +
> > +	if [ $ovl_lower_dir != "/" ]; then
> > +		log_begin_msg "Mounting overlay $ovl_lower_dir"
> > +
> > +		# remove the first '/' and replace all '/' with '_'
> > +		# on variable $ovl_lower_dir
> > +		work_dir_name=$(awk -v var=$ovl_lower_dir \
> > +			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
> > +
> > +		lower_dir=${rootmnt}${ovl_lower_dir}
> > +		upper_dir=${root_mount_storage}${ovl_lower_dir}
> > +		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> > +		mount_dir=${lower_dir}
> > +	else
> > +		log_begin_msg "Mounting root overlay"
> > +
> > +		# When overlaying root we need to move the mountpoint
> > +		# from ${rootmnt} to a dedicated mountpoint inside
> > +		# the storage dir to prevent a "recursion" within
> > +		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
> > +		# While this is not a limitation of overlay itself
> > +		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
> > +		# outside the tree of new root which panics when switching root
> > +
> > +		# create lower dir to move ro-rootfs to
> > +		lower_dir=${root_mount_storage}/lower-root
> > +		mkdir -p ${lower_dir}
> > +
> > +		# make the readonly root available
> > +		mount -n -o move ${rootmnt} ${lower_dir}
> > +		root_moved=true
> >   
> > -	lower_dir=${rootmnt}${ovl_lower_dir}
> > -	upper_dir=${root_mount_storage}${ovl_lower_dir}
> > -	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
> > +		upper_dir=${root_mount_storage}/root
> > +		work_dir=${root_mount_storage}/.root-atomic
> > +
> > +		mount_dir=${rootmnt}
> > +	fi
> >   
> >   	mkdir -p ${upper_dir}
> >   	mkdir -p ${work_dir}
> >   
> > +	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
> >   	if ! mount -t overlay \
> >   		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
> > -		   overlay ${lower_dir}; then
> > +		   overlay ${mount_dir}; then
> >   		panic "Can't mount overlay for '$ovl_lower_dir' !"
> >   	fi
> > +
> > +	if ${root_moved}; then
> > +		# Loop over all mountpoints already setup before we established the root overlay
> > +		# and move them to the resulting overlayed rootfs
> > +		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
> > +		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
> > +			while read -r old_mountpoint
> > +			do
> > +				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
> > +
> > +				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
> > +				mount -n -o move ${old_mountpoint} ${new_mountpoint}
> > +			done
> > +	fi
> > +
> > +	log_end_msg
> >   done
> >   
> > +# Finally, move the overlay workdir from initramfs to the final 
> > +rootfs [ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
> > +mount -n -o move ${storage_mount_point} 
> > +${rootmnt}${storage_mount_point}
> > +
> > +log_success_msg "Overlays setup successfully"
> > +
> >   exit 0
> > diff --git 
> > a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
> > bb 
> > b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
> > bb
> > index ec7f85b..af610e5 100644
> > --- 
> > a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
> > bb
> > +++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_
> > +++ 0.3.bb
> > @@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
> >   DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
> >   
> >   HOOK_ADD_MODULES = "overlay"
> > -HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
> > +HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
> >   
> >   SCRIPT_PREREQ="crypt"
> >   

Re: [cip-dev] [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Nussel, Ludwig]

On Thu, 2025-02-20 at 13:00 +0000, Heinisch, Alexander via lists.cip-
project.org wrote:
> > On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
> > > From: Alexander Heinisch <alexander.heinisch@siemens.com>
> > > 
> > > In addition to the overlays for e.g. /etc we now support
> > > persistent 
> > > overlays for rootfs as well.
> > > To enable this behaviour you can either set
> > > INITRAMFS_OVERLAY_PATHS = "/"
> > > or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> > > 
> > 
> > I would like to avoid that anti pattern of creating an overlay over
> > the complete rootfs. What are the debug case you need a full
> > overlay?
> > 
> To be very clear: This is not meant for debugging in the field!
> Never, ever.
> 
> The intention of having a root-overlay for us was to have an easy way
> to develop and extend recipes based on our existing images. That
> means,
> we wanted to have the possibility to install packages with apt, or
> copy 
> additional binaries, services, configs on the device without having
> to 
> rebuild our image all the time, still maintaining to have a working
> environment
> as close as possible to the production image. 
> That enables us to develop features in a debian native environment,
> still being
> close to the features present in the final target image. 
> After being confident, the steps done and changes reflected on the
> overlay can 
> be transferred fairly easy to a recipe and end up in a production
> image (image without the overlay).

For that it should still be sufficient to overlay /usr only as that one
holds the read-only OS parts for any halfway modern OS that has
usrmerge. /etc already has a separate overlay.

What we came up with is to trigger the /usr overlay based on presence
of a file in /etc. The backing store dir in /var is then versioned
based on the revision/snapshot of the OS in /usr (eg BUILD_ID in os-
release). That way the overlay is only used for a given build of the
OS. A new build of the image can then be deployed without old overlays
interfering.

IIRC recently systemd-sysext also gained some options to provide rw
overlays. That might be the way going forward.


cu
Ludwig

-- 
(o_ Ludwig Nussel
//\ Siemens AG / SI E R&D IOT
V_/_ www.siemens.com

RE: [isar-cip-core][PATCH 0/3] Added support for rootfs-overlay (for development).

[Heinisch, Alexander]

> On 17.02.25 11:00, alexander.heinisch@siemens.com wrote:
> > From: Alexander Heinisch <alexander.heinisch@siemens.com>
> > 
> > Updateable images based on isar-cip-core come with immutable rootfs 
> > out of the box. During development this oftentimes comes with the 
> > penalty of having to either rebuild images with additional files, packages aso.
> > to accomplish development and debugging tasks or derive from the 
> > production image in a way that some fundamental features like verity, squashfs, swupdate, aso.
> > are not part of such.
> > 
> > Both cases are far from optimal. With this patch series we want to 
> > share our attempt to streamline development by staying as close as 
> > possible to our production image by overlaying the squashfs or erofs 
> > based ro filesystem with an overlay. To make changes redundant accross 
> > reboots, we decided to use a persistent storage option in favour of a tmpfs based approach.
> > (Although, technically there is no limitation on doing so)
> > 
> > Keep in mind, support for the root overlay is thought as a development 
> > feature! Thus, it is not thought for production, as it invalidates 
> > many of the properties we have from ro-filesystems. Also, keep in mind 
> > that changes on the overlay may partially hide updates or lead to "unexpected"
> > results after applying updates.
> > 
> > Alexander Heinisch (3):
> >   Removed "ro" option from read-only-rootfs's fstab.
> >   Fix return value handling on filesystem check
> >   Added support for rootfs-overlay.
> > 
> >  classes/read-only-rootfs.bbclass              |  2 +-
> >  .../files/local-bottom.tmpl                   | 97 ++++++++++++++++---
> >  .../initramfs-overlay-hook_0.3.bb             |  2 +-
> >  3 files changed, 87 insertions(+), 14 deletions(-)
> > 
> 
> Thanks, already applied patches 1 and 2. 3 is still under review and will need at least some rewording.

Unfortunately, we identified some issues with 
>> Removed "ro" option from read-only-rootfs's fstab. <<
in a downstream layer! - will share our observations in the other thread.

Sry

> 
> Jan
> 
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center

RE: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Heinisch, Alexander]

> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Heinisch, Alexander via lists.cip-project.org
> Sent: Montag, 17. Februar 2025 11:00
> To: cip-dev@lists.cip-project.org
> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; Heinisch, Alexander (FT RPD CED SES-AT) <alexander.heinisch@siemens.com>
> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.
> 
> From: Alexander Heinisch <alexander.heinisch@siemens.com>
> 
> Since we only support erofs and squashfs this option is not needed any more. Further, it causes potential overlay rootfs variants to be remounted read-only.
> 
> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
> ---
>  classes/read-only-rootfs.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only-rootfs.bbclass
> index 35a3ab3..83ddc33 100644
> --- a/classes/read-only-rootfs.bbclass
> +++ b/classes/read-only-rootfs.bbclass
> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>  image_configure_fstab() {
>      sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin /etc/fstab
> -/dev/root	/		auto		defaults,ro			0	0
> +/dev/root	/		auto		defaults			0	0
>  LABEL=var	/var		auto		defaults			0	0
>  proc		/proc		proc		nosuid,noexec,nodev		0	0
>  sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
> --
> 2.39.5

We observed `systemd-remount-fs` failing in a downstream layer.
(using a ro squashfs and apply full disk encryption on top)
Although, the service is executed in isar-cip-core as well, upstream,
the service can be executed successfully.

In essence what happens is that, the `systemd-remount-fs`
remounts various mountpoints based on their settings in
`/etc/fstab`

```
root@device-mgmt:~# cat /etc/fstab
# Begin /etc/fstab
/dev/root	/		auto		defaults			0	0
LABEL=var	/var		auto		defaults			0	0
proc		/proc		proc		nosuid,noexec,nodev		0	0
sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
devpts		/dev/pts	devpts		gid=5,mode=620			0	0
tmpfs		/run		tmpfs		nodev,nosuid,size=500M,mode=755	0	0
devtmpfs	/dev		devtmpfs	mode=0755,nosuid		0	0
# End /etc/fstab
```

Since `defaults` in fstab renders to: `rw,suid,dev,exec,auto,nouser,async`
the service fails trying to remount the `squashfs` as `rw` as shown below:

```
root@device-mgmt:~# systemctl status systemd-remount-fs
× systemd-remount-fs.service - Remount Root and Kernel File Systems
     Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; enabled-ru>
     Active: failed (Result: exit-code) since Fri 2025-02-21 08:55:39 UTC; 5min>
       Docs: man:systemd-remount-fs.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
    Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, statu>
   Main PID: 7530 (code=exited, status=1/FAILURE)
        CPU: 15ms
```

```
root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs
Feb 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /: cannot remount /dev/root read-write, is write-protected.
Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:        dmesg(1) may have more information after failed mount system call.
Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/mount for / exited with exit status 32.
Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
```

The rootfs is mounted as follows:

```
root@device-mgmt:~# mount | grep " / "
/dev/mapper/verityroot on / type squashfs (ro,relatime,errors=continue)
```

I understand why this happens on our system, but could not clarify why it does 
not happen upstream! So any idea, welcome!

Nevertheless, after rethinking this patch. Maybe it was advisible to not use
`default` here and either use a options like `suid,dev,exec,auto,nouser,async`
or drop the line for `/dev/root` at all, keeping the mounts as they were setup
in the initramfs.

Any opinions, recommendations, hints, ...?

Thx in advance,
BR Alexander

RE: [cip-dev] [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Heinisch, Alexander]

[-- Attachment #1: Type: text/plain, Size: 3607 bytes --]

> From: cip-dev@lists.cip-project.org<mailto:cip-dev@lists.cip-project.org> cip-dev@lists.cip-project.org<mailto:cip-dev@lists.cip-project.org> On Behalf Of Nussel, Ludwig via lists.cip-project.org
> Sent: Donnerstag, 20. Februar 2025 14:12
> To: cip-dev@lists.cip-project.org<mailto:cip-dev@lists.cip-project.org>
> Subject: Re: [cip-dev] [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.
>
> On Thu, 2025-02-20 at 13:00 +0000, Heinisch, Alexander via lists.cip- project.org wrote:
> > > On 2/17/25 11:00, alexander.heinisch@siemens.com<mailto:alexander.heinisch@siemens.com> wrote:
> > > > From: Alexander Heinisch alexander.heinisch@siemens.com<mailto:alexander.heinisch@siemens.com>
> > > >
> > > > In addition to the overlays for e.g. /etc we now support
> > > > persistent overlays for rootfs as well.
> > > > To enable this behaviour you can either set
> > > > INITRAMFS_OVERLAY_PATHS = "/"
> > > > or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
> > > >
> > >
> > > I would like to avoid that anti pattern of creating an overlay over
> > > the complete rootfs. What are the debug case you need a full
> > > overlay?
> > >
> > To be very clear: This is not meant for debugging in the field!
> > Never, ever.
> >
> > The intention of having a root-overlay for us was to have an easy way
> > to develop and extend recipes based on our existing images. That
> > means,
> > we wanted to have the possibility to install packages with apt, or
> > copy
> > additional binaries, services, configs on the device without having
> > to
> > rebuild our image all the time, still maintaining to have a working
> > environment
> > as close as possible to the production image.
> > That enables us to develop features in a debian native environment,
> > still being
> > close to the features present in the final target image.
> > After being confident, the steps done and changes reflected on the
> > overlay can
> > be transferred fairly easy to a recipe and end up in a production
> > image (image without the overlay).
>
> For that it should still be sufficient to overlay /usr only as that one
> holds the read-only OS parts for any halfway modern OS that has
> usrmerge. /etc already has a separate overlay.
>

Probably that's the case for most partitionings. Still having a complete
root overlay is more generic imo.
But I am open for that!

>
> What we came up with is to trigger the /usr overlay based on presence
> of a file in /etc. The backing store dir in /var is then versioned
> based on the revision/snapshot of the OS in /usr (eg BUILD_ID in os-
> release). That way the overlay is only used for a given build of the
> OS. A new build of the image can then be deployed without old overlays
> interfering.

Sounds interesting, are there patches available for this?

Anyhow, I would not go for an enablement of such behaviour via /etc or any other rw-able storage,
as this may cause security implications and lots of argumentations during TRA :-)
which can easily be avoided due to the fact of having separate images for development
and production (where this behavior is disabled by not having `/` listed in
`INITRAMFS_OVERLAY_PATHS`)

>
> IIRC recently systemd-sysext also gained some options to provide rw
> overlays. That might be the way going forward.

If limiting the overlay to /usr systemd-sysext could help.
But that requires systemd 256

BR Alexander

>
>
> cu
> Ludwig
>
> --
> (o_ Ludwig Nussel
> //\ Siemens AG / SI E R&D IOT
> V_/_ www.siemens.com<http://www.siemens.com>

[-- Attachment #2: Type: text/html, Size: 17025 bytes --]

Re: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Jan Kiszka]

On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Heinisch, Alexander via lists.cip-project.org
>> Sent: Montag, 17. Februar 2025 11:00
>> To: cip-dev@lists.cip-project.org
>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; Heinisch, Alexander (FT RPD CED SES-AT) <alexander.heinisch@siemens.com>
>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.
>>
>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>
>> Since we only support erofs and squashfs this option is not needed any more. Further, it causes potential overlay rootfs variants to be remounted read-only.
>>
>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>> ---
>>  classes/read-only-rootfs.bbclass | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only-rootfs.bbclass
>> index 35a3ab3..83ddc33 100644
>> --- a/classes/read-only-rootfs.bbclass
>> +++ b/classes/read-only-rootfs.bbclass
>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>  image_configure_fstab() {
>>      sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin /etc/fstab
>> -/dev/root	/		auto		defaults,ro			0	0
>> +/dev/root	/		auto		defaults			0	0
>>  LABEL=var	/var		auto		defaults			0	0
>>  proc		/proc		proc		nosuid,noexec,nodev		0	0
>>  sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
>> --
>> 2.39.5
> 
> We observed `systemd-remount-fs` failing in a downstream layer.
> (using a ro squashfs and apply full disk encryption on top)
> Although, the service is executed in isar-cip-core as well, upstream,
> the service can be executed successfully.
> 
> In essence what happens is that, the `systemd-remount-fs`
> remounts various mountpoints based on their settings in
> `/etc/fstab`
> 
> ```
> root@device-mgmt:~# cat /etc/fstab
> # Begin /etc/fstab
> /dev/root	/		auto		defaults			0	0
> LABEL=var	/var		auto		defaults			0	0
> proc		/proc		proc		nosuid,noexec,nodev		0	0
> sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
> devpts		/dev/pts	devpts		gid=5,mode=620			0	0
> tmpfs		/run		tmpfs		nodev,nosuid,size=500M,mode=755	0	0
> devtmpfs	/dev		devtmpfs	mode=0755,nosuid		0	0
> # End /etc/fstab
> ```
> 
> Since `defaults` in fstab renders to: `rw,suid,dev,exec,auto,nouser,async`
> the service fails trying to remount the `squashfs` as `rw` as shown below:
> 
> ```
> root@device-mgmt:~# systemctl status systemd-remount-fs
> × systemd-remount-fs.service - Remount Root and Kernel File Systems
>      Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; enabled-ru>
>      Active: failed (Result: exit-code) since Fri 2025-02-21 08:55:39 UTC; 5min>
>        Docs: man:systemd-remount-fs.service(8)
>              https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
>     Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, statu>
>    Main PID: 7530 (code=exited, status=1/FAILURE)
>         CPU: 15ms
> ```
> 
> ```
> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs
> Feb 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /: cannot remount /dev/root read-write, is write-protected.
> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:        dmesg(1) may have more information after failed mount system call.
> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/mount for / exited with exit status 32.
> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
> ```
> 
> The rootfs is mounted as follows:
> 
> ```
> root@device-mgmt:~# mount | grep " / "
> /dev/mapper/verityroot on / type squashfs (ro,relatime,errors=continue)
> ```
> 
> I understand why this happens on our system, but could not clarify why it does 
> not happen upstream! So any idea, welcome!
> 

The same issue happens in upstream: secure boot on, data partition
encrypted, remount-fs fails. Which makes me wonder if our boot tests
should check for failing system services...

> Nevertheless, after rethinking this patch. Maybe it was advisible to not use
> `default` here and either use a options like `suid,dev,exec,auto,nouser,async`
> or drop the line for `/dev/root` at all, keeping the mounts as they were setup
> in the initramfs.
> 
> Any opinions, recommendations, hints, ...?
> 

According to the man pages, a left-out rw/ro means that mount will first
try rw and then, if it fails, ro. That would give us the desired
flexibility. Hard-coding the other values of "defaults" is the downside.

On the other hand, what would be the mount options that the initramfs
will use? We do not have direct control over them as well, do we?

Quirin, other reasons why we want / in our fstab?

Another option: "ro" a parameter of read-only-rootfs.bbclass so that
your scenario can override the default.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Quirin Gylstorff]

On 2/24/25 07:09, Jan Kiszka wrote:
> On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> On Behalf Of Heinisch, Alexander via lists.cip-project.org
>>> Sent: Montag, 17. Februar 2025 11:00
>>> To: cip-dev@lists.cip-project.org
>>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; Heinisch, Alexander (FT RPD CED SES-AT) <alexander.heinisch@siemens.com>
>>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.
>>>
>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>
>>> Since we only support erofs and squashfs this option is not needed any more. Further, it causes potential overlay rootfs variants to be remounted read-only.
>>>
>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>> ---
>>>   classes/read-only-rootfs.bbclass | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only-rootfs.bbclass
>>> index 35a3ab3..83ddc33 100644
>>> --- a/classes/read-only-rootfs.bbclass
>>> +++ b/classes/read-only-rootfs.bbclass
>>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>>   image_configure_fstab() {
>>>       sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin /etc/fstab
>>> -/dev/root	/		auto		defaults,ro			0	0
>>> +/dev/root	/		auto		defaults			0	0
>>>   LABEL=var	/var		auto		defaults			0	0
>>>   proc		/proc		proc		nosuid,noexec,nodev		0	0
>>>   sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
>>> --
>>> 2.39.5
>>
>> We observed `systemd-remount-fs` failing in a downstream layer.
>> (using a ro squashfs and apply full disk encryption on top)
>> Although, the service is executed in isar-cip-core as well, upstream,
>> the service can be executed successfully.
>>
>> In essence what happens is that, the `systemd-remount-fs`
>> remounts various mountpoints based on their settings in
>> `/etc/fstab`
>>
>> ```
>> root@device-mgmt:~# cat /etc/fstab
>> # Begin /etc/fstab
>> /dev/root	/		auto		defaults			0	0
>> LABEL=var	/var		auto		defaults			0	0
>> proc		/proc		proc		nosuid,noexec,nodev		0	0
>> sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
>> devpts		/dev/pts	devpts		gid=5,mode=620			0	0
>> tmpfs		/run		tmpfs		nodev,nosuid,size=500M,mode=755	0	0
>> devtmpfs	/dev		devtmpfs	mode=0755,nosuid		0	0
>> # End /etc/fstab
>> ```
>>
>> Since `defaults` in fstab renders to: `rw,suid,dev,exec,auto,nouser,async`
>> the service fails trying to remount the `squashfs` as `rw` as shown below:
>>
>> ```
>> root@device-mgmt:~# systemctl status systemd-remount-fs
>> × systemd-remount-fs.service - Remount Root and Kernel File Systems
>>       Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; enabled-ru>
>>       Active: failed (Result: exit-code) since Fri 2025-02-21 08:55:39 UTC; 5min>
>>         Docs: man:systemd-remount-fs.service(8)
>>               https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
>>      Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, statu>
>>     Main PID: 7530 (code=exited, status=1/FAILURE)
>>          CPU: 15ms
>> ```
>>
>> ```
>> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs
>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd-remount-fs.service - Remount Root and Kernel File Systems...
>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /: cannot remount /dev/root read-write, is write-protected.
>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:        dmesg(1) may have more information after failed mount system call.
>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/mount for / exited with exit status 32.
>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
>> ```
>>
>> The rootfs is mounted as follows:
>>
>> ```
>> root@device-mgmt:~# mount | grep " / "
>> /dev/mapper/verityroot on / type squashfs (ro,relatime,errors=continue)
>> ```
>>
>> I understand why this happens on our system, but could not clarify why it does
>> not happen upstream! So any idea, welcome!
>>
> 
> The same issue happens in upstream: secure boot on, data partition
> encrypted, remount-fs fails. Which makes me wonder if our boot tests
> should check for failing system services...
> 
>> Nevertheless, after rethinking this patch. Maybe it was advisible to not use
>> `default` here and either use a options like `suid,dev,exec,auto,nouser,async`
>> or drop the line for `/dev/root` at all, keeping the mounts as they were setup
>> in the initramfs.
>>
>> Any opinions, recommendations, hints, ...?
>>
> 
> According to the man pages, a left-out rw/ro means that mount will first
> try rw and then, if it fails, ro. That would give us the desired
> flexibility. Hard-coding the other values of "defaults" is the downside.
> 
Will no also lead to error of systemd-remount?

> On the other hand, what would be the mount options that the initramfs
> will use? We do not have direct control over them as well, do we?

The initramfs in standard Debian contains the fstab - so it would/could 
use the options from there.
> 
> Quirin, other reasons why we want / in our fstab?

The fstab of the readonly rootfs is an evolution of the original fstab
from ISAR. It is also there as we disabled the fstab generation from
wic.

> 
> Another option: "ro" a parameter of read-only-rootfs.bbclass so that
> your scenario can override the default.

I would make it to option for all the fstab options.

> 
> Jan
> 
Quirin

Re: [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Quirin Gylstorff]

On 2/20/25 14:00, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>> On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>
>>> In addition to the overlays for e.g. /etc we now support persistent
>>> overlays for rootfs as well.
>>> To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
>>> or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
>>>
>>
>> I would like to avoid that anti pattern of creating an overlay over the complete rootfs. What are the debug case you need a full overlay?
>>
> To be very clear: This is not meant for debugging in the field! Never, ever.
> 
> The intention of having a root-overlay for us was to have an easy way
> to develop and extend recipes based on our existing images. That means,
> we wanted to have the possibility to install packages with apt, or copy
> additional binaries, services, configs on the device without having to
> rebuild our image all the time, still maintaining to have a working environment
> as close as possible to the production image.
> That enables us to develop features in a debian native environment, still being
> close to the features present in the final target image.
> After being confident, the steps done and changes reflected on the overlay can
> be transferred fairly easy to a recipe and end up in a production image (image without the overlay).

Ok, this still does not explain why
Overlaying `/usr` should be enough for these use cases ?

Also this kind of testing  ensures that you will not find issues 
realated to read only rootfs in your packages during development.

Quirin
> 
> BR Alexander
>   
>> Quirin
>>
>>> This helps when developing or debugging an existing image, while still
>>> maintaining being as close to the prod image as possible.
>>> That way all image features like swupdate / verity / aso. remain
>>> intact with the sole exeception of having the possibility to work on a
>>> rw rootfs.
>>> Bare in mind while this comes in handy for debugging or developing new
>>> features, such feature has to be used with care. e.g. when updating
>>> the rootfs, overwritten files from the overlay could potentially cause
>>> unexpected results!
>>>
>>> Compared to typical overlays we have to move the storage device used
>>> for persistence off the previous rootfs. Otherwise, the recursion in
>>> the overlay setup would cause panics when finally switching root.
>>> In addition to that we move all mountpoints setup prior to overlaying
>>> to the new overlayed root.
>>>
>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>> ---
>>>    .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
>>>    .../initramfs-overlay-hook_0.3.bb             |  2 +-
>>>    2 files changed, 83 insertions(+), 11 deletions(-)
>>>
>>> diff --git
>>> a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>> b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>> index 71cc63c..a17e4a4 100644
>>> --- a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>> +++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>> @@ -6,14 +6,19 @@
>>>    # Authors:
>>>    #  Jan Kiszka <jan.kiszka@siemens.com>
>>>    #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> +#  Alexander Heinisch <alexander.heinisch@siemens.com>
>>>    #
>>>    
>>>    ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
>>>    ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
>>>    
>>> -root_mount_storage=${rootmnt}${ovl_storage_path}
>>> +root_mount_storage=${ovl_storage_path}
>>>    storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
>>>    
>>> +# Setup mountpoint for persistent storage # Either, it already got
>>> +setup by a previous initramfs script, # then we have to move it from
>>> +$rootmnt to $storage_mount_point or it has not # been setup yet, then
>>> +check it and mount it under $storage_mount_point
>>>    if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>>>    	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
>>>    	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
>>> @@ -21,6 +26,8 @@ if ! mountpoint -q
>>> "${rootmnt}${storage_mount_point}"; then
>>>    
>>>    	partition_fstype=$(get_fstype "${ovl_partition_device}")
>>>    
>>> +	# FS checks should be moved to a separate hook / or at least initramfs lib
>>> +	# so generic functionality can be reused here!
>>>    	case $partition_fstype in
>>>    	ext*)
>>>    		fsck_ret=0
>>> @@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>>>    		fi
>>>    		;;
>>>    	esac
>>> +
>>> +	# Mount overlay device to initramfs - will be moved to rootfs at the end
>>> +	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
>>>    	if ! mount -t ${partition_fstype} \
>>>    		 -o ${ovl_mount_option} \
>>>    		 ${ovl_partition_device} \
>>> -		 ${rootmnt}${storage_mount_point}; then
>>> +		 ${storage_mount_point}; then
>>>    		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
>>>    	fi
>>> +else
>>> +	# If the storage device is already mounted on the final rootfs
>>> +	# move it to the initramfs. Otherwise, we cannot move back the
>>> +	# storage mount to the resulting rootfs in the end.
>>> +
>>> +	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
>>> +	mount -n -o move ${rootmnt}${storage_mount_point}
>>> +${storage_mount_point}
>>>    fi
>>>    
>>> +[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
>>> +
>>>    for ovl_lower_dir in ${ovl_lower_dirs}; do
>>> -	# remove the first '/' and replace all '/' with '_'
>>> -	# on variable $ovl_lower_dir
>>> -	work_dir_name=$(awk -v var=$ovl_lower_dir \
>>> -		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
>>> +	root_moved=false
>>> +
>>> +	if [ $ovl_lower_dir != "/" ]; then
>>> +		log_begin_msg "Mounting overlay $ovl_lower_dir"
>>> +
>>> +		# remove the first '/' and replace all '/' with '_'
>>> +		# on variable $ovl_lower_dir
>>> +		work_dir_name=$(awk -v var=$ovl_lower_dir \
>>> +			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
>>> +
>>> +		lower_dir=${rootmnt}${ovl_lower_dir}
>>> +		upper_dir=${root_mount_storage}${ovl_lower_dir}
>>> +		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
>>> +		mount_dir=${lower_dir}
>>> +	else
>>> +		log_begin_msg "Mounting root overlay"
>>> +
>>> +		# When overlaying root we need to move the mountpoint
>>> +		# from ${rootmnt} to a dedicated mountpoint inside
>>> +		# the storage dir to prevent a "recursion" within
>>> +		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
>>> +		# While this is not a limitation of overlay itself
>>> +		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
>>> +		# outside the tree of new root which panics when switching root
>>> +
>>> +		# create lower dir to move ro-rootfs to
>>> +		lower_dir=${root_mount_storage}/lower-root
>>> +		mkdir -p ${lower_dir}
>>> +
>>> +		# make the readonly root available
>>> +		mount -n -o move ${rootmnt} ${lower_dir}
>>> +		root_moved=true
>>>    
>>> -	lower_dir=${rootmnt}${ovl_lower_dir}
>>> -	upper_dir=${root_mount_storage}${ovl_lower_dir}
>>> -	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
>>> +		upper_dir=${root_mount_storage}/root
>>> +		work_dir=${root_mount_storage}/.root-atomic
>>> +
>>> +		mount_dir=${rootmnt}
>>> +	fi
>>>    
>>>    	mkdir -p ${upper_dir}
>>>    	mkdir -p ${work_dir}
>>>    
>>> +	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
>>>    	if ! mount -t overlay \
>>>    		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
>>> -		   overlay ${lower_dir}; then
>>> +		   overlay ${mount_dir}; then
>>>    		panic "Can't mount overlay for '$ovl_lower_dir' !"
>>>    	fi
>>> +
>>> +	if ${root_moved}; then
>>> +		# Loop over all mountpoints already setup before we established the root overlay
>>> +		# and move them to the resulting overlayed rootfs
>>> +		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
>>> +		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
>>> +			while read -r old_mountpoint
>>> +			do
>>> +				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
>>> +
>>> +				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
>>> +				mount -n -o move ${old_mountpoint} ${new_mountpoint}
>>> +			done
>>> +	fi
>>> +
>>> +	log_end_msg
>>>    done
>>>    
>>> +# Finally, move the overlay workdir from initramfs to the final
>>> +rootfs [ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
>>> +mount -n -o move ${storage_mount_point}
>>> +${rootmnt}${storage_mount_point}
>>> +
>>> +log_success_msg "Overlays setup successfully"
>>> +
>>>    exit 0
>>> diff --git
>>> a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
>>> bb
>>> b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
>>> bb
>>> index ec7f85b..af610e5 100644
>>> ---
>>> a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
>>> bb
>>> +++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_
>>> +++ 0.3.bb
>>> @@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
>>>    DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
>>>    
>>>    HOOK_ADD_MODULES = "overlay"
>>> -HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
>>> +HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
>>>    
>>>    SCRIPT_PREREQ="crypt"
>>>    
> 

Re: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Jan Kiszka]

On 24.02.25 13:17, Quirin Gylstorff wrote:
> 
> 
> On 2/24/25 07:09, Jan Kiszka wrote:
>> On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org>
>>>> On Behalf Of Heinisch, Alexander via lists.cip-project.org
>>>> Sent: Montag, 17. Februar 2025 11:00
>>>> To: cip-dev@lists.cip-project.org
>>>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff,
>>>> Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; Heinisch,
>>>> Alexander (FT RPD CED SES-AT) <alexander.heinisch@siemens.com>
>>>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option
>>>> from read-only-rootfs's fstab.
>>>>
>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>
>>>> Since we only support erofs and squashfs this option is not needed
>>>> any more. Further, it causes potential overlay rootfs variants to be
>>>> remounted read-only.
>>>>
>>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>> ---
>>>>   classes/read-only-rootfs.bbclass | 2 +-
>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only-
>>>> rootfs.bbclass
>>>> index 35a3ab3..83ddc33 100644
>>>> --- a/classes/read-only-rootfs.bbclass
>>>> +++ b/classes/read-only-rootfs.bbclass
>>>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>>>   image_configure_fstab() {
>>>>       sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin /etc/fstab
>>>> -/dev/root    /        auto        defaults,ro            0    0
>>>> +/dev/root    /        auto        defaults            0    0
>>>>   LABEL=var    /var        auto        defaults            0    0
>>>>   proc        /proc        proc        nosuid,noexec,nodev       
>>>> 0    0
>>>>   sysfs        /sys        sysfs        nosuid,noexec,nodev       
>>>> 0    0
>>>> -- 
>>>> 2.39.5
>>>
>>> We observed `systemd-remount-fs` failing in a downstream layer.
>>> (using a ro squashfs and apply full disk encryption on top)
>>> Although, the service is executed in isar-cip-core as well, upstream,
>>> the service can be executed successfully.
>>>
>>> In essence what happens is that, the `systemd-remount-fs`
>>> remounts various mountpoints based on their settings in
>>> `/etc/fstab`
>>>
>>> ```
>>> root@device-mgmt:~# cat /etc/fstab
>>> # Begin /etc/fstab
>>> /dev/root    /        auto        defaults            0    0
>>> LABEL=var    /var        auto        defaults            0    0
>>> proc        /proc        proc        nosuid,noexec,nodev        0    0
>>> sysfs        /sys        sysfs        nosuid,noexec,nodev        0    0
>>> devpts        /dev/pts    devpts        gid=5,mode=620            0    0
>>> tmpfs        /run        tmpfs       
>>> nodev,nosuid,size=500M,mode=755    0    0
>>> devtmpfs    /dev        devtmpfs    mode=0755,nosuid        0    0
>>> # End /etc/fstab
>>> ```
>>>
>>> Since `defaults` in fstab renders to:
>>> `rw,suid,dev,exec,auto,nouser,async`
>>> the service fails trying to remount the `squashfs` as `rw` as shown
>>> below:
>>>
>>> ```
>>> root@device-mgmt:~# systemctl status systemd-remount-fs
>>> × systemd-remount-fs.service - Remount Root and Kernel File Systems
>>>       Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service;
>>> enabled-ru>
>>>       Active: failed (Result: exit-code) since Fri 2025-02-21
>>> 08:55:39 UTC; 5min>
>>>         Docs: man:systemd-remount-fs.service(8)
>>>               https://www.freedesktop.org/wiki/Software/systemd/
>>> APIFileSystems
>>>      Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs
>>> (code=exited, statu>
>>>     Main PID: 7530 (code=exited, status=1/FAILURE)
>>>          CPU: 15ms
>>> ```
>>>
>>> ```
>>> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs
>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd-
>>> remount-fs.service - Remount Root and Kernel File Systems...
>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /:
>>> cannot remount /dev/root read-write, is write-protected.
>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:       
>>> dmesg(1) may have more information after failed mount system call.
>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/
>>> mount for / exited with exit status 32.
>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>> fs.service: Main process exited, code=exited, status=1/FAILURE
>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>> fs.service: Failed with result 'exit-code'.
>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start
>>> systemd-remount-fs.service - Remount Root and Kernel File Systems.
>>> ```
>>>
>>> The rootfs is mounted as follows:
>>>
>>> ```
>>> root@device-mgmt:~# mount | grep " / "
>>> /dev/mapper/verityroot on / type squashfs (ro,relatime,errors=continue)
>>> ```
>>>
>>> I understand why this happens on our system, but could not clarify
>>> why it does
>>> not happen upstream! So any idea, welcome!
>>>
>>
>> The same issue happens in upstream: secure boot on, data partition
>> encrypted, remount-fs fails. Which makes me wonder if our boot tests
>> should check for failing system services...
>>
>>> Nevertheless, after rethinking this patch. Maybe it was advisible to
>>> not use
>>> `default` here and either use a options like
>>> `suid,dev,exec,auto,nouser,async`
>>> or drop the line for `/dev/root` at all, keeping the mounts as they
>>> were setup
>>> in the initramfs.
>>>
>>> Any opinions, recommendations, hints, ...?
>>>
>>
>> According to the man pages, a left-out rw/ro means that mount will first
>> try rw and then, if it fails, ro. That would give us the desired
>> flexibility. Hard-coding the other values of "defaults" is the downside.
>>
> Will no also lead to error of systemd-remount?
> 

It does not help, yes, just tried in vain.

>> On the other hand, what would be the mount options that the initramfs
>> will use? We do not have direct control over them as well, do we?
> 
> The initramfs in standard Debian contains the fstab - so it would/could
> use the options from there.
>>
>> Quirin, other reasons why we want / in our fstab?
> 
> The fstab of the readonly rootfs is an evolution of the original fstab
> from ISAR. It is also there as we disabled the fstab generation from
> wic.
> 
>>
>> Another option: "ro" a parameter of read-only-rootfs.bbclass so that
>> your scenario can override the default.
> 
> I would make it to option for all the fstab options.
> 

Ok, let's go for some config variable then.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Heinisch, Alexander]

>On 2/20/25 14:00, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>> On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>
>>>> In addition to the overlays for e.g. /etc we now support persistent 
>>>> overlays for rootfs as well.
>>>> To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
>>>> or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
>>>>
>>>
>>> I would like to avoid that anti pattern of creating an overlay over the complete rootfs. What are the debug case you need a full overlay?
>>>
>> To be very clear: This is not meant for debugging in the field! Never, ever.
>> 
>> The intention of having a root-overlay for us was to have an easy way 
>> to develop and extend recipes based on our existing images. That 
>> means, we wanted to have the possibility to install packages with apt, 
>> or copy additional binaries, services, configs on the device without 
>> having to rebuild our image all the time, still maintaining to have a 
>> working environment as close as possible to the production image.
>> That enables us to develop features in a debian native environment, 
>> still being close to the features present in the final target image.
>> After being confident, the steps done and changes reflected on the 
>> overlay can be transferred fairly easy to a recipe and end up in a production image (image without the overlay).
>
>Ok, this still does not explain why
>Overlaying `/usr` should be enough for these use cases ?

I explained that in the thread with Ludwig: 
"
Probably that's the case for most partitionings. Still having a complete 
root overlay is more generic imo.
But I am open for that!
"

>
>Also this kind of testing  ensures that you will not find issues realated to read only rootfs in your packages during development.

Same holds true for overlaying /usr. Once I have fundamental parts of the
ro filesystem overlayed, issues may be hidden during that time.
Anyhow, as said, this is only a feature we enable when developing new recipes.
Once the recipe is done we rebuild an image without root-overlay and 
testing is done without such overlay!

TLDR: I am also fine with overlaying just /usr to simplify the setup.
The initial idea was to have a generic solution for a full overlay
including custom downstream partitionings / mountpoints for external storage.

But I have to admit, it brings additional complexity and if we decide against
such generic approach I am also fine with that!

BR Alexander

>
>Quirin
>> 
>> BR Alexander
>>   
>>> Quirin
>>>
>>>> This helps when developing or debugging an existing image, while 
>>>> still maintaining being as close to the prod image as possible.
>>>> That way all image features like swupdate / verity / aso. remain 
>>>> intact with the sole exeception of having the possibility to work on 
>>>> a rw rootfs.
>>>> Bare in mind while this comes in handy for debugging or developing 
>>>> new features, such feature has to be used with care. e.g. when 
>>>> updating the rootfs, overwritten files from the overlay could 
>>>> potentially cause unexpected results!
>>>>
>>>> Compared to typical overlays we have to move the storage device used 
>>>> for persistence off the previous rootfs. Otherwise, the recursion in 
>>>> the overlay setup would cause panics when finally switching root.
>>>> In addition to that we move all mountpoints setup prior to 
>>>> overlaying to the new overlayed root.
>>>>
>>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>> ---
>>>>    .../files/local-bottom.tmpl                   | 92 +++++++++++++++++--
>>>>    .../initramfs-overlay-hook_0.3.bb             |  2 +-
>>>>    2 files changed, 83 insertions(+), 11 deletions(-)
>>>>
>>>> diff --git
>>>> a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>>> b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>>> index 71cc63c..a17e4a4 100644
>>>> --- 
>>>> a/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tmpl
>>>> +++ b/recipes-initramfs/initramfs-overlay-hook/files/local-bottom.tm
>>>> +++ pl
>>>> @@ -6,14 +6,19 @@
>>>>    # Authors:
>>>>    #  Jan Kiszka <jan.kiszka@siemens.com>
>>>>    #  Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>> +#  Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>    #
>>>>    
>>>>    ovl_storage_path="${INITRAMFS_OVERLAY_STORAGE_PATH}"
>>>>    ovl_lower_dirs="${INITRAMFS_OVERLAY_PATHS}"
>>>>    
>>>> -root_mount_storage=${rootmnt}${ovl_storage_path}
>>>> +root_mount_storage=${ovl_storage_path}
>>>>    storage_mount_point="$(echo "${ovl_storage_path}" | awk -F/ '{print FS$2}' )"
>>>>    
>>>> +# Setup mountpoint for persistent storage # Either, it already got 
>>>> +setup by a previous initramfs script, # then we have to move it 
>>>> +from $rootmnt to $storage_mount_point or it has not # been setup 
>>>> +yet, then check it and mount it under $storage_mount_point
>>>>    if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>>>>    	ovl_partition_device="${INITRAMFS_OVERLAY_STORAGE_DEVICE}"
>>>>    	ovl_mount_option="${INITRAMFS_OVERLAY_MOUNT_OPTION}"
>>>> @@ -21,6 +26,8 @@ if ! mountpoint -q 
>>>> "${rootmnt}${storage_mount_point}"; then
>>>>    
>>>>    	partition_fstype=$(get_fstype "${ovl_partition_device}")
>>>>    
>>>> +	# FS checks should be moved to a separate hook / or at least initramfs lib
>>>> +	# so generic functionality can be reused here!
>>>>    	case $partition_fstype in
>>>>    	ext*)
>>>>    		fsck_ret=0
>>>> @@ -33,32 +40,97 @@ if ! mountpoint -q "${rootmnt}${storage_mount_point}"; then
>>>>    		fi
>>>>    		;;
>>>>    	esac
>>>> +
>>>> +	# Mount overlay device to initramfs - will be moved to rootfs at the end
>>>> +	[ "$debug" = "y" ] && echo "Mount overlay device to initramfs - will be moved to rootfs at the end"
>>>>    	if ! mount -t ${partition_fstype} \
>>>>    		 -o ${ovl_mount_option} \
>>>>    		 ${ovl_partition_device} \
>>>> -		 ${rootmnt}${storage_mount_point}; then
>>>> +		 ${storage_mount_point}; then
>>>>    		panic "Can't mount ${storage_mount_point} partition - overlay will not work!"
>>>>    	fi
>>>> +else
>>>> +	# If the storage device is already mounted on the final rootfs
>>>> +	# move it to the initramfs. Otherwise, we cannot move back the
>>>> +	# storage mount to the resulting rootfs in the end.
>>>> +
>>>> +	[ "$debug" = "y" ] && echo "Move ${rootmnt}${storage_mount_point} to ${storage_mount_point}"
>>>> +	mount -n -o move ${rootmnt}${storage_mount_point} 
>>>> +${storage_mount_point}
>>>>    fi
>>>>    
>>>> +[ "$debug" = "y" ] && echo "Iterate overlay mount spec..."
>>>> +
>>>>    for ovl_lower_dir in ${ovl_lower_dirs}; do
>>>> -	# remove the first '/' and replace all '/' with '_'
>>>> -	# on variable $ovl_lower_dir
>>>> -	work_dir_name=$(awk -v var=$ovl_lower_dir \
>>>> -		'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
>>>> +	root_moved=false
>>>> +
>>>> +	if [ $ovl_lower_dir != "/" ]; then
>>>> +		log_begin_msg "Mounting overlay $ovl_lower_dir"
>>>> +
>>>> +		# remove the first '/' and replace all '/' with '_'
>>>> +		# on variable $ovl_lower_dir
>>>> +		work_dir_name=$(awk -v var=$ovl_lower_dir \
>>>> +			'BEGIN{subvar=substr(var,2);gsub("/","_",subvar);print subvar}')
>>>> +
>>>> +		lower_dir=${rootmnt}${ovl_lower_dir}
>>>> +		upper_dir=${root_mount_storage}${ovl_lower_dir}
>>>> +		work_dir=${root_mount_storage}/.${work_dir_name}-atomic
>>>> +		mount_dir=${lower_dir}
>>>> +	else
>>>> +		log_begin_msg "Mounting root overlay"
>>>> +
>>>> +		# When overlaying root we need to move the mountpoint
>>>> +		# from ${rootmnt} to a dedicated mountpoint inside
>>>> +		# the storage dir to prevent a "recursion" within
>>>> +		# ${lowerdir} and the final mountpoint of the overlay: ${rootmnt}
>>>> +		# While this is not a limitation of overlay itself
>>>> +		# such "recursion" leaves the mountpoint for lower-dir (the ro-rootfs)
>>>> +		# outside the tree of new root which panics when switching root
>>>> +
>>>> +		# create lower dir to move ro-rootfs to
>>>> +		lower_dir=${root_mount_storage}/lower-root
>>>> +		mkdir -p ${lower_dir}
>>>> +
>>>> +		# make the readonly root available
>>>> +		mount -n -o move ${rootmnt} ${lower_dir}
>>>> +		root_moved=true
>>>>    
>>>> -	lower_dir=${rootmnt}${ovl_lower_dir}
>>>> -	upper_dir=${root_mount_storage}${ovl_lower_dir}
>>>> -	work_dir=${root_mount_storage}/.${work_dir_name}-atomic
>>>> +		upper_dir=${root_mount_storage}/root
>>>> +		work_dir=${root_mount_storage}/.root-atomic
>>>> +
>>>> +		mount_dir=${rootmnt}
>>>> +	fi
>>>>    
>>>>    	mkdir -p ${upper_dir}
>>>>    	mkdir -p ${work_dir}
>>>>    
>>>> +	[ "$debug" = "y" ] && echo "Setup overlay L:${lower_dir} + U:${upper_dir} -> ${mount_dir}"
>>>>    	if ! mount -t overlay \
>>>>    		   -o lowerdir=${lower_dir},upperdir=${upper_dir},workdir=${work_dir} \
>>>> -		   overlay ${lower_dir}; then
>>>> +		   overlay ${mount_dir}; then
>>>>    		panic "Can't mount overlay for '$ovl_lower_dir' !"
>>>>    	fi
>>>> +
>>>> +	if ${root_moved}; then
>>>> +		# Loop over all mountpoints already setup before we established the root overlay
>>>> +		# and move them to the resulting overlayed rootfs
>>>> +		[ "$debug" = "y" ] && echo "Move all mountpoints already setup before we established the root overlay..."
>>>> +		cat /proc/mounts | grep "${root_mount_storage}/lower-root/" | awk '{print $2}' | \
>>>> +			while read -r old_mountpoint
>>>> +			do
>>>> +				new_mountpoint="${rootmnt}/${old_mountpoint#${root_mount_storage}/lower-root/}"
>>>> +
>>>> +				[ "$debug" = "y" ] && echo "Move ${old_mountpoint} to ${new_mountpoint}"
>>>> +				mount -n -o move ${old_mountpoint} ${new_mountpoint}
>>>> +			done
>>>> +	fi
>>>> +
>>>> +	log_end_msg
>>>>    done
>>>>    
>>>> +# Finally, move the overlay workdir from initramfs to the final 
>>>> +rootfs [ "$debug" = "y" ] && echo "Move ${storage_mount_point} to ${rootmnt}${storage_mount_point}"
>>>> +mount -n -o move ${storage_mount_point} 
>>>> +${rootmnt}${storage_mount_point}
>>>> +
>>>> +log_success_msg "Overlays setup successfully"
>>>> +
>>>>    exit 0
>>>> diff --git
>>>> a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
>>>> bb
>>>> b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
>>>> bb
>>>> index ec7f85b..af610e5 100644
>>>> ---
>>>> a/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hook_0.3.
>>>> bb
>>>> +++ b/recipes-initramfs/initramfs-overlay-hook/initramfs-overlay-hoo
>>>> +++ k_
>>>> +++ 0.3.bb
>>>> @@ -45,7 +45,7 @@ TEMPLATE_VARS += " INITRAMFS_OVERLAY_STORAGE_PATH \
>>>>    DEBIAN_DEPENDS .= ", awk, coreutils, util-linux"
>>>>    
>>>>    HOOK_ADD_MODULES = "overlay"
>>>> -HOOK_COPY_EXECS = "mountpoint awk e2fsck mke2fs"
>>>> +HOOK_COPY_EXECS = "mkdir mountpoint awk e2fsck mke2fs grep"
>>>>    
>>>>    SCRIPT_PREREQ="crypt"
>>>>    
>>

Re: [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Jan Kiszka]

On 24.02.25 13:49, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>> On 2/20/25 14:00, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>> On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
>>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>
>>>>> In addition to the overlays for e.g. /etc we now support persistent 
>>>>> overlays for rootfs as well.
>>>>> To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
>>>>> or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
>>>>>
>>>>
>>>> I would like to avoid that anti pattern of creating an overlay over the complete rootfs. What are the debug case you need a full overlay?
>>>>
>>> To be very clear: This is not meant for debugging in the field! Never, ever.
>>>
>>> The intention of having a root-overlay for us was to have an easy way 
>>> to develop and extend recipes based on our existing images. That 
>>> means, we wanted to have the possibility to install packages with apt, 
>>> or copy additional binaries, services, configs on the device without 
>>> having to rebuild our image all the time, still maintaining to have a 
>>> working environment as close as possible to the production image.
>>> That enables us to develop features in a debian native environment, 
>>> still being close to the features present in the final target image.
>>> After being confident, the steps done and changes reflected on the 
>>> overlay can be transferred fairly easy to a recipe and end up in a production image (image without the overlay).
>>
>> Ok, this still does not explain why
>> Overlaying `/usr` should be enough for these use cases ?
> 
> I explained that in the thread with Ludwig: 
> "
> Probably that's the case for most partitionings. Still having a complete 
> root overlay is more generic imo.
> But I am open for that!
> "
> 
>>
>> Also this kind of testing  ensures that you will not find issues realated to read only rootfs in your packages during development.
> 
> Same holds true for overlaying /usr. Once I have fundamental parts of the
> ro filesystem overlayed, issues may be hidden during that time.
> Anyhow, as said, this is only a feature we enable when developing new recipes.
> Once the recipe is done we rebuild an image without root-overlay and 
> testing is done without such overlay!
> 
> TLDR: I am also fine with overlaying just /usr to simplify the setup.
> The initial idea was to have a generic solution for a full overlay
> including custom downstream partitionings / mountpoints for external storage.
> 
> But I have to admit, it brings additional complexity and if we decide against
> such generic approach I am also fine with that!
> 

Should we simply revert patch 1 until it is clear if we still need it
(in a different form then)?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Heinisch, Alexander]

>On 24.02.25 13:17, Quirin Gylstorff wrote:
>> 
>> 
>> On 2/24/25 07:09, Jan Kiszka wrote:
>>> On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> 
>>>>> On Behalf Of Heinisch, Alexander via lists.cip-project.org
>>>>> Sent: Montag, 17. Februar 2025 11:00
>>>>> To: cip-dev@lists.cip-project.org
>>>>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, 
>>>>> Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; 
>>>>> Heinisch, Alexander (FT RPD CED SES-AT) 
>>>>> <alexander.heinisch@siemens.com>
>>>>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option 
>>>>> from read-only-rootfs's fstab.
>>>>>
>>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>
>>>>> Since we only support erofs and squashfs this option is not needed 
>>>>> any more. Further, it causes potential overlay rootfs variants to 
>>>>> be remounted read-only.
>>>>>
>>>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>> ---
>>>>>   classes/read-only-rootfs.bbclass | 2 +-
>>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only- 
>>>>> rootfs.bbclass index 35a3ab3..83ddc33 100644
>>>>> --- a/classes/read-only-rootfs.bbclass
>>>>> +++ b/classes/read-only-rootfs.bbclass
>>>>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>>>>   image_configure_fstab() {
>>>>>       sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin 
>>>>> /etc/fstab -/dev/root    /        auto        defaults,ro            
>>>>> 0    0
>>>>> +/dev/root    /        auto        defaults            0    0
>>>>>   LABEL=var    /var        auto        defaults            0    0
>>>>>   proc        /proc        proc        nosuid,noexec,nodev
>>>>> 0    0
>>>>>   sysfs        /sys        sysfs        nosuid,noexec,nodev
>>>>> 0    0
>>>>> --
>>>>> 2.39.5
>>>>
>>>> We observed `systemd-remount-fs` failing in a downstream layer.
>>>> (using a ro squashfs and apply full disk encryption on top) 
>>>> Although, the service is executed in isar-cip-core as well, 
>>>> upstream, the service can be executed successfully.
>>>>
>>>> In essence what happens is that, the `systemd-remount-fs` remounts 
>>>> various mountpoints based on their settings in `/etc/fstab`
>>>>
>>>> ```
>>>> root@device-mgmt:~# cat /etc/fstab
>>>> # Begin /etc/fstab
>>>> /dev/root    /        auto        defaults            0    0 
>>>> LABEL=var    /var        auto        defaults            0    0 proc        
>>>> /proc        proc        nosuid,noexec,nodev        0    0 sysfs        
>>>> /sys        sysfs        nosuid,noexec,nodev        0    0 devpts        
>>>> /dev/pts    devpts        gid=5,mode=620            0    0 tmpfs        
>>>> /run        tmpfs
>>>> nodev,nosuid,size=500M,mode=755    0    0 devtmpfs    /dev        
>>>> devtmpfs    mode=0755,nosuid        0    0 # End /etc/fstab ```
>>>>
>>>> Since `defaults` in fstab renders to:
>>>> `rw,suid,dev,exec,auto,nouser,async`
>>>> the service fails trying to remount the `squashfs` as `rw` as shown
>>>> below:
>>>>
>>>> ```
>>>> root@device-mgmt:~# systemctl status systemd-remount-fs × 
>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems
>>>>       Loaded: loaded 
>>>> (/lib/systemd/system/systemd-remount-fs.service;
>>>> enabled-ru>
>>>>       Active: failed (Result: exit-code) since Fri 2025-02-21
>>>> 08:55:39 UTC; 5min>
>>>>         Docs: man:systemd-remount-fs.service(8)
>>>>               
>>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
>>>> w.freedesktop.org%2Fwiki%2FSoftware%2Fsystemd%2F&data=05%7C02%7Calex
>>>> ander.heinisch%40siemens.com%7Cde0b4a7d3b494f87bd1a08dd54d0ac65%7C38
>>>> ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638759977392750433%7CUnknow
>>>> n%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJX
>>>> aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X5%2BBYi
>>>> WGiccJJ8qUcawdQ40DExa71H3avr0h15jJaU4%3D&reserved=0
>>>> APIFileSystems
>>>>      Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs
>>>> (code=exited, statu>
>>>>     Main PID: 7530 (code=exited, status=1/FAILURE)
>>>>          CPU: 15ms
>>>> ```
>>>>
>>>> ```
>>>> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs Feb 
>>>> 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd- 
>>>> remount-fs.service - Remount Root and Kernel File Systems...
>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /:
>>>> cannot remount /dev/root read-write, is write-protected.
>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:
>>>> dmesg(1) may have more information after failed mount system call.
>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/ 
>>>> mount for / exited with exit status 32.
>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>> fs.service: Main process exited, code=exited, status=1/FAILURE Feb 
>>>> 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>> fs.service: Failed with result 'exit-code'.
>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start 
>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems.
>>>> ```
>>>>
>>>> The rootfs is mounted as follows:
>>>>
>>>> ```
>>>> root@device-mgmt:~# mount | grep " / "
>>>> /dev/mapper/verityroot on / type squashfs 
>>>> (ro,relatime,errors=continue) ```
>>>>
>>>> I understand why this happens on our system, but could not clarify 
>>>> why it does not happen upstream! So any idea, welcome!
>>>>
>>>
>>> The same issue happens in upstream: secure boot on, data partition
>>> encrypted, remount-fs fails. Which makes me wonder if our boot tests
>>> should check for failing system services...

Such check for failing services definitely makes sense!

The good thing, I could reproduce using the config you described.
The bad news, it still works with a slightly different config.

TLDR:
In the failing setup we use dm-verity and in
the working setup we use the A/B-rootfs.

In both cases the filesystem is a squashfs 
(once mounted directly and once via dm-verity mapping)
Thus, in both cases it cannot be mounted rw.
Options used to remount are similar.


Details:

In my isar-cip-core setup this does not happen!

I did a clean build using ./kas-container build using following config:

```
$ cat .config.yaml
#
# Automatically generated by kas 4.7
#
_source_dir: /repo
build_system: isar
header:
  includes:
  - kas-cip.yml
  - kas/board/qemu-amd64.yml
  - kas/opt/6.1.yml
  - kas/opt/bookworm.yml
  - kas/opt/expand-on-first-boot.yml
  - kas/opt/ebg-swu.yml
  - kas/opt/encrypt-data.yml
  - kas/opt/encrypt-all.yml
  version: 18
local_conf_header:
  __menu_config_vars: WDOG_TIMEOUT = "60"
menu_configuration:
  DEBIAN_BOOKWORM: true
  DEBIAN_BULLSEYE: false
  DEBIAN_BUSTER: false
  DEBIAN_TRIXIE: false
  EXPAND_ON_FIRST_BOOT: true
  IMAGE_COMPLETE_SWUPDATE: true
  IMAGE_DATA_ENCRYPTION: true
  IMAGE_DELTA_SWUPDATE: false
  IMAGE_FULL_ENCRYPTION: true
  IMAGE_RO_ROOTFS_EROFS: false
  IMAGE_RO_ROOTFS_SQUASHFS: true
  IMAGE_SECURE_BOOT: false
  IMAGE_SECURITY: false
  IMAGE_SWUPDATE: true
  IMAGE_TESTING: false
  KERNEL_4_19: false
  KERNEL_4_4: false
  KERNEL_5_10: false
  KERNEL_6_1: true
  KERNEL_RT: false
  TARGET_BBB: false
  TARGET_HIHOPE_RZG2M: false
  TARGET_IWG20D: false
  TARGET_QEMU_AMD64: true
  TARGET_QEMU_ARM: false
  TARGET_QEMU_ARM64: false
  TARGET_QEMU_RISCV64: false
  TARGET_X86_UEFI: false
  WDOG_TIMEOUT: 60
```

VM Setup: `./start-qemu.sh`

During bootup:
```
...
[  441.107438] systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[  OK  ] Finished systemd-remount-f…ount Root and Kernel File Systems.
...
```

```
# systemctl list-units --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.

# systemctl status systemd-remount-fs
● systemd-remount-fs.service - Remount Root and Kernel File Systems
     Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: enabled)
     Active: active (exited) since Mon 2025-02-24 10:16:29 UTC; 3min 5s ago
       Docs: man:systemd-remount-fs.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
    Process: 472 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, status=0/SUCCESS)
   Main PID: 472 (code=exited, status=0/SUCCESS)
        CPU: 868ms
```

```
# cat /etc/fstab 
# Begin /etc/fstab
/dev/root	/		auto		defaults			0	0
LABEL=var	/var		auto		defaults			0	0
proc		/proc		proc		nosuid,noexec,nodev		0	0
sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
devpts		/dev/pts	devpts		gid=5,mode=620			0	0
tmpfs		/run		tmpfs		nodev,nosuid,size=500M,mode=755	0	0
devtmpfs	/dev		devtmpfs	mode=0755,nosuid		0	0
# End /etc/fstab
```

```
# mount | grep " / "
/dev/mapper/fedcba98-7654-3210-cafe-5e0710000001 on / type squashfs (ro,relatime,errors=continue)
```

-------------
-------------
-------------

But I can confirm in a setup like you mentioned it is reproducible (for a reasons
I don't understand yet):

```
$ cat .config.yaml
#
# Automatically generated by kas 4.7
#
_source_dir: /repo
build_system: isar
header:
  includes:
  - kas-cip.yml
  - kas/board/qemu-amd64.yml
  - kas/opt/6.1.yml
  - kas/opt/bookworm.yml
  - kas/opt/expand-on-first-boot.yml
  - kas/opt/ebg-secure-boot-snakeoil.yml
  - kas/opt/encrypt-data.yml
  - kas/opt/encrypt-all.yml
  version: 18
local_conf_header:
  __menu_config_vars: WDOG_TIMEOUT = "60"
menu_configuration:
  DEBIAN_BOOKWORM: true
  DEBIAN_BULLSEYE: false
  DEBIAN_BUSTER: false
  DEBIAN_TRIXIE: false
  EXPAND_ON_FIRST_BOOT: true
  IMAGE_COMPLETE_SWUPDATE: true
  IMAGE_DATA_ENCRYPTION: true
  IMAGE_DELTA_SWUPDATE: false
  IMAGE_FULL_ENCRYPTION: true
  IMAGE_RO_ROOTFS_EROFS: false
  IMAGE_RO_ROOTFS_SQUASHFS: true
  IMAGE_SECURE_BOOT: true
  IMAGE_SECURITY: false
  IMAGE_SWUPDATE: true
  IMAGE_TESTING: false
  KERNEL_4_19: false
  KERNEL_4_4: false
  KERNEL_5_10: false
  KERNEL_6_1: true
  KERNEL_RT: false
  TARGET_BBB: false
  TARGET_HIHOPE_RZG2M: false
  TARGET_IWG20D: false
  TARGET_QEMU_AMD64: true
  TARGET_QEMU_ARM: false
  TARGET_QEMU_ARM64: false
  TARGET_QEMU_RISCV64: false
  TARGET_X86_UEFI: false
  WDOG_TIMEOUT: 60
```

```
[FAILED] Failed to start systemd-re…ount Root and Kernel File Systems.
See 'systemctl status systemd-remount-fs.service' for details.
```

```
# systemctl list-units --failed
  UNIT                       LOAD   ACTIVE SUB    DESCRIPTION                         
● systemd-remount-fs.service loaded failed failed Remount Root and Kernel File Systems

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
```

```
# systemctl status systemd-remount-fs
× systemd-remount-fs.service - Remount Root and Kernel File Systems
     Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: enabled)
     Active: failed (Result: exit-code) since Mon 2025-02-24 11:47:05 UTC; 1min 35s ago
       Docs: man:systemd-remount-fs.service(8)
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
    Process: 2665 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE)
   Main PID: 2665 (code=exited, status=1/FAILURE)
        CPU: 794ms

Feb 24 11:47:05 demo systemd-remount-fs[2665]: /bin/mount for / exited with exit status 32.
Feb 24 11:47:05 demo systemd-remount-fs[2668]: mount: /: cannot remount /dev/root read-write, is write-protected.
Feb 24 11:47:05 demo systemd-remount-fs[2668]:        dmesg(1) may have more information after failed mount system call.
Feb 24 11:47:05 demo systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 11:47:05 demo systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Feb 24 11:47:05 demo systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
Notice: journal has been rotated since unit was started, output may be incomplete.

# journalctl -u systemd-remount-fs
Feb 24 11:47:05 demo systemd-remount-fs[2665]: /bin/mount for / exited with exit status 32.
Feb 24 11:47:05 demo systemd-remount-fs[2668]: mount: /: cannot remount /dev/root read-write, is write-protected.
Feb 24 11:47:05 demo systemd-remount-fs[2668]:        dmesg(1) may have more information after failed mount system call.
Feb 24 11:47:05 demo systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Feb 24 11:47:05 demo systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Feb 24 11:47:05 demo systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
```

```
# cat /etc/fstab 
# Begin /etc/fstab
/dev/root	/		auto		defaults			0	0
LABEL=var	/var		auto		defaults			0	0
proc		/proc		proc		nosuid,noexec,nodev		0	0
sysfs		/sys		sysfs		nosuid,noexec,nodev		0	0
devpts		/dev/pts	devpts		gid=5,mode=620			0	0
tmpfs		/run		tmpfs		nodev,nosuid,size=500M,mode=755	0	0
devtmpfs	/dev		devtmpfs	mode=0755,nosuid		0	0
# End /etc/fstab
```

```
# mount | grep " / "
/dev/mapper/verityroot on / type squashfs (ro,relatime,errors=continue)
```

>>>
>>>> Nevertheless, after rethinking this patch. Maybe it was advisible to
>>>> not use
>>>> `default` here and either use a options like
>>>> `suid,dev,exec,auto,nouser,async`
>>>> or drop the line for `/dev/root` at all, keeping the mounts as they
>>>> were setup
>>>> in the initramfs.
>>>>
>>>> Any opinions, recommendations, hints, ...?
>>>>
>>>
>>> According to the man pages, a left-out rw/ro means that mount will first
>>> try rw and then, if it fails, ro. That would give us the desired
>>> flexibility. Hard-coding the other values of "defaults" is the downside.
>>>
>> Will no also lead to error of systemd-remount?
>> 
>
>It does not help, yes, just tried in vain.
>


From: https://man7.org/linux/man-pages/man8/mount.8.html
```
 Note that the real set of all default mount options depends on
           the kernel and filesystem type. See the beginning of this
           section for more details.
```

So having "defaults" kind of depends on multiple factors, so it seems to be
a better fit to specify the options we actually want leaving out the ro and rw parts,
so fallback-behaviour as described above applies.


>>> On the other hand, what would be the mount options that the initramfs
>>> will use? We do not have direct control over them as well, do we?
>> 
>> The initramfs in standard Debian contains the fstab - so it would/could
>> use the options from there.
>>>
>>> Quirin, other reasons why we want / in our fstab?
>> 
>> The fstab of the readonly rootfs is an evolution of the original fstab
>> from ISAR. It is also there as we disabled the fstab generation from
>> wic.
>> 
>>>
>>> Another option: "ro" a parameter of read-only-rootfs.bbclass so that
>>> your scenario can override the default.
>> 
>> I would make it to option for all the fstab options.
>> 
>
>Ok, let's go for some config variable then.

Would not like to set another option for something which is obvious by the filessystem type already.

Short term: Since fstab has no real use in most of the configurations 
(except for mounting `/var` when using unencrypted data partitions)
I'd simply drop the `/` entry in fstab.

But before, I would like to understand, why this happens at all!



---
Offtopic: Long term we should reconsolidate usage of fstab / cryptab, but probably that
goes along with a broader discussion on systemd based initramfs hooks / dracut / ...:
For fstab there are primarily 2 options,
1.) rely on fstab, then we or some (systemd) generator tool should put everything 
    we mount in fstab (incl. encryption setup via `/etc/crypttab`) or 
2.) not rely on fstab at all, thus dropping the `/` entry and probably also
    put `/var` for unencrypted setups to initramfs (also with focus for A/B 
	data partitions).
---

>
>Jan
>
>-- 
>Siemens AG, Foundational Technologies
>Linux Expert Center

Sry for the long post,

BR Alexander

RE: [isar-cip-core][PATCH 3/3] Added support for rootfs-overlay.

[Heinisch, Alexander]

>On 24.02.25 13:49, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>> On 2/20/25 14:00, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>>> On 2/17/25 11:00, alexander.heinisch@siemens.com wrote:
>>>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>>
>>>>>> In addition to the overlays for e.g. /etc we now support 
>>>>>> persistent overlays for rootfs as well.
>>>>>> To enable this behaviour you can either set INITRAMFS_OVERLAY_PATHS = "/"
>>>>>> or extend your existing config INITRAMFS_OVERLAY_PATHS = "/ /etc"
>>>>>>
>>>>>
>>>>> I would like to avoid that anti pattern of creating an overlay over the complete rootfs. What are the debug case you need a full overlay?
>>>>>
>>>> To be very clear: This is not meant for debugging in the field! Never, ever.
>>>>
>>>> The intention of having a root-overlay for us was to have an easy 
>>>> way to develop and extend recipes based on our existing images. That 
>>>> means, we wanted to have the possibility to install packages with 
>>>> apt, or copy additional binaries, services, configs on the device 
>>>> without having to rebuild our image all the time, still maintaining 
>>>> to have a working environment as close as possible to the production image.
>>>> That enables us to develop features in a debian native environment, 
>>>> still being close to the features present in the final target image.
>>>> After being confident, the steps done and changes reflected on the 
>>>> overlay can be transferred fairly easy to a recipe and end up in a production image (image without the overlay).
>>>
>>> Ok, this still does not explain why
>>> Overlaying `/usr` should be enough for these use cases ?
>> 
>> I explained that in the thread with Ludwig: 
>> "
>> Probably that's the case for most partitionings. Still having a 
>> complete root overlay is more generic imo.
>> But I am open for that!
>> "
>> 
>>>
>>> Also this kind of testing  ensures that you will not find issues realated to read only rootfs in your packages during development.
>> 
>> Same holds true for overlaying /usr. Once I have fundamental parts of 
>> the ro filesystem overlayed, issues may be hidden during that time.
>> Anyhow, as said, this is only a feature we enable when developing new recipes.
>> Once the recipe is done we rebuild an image without root-overlay and 
>> testing is done without such overlay!
>> 
>> TLDR: I am also fine with overlaying just /usr to simplify the setup.
>> The initial idea was to have a generic solution for a full overlay 
>> including custom downstream partitionings / mountpoints for external storage.
>> 
>> But I have to admit, it brings additional complexity and if we decide 
>> against such generic approach I am also fine with that!
>> 
>
>Should we simply revert patch 1 until it is clear if we still need it (in a different form then)?

Yes! (sry for that, ...)

But no matter if we keep the root-overlay or not, we should find
clarity on that issue.

BR Alexander

>
>Jan
>
>--
>Siemens AG, Foundational Technologies
>Linux Expert Center
>

Re: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Jan Kiszka]

On 24.02.25 14:03, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>> On 24.02.25 13:17, Quirin Gylstorff wrote:
>>>
>>>
>>> On 2/24/25 07:09, Jan Kiszka wrote:
>>>> On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> 
>>>>>> On Behalf Of Heinisch, Alexander via lists.cip-project.org
>>>>>> Sent: Montag, 17. Februar 2025 11:00
>>>>>> To: cip-dev@lists.cip-project.org
>>>>>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, 
>>>>>> Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; 
>>>>>> Heinisch, Alexander (FT RPD CED SES-AT) 
>>>>>> <alexander.heinisch@siemens.com>
>>>>>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option 
>>>>>> from read-only-rootfs's fstab.
>>>>>>
>>>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>>
>>>>>> Since we only support erofs and squashfs this option is not needed 
>>>>>> any more. Further, it causes potential overlay rootfs variants to 
>>>>>> be remounted read-only.
>>>>>>
>>>>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>> ---
>>>>>>   classes/read-only-rootfs.bbclass | 2 +-
>>>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only- 
>>>>>> rootfs.bbclass index 35a3ab3..83ddc33 100644
>>>>>> --- a/classes/read-only-rootfs.bbclass
>>>>>> +++ b/classes/read-only-rootfs.bbclass
>>>>>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>>>>>   image_configure_fstab() {
>>>>>>       sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin 
>>>>>> /etc/fstab -/dev/root    /        auto        defaults,ro            
>>>>>> 0    0
>>>>>> +/dev/root    /        auto        defaults            0    0
>>>>>>   LABEL=var    /var        auto        defaults            0    0
>>>>>>   proc        /proc        proc        nosuid,noexec,nodev
>>>>>> 0    0
>>>>>>   sysfs        /sys        sysfs        nosuid,noexec,nodev
>>>>>> 0    0
>>>>>> --
>>>>>> 2.39.5
>>>>>
>>>>> We observed `systemd-remount-fs` failing in a downstream layer.
>>>>> (using a ro squashfs and apply full disk encryption on top) 
>>>>> Although, the service is executed in isar-cip-core as well, 
>>>>> upstream, the service can be executed successfully.
>>>>>
>>>>> In essence what happens is that, the `systemd-remount-fs` remounts 
>>>>> various mountpoints based on their settings in `/etc/fstab`
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# cat /etc/fstab
>>>>> # Begin /etc/fstab
>>>>> /dev/root    /        auto        defaults            0    0 
>>>>> LABEL=var    /var        auto        defaults            0    0 proc        
>>>>> /proc        proc        nosuid,noexec,nodev        0    0 sysfs        
>>>>> /sys        sysfs        nosuid,noexec,nodev        0    0 devpts        
>>>>> /dev/pts    devpts        gid=5,mode=620            0    0 tmpfs        
>>>>> /run        tmpfs
>>>>> nodev,nosuid,size=500M,mode=755    0    0 devtmpfs    /dev        
>>>>> devtmpfs    mode=0755,nosuid        0    0 # End /etc/fstab ```
>>>>>
>>>>> Since `defaults` in fstab renders to:
>>>>> `rw,suid,dev,exec,auto,nouser,async`
>>>>> the service fails trying to remount the `squashfs` as `rw` as shown
>>>>> below:
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# systemctl status systemd-remount-fs × 
>>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems
>>>>>       Loaded: loaded 
>>>>> (/lib/systemd/system/systemd-remount-fs.service;
>>>>> enabled-ru>
>>>>>       Active: failed (Result: exit-code) since Fri 2025-02-21
>>>>> 08:55:39 UTC; 5min>
>>>>>         Docs: man:systemd-remount-fs.service(8)
>>>>>               
>>>>> https://ww
>>>>> w.freedesktop.org%2Fwiki%2FSoftware%2Fsystemd%2F&data=05%7C02%7Calex
>>>>> ander.heinisch%40siemens.com%7Cde0b4a7d3b494f87bd1a08dd54d0ac65%7C38
>>>>> ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638759977392750433%7CUnknow
>>>>> n%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJX
>>>>> aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X5%2BBYi
>>>>> WGiccJJ8qUcawdQ40DExa71H3avr0h15jJaU4%3D&reserved=0
>>>>> APIFileSystems
>>>>>      Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs
>>>>> (code=exited, statu>
>>>>>     Main PID: 7530 (code=exited, status=1/FAILURE)
>>>>>          CPU: 15ms
>>>>> ```
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs Feb 
>>>>> 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd- 
>>>>> remount-fs.service - Remount Root and Kernel File Systems...
>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /:
>>>>> cannot remount /dev/root read-write, is write-protected.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:
>>>>> dmesg(1) may have more information after failed mount system call.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/ 
>>>>> mount for / exited with exit status 32.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>>> fs.service: Main process exited, code=exited, status=1/FAILURE Feb 
>>>>> 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>>> fs.service: Failed with result 'exit-code'.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start 
>>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems.
>>>>> ```
>>>>>
>>>>> The rootfs is mounted as follows:
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# mount | grep " / "
>>>>> /dev/mapper/verityroot on / type squashfs 
>>>>> (ro,relatime,errors=continue) ```
>>>>>
>>>>> I understand why this happens on our system, but could not clarify 
>>>>> why it does not happen upstream! So any idea, welcome!
>>>>>
>>>>
>>>> The same issue happens in upstream: secure boot on, data partition
>>>> encrypted, remount-fs fails. Which makes me wonder if our boot tests
>>>> should check for failing system services...
> 
> Such check for failing services definitely makes sense!
> 
> The good thing, I could reproduce using the config you described.
> The bad news, it still works with a slightly different config.
> 
> TLDR:
> In the failing setup we use dm-verity and in
> the working setup we use the A/B-rootfs.
> 
> In both cases the filesystem is a squashfs 
> (once mounted directly and once via dm-verity mapping)
> Thus, in both cases it cannot be mounted rw.
> Options used to remount are similar.
> 
> 
> Details:
> 
> In my isar-cip-core setup this does not happen!
> 
> I did a clean build using ./kas-container build using following config:
> 

<snip>

Disable full disk encryption and also exand-on-first-boot to get the
error with upstream.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Heinisch, Alexander]

>On 24.02.25 14:03, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>> On 24.02.25 13:17, Quirin Gylstorff wrote:
>>>>
>>>>
>>>> On 2/24/25 07:09, Jan Kiszka wrote:
>>>>> On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>>>>> From: cip-dev@lists.cip-project.org 
>>>>>>> <cip-dev@lists.cip-project.org> On Behalf Of Heinisch, Alexander 
>>>>>>> via lists.cip-project.org
>>>>>>> Sent: Montag, 17. Februar 2025 11:00
>>>>>>> To: cip-dev@lists.cip-project.org
>>>>>>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, 
>>>>>>> Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; 
>>>>>>> Heinisch, Alexander (FT RPD CED SES-AT) 
>>>>>>> <alexander.heinisch@siemens.com>
>>>>>>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option 
>>>>>>> from read-only-rootfs's fstab.
>>>>>>>
>>>>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>>>
>>>>>>> Since we only support erofs and squashfs this option is not 
>>>>>>> needed any more. Further, it causes potential overlay rootfs 
>>>>>>> variants to be remounted read-only.
>>>>>>>
>>>>>>> Signed-off-by: Alexander Heinisch 
>>>>>>> <alexander.heinisch@siemens.com>
>>>>>>> ---
>>>>>>>   classes/read-only-rootfs.bbclass | 2 +-
>>>>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/classes/read-only-rootfs.bbclass 
>>>>>>> b/classes/read-only- rootfs.bbclass index 35a3ab3..83ddc33 100644
>>>>>>> --- a/classes/read-only-rootfs.bbclass
>>>>>>> +++ b/classes/read-only-rootfs.bbclass
>>>>>>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>>>>>>   image_configure_fstab() {
>>>>>>>       sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin 
>>>>>>> /etc/fstab -/dev/root    /        auto        defaults,ro            
>>>>>>> 0    0
>>>>>>> +/dev/root    /        auto        defaults            0    0
>>>>>>>   LABEL=var    /var        auto        defaults            0    0
>>>>>>>   proc        /proc        proc        nosuid,noexec,nodev
>>>>>>> 0    0
>>>>>>>   sysfs        /sys        sysfs        nosuid,noexec,nodev
>>>>>>> 0    0
>>>>>>> --
>>>>>>> 2.39.5
>>>>>>
>>>>>> We observed `systemd-remount-fs` failing in a downstream layer.
>>>>>> (using a ro squashfs and apply full disk encryption on top) 
>>>>>> Although, the service is executed in isar-cip-core as well, 
>>>>>> upstream, the service can be executed successfully.
>>>>>>
>>>>>> In essence what happens is that, the `systemd-remount-fs` remounts 
>>>>>> various mountpoints based on their settings in `/etc/fstab`
>>>>>>
>>>>>> ```
>>>>>> root@device-mgmt:~# cat /etc/fstab # Begin /etc/fstab
>>>>>> /dev/root    /        auto        defaults            0    0 
>>>>>> LABEL=var    /var        auto        defaults            0    0 proc        
>>>>>> /proc        proc        nosuid,noexec,nodev        0    0 sysfs        
>>>>>> /sys        sysfs        nosuid,noexec,nodev        0    0 devpts        
>>>>>> /dev/pts    devpts        gid=5,mode=620            0    0 tmpfs        
>>>>>> /run        tmpfs
>>>>>> nodev,nosuid,size=500M,mode=755    0    0 devtmpfs    /dev        
>>>>>> devtmpfs    mode=0755,nosuid        0    0 # End /etc/fstab ```
>>>>>>
>>>>>> Since `defaults` in fstab renders to:
>>>>>> `rw,suid,dev,exec,auto,nouser,async`
>>>>>> the service fails trying to remount the `squashfs` as `rw` as 
>>>>>> shown
>>>>>> below:
>>>>>>
>>>>>> ```
>>>>>> root@device-mgmt:~# systemctl status systemd-remount-fs × 
>>>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems
>>>>>>       Loaded: loaded
>>>>>> (/lib/systemd/system/systemd-remount-fs.service;
>>>>>> enabled-ru>
>>>>>>       Active: failed (Result: exit-code) since Fri 2025-02-21
>>>>>> 08:55:39 UTC; 5min>
>>>>>>         Docs: man:systemd-remount-fs.service(8)
>>>>>>               
>>>>>> https://ww
>>>>>> w.freedesktop.org%2Fwiki%2FSoftware%2Fsystemd%2F&data=05%7C02%7Cal
>>>>>> ex
>>>>>> ander.heinisch%40siemens.com%7Cde0b4a7d3b494f87bd1a08dd54d0ac65%7C
>>>>>> 38 
>>>>>> ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638759977392750433%7CUnkn
>>>>>> ow 
>>>>>> n%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOi
>>>>>> JX 
>>>>>> aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X5%2BB
>>>>>> Yi
>>>>>> WGiccJJ8qUcawdQ40DExa71H3avr0h15jJaU4%3D&reserved=0
>>>>>> APIFileSystems
>>>>>>      Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs
>>>>>> (code=exited, statu>
>>>>>>     Main PID: 7530 (code=exited, status=1/FAILURE)
>>>>>>          CPU: 15ms
>>>>>> ```
>>>>>>
>>>>>> ```
>>>>>> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs 
>>>>>> Feb
>>>>>> 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd- 
>>>>>> remount-fs.service - Remount Root and Kernel File Systems...
>>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /:
>>>>>> cannot remount /dev/root read-write, is write-protected.
>>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:
>>>>>> dmesg(1) may have more information after failed mount system call.
>>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/ 
>>>>>> mount for / exited with exit status 32.
>>>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>>>> fs.service: Main process exited, code=exited, status=1/FAILURE Feb
>>>>>> 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>>>> fs.service: Failed with result 'exit-code'.
>>>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start 
>>>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems.
>>>>>> ```
>>>>>>
>>>>>> The rootfs is mounted as follows:
>>>>>>
>>>>>> ```
>>>>>> root@device-mgmt:~# mount | grep " / "
>>>>>> /dev/mapper/verityroot on / type squashfs
>>>>>> (ro,relatime,errors=continue) ```
>>>>>>
>>>>>> I understand why this happens on our system, but could not clarify 
>>>>>> why it does not happen upstream! So any idea, welcome!
>>>>>>
>>>>>
>>>>> The same issue happens in upstream: secure boot on, data partition 
>>>>> encrypted, remount-fs fails. Which makes me wonder if our boot 
>>>>> tests should check for failing system services...
>> 
>> Such check for failing services definitely makes sense!
>> 
>> The good thing, I could reproduce using the config you described.
>> The bad news, it still works with a slightly different config.
>> 
>> TLDR:
>> In the failing setup we use dm-verity and in the working setup we use 
>> the A/B-rootfs.
>> 
>> In both cases the filesystem is a squashfs (once mounted directly and 
>> once via dm-verity mapping) Thus, in both cases it cannot be mounted 
>> rw.
>> Options used to remount are similar.
>> 
>> 
>> Details:
>> 
>> In my isar-cip-core setup this does not happen!
>> 
>> I did a clean build using ./kas-container build using following config:
>> 
>
><snip>
>
>Disable full disk encryption and also exand-on-first-boot to get the error with upstream.

No, that's not the case!
The difference in failing or not is by enabling verity setup.

works: (no encryption, no verity)
/dev/nvme0n1p4 on / type squashfs (ro,relatime,errors=continue)

fails: (no encryption, verity)
/dev/mapper/verityroot on / type squashfs (ro,relatime,errors=continue)

BR Alexander

>
>Jan
>
>--
>Siemens AG, Foundational Technologies
>Linux Expert Center
>