Re: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option from read-only-rootfs's fstab.

[Jan Kiszka <jan.kiszka@siemens.com>]

On 24.02.25 14:03, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>> On 24.02.25 13:17, Quirin Gylstorff wrote:
>>>
>>>
>>> On 2/24/25 07:09, Jan Kiszka wrote:
>>>> On 21.02.25 13:31, Heinisch, Alexander (FT RPD CED SES-AT) wrote:
>>>>>> From: cip-dev@lists.cip-project.org <cip-dev@lists.cip-project.org> 
>>>>>> On Behalf Of Heinisch, Alexander via lists.cip-project.org
>>>>>> Sent: Montag, 17. Februar 2025 11:00
>>>>>> To: cip-dev@lists.cip-project.org
>>>>>> Cc: Kiszka, Jan (FT RPD CED) <jan.kiszka@siemens.com>; Gylstorff, 
>>>>>> Quirin (FT RPD CED OES-DE) <quirin.gylstorff@siemens.com>; 
>>>>>> Heinisch, Alexander (FT RPD CED SES-AT) 
>>>>>> <alexander.heinisch@siemens.com>
>>>>>> Subject: [cip-dev] [isar-cip-core][PATCH 1/3] Removed "ro" option 
>>>>>> from read-only-rootfs's fstab.
>>>>>>
>>>>>> From: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>>
>>>>>> Since we only support erofs and squashfs this option is not needed 
>>>>>> any more. Further, it causes potential overlay rootfs variants to 
>>>>>> be remounted read-only.
>>>>>>
>>>>>> Signed-off-by: Alexander Heinisch <alexander.heinisch@siemens.com>
>>>>>> ---
>>>>>>   classes/read-only-rootfs.bbclass | 2 +-
>>>>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/classes/read-only-rootfs.bbclass b/classes/read-only- 
>>>>>> rootfs.bbclass index 35a3ab3..83ddc33 100644
>>>>>> --- a/classes/read-only-rootfs.bbclass
>>>>>> +++ b/classes/read-only-rootfs.bbclass
>>>>>> @@ -41,7 +41,7 @@ SQUASHFS_EXCLUDE_DIRS = "${RO_ROOTFS_EXCLUDE_DIRS}"
>>>>>>   image_configure_fstab() {
>>>>>>       sudo tee '${IMAGE_ROOTFS}/etc/fstab' << EOF  # Begin 
>>>>>> /etc/fstab -/dev/root    /        auto        defaults,ro            
>>>>>> 0    0
>>>>>> +/dev/root    /        auto        defaults            0    0
>>>>>>   LABEL=var    /var        auto        defaults            0    0
>>>>>>   proc        /proc        proc        nosuid,noexec,nodev
>>>>>> 0    0
>>>>>>   sysfs        /sys        sysfs        nosuid,noexec,nodev
>>>>>> 0    0
>>>>>> --
>>>>>> 2.39.5
>>>>>
>>>>> We observed `systemd-remount-fs` failing in a downstream layer.
>>>>> (using a ro squashfs and apply full disk encryption on top) 
>>>>> Although, the service is executed in isar-cip-core as well, 
>>>>> upstream, the service can be executed successfully.
>>>>>
>>>>> In essence what happens is that, the `systemd-remount-fs` remounts 
>>>>> various mountpoints based on their settings in `/etc/fstab`
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# cat /etc/fstab
>>>>> # Begin /etc/fstab
>>>>> /dev/root    /        auto        defaults            0    0 
>>>>> LABEL=var    /var        auto        defaults            0    0 proc        
>>>>> /proc        proc        nosuid,noexec,nodev        0    0 sysfs        
>>>>> /sys        sysfs        nosuid,noexec,nodev        0    0 devpts        
>>>>> /dev/pts    devpts        gid=5,mode=620            0    0 tmpfs        
>>>>> /run        tmpfs
>>>>> nodev,nosuid,size=500M,mode=755    0    0 devtmpfs    /dev        
>>>>> devtmpfs    mode=0755,nosuid        0    0 # End /etc/fstab ```
>>>>>
>>>>> Since `defaults` in fstab renders to:
>>>>> `rw,suid,dev,exec,auto,nouser,async`
>>>>> the service fails trying to remount the `squashfs` as `rw` as shown
>>>>> below:
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# systemctl status systemd-remount-fs × 
>>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems
>>>>>       Loaded: loaded 
>>>>> (/lib/systemd/system/systemd-remount-fs.service;
>>>>> enabled-ru>
>>>>>       Active: failed (Result: exit-code) since Fri 2025-02-21
>>>>> 08:55:39 UTC; 5min>
>>>>>         Docs: man:systemd-remount-fs.service(8)
>>>>>               
>>>>> https://ww
>>>>> w.freedesktop.org%2Fwiki%2FSoftware%2Fsystemd%2F&data=05%7C02%7Calex
>>>>> ander.heinisch%40siemens.com%7Cde0b4a7d3b494f87bd1a08dd54d0ac65%7C38
>>>>> ae3bcd95794fd4addab42e1495d55a%7C1%7C0%7C638759977392750433%7CUnknow
>>>>> n%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJX
>>>>> aW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X5%2BBYi
>>>>> WGiccJJ8qUcawdQ40DExa71H3avr0h15jJaU4%3D&reserved=0
>>>>> APIFileSystems
>>>>>      Process: 7530 ExecStart=/lib/systemd/systemd-remount-fs
>>>>> (code=exited, statu>
>>>>>     Main PID: 7530 (code=exited, status=1/FAILURE)
>>>>>          CPU: 15ms
>>>>> ```
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# journalctl --no-pager -u systemd-remount-fs Feb 
>>>>> 21 08:55:39 device-mgmt.local systemd[1]: Starting systemd- 
>>>>> remount-fs.service - Remount Root and Kernel File Systems...
>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]: mount: /:
>>>>> cannot remount /dev/root read-write, is write-protected.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7531]:
>>>>> dmesg(1) may have more information after failed mount system call.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd-remount-fs[7530]: /bin/ 
>>>>> mount for / exited with exit status 32.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>>> fs.service: Main process exited, code=exited, status=1/FAILURE Feb 
>>>>> 21 08:55:39 device-mgmt.local systemd[1]: systemd-remount-
>>>>> fs.service: Failed with result 'exit-code'.
>>>>> Feb 21 08:55:39 device-mgmt.local systemd[1]: Failed to start 
>>>>> systemd-remount-fs.service - Remount Root and Kernel File Systems.
>>>>> ```
>>>>>
>>>>> The rootfs is mounted as follows:
>>>>>
>>>>> ```
>>>>> root@device-mgmt:~# mount | grep " / "
>>>>> /dev/mapper/verityroot on / type squashfs 
>>>>> (ro,relatime,errors=continue) ```
>>>>>
>>>>> I understand why this happens on our system, but could not clarify 
>>>>> why it does not happen upstream! So any idea, welcome!
>>>>>
>>>>
>>>> The same issue happens in upstream: secure boot on, data partition
>>>> encrypted, remount-fs fails. Which makes me wonder if our boot tests
>>>> should check for failing system services...
> 
> Such check for failing services definitely makes sense!
> 
> The good thing, I could reproduce using the config you described.
> The bad news, it still works with a slightly different config.
> 
> TLDR:
> In the failing setup we use dm-verity and in
> the working setup we use the A/B-rootfs.
> 
> In both cases the filesystem is a squashfs 
> (once mounted directly and once via dm-verity mapping)
> Thus, in both cases it cannot be mounted rw.
> Options used to remount are similar.
> 
> 
> Details:
> 
> In my isar-cip-core setup this does not happen!
> 
> I did a clean build using ./kas-container build using following config:
> 

<snip>

Disable full disk encryption and also exand-on-first-boot to get the
error with upstream.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center