overlayfs-etc on top of dm-verity?

overlayfs-etc on top of dm-verity?

[Michael Opdenacker]

Hi Slava and community,

Do you know if overlayfs and in particular our overlayfs-etc class works 
when /etc is on a dm-verity root filesystem?

Without dm-verity (regular ext4 or erofs root filesystem), everything 
looks all right:
# mount | grep overlay
/data/overlay-etc/upper on /etc type overlay 
(rw,relatime,lowerdir=/etc,upperdir=/data/overlay-etc/upper,workdir=/data/overlay-etc/work,uuid=on)

When /etc is on /dev/mapper/rootfs (dm-verity), everything seems messed up:
# mount | grep overlay
overlay on /var/cache type overlay 
(rw,relatime,lowerdir=/var/cache,upperdir=/var/volatile/cache,workdir=/var/volatile/.cache-work,uuid=on)
overlay on /var/lib type overlay 
(rw,relatime,lowerdir=/var/lib,upperdir=/var/volatile/lib,workdir=/var/volatile/.lib-work,uuid=on)
overlay on /var/spool type overlay 
(rw,relatime,lowerdir=/var/spool,upperdir=/var/volatile/spool,workdir=/var/volatile/.spool-work,uuid=on)
overlay on /srv type overlay 
(rw,relatime,lowerdir=/srv,upperdir=/var/volatile/srv,workdir=/var/volatile/.srv-work,uuid=on)

Systemd may be messing up, as only in this case, it does:
          Starting Bind mount volatile /var/cache...
          Starting Bind mount volatile /var/lib...
          Starting Bind mount volatile /var/spool...
          Starting Bind mount volatile /srv...

But these bind mounts show up as overlay mounts!

Has anyone already encountered such an issue?
Thanks in advance,
Cheers
Michael.

-- 
Root Commit
Embedded Linux Training and Consulting
https://rootcommit.com

Re: overlayfs-etc on top of dm-verity?

[Michael Opdenacker]

Greetings,

On 3/25/26 10:20 PM, Michael Opdenacker wrote:
> Hi Slava and community,
>
> Do you know if overlayfs and in particular our overlayfs-etc class 
> works when /etc is on a dm-verity root filesystem?
>
> Without dm-verity (regular ext4 or erofs root filesystem), everything 
> looks all right:
> # mount | grep overlay
> /data/overlay-etc/upper on /etc type overlay 
> (rw,relatime,lowerdir=/etc,upperdir=/data/overlay-etc/upper,workdir=/data/overlay-etc/work,uuid=on)
>
> When /etc is on /dev/mapper/rootfs (dm-verity), everything seems 
> messed up:
> # mount | grep overlay
> overlay on /var/cache type overlay 
> (rw,relatime,lowerdir=/var/cache,upperdir=/var/volatile/cache,workdir=/var/volatile/.cache-work,uuid=on)
> overlay on /var/lib type overlay 
> (rw,relatime,lowerdir=/var/lib,upperdir=/var/volatile/lib,workdir=/var/volatile/.lib-work,uuid=on)
> overlay on /var/spool type overlay 
> (rw,relatime,lowerdir=/var/spool,upperdir=/var/volatile/spool,workdir=/var/volatile/.spool-work,uuid=on)
> overlay on /srv type overlay 
> (rw,relatime,lowerdir=/srv,upperdir=/var/volatile/srv,workdir=/var/volatile/.srv-work,uuid=on)
>
> Systemd may be messing up, as only in this case, it does:
>          Starting Bind mount volatile /var/cache...
>          Starting Bind mount volatile /var/lib...
>          Starting Bind mount volatile /var/spool...
>          Starting Bind mount volatile /srv...
>
> But these bind mounts show up as overlay mounts!
>
> Has anyone already encountered such an issue?

I eventually managed to get /etc mounted as an overlay. It seems that 
/sbin/init was started instead of /sbin/preinit as specified in the 
kernel command line.
I hardcoded the call to /sbin/preinit by customizing 
openembedded-core/meta/recipes-core/initrdscripts/initramfs-framework/finish 
(in a bbappend file, of course).

The code looks right though, I need to understand why this happens.

Another weirdness that remains is these volatile mounts for /var/cache/, 
/var/lib, /var/spool and /srv, which I didn't have with a regular 
read-only root filesystem.
I'll keep you posted.
Cheers
Michael.

-- 
Root Commit
Embedded Linux Training and Consulting
https://rootcommit.com

Re: overlayfs-etc on top of dm-verity?

[Michael Opdenacker]

Greetings,

So follow-up on this issue...

On 3/26/26 10:56 AM, Michael Opdenacker wrote:
>
>
> On 3/25/26 10:20 PM, Michael Opdenacker wrote:
>
> I eventually managed to get /etc mounted as an overlay. It seems that 
> /sbin/init was started instead of /sbin/preinit as specified in the 
> kernel command line.
> I hardcoded the call to /sbin/preinit by customizing 
> openembedded-core/meta/recipes-core/initrdscripts/initramfs-framework/finish 
> (in a bbappend file, of course).
>
> The code looks right though, I need to understand why this happens.


The issue (the initramfs scripts not calling the init script specified 
in the kernel command line) was caused by an issue with kernel 
parameters like:
opt="value", that happen when you use the kernel's "bootconfig" 
configuration options.

This fix I've just submitted solves the issue: 
https://lore.kernel.org/openembedded-core/20260326173432.3286250-1-michael.opdenacker@rootcommit.com/T/#u

Cheers
Michael.

Root Commit
Embedded Linux Training and Consulting
https://rootcommit.com

Re: [yocto] overlayfs-etc on top of dm-verity?

[Francesco Valla]

Hi Michael,

On Thu, Mar 26, 2026 at 09:56:21AM +0000, Michael Opdenacker via lists.yoctoproject.org wrote:
> Greetings,
> 
> On 3/25/26 10:20 PM, Michael Opdenacker wrote:
> > Hi Slava and community,
> > 
> > Do you know if overlayfs and in particular our overlayfs-etc class works
> > when /etc is on a dm-verity root filesystem?
> > 
> > Without dm-verity (regular ext4 or erofs root filesystem), everything
> > looks all right:
> > # mount | grep overlay
> > /data/overlay-etc/upper on /etc type overlay (rw,relatime,lowerdir=/etc,upperdir=/data/overlay-etc/upper,workdir=/data/overlay-etc/work,uuid=on)
> > 
> > When /etc is on /dev/mapper/rootfs (dm-verity), everything seems messed
> > up:
> > # mount | grep overlay
> > overlay on /var/cache type overlay (rw,relatime,lowerdir=/var/cache,upperdir=/var/volatile/cache,workdir=/var/volatile/.cache-work,uuid=on)
> > overlay on /var/lib type overlay (rw,relatime,lowerdir=/var/lib,upperdir=/var/volatile/lib,workdir=/var/volatile/.lib-work,uuid=on)
> > overlay on /var/spool type overlay (rw,relatime,lowerdir=/var/spool,upperdir=/var/volatile/spool,workdir=/var/volatile/.spool-work,uuid=on)
> > overlay on /srv type overlay (rw,relatime,lowerdir=/srv,upperdir=/var/volatile/srv,workdir=/var/volatile/.srv-work,uuid=on)
> > 
> > Systemd may be messing up, as only in this case, it does:
> > � � � � �Starting Bind mount volatile /var/cache...
> > � � � � �Starting Bind mount volatile /var/lib...
> > � � � � �Starting Bind mount volatile /var/spool...
> > � � � � �Starting Bind mount volatile /srv...
> > 
> > But these bind mounts show up as overlay mounts!
> > 
> > Has anyone already encountered such an issue?
> 
> I eventually managed to get /etc mounted as an overlay. It seems that
> /sbin/init was started instead of /sbin/preinit as specified in the kernel
> command line.
> I hardcoded the call to /sbin/preinit by customizing
> openembedded-core/meta/recipes-core/initrdscripts/initramfs-framework/finish
> (in a bbappend file, of course).
> 
> The code looks right though, I need to understand why this happens.
> 
> Another weirdness that remains is these volatile mounts for /var/cache/,
> /var/lib, /var/spool and /srv, which I didn't have with a regular read-only
> root filesystem.

AFAIK, this should be the regular behavior on a read-only root
filesystem. The overlayfs mounts are created by services generated by:

  meta/recipes-core/volatile-binds/volatile-binds.bb

depending on the content of the VOLATILE_BINDS variable. For each couple
of upperdir-lowerdir specified there, a service is generated that
starts only if upperdir's parent is writable and lowerdir is not.

E.g.:
  lowerdir=/srv
  upperdir=/var/volatile/srv

In a vanilla openembedded-core system, a tmpfs is mounted on /var/volatile
by the fstab (that is, by the fstab systemd generator), so the
upperdir's parent directory (which is the same /var/volatile) is writable.

You can force a copy+bind behavior setting AVOID_OVERLAYFS=1.

> I'll keep you posted.
> Cheers
> Michael.
> 
> -- 
> Root Commit
> Embedded Linux Training and Consulting
> https://rootcommit.com
> 

Best regards,
Francesco

Re: [yocto] overlayfs-etc on top of dm-verity?

[Michael Opdenacker]

Hi Francesco,

On 3/26/26 11:19 PM, Francesco Valla wrote:
>> The code looks right though, I need to understand why this happens.
>>
>> Another weirdness that remains is these volatile mounts for /var/cache/,
>> /var/lib, /var/spool and /srv, which I didn't have with a regular read-only
>> root filesystem.
> AFAIK, this should be the regular behavior on a read-only root
> filesystem. The overlayfs mounts are created by services generated by:
>
>    meta/recipes-core/volatile-binds/volatile-binds.bb
>
> depending on the content of the VOLATILE_BINDS variable. For each couple
> of upperdir-lowerdir specified there, a service is generated that
> starts only if upperdir's parent is writable and lowerdir is not.
>
> E.g.:
>    lowerdir=/srv
>    upperdir=/var/volatile/srv
>
> In a vanilla openembedded-core system, a tmpfs is mounted on /var/volatile
> by the fstab (that is, by the fstab systemd generator), so the
> upperdir's parent directory (which is the same /var/volatile) is writable.
>
> You can force a copy+bind behavior setting AVOID_OVERLAYFS=1.

I didn't thank you for these great clarifications and tips. It's good to 
understand the "magic".
I eventually dropped overlayfs-etc and started implementing specific 
mount points using VOLATILE_BINDS indeed (with AVOID_OVERLAYFS=1).

It's a very easy to use mechanism thanks to the volatile-binds.bb recipe.
Thanks again, you made my day!
Cheers
Michael.

-- 
Root Commit
Embedded Linux Training and Consulting
https://rootcommit.com

Re: overlayfs-etc on top of dm-verity?

[Ayoub Zaki]

[-- Attachment #1: Type: text/plain, Size: 374 bytes --]

Hi Michael,
From a security perspective I would strongly advise against overlaying the
entire /etc as it undermines the integrity provided by secure boot. Instead
only overlay the specific files that actually need to be modified. In
addition, consider enforcing integrity protection on the upper layer and
consider alternatively switch to bind mounts for those files.

Best

[-- Attachment #2: Type: text/html, Size: 464 bytes --]

Re: overlayfs-etc on top of dm-verity?

[Michael Opdenacker]

Hi Ayoub,

On 3/27/26 2:26 PM, Ayoub Zaki wrote:
> Hi Michael,
> From a security perspective I would strongly advise against overlaying 
> the entire /etc as it undermines the integrity provided by secure 
> boot. Instead only overlay the specific files that actually need to be 
> modified. In addition, consider enforcing integrity protection on the 
> upper layer and consider alternatively switch to bind mounts for those 
> files.
>
Thanks for the advice and reminder! This definitely makes sense and I'll 
review which /etc files we need to modify, if any. I initially suggested 
this to allow for changing root/user passwords at first boot, but we may 
be able to do without users and passwords after all.

Cheers
Michael.

-- 
Root Commit
Embedded Linux Training and Consulting
https://rootcommit.com