Frequent errors with OverlayFS on root

Frequent errors with OverlayFS on root

[Mauro Condarelli]

Hi,
most likely this is not the right place to ask, please redirect me as needed.

I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
rootfs (SquashFS)

Essentially (actual script is more complex, of course) boot-sequence includes:

# /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
# /dev/mmcblk0p6: SquashFS mounted on /
mount /dev/mmcblk0p5 /overlay
mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
cd /overlay/newroot
pivot_root . oldroot
mount --move oldroot/dev /dev
mount --move oldroot/proc /proc

This works as expected, but, too often for comfort, some file
(and sometime also directories) become unavailable due to error:

overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).

File name changes, of course, but rest is fairly constant.

This always happens when some file is written.
Error persists reboots.
Only way I found to "cure" the system is to go on "upper" and delete the file
thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")

This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
on a custom target based on a SoC (MT7628).

I am available to do any required test, but I have no idea about where to start.

Any hint (or redirect) would be greatly appreciated.

Many Thanks in Advance
Mauro

Re: Frequent errors with OverlayFS on root

[Amir Goldstein]

On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
>
> Hi,
> most likely this is not the right place to ask, please redirect me as needed.
>
> I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
> rootfs (SquashFS)
>
> Essentially (actual script is more complex, of course) boot-sequence includes:
>
> # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
> # /dev/mmcblk0p6: SquashFS mounted on /
> mount /dev/mmcblk0p5 /overlay
> mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
> cd /overlay/newroot
> pivot_root . oldroot
> mount --move oldroot/dev /dev
> mount --move oldroot/proc /proc
>
> This works as expected, but, too often for comfort, some file
> (and sometime also directories) become unavailable due to error:
>
> overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
>
> File name changes, of course, but rest is fairly constant.
>
> This always happens when some file is written.
> Error persists reboots.
> Only way I found to "cure" the system is to go on "upper" and delete the file
> thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
>
> This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
> on a custom target based on a SoC (MT7628).
>
> I am available to do any required test, but I have no idea about where to start.
>
> Any hint (or redirect) would be greatly appreciated.

This is probably your problem:
https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/

If it is, it should be solved by commit a888db310195
ovl: fix regression with re-formatted lower squashfs
in upstream kernel v5.9-rc1 or in stable kernel >= 5.7.10.

Thanks,
Amir.

Re: Frequent errors with OverlayFS on root

[Amir Goldstein]

On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
>
> Thanks Amir,
> comments inline below
>
> Regards
> Mauro
>
> On 9/1/20 8:43 PM, Amir Goldstein wrote:
>
> On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
>
> Hi,
> most likely this is not the right place to ask, please redirect me as needed.
>
> I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
> rootfs (SquashFS)
>
> Essentially (actual script is more complex, of course) boot-sequence includes:
>
> # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
> # /dev/mmcblk0p6: SquashFS mounted on /
> mount /dev/mmcblk0p5 /overlay
> mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
> cd /overlay/newroot
> pivot_root . oldroot
> mount --move oldroot/dev /dev
> mount --move oldroot/proc /proc
>
> This works as expected, but, too often for comfort, some file
> (and sometime also directories) become unavailable due to error:
>
> overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
>
> File name changes, of course, but rest is fairly constant.
>
> This always happens when some file is written.
> Error persists reboots.
> Only way I found to "cure" the system is to go on "upper" and delete the file
> thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
>
> This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
> on a custom target based on a SoC (MT7628).
>
> I am available to do any required test, but I have no idea about where to start.
>
> Any hint (or redirect) would be greatly appreciated.
>
> This is probably your problem:
> https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
>
> It surely looks like it is.
> I tried to follow the thread, but I'm unsure i grokked it .
>
> I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
> be able to update the whole system (changing SquashFS) while
> retaining customization (essentially tweaks in /etc/, host certificates
> and similar stuff) in an embedded target.
>
> To complicate matters I also have a dual system for fallback in case
> of faulty upgrade; this means I can switch from:
>   lower=part6, upper=part5, update will go into part7
> to
>   lower=part7, upper=part5, update will go into part6
>
> I don't need nfs at all, so exporting OverlayFS is not an issue,
> but it's unclear to me if this "lower swapping" is
> actually supported as I see:
>
> > > /me is again wondering what's the use case of modifying lower layer
> > > with an existing upper. Is it fair to say, no don't recreate/modify
> > > lower layers and use with existing upper.
> > >
> >
> > It's fine by me to document that this is not supported.
>
> ... which is scary.
> Note I do *not* need to modify lower on-the-fly, when I
> swap systems, that will happen at reboot.

Don't be scared :-)
Your reaction is a good answer to Vivek's question above -
No, it is not fair to say don't re-create lower and use existing upper

But we may decide this is forbidden in case user opts-in to
new features. The mentioned fix commit fixes the regression
for mounts without mount options (index, xino, metacopy),
just like your use case.

>
> If it is, it should be solved by commit a888db310195
> ovl: fix regression with re-formatted lower squashfs
> in upstream kernel v5.9-rc1 or in stable kernel >= 5.7.10.
>
> I'm currently at kernel v5.7.0, but upgrading to
> stable v5.8.5 shouldn't be a problem (I would like
> to avoid Release Candidate kernels, if possible).
>
> Can You confirm v5.8.+ is supposed to work in my
> use case?

Yes, it was fixed in v5.8.0 (and not in v5.9-rc1 as I wrote).

Thanks,
Amir.

Re: Frequent errors with OverlayFS on root

[Vivek Goyal]

On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote:
> On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
> >
> > Thanks Amir,
> > comments inline below
> >
> > Regards
> > Mauro
> >
> > On 9/1/20 8:43 PM, Amir Goldstein wrote:
> >
> > On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
> >
> > Hi,
> > most likely this is not the right place to ask, please redirect me as needed.
> >
> > I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
> > rootfs (SquashFS)
> >
> > Essentially (actual script is more complex, of course) boot-sequence includes:
> >
> > # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
> > # /dev/mmcblk0p6: SquashFS mounted on /
> > mount /dev/mmcblk0p5 /overlay
> > mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
> > cd /overlay/newroot
> > pivot_root . oldroot
> > mount --move oldroot/dev /dev
> > mount --move oldroot/proc /proc
> >
> > This works as expected, but, too often for comfort, some file
> > (and sometime also directories) become unavailable due to error:
> >
> > overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
> >
> > File name changes, of course, but rest is fairly constant.
> >
> > This always happens when some file is written.
> > Error persists reboots.
> > Only way I found to "cure" the system is to go on "upper" and delete the file
> > thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
> >
> > This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
> > on a custom target based on a SoC (MT7628).
> >
> > I am available to do any required test, but I have no idea about where to start.
> >
> > Any hint (or redirect) would be greatly appreciated.
> >
> > This is probably your problem:
> > https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
> >
> > It surely looks like it is.
> > I tried to follow the thread, but I'm unsure i grokked it .
> >
> > I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
> > be able to update the whole system (changing SquashFS) while
> > retaining customization (essentially tweaks in /etc/, host certificates
> > and similar stuff) in an embedded target.
> >
> > To complicate matters I also have a dual system for fallback in case
> > of faulty upgrade; this means I can switch from:
> >   lower=part6, upper=part5, update will go into part7
> > to
> >   lower=part7, upper=part5, update will go into part6
> >
> > I don't need nfs at all, so exporting OverlayFS is not an issue,
> > but it's unclear to me if this "lower swapping" is
> > actually supported as I see:
> >
> > > > /me is again wondering what's the use case of modifying lower layer
> > > > with an existing upper. Is it fair to say, no don't recreate/modify
> > > > lower layers and use with existing upper.
> > > >
> > >
> > > It's fine by me to document that this is not supported.
> >
> > ... which is scary.
> > Note I do *not* need to modify lower on-the-fly, when I
> > swap systems, that will happen at reboot.
> 
> Don't be scared :-)
> Your reaction is a good answer to Vivek's question above -
> No, it is not fair to say don't re-create lower and use existing upper

This seems like a wrong use case to me. So in above example, say
following happens.

- Mount overlay
- modify sshd_config (it gets copied up).
- Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated.
- Mount overlay and user now sees old copied up sshd_config. (If it
  works).

Conceptually it is not making much sense to me. What's the point of
upgrading lower because after mounting overlay you might still see
old file. And to make it worse, behavior is underministic beacuse
if file has not been copied up, you will see new file.

To me, this sounds like a volatile overlay use case. Any changes to
overlay should be thrown away when lower sqashfs is updated and a
fresh upper should be set.

Thanks
Vivek

Re: Frequent errors with OverlayFS on root

[Amir Goldstein]

On Wed, Sep 2, 2020 at 4:29 PM Vivek Goyal <vgoyal@redhat.com> wrote:
>
> On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote:
> > On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
> > >
> > > Thanks Amir,
> > > comments inline below
> > >
> > > Regards
> > > Mauro
> > >
> > > On 9/1/20 8:43 PM, Amir Goldstein wrote:
> > >
> > > On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
> > >
> > > Hi,
> > > most likely this is not the right place to ask, please redirect me as needed.
> > >
> > > I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
> > > rootfs (SquashFS)
> > >
> > > Essentially (actual script is more complex, of course) boot-sequence includes:
> > >
> > > # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
> > > # /dev/mmcblk0p6: SquashFS mounted on /
> > > mount /dev/mmcblk0p5 /overlay
> > > mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
> > > cd /overlay/newroot
> > > pivot_root . oldroot
> > > mount --move oldroot/dev /dev
> > > mount --move oldroot/proc /proc
> > >
> > > This works as expected, but, too often for comfort, some file
> > > (and sometime also directories) become unavailable due to error:
> > >
> > > overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
> > >
> > > File name changes, of course, but rest is fairly constant.
> > >
> > > This always happens when some file is written.
> > > Error persists reboots.
> > > Only way I found to "cure" the system is to go on "upper" and delete the file
> > > thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
> > >
> > > This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
> > > on a custom target based on a SoC (MT7628).
> > >
> > > I am available to do any required test, but I have no idea about where to start.
> > >
> > > Any hint (or redirect) would be greatly appreciated.
> > >
> > > This is probably your problem:
> > > https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
> > >
> > > It surely looks like it is.
> > > I tried to follow the thread, but I'm unsure i grokked it .
> > >
> > > I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
> > > be able to update the whole system (changing SquashFS) while
> > > retaining customization (essentially tweaks in /etc/, host certificates
> > > and similar stuff) in an embedded target.
> > >
> > > To complicate matters I also have a dual system for fallback in case
> > > of faulty upgrade; this means I can switch from:
> > >   lower=part6, upper=part5, update will go into part7
> > > to
> > >   lower=part7, upper=part5, update will go into part6
> > >
> > > I don't need nfs at all, so exporting OverlayFS is not an issue,
> > > but it's unclear to me if this "lower swapping" is
> > > actually supported as I see:
> > >
> > > > > /me is again wondering what's the use case of modifying lower layer
> > > > > with an existing upper. Is it fair to say, no don't recreate/modify
> > > > > lower layers and use with existing upper.
> > > > >
> > > >
> > > > It's fine by me to document that this is not supported.
> > >
> > > ... which is scary.
> > > Note I do *not* need to modify lower on-the-fly, when I
> > > swap systems, that will happen at reboot.
> >
> > Don't be scared :-)
> > Your reaction is a good answer to Vivek's question above -
> > No, it is not fair to say don't re-create lower and use existing upper
>
> This seems like a wrong use case to me. So in above example, say
> following happens.
>
> - Mount overlay
> - modify sshd_config (it gets copied up).
> - Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated.
> - Mount overlay and user now sees old copied up sshd_config. (If it
>   works).
>
> Conceptually it is not making much sense to me. What's the point of
> upgrading lower because after mounting overlay you might still see
> old file. And to make it worse, behavior is underministic beacuse
> if file has not been copied up, you will see new file.
>

That is exactly the case with package managers that update system
files, except package managers can prompt user to resolve conflicts.

Think of this is a package manager that manages a single package
called "system". If that package manager (usually called firmware upgrade)
wanted to, it could compare for every upper file to origin from system image A
and to same path from system image B before making the upgrade and
prompt the user to resolve the conflict just like a regular package manager.

But if said package manager does not resolve conflicts it is the same as
a package manager configured to auto resolve conflicts by choosing the
local modification. Not perfect, but people do live with such setups even
without overlayfs.

Thanks,
Amir.

Re: Frequent errors with OverlayFS on root

[Mauro Condarelli]

Hi Vivek,
comments inline below.

On 9/2/20 3:29 PM, Vivek Goyal wrote:
> On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote:
>> On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
>>> Thanks Amir,
>>> comments inline below
>>>
>>> Regards
>>> Mauro
>>>
>>> On 9/1/20 8:43 PM, Amir Goldstein wrote:
>>>
>>> On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
>>>
>>> Hi,
>>> most likely this is not the right place to ask, please redirect me as needed.
>>>
>>> I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
>>> rootfs (SquashFS)
>>>
>>> Essentially (actual script is more complex, of course) boot-sequence includes:
>>>
>>> # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
>>> # /dev/mmcblk0p6: SquashFS mounted on /
>>> mount /dev/mmcblk0p5 /overlay
>>> mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
>>> cd /overlay/newroot
>>> pivot_root . oldroot
>>> mount --move oldroot/dev /dev
>>> mount --move oldroot/proc /proc
>>>
>>> This works as expected, but, too often for comfort, some file
>>> (and sometime also directories) become unavailable due to error:
>>>
>>> overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
>>>
>>> File name changes, of course, but rest is fairly constant.
>>>
>>> This always happens when some file is written.
>>> Error persists reboots.
>>> Only way I found to "cure" the system is to go on "upper" and delete the file
>>> thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
>>>
>>> This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
>>> on a custom target based on a SoC (MT7628).
>>>
>>> I am available to do any required test, but I have no idea about where to start.
>>>
>>> Any hint (or redirect) would be greatly appreciated.
>>>
>>> This is probably your problem:
>>> https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
>>>
>>> It surely looks like it is.
>>> I tried to follow the thread, but I'm unsure i grokked it .
>>>
>>> I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
>>> be able to update the whole system (changing SquashFS) while
>>> retaining customization (essentially tweaks in /etc/, host certificates
>>> and similar stuff) in an embedded target.
>>>
>>> To complicate matters I also have a dual system for fallback in case
>>> of faulty upgrade; this means I can switch from:
>>>   lower=part6, upper=part5, update will go into part7
>>> to
>>>   lower=part7, upper=part5, update will go into part6
>>>
>>> I don't need nfs at all, so exporting OverlayFS is not an issue,
>>> but it's unclear to me if this "lower swapping" is
>>> actually supported as I see:
>>>
>>>>> /me is again wondering what's the use case of modifying lower layer
>>>>> with an existing upper. Is it fair to say, no don't recreate/modify
>>>>> lower layers and use with existing upper.
>>>>>
>>>> It's fine by me to document that this is not supported.
>>> ... which is scary.
>>> Note I do *not* need to modify lower on-the-fly, when I
>>> swap systems, that will happen at reboot.
>> Don't be scared :-)
>> Your reaction is a good answer to Vivek's question above -
>> No, it is not fair to say don't re-create lower and use existing upper
> This seems like a wrong use case to me. So in above example, say
> following happens.
>
> - Mount overlay
> - modify sshd_config (it gets copied up).
> - Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated.
> - Mount overlay and user now sees old copied up sshd_config. (If it
>   works).
>
> Conceptually it is not making much sense to me. What's the point of
> upgrading lower because after mounting overlay you might still see
> old file. And to make it worse, behavior is underministic beacuse
> if file has not been copied up, you will see new file.
It might not make much sense to you, but it's EXACTLY what I need.
This OverlayFS is used *only* for permanent configuration.
Most of the times I'm adding new files (like host certificates and
similar things), but sometimes applications provide "default
configuration" I might need to override.
If/when applications "upgrade their defaults" I definitely do not
want them to override my choices.

> To me, this sounds like a volatile overlay use case. Any changes to
> overlay should be thrown away when lower sqashfs is updated and a
> fresh upper should be set.
I beg to differ.
While You are right in the general case it's possible to end up with
a tangled mess of old/new files none can sort out, your solution
resembles a lot to Alexander "solution" of the gordian knot.

In practice problems can (and will) arise in development
when most of changes are done by hand for testing (so some
unwanted "permanent" change can creep in; in production
there's a well defined flow that should prevent mishap.

Some of those configurations could be handled via other means,
but they are invariably more complex and less adaptable to
"unforeseen" needs/changes (which *will* happen).
> Thanks
> Vivek
Regards
Mauro

Re: Frequent errors with OverlayFS on root

[Vivek Goyal]

On Wed, Sep 02, 2020 at 05:33:53PM +0200, Mauro Condarelli wrote:
> Hi Vivek,
> comments inline below.
> 
> On 9/2/20 3:29 PM, Vivek Goyal wrote:
> > On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote:
> >> On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
> >>> Thanks Amir,
> >>> comments inline below
> >>>
> >>> Regards
> >>> Mauro
> >>>
> >>> On 9/1/20 8:43 PM, Amir Goldstein wrote:
> >>>
> >>> On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
> >>>
> >>> Hi,
> >>> most likely this is not the right place to ask, please redirect me as needed.
> >>>
> >>> I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
> >>> rootfs (SquashFS)
> >>>
> >>> Essentially (actual script is more complex, of course) boot-sequence includes:
> >>>
> >>> # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
> >>> # /dev/mmcblk0p6: SquashFS mounted on /
> >>> mount /dev/mmcblk0p5 /overlay
> >>> mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
> >>> cd /overlay/newroot
> >>> pivot_root . oldroot
> >>> mount --move oldroot/dev /dev
> >>> mount --move oldroot/proc /proc
> >>>
> >>> This works as expected, but, too often for comfort, some file
> >>> (and sometime also directories) become unavailable due to error:
> >>>
> >>> overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
> >>>
> >>> File name changes, of course, but rest is fairly constant.
> >>>
> >>> This always happens when some file is written.
> >>> Error persists reboots.
> >>> Only way I found to "cure" the system is to go on "upper" and delete the file
> >>> thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
> >>>
> >>> This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
> >>> on a custom target based on a SoC (MT7628).
> >>>
> >>> I am available to do any required test, but I have no idea about where to start.
> >>>
> >>> Any hint (or redirect) would be greatly appreciated.
> >>>
> >>> This is probably your problem:
> >>> https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
> >>>
> >>> It surely looks like it is.
> >>> I tried to follow the thread, but I'm unsure i grokked it .
> >>>
> >>> I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
> >>> be able to update the whole system (changing SquashFS) while
> >>> retaining customization (essentially tweaks in /etc/, host certificates
> >>> and similar stuff) in an embedded target.
> >>>
> >>> To complicate matters I also have a dual system for fallback in case
> >>> of faulty upgrade; this means I can switch from:
> >>>   lower=part6, upper=part5, update will go into part7
> >>> to
> >>>   lower=part7, upper=part5, update will go into part6
> >>>
> >>> I don't need nfs at all, so exporting OverlayFS is not an issue,
> >>> but it's unclear to me if this "lower swapping" is
> >>> actually supported as I see:
> >>>
> >>>>> /me is again wondering what's the use case of modifying lower layer
> >>>>> with an existing upper. Is it fair to say, no don't recreate/modify
> >>>>> lower layers and use with existing upper.
> >>>>>
> >>>> It's fine by me to document that this is not supported.
> >>> ... which is scary.
> >>> Note I do *not* need to modify lower on-the-fly, when I
> >>> swap systems, that will happen at reboot.
> >> Don't be scared :-)
> >> Your reaction is a good answer to Vivek's question above -
> >> No, it is not fair to say don't re-create lower and use existing upper
> > This seems like a wrong use case to me. So in above example, say
> > following happens.
> >
> > - Mount overlay
> > - modify sshd_config (it gets copied up).
> > - Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated.
> > - Mount overlay and user now sees old copied up sshd_config. (If it
> >   works).
> >
> > Conceptually it is not making much sense to me. What's the point of
> > upgrading lower because after mounting overlay you might still see
> > old file. And to make it worse, behavior is underministic beacuse
> > if file has not been copied up, you will see new file.
> It might not make much sense to you, but it's EXACTLY what I need.
> This OverlayFS is used *only* for permanent configuration.
> Most of the times I'm adding new files (like host certificates and
> similar things), but sometimes applications provide "default
> configuration" I might need to override.
> If/when applications "upgrade their defaults" I definitely do not
> want them to override my choices.

Some packages solved that problem by keeping system default
separate from user default files and over upgrade user files are
left untouched.

So basically squashfs image will be udpated. I am not sure how this
will work for all the cases. What if application is updated and
it decides to rename its config file from foo.txt to bar.txt. Now
when application is launched, user defaults are lost anyway.

IOW, what is rootfs image provider expecting. Are they expecting
overlayfs to be there and are they writing applications in backward
compatible manner. I don't generally boot into such systems so
I have no idea. 

Thanks
Vivek

Re: Frequent errors with OverlayFS on root

[Mauro Condarelli]

Hi Vivek,
comments inline below

Regards
Mauro

On 9/2/20 10:20 PM, Vivek Goyal wrote:
> On Wed, Sep 02, 2020 at 05:33:53PM +0200, Mauro Condarelli wrote:
>> Hi Vivek,
>> comments inline below.
>>
>> On 9/2/20 3:29 PM, Vivek Goyal wrote:
>>> On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote:
>>>> On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
>>>>> Thanks Amir,
>>>>> comments inline below
>>>>>
>>>>> Regards
>>>>> Mauro
>>>>>
>>>>> On 9/1/20 8:43 PM, Amir Goldstein wrote:
>>>>>
>>>>> On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
>>>>>
>>>>> Hi,
>>>>> most likely this is not the right place to ask, please redirect me as needed.
>>>>>
>>>>> I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
>>>>> rootfs (SquashFS)
>>>>>
>>>>> Essentially (actual script is more complex, of course) boot-sequence includes:
>>>>>
>>>>> # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
>>>>> # /dev/mmcblk0p6: SquashFS mounted on /
>>>>> mount /dev/mmcblk0p5 /overlay
>>>>> mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
>>>>> cd /overlay/newroot
>>>>> pivot_root . oldroot
>>>>> mount --move oldroot/dev /dev
>>>>> mount --move oldroot/proc /proc
>>>>>
>>>>> This works as expected, but, too often for comfort, some file
>>>>> (and sometime also directories) become unavailable due to error:
>>>>>
>>>>> overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
>>>>>
>>>>> File name changes, of course, but rest is fairly constant.
>>>>>
>>>>> This always happens when some file is written.
>>>>> Error persists reboots.
>>>>> Only way I found to "cure" the system is to go on "upper" and delete the file
>>>>> thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
>>>>>
>>>>> This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
>>>>> on a custom target based on a SoC (MT7628).
>>>>>
>>>>> I am available to do any required test, but I have no idea about where to start.
>>>>>
>>>>> Any hint (or redirect) would be greatly appreciated.
>>>>>
>>>>> This is probably your problem:
>>>>> https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
>>>>>
>>>>> It surely looks like it is.
>>>>> I tried to follow the thread, but I'm unsure i grokked it .
>>>>>
>>>>> I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
>>>>> be able to update the whole system (changing SquashFS) while
>>>>> retaining customization (essentially tweaks in /etc/, host certificates
>>>>> and similar stuff) in an embedded target.
>>>>>
>>>>> To complicate matters I also have a dual system for fallback in case
>>>>> of faulty upgrade; this means I can switch from:
>>>>>   lower=part6, upper=part5, update will go into part7
>>>>> to
>>>>>   lower=part7, upper=part5, update will go into part6
>>>>>
>>>>> I don't need nfs at all, so exporting OverlayFS is not an issue,
>>>>> but it's unclear to me if this "lower swapping" is
>>>>> actually supported as I see:
>>>>>
>>>>>>> /me is again wondering what's the use case of modifying lower layer
>>>>>>> with an existing upper. Is it fair to say, no don't recreate/modify
>>>>>>> lower layers and use with existing upper.
>>>>>>>
>>>>>> It's fine by me to document that this is not supported.
>>>>> ... which is scary.
>>>>> Note I do *not* need to modify lower on-the-fly, when I
>>>>> swap systems, that will happen at reboot.
>>>> Don't be scared :-)
>>>> Your reaction is a good answer to Vivek's question above -
>>>> No, it is not fair to say don't re-create lower and use existing upper
>>> This seems like a wrong use case to me. So in above example, say
>>> following happens.
>>>
>>> - Mount overlay
>>> - modify sshd_config (it gets copied up).
>>> - Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated.
>>> - Mount overlay and user now sees old copied up sshd_config. (If it
>>>   works).
>>>
>>> Conceptually it is not making much sense to me. What's the point of
>>> upgrading lower because after mounting overlay you might still see
>>> old file. And to make it worse, behavior is underministic beacuse
>>> if file has not been copied up, you will see new file.
>> It might not make much sense to you, but it's EXACTLY what I need.
>> This OverlayFS is used *only* for permanent configuration.
>> Most of the times I'm adding new files (like host certificates and
>> similar things), but sometimes applications provide "default
>> configuration" I might need to override.
>> If/when applications "upgrade their defaults" I definitely do not
>> want them to override my choices.
> Some packages solved that problem by keeping system default
> separate from user default files and over upgrade user files are
> left untouched.
>
> So basically squashfs image will be udpated. I am not sure how this
> will work for all the cases. What if application is updated and
> it decides to rename its config file from foo.txt to bar.txt. Now
> when application is launched, user defaults are lost anyway.
I'm in a somewhat middle position.

Targets are embedded systems and it wouldn't make any sense
to implement a fully flexible update system (deb/rpm/ipkg/...);
I just have two "packages": system and application.

I *am* the "rootfs (aka: system) image provider".

OTOH I'm not responsible for the zillions packages composing a RootFS.
A vast majority of packages provides "default configurations", often
completely commented-out and heavily commented "for reference".
I don't think it would be a good idea to inspect each and all the
packages to remove such "useless" files.

In some instances such configuration I need to override "globally"
(i.e.: for all targets in the same manner) and that's easy because
I can override in generated RootFS (lower SquashFS).

In other instances changes need to be done on a per-target
basis (note: I have ~20k targets to handle in places I cannot
physically reach after installation... I mean: i *can* reach, but it
would be expensive, you don't want to do it if net is still up).

Here I have two cases:
There are "once in the life" configurations that could conceivably
be done at a central point (e.g.: client certificates), but that would
mean to send different update images to each target, a logistic
nightmare.
There are other configs that need to be done at installation
and possibly changed later (e.g.: WiFI setup).
In both cases configurations *must* survive a full upgrade.

> IOW, what is rootfs image provider expecting. Are they expecting
> overlayfs to be there and are they writing applications in backward
> compatible manner. I don't generally boot into such systems so
> I have no idea. 
Bottom line:
I *need* to be able to change "lower" keeping "upper", otherwise
OverlayFS is simply not the right tool for me.
Alternative would be to save all configs somewhere and
restore them after an update, which is highly "inconvenient"
for many reasons, especially in system that may (and will!!)
need to be updated in unforeseeable ways.
Please do *not* enforce cleanup of "upper" on "lower" changes.

> Thanks
> Vivek
Regards
Mauro

Re: Frequent errors with OverlayFS on root

[Amir Goldstein]

> > So basically squashfs image will be udpated. I am not sure how this
> > will work for all the cases. What if application is updated and
> > it decides to rename its config file from foo.txt to bar.txt. Now
> > when application is launched, user defaults are lost anyway.
> I'm in a somewhat middle position.
>
> Targets are embedded systems and it wouldn't make any sense
> to implement a fully flexible update system (deb/rpm/ipkg/...);
> I just have two "packages": system and application.
>
> I *am* the "rootfs (aka: system) image provider".
>

FYI, I am not a stakeholder. Just chose the "defending" side, to counter
Vivek's opinion for the sake of the argument.

FYI2, overlayfs maintainer as well as Linux maintainer have always
followed the practice of not breaking existing user workloads, so the
status quo is playing to your side anyway.

I assume, although you did not mention this, that you maintain an
OpenWRT derivative.

Just so you know, Android is another "system image provider" project
whose developers were talking about using overlayfs to provide
"debug mode", wherein developers can modify system files.
This is the use case for the proposed override_creds=off feature [1].

I do not know if this mode was deployed on existing phones, but anyway,
it is not enabled in production mode, so it limits very much the problem
of conflict resolution.

Thanks,
Amir.

[1] https://lore.kernel.org/linux-unionfs/20191104215253.141818-1-salyzyn@android.com/

Re: Frequent errors with OverlayFS on root

[Mauro Condarelli]

Thanks Amir,
comments inline below.

On 9/3/20 10:30 AM, Amir Goldstein wrote:
>>> So basically squashfs image will be udpated. I am not sure how this
>>> will work for all the cases. What if application is updated and
>>> it decides to rename its config file from foo.txt to bar.txt. Now
>>> when application is launched, user defaults are lost anyway.
>> I'm in a somewhat middle position.
>>
>> Targets are embedded systems and it wouldn't make any sense
>> to implement a fully flexible update system (deb/rpm/ipkg/...);
>> I just have two "packages": system and application.
>>
>> I *am* the "rootfs (aka: system) image provider".
>>
> FYI, I am not a stakeholder. Just chose the "defending" side, to counter
> Vivek's opinion for the sake of the argument.
I guessed that ;)

> FYI2, overlayfs maintainer as well as Linux maintainer have always
> followed the practice of not breaking existing user workloads, so the
> status quo is playing to your side anyway.
I'm *very* glad to hear this, but if there's any chance this will
change in a foreseeable future I'd want to know it *now*,
changing later would be a disaster (for me, of course).

> I assume, although you did not mention this, that you maintan an
> OpenWRT derivative.
No, that's a fair guess, but I'm actually rolling out a full system
based on Buildroot with current upstreamed u-boot/kernel/apps.

> Just so you know, Android is another "system image provider" project
> whose developers were talking about using overlayfs to provide
> "debug mode", wherein developers can modify system files.
> This is the use case for the proposed override_creds=off feature [1].
>
> I do not know if this mode was deployed on existing phones, but anyway,
> it is not enabled in production mode, so it limits very much the problem
> of conflict resolution.
I'm not sure about what You mean exactly, but I'll try to check it
if You so suggest.

> Thanks,
> Amir.
>
> [1] https://lore.kernel.org/linux-unionfs/20191104215253.141818-1-salyzyn@android.com/
Regards
Mauro

Re: Frequent errors with OverlayFS on root

[Amir Goldstein]

> > Just so you know, Android is another "system image provider" project
> > whose developers were talking about using overlayfs to provide
> > "debug mode", wherein developers can modify system files.
> > This is the use case for the proposed override_creds=off feature [1].
> >
> > I do not know if this mode was deployed on existing phones, but anyway,
> > it is not enabled in production mode, so it limits very much the problem
> > of conflict resolution.
> I'm not sure about what You mean exactly, but I'll try to check it
> if You so suggest.
>

There is nothing for you to check.
Only informing you of another user that may be doing the same thing that
you do and expects it to continue working.

FWIW, if Android *are* using overlayfs like this they should encounter
the same problem you encountered and the upstream fix will not solve it....

Thanks,
Amir.

Re: Frequent errors with OverlayFS on root

[Amir Goldstein]

On Thu, Sep 3, 2020 at 12:00 PM Amir Goldstein <amir73il@gmail.com> wrote:
>
> > > Just so you know, Android is another "system image provider" project
> > > whose developers were talking about using overlayfs to provide
> > > "debug mode", wherein developers can modify system files.
> > > This is the use case for the proposed override_creds=off feature [1].
> > >
> > > I do not know if this mode was deployed on existing phones, but anyway,
> > > it is not enabled in production mode, so it limits very much the problem
> > > of conflict resolution.
> > I'm not sure about what You mean exactly, but I'll try to check it
> > if You so suggest.
> >
>
> There is nothing for you to check.
> Only informing you of another user that may be doing the same thing that
> you do and expects it to continue working.
>
> FWIW, if Android *are* using overlayfs like this they should encounter
> the same problem you encountered and the upstream fix will not solve it....
>

Correction. I made the same mistake before when posting the patch
and Miklos corrected me. Lower fs with UUID (ext4/f2fs) does not suffer from
the specific problem that the fix commit fixes.

Thanks,
Amir.

Re: Frequent errors with OverlayFS on root

[Mauro Condarelli]

On 9/3/20 11:00 AM, Amir Goldstein wrote:
>>> Just so you know, Android is another "system image provider" project
>>> whose developers were talking about using overlayfs to provide
>>> "debug mode", wherein developers can modify system files.
>>> This is the use case for the proposed override_creds=off feature [1].
>>>
>>> I do not know if this mode was deployed on existing phones, but anyway,
>>> it is not enabled in production mode, so it limits very much the problem
>>> of conflict resolution.
>> I'm not sure about what You mean exactly, but I'll try to check it
>> if You so suggest.
>>
> There is nothing for you to check.
> Only informing you of another user that may be doing the same thing that
> you do and expects it to continue working.
>
> FWIW, if Android *are* using overlayfs like this they should encounter
> the same problem you encountered and the upstream fix will not solve it....
>
> Thanks,
> Amir.
FYI: using OverlayFS this way and for this use-case was suggested
by SWUpdate maintainer (Stefano Babic) so I guess there are other
people relying on "correct" behavior around the world .

Regards
Mauro

Re: Frequent errors with OverlayFS on root

[Miklos Szeredi]

On Thu, Sep 3, 2020 at 11:24 AM Mauro Condarelli <mc5686@mclink.it> wrote:

> FYI: using OverlayFS this way and for this use-case was suggested
> by SWUpdate maintainer (Stefano Babic) so I guess there are other
> people relying on "correct" behavior around the world .

So my opinion is that this use case (modifying or even completely
swapping out the lower layer while it is not mounted) should work
perfectly fine with a "plain" configuration.

Doing this with redirect_dir, metacopy or index does sound somewhat
scary and some of that obviously won't even work because of the broken
origins.

Thanks,
Miklos

Re: Frequent errors with OverlayFS on root

[Vivek Goyal]

On Thu, Sep 03, 2020 at 11:53:03AM +0200, Miklos Szeredi wrote:
> On Thu, Sep 3, 2020 at 11:24 AM Mauro Condarelli <mc5686@mclink.it> wrote:
> 
> > FYI: using OverlayFS this way and for this use-case was suggested
> > by SWUpdate maintainer (Stefano Babic) so I guess there are other
> > people relying on "correct" behavior around the world .
> 
> So my opinion is that this use case (modifying or even completely
> swapping out the lower layer while it is not mounted) should work
> perfectly fine with a "plain" configuration.
> 
> Doing this with redirect_dir, metacopy or index does sound somewhat
> scary and some of that obviously won't even work because of the broken
> origins.

Hi Miklos,

Shall we document that changing/swapping lower layers with existing
upper is not supported configuration with advanced features like
metacopy/redirect_dir/index.

This topic has come up now multiple times recently. It will be good
to make it very clear in documentation.

Thanks
Vivek