Re: Frequent errors with OverlayFS on root

[Vivek Goyal <vgoyal@redhat.com>]

On Wed, Sep 02, 2020 at 06:34:12AM +0300, Amir Goldstein wrote:
> On Wed, Sep 2, 2020 at 2:37 AM Mauro Condarelli <mc5686@mclink.it> wrote:
> >
> > Thanks Amir,
> > comments inline below
> >
> > Regards
> > Mauro
> >
> > On 9/1/20 8:43 PM, Amir Goldstein wrote:
> >
> > On Tue, Sep 1, 2020 at 9:01 PM Mauro Condarelli <mc5686@mclink.it> wrote:
> >
> > Hi,
> > most likely this is not the right place to ask, please redirect me as needed.
> >
> > I'm trying to use OverlayFS to add (limited) write capability to a ReadOnly
> > rootfs (SquashFS)
> >
> > Essentially (actual script is more complex, of course) boot-sequence includes:
> >
> > # /dev/mmcblk0p5: ext4 (upper+work+nwwroot+newroot/oldroot)
> > # /dev/mmcblk0p6: SquashFS mounted on /
> > mount /dev/mmcblk0p5 /overlay
> > mount -t overlay overlay -o lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work  /overlay/newroot
> > cd /overlay/newroot
> > pivot_root . oldroot
> > mount --move oldroot/dev /dev
> > mount --move oldroot/proc /proc
> >
> > This works as expected, but, too often for comfort, some file
> > (and sometime also directories) become unavailable due to error:
> >
> > overlayfs: invalid origin (ssh/sshd_config, ftype=8000, origin ftype=4000).
> >
> > File name changes, of course, but rest is fairly constant.
> >
> > This always happens when some file is written.
> > Error persists reboots.
> > Only way I found to "cure" the system is to go on "upper" and delete the file
> > thus going back to "lower" version (in this case I should delete "/oldroot/overlay/upper/etc/ssh/sshd_config")
> >
> > This is a self-built kernel (Linux vocore 5.7.0 #2 PREEMPT Mon Aug 3 09:19:06 CEST 2020 mips GNU/Linux)
> > on a custom target based on a SoC (MT7628).
> >
> > I am available to do any required test, but I have no idea about where to start.
> >
> > Any hint (or redirect) would be greatly appreciated.
> >
> > This is probably your problem:
> > https://lore.kernel.org/linux-unionfs/32532923.JtPX5UtSzP@fgdesktop/
> >
> > It surely looks like it is.
> > I tried to follow the thread, but I'm unsure i grokked it .
> >
> > I am using OverlayFS (ext4 over a SquashFS) on rootfs in order to
> > be able to update the whole system (changing SquashFS) while
> > retaining customization (essentially tweaks in /etc/, host certificates
> > and similar stuff) in an embedded target.
> >
> > To complicate matters I also have a dual system for fallback in case
> > of faulty upgrade; this means I can switch from:
> >   lower=part6, upper=part5, update will go into part7
> > to
> >   lower=part7, upper=part5, update will go into part6
> >
> > I don't need nfs at all, so exporting OverlayFS is not an issue,
> > but it's unclear to me if this "lower swapping" is
> > actually supported as I see:
> >
> > > > /me is again wondering what's the use case of modifying lower layer
> > > > with an existing upper. Is it fair to say, no don't recreate/modify
> > > > lower layers and use with existing upper.
> > > >
> > >
> > > It's fine by me to document that this is not supported.
> >
> > ... which is scary.
> > Note I do *not* need to modify lower on-the-fly, when I
> > swap systems, that will happen at reboot.
> 
> Don't be scared :-)
> Your reaction is a good answer to Vivek's question above -
> No, it is not fair to say don't re-create lower and use existing upper

This seems like a wrong use case to me. So in above example, say
following happens.

- Mount overlay
- modify sshd_config (it gets copied up).
- Lower squahsfs gets updated and /etc/sshd/sshd_config gets updated.
- Mount overlay and user now sees old copied up sshd_config. (If it
  works).

Conceptually it is not making much sense to me. What's the point of
upgrading lower because after mounting overlay you might still see
old file. And to make it worse, behavior is underministic beacuse
if file has not been copied up, you will see new file.

To me, this sounds like a volatile overlay use case. Any changes to
overlay should be thrown away when lower sqashfs is updated and a
fresh upper should be set.

Thanks
Vivek