[RFC] Pre-seeded files/directories for UBIFS

[RFC] Pre-seeded files/directories for UBIFS

[Richard Weinberger]

Hi!

These days I had an interesting discussion with Christoph about overlayfs and
its burden. The main use-case of overlayfs in combination with UBIFS is having a
squashfs as lower and UBIFS as upper directory. Such that all changes to the
read-only squashfs go into UBIFS. Upon a factory reset all files within the
UBIFS will be removed  and the merged directory is clean again. Christoph argued
that such a functionality  could be achieved without overlayfs if the filesystem
supported something like pre-seeded  files or directories. This would lower
memory pressure and complexity.

Today I had a thought about this and I'm pretty sure we can implement this in
UBIFS with not too much effort. The basic idea is marking files or whole
directories as seed upon mkfs.ubifs time.
If these files will be changed at run-time, the original contents will stay on
the medium and at any time these files can be reverted back to their seed state.
This includes file contents, attributes and extended attributes. I can think of
an UBIFS specific IOCTL to put files into seed state and to revert them again.

Since UBIFS is already a pure copy-on-write filesystem, all we have to do is
teaching the index tree about seeds. We could add a flag to the UBIFS key which
indicates that the node behind this key is seeded.
i.e. file "foo" is seeded and the corresponding inode number is 0x1234,
then every key of every UBIFS node that belongs to that file will wear the new
flag UBIFS_SEED_KEY.
ubifs_ino_node: 0x1234 | UBIFS_INO_KEY | UBIFS_SEED_KEY
ubifs_data_node: 0x1234 | <BLOCK_NO> | UBIFS_DATA_KEY | UBIFS_SEED_KEY

The inode itself will have a flag which denotes whether this file is seeded and
whether some modifications have happened. This will allow us to lookup directly
in the index tree with UBIFS_SEED_KEY set or no. Otherwise we'd have to do two
lookups every time.

If a seeded node faces a modification it will stay referenced in the index tree
and a copy without UBIFS_SEED_KEY is made. Upon next lookup the new node will
be used automatically. Reverting to the original state means purging all nodes
that don't have the UBIFS_SEED_KEY flag.

There are corner cases to consider, mostly for lookup of data nodes.
Currently a missing data node denotes a hole in a file.
With seeded files a missing data node can also mean that we need to fall back
to a UBIFS_SEED_KEY lookup.

Storing UBIFS_SEED_KEY itself into the UBIFS key is also not trivial since
almost all bits of the 32bit tuple are in use. But I'm sure we find some way
to encode it.

Another thing to consider are seeded directory entries. This requires us to
implement our own whiteout mechanism. But this could also be re-used for
overlayfs whiteouts.

That said, I consider this feature as doable but not trivial.
Artem, Adrian, you designed UBIFS, what do you think?
Maybe I missed some major show-stopper. :)

Thanks,
//richard

Re: [RFC] Pre-seeded files/directories for UBIFS

[Florian Fainelli]

Hello,

On 05/20/2017 09:12 AM, Richard Weinberger wrote:
> Hi!
> 
> These days I had an interesting discussion with Christoph about overlayfs and
> its burden. The main use-case of overlayfs in combination with UBIFS is having a
> squashfs as lower and UBIFS as upper directory. Such that all changes to the
> read-only squashfs go into UBIFS. Upon a factory reset all files within the
> UBIFS will be removed  and the merged directory is clean again. Christoph argued
> that such a functionality  could be achieved without overlayfs if the filesystem
> supported something like pre-seeded  files or directories. This would lower
> memory pressure and complexity.

As you may know, OpenWrt/LEDE have been using this scheme for many years
now (before it was named overlayfs, this was called mini fanout overlay
~10 yrs ago) with squashfs + jffs2 before on P-NOR flashes. There are
still devices like those that benefit from squashfs(ro)+jffs2(rw), so
while bringing a similar functionality using UBIFS exclusively would be
interesting, it would still make Linux distribution want to support a
more generic scheme which is using overlayfs as well.

> 
> Today I had a thought about this and I'm pretty sure we can implement this in
> UBIFS with not too much effort. The basic idea is marking files or whole
> directories as seed upon mkfs.ubifs time.
> If these files will be changed at run-time, the original contents will stay on
> the medium and at any time these files can be reverted back to their seed state.
> This includes file contents, attributes and extended attributes. I can think of
> an UBIFS specific IOCTL to put files into seed state and to revert them again.
> 
> Since UBIFS is already a pure copy-on-write filesystem, all we have to do is
> teaching the index tree about seeds. We could add a flag to the UBIFS key which
> indicates that the node behind this key is seeded.
> i.e. file "foo" is seeded and the corresponding inode number is 0x1234,
> then every key of every UBIFS node that belongs to that file will wear the new
> flag UBIFS_SEED_KEY.
> ubifs_ino_node: 0x1234 | UBIFS_INO_KEY | UBIFS_SEED_KEY
> ubifs_data_node: 0x1234 | <BLOCK_NO> | UBIFS_DATA_KEY | UBIFS_SEED_KEY
> 
> The inode itself will have a flag which denotes whether this file is seeded and
> whether some modifications have happened. This will allow us to lookup directly
> in the index tree with UBIFS_SEED_KEY set or no. Otherwise we'd have to do two
> lookups every time.
> 
> If a seeded node faces a modification it will stay referenced in the index tree
> and a copy without UBIFS_SEED_KEY is made. Upon next lookup the new node will
> be used automatically. Reverting to the original state means purging all nodes
> that don't have the UBIFS_SEED_KEY flag.
> 
> There are corner cases to consider, mostly for lookup of data nodes.
> Currently a missing data node denotes a hole in a file.
> With seeded files a missing data node can also mean that we need to fall back
> to a UBIFS_SEED_KEY lookup.
> 
> Storing UBIFS_SEED_KEY itself into the UBIFS key is also not trivial since
> almost all bits of the 32bit tuple are in use. But I'm sure we find some way
> to encode it.
> 
> Another thing to consider are seeded directory entries. This requires us to
> implement our own whiteout mechanism. But this could also be re-used for
> overlayfs whiteouts.
> 
> That said, I consider this feature as doable but not trivial.
> Artem, Adrian, you designed UBIFS, what do you think?
> Maybe I missed some major show-stopper. :)
> 
> Thanks,
> //richard
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
> 

-- 
Florian

Re: [RFC] Pre-seeded files/directories for UBIFS

[Richard Weinberger]

Florian,

Am 20.05.2017 um 20:39 schrieb Florian Fainelli:
> Hello,
> 
> On 05/20/2017 09:12 AM, Richard Weinberger wrote:
>> Hi!
>>
>> These days I had an interesting discussion with Christoph about overlayfs and
>> its burden. The main use-case of overlayfs in combination with UBIFS is having a
>> squashfs as lower and UBIFS as upper directory. Such that all changes to the
>> read-only squashfs go into UBIFS. Upon a factory reset all files within the
>> UBIFS will be removed  and the merged directory is clean again. Christoph argued
>> that such a functionality  could be achieved without overlayfs if the filesystem
>> supported something like pre-seeded  files or directories. This would lower
>> memory pressure and complexity.
> 
> As you may know, OpenWrt/LEDE have been using this scheme for many years
> now (before it was named overlayfs, this was called mini fanout overlay
> ~10 yrs ago) with squashfs + jffs2 before on P-NOR flashes. There are
> still devices like those that benefit from squashfs(ro)+jffs2(rw), so
> while bringing a similar functionality using UBIFS exclusively would be
> interesting, it would still make Linux distribution want to support a
> more generic scheme which is using overlayfs as well.

UBIFS will still support overlayfs and adding a pre-seed mechanism
to UBIFS will not block generic solutions.
Maybe we can add a generic interface to VFS at some point if other filesystems
support that too. :-)

Thanks,
//richard

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Ralph Sennhauser]

On Sat, 20 May 2017 11:39:33 -0700
Florian Fainelli <f.fainelli@gmail.com> wrote:

> Hello,
> 
> On 05/20/2017 09:12 AM, Richard Weinberger wrote:
> > Hi!
> > 
> > These days I had an interesting discussion with Christoph about
> > overlayfs and its burden. The main use-case of overlayfs in
> > combination with UBIFS is having a squashfs as lower and UBIFS as
> > upper directory. Such that all changes to the read-only squashfs go
> > into UBIFS. Upon a factory reset all files within the UBIFS will be
> > removed  and the merged directory is clean again. Christoph argued
> > that such a functionality  could be achieved without overlayfs if
> > the filesystem supported something like pre-seeded  files or
> > directories. This would lower memory pressure and complexity.  
> 
> As you may know, OpenWrt/LEDE have been using this scheme for many
> years now (before it was named overlayfs, this was called mini fanout
> overlay ~10 yrs ago) with squashfs + jffs2 before on P-NOR flashes.
> There are still devices like those that benefit from
> squashfs(ro)+jffs2(rw), so while bringing a similar functionality
> using UBIFS exclusively would be interesting, it would still make
> Linux distribution want to support a more generic scheme which is
> using overlayfs as well.
> 

There is also the size consideration. Unless a seeded ubifs can get
close to squashfs in terms of compression there would still be a
use-case for squashfs with an ubifs overlay. My current root as ubifs
instead of squashfs is 76.8% bigger.

> > 
> > Today I had a thought about this and I'm pretty sure we can
> > implement this in UBIFS with not too much effort. The basic idea is
> > marking files or whole directories as seed upon mkfs.ubifs time.
> > If these files will be changed at run-time, the original contents
> > will stay on the medium and at any time these files can be reverted
> > back to their seed state. This includes file contents, attributes
> > and extended attributes. I can think of an UBIFS specific IOCTL to
> > put files into seed state and to revert them again.
> > 
> > Since UBIFS is already a pure copy-on-write filesystem, all we have
> > to do is teaching the index tree about seeds. We could add a flag
> > to the UBIFS key which indicates that the node behind this key is
> > seeded. i.e. file "foo" is seeded and the corresponding inode
> > number is 0x1234, then every key of every UBIFS node that belongs
> > to that file will wear the new flag UBIFS_SEED_KEY.
> > ubifs_ino_node: 0x1234 | UBIFS_INO_KEY | UBIFS_SEED_KEY
> > ubifs_data_node: 0x1234 | <BLOCK_NO> | UBIFS_DATA_KEY |
> > UBIFS_SEED_KEY
> > 
> > The inode itself will have a flag which denotes whether this file
> > is seeded and whether some modifications have happened. This will
> > allow us to lookup directly in the index tree with UBIFS_SEED_KEY
> > set or no. Otherwise we'd have to do two lookups every time.
> > 
> > If a seeded node faces a modification it will stay referenced in
> > the index tree and a copy without UBIFS_SEED_KEY is made. Upon next
> > lookup the new node will be used automatically. Reverting to the
> > original state means purging all nodes that don't have the
> > UBIFS_SEED_KEY flag.
> > 
> > There are corner cases to consider, mostly for lookup of data nodes.
> > Currently a missing data node denotes a hole in a file.
> > With seeded files a missing data node can also mean that we need to
> > fall back to a UBIFS_SEED_KEY lookup.
> > 
> > Storing UBIFS_SEED_KEY itself into the UBIFS key is also not
> > trivial since almost all bits of the 32bit tuple are in use. But
> > I'm sure we find some way to encode it.
> > 
> > Another thing to consider are seeded directory entries. This
> > requires us to implement our own whiteout mechanism. But this could
> > also be re-used for overlayfs whiteouts.
> > 
> > That said, I consider this feature as doable but not trivial.
> > Artem, Adrian, you designed UBIFS, what do you think?
> > Maybe I missed some major show-stopper. :)
> > 
> > Thanks,
> > //richard
> > 
> > ______________________________________________________
> > Linux MTD discussion mailing list
> > http://lists.infradead.org/mailman/listinfo/linux-mtd/
> >   
> 

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Richard Weinberger]

Ralph,

Am 20.05.2017 um 21:36 schrieb Ralph Sennhauser:
>>> These days I had an interesting discussion with Christoph about
>>> overlayfs and its burden. The main use-case of overlayfs in
>>> combination with UBIFS is having a squashfs as lower and UBIFS as
>>> upper directory. Such that all changes to the read-only squashfs go
>>> into UBIFS. Upon a factory reset all files within the UBIFS will be
>>> removed  and the merged directory is clean again. Christoph argued
>>> that such a functionality  could be achieved without overlayfs if
>>> the filesystem supported something like pre-seeded  files or
>>> directories. This would lower memory pressure and complexity.  
>>
>> As you may know, OpenWrt/LEDE have been using this scheme for many
>> years now (before it was named overlayfs, this was called mini fanout
>> overlay ~10 yrs ago) with squashfs + jffs2 before on P-NOR flashes.
>> There are still devices like those that benefit from
>> squashfs(ro)+jffs2(rw), so while bringing a similar functionality
>> using UBIFS exclusively would be interesting, it would still make
>> Linux distribution want to support a more generic scheme which is
>> using overlayfs as well.
>>
> 
> There is also the size consideration. Unless a seeded ubifs can get
> close to squashfs in terms of compression there would still be a
> use-case for squashfs with an ubifs overlay. My current root as ubifs
> instead of squashfs is 76.8% bigger.

You seem to misunderstand this feature, the goal is not to void all
uses of squashfs.
I'm pretty sure for the LEDE usecase squashfs is the better choice.

Thanks,
//richard

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Ralph Sennhauser]

Hi Richard,

On Sat, 20 May 2017 21:57:36 +0200
Richard Weinberger <richard@nod.at> wrote:

> Ralph,
> 
> Am 20.05.2017 um 21:36 schrieb Ralph Sennhauser:
> >>> These days I had an interesting discussion with Christoph about
> >>> overlayfs and its burden. The main use-case of overlayfs in
> >>> combination with UBIFS is having a squashfs as lower and UBIFS as
> >>> upper directory. Such that all changes to the read-only squashfs
> >>> go into UBIFS. Upon a factory reset all files within the UBIFS
> >>> will be removed  and the merged directory is clean again.
> >>> Christoph argued that such a functionality  could be achieved
> >>> without overlayfs if the filesystem supported something like
> >>> pre-seeded  files or directories. This would lower memory
> >>> pressure and complexity.    
> >>
> >> As you may know, OpenWrt/LEDE have been using this scheme for many
> >> years now (before it was named overlayfs, this was called mini
> >> fanout overlay ~10 yrs ago) with squashfs + jffs2 before on P-NOR
> >> flashes. There are still devices like those that benefit from
> >> squashfs(ro)+jffs2(rw), so while bringing a similar functionality
> >> using UBIFS exclusively would be interesting, it would still make
> >> Linux distribution want to support a more generic scheme which is
> >> using overlayfs as well.
> >>  
> > 
> > There is also the size consideration. Unless a seeded ubifs can get
> > close to squashfs in terms of compression there would still be a
> > use-case for squashfs with an ubifs overlay. My current root as
> > ubifs instead of squashfs is 76.8% bigger.  
> 
> You seem to misunderstand this feature, the goal is not to void all
> uses of squashfs.
> I'm pretty sure for the LEDE usecase squashfs is the better choice.

Probably depends on the device but is beyond the point. Just wanted to
mention in response to the main point being the factory reset of the
squashfs + ubifs overlay setup that size is just as important or even
more important at least for some.

Whether you want to implement and maintain another solution to the
factory reset problem in ubifs which falls short of full snapshot
support is up to you.

Ralph

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Geert Uytterhoeven]

On Sat, May 20, 2017 at 9:36 PM, Ralph Sennhauser
<ralph.sennhauser@gmail.com> wrote:
> There is also the size consideration. Unless a seeded ubifs can get
> close to squashfs in terms of compression there would still be a
> use-case for squashfs with an ubifs overlay. My current root as ubifs
> instead of squashfs is 76.8% bigger.

As seeded files are stored and kept unmodified, they could be stored in
compressed form?

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Richard Weinberger]

Geert,

Am 21.05.2017 um 10:37 schrieb Geert Uytterhoeven:
> On Sat, May 20, 2017 at 9:36 PM, Ralph Sennhauser
> <ralph.sennhauser@gmail.com> wrote:
>> There is also the size consideration. Unless a seeded ubifs can get
>> close to squashfs in terms of compression there would still be a
>> use-case for squashfs with an ubifs overlay. My current root as ubifs
>> instead of squashfs is 76.8% bigger.
> 
> As seeded files are stored and kept unmodified, they could be stored in
> compressed form?

This is what UBIFS already does.

Thanks,
//richard

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Ralph Sennhauser]

Hi Richard

On Sun, 21 May 2017 10:40:05 +0200
Richard Weinberger <richard@nod.at> wrote:

> Geert,
> 
> Am 21.05.2017 um 10:37 schrieb Geert Uytterhoeven:
> > On Sat, May 20, 2017 at 9:36 PM, Ralph Sennhauser
> > <ralph.sennhauser@gmail.com> wrote:  
> >> There is also the size consideration. Unless a seeded ubifs can get
> >> close to squashfs in terms of compression there would still be a
> >> use-case for squashfs with an ubifs overlay. My current root as
> >> ubifs instead of squashfs is 76.8% bigger.  
> > 
> > As seeded files are stored and kept unmodified, they could be
> > stored in compressed form?  
> 
> This is what UBIFS already does.

Right, or it would be some 400-500% difference.

Another advantage of the squashfs with overlay came to mind. OpenWrt
doesn't do a factory reset per se but boots into a "failsafe" mode which
just doesn't mount the overlay. This allows to fix the overlay (some
screwed up config value which prevents connection) or to reset the
password for example without having to reinstall packages and
reconfigure the device from ground up. Wiping the overlay is basically
the last resort.

With the expectation distributions to use the same approach for as many
devices possible I see the seeded ubifs running out of users fast. Some
commercial vendor backporting it to a 3.10 kernel maybe?

If the seeded ubifs could be generalized to snapshot support ala btrfs
that would change things a lot as it would enable uses far beyond just
factory reset. No idea how feasible that is but might be worth
considering instead.

Ralph

> 
> Thanks,
> //richard

Re: [OpenWrt-Devel] [RFC] Pre-seeded files/directories for UBIFS

[Richard Weinberger]

Ralph,

Am 21.05.2017 um 14:23 schrieb Ralph Sennhauser:
> If the seeded ubifs could be generalized to snapshot support ala btrfs
> that would change things a lot as it would enable uses far beyond just
> factory reset. No idea how feasible that is but might be worth
> considering instead.

I thought about that too but this requires much more invasive changes
in UBIFS.
So I'm not sure whether it is worth the hassle.

Thanks,
//richard