[Buildroot] rootfs overlay best practices

[Buildroot] rootfs overlay best practices

[Patrick Doyle]

Rather than invent my own, I thought I would query the list first.

NOTE: This post does not refer to the root file system overlay one can
use to customize the build process.  I am looking for information
regarding non-volatile storage of configuration data.

So here is the situation.  I have compiled my (very tiny!) rootfs into
my kernel and boot with that as my rootfs.  But I would like to allow
myself the luxury of making changes to the rootfs by overlaying
non-volatile storage over the rootfs, or, perhaps over selected
portions of the rootfs.

How do folks typically manage this?

I could mount my overlay nonvolatile storage right at / (I think... I
haven't actually tried this yet!) and I think that means that any
change I make to the running system will be preserved.  But there are
probably some weird issues with mount points and such.

Or, I could mount my overlay storage someplace like /config, create
symlinks from selected (or all?) files in /etc to /config/etc, and
thus be able to modify e.g. /config/etc/hostname to change the name of
my device.  It seems like it would be pretty straightforward to write
a post-build script that duplicates /etc into /config/etc and replaces
the files in /etc with symlinks.

Or I could overthink this too much and perhaps I should just stop here
and seek the wisdom of the community.  This is the option I have
selected :-)

--wpd

[Buildroot] rootfs overlay best practices

[Arnout Vandecappelle]

On 04/21/16 16:24, Patrick Doyle wrote:
> Rather than invent my own, I thought I would query the list first.
>
> NOTE: This post does not refer to the root file system overlay one can
> use to customize the build process.  I am looking for information
> regarding non-volatile storage of configuration data.
>
> So here is the situation.  I have compiled my (very tiny!) rootfs into
> my kernel and boot with that as my rootfs.  But I would like to allow
> myself the luxury of making changes to the rootfs by overlaying
> non-volatile storage over the rootfs, or, perhaps over selected
> portions of the rootfs.
>
> How do folks typically manage this?
>
> I could mount my overlay nonvolatile storage right at / (I think... I
> haven't actually tried this yet!) and I think that means that any
> change I make to the running system will be preserved.  But there are
> probably some weird issues with mount points and such.

  Well, no, that doesn't work actually. The underlying initramfs would be hidden 
completely, so your overlay would need to contain a full rootfs as well. And 
this is wasteful because the initramfs would still occupy memory while it can't 
be used anymore. So what usual systems do is to pivot_root from the initramfs to 
the final rootfs. But don't think that that's what you want to do.

>
> Or, I could mount my overlay storage someplace like /config, create
> symlinks from selected (or all?) files in /etc to /config/etc, and
> thus be able to modify e.g. /config/etc/hostname to change the name of
> my device.  It seems like it would be pretty straightforward to write
> a post-build script that duplicates /etc into /config/etc and replaces
> the files in /etc with symlinks.

  Yes, but this doesn't allow you to overwrite/add files at will, only the files 
that you decided at build time that can be overridden. Also, you still have to 
pre-populate your writable storage with the original version of those files.

>
> Or I could overthink this too much and perhaps I should just stop here
> and seek the wisdom of the community.  This is the option I have
> selected :-)

  The usual approach is an overlayfs. Given the subject of this mail, I thought 
you'd already know about it :-) It's been upstream since 3.10 [1]. You basically 
mount a layered filesystem, specifying the lower layer (= your initramfs) and an 
upper layer (= the writable overlay). Any file you write will be written to the 
upper layer, the lower layer stays unaffected. You can also delete files etc. as 
you like.

  The Arch wiki, as is often the case, has the best documentation. [2]

  Unfortunately, using this is a rootfs is not so trivial, because you can't use 
it as the boot-time rootfs. So you have to make an init script that builds the 
overlay and then pivot_roots it. Or limit it to a subdirectory (e.g. /etc). Or 
symlink the important bits into the overlayfs mountpoint.


  As an alternative to overlayfs, you can also use unionfs-fuse. It's a 
userspace (FUSE) implementation of the same concept. Useful for older kernels.


  Generally, I prefer your second approach: symlinking the relevant 
files/directories to a config partition, because that gives you tight control 
about what is writable. You can combine that with a script that formats the 
writable partition if it doesn't exist and prepopulates it with the initramfs 
version of the config partition. The only problem there is when you do an update 
of the initramfs and it has a new config entry which doesn't exist yet in the 
writable partition.


  Regards,
  Arnout


[1] https://lwn.net/Articles/542707/
[2] https://wiki.archlinux.org/index.php/Overlay_filesystem


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] rootfs overlay best practices

[Patrick Doyle]

Hello Arnout,
Thank you for your reply.

On Thu, Apr 21, 2016 at 6:09 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 04/21/16 16:24, Patrick Doyle wrote:
>>
>> Rather than invent my own, I thought I would query the list first.
>>
>
>  The usual approach is an overlayfs. Given the subject of this mail, I
> thought you'd already know about it :-)
I am aware of it, but have never set one up, and hence my questions.

> It's been upstream since 3.10 [1].
> You basically mount a layered filesystem, specifying the lower layer (= your
> initramfs) and an upper layer (= the writable overlay). Any file you write
> will be written to the upper layer, the lower layer stays unaffected. You
> can also delete files etc. as you like.
Yes.  That is exactly what I want.  What I didn't know was how to set
that up for my entire rootfs, or if it was reasonable, or even
possible to set it up for my rootfs.

>  Unfortunately, using this is a rootfs is not so trivial, because you can't
> use it as the boot-time rootfs.
Yes.  That's what I just learned when I tried:

# mount -t overlay overlay
-olowerdir=/,upperdir=/storage,workdir=/storage/work /
overlayfs: workdir and upperdir must be separate subtrees
mount: mounting overlay on / failed: Invalid argument

and

# mount -t overlay overlay -olowerdir=/,upperdir=/storage,workdir=/work /
overlayfs: workdir and upperdir must reside under the same mount
mount: mounting overlay on / failed: Invalid argument


> So you have to make an init script that
> builds the overlay and then pivot_roots it. Or limit it to a subdirectory
> (e.g. /etc). Or symlink the important bits into the overlayfs mountpoint.
Yes -- that was the intent of my email -- how do folks handle this
situation in the real word?  Which approach do you use?  Which works
the best for you? (And by "you", I mean "buildroot community", not
specifically "Arnout")

>  As an alternative to overlayfs, you can also use unionfs-fuse. It's a
> userspace (FUSE) implementation of the same concept. Useful for older
> kernels.
Thanks.  That's good to know, but I'm using a 4.1 kernel, so I think
that overlayfs would be the preferred way to go here, wouldn't it?

>  Generally, I prefer your second approach: symlinking the relevant
> files/directories to a config partition, because that gives you tight
> control about what is writable. You can combine that with a script that
> formats the writable partition if it doesn't exist and prepopulates it with
> the initramfs version of the config partition. The only problem there is
> when you do an update of the initramfs and it has a new config entry which
> doesn't exist yet in the writable partition.
OK, thanks.

--wpd

[Buildroot] rootfs overlay best practices

[Arnout Vandecappelle]

On 04/22/16 15:35, Patrick Doyle wrote:
> Hello Arnout,
> Thank you for your reply.
>
> On Thu, Apr 21, 2016 at 6:09 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> On 04/21/16 16:24, Patrick Doyle wrote:
[snip]
>>   Unfortunately, using this is a rootfs is not so trivial, because you can't
>> use it as the boot-time rootfs.
> Yes.  That's what I just learned when I tried:
>
> # mount -t overlay overlay
> -olowerdir=/,upperdir=/storage,workdir=/storage/work /
> overlayfs: workdir and upperdir must be separate subtrees
> mount: mounting overlay on / failed: Invalid argument
>
> and
>
> # mount -t overlay overlay -olowerdir=/,upperdir=/storage,workdir=/work /
> overlayfs: workdir and upperdir must reside under the same mount
> mount: mounting overlay on / failed: Invalid argument

  Googling (or duckducking) "overlayfs root" gives you plenty of explanations of 
how to do it properly.

  It would actually be nice if buildroot had an option in the filesystem menu to 
set up a rootfs overlay for you...


>> So you have to make an init script that
>> builds the overlay and then pivot_roots it. Or limit it to a subdirectory
>> (e.g. /etc). Or symlink the important bits into the overlayfs mountpoint.
> Yes -- that was the intent of my email -- how do folks handle this
> situation in the real word?  Which approach do you use?  Which works
> the best for you? (And by "you", I mean "buildroot community", not
> specifically "Arnout")

  Well, I work on many different project and it's different for each project 
:-). In one project, we started with unionfs-fuse but in the end switched to 
symlinking to a writable partition.

>
>>   As an alternative to overlayfs, you can also use unionfs-fuse. It's a
>> userspace (FUSE) implementation of the same concept. Useful for older
>> kernels.
> Thanks.  That's good to know, but I'm using a 4.1 kernel, so I think
> that overlayfs would be the preferred way to go here, wouldn't it?

  Yes, overlayfs has much better performance and I think it's also a bit more 
robust. unionfs-fuse can do a few things more IIRC, but probably nothing important.


  Regards,
  Arnout

[snip]

-- 
Arnout Vandecappelle      arnout dot vandecappelle at essensium dot com
Senior Embedded Software Architect . . . . . . +32-478-010353 (mobile)
Essensium, Mind division . . . . . . . . . . . . . . http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium . . . . . BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] rootfs overlay best practices

[Steve Calfee]

Hi Patrick,

I like these user questions on the mail list. See below

On Fri, Apr 22, 2016 at 6:35 AM, Patrick Doyle <wpdster@gmail.com> wrote:
> Yes -- that was the intent of my email -- how do folks handle this
> situation in the real word?  Which approach do you use?  Which works
> the best for you? (And by "you", I mean "buildroot community", not
> specifically "Arnout")
>
Personally, I like to keep things simple. Makes it easier to maintain,
update in the field etc. It does require a little more time when
setting up a new embedded service like say MySQL or something.

I like to run most of the system directly from a read only filesystem.
Stuff that needs persistent data should be on another read/write
filesystem. Overlay or unionfs hide what is changeable and what has
been changed. It easy to have hacks done on a dev system not get
propaged to a production system causing failures.

So I have most of the rootfs in one flash partition and have another
partition just for writeable data. So mysql will run out of rootfs
flash and access its db and writeable files from the other partition.
Lots of fuss setting up the initial layout- editing configs etc, but
nice when running.

Also, you can re-virginize a system by clearing the writable
partitiion. If done correctly a reboot will then reestablish the
needed files in the rw partition.

Regards, Steve