Support for writable files with a read-only root file system

Support for writable files with a read-only root file system

[Matt Schuckmann]

I've been investigating the support for read only root file systems and trying to suss out how to support persistent writeable files in my image where the root file system is read only, this seems like a very common thing for an embedded Linux system to need but I'm not seeing the support for it. 
 
I should note that I'm working off of the Dylan branch, I'm not sure if things have changed in a newer branch. 

So far I've had to read the code to figure out as much as I have, and I haven't found any documentation for this, am I missing something? 

It appears that my recipe(s) are supposed to install a volatiles file under ${D}${sysconfdir}/default/volatiles/ that lists out the volatile directories, links and files that need to be created in the image. 

For writeable persistent files I would presume that I should specify links that point to some place on a write partition. 

What I don't see is any sort of mechanism for setting up the default files on that writeable partition, am I on my own for this or is there some other recommended mechanism. 

In my early research I found the MentorEmbedded/meta-ro-rootfs layer that had a nifty way of specifying VOLATILE_BINDS to create a list of binds that should occur at boot up and in the process creating the mounts if the target of the bind didn't exist a copy was made from the read only location, this appears to have provided a sort of default to go into the writeable location. Was this functionality abandoned when support for ro-rootfs was brought into oe-core? If so why? 

A good example for the kind of writeable persistent file that I intend to have is /etc/network/interfaces
How do I go about letting the init_ifupdown recipe install it normally and then have another recipe or even the image configure it to be either a link or a bind mount at another location while still preserving the default contents. 

I hope I'm making sense here. 

Thanks,
Matt S. 

Re: Support for writable files with a read-only root file system

[Mike Looijmans]

The one-size-fits-all solution is to use an unionfs, and a writable part in 
RAM or mounted elsewhere, e.g. in flash (typically a tiny jffs2 partition).

It requires quite some scripting to get right, you have to boot the system and 
then mount the overlay, the unionfs to join it with the rootfs, and then 
"pivot" it to make it the new root.

After that you get a root where you can modify any file, and even persist that 
change (if you use a persistent overlay).



On 11/26/2014 08:51 AM, Matt Schuckmann wrote:
> I've been investigating the support for read only root file systems and trying to suss out how to support persistent writeable files in my image where the root file system is read only, this seems like a very common thing for an embedded Linux system to need but I'm not seeing the support for it.
>
> I should note that I'm working off of the Dylan branch, I'm not sure if things have changed in a newer branch.
>
> So far I've had to read the code to figure out as much as I have, and I haven't found any documentation for this, am I missing something?
>
> It appears that my recipe(s) are supposed to install a volatiles file under ${D}${sysconfdir}/default/volatiles/ that lists out the volatile directories, links and files that need to be created in the image.
>
> For writeable persistent files I would presume that I should specify links that point to some place on a write partition.
>
> What I don't see is any sort of mechanism for setting up the default files on that writeable partition, am I on my own for this or is there some other recommended mechanism.
>
> In my early research I found the MentorEmbedded/meta-ro-rootfs layer that had a nifty way of specifying VOLATILE_BINDS to create a list of binds that should occur at boot up and in the process creating the mounts if the target of the bind didn't exist a copy was made from the read only location, this appears to have provided a sort of default to go into the writeable location. Was this functionality abandoned when support for ro-rootfs was brought into oe-core? If so why?
>
> A good example for the kind of writeable persistent file that I intend to have is /etc/network/interfaces
> How do I go about letting the init_ifupdown recipe install it normally and then have another recipe or even the image configure it to be either a link or a bind mount at another location while still preserving the default contents.
>
> I hope I'm making sense here.
>
> Thanks,
> Matt S.
>



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Support for writable files with a read-only root file system

[Jim Rafert]

Hi Mike,

It sounds like using volatiles is the opposite of what you want.  Unless I miss the point,
what you want is to have your root file system mounted read-only with only select files being
writable, and persistent.  Volatiles are by definition not persistent. Volatiles may be a part of your
solution for files that can be non-persistent, but can't be the whole solution.

If you can make a small number of entire directory hierarchies writable (like /etc, /mnt), you can make
small partitions that are mounted by /etc/fstab.  If you want to be more selective, you need to have 
a writable file system where the writable files will reside, with links in the root file system that point 
to them.

-Jim-
________________________________________
From: Mike Looijmans [mike.looijmans@topic.nl]
Sent: Wednesday, November 26, 2014 7:38 AM
To: yocto@yoctoproject.org
Subject: Re: [yocto] Support for writable files with a read-only root file system

The one-size-fits-all solution is to use an unionfs, and a writable part in
RAM or mounted elsewhere, e.g. in flash (typically a tiny jffs2 partition).

It requires quite some scripting to get right, you have to boot the system and
then mount the overlay, the unionfs to join it with the rootfs, and then
"pivot" it to make it the new root.

After that you get a root where you can modify any file, and even persist that
change (if you use a persistent overlay).



On 11/26/2014 08:51 AM, Matt Schuckmann wrote:
> I've been investigating the support for read only root file systems and trying to suss out how to support persistent writeable files in my image where the root file system is read only, this seems like a very common thing for an embedded Linux system to need but I'm not seeing the support for it.
>
> I should note that I'm working off of the Dylan branch, I'm not sure if things have changed in a newer branch.
>
> So far I've had to read the code to figure out as much as I have, and I haven't found any documentation for this, am I missing something?
>
> It appears that my recipe(s) are supposed to install a volatiles file under ${D}${sysconfdir}/default/volatiles/ that lists out the volatile directories, links and files that need to be created in the image.
>
> For writeable persistent files I would presume that I should specify links that point to some place on a write partition.
>
> What I don't see is any sort of mechanism for setting up the default files on that writeable partition, am I on my own for this or is there some other recommended mechanism.
>
> In my early research I found the MentorEmbedded/meta-ro-rootfs layer that had a nifty way of specifying VOLATILE_BINDS to create a list of binds that should occur at boot up and in the process creating the mounts if the target of the bind didn't exist a copy was made from the read only location, this appears to have provided a sort of default to go into the writeable location. Was this functionality abandoned when support for ro-rootfs was brought into oe-core? If so why?
>
> A good example for the kind of writeable persistent file that I intend to have is /etc/network/interfaces
> How do I go about letting the init_ifupdown recipe install it normally and then have another recipe or even the image configure it to be either a link or a bind mount at another location while still preserving the default contents.
>
> I hope I'm making sense here.
>
> Thanks,
> Matt S.
>



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: Support for writable files with a read-only root file system

[Matt Schuckmann]

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org [mailto:yocto-
> bounces@yoctoproject.org] On Behalf Of Jim Rafert
> Sent: Wednesday, November 26, 2014 8:33 AM
> To: Mike Looijmans; yocto@yoctoproject.org
> Subject: Re: [yocto] Support for writable files with a read-only root
> file system
> 
> Hi Mike,
> 
> It sounds like using volatiles is the opposite of what you want.
> Unless I miss the point, what you want is to have your root file system
> mounted read-only with only select files being writable, and
> persistent.  Volatiles are by definition not persistent. Volatiles may
> be a part of your solution for files that can be non-persistent, but
> can't be the whole solution.

> 
> If you can make a small number of entire directory hierarchies writable
> (like /etc, /mnt), you can make small partitions that are mounted by
> /etc/fstab.  If you want to be more selective, you need to have a
> writable file system where the writable files will reside, with links
> in the root file system that point to them.
> 

Hi Jim, 

Yes you are correct I want a small number of files into to reside in a writeable partition with links in the root file system to point to them. I guess I latched on to the volatiles mechanism because it seems close to meeting my needs.

Making entire directory hierarchies like /etc writeable seems risky, things that you never want changed could get changed. Furthermore it makes it harder to deal with setting up default versions of files. 

Is there really no mechanism for this type of thing in OE? It seems like such a common thing for anybody creating a system with a ro root file system to need or am I missing something. 

If there is not such a mechanism then I will probably recreate the VOLATILE_BINDS mechanism that was in the MentorEmbedded layer and maybe call it PERSISTENT_BINDS or some such. I'm pretty new to OE so whatever I come up with will be pretty rough but I'd be willing to share my solution if there was interest. 

Thanks,
Matt S. 


> -Jim-
> ________________________________________
> From: Mike Looijmans [mike.looijmans@topic.nl]
> Sent: Wednesday, November 26, 2014 7:38 AM
> To: yocto@yoctoproject.org
> Subject: Re: [yocto] Support for writable files with a read-only root
> file system
> 
> The one-size-fits-all solution is to use an unionfs, and a writable
> part in RAM or mounted elsewhere, e.g. in flash (typically a tiny jffs2
> partition).
> 
> It requires quite some scripting to get right, you have to boot the
> system and then mount the overlay, the unionfs to join it with the
> rootfs, and then "pivot" it to make it the new root.
> 
> After that you get a root where you can modify any file, and even
> persist that change (if you use a persistent overlay).
> 
> 
> 
> On 11/26/2014 08:51 AM, Matt Schuckmann wrote:
> > I've been investigating the support for read only root file systems
> and trying to suss out how to support persistent writeable files in my
> image where the root file system is read only, this seems like a very
> common thing for an embedded Linux system to need but I'm not seeing
> the support for it.
> >
> > I should note that I'm working off of the Dylan branch, I'm not sure
> if things have changed in a newer branch.
> >
> > So far I've had to read the code to figure out as much as I have, and
> I haven't found any documentation for this, am I missing something?
> >
> > It appears that my recipe(s) are supposed to install a volatiles file
> under ${D}${sysconfdir}/default/volatiles/ that lists out the volatile
> directories, links and files that need to be created in the image.
> >
> > For writeable persistent files I would presume that I should specify
> links that point to some place on a write partition.
> >
> > What I don't see is any sort of mechanism for setting up the default
> files on that writeable partition, am I on my own for this or is there
> some other recommended mechanism.
> >
> > In my early research I found the MentorEmbedded/meta-ro-rootfs layer
> that had a nifty way of specifying VOLATILE_BINDS to create a list of
> binds that should occur at boot up and in the process creating the
> mounts if the target of the bind didn't exist a copy was made from the
> read only location, this appears to have provided a sort of default to
> go into the writeable location. Was this functionality abandoned when
> support for ro-rootfs was brought into oe-core? If so why?
> >
> > A good example for the kind of writeable persistent file that I
> intend
> > to have is /etc/network/interfaces How do I go about letting the
> init_ifupdown recipe install it normally and then have another recipe
> or even the image configure it to be either a link or a bind mount at
> another location while still preserving the default contents.
> >
> > I hope I'm making sense here.
> >
> > Thanks,
> > Matt S.
> >
> 
> 
> 
> Met vriendelijke groet / kind regards,
> 
> Mike Looijmans
> System Expert
> 
> 
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) (0) 499 33 69 79
> Telefax:  (+31) (0) 499 33 69 70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
> 
> Please consider the environment before printing this e-mail
> 
> Topic zoekt gedreven (embedded) software specialisten!
> http://topic.nl/vacatures/topic-zoekt-software-engineers/
> 
> 
> --
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: Support for writable files with a read-only root file system

[Anders Darander]

* Matt Schuckmann <Matt.Schuckmann@planar.com> [141126 18:22]:
> Yes you are correct I want a small number of files into to reside in a
> writeable partition with links in the root file system to point to
> them. I guess I latched on to the volatiles mechanism because it seems
> close to meeting my needs.

The volatile-binds from meta-ro-rootfs can be used to handle exactly
this. (I.e. they doesn't have to point to tmpfs).

The 3rd alternative is the old classical approach of using a symbolic
links...

> Making entire directory hierarchies like /etc writeable seems risky,
> things that you never want changed could get changed. Furthermore it
> makes it harder to deal with setting up default versions of files. 

> Is there really no mechanism for this type of thing in OE? It seems
> like such a common thing for anybody creating a system with a ro root
> file system to need or am I missing something. 

It depends on your which version you're running. Up to daisy, you'd have
to use meta-ro-rootfs to get access to volatile-binds.

From dizzy onward, volatile-bind was merged to oe-core, so there's no
need for an additional layer nowadays.

> If there is not such a mechanism then I will probably recreate the
> VOLATILE_BINDS mechanism that was in the MentorEmbedded layer and
> maybe call it PERSISTENT_BINDS or some such. I'm pretty new to OE so
> whatever I come up with will be pretty rough but I'd be willing to
> share my solution if there was interest. 

Use volatile-binds.

Cheers,
Anders

-- 
Never make any mistaeks.
		-- Anonymous, in a mail discussion about to a kernel bug report

Re: Support for writable files with a read-only root file system

[Jens Rehsack]

Am 28.11.2014 um 14:51 schrieb Anders Darander <anders@chargestorm.se>:

> * Matt Schuckmann <Matt.Schuckmann@planar.com> [141126 18:22]:
>> Yes you are correct I want a small number of files into to reside in a
>> writeable partition with links in the root file system to point to
>> them. I guess I latched on to the volatiles mechanism because it seems
>> close to meeting my needs.
> 
> The volatile-binds from meta-ro-rootfs can be used to handle exactly
> this. (I.e. they doesn't have to point to tmpfs).
> 
> The 3rd alternative is the old classical approach of using a symbolic
> links...

In our project we mix the usage of volatiles, symblic links and unionfs.
You can take a look into https://github.com/rehsack/meta-jens (or
https://github.com/rehsack/yocto-builddir/tree/master/conf for other
used layers) to get a picture.

All used utilities for basic results (minimal image ...) like mkimgs,
flash-device.sh for burning mmc's from nfs boot etc. are included.

I think that should work with branches before daisy as well.

Good luck
-- 
Jens Rehsack
rehsack@gmail.com