[PATCH 0/5] Remove /var/cache from volatiles directory

[PATCH 0/5] Remove /var/cache from volatiles directory

[Laurentiu Palcu]

Hi,

Re-generating applications' cache every time the system is rebooted is not a
very efficient process for an embedded device. Usually, the cache directory is
used by applications to store data resulting from time consmuming I/O or
calculation. So, this patchset will take out /var/cache from tmpfs.

Thanks,
Laurentiu

The following changes since commit 75f470cd18d693a9a96d9849291c2c8de4dcbeb8:

  qt4: Add space for _appends (2013-02-01 22:49:47 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lpalcu/remove_cache_from_volatiles
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/remove_cache_from_volatiles

Laurentiu Palcu (5):
  base-files: remove /var/cache from volatiles
  fs-perms: remove /var/cache from volatiles
  initscripts: remove /var/cache from volatiles
  systemd: remove /var/cache from volatiles
  rpm: remove /var/volatiles/cache/rpm from the FILEs list

 meta/files/fs-perms.txt                            |    1 -
 meta/recipes-core/base-files/base-files_3.0.14.bb  |    4 ++--
 .../initscripts/initscripts-1.0/volatiles          |    2 --
 .../systemd/systemd/00-create-volatile.conf        |    1 -
 meta/recipes-devtools/rpm/rpm_5.4.9.bb             |    1 -
 5 files changed, 2 insertions(+), 7 deletions(-)

-- 
1.7.9.5

[PATCH 1/5] base-files: remove /var/cache from volatiles

[Laurentiu Palcu]

Having the cache in tmpfs implies cache regeneration after every reboot.
For an embedded device this might not be very efficient. So, it is
better for the cache to be persistent between reboots.

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/recipes-core/base-files/base-files_3.0.14.bb |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/base-files/base-files_3.0.14.bb b/meta/recipes-core/base-files/base-files_3.0.14.bb
index a90e995..6a3db50 100644
--- a/meta/recipes-core/base-files/base-files_3.0.14.bb
+++ b/meta/recipes-core/base-files/base-files_3.0.14.bb
@@ -42,7 +42,7 @@ dirs755 = "/bin /boot /dev ${sysconfdir} ${sysconfdir}/default \
            ${mandir} ${datadir}/misc ${localstatedir} \
            ${localstatedir}/backups ${localstatedir}/lib \
            /sys ${localstatedir}/lib/misc ${localstatedir}/spool \
-           ${localstatedir}/volatile ${localstatedir}/volatile/cache \
+           ${localstatedir}/volatile \
            ${localstatedir}/volatile/lock/subsys \
            ${localstatedir}/volatile/log \
            ${localstatedir}/volatile/run \
@@ -55,7 +55,7 @@ dirs3755 = "/srv  \
             ${prefix}/local/share ${prefix}/local/src"
 dirs4775 = "/var/mail"
 
-volatiles = "cache run log lock tmp"
+volatiles = "run log lock tmp"
 conffiles = "${sysconfdir}/debian_version ${sysconfdir}/host.conf \
              ${sysconfdir}/inputrc ${sysconfdir}/issue /${sysconfdir}/issue.net \
              ${sysconfdir}/nsswitch.conf ${sysconfdir}/profile \
-- 
1.7.9.5

[PATCH 2/5] fs-perms: remove /var/cache from volatiles

[Laurentiu Palcu]

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/files/fs-perms.txt |    1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/files/fs-perms.txt b/meta/files/fs-perms.txt
index f5a2b69..ef44e26 100644
--- a/meta/files/fs-perms.txt
+++ b/meta/files/fs-perms.txt
@@ -45,7 +45,6 @@ ${oldincludedir}	0755	root	root	true	0644	root	root
 
 # Items from base-files
 # Links
-${localstatedir}/cache	link	volatile/cache
 ${localstatedir}/run	link	volatile/run
 ${localstatedir}/log	link	volatile/log
 ${localstatedir}/lock	link	volatile/lock
-- 
1.7.9.5

[PATCH 3/5] initscripts: remove /var/cache from volatiles

[Laurentiu Palcu]

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 .../initscripts/initscripts-1.0/volatiles          |    2 --
 1 file changed, 2 deletions(-)

diff --git a/meta/recipes-core/initscripts/initscripts-1.0/volatiles b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
index e0741aa..087ab0a 100644
--- a/meta/recipes-core/initscripts/initscripts-1.0/volatiles
+++ b/meta/recipes-core/initscripts/initscripts-1.0/volatiles
@@ -21,12 +21,10 @@
 # This explains the order of "link before object" as in the example above, where
 # a link will be created at /var/test pointing to /tmp/testfile and due to this
 # link the file defined as /var/test will actually be created as /tmp/testfile.
-d root root 0755 /var/volatile/cache none
 d root root 1777 /var/volatile/lock none
 d root root 0755 /var/volatile/log none
 d root root 0755 /var/volatile/run none
 d root root 1777 /var/volatile/tmp none
-l root root 0755 /var/cache /var/volatile/cache
 l root root 1777 /var/lock /var/volatile/lock
 l root root 0755 /var/log /var/volatile/log
 l root root 0755 /var/run /var/volatile/run
-- 
1.7.9.5

[PATCH 4/5] systemd: remove /var/cache from volatiles

[Laurentiu Palcu]

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 .../systemd/systemd/00-create-volatile.conf        |    1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-core/systemd/systemd/00-create-volatile.conf b/meta/recipes-core/systemd/systemd/00-create-volatile.conf
index d160a9b..6163a9c 100644
--- a/meta/recipes-core/systemd/systemd/00-create-volatile.conf
+++ b/meta/recipes-core/systemd/systemd/00-create-volatile.conf
@@ -3,7 +3,6 @@
 # inside /var/log.
 
 
-d		/var/volatile/cache	-	-	-	-
 d		/var/volatile/log		-	-	-	-
 d		/var/volatile/lock		-	-	-	-
 d		/var/volatile/run		-	-	-	-
-- 
1.7.9.5

[PATCH 5/5] rpm: remove /var/volatiles/cache/rpm from the FILEs list

[Laurentiu Palcu]

Since /var/cache is not in volatiles anymore, this entry has to go.

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/recipes-devtools/rpm/rpm_5.4.9.bb |    1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-devtools/rpm/rpm_5.4.9.bb b/meta/recipes-devtools/rpm/rpm_5.4.9.bb
index 881b960..39b0481 100644
--- a/meta/recipes-devtools/rpm/rpm_5.4.9.bb
+++ b/meta/recipes-devtools/rpm/rpm_5.4.9.bb
@@ -223,7 +223,6 @@ FILES_${PN} =  "${bindir}/rpm \
 		${libdir}/rpm/bin/wget \
 		/var/lib/rpm \
 		/var/cache/rpm \
-		/var/volatile/cache/rpm \
 		"
 
 FILES_${PN}-dbg += "${libdir}/rpm/.debug \
-- 
1.7.9.5

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Mike Looijmans]

On 02/04/2013 10:26 AM, Laurentiu Palcu wrote:
> Hi,
>
> Re-generating applications' cache every time the system is rebooted is not a
> very efficient process for an embedded device. Usually, the cache directory is
> used by applications to store data resulting from time consmuming I/O or
> calculation. So, this patchset will take out /var/cache from tmpfs.

What if you have a READONLY rootfs? This patch will "break" those 
devices I think.


>
> Thanks,
> Laurentiu
>
> The following changes since commit 75f470cd18d693a9a96d9849291c2c8de4dcbeb8:
>
>    qt4: Add space for _appends (2013-02-01 22:49:47 +0000)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib lpalcu/remove_cache_from_volatiles
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/remove_cache_from_volatiles
>
> Laurentiu Palcu (5):
>    base-files: remove /var/cache from volatiles
>    fs-perms: remove /var/cache from volatiles
>    initscripts: remove /var/cache from volatiles
>    systemd: remove /var/cache from volatiles
>    rpm: remove /var/volatiles/cache/rpm from the FILEs list
>
>   meta/files/fs-perms.txt                            |    1 -
>   meta/recipes-core/base-files/base-files_3.0.14.bb  |    4 ++--
>   .../initscripts/initscripts-1.0/volatiles          |    2 --
>   .../systemd/systemd/00-create-volatile.conf        |    1 -
>   meta/recipes-devtools/rpm/rpm_5.4.9.bb             |    1 -
>   5 files changed, 2 insertions(+), 7 deletions(-)
>

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Laurentiu Palcu]

On 02/04/2013 11:46 AM, Mike Looijmans wrote:
> On 02/04/2013 10:26 AM, Laurentiu Palcu wrote:
>> > Hi,
>> >
>> > Re-generating applications' cache every time the system is rebooted is not a
>> > very efficient process for an embedded device. Usually, the cache directory is
>> > used by applications to store data resulting from time consmuming I/O or
>> > calculation. So, this patchset will take out /var/cache from tmpfs.
> What if you have a READONLY rootfs? This patch will "break" those 
> devices I think.
> 
> 

Not necessarily. We are trying to have all postinstalls run on host, so
we don't have to do any postinstall activity on target. Any cache
generated during postinstall, will be generated at do_rootfs time. Then,
we can deploy the image with the cache already in place.

Also, on a RO rootfs, there is really no need to update the cache since
the cached files do not really change. Hence, the cache created on host
should, in theory, be valid.

If the above logic is missing anything, feel free to correct me.

Thanks,
Laurentiu

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Richard Purdie]

On Mon, 2013-02-04 at 12:03 +0200, Laurentiu Palcu wrote:
> 
> On 02/04/2013 11:46 AM, Mike Looijmans wrote:
> > On 02/04/2013 10:26 AM, Laurentiu Palcu wrote:
> >> > Hi,
> >> >
> >> > Re-generating applications' cache every time the system is rebooted is not a
> >> > very efficient process for an embedded device. Usually, the cache directory is
> >> > used by applications to store data resulting from time consmuming I/O or
> >> > calculation. So, this patchset will take out /var/cache from tmpfs.
> > What if you have a READONLY rootfs? This patch will "break" those 
> > devices I think.
> > 
> > 
> 
> Not necessarily. We are trying to have all postinstalls run on host, so
> we don't have to do any postinstall activity on target. Any cache
> generated during postinstall, will be generated at do_rootfs time. Then,
> we can deploy the image with the cache already in place.
> 
> Also, on a RO rootfs, there is really no need to update the cache since
> the cached files do not really change. Hence, the cache created on host
> should, in theory, be valid.
> 
> If the above logic is missing anything, feel free to correct me.

If this is an issue and something does need to touch the cache
directory, we can mount a unionfs over /var/cache like we're doing
for /var/lib in the readonly rootfs patch series.

Ultimately, I think we're better served with that approach all round.

Cheers,

Richard

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Phil Blundell]

On Mon, 2013-02-04 at 12:03 +0200, Laurentiu Palcu wrote:
> Not necessarily. We are trying to have all postinstalls run on host, so
> we don't have to do any postinstall activity on target. Any cache
> generated during postinstall, will be generated at do_rootfs time. Then,
> we can deploy the image with the cache already in place.
> 
> Also, on a RO rootfs, there is really no need to update the cache since
> the cached files do not really change. Hence, the cache created on host
> should, in theory, be valid.

What sort of files in /var/cache are you trying to address with this
change?  The impression I have is that most of the things in there are
created dynamically by applications running on the target system and not
by postinst scripts that could be run on the host.

For example, an (admittedly somewhat unscientific) survey on my desktop
machine suggests that fontconfig, samba and cups, at least, are creating
files in /var/cache at run time with contents that you wouldn't be able
to predict in advance.  (Apt also uses /var/cache as its download area,
but I guess if you have a read-only roofs then you aren't going to be
running apt very much so that one probably isn't an issue.)  I wasn't
able to immediately identify any files in there that would be obvious
candidates for creating in a postinst and then leaving untouched
afterwards.

p.

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Burton, Ross]

On 4 February 2013 17:47, Phil Blundell <pb@pbcl.net> wrote:
> For example, an (admittedly somewhat unscientific) survey on my desktop
> machine suggests that fontconfig, samba and cups, at least, are creating
> files in /var/cache at run time with contents that you wouldn't be able
> to predict in advance

The fontconfig cache is perfectly capable of generation at image
creation time, and that's actually what prompted removal of /var/cache
from volatiles.

If samba and cups are using /var/cache as something that isn't a
persistent cache, then the should use something like /var/run/
instead.

Ross

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Enrico Scholz]

"Burton, Ross" <ross.burton-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
writes:

> The fontconfig cache is perfectly capable of generation at image
> creation time,

really? The cache files are architecture dependent and I am not aware of
a cross-fc-cache tool.



Enrico

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Burton, Ross]

On 4 February 2013 18:25, Enrico Scholz <enrico.scholz@sigma-chemnitz.de> wrote:
> really? The cache files are architecture dependent and I am not aware of
> a cross-fc-cache tool.

That's right, they are.  Laurentiu's been working on that too:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=lpalcu/fontconfig&id=fe7c0e0b2c92c36466ac6de02a98d13b3985282e

Ross

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[ChenQi]

On 02/04/2013 05:26 PM, Laurentiu Palcu wrote:
> Hi,
>
> Re-generating applications' cache every time the system is rebooted is not a
> very efficient process for an embedded device. Usually, the cache directory is
> used by applications to store data resulting from time consmuming I/O or
> calculation. So, this patchset will take out /var/cache from tmpfs.
>
> Thanks,
> Laurentiu
>
> The following changes since commit 75f470cd18d693a9a96d9849291c2c8de4dcbeb8:
>
>    qt4: Add space for _appends (2013-02-01 22:49:47 +0000)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib lpalcu/remove_cache_from_volatiles
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/remove_cache_from_volatiles
>
> Laurentiu Palcu (5):
>    base-files: remove /var/cache from volatiles
>    fs-perms: remove /var/cache from volatiles
>    initscripts: remove /var/cache from volatiles
>    systemd: remove /var/cache from volatiles
>    rpm: remove /var/volatiles/cache/rpm from the FILEs list
>
>   meta/files/fs-perms.txt                            |    1 -
>   meta/recipes-core/base-files/base-files_3.0.14.bb  |    4 ++--
>   .../initscripts/initscripts-1.0/volatiles          |    2 --
>   .../systemd/systemd/00-create-volatile.conf        |    1 -
>   meta/recipes-devtools/rpm/rpm_5.4.9.bb             |    1 -
>   5 files changed, 2 insertions(+), 7 deletions(-)
>
what about /var/log?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=3397

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Laurentiu Palcu]

On 02/05/2013 04:37 AM, ChenQi wrote:
> what about /var/log?
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=3397
I wasn't aware this bug existed... Sorry for the inconvenience. I'm not
working on moving /var/log. I had to move the /var/cache though because
of the postinstall work I've been doing.

Thanks,
Laurentiu

Re: [PATCH 0/5] Remove /var/cache from volatiles directory

[Saul Wold]

On 02/04/2013 01:26 AM, Laurentiu Palcu wrote:
> Hi,
>
> Re-generating applications' cache every time the system is rebooted is not a
> very efficient process for an embedded device. Usually, the cache directory is
> used by applications to store data resulting from time consmuming I/O or
> calculation. So, this patchset will take out /var/cache from tmpfs.
>
I did a basic test with this change along with the "read-only-rootfs" 
IMAGE_FEATURE and things are looking pretty good.  The only thing I 
noticed is that we leave some /etc/*- files around, I know it's not this 
patch.

Acked-by: Saul Wold <sgw@linux.intel.com>

Thanks
	Sau!
> Thanks,
> Laurentiu
>
> The following changes since commit 75f470cd18d693a9a96d9849291c2c8de4dcbeb8:
>
>    qt4: Add space for _appends (2013-02-01 22:49:47 +0000)
>
> are available in the git repository at:
>
>    git://git.yoctoproject.org/poky-contrib lpalcu/remove_cache_from_volatiles
>    http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/remove_cache_from_volatiles
>
> Laurentiu Palcu (5):
>    base-files: remove /var/cache from volatiles
>    fs-perms: remove /var/cache from volatiles
>    initscripts: remove /var/cache from volatiles
>    systemd: remove /var/cache from volatiles
>    rpm: remove /var/volatiles/cache/rpm from the FILEs list
>
>   meta/files/fs-perms.txt                            |    1 -
>   meta/recipes-core/base-files/base-files_3.0.14.bb  |    4 ++--
>   .../initscripts/initscripts-1.0/volatiles          |    2 --
>   .../systemd/systemd/00-create-volatile.conf        |    1 -
>   meta/recipes-devtools/rpm/rpm_5.4.9.bb             |    1 -
>   5 files changed, 2 insertions(+), 7 deletions(-)
>