[PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Jason Wessel]

The end user of oe-core should be free to turn off ccache, or use an
external ccache without impacting the sstate sums.

Example of using a different ccache directory via local.conf

CCACHE_DIR = "/tmp/ccache"

Example of turning off ccache:

export CCACHE_DISABLE = "1"
BB_HASHBASE_WHITELIST += "CCACHE_DISABLE"

Jason.

[PATCH 1/2] bitbake.conf: Move default hash policy initialization before conf file inclusion

[Jason Wessel]

It is desirable to be able to augment the default hash policy from
local.conf if adding new exported variables that do not impact the end
result of a build, and in order to do this the default initialization
must occur prior to the conf file inclusion.

This is an example of a change that should not impact the sstate sum
that an end user might change in the local.conf.

export CCACHE_DISABLE = "1"
BB_HASHBASE_WHITELIST += "CCACHE_DISABLE"

There are a myriad of reasons you might set this ranging from
debugging a ccache issue to the desire to never construct the ccache
at all, either way the end result of the binaries produced is always
the same.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/conf/bitbake.conf |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index dc68690..12bca0e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -653,6 +653,11 @@ MACHINEOVERRIDES[vardepsexclude] = "MACHINE"
 CPU_FEATURES ?= ""
 CPU_FEATURES_arm ?= "vfp"
 
+# Setup our default hash policy
+BB_SIGNATURE_HANDLER ?= "OEBasic"
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE"
+BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS SSH_AGENT_PID XDG_SESSION_COOKIE SSH_AUTH_SOCK XAUTHORITY"
+
 ##################################################################
 # Include the rest of the config files.
 ##################################################################
@@ -744,10 +749,5 @@ TRANSLATED_TARGET_ARCH ??= "${@d.getVar('TARGET_ARCH', True).replace("_", "-")}"
 # Complete output from bitbake, if you're changing it, make sure to use existing directory
 BB_CONSOLELOG ?= "${TMPDIR}/cooker.log.${DATETIME}"
 
-# Setup our default hash policy
-BB_SIGNATURE_HANDLER ?= "OEBasic"
-BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE"
-BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS SSH_AGENT_PID XDG_SESSION_COOKIE SSH_AUTH_SOCK XAUTHORITY"
-
 MLPREFIX ??= ""
 MULTILIB_VARIANTS ??= ""
-- 
1.7.1

[PATCH 2/2] bitbake.conf: Add CCACHE_DIR to BB_HASHBASE_WHITELIST

[Jason Wessel]

The location of the ccache should not affect the sum in the sstate
files.

The end user should be free to use an external ccache via defining
CCACHE_DIR to a new value in local.conf or the default per package
local ccache in the temp directory.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/conf/bitbake.conf |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 12bca0e..8fe3c2c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -655,7 +655,7 @@ CPU_FEATURES_arm ?= "vfp"
 
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "OEBasic"
-BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE"
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS SSH_AGENT_PID XDG_SESSION_COOKIE SSH_AUTH_SOCK XAUTHORITY"
 
 ##################################################################
-- 
1.7.1

Re: [PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Jason Wessel]

On 05/14/2012 11:09 AM, Jason Wessel wrote:
> The end user of oe-core should be free to turn off ccache, or use an
> external ccache without impacting the sstate sums.
> 
> Example of using a different ccache directory via local.conf
> 
> CCACHE_DIR = "/tmp/ccache"
> 
> Example of turning off ccache:
> 
> export CCACHE_DISABLE = "1"
> BB_HASHBASE_WHITELIST += "CCACHE_DISABLE"


Oops.  I got this totally wrong...  It is bogus to move around the defaults in the core bitbake.conf.

All that needs to be done for the second case is to add to my local.conf:

export CCACHE_DISABLE = "1"
BB_HASHBASE_WHITELIST_append += "CCACHE_DISABLE"

And that doesn't require any changes to the bitbake.conf.

Sorry for the noise.   I'll send a patch just to add CCACHE_DIR to the BB_HASHBASE_WHITELIST.

Cheers,
Jason.

Re: [PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Marko Lindqvist]

On 14 May 2012 19:09, Jason Wessel <jason.wessel@windriver.com> wrote:
> The end user of oe-core should be free to turn off ccache, or use an
> external ccache without impacting the sstate sums.

 ...

> I'll send a patch just to add CCACHE_DIR to the BB_HASHBASE_WHITELIST.

 External cache makes sense for anything being built by system
compiler. Caching results of anything produced by OE-built compilers
to user's normal cache is problematic. Rebuild of the compiler itself
will invalidate any such cached files, but caching them in first place
may push more valid results out from the cache. So you are proposing
that user has to opt-out from that behavior (by unsetting his normal
CCACHE_DIR before running bitbake) and no longer opt-in (by adding
CCACHE_DIR to whitelist)


 - ML

Re: [PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Chris Larson]

On Mon, May 14, 2012 at 10:18 AM, Marko Lindqvist <cazfi74@gmail.com> wrote:
>> I'll send a patch just to add CCACHE_DIR to the BB_HASHBASE_WHITELIST.
>
>  External cache makes sense for anything being built by system
> compiler. Caching results of anything produced by OE-built compilers
> to user's normal cache is problematic. Rebuild of the compiler itself
> will invalidate any such cached files, but caching them in first place
> may push more valid results out from the cache. So you are proposing
> that user has to opt-out from that behavior (by unsetting his normal
> CCACHE_DIR before running bitbake) and no longer opt-in (by adding
> CCACHE_DIR to whitelist)

Whether you like or don't like ccache is irrelevant. I personally
don't use it, but regardless, obviously CCACHE_DIR doesn't belong in
the shared state checksums, and CCACHE is the var you use to opt in vs
out of ccache usage.
-- 
Christopher Larson

Re: [PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Jason Wessel]

On 05/14/2012 12:18 PM, Marko Lindqvist wrote:
> On 14 May 2012 19:09, Jason Wessel <jason.wessel@windriver.com> wrote:
>> The end user of oe-core should be free to turn off ccache, or use an
>> external ccache without impacting the sstate sums.
> 
>  ...
> 
>  External cache makes sense for anything being built by system
> compiler. Caching results of anything produced by OE-built compilers
> to user's normal cache is problematic. Rebuild of the compiler itself
> will invalidate any such cached files, but caching them in first place
> may push more valid results out from the cache. 

This is only an issue if the compiler has changed.  In my case it is
not changing, or at least it should be the same each time.  I use one
external ccache for all the embedded target work on a per build server
basis and have done so for years for non-bitbake environments.  I
would like the same functionality from bitbake based builds.

Right now with the external ccache my kernel compiles alone go from >
6 min down to < 30 seconds, so there is a definite desire on my side
to have this work.


> So you are proposing that user has to opt-out from that behavior (by
> unsetting his normal CCACHE_DIR before running bitbake) and no
> longer opt-in (by adding CCACHE_DIR to whitelist)

The defaults remain the defaults, I am not changing anything to that
end.  I only aim to have configurability via the local.conf for
various use cases surrounding ccache.  As I mentioned before I'll just
submit a new patch to add CCACHE_DIR to the hash whitelist so it is
not in the ssatate sum.  All the other problems were a result of pilot
error.

Cheers,
Jason.

Re: [PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Marko Lindqvist]

As you can see from my answer to last part I quote, I'm not against
your patch, so don't read first parts of this email with that mindset.
They are more generic discussion about ccache caches.

On 14 May 2012 23:58, Jason Wessel <jason.wessel@windriver.com> wrote:
>
> This is only an issue if the compiler has changed.  In my case it is
> not changing, or at least it should be the same each time.

 Several years back ccache detected if compiler is the same by binary
size and timestamp, so compiler rebuild (with new timestamp) resulted
contents of the cache getting invalidated. They talked about changing
it to check md5sum of the compiler binary instead, but I haven't
followed if that's implemented already.

>  I use one
> external ccache for all the embedded target work on a per build server
> basis and have done so for years for non-bitbake environments.  I
> would like the same functionality from bitbake based builds.
>
> Right now with the external ccache my kernel compiles alone go from >
> 6 min down to < 30 seconds, so there is a definite desire on my side
> to have this work.

 Yes, I just wanted to make sure it works best way possible in every
possible use-case.

 Optimal results would come from using external (what I refer here as
my "normal" cache) cache for anything outside OpenEmbedded scope, and
for initial cross-compiler builds of OpenEmbedded (as that is done
with external compiler that will remain the same even when one does
complete rebuild), and second ("internal") cache for the rest of
OpenEmbedded build. Contents of latter gets invalid when one does full
rebuild, so it can be removed before one.
 I don't know, but assume that it would be a bit hard to select
correct one from two caches in OpenEmbedded builds (external one for
some builds, internal for others) so optimal realistic setup is that
user's normal external cache is not used at all inside OpenEmbedded.
Now, if user wants to override cache location (and thus create what
one may consider "external" cache) he should be allowed to do so, but
thing here is that the cache one uses normally does not get polluted
accidentally.

>> So you are proposing that user has to opt-out from that behavior (by
>> unsetting his normal CCACHE_DIR before running bitbake) and no
>> longer opt-in (by adding CCACHE_DIR to whitelist)
>
> The defaults remain the defaults, I am not changing anything to that
> end.  I only aim to have configurability via the local.conf for
> various use cases surrounding ccache.  As I mentioned before I'll just
> submit a new patch to add CCACHE_DIR to the hash whitelist so it is
> not in the ssatate sum.  All the other problems were a result of pilot
> error.

 Sorry, I earlier missed the "hash" part, and thought that you were
changing it so that OpenEmbedded would use external CCACHE_DIR unlike
before. That is: to change it to pollute my normal cache.


 - ML

Re: [PATCH 0/2] bitbake.conf: gracefully work with ccache environment variables

[Jason Wessel]

On 05/14/2012 05:00 PM, Marko Lindqvist wrote:
> As you can see from my answer to last part I quote, I'm not against
> your patch, so don't read first parts of this email with that mindset.
> They are more generic discussion about ccache caches.


No worries.

> 
> On 14 May 2012 23:58, Jason Wessel <jason.wessel@windriver.com> wrote:
>>
>> This is only an issue if the compiler has changed.  In my case it is
>> not changing, or at least it should be the same each time.
> 
>  Several years back ccache detected if compiler is the same by binary
> size and timestamp, so compiler rebuild (with new timestamp) resulted
> contents of the cache getting invalidated. They talked about changing
> it to check md5sum of the compiler binary instead, but I haven't
> followed if that's implemented already.


I am not exactly sure if ccache is invaliding based on the compiler or
not.  It is entirely possible the cache is completely invalid for that
case.  For my local use case this is not an issue because the compiler
is _always_ use the same pre-built version.

A modern ccache does have some extra configuration for doing "wild"
things like relative locations (from ccache docs see CCACHE_BASEDIR)
where you can literally share the cache across multiple build
directories.  I have no idea if this feature actually works though.  I
tend to only use basic features of the ccache and build in the same
set of directories every time.

>>
>> Right now with the external ccache my kernel compiles alone go from >
>> 6 min down to < 30 seconds, so there is a definite desire on my side
>> to have this work.
> 
>  Yes, I just wanted to make sure it works best way possible in every
> possible use-case.
> 
>  Optimal results would come from using external (what I refer here as
> my "normal" cache) cache for anything outside OpenEmbedded scope, and
> for initial cross-compiler builds of OpenEmbedded (as that is done
> with external compiler that will remain the same even when one does
> complete rebuild), and second ("internal") cache for the rest of
> OpenEmbedded build. Contents of latter gets invalid when one does full
> rebuild, so it can be removed before one.
>  I don't know, but assume that it would be a bit hard to select
> correct one from two caches in OpenEmbedded builds (external one for
> some builds, internal for others) so optimal realistic setup is that
> user's normal external cache is not used at all inside OpenEmbedded.
> Now, if user wants to override cache location (and thus create what
> one may consider "external" cache) he should be allowed to do so, but
> thing here is that the cache one uses normally does not get polluted
> accidentally.


It appears that the default way the ccache is setup, it is already
doing the right thing, meaning the defaults for bitbake are fine.

For me, there are two basic cases.

 1) create a "project" directory make a ccache dir in there and then
    set CCACHE_DIR in the local.conf
 2) Use an external shared directory for the ccache with a deeper
    than the default hash and size

Both these configurations greatly speed up the incremental rebuilds
and build from scratch cases.

Jason.