[PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[Jason Wessel]

I am not exactly sure how to fix this, so I thought I might ask in the
form of a working patch.  The problem is that I want to use an
external CCACHE_DIR on some build servers, but use the defaults on
others.  Ultimately the sstate sums should be the same in either case,
but they are not due to the way that bitbake tracks the "export"
variables for inclusion i the sum dependencies.  My example test is to
simply set CCACHE_DIR = "/tmp/ccache" in the local.conf and recompile
the quilt-native package.

I ended up adding another check to the dependency generator because
simply trying to use "unexport" had undesired effects.  If there is a
better or correct way of fixing this such that CCACHE_DIR will not
impact the sstate sum, I would really like to know how to do this. :-)

Thanks,
Jason.

[PATCH 1/2] bitbake/lib/bb/data.py: Allow an exported variable to be excluded from dependency processing

[Jason Wessel]

This change is intended to allow an exported variable to not impact
the sstate sum in oe-core.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 lib/bb/data.py |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/bb/data.py b/lib/bb/data.py
index c0636e1..a68ab43 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -313,7 +313,7 @@ def build_dependencies(key, keys, shelldeps, vardepvals, d):
 def generate_dependencies(d):
 
     keys = set(key for key in d.keys() if not key.startswith("__"))
-    shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport"))
+    shelldeps = set(key for key in keys if d.getVarFlag(key, "export") and not d.getVarFlag(key, "unexport") and not d.getVarFlag(key, "undep"))
     vardepvals = set(key for key in keys if d.getVarFlag(key, "vardepvalue"))
 
     deps = {}
-- 
1.7.1

[PATCH 2/2] bitbake.conf: A change to CCACHE_DIR should not change the sstate sum

[Jason Wessel]

If the location of the CCACHE_DIR changes it should not impact the
sstate sum.  This allows a build generated with an external CCACHE_DIR
to have the same sstate sum as a build that uses the ccache dir
located in the temp build area.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/conf/bitbake.conf |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9f4e4d4..9085779 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -413,6 +413,8 @@ CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}"
 TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
 
 export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
+# The sstate cache dependencies should not depend on the value of CCACHE_DIR
+CCACHE_DIR[undep] = "1"
 export CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
 export CXX = "${CCACHE}${HOST_PREFIX}g++ ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
 export F77 = "${CCACHE}${HOST_PREFIX}g77 ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
-- 
1.7.1

Re: [PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[Chris Larson]

On Sun, May 13, 2012 at 7:28 PM, Jason Wessel
<jason.wessel@windriver.com> wrote:
> I am not exactly sure how to fix this, so I thought I might ask in the
> form of a working patch.  The problem is that I want to use an
> external CCACHE_DIR on some build servers, but use the defaults on
> others.  Ultimately the sstate sums should be the same in either case,
> but they are not due to the way that bitbake tracks the "export"
> variables for inclusion i the sum dependencies.  My example test is to
> simply set CCACHE_DIR = "/tmp/ccache" in the local.conf and recompile
> the quilt-native package.
>
> I ended up adding another check to the dependency generator because
> simply trying to use "unexport" had undesired effects.  If there is a
> better or correct way of fixing this such that CCACHE_DIR will not
> impact the sstate sum, I would really like to know how to do this. :-)

Add it to BB_HASHBASE_WHITELIST next to other variables like TMPDIR,
FILE, and BBPATH.
-- 
Christopher Larson

Re: [PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[Jason Wessel]

On 05/13/2012 09:47 PM, Chris Larson wrote:
> On Sun, May 13, 2012 at 7:28 PM, Jason Wessel
> <jason.wessel@windriver.com> wrote:
>> I am not exactly sure how to fix this, so I thought I might ask in the
>> form of a working patch.  The problem is that I want to use an
>> external CCACHE_DIR on some build servers, but use the defaults on
>> others.  Ultimately the sstate sums should be the same in either case,
>> but they are not due to the way that bitbake tracks the "export"
>> variables for inclusion i the sum dependencies.  My example test is to
>> simply set CCACHE_DIR = "/tmp/ccache" in the local.conf and recompile
>> the quilt-native package.
>>
>> I ended up adding another check to the dependency generator because
>> simply trying to use "unexport" had undesired effects.  If there is a
>> better or correct way of fixing this such that CCACHE_DIR will not
>> impact the sstate sum, I would really like to know how to do this. :-)
> 
> Add it to BB_HASHBASE_WHITELIST next to other variables like TMPDIR,
> FILE, and BBPATH.


Many thanks for the response, this definitely works for the CCACHE_DIR case.

I do have a question however.  With the patch I had created I was also able to add new exported variables to the local.conf like CCACHE_DISABLE on demand  Example:

export CCACHE_DISABLE = "1"
CCACHE_DISABLE[undep] = "1"

I would like to be able to do the same thing with the BB_HASHBASE_WHITELIST, like:

export CCACHE_DISABLE = "1"
BB_HASHBASE_WHITELIST += "CCACHE_DISABLE"

My question is if it would be ok to move the hashbase setup above all the requires *.conf lines such that this possible?

Thanks,
Jason.

Re: [PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[Khem Raj]

On Sun, May 13, 2012 at 7:28 PM, Jason Wessel
<jason.wessel@windriver.com> wrote:
> I am not exactly sure how to fix this, so I thought I might ask in the
> form of a working patch.  The problem is that I want to use an
> external CCACHE_DIR on some build servers, but use the defaults on
> others.  Ultimately the sstate sums should be the same in either case,

hmmm so how do you ensure that both ccache stashes are exactly same ?

Re: [PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[Jason Wessel]

On 05/14/2012 04:33 PM, Khem Raj wrote:
> On Sun, May 13, 2012 at 7:28 PM, Jason Wessel
> <jason.wessel@windriver.com> wrote:
>> I am not exactly sure how to fix this, so I thought I might ask in the
>> form of a working patch.  The problem is that I want to use an
>> external CCACHE_DIR on some build servers, but use the defaults on
>> others.  Ultimately the sstate sums should be the same in either case,
> hmmm so how do you ensure that both ccache stashes are exactly same ?

Why do they need to be the same?  

The end result of the compiler is the same with ccache, and that is all that matters in this case, meaning you get the same .o with "ccache gcc".  I might have several ccache directories with different sizes on different systems (it takes ~3.4 gigs for a full ccache).

CCACHE_DIR=/tmp/place1 ccache target-gcc ....
CCACHE_DIR=/tmp/place2 ccache target-gcc ....

Regardless of the state of the CCACHE_DIR the .o will be the same.

Jason.

Re: [PATCH 0/2] RFC - CCACHE_DIR to not impact sstate

[Khem Raj]

On Mon, May 14, 2012 at 2:50 PM, Jason Wessel
<jason.wessel@windriver.com> wrote:
>
> Regardless of the state of the CCACHE_DIR the .o will be the same.

and .o are part of checksum right ?