[PATCH] ccache: Separate out into its own class

[PATCH] ccache: Separate out into its own class

[Richard Purdie]

Currently, ccache is used if it is present. When building from scratch it gives
no performance improvement and creates a ton of empty directories even when its
not in use.

This change moves ccache support to a bbclass file which the user can choose to
enable. This should make builds more determinstic and make it easier/clearer
to the end user when its being used and when it is not.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index b166d62..17aa5e1 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -271,7 +271,7 @@ python base_eventhandler() {
 }
 
 addtask configure after do_patch
-do_configure[dirs] = "${CCACHE_DIR} ${S} ${B}"
+do_configure[dirs] = "${S} ${B}"
 do_configure[deptask] = "do_populate_sysroot"
 base_do_configure() {
 	:
diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
new file mode 100644
index 0000000..10f9b9f
--- a/dev/null
+++ b/meta/classes/ccache.bbclass
@@ -0,0 +1,5 @@
+CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}"
+export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
+
+do_configure[dirs] =+ "${CCACHE_DIR}"
+do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
diff --git a/meta/classes/kernel-yocto.bbclass b/meta/classes/kernel-yocto.bbclass
index e931630..2a3373c 100644
--- a/meta/classes/kernel-yocto.bbclass
+++ b/meta/classes/kernel-yocto.bbclass
@@ -161,7 +161,7 @@ do_kernel_checkout[dirs] = "${S}"
 
 addtask kernel_checkout before do_patch after do_unpack
 
-do_kernel_configme[dirs] = "${CCACHE_DIR} ${S} ${B}"
+do_kernel_configme[dirs] = "${S} ${B}"
 do_kernel_configme() {
 	echo "[INFO] doing kernel configme"
 
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 5fda36b..48287a7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -409,10 +409,9 @@ export PATH
 # Build utility info.
 ##################################################################
 
-CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}"
+CCACHE ??= ""
 TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
 
-export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
 export CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
 export CXX = "${CCACHE}${HOST_PREFIX}g++ ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
 export F77 = "${CCACHE}${HOST_PREFIX}g77 ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
@@ -746,7 +745,7 @@ BB_CONSOLELOG ?= "${TMPDIR}/cooker.log.${DATETIME}"
 
 # Setup our default hash policy
 BB_SIGNATURE_HANDLER ?= "OEBasic"
-BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN"
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS SSH_AGENT_PID XDG_SESSION_COOKIE SSH_AUTH_SOCK XAUTHORITY"
 
 MLPREFIX ??= ""

Re: [PATCH] ccache: Separate out into its own class

[Jason Wessel]

On 05/29/2012 07:29 AM, Richard Purdie wrote:
> Currently, ccache is used if it is present. When building from scratch it gives
> no performance improvement and creates a ton of empty directories even when its
> not in use.
>
> This change moves ccache support to a bbclass file which the user can choose to
> enable. This should make builds more determinstic and make it easier/clearer
> to the end user when its being used and when it is not.

Are the sstate sums the same with and without the use of the new bbclass ccache?

It is not clear to me that this is or is not the case based on this single commit header.

Jason.

Re: [PATCH] ccache: Separate out into its own class

[Richard Purdie]

On Tue, 2012-05-29 at 07:36 -0500, Jason Wessel wrote:
> On 05/29/2012 07:29 AM, Richard Purdie wrote:
> > Currently, ccache is used if it is present. When building from scratch it gives
> > no performance improvement and creates a ton of empty directories even when its
> > not in use.
> >
> > This change moves ccache support to a bbclass file which the user can choose to
> > enable. This should make builds more determinstic and make it easier/clearer
> > to the end user when its being used and when it is not.
> 
> Are the sstate sums the same with and without the use of the new bbclass ccache?
> 
> It is not clear to me that this is or is not the case based on this single commit header.

There is a problem with the existing sstate checksums since CCACHE
itself is not being excluded, only CCACHE_DIR so you'd end up with
sstate built using ccache and sstate built without using cccache with
different checksums. This change unifies those to have matching sstate
checksums.

So with and without use of the class, the checksums should now be the
same.

Cheers,

Richard

Re: [PATCH] ccache: Separate out into its own class

[Enrico Scholz]

Richard Purdie
<richard.purdie-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org>
writes:

> Currently, ccache is used if it is present. When building from scratch
> it gives no performance improvement

Here, I have a 10% cache hit ratio in a scratch build:

cache hit (direct)                  2841
cache hit (preprocessed)            1370
cache miss                         39957
called for link                     3369
called for preprocessing            3207
multiple source files                  8
compile failed                      2531
preprocessor error                   432
couldn't find the compiler           111
bad compiler arguments               468
unsupported source language          532
autoconf compile/link               6613
unsupported compiler option          132
no input file                       2919
files in cache                     94205
cache size                           2.0 Gbytes
max cache size                       3.0 Gbytes



Enrico

Re: [PATCH] ccache: Separate out into its own class

[Richard Purdie]

On Tue, 2012-05-29 at 15:40 +0200, Enrico Scholz wrote:
> Richard Purdie
> <richard.purdie-hQyY1W1yCW8ekmWlsbkhG0B+6BGkLq7r@public.gmane.org>
> writes:
> 
> > Currently, ccache is used if it is present. When building from scratch
> > it gives no performance improvement
> 
> Here, I have a 10% cache hit ratio in a scratch build:
> 
> cache hit (direct)                  2841
> cache hit (preprocessed)            1370
> cache miss                         39957
> called for link                     3369
> called for preprocessing            3207
> multiple source files                  8
> compile failed                      2531
> preprocessor error                   432
> couldn't find the compiler           111
> bad compiler arguments               468
> unsupported source language          532
> autoconf compile/link               6613
> unsupported compiler option          132
> no input file                       2919
> files in cache                     94205
> cache size                           2.0 Gbytes
> max cache size                       3.0 Gbytes
> 

That's good data, thanks for sharing it. In any measurement I've made,
it doesn't seem to help the wall time for the build though :(

We're not removing the ability to use ccache, just changing the way its
enabled which all things considered I still think is a good move...

Cheers,

Richard