[Buildroot] ccache problem with codesourcery toolchain

[Buildroot] ccache problem with codesourcery toolchain

[D M]

Hi,

I am working on a project based on the 2011.08 release of buildroot. ?We found that the ccache savings was not as good as with our older version of buildroot (which used an older version of ccache). ?For example, starting from an empty cache directory, our compile takes about half an hour. ?With the newer version of ccache, we only save about 3 minutes; with the older version, we saved about 13 minutes. ?We traced the problem to a feature introduced in ccache 3.0, the?CCACHE_COMPILERCHECK option. ?Quoting from ccache manual:?

? ?"By default, ccache includes the modification time (?mtime?) and size of the compiler in the hash to ensure that results retrieved from the cache are accurate. The CACHE_COMPILERCHECK environment variable can be used to select another strategy." ?

The default is "mtime", but if we select "none", we get back all of our savings. ?Again quoting from the manual: ?

? ? "none - Don?t hash anything. This may be good for situations where you can safely use the cached results even though the compiler?s mtime or size has changed (e.g. if the compiler is built as part of your build system and the compiler?s source has not changed, or if the compiler only has changes that don?t affect code generation)."

Since we are using buildroot's capability to automatically download and extract the CodeSourcery ARM toolchain, the "mtime" used in the hash is different every time we do a full recompile. ?So a value of "none" seems appropriate when use buildroot this way.

Have others ran into this? ?Since buildroot has this kind of CodeSourcery toolchain support, I was wondering if there should be an option in menuconfig to specify CCACHE_COMPILERCHECK, to, for example, prepend a "CCACHE_COMPILERCHECK=none" to the beginning of the CCACHE variable that is used to make TARGET_CC, TARGET_CXX, HOSTCC, HOSTCXX.

Thanks,
Danomi -

[Buildroot] ccache problem with codesourcery toolchain

[Thomas Petazzoni]

Hello Danomi,

Le Sat, 4 Feb 2012 17:05:01 -0800 (PST),
D M <d_mo1234@yahoo.com> a ?crit :

> I am working on a project based on the 2011.08 release of buildroot.
> ?We found that the ccache savings was not as good as with our older
> version of buildroot (which used an older version of ccache). ?For
> example, starting from an empty cache directory, our compile takes
> about half an hour. ?With the newer version of ccache, we only save
> about 3 minutes; with the older version, we saved about 13 minutes.
> ?We traced the problem to a feature introduced in ccache 3.0,
> the?CCACHE_COMPILERCHECK option. ?Quoting from ccache manual:?
> 
> ? ?"By default, ccache includes the modification time (?mtime?) and
> size of the compiler in the hash to ensure that results retrieved
> from the cache are accurate. The CACHE_COMPILERCHECK environment
> variable can be used to select another strategy." ?
> 
> The default is "mtime", but if we select "none", we get back all of
> our savings. ?Again quoting from the manual: ?
> 
> ? ? "none - Don?t hash anything. This may be good for situations
> where you can safely use the cached results even though the
> compiler?s mtime or size has changed (e.g. if the compiler is built
> as part of your build system and the compiler?s source has not
> changed, or if the compiler only has changes that don?t affect code
> generation)."
> 
> Since we are using buildroot's capability to automatically download
> and extract the CodeSourcery ARM toolchain, the "mtime" used in the
> hash is different every time we do a full recompile. ?So a value of
> "none" seems appropriate when use buildroot this way.
> 
> Have others ran into this? ?Since buildroot has this kind of
> CodeSourcery toolchain support, I was wondering if there should be an
> option in menuconfig to specify CCACHE_COMPILERCHECK, to, for
> example, prepend a "CCACHE_COMPILERCHECK=none" to the beginning of
> the CCACHE variable that is used to make TARGET_CC, TARGET_CXX,
> HOSTCC, HOSTCXX.

Good research! Obviously, using the default value of
CCACHE_COMPILERCHECK=mtime will completely defeat the compiler cache in
the context of Buildroot, because the compiler gets re-created pretty
much at every build, so definitely we should arrange for
CCACHE_COMPILERCHECK=none to be set in the ccache environment, or even
directly modify the ccache code to make it the default.

The thing I am curious about is where do the 3 minutes time reduction
come from even when CCACHE_COMPILERCHECK=mtime? Maybe it's the time
reduction gained on compiling host packages, since those always use the
same host compiler.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] ccache problem with codesourcery toolchain

[D M]

Hi Thomas,

>?or even?directly modify the ccache code to make it the default.

I hadn't considered patching ccache itself to make "none" the
default. ?I wasn't sure if other use-cases of buildroot would
actually want to have the "mtime" setting - like maybe a completely
external toolchain? ?So I was envisioning a sub-menu where you could
select 1 of the 3 options to be explicitly set in the CCACHE var, or maybe
1 of 4 to have a dont-specify-anything option; or a text field, to type in
whatever you want - in case you want to specify additional env vars, like
the CCACHE_BASEDIR var, so that parallel buildroot trees can use the?
same cached?objects. ?But for my particular use-case, a patch is as good?
a solution?as any.

> where do the 3 minutes time reduction?come from even when?
> CCACHE_COMPILERCHECK=mtime? ?Maybe it's the time
> reduction gained on compiling host packages,

I didn't try to track that down, but you're probably correct, since the
host tool compiler is the only compiler that didn't change, mtime-wise.

Dan -



----- Original Message -----
From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To: D M <d_mo1234@yahoo.com>
Cc: "buildroot at busybox.net" <buildroot@busybox.net>
Sent: Sunday, February 5, 2012 2:45 AM
Subject: Re: [Buildroot] ccache problem with codesourcery toolchain

Hello Danomi,

Le Sat, 4 Feb 2012 17:05:01 -0800 (PST),
D M <d_mo1234@yahoo.com> a ?crit :

> I am working on a project based on the 2011.08 release of buildroot.
> ?We found that the ccache savings was not as good as with our older
> version of buildroot (which used an older version of ccache). ?For
> example, starting from an empty cache directory, our compile takes
> about half an hour. ?With the newer version of ccache, we only save
> about 3 minutes; with the older version, we saved about 13 minutes.
> ?We traced the problem to a feature introduced in ccache 3.0,
> the?CCACHE_COMPILERCHECK option. ?Quoting from ccache manual:?
> 
> ? ?"By default, ccache includes the modification time (?mtime?) and
> size of the compiler in the hash to ensure that results retrieved
> from the cache are accurate. The CACHE_COMPILERCHECK environment
> variable can be used to select another strategy." ?
> 
> The default is "mtime", but if we select "none", we get back all of
> our savings. ?Again quoting from the manual: ?
> 
> ? ? "none - Don?t hash anything. This may be good for situations
> where you can safely use the cached results even though the
> compiler?s mtime or size has changed (e.g. if the compiler is built
> as part of your build system and the compiler?s source has not
> changed, or if the compiler only has changes that don?t affect code
> generation)."
> 
> Since we are using buildroot's capability to automatically download
> and extract the CodeSourcery ARM toolchain, the "mtime" used in the
> hash is different every time we do a full recompile. ?So a value of
> "none" seems appropriate when use buildroot this way.
> 
> Have others ran into this? ?Since buildroot has this kind of
> CodeSourcery toolchain support, I was wondering if there should be an
> option in menuconfig to specify CCACHE_COMPILERCHECK, to, for
> example, prepend a "CCACHE_COMPILERCHECK=none" to the beginning of
> the CCACHE variable that is used to make TARGET_CC, TARGET_CXX,
> HOSTCC, HOSTCXX.

Good research! Obviously, using the default value of
CCACHE_COMPILERCHECK=mtime will completely defeat the compiler cache in
the context of Buildroot, because the compiler gets re-created pretty
much at every build, so definitely we should arrange for
CCACHE_COMPILERCHECK=none to be set in the ccache environment, or even
directly modify the ccache code to make it the default.

The thing I am curious about is where do the 3 minutes time reduction
come from even when CCACHE_COMPILERCHECK=mtime? Maybe it's the time
reduction gained on compiling host packages, since those always use the
same host compiler.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] ccache problem with codesourcery toolchain

[Arnout Vandecappelle]

On Sunday 05 February 2012 19:11:20 D M wrote:
> > or even directly modify the ccache code to make it the default.
> 
> I hadn't considered patching ccache itself to make "none" the
> default.  I wasn't sure if other use-cases of buildroot would
> actually want to have the "mtime" setting - like maybe a completely
> external toolchain?

 You're right: if you do a make clean, change the config of the toolchain,
and make again, then we don't want to take object files out of the cache...
Of course, we already had this situation in earlier versions of ccache.

> So I was envisioning a sub-menu where you could
> select 1 of the 3 options to be explicitly set in the CCACHE var, or maybe
> 1 of 4 to have a dont-specify-anything option; or a text field, to type in
> whatever you want - in case you want to specify additional env vars, like
> the CCACHE_BASEDIR var, so that parallel buildroot trees can use the 
> same cached objects.

 I think that's a bit too complex.

 I see two generic solutions:

- Stick to mtime, but make sure the mtime of external toolchains isn't 
changed.  Since we actually create a toolchain wrapper, we'd need to do
a 'touch -r <real-toolchain>' on it.

- Use the command string option of CCACHE_COMPILERCHECK.  We can
calculate a static hash of the compiler/kernel-headers/libc/whatnot
configuration, and use 'echo <hash>' as the value of
CCACHE_COMPILERCHECK.  That's quite a bit more complex, but it
makes it possible to use the compiler cache also for an internal
toolchain after you've done a make clean.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] ccache problem with codesourcery toolchain

[D M]

I like Arnout's "touch -r" idea. ?Please find a patch attached to
this email. ?It did the trick for me.

Dan -




----- Original Message -----
From: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
To: D M <d_mo1234@yahoo.com>
Cc: "buildroot at busybox.net" <buildroot@busybox.net>
Sent: Sunday, February 5, 2012 2:45 AM
Subject: Re: [Buildroot] ccache problem with codesourcery toolchain

Hello Danomi,

Le Sat, 4 Feb 2012 17:05:01 -0800 (PST),
D M <d_mo1234@yahoo.com> a ?crit :

> I am working on a project based on the 2011.08 release of buildroot.
> ?We found that the ccache savings was not as good as with our older
> version of buildroot (which used an older version of ccache). ?For
> example, starting from an empty cache directory, our compile takes
> about half an hour. ?With the newer version of ccache, we only save
> about 3 minutes; with the older version, we saved about 13 minutes.
> ?We traced the problem to a feature introduced in ccache 3.0,
> the?CCACHE_COMPILERCHECK option. ?Quoting from ccache manual:?
> 
> ? ?"By default, ccache includes the modification time (?mtime?) and
> size of the compiler in the hash to ensure that results retrieved
> from the cache are accurate. The CACHE_COMPILERCHECK environment
> variable can be used to select another strategy." ?
> 
> The default is "mtime", but if we select "none", we get back all of
> our savings. ?Again quoting from the manual: ?
> 
> ? ? "none - Don?t hash anything. This may be good for situations
> where you can safely use the cached results even though the
> compiler?s mtime or size has changed (e.g. if the compiler is built
> as part of your build system and the compiler?s source has not
> changed, or if the compiler only has changes that don?t affect code
> generation)."
> 
> Since we are using buildroot's capability to automatically download
> and extract the CodeSourcery ARM toolchain, the "mtime" used in the
> hash is different every time we do a full recompile. ?So a value of
> "none" seems appropriate when use buildroot this way.
> 
> Have others ran into this? ?Since buildroot has this kind of
> CodeSourcery toolchain support, I was wondering if there should be an
> option in menuconfig to specify CCACHE_COMPILERCHECK, to, for
> example, prepend a "CCACHE_COMPILERCHECK=none" to the beginning of
> the CCACHE variable that is used to make TARGET_CC, TARGET_CXX,
> HOSTCC, HOSTCXX.

Good research! Obviously, using the default value of
CCACHE_COMPILERCHECK=mtime will completely defeat the compiler cache in
the context of Buildroot, because the compiler gets re-created pretty
much at every build, so definitely we should arrange for
CCACHE_COMPILERCHECK=none to be set in the ccache environment, or even
directly modify the ccache code to make it the default.

The thing I am curious about is where do the 3 minutes time reduction
come from even when CCACHE_COMPILERCHECK=mtime? Maybe it's the time
reduction gained on compiling host packages, since those always use the
same host compiler.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ext-toolchain-set-wrapper-date-for-ccache-mtime.patch
Type: application/octet-stream
Size: 1253 bytes
Desc: not available
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120208/42e9d2cb/attachment-0001.obj>

[Buildroot] ccache problem with codesourcery toolchain

[Arnout Vandecappelle]

On Thursday 09 February 2012 03:39:16 D M wrote:
> I like Arnout's "touch -r" idea.  Please find a patch attached to
> this email.  It did the trick for me.

 Thank you for this contribution!  It will help to speed up builds.

 We prefer to receive patches in the format generated by git-send-email,
because that is easy to add to our git repository.  For convenience, I
have applied your patch to my own git repository and I'll resend it as
a reply to this mail.

 We will also need a Signed-off-by tag on your patch, i.e. a line saying

Signed-off-by: Your Name <your@email.address>

This is a short way for you to assert that you are entitled to contribute
the patch under buildroot's GPL license.  See 
http://kerneltrap.org/files/Jeremy/DCO.txt for more details.  We also
require contributers to use their real name.

 Finally, I've added a small fix to your patch: the compiler wrapper
was recompiled all the time because the timestamp was changed, so I
removed it as a target.  (It's a small fix conceptually, but it modifies
many lines :-)


 Regards,
 Arnout


-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120210/71560ea5/attachment.html>

[Buildroot] [PATCH] ext-toolchain: set mtime of toolchain wrapper for ccache

[Arnout Vandecappelle]

From: D M <d_mo1234@yahoo.com>

Also removed the ext-toolchain-wrapper as a target.

When using recent ccache with an external toolchain, a rebuild after
'make clean' will not make use of the cache.  ccache by default looks
at the mtime of the compiler executable and includes that in the
hash of the compiled file.  Since we build a wrapper around the
external toolchain, the mtime will have changed after the recompile
so everything will be rebuild.

As a solution, we give the wrapper executable the same mtime as the
external compiler executable, which will stay the same when rebuilding.
Unfortunately, this would mean that the ext-toolchain-wrapper is
rebuilt all the time (because its mtime is before that of the
ext-toolchain-installed stamp file).  Therefore, ext-toolchain-wrapper
is removed as a target.  Instead, the ext-toolchain-installed target
includes the building of the wrapper.

Alternative solutions are:

- Setting the CCACHE_COMPILERCHECK environment variable to "none", so
the compiler mtime is not included in the hash.  But then, if we change
the uClibc configuration, do a make clean and rebuild, most files will
still use the old object file.

- Setting the CCACHE_COMPILERCHECK environment variable to "hash", which
will calculate a hash of the compiler executable.  However, this hash is
recalculated for every file, so that's expensive.

- Setting the CCACHE_COMPILERCHECK environment variable to "echo $(HASH)",
where $(HASH) is a value that we compute from the toolchain configuration.

Although the last option is the best one, it costs a lot more effort to
implement.  This patch is a much simpler solution.

This patch only works for an external toolchain.  When using an internal
or crosstool-ng toolchain, the cache will still not be used.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 toolchain/toolchain-external/ext-tool.mk |   26 ++++++++++++++------------
 1 files changed, 14 insertions(+), 12 deletions(-)

diff --git a/toolchain/toolchain-external/ext-tool.mk b/toolchain/toolchain-external/ext-tool.mk
index 2ac4973..e914ed0 100644
--- a/toolchain/toolchain-external/ext-tool.mk
+++ b/toolchain/toolchain-external/ext-tool.mk
@@ -315,29 +315,31 @@ $(STAMP_DIR)/ext-toolchain-installed: $(TOOLCHAIN_EXTERNAL_DEPENDENCIES)
 	$(call copy_toolchain_sysroot,$${SYSROOT_DIR},$${ARCH_SYSROOT_DIR},$${ARCH_SUBDIR}) ; \
 	if [ -L $${ARCH_SYSROOT_DIR}/lib64 ] ; then \
 		$(call create_lib64_symlinks) ; \
-	fi ; \
-	touch $@
-
-# Build toolchain wrapper for preprocessor, C and C++ compiler, and setup
-# symlinks for everything else
-$(HOST_DIR)/usr/bin/ext-toolchain-wrapper: $(STAMP_DIR)/ext-toolchain-installed
-	mkdir -p $(HOST_DIR)/usr/bin; cd $(HOST_DIR)/usr/bin; \
+	fi
+	# Build toolchain wrapper for preprocessor, C and C++ compiler, and
+	# setup symlinks for everything else.  Give the toolchain wrapper the
+	# same timestamp as one of the executables it wraps, so ccache still
+	# works.
+	$(Q)mkdir -p $(HOST_DIR)/usr/bin
+	$(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_EXTERNAL_WRAPPER_ARGS) -s \
+		toolchain/toolchain-external/ext-toolchain-wrapper.c -o $(HOST_DIR)/usr/bin/ext-toolchain-wrapper
+	$(Q)cd $(HOST_DIR)/usr/bin; \
 	for i in $(TOOLCHAIN_EXTERNAL_CROSS)*; do \
 		base=$${i##*/}; \
 		case "$$base" in \
 		*cc|*cc-*|*++|*++-*|*cpp) \
-			ln -sf $(@F) $$base; \
+			ln -sf ext-toolchain-wrapper $$base; \
+			touch -r $$i ext-toolchain-wrapper; \
 			;; \
 		*) \
 			ln -sf $$i .; \
 			;; \
 		esac; \
-	done ;
-	$(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_EXTERNAL_WRAPPER_ARGS) -s \
-		toolchain/toolchain-external/ext-toolchain-wrapper.c -o $@
+	done
+	$(Q)touch $@
 
 # 'uclibc' is the target to provide toolchain / staging dir
-uclibc: dependencies $(HOST_DIR)/usr/bin/ext-toolchain-wrapper
+uclibc: dependencies $(STAMP_DIR)/ext-toolchain-installed
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD),y)
 # download ext toolchain if so configured
-- 
1.7.8.3