[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Karoly Kasza]

This patch adds a wrapper script to call ccache, which will differentiate
if the HOST or either internal or external TARGET compiler is called.

Ccache needs to know what compiler we use, but can be simply fooled to fetch
invalid objects. To circumvent this, it uses different algorythms to
distiguish the compilers. This needs to be tuned for Buildroot's usage,
as the internal toolchains' characteristics change with every recompilation,
while ccache can not directly access the external toolchains (symlinks).

This patch does the following:
- Reenable using mtime (hash of binary size and modification date) as the
  default compiler_check algorythm. This will be used for the host compiler.
- Renames the ccache binary, so the wrapper script can be installed on it's
  place.
- Creates a wrapper script with hardcoded values for performance, which will
  disable the mtime compiler check and add an extra file to hash if it is called
  for a target compiler (either internal or external). It is basically all the
  same if we are using internal or external toolchains, ccache only sees a path.
- Creates a checksum file for the above method from the extract of the current
  Buildroot config, which should not change when configuring packages or
  recompiling the whole project. However it will change if the compiler options
  change, thus safely pointing to cached objects.

Signed-off-by: Karoly Kasza <kaszak@gmail.com>
---
 Config.in                |    7 -------
 package/ccache/ccache.mk |   25 ++++++++++++++++++++-----
 2 files changed, 20 insertions(+), 12 deletions(-)

diff --git a/Config.in b/Config.in
index 2b39d6a..6e7f722 100644
--- a/Config.in
+++ b/Config.in
@@ -257,13 +257,6 @@ config BR2_CCACHE
 	  up future builds. By default, the cache is stored in
 	  $HOME/.buildroot-ccache.
 
-	  Note that Buildroot does not try to invalidate the cache
-	  contents when the compiler changes in an incompatible
-	  way. Therefore, if you make a change to the compiler version
-	  and/or configuration, you are responsible for purging the
-	  ccache cache by removing the $HOME/.buildroot-ccache
-	  directory.
-
 if BR2_CCACHE
 
 config BR2_CCACHE_DIR
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 52b5c67..ee9085d 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
 #    is already used by autotargets for the ccache package.
 #    BR_CACHE_DIR is exported by Makefile based on config option
 #    BR2_CCACHE_DIR.
-#  - ccache shouldn't use the compiler binary mtime to detect a change in
-#    the compiler, because in the context of Buildroot, that completely
-#    defeats the purpose of ccache. Of course, that leaves the user
-#    responsible for purging its cache when the compiler changes.
 #  - Change hard-coded last-ditch default to match path in .config, to avoid
 #    the need to specify BR_CACHE_DIR when invoking ccache directly.
+#  - Rename ccache binary to ccache.bin, so later the wrapper can be safely
+#    named ccache, as referred everywhere
 define HOST_CCACHE_PATCH_CONFIGURATION
 	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c
-	sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c
 	sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c
+	sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h
 endef
 
 HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
@@ -58,6 +56,23 @@ endef
 HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
 endif
 
+# Install a wrapper script, which, when using the TARGET_ toolchain, will make
+# ccache use the hash of the toolchain options instead of the compiler's mtime
+define HOST_CCACHE_INSTALL_WRAPPER
+	mv $(CCACHE) $(CCACHE).bin
+	echo "#!/bin/sh" > $(CCACHE)
+	echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE)
+	echo "   [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE)
+	echo "	export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE)
+	echo "fi" >> $(CCACHE)
+	echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE)
+	grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \
+	grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash
+	chmod 755 $(CCACHE)
+endef
+
+HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_INSTALL_WRAPPER
+
 $(eval $(host-autotools-package))
 
 ifeq ($(BR2_CCACHE),y)
-- 
1.7.10.4

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Danomi Manchego]

Karoly,

On Wed, Apr 29, 2015 at 8:05 AM, Karoly Kasza <kaszak@gmail.com> wrote:
> This patch adds a wrapper script to call ccache, which will differentiate
> if the HOST or either internal or external TARGET compiler is called.
>
> Ccache needs to know what compiler we use, but can be simply fooled to fetch
> invalid objects. To circumvent this, it uses different algorythms to
> distiguish the compilers. This needs to be tuned for Buildroot's usage,
> as the internal toolchains' characteristics change with every recompilation,
> while ccache can not directly access the external toolchains (symlinks).
>
> This patch does the following:
> - Reenable using mtime (hash of binary size and modification date) as the
>   default compiler_check algorythm. This will be used for the host compiler.
> - Renames the ccache binary, so the wrapper script can be installed on it's
>   place.
> - Creates a wrapper script with hardcoded values for performance, which will
>   disable the mtime compiler check and add an extra file to hash if it is called
>   for a target compiler (either internal or external). It is basically all the
>   same if we are using internal or external toolchains, ccache only sees a path.
> - Creates a checksum file for the above method from the extract of the current
>   Buildroot config, which should not change when configuring packages or
>   recompiling the whole project. However it will change if the compiler options
>   change, thus safely pointing to cached objects.
>
> Signed-off-by: Karoly Kasza <kaszak@gmail.com>
> ---
>  Config.in                |    7 -------
>  package/ccache/ccache.mk |   25 ++++++++++++++++++++-----
>  2 files changed, 20 insertions(+), 12 deletions(-)
>
> diff --git a/Config.in b/Config.in
> index 2b39d6a..6e7f722 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -257,13 +257,6 @@ config BR2_CCACHE
>           up future builds. By default, the cache is stored in
>           $HOME/.buildroot-ccache.
>
> -         Note that Buildroot does not try to invalidate the cache
> -         contents when the compiler changes in an incompatible
> -         way. Therefore, if you make a change to the compiler version
> -         and/or configuration, you are responsible for purging the
> -         ccache cache by removing the $HOME/.buildroot-ccache
> -         directory.
> -
>  if BR2_CCACHE
>
>  config BR2_CCACHE_DIR
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 52b5c67..ee9085d 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
>  #    is already used by autotargets for the ccache package.
>  #    BR_CACHE_DIR is exported by Makefile based on config option
>  #    BR2_CCACHE_DIR.
> -#  - ccache shouldn't use the compiler binary mtime to detect a change in
> -#    the compiler, because in the context of Buildroot, that completely
> -#    defeats the purpose of ccache. Of course, that leaves the user
> -#    responsible for purging its cache when the compiler changes.
>  #  - Change hard-coded last-ditch default to match path in .config, to avoid
>  #    the need to specify BR_CACHE_DIR when invoking ccache directly.
> +#  - Rename ccache binary to ccache.bin, so later the wrapper can be safely
> +#    named ccache, as referred everywhere
>  define HOST_CCACHE_PATCH_CONFIGURATION
>         sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c
> -       sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c
>         sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c
> +       sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h
>  endef
>
>  HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
> @@ -58,6 +56,23 @@ endef
>  HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
>  endif
>
> +# Install a wrapper script, which, when using the TARGET_ toolchain, will make
> +# ccache use the hash of the toolchain options instead of the compiler's mtime
> +define HOST_CCACHE_INSTALL_WRAPPER
> +       mv $(CCACHE) $(CCACHE).bin
> +       echo "#!/bin/sh" > $(CCACHE)
> +       echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE)
> +       echo "   [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE)
> +       echo "  export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE)
> +       echo "fi" >> $(CCACHE)
> +       echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE)
> +       grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \
> +       grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash

Does the list used for grep need these as well, as they influence the
ext-toolchain-wrapper.c?

* BR2_TARGET_OPTIMIZATION
* BR2_SOFT_FLOAT

Danomi -


> +       chmod 755 $(CCACHE)
> +endef
> +
> +HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_INSTALL_WRAPPER
> +
>  $(eval $(host-autotools-package))
>
>  ifeq ($(BR2_CCACHE),y)
> --
> 1.7.10.4
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Károly Kasza]

Hi Danomi,

> +       grep
> "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC"
> $(BR2_CONFIG) | \
> > +       grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash
>
> Does the list used for grep need these as well, as they influence the
> ext-toolchain-wrapper.c?
>
> * BR2_TARGET_OPTIMIZATION
> * BR2_SOFT_FLOAT
>
>
This patch don't really calculates with ext-toolchain-wrapper.c, but it's a
good question if these config options should be factored in, I am not sure.
If these are passed at compilation time to the target compiler (by the
wrapper binary), then theoretically ccache gets them and calculates them
in, as ccache is not influenced by the wrapper (mtime turned off) - so we
don't need them in the hash file.
If the target compiler binary changes based on their values, then they
should be calculated in - I think this doesn't happen, as the external
binaries won't be changed.
Opinions?

(It's not a be a big deal to add them also into the grep line, but if we
restrict the hash too much, then it won't match the cache hits even when it
should. If we don't restrict it enough, it may generate false cache hits
and through that corrupted binaries.)

BR
Karoly
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20150430/47bff646/attachment.html>

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Arnout Vandecappelle]

On 04/30/15 20:17, K?roly Kasza wrote:
> Hi Danomi,
> 
>     > +       grep
>     "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC"
>     $(BR2_CONFIG) | \
>     > +       grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash
> 
>     Does the list used for grep need these as well, as they influence the
>     ext-toolchain-wrapper.c?
> 
>     * BR2_TARGET_OPTIMIZATION
>     * BR2_SOFT_FLOAT
> 
> 
> This patch don't really calculates with ext-toolchain-wrapper.c, but it's a good
> question if these config options should be factored in, I am not sure. If these
> are passed at compilation time to the target compiler (by the wrapper binary),
> then theoretically ccache gets them and calculates them in, as ccache is not
> influenced by the wrapper (mtime turned off) - so we don't need them in the hash
> file.

 The wrapper adds these values, and ccache calls the wrapper. So if these values
change, ccache won't notice it (at least if the wrapper is used).

 The list of affected options is actually a lot longer I'm afraid. I think this
is the full list of options affecting the compiler. Some of these are Config.in
options, some of these are options calculated in the makefiles, but all of them
are available to make.

Options that affect the build of the compiler (internal toolchain):
BR2_GCC_TARGET_ARCH
BR2_GCC_TARGET_ABI
BR2_GCC_TARGET_CPU
BR2_GCC_TARGET_CPU_REVISION
BR2_GCC_TARGET_FPU
BR2_GCC_TARGET_FLOAT_ABI
BR2_GCC_TARGET_MODE
BR2_EXTRA_GCC_CONFIG_OPTIONS
BR2_SOFT_FLOAT
BR2_PTHREADS_*
BR2_TOOLCHAIN_BUILDROOT_LIBC
BR2_TOOLCHAIN_BUILDROOT_WCHAR
BR2_GCC_ENABLE_TLS
BR2_GCC_ENABLE_LTO
BR2_GCC_ENABLE_GRAPHITE
BR2_GCC_ENABLE_OPENMP (probably not needed, only affects headers/libraries)
BR2_GCC_ENABLE_LIBMUDFLAP (probably not needed, only affects headers/libraries)
GCC_VERSION
The contents of package/gcc/$(GCC_VERSION)/*
The contents of $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,\
			$(BR2_XTENSA_CORE_NAME)).tar
The contents of package/gcc/gcc.mk (although that may be too conservative)
The contents of package/gcc/gcc-final.mk (although that may be too conservative)

 Note that the libc options are not relevant, since they will affect only
libraries and header files. ccache doesn't deal with libraries, and ccache will
still fingerprint header files.

 A simpler way of getting the correct fingerprint would be to look at
HOST_GCC_FINAL_CONF_OPTS, after removing all occurences of $(HOST_DIR) and
$(TARGET_DIR) from it. It's slightly too conservative since some unimportant
options (e.g. BR2_INSTALL_LIBSTDCPP) still appear in there, but on the other
hand it's more future safe since any option that affects the compiler will
appear in it.


Options that affect external toolchains:
The BR2_GCC_TARGET_* options mentioned above
BR2_TARGET_OPTIMIZATION
BR2_SOFT_FLOAT
The contents of $(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE)
The contents of $(BR2_TOOLCHAIN_EXTERNAL_PATH) (if preinstalled)

 As it happens, all of the flags that affect the toolchain wrapper are already
in TOOLCHAIN_EXTERNAL_CFLAGS.


 So I think the easiest solution is to calculate the ccache hash in two places:

- In HOST_GCC_FINAL_POST_PATCH_HOOKS, calculate a hash of the contents of $(@D)
and $(subst $(HOST_DIR),@,$(HOST_GCC_FINAL_CONF_OPTS))

- In TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER, calculate a hash of
$(DL_DIR)/$(TOOLCHAIN_EXTERNAL_SOURCE) or of $(BR2_TOOLCHAIN_EXTERNAL_PATH) and
$(TOOLCHAIN_EXTERNAL_CFLAGS)


 Obviously, this needs a bit of experimentation to see if it really works.

 Regards,
 Arnout


> If the target compiler binary changes based on their values, then they should be
> calculated in - I think this doesn't happen, as the external binaries won't be
> changed.
> Opinions?
> 
> (It's not a be a big deal to add them also into the grep line, but if we
> restrict the hash too much, then it won't match the cache hits even when it
> should. If we don't restrict it enough, it may generate false cache hits and
> through that corrupted binaries.)
> 
> BR
> Karoly
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Arnout Vandecappelle]

On 04/29/15 14:05, Karoly Kasza wrote:
> This patch adds a wrapper script to call ccache, which will differentiate
> if the HOST or either internal or external TARGET compiler is called.
> 
> Ccache needs to know what compiler we use, but can be simply fooled to fetch
> invalid objects. To circumvent this, it uses different algorythms to
> distiguish the compilers. This needs to be tuned for Buildroot's usage,
> as the internal toolchains' characteristics change with every recompilation,
> while ccache can not directly access the external toolchains (symlinks).
> 
> This patch does the following:
> - Reenable using mtime (hash of binary size and modification date) as the
>   default compiler_check algorythm. This will be used for the host compiler.
> - Renames the ccache binary, so the wrapper script can be installed on it's
>   place.
> - Creates a wrapper script with hardcoded values for performance, which will
>   disable the mtime compiler check and add an extra file to hash if it is called
>   for a target compiler (either internal or external). It is basically all the
>   same if we are using internal or external toolchains, ccache only sees a path.
> - Creates a checksum file for the above method from the extract of the current
>   Buildroot config, which should not change when configuring packages or
>   recompiling the whole project. However it will change if the compiler options
>   change, thus safely pointing to cached objects.
> 
> Signed-off-by: Karoly Kasza <kaszak@gmail.com>
> ---
>  Config.in                |    7 -------
>  package/ccache/ccache.mk |   25 ++++++++++++++++++++-----
>  2 files changed, 20 insertions(+), 12 deletions(-)
> 
> diff --git a/Config.in b/Config.in
> index 2b39d6a..6e7f722 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -257,13 +257,6 @@ config BR2_CCACHE
>  	  up future builds. By default, the cache is stored in
>  	  $HOME/.buildroot-ccache.
>  
> -	  Note that Buildroot does not try to invalidate the cache
> -	  contents when the compiler changes in an incompatible
> -	  way. Therefore, if you make a change to the compiler version
> -	  and/or configuration, you are responsible for purging the
> -	  ccache cache by removing the $HOME/.buildroot-ccache
> -	  directory.
> -
>  if BR2_CCACHE
>  
>  config BR2_CCACHE_DIR
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 52b5c67..ee9085d 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
>  #    is already used by autotargets for the ccache package.
>  #    BR_CACHE_DIR is exported by Makefile based on config option
>  #    BR2_CCACHE_DIR.
> -#  - ccache shouldn't use the compiler binary mtime to detect a change in
> -#    the compiler, because in the context of Buildroot, that completely
> -#    defeats the purpose of ccache. Of course, that leaves the user
> -#    responsible for purging its cache when the compiler changes.
>  #  - Change hard-coded last-ditch default to match path in .config, to avoid
>  #    the need to specify BR_CACHE_DIR when invoking ccache directly.
> +#  - Rename ccache binary to ccache.bin, so later the wrapper can be safely
> +#    named ccache, as referred everywhere
>  define HOST_CCACHE_PATCH_CONFIGURATION
>  	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c
> -	sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c
>  	sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c
> +	sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h
>  endef
>  
>  HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
> @@ -58,6 +56,23 @@ endef
>  HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
>  endif
>  
> +# Install a wrapper script, which, when using the TARGET_ toolchain, will make
> +# ccache use the hash of the toolchain options instead of the compiler's mtime
> +define HOST_CCACHE_INSTALL_WRAPPER
> +	mv $(CCACHE) $(CCACHE).bin
> +	echo "#!/bin/sh" > $(CCACHE)
> +	echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE)
> +	echo "   [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE)
> +	echo "	export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE)
> +	echo "fi" >> $(CCACHE)
> +	echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE)

 I don't really like this additional wrapper script. Can't we just define two
variables, HOST_CCACHE and TARGET_CCACHE:

HOST_CCACHE = $(HOST_DIR)/usr/bin/ccache
TARGET_CCACHE = \
	CCACHE_COMPILERCHECK=none \
	CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash \
	$(HOST_DIR)/usr/bin/ccache

TARGET_CC := $(TARGET_CCACHE) $(TARGET_CC)

(and subsitute the other references to $(CCACHE) as appropriate)

 The handling of cmake and qmake will require a bit of thinking however.


 If you do keep the wrapper script, then I'd prefer if you create it as
package/ccache/ccache_wrapper.in and fill in the relevant variables with sed.


> +	grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \
> +	grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash

 Since CCACHE_EXTRAFILES is anyway taken together with the input file into one
big hash, I wouldn't bother with hashing it here, but just put the actual
options to be hashed in that file. That gives a good reference when you want to
debug ccache, and the speed penalty is negligible (the overhead of opening the
file is larger than reading and hashing it).


 Regards,
 Arnout


> +	chmod 755 $(CCACHE)
> +endef
> +
> +HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_INSTALL_WRAPPER
> +
>  $(eval $(host-autotools-package))
>  
>  ifeq ($(BR2_CCACHE),y)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Thomas De Schampheleire]

Hi Karoly,

On Fri, May 1, 2015 at 1:05 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 04/29/15 14:05, Karoly Kasza wrote:
>> This patch adds a wrapper script to call ccache, which will differentiate
>> if the HOST or either internal or external TARGET compiler is called.
>>
>> Ccache needs to know what compiler we use, but can be simply fooled to fetch
>> invalid objects. To circumvent this, it uses different algorythms to
>> distiguish the compilers. This needs to be tuned for Buildroot's usage,
>> as the internal toolchains' characteristics change with every recompilation,
>> while ccache can not directly access the external toolchains (symlinks).
>>
>> This patch does the following:
>> - Reenable using mtime (hash of binary size and modification date) as the
>>   default compiler_check algorythm. This will be used for the host compiler.
>> - Renames the ccache binary, so the wrapper script can be installed on it's
>>   place.
>> - Creates a wrapper script with hardcoded values for performance, which will
>>   disable the mtime compiler check and add an extra file to hash if it is called
>>   for a target compiler (either internal or external). It is basically all the
>>   same if we are using internal or external toolchains, ccache only sees a path.
>> - Creates a checksum file for the above method from the extract of the current
>>   Buildroot config, which should not change when configuring packages or
>>   recompiling the whole project. However it will change if the compiler options
>>   change, thus safely pointing to cached objects.
>>
>> Signed-off-by: Karoly Kasza <kaszak@gmail.com>
>> ---
>>  Config.in                |    7 -------
>>  package/ccache/ccache.mk |   25 ++++++++++++++++++++-----
>>  2 files changed, 20 insertions(+), 12 deletions(-)
>>
>> diff --git a/Config.in b/Config.in
>> index 2b39d6a..6e7f722 100644
>> --- a/Config.in
>> +++ b/Config.in
>> @@ -257,13 +257,6 @@ config BR2_CCACHE
>>         up future builds. By default, the cache is stored in
>>         $HOME/.buildroot-ccache.
>>
>> -       Note that Buildroot does not try to invalidate the cache
>> -       contents when the compiler changes in an incompatible
>> -       way. Therefore, if you make a change to the compiler version
>> -       and/or configuration, you are responsible for purging the
>> -       ccache cache by removing the $HOME/.buildroot-ccache
>> -       directory.
>> -
>>  if BR2_CCACHE
>>
>>  config BR2_CCACHE_DIR
>> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
>> index 52b5c67..ee9085d 100644
>> --- a/package/ccache/ccache.mk
>> +++ b/package/ccache/ccache.mk
>> @@ -26,16 +26,14 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
>>  #    is already used by autotargets for the ccache package.
>>  #    BR_CACHE_DIR is exported by Makefile based on config option
>>  #    BR2_CCACHE_DIR.
>> -#  - ccache shouldn't use the compiler binary mtime to detect a change in
>> -#    the compiler, because in the context of Buildroot, that completely
>> -#    defeats the purpose of ccache. Of course, that leaves the user
>> -#    responsible for purging its cache when the compiler changes.
>>  #  - Change hard-coded last-ditch default to match path in .config, to avoid
>>  #    the need to specify BR_CACHE_DIR when invoking ccache directly.
>> +#  - Rename ccache binary to ccache.bin, so later the wrapper can be safely
>> +#    named ccache, as referred everywhere
>>  define HOST_CCACHE_PATCH_CONFIGURATION
>>       sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c
>> -     sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c
>>       sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c
>> +     sed -i 's,#define MYNAME "ccache",#define MYNAME "ccache.bin",' $(@D)/ccache.h
>>  endef
>>
>>  HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
>> @@ -58,6 +56,23 @@ endef
>>  HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
>>  endif
>>
>> +# Install a wrapper script, which, when using the TARGET_ toolchain, will make
>> +# ccache use the hash of the toolchain options instead of the compiler's mtime
>> +define HOST_CCACHE_INSTALL_WRAPPER
>> +     mv $(CCACHE) $(CCACHE).bin
>> +     echo "#!/bin/sh" > $(CCACHE)
>> +     echo "if [ \"$$"'1'"\" = \"$(TARGET_CC_NOCCACHE)\" ] || \\" >> $(CCACHE)
>> +     echo "   [ \"$$"'1'"\" = \"$(TARGET_CXX_NOCCACHE)\" ]; then" >> $(CCACHE)
>> +     echo "  export CCACHE_COMPILERCHECK=none CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash" >> $(CCACHE)
>> +     echo "fi" >> $(CCACHE)
>> +     echo "$(CCACHE).bin \"$$"'@'"\"" >> $(CCACHE)
>
>  I don't really like this additional wrapper script. Can't we just define two
> variables, HOST_CCACHE and TARGET_CCACHE:
>
> HOST_CCACHE = $(HOST_DIR)/usr/bin/ccache
> TARGET_CCACHE = \
>         CCACHE_COMPILERCHECK=none \
>         CCACHE_EXTRAFILES=$(HOST_DIR)/ccache_toolchain.config.hash \
>         $(HOST_DIR)/usr/bin/ccache
>
> TARGET_CC := $(TARGET_CCACHE) $(TARGET_CC)
>
> (and subsitute the other references to $(CCACHE) as appropriate)
>
>  The handling of cmake and qmake will require a bit of thinking however.
>
>
>  If you do keep the wrapper script, then I'd prefer if you create it as
> package/ccache/ccache_wrapper.in and fill in the relevant variables with sed.
>
>
>> +     grep "BR2_ARCH\|BR2_TOOLCHAIN\|BR2_GCC\|BR2_BINUTILS\|BR2_UCLIBC\|BR2_GLIBC\|BR2_EGLIBC" $(BR2_CONFIG) | \
>> +     grep -v "#" | md5sum > $(HOST_DIR)/ccache_toolchain.config.hash
>
>  Since CCACHE_EXTRAFILES is anyway taken together with the input file into one
> big hash, I wouldn't bother with hashing it here, but just put the actual
> options to be hashed in that file. That gives a good reference when you want to
> debug ccache, and the speed penalty is negligible (the overhead of opening the
> file is larger than reading and hashing it).
>
>


Did you have a chance to look any further into this topic ?

Thanks,
Thomas

[Buildroot] [PATCH] package/ccache: add universal wrapper script

[Károly Kasza]

Hi Thomas, all

I had (have) an IRL interrupt, but I still plan to take this topic further,
based on Arnout's comments.
Also, ccache released a new minor version, I'll send a bump in a few days,
as soon as I have time to test it properly.
The new CCACHE_COMPILERCHECK=string:<value> option could provide useful
after the bump.

Kind regards,
Karoly

>
> Did you have a chance to look any further into this topic ?
>
> Thanks,
> Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20150529/37ebcd6d/attachment.html>