[Buildroot] [PATCH 0/3] ccache compilercheck and automated option setup

[Buildroot] [PATCH 0/3] ccache compilercheck and automated option setup

[Danomi Manchego]

Thomas De Schampheleire recently inquired about a mailing list thread started back in April about ccache.  So I thought I would submit my changes as an RFC.  I can imagine that these changes might not be accepted on grounds of adding too much complication, or being too specific to my use cases.  Which would be fine - I'm just sharing the fruits of some of the helpful discussions that started on the mailing list.


Patch 1, "ccache: change compilercheck to use compiler and toolchain info":

The first patch addresses a case where ccache can be fooled in certain circumstances, resulting in using objects compiled with, say, the wrong toolchain.  (This was discovered when compiling kmod from the same project (clean, build with toolchain 1, clean, build with toolchain 2).  This original thread on the mailing list was:

http://lists.busybox.net/pipermail/buildroot/2013-April/070819.html

The patch resolves this by incorporating feedback from the compiler itself, using the compiler's "-v" output.  In fact, this is mentioned at http://ccache.samba.org/manual.html in the CCACHE_COMPILERCHECK section, and I think by inference in the "Using ccache with other compiler wrappers" section.

However, as Arnout pointed out, the -v is not quite enough to capture other changes incorporated into the toolchain wrapper.  To capture these, the patch greps out the settings from the "Target Architecture" and "Toolchain" sections of .config, and incorporates the md5sum of those settings into the ccache hash.  (The md5sum is used to not add too much more to ccache's extra checking.)

The results are these:

- Original ccache compilercheck expression (not good for buildroot):

	compilercheck = getenv("CCACHE_COMPILERCHECK");

- Current compilercheck expression in buildroot:

	compilercheck = "none";

- Example of compilercheck expression with this patch:

	compilercheck = "%compiler% -v; echo 'fd787552bdf20802b759cae764d03a48  -'";

With this patch, changes in both completely external toolchains and in buildroot's toolchain setup are factored into ccache's considerations.

The downside is that if the compiler's -v output has any absolute paths in it, and if the compiler is located in the output directory, then the resulting hashes would be unique per output directory - which would effect cases where groups might be trying to share cache directories.


Patch 2, "ccache: change default cache directory path to match config setting":

The goal of this patch is to let us call the ccache executable outside of buildroot's Makefile without defining a "BUILDROOT_CACHE_DIR" environment variable, for either setup (which overlaps the recent ccache.mk mod by Tzu-Jung Lee), or for cached compiler invocation outside of buildroot.

Commit 433290761fceb476b095548eec10adf72405e050 changed the hard-coded
ccache directory location to use BUILDROOT_CACHE_DIR, which is exported
by Makefile based on the BR2_CCACHE_DIR config option.  This allowed the
cache location to be changed on-the-fly by setting an environment
variable, but left the default location of ccache's normal default at
"$HOME/.ccache".  Since this location does not match the default for
BR2_CCACHE_DIR, it is basically almost never correct, so direct invocation
of ccache outside of the buildroot Makefile becomes cumbersome.

This patch changes the last-ditch cache location from "$HOME/.ccache" to
the BR2_CCACHE_DIR value at the time of host-ccache compilation.

- Original ccache default cache directory location:

	cache_dir = getenv("CCACHE_DIR");
	if (cache_dir) {
		cache_dir = x_strdup(cache_dir);
	} else {
		const char *home_directory = get_home_directory();
		if (home_directory) {
			cache_dir = format("%s/.ccache", home_directory);
		}
	}

- Current default cache directory location in buildroot:

	cache_dir = getenv("BUILDROOT_CACHE_DIR");
	if (cache_dir) {
		cache_dir = x_strdup(cache_dir);
	} else {
		const char *home_directory = get_home_directory();
		if (home_directory) {
			cache_dir = format("%s/.ccache", home_directory);
		}
	}

- Example of default cache directory location with this patch:

	cache_dir = getenv("BUILDROOT_CACHE_DIR");
	if (cache_dir) {
		cache_dir = x_strdup(cache_dir);
	} else {
		const char *home_directory = get_home_directory();
		if (home_directory) {
			cache_dir = format("/home/bob/.buildroot-ccache", home_directory);
		}
	}

Please note that the ability to override the location by BUILDROOT_CACHE_DIR is left intact.


Patch 3, "ccache: provide capability to do initial ccache setup":

We found that once a project gets large enough in terms of packages (for example, a multimedia project with gstreamer, lots of plugins, etc.), it begins to hit or surpass ccache's default maximum cache size - so ccache ends up pushing out old content just to get through a re-compile.  So we came up with this patch to provide a way to set cache limits on a defconfig basis, so that nice settings are pushed out to everyone on the project.  (This also has the virtue of re-establishing such a setup even if you manually delete the cache directory for whatever reason.)  If no settings are specified, then the ccache settings are left untouched.


Danomi Manchego (3):
  ccache: change compilercheck to use compiler and toolchain info
  ccache: change default cache directory path to match config setting
  ccache: provide capability to do initial ccache setup

 Config.in                |   14 +++++++++++++-
 package/ccache/ccache.mk |   40 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 52 insertions(+), 2 deletions(-)

-- 
1.7.9.5

[Buildroot] [PATCH 1/3] ccache: change compilercheck to use compiler and toolchain info

[Danomi Manchego]

When CCACHE_COMPILERCHECK is set to "none", then ccache can be fooled in certain
circumstances, resulting in using objects compiled with, say, the wrong toolchain.
(This was discovered when compiling kmod from the same project, but with different
external toolchains.)

So let's try to make ccache use as safe as possible:

- Use "%compiler% -v" in CCACHE_COMPILERCHECK to capture changes purely in an
  externally provided toolchain.  See CCACHE_COMPILERCHECK and wrapper sections
  at http://ccache.samba.org/manual.html for more info.

- Additionally, use the hash of part of the .config to describe toolchain and
  C-library configuration that cannot be captured by the -v output:

  + Use first section of .config, "Target Architecture" - "1,/^# Commands/", to
    capture arch/cpu settings built into the external toolchain wrapper (for when
    external toolchain it used) and the toolchain/c-library compilation settings
    (for when toolchain is built).

  + Use "Toolchain" section of .config - "/^\# Toolchain/,/^\# System config/",
    for additional toolchain settings.

  + Filter out blanks, comments, and dont-care stuff, then sort, to try to make
    immune to minor kconfig organizational changes.

Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>

---

Mailing list thread: http://lists.busybox.net/pipermail/buildroot/2013-April/070819.html
---
 package/ccache/ccache.mk |   25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 82a53f3..9ad5129 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -26,13 +26,36 @@ HOST_CCACHE_CONF_OPT += ccache_cv_zlib_1_2_3=no
 #    is already used by autotargets for the ccache package.
 #    BUILDROOT_CACHE_DIR is exported by Makefile based on config option
 #    BR2_CCACHE_DIR.
+#
 #  - ccache shouldn't use the compiler binary mtime to detect a change in
 #    the compiler, because in the context of Buildroot, that completely
 #    defeats the purpose of ccache. Of course, that leaves the user
 #    responsible for purging its cache when the compiler changes.
+#
+#    But let's try to make ccache use as safe as possible.  Let's use
+#    "%compiler% -v" in CCACHE_COMPILERCHECK to capture changes in external
+#    toolchains.  See CCACHE_COMPILERCHECK and wrapper sections at
+#    http://ccache.samba.org/manual.html for more info.
+#
+#    Additionally, use the hash of part of the .config to describe toolchain and
+#    C-library configuration that cannot be captured by the compiler's -v output:
+#
+#    + Use first section of .config, "Target Architecture" - "1,/^# Commands/", to
+#      capture arch/cpu settings built into the external toolchain wrapper (for when
+#      external toolchain it used) and the toolchain/c-library compilation settings
+#      (for when toolchain is built).
+#
+#    + Use "Toolchain" section of .config - "/^\# Toolchain/,/^\# System config/",
+#      for additional toolchain settings.
+#
+#    + Filter out blanks, comments, and dont-care stuff, then sort, to try to make
+#      immune to minor kconfig organizational changes.
 define HOST_CCACHE_PATCH_CONFIGURATION
 	sed -i 's,getenv("CCACHE_DIR"),getenv("BUILDROOT_CACHE_DIR"),' $(@D)/ccache.c
-	sed -i 's,getenv("CCACHE_COMPILERCHECK"),"none",' $(@D)/ccache.c
+	sed -n '1,/^# Commands/p; /^\# Toolchain/,/^\# System config/p' $(BUILDROOT_CONFIG) | \
+		grep -v -e '^$$' -e '^\#' -e GDB -e ECLIPSE | \
+		sort > $(STAMP_DIR)/ccache-toolchain-config
+	sed -i "s,getenv(\"CCACHE_COMPILERCHECK\"),\"%compiler% -v; echo \'$$(md5sum < $(STAMP_DIR)/ccache-toolchain-config)\'\"," $(@D)/ccache.c
 endef
 
 HOST_CCACHE_POST_CONFIGURE_HOOKS += \
-- 
1.7.9.5

[Buildroot] [PATCH 2/3] ccache: change default cache directory path to match config setting

[Danomi Manchego]

Commit 433290761fceb476b095548eec10adf72405e050 changed the hard-coded
ccache directory location to use BUILDROOT_CACHE_DIR, which is exported
by Makefile based on the BR2_CCACHE_DIR config option.  This allowed the
cache location to be changed on-the-fly by setting an environment
variable, but left the default location of ccache's normal default at
"$HOME/.ccache".  Since this location does not match the default for
BR2_CCACHE_DIR, it is basically almost never correct, so direct invocation
of ccache outside of the buildroot Makefile becomes cumbersome.

This patch changes the last-ditch cache location from "$HOME/.ccache" to
the BR2_CCACHE_DIR value at the time of host-ccache compilation.  Note
that the ability to override the location by BUILDROOT_CACHE_DIR is left
intact.

Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
---
 package/ccache/ccache.mk |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 9ad5129..1ebe974 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -50,12 +50,16 @@ HOST_CCACHE_CONF_OPT += ccache_cv_zlib_1_2_3=no
 #
 #    + Filter out blanks, comments, and dont-care stuff, then sort, to try to make
 #      immune to minor kconfig organizational changes.
+#
+#  - Change hard-coded last-ditch default to match path in .config, to avoid the
+#    need to specify BUILDROOT_CACHE_DIR when invoking ccache directly.
 define HOST_CCACHE_PATCH_CONFIGURATION
 	sed -i 's,getenv("CCACHE_DIR"),getenv("BUILDROOT_CACHE_DIR"),' $(@D)/ccache.c
 	sed -n '1,/^# Commands/p; /^\# Toolchain/,/^\# System config/p' $(BUILDROOT_CONFIG) | \
 		grep -v -e '^$$' -e '^\#' -e GDB -e ECLIPSE | \
 		sort > $(STAMP_DIR)/ccache-toolchain-config
 	sed -i "s,getenv(\"CCACHE_COMPILERCHECK\"),\"%compiler% -v; echo \'$$(md5sum < $(STAMP_DIR)/ccache-toolchain-config)\'\"," $(@D)/ccache.c
+	sed -i 's,"%s/.ccache","$(BUILDROOT_CACHE_DIR)",' $(@D)/ccache.c
 endef
 
 HOST_CCACHE_POST_CONFIGURE_HOOKS += \
-- 
1.7.9.5

[Buildroot] [PATCH 3/3] ccache: provide capability to do initial ccache setup

[Danomi Manchego]

For example, if your project is known to require more space
than the default max cache size, then you might want to
automatically increase the cache size to a suitable amount
using the -M option.

Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
---
 Config.in                |   15 ++++++++++++++-
 package/ccache/ccache.mk |   11 +++++++++++
 2 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/Config.in b/Config.in
index af720ec..c2971df 100644
--- a/Config.in
+++ b/Config.in
@@ -222,7 +222,7 @@ config BR2_CCACHE
 	help
 	  This option will enable the use of ccache, a compiler
 	  cache. It will cache the result of previous builds to speed
-	  up future builds. The cache is stored in
+	  up future builds. By default, the cache is stored in
 	  $HOME/.buildroot-ccache.
 
 	  Note that Buildroot does not try to invalidate the cache
@@ -239,6 +239,19 @@ config BR2_CCACHE_DIR
 	help
 	  Where ccache should store cached files.
 
+config BR2_CCACHE_INITIAL_SETUP
+	string "Compiler cache initial setup"
+	depends on BR2_CCACHE
+	help
+	  Additional ccache setup options, such as max-files or max-size.
+
+	  For example, if your project is known to require more space
+	  than the default max cache size, then you might want to
+	  automatically increase the cache size to a suitable amount
+	  using the -M option.
+
+	  Applied at time of ccache compilation.
+
 config BR2_DEPRECATED
 	bool "Show packages that are deprecated or obsolete"
 	help
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 1ebe974..cea0701 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -21,6 +21,17 @@ CCACHE_LICENSE_FILES = LICENSE.txt GPL-3.0.txt
 # has zero dependency besides the C library.
 HOST_CCACHE_CONF_OPT += ccache_cv_zlib_1_2_3=no
 
+# Provide capability to do initial ccache setup (e.g. increase default size)
+HOST_CCACHE_INITIAL_SETUP = $(call qstrip,$(BR2_CCACHE_INITIAL_SETUP))
+ifneq ($(HOST_CCACHE_INITIAL_SETUP),)
+  define HOST_CCACHE_DO_INITIAIL_SETUP
+	@$(call MESSAGE,"Do initial host ccache setup")
+	$(CCACHE) $(HOST_CCACHE_INITIAL_SETUP)
+	$(CCACHE) -s
+  endef
+  HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAIL_SETUP
+endif
+
 # Patch host-ccache as follows:
 #  - Use BUILDROOT_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
 #    is already used by autotargets for the ccache package.
-- 
1.7.9.5

[Buildroot] [PATCH 1/3] ccache: change compilercheck to use compiler and toolchain info

[Arnout Vandecappelle]

On 31/10/13 03:54, Danomi Manchego wrote:
> When CCACHE_COMPILERCHECK is set to "none", then ccache can be fooled in certain
> circumstances, resulting in using objects compiled with, say, the wrong toolchain.
> (This was discovered when compiling kmod from the same project, but with different
> external toolchains.)
>
> So let's try to make ccache use as safe as possible:
>
> - Use "%compiler% -v" in CCACHE_COMPILERCHECK to capture changes purely in an
>    externally provided toolchain.  See CCACHE_COMPILERCHECK and wrapper sections
>    at http://ccache.samba.org/manual.html for more info.

  As I said in the previous discussion, this makes the ccache almost 
useless unless absolute paths are filtered out from the output. Or am I 
mistaken?

>
> - Additionally, use the hash of part of the .config to describe toolchain and
>    C-library configuration that cannot be captured by the -v output:
>
>    + Use first section of .config, "Target Architecture" - "1,/^# Commands/", to
>      capture arch/cpu settings built into the external toolchain wrapper (for when
>      external toolchain it used) and the toolchain/c-library compilation settings
>      (for when toolchain is built).
>
>    + Use "Toolchain" section of .config - "/^\# Toolchain/,/^\# System config/",
>      for additional toolchain settings.
>
>    + Filter out blanks, comments, and dont-care stuff, then sort, to try to make
>      immune to minor kconfig organizational changes.

  If we're going to change this again, it should really be perfect this 
time :-) so let's analyse it carefully.

  The compiler that is called can be one of the following: HOSTCC, 
HOSTCXX, TARGET_CC, TARGET_CXX, or some other cross-tool that is called 
through CROSS_COMPILE. The last one we can safely ignore because ccache 
doesn't do anything with non-C-code. CC and CXX we can safely treat the 
same, because ccache always includes the basename into the hash. So we 
just have to support HOSTCC and TARGET_CC.

  For HOSTCC, mtime and %compiler% -v are probably equally good. mtime 
would be preferred, because it's much faster.

  For TARGET_CC, we again have several situations:

* Predefined external toolchain: the hash of the selected toolchain + the 
options passed when compiling the wrapper are enough. Since the latter 
also depend exclusively on the toolchain config options, hashing those is 
probably sufficient. Note that it doesn't really matter if the toolchain 
is preinstalled or downloaded (assuming the preinstalled one is the 
correct one).

* Custom external toolchain: we can assume that mtime of the real 
toolchain (not the wrapper) is pretty accurate here (when combined with 
the options embedded in the external toolchain wrapper). %compiler% -v 
doesn't work well, because it contains an absolute path to the 
lto-wrapper so its hash depends on where you extract the toolchain - if 
it's a downloaded toolchain, it will be different for different output 
directories. Purely the hash of the toolchain options doesn't work very 
well because the toolchain may be changed externally while staying in the 
same location (URL or path).

* Buildroot toolchain: in most cases, the arch and toolchain options 
(gcc, uclibc, binutils, elf2flt) accurately determine the compiler hash. 
There is one exception: when something changes in buildroot itself 
(adding a gcc patch, changing the way the compiler is built, ...), this 
will not be detected. But I think we can assume that a buildroot 
developer who is busy with that is smart enough to clean the cache. 
%compiler% -v doesn't work because it contains paths to the output 
directory and buildroot's git hash. mtime doesn't work because it changes 
every time the compiler is rebuilt.

To summarize, the ideal hash for each case is:

host gcc: mtime or %compiler% -v

predefined external toolchain: toolchain choice + wrapper options

custom external toolchain: mtime of real toolchain + wrapper options

buildroot toolchain: toolchain options


  So, how can we make optimal use of this? For the target toolchain, we 
can pre-compute the required information and set CCACHE_COMPILERCHECK to 
'echo $(TOOLCHAIN_HASH)'. That means we loose caching for the host 
packages, but has no other adverse effects.

  The stupid thing is that we know exactly what the compiler hash is at 
the time that we call ccache. So an even better solution would be if 
ccache had a way to pass options on the command line, so we can pass a 
different hash depending on which compiler is called.

  One more option is to always generate a wrapper, also for the internal 
toolchain. That one can set CCACHE_EXTRAFILES to a file that contains the 
toolchain options, and set CCACHE_COMPILERCHECK to none. The external 
toolchain wrapper could similarly be extended to call ccache instead of 
the compiler - and then it can just use mtime. And for the host compiler 
nothing needs to be done, it can just default to mtime.

  Does this sound more or less sane?

  Regards,
  Arnout

>
> Signed-off-by: Danomi Manchego <danomimanchego123@gmail.com>
>
> ---
>
> Mailing list thread: http://lists.busybox.net/pipermail/buildroot/2013-April/070819.html
> ---
>   package/ccache/ccache.mk |   25 ++++++++++++++++++++++++-
>   1 file changed, 24 insertions(+), 1 deletion(-)
>
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 82a53f3..9ad5129 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -26,13 +26,36 @@ HOST_CCACHE_CONF_OPT += ccache_cv_zlib_1_2_3=no
>   #    is already used by autotargets for the ccache package.
>   #    BUILDROOT_CACHE_DIR is exported by Makefile based on config option
>   #    BR2_CCACHE_DIR.
> +#
>   #  - ccache shouldn't use the compiler binary mtime to detect a change in
>   #    the compiler, because in the context of Buildroot, that completely
>   #    defeats the purpose of ccache. Of course, that leaves the user
>   #    responsible for purging its cache when the compiler changes.
> +#
> +#    But let's try to make ccache use as safe as possible.  Let's use
> +#    "%compiler% -v" in CCACHE_COMPILERCHECK to capture changes in external
> +#    toolchains.  See CCACHE_COMPILERCHECK and wrapper sections at
> +#    http://ccache.samba.org/manual.html for more info.
> +#
> +#    Additionally, use the hash of part of the .config to describe toolchain and
> +#    C-library configuration that cannot be captured by the compiler's -v output:
> +#
> +#    + Use first section of .config, "Target Architecture" - "1,/^# Commands/", to
> +#      capture arch/cpu settings built into the external toolchain wrapper (for when
> +#      external toolchain it used) and the toolchain/c-library compilation settings
> +#      (for when toolchain is built).
> +#
> +#    + Use "Toolchain" section of .config - "/^\# Toolchain/,/^\# System config/",
> +#      for additional toolchain settings.
> +#
> +#    + Filter out blanks, comments, and dont-care stuff, then sort, to try to make
> +#      immune to minor kconfig organizational changes.
>   define HOST_CCACHE_PATCH_CONFIGURATION
>   	sed -i 's,getenv("CCACHE_DIR"),getenv("BUILDROOT_CACHE_DIR"),' $(@D)/ccache.c
> -	sed -i 's,getenv("CCACHE_COMPILERCHECK"),"none",' $(@D)/ccache.c
> +	sed -n '1,/^# Commands/p; /^\# Toolchain/,/^\# System config/p' $(BUILDROOT_CONFIG) | \
> +		grep -v -e '^$$' -e '^\#' -e GDB -e ECLIPSE | \
> +		sort > $(STAMP_DIR)/ccache-toolchain-config
> +	sed -i "s,getenv(\"CCACHE_COMPILERCHECK\"),\"%compiler% -v; echo \'$$(md5sum < $(STAMP_DIR)/ccache-toolchain-config)\'\"," $(@D)/ccache.c
>   endef
>
>   HOST_CCACHE_POST_CONFIGURE_HOOKS += \
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 1/3] ccache: change compilercheck to use compiler and toolchain info

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

[Nice overview snipped]

 >  One more option is to always generate a wrapper, also for the
 > internal toolchain. That one can set CCACHE_EXTRAFILES to a file that
 > contains the toolchain options, and set CCACHE_COMPILERCHECK to
 > none. The external toolchain wrapper could similarly be extended to
 > call ccache instead of the compiler - and then it can just use
 > mtime. And for the host compiler nothing needs to be done, it can just
 > default to mtime.

I'm starting to think this is the best approach, yes.

-- 
Bye, Peter Korsgaard