Re: [Buildroot] [PATCH v4] package/ccache: bump version to 4.6.1

["Yann E. MORIN" <yann.morin.1998@free.fr>]

Dominik, Anders, All,

+James

On 2022-08-07 09:38 +0000, Dominik Michael Rauh spake thusly:
> Since version 4.0, the following major changes have been made to ccache,
> which affect its integration with Buildroot:
> 
> - Changed the default cache directory location to follow the XDG base directory specification.
> - Changed compression algorithm from Deflate (zlib) to Zstandard, enabled by default.
> - CMake is now used instead of Autoconf for configuration and building.
> 
> Additionally with version 4.4, an optional Redis backend for secondary
> storage on any server that supports the Redis protocol (by means of
> hiredis) has been added.
> 
> In contrast to ccache versions < 4, that came bundled with zlib, now
> zstd can merely be downloaded at compile time. Thus using the version
> that comes with Buildroot is the more sensible option. The same applies
> to hiredis.
> 
> Due to this, all of ccache's new dependencies (host-cmake, host-hiredis,
> host-pkgconf and host-zstd) have to be built before ccache.
> 
> Also add a section to Buildroot's ccache support manual on the new
> secondary storage backends.
> 
> Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
> Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>

Thanks a lot for this original patch! It helped me a lot when I was
reviewing and applying the series from James.

I applied patches from James instead, because they were split into more
manageable patches, rather than a big patch that changed everything at
once and, unless I completely borked and botched those patches when
aplying, could be applied as preparatory patches without actually
requiring bumping ccache.

Sorry that we did not have time to review and apply your patch sp far.
Still, I did refer a lot to this patch to understand parts of the
changes!

Now, we are left with just the need to bump ccache, which should be
slightly easier now.

Regards,
Yann E. MORIN.

> ---
> Changes v3-> v4 (after review by Thomas Petazzoni):
> - Rectified "BR2_CCACHE_REDIS_BACKEND" option
> 
> Changes v2 -> v3:
> - Add documentation
> 
> Changes v1 -> v2:
> - Improve commmit message
> - Disable building host-ccache's docs
> 
>  Config.in                                     |  6 +++
>  docs/manual/ccache-support.txt                | 17 +++++--
>  ...cache_dir-through-CCACHE_DIR-env-var.patch | 42 +++++++++++++++++
>  package/ccache/ccache.hash                    |  6 +--
>  package/ccache/ccache.mk                      | 45 ++++++++++---------
>  package/cmake/cmake.mk                        |  9 +++-
>  package/hiredis/hiredis.mk                    |  6 +++
>  package/pkg-generic.mk                        |  2 +-
>  package/pkgconf/pkgconf.mk                    |  5 +++
>  package/zstd/zstd.mk                          |  5 +++
>  10 files changed, 114 insertions(+), 29 deletions(-)
>  create mode 100644 package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
> 
> diff --git a/Config.in b/Config.in
> index 3c57c591a8..dbc49d0250 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -376,6 +376,12 @@ config BR2_CCACHE_USE_BASEDIR
>  	  the section "COMPILING IN DIFFERENT DIRECTORIES" in the ccache
>  	  manual for more information.
>  
> +config BR2_CCACHE_REDIS_BACKEND
> +	bool "Enable Redis storage backend"
> +	help
> +	  Enable ccache to use the Redis protocol to communicate with a
> +	  server for the purpose of sharing cache results with others.
> +
>  endif
>  
>  config BR2_ENABLE_DEBUG
> diff --git a/docs/manual/ccache-support.txt b/docs/manual/ccache-support.txt
> index ab4f10be23..1428c81a27 100644
> --- a/docs/manual/ccache-support.txt
> +++ b/docs/manual/ccache-support.txt
> @@ -4,7 +4,7 @@
>  [[ccache]]
>  ==== Using +ccache+ in Buildroot
>  
> -http://ccache.samba.org[ccache] is a compiler cache. It stores the
> +https://ccache.dev[ccache] is a compiler cache. It stores the
>  object files resulting from each compilation process, and is able to
>  skip future compilation of the same source file (with same compiler
>  and same arguments) by using the pre-existing object files. When doing
> @@ -52,9 +52,9 @@ relative paths in the object file. Therefore, for example, the debugger
>  will no longer find the file, unless you cd to the output directory
>  first.
>  
> -See https://ccache.samba.org/manual.html#_compiling_in_different_directories[the
> -ccache manual's section on "Compiling in different directories"] for
> -more details about this rewriting of absolute paths.
> +See https://ccache.dev/manual/latest.html#_compiling_in_different_directories[
> +the ccache manual's section on "Compiling in different directories"]
> +for more details about this rewriting of absolute paths.
>  
>  When +ccache+ is enabled in Buildroot using the +BR2_CCACHE=y+ option:
>  
> @@ -67,3 +67,12 @@ One can override this behavior using the +BR2_USE_CCACHE+ environment
>  variable: when set to +1+, usage of ccache is enabled (default during
>  the Buildroot build), when unset or set to a value different from +1+,
>  usage of ccache is disabled.
> +
> ++ccache+ has support for using the Redis protocol to communicate with
> +a server for the purpose of sharing cache results with others e.g. to
> +speed up the compilation process in a distributed CI/CD pipeline. For
> +this, the option +BR2_CCACHE_REDIS_BACKEND+ of +ccache+ has to be set.
> +Please see https://github.com/ccache/ccache/wiki/Redis-storage[ccache's
> +Wiki on how to configure this Redis storage backend] and its
> +https://ccache.dev/manual/latest.html#_secondary_storage_backends[manual's
> +section on "Secondary storage backends"] for general information.
> diff --git a/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
> new file mode 100644
> index 0000000000..dc2c86ddf8
> --- /dev/null
> +++ b/package/ccache/0001-Allow-to-patch-cache_dir-through-CCACHE_DIR-env-var.patch
> @@ -0,0 +1,42 @@
> +From 09231764c72bad287dd94c0dd67dff07f288b559 Mon Sep 17 00:00:00 2001
> +From: Dominik Michael Rauh <dmrauh@posteo.de>
> +Date: Thu, 4 Aug 2022 14:46:16 +0200
> +Subject: [PATCH] Allow to patch "cache_dir" through "CCACHE_DIR" env var
> +MIME-Version: 1.0
> +Content-Type: text/plain; charset=UTF-8
> +Content-Transfer-Encoding: 8bit
> +
> +Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com>
> +Signed-off-by: Dominik Michael Rauh <dmrauh@posteo.de>
> +[Upstream: not applicable due to only being needed by Buildroot]
> +---
> + src/Config.cpp | 6 +++++-
> + 1 file changed, 5 insertions(+), 1 deletion(-)
> +
> +diff --git a/src/Config.cpp b/src/Config.cpp
> +index ee4cd9f7..8b40f960 100644
> +--- a/src/Config.cpp
> ++++ b/src/Config.cpp
> +@@ -162,7 +162,6 @@ const std::unordered_map<std::string, std::string> k_env_variable_table = {
> +   {"DEBUG", "debug"},
> +   {"DEBUGDIR", "debug_dir"},
> +   {"DEPEND", "depend_mode"},
> +-  {"DIR", "cache_dir"},
> +   {"DIRECT", "direct_mode"},
> +   {"DISABLE", "disable"},
> +   {"EXTENSION", "cpp_extension"},
> +@@ -613,6 +612,11 @@ Config::update_from_environment()
> +       throw core::Error("CCACHE_{}{}: {}", negate ? "NO" : "", key, e.what());
> +     }
> +   }
> ++
> ++  const char* const env_ccache_dir = getenv("CCACHE_DIR");
> ++  if (env_ccache_dir && *env_ccache_dir) {
> ++    set_item("cache_dir", env_ccache_dir, "DIR", false, "environment");
> ++  }
> + }
> + 
> + std::string
> +-- 
> +2.30.2
> +
> diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
> index 16cfad602c..980ac46d77 100644
> --- a/package/ccache/ccache.hash
> +++ b/package/ccache/ccache.hash
> @@ -1,4 +1,4 @@
>  # sha256 computed locally
> -sha256  a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400  ccache-3.7.12.tar.xz
> -sha256  ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361  GPL-3.0.txt
> -sha256  4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59  LICENSE.adoc
> +sha256  e5d47bd3cbb504ada864124690e7c0d28ecb1f9aeac22a9976025aed9633f3d1  ccache-4.6.1.tar.xz
> +sha256  80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5  GPL-3.0.txt
> +sha256  c5b8e7e1c59184fd9c0e823e6f019ec261d3ee8e2860dd5e3ad98842b730994a  LICENSE.adoc
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 97aa8b1c75..a6b15a6238 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -4,27 +4,28 @@
>  #
>  ################################################################################
>  
> -CCACHE_VERSION = 3.7.12
> +CCACHE_VERSION = 4.6.1
>  CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
>  CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
>  CCACHE_LICENSE = GPL-3.0+, others
>  CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
> +HOST_CCACHE_DEPENDENCIES = host-zstd
>  
> -# Force ccache to use its internal zlib. The problem is that without
> -# this, ccache would link against the zlib of the build system, but we
> -# might build and install a different version of zlib in $(O)/host
> -# afterwards, which ccache will pick up. This might break if there is
> -# a version mismatch. A solution would be to add host-zlib has a
> -# dependency of ccache, but it would require tuning the zlib .mk file
> -# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
> -# path: tell ccache to use its internal copy of zlib, so that ccache
> -# has zero dependency besides the C library.
> -HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
> +HOST_CCACHE_CONF_OPTS += \
> +	-DENABLE_DOCUMENTATION=OFF \
> +	-DENABLE_TESTING=OFF
>  
>  # We are ccache, so we can't use ccache
> -HOST_CCACHE_CONF_ENV = \
> -	CC="$(HOSTCC_NOCCACHE)" \
> -	CXX="$(HOSTCXX_NOCCACHE)"
> +HOST_CCACHE_CONF_OPTS += \
> +	-UCMAKE_C_COMPILER_LAUNCHER \
> +	-UCMAKE_CXX_COMPILER_LAUNCHER
> +
> +ifeq ($(BR2_CCACHE_REDIS_BACKEND),y)
> +HOST_CCACHE_DEPENDENCIES += host-hiredis
> +HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=ON
> +else
> +HOST_CCACHE_CONF_OPTS += -DREDIS_STORAGE_BACKEND=OFF
> +endif
>  
>  # Patch host-ccache as follows:
>  #  - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
> @@ -33,13 +34,17 @@ HOST_CCACHE_CONF_ENV = \
>  #    BR2_CCACHE_DIR.
>  #  - Change hard-coded last-ditch default to match path in .config, to avoid
>  #    the need to specify BR_CACHE_DIR when invoking ccache directly.
> -#    CCache replaces "%s" with the home directory of the current user,
> -#    So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
> -HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
> +#    CCache replaces "home_dir" with the home directory of the current user,
> +#    so rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
> +HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
>  
>  define HOST_CCACHE_PATCH_CONFIGURATION
> -	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
> -	sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
> +	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
> +	sed -i 's,home_dir + "/.ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
> +	sed -i 's,getenv("XDG_CACHE_HOME"),nullptr,' $(@D)/src/Config.cpp
> +	sed -i 's,home_dir + "/.cache/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
> +	sed -i 's,getenv("XDG_CONFIG_HOME"),nullptr,' $(@D)/src/Config.cpp
> +	sed -i 's,home_dir + "/.config/ccache",$(HOST_CCACHE_DEFAULT_CCACHE_DIR),' $(@D)/src/Config.cpp
>  endef
>  
>  HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
> @@ -62,7 +67,7 @@ endef
>  HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
>  endif
>  
> -$(eval $(host-autotools-package))
> +$(eval $(host-cmake-package))
>  
>  ifeq ($(BR2_CCACHE),y)
>  ccache-stats: host-ccache
> diff --git a/package/cmake/cmake.mk b/package/cmake/cmake.mk
> index 053658fad6..eae12c1465 100644
> --- a/package/cmake/cmake.mk
> +++ b/package/cmake/cmake.mk
> @@ -40,9 +40,16 @@ CMAKE_CONF_OPTS = \
>  HOST_CMAKE_CFLAGS = $(shell echo $(HOST_CFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
>  HOST_CMAKE_CXXFLAGS = $(shell echo $(HOST_CXXFLAGS) | sed -r "s%$(HOST_CPPFLAGS)%%")
>  
> +# we are built before ccache
> +HOST_CMAKE_CONFIGURE_OPTS = \
> +	$(HOST_CONFIGURE_OPTS) \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	GCC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  define HOST_CMAKE_CONFIGURE_CMDS
>  	(cd $(@D); \
> -		$(HOST_CONFIGURE_OPTS) \
> +		$(HOST_CMAKE_CONFIGURE_OPTS) \
>  		CFLAGS="$(HOST_CMAKE_CFLAGS)" \
>  		./bootstrap --prefix=$(HOST_DIR) \
>  			--parallel=$(PARALLEL_JOBS) -- \
> diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
> index a571951fa3..2cb9991029 100644
> --- a/package/hiredis/hiredis.mk
> +++ b/package/hiredis/hiredis.mk
> @@ -20,4 +20,10 @@ else
>  HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
>  endif
>  
> +# we are built before ccache
> +HOST_HIREDIS_CONF_OPTS += \
> +	-UCMAKE_C_COMPILER_LAUNCHER \
> +	-UCMAKE_CXX_COMPILER_LAUNCHER
> +
>  $(eval $(cmake-package))
> +$(eval $(host-cmake-package))
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index b233b07548..cba94052a0 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
>  endif
>  
>  ifeq ($$(BR2_CCACHE),y)
> -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
> +ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
>  $(2)_DEPENDENCIES += host-ccache
>  endif
>  endif
> diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
> index 5d65f69c10..6b187b0238 100644
> --- a/package/pkgconf/pkgconf.mk
> +++ b/package/pkgconf/pkgconf.mk
> @@ -32,6 +32,11 @@ define HOST_PKGCONF_SHARED
>  	$(SED) 's,@STATIC@,,' $(HOST_DIR)/bin/pkg-config
>  endef
>  
> +# we are built before ccache
> +HOST_PKGCONF_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  PKGCONF_POST_INSTALL_TARGET_HOOKS += PKGCONF_LINK_PKGCONFIG
>  HOST_PKGCONF_POST_INSTALL_HOOKS += HOST_PKGCONF_INSTALL_WRAPPER
>  
> diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
> index e8f6315222..edc4f1c624 100644
> --- a/package/zstd/zstd.mk
> +++ b/package/zstd/zstd.mk
> @@ -97,6 +97,11 @@ endef
>  
>  HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
>  
> +# we are built before ccache
> +HOST_ZSTD_OPTS += \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  define HOST_ZSTD_BUILD_CMDS
>  	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
>  		-C $(@D) zstd-release lib-release
> -- 
> 2.37.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot