[Buildroot] [PATCH 1/4] package/hiredis: enable host package

[Buildroot] [PATCH 1/4] package/hiredis: enable host package

[James Hilliard]

The host-hiredis package is needed for the upcoming ccache update.

Note that we must disable ccache when building host-hiredis itself
as host-ccache depends on host-hiredis.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/hiredis/hiredis.mk | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
index f79b6c757a..f2a4836310 100644
--- a/package/hiredis/hiredis.mk
+++ b/package/hiredis/hiredis.mk
@@ -22,4 +22,13 @@ else
 HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
 endif
 
+# We are a ccache dependency, so we can't use ccache
+HOST_HIREDIS_CONF_OPTS += \
+	-DCMAKE_C_COMPILER_LAUNCHER="" \
+	-DCMAKE_CXX_COMPILER_LAUNCHER="" \
+	-DCMAKE_BUILD_TYPE=Release \
+	-DDISABLE_TESTS=ON \
+	-DENABLE_SSL=OFF
+
 $(eval $(cmake-package))
+$(eval $(host-cmake-package))
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/4] package/zstd: disable ccache for host package

[James Hilliard]

We need to disable ccache when building host-zstd as the upcoming
ccache update depends on host-zstd.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/zstd/zstd.mk | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
index e8f6315222..cb4f4cc611 100644
--- a/package/zstd/zstd.mk
+++ b/package/zstd/zstd.mk
@@ -98,12 +98,16 @@ endef
 HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
 
 define HOST_ZSTD_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) \
+	CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)" \
+	$(MAKE) $(HOST_ZSTD_OPTS) \
 		-C $(@D) zstd-release lib-release
 endef
 
 define HOST_ZSTD_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) \
+	CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)" \
+	$(MAKE) $(HOST_ZSTD_OPTS) \
 		-C $(@D) install
 endef
 
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/4] package/zstd: disable ccache for host package

[Yann E. MORIN]

James, All,

+Dominik +Anders

On 2023-01-06 18:04 -0700, James Hilliard spake thusly:
> We need to disable ccache when building host-zstd as the upcoming
> ccache update depends on host-zstd.
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
>  package/zstd/zstd.mk | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk
> index e8f6315222..cb4f4cc611 100644
> --- a/package/zstd/zstd.mk
> +++ b/package/zstd/zstd.mk
> @@ -98,12 +98,16 @@ endef
>  HOST_ZSTD_OPTS += PREFIX=$(HOST_DIR)
>  
>  define HOST_ZSTD_BUILD_CMDS
> -	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
> +	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) \
> +	CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)" \
> +	$(MAKE) $(HOST_ZSTD_OPTS) \
>  		-C $(@D) zstd-release lib-release
>  endef
>  
>  define HOST_ZSTD_INSTALL_CMDS
> -	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) $(HOST_ZSTD_OPTS) \
> +	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) \
> +	CC="$(HOSTCC_NOCCACHE)" CXX="$(HOSTCXX_NOCCACHE)" \
> +	$(MAKE) $(HOST_ZSTD_OPTS) \

Rather than duplicating the environment in both _CMDS, I've introduce
HOST_ZSTD_ENV.

Applied to master, thanks.

Regards,
Yann E. MORIN.

>  		-C $(@D) install
>  endef
>  
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 3/4] package/pkgconf: disable ccache for host package

[James Hilliard]

We need to disable ccache when building host-pkgconf as the upcoming
ccache update depends on host-pkgconf indirectly due to host-cmake
depending on host-pkgconf which will be required due to host-ccache
migrating from autotools to cmake package infrastructure.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/pkgconf/pkgconf.mk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index 5d65f69c10..e9587fae6b 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -11,6 +11,11 @@ PKGCONF_LICENSE = pkgconf license
 PKGCONF_LICENSE_FILES = COPYING
 PKGCONF_CPE_ID_VENDOR = pkgconf
 
+# We are a ccache dependency, so we can't use ccache
+HOST_PKGCONF_CONF_ENV = \
+	CC="$(HOSTCC_NOCCACHE)" \
+	CXX="$(HOSTCXX_NOCCACHE)"
+
 PKG_CONFIG_HOST_BINARY = $(HOST_DIR)/bin/pkg-config
 
 define PKGCONF_LINK_PKGCONFIG
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 3/4] package/pkgconf: disable ccache for host package

[Yann E. MORIN]

James, All,

On 2023-01-06 18:04 -0700, James Hilliard spake thusly:
> We need to disable ccache when building host-pkgconf as the upcoming
> ccache update depends on host-pkgconf indirectly due to host-cmake
> depending on host-pkgconf which will be required due to host-ccache
> migrating from autotools to cmake package infrastructure.

I've added a note about having to take care of propagating the no-cache
build doen to the autotools-related packages if pkgconf ever needs
_AUTORECONF=YES

> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  package/pkgconf/pkgconf.mk | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
> index 5d65f69c10..e9587fae6b 100644
> --- a/package/pkgconf/pkgconf.mk
> +++ b/package/pkgconf/pkgconf.mk
> @@ -11,6 +11,11 @@ PKGCONF_LICENSE = pkgconf license
>  PKGCONF_LICENSE_FILES = COPYING
>  PKGCONF_CPE_ID_VENDOR = pkgconf
>  
> +# We are a ccache dependency, so we can't use ccache
> +HOST_PKGCONF_CONF_ENV = \
> +	CC="$(HOSTCC_NOCCACHE)" \
> +	CXX="$(HOSTCXX_NOCCACHE)"
> +
>  PKG_CONFIG_HOST_BINARY = $(HOST_DIR)/bin/pkg-config
>  
>  define PKGCONF_LINK_PKGCONFIG
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 4/4] package/ccache: bump to version 4.7.4

[James Hilliard]

Migrate to cmake package infrastructure.

Add new host-hiredis host-zstd dependencies.

Add new ccache dependency exclusions to pkg-generic.

Migrate HOST_CCACHE_PATCH_CONFIGURATION to handle updated
source files/format.

License hashes changed due to migrating urls to https:
https://github.com/ccache/ccache/commit/a0f32f161f1b8b9c5d287ca8abe88e3fd1e940a2

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/ccache/ccache.hash |  6 +++---
 package/ccache/ccache.mk   | 30 +++++++++++-------------------
 package/pkg-generic.mk     |  2 +-
 3 files changed, 15 insertions(+), 23 deletions(-)

diff --git a/package/ccache/ccache.hash b/package/ccache/ccache.hash
index 16cfad602c..9b11f026ec 100644
--- a/package/ccache/ccache.hash
+++ b/package/ccache/ccache.hash
@@ -1,4 +1,4 @@
 # sha256 computed locally
-sha256  a02f4e8360dc6618bc494ca35b0ae21cea080f804a4898eab1ad3fcd108eb400  ccache-3.7.12.tar.xz
-sha256  ec6b1a326ff93b2cc21df88a697ae470ff6927a55b8929e7e491b315e1563361  GPL-3.0.txt
-sha256  4f63223bcdee822d55a6768cc8399ffe06401d7a03cfe79e1dc305f50dc22c59  LICENSE.adoc
+sha256  df0c64d15d3efaf0b4f6837dd6b1467e40eeaaa807db25ce79c3a08a46a84e36  ccache-4.7.4.tar.xz
+sha256  80b5112739a423dfac7bed1ca8a1df3cccda3d794425441997d4462b83db4dd5  GPL-3.0.txt
+sha256  56cf64f68b7b8150300acd79899941615523ea5c56c120e92f5b175b7cc78e3c  LICENSE.adoc
diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
index 97aa8b1c75..0c208cb9e6 100644
--- a/package/ccache/ccache.mk
+++ b/package/ccache/ccache.mk
@@ -4,27 +4,19 @@
 #
 ################################################################################
 
-CCACHE_VERSION = 3.7.12
+CCACHE_VERSION = 4.7.4
 CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
 CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
 CCACHE_LICENSE = GPL-3.0+, others
 CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
-
-# Force ccache to use its internal zlib. The problem is that without
-# this, ccache would link against the zlib of the build system, but we
-# might build and install a different version of zlib in $(O)/host
-# afterwards, which ccache will pick up. This might break if there is
-# a version mismatch. A solution would be to add host-zlib has a
-# dependency of ccache, but it would require tuning the zlib .mk file
-# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
-# path: tell ccache to use its internal copy of zlib, so that ccache
-# has zero dependency besides the C library.
-HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
+HOST_CCACHE_DEPENDENCIES = host-hiredis host-zstd
 
 # We are ccache, so we can't use ccache
-HOST_CCACHE_CONF_ENV = \
-	CC="$(HOSTCC_NOCCACHE)" \
-	CXX="$(HOSTCXX_NOCCACHE)"
+HOST_CCACHE_CONF_OPTS += \
+	-DCMAKE_C_COMPILER_LAUNCHER="" \
+	-DCMAKE_CXX_COMPILER_LAUNCHER="" \
+	-DZSTD_FROM_INTERNET=OFF \
+	-DHIREDIS_FROM_INTERNET=OFF
 
 # Patch host-ccache as follows:
 #  - Use BR_CACHE_DIR instead of CCACHE_DIR, because CCACHE_DIR
@@ -35,11 +27,11 @@ HOST_CCACHE_CONF_ENV = \
 #    the need to specify BR_CACHE_DIR when invoking ccache directly.
 #    CCache replaces "%s" with the home directory of the current user,
 #    So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
-HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
+HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,%,$(BR_CACHE_DIR))
 
 define HOST_CCACHE_PATCH_CONFIGURATION
-	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
-	sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
+	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
+	sed -i 's,".ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/Config.cpp
 endef
 
 HOST_CCACHE_POST_PATCH_HOOKS += HOST_CCACHE_PATCH_CONFIGURATION
@@ -62,7 +54,7 @@ endef
 HOST_CCACHE_POST_INSTALL_HOOKS += HOST_CCACHE_DO_INITIAL_SETUP
 endif
 
-$(eval $(host-autotools-package))
+$(eval $(host-cmake-package))
 
 ifeq ($(BR2_CCACHE),y)
 ccache-stats: host-ccache
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index f24e03a325..0863444221 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
 endif
 
 ifeq ($$(BR2_CCACHE),y)
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
+ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
 $(2)_DEPENDENCIES += host-ccache
 endif
 endif
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 4/4] package/ccache: bump to version 4.7.4

[Yann E. MORIN]

James, All,

+Dominik, +Anders

On 2023-01-06 18:04 -0700, James Hilliard spake thusly:
> Migrate to cmake package infrastructure.
> 
> Add new host-hiredis host-zstd dependencies.
> 
> Add new ccache dependency exclusions to pkg-generic.
> 
> Migrate HOST_CCACHE_PATCH_CONFIGURATION to handle updated
> source files/format.
> 
> License hashes changed due to migrating urls to https:
> https://github.com/ccache/ccache/commit/a0f32f161f1b8b9c5d287ca8abe88e3fd1e940a2
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
>  package/ccache/ccache.hash |  6 +++---
>  package/ccache/ccache.mk   | 30 +++++++++++-------------------

This misses a critical part: what prevents cmake from using ccache to
build?

Dominik and Anders got that correct in their patch:

    +# we are built before ccache
    +HOST_CMAKE_CONFIGURE_OPTS = \
    +	$(HOST_CONFIGURE_OPTS) \
    +	CC="$(HOSTCC_NOCCACHE)" \
    +	GCC="$(HOSTCC_NOCCACHE)" \
    +	CXX="$(HOSTCXX_NOCCACHE)"
    +
     define HOST_CMAKE_CONFIGURE_CMDS
     	(cd $(@D); \
    -		$(HOST_CONFIGURE_OPTS) \
    +		$(HOST_CMAKE_CONFIGURE_OPTS) \
     		CFLAGS="$(HOST_CMAKE_CFLAGS)" \

(Arguably, CFLAGS assignment should have been moved into
HOST_CMAKE_CONFIGURE_OPTS as well.)

[--SNIP--]
> diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> index 97aa8b1c75..0c208cb9e6 100644
> --- a/package/ccache/ccache.mk
> +++ b/package/ccache/ccache.mk
> @@ -4,27 +4,19 @@
>  #
>  ################################################################################
>  
> -CCACHE_VERSION = 3.7.12
> +CCACHE_VERSION = 4.7.4
>  CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
>  CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
>  CCACHE_LICENSE = GPL-3.0+, others
>  CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
> -
> -# Force ccache to use its internal zlib. The problem is that without
> -# this, ccache would link against the zlib of the build system, but we
> -# might build and install a different version of zlib in $(O)/host
> -# afterwards, which ccache will pick up. This might break if there is
> -# a version mismatch. A solution would be to add host-zlib has a
> -# dependency of ccache, but it would require tuning the zlib .mk file
> -# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
> -# path: tell ccache to use its internal copy of zlib, so that ccache
> -# has zero dependency besides the C library.
> -HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
> +HOST_CCACHE_DEPENDENCIES = host-hiredis host-zstd

Dominik and Anders made hiredis an optional dependency. Given that
hiredis builds very, very fast (just about 3s here), I wonder if that
really makes sense to make it optional; hiredis has no dependency that
are not dependencies of ccache, so there is no hidden overhead.

If hiredis support is compiled in, does that make it mandatory to use a
hiredis server, or does ccache still uses local files by default?

If the latter, then I think a mandatory dependency is OK.

Dominik, Anders, did you make it a optional because of a previous review
by Thomas asking so?

[--SNIP--]
> +HOST_CCACHE_CONF_OPTS += \
> +	-DCMAKE_C_COMPILER_LAUNCHER="" \
> +	-DCMAKE_CXX_COMPILER_LAUNCHER="" \

Same as for hiredis: use -U.

[--SNIP--]
> @@ -35,11 +27,11 @@ HOST_CCACHE_CONF_ENV = \
>  #    the need to specify BR_CACHE_DIR when invoking ccache directly.
>  #    CCache replaces "%s" with the home directory of the current user,
>  #    So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
> -HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
> +HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,%,$(BR_CACHE_DIR))

Err, this is probably bad and the comment above no longer matches the
code.

Did you meant using 'home_dir +' to replace '%s' (see Dominik and
Anders' patch):

    HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")

>  define HOST_CCACHE_PATCH_CONFIGURATION
> -	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
> -	sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
> +	sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
> +	sed -i 's,".ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/Config.cpp

This needs a bit of consolidation and further explanations, becasue
ccache now has more locations where it looks for, and we do not want it
to default to those:

    home_dir + "/.ccache"           -> the legacy location
    home_dir + "/.cache/ccache"     -> new, XDG-based
    home_dir + "/.config/ccache"    -> new, XDG-based

Also, it seems that we need a patch to use BR_CACHE_DIR from the
environment...

[--SNIP--]
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index f24e03a325..0863444221 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
>  endif
>  
>  ifeq ($$(BR2_CCACHE),y)
> -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
> +ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)

I've applied that part with just host-hiredis, host-pkgconf, and host-zstd,
and left host-cmake out for now.

Regards,
Yann E. MORIN.

>  $(2)_DEPENDENCIES += host-ccache
>  endif
>  endif
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 4/4] package/ccache: bump to version 4.7.4

[James Hilliard]

On Sun, Jan 8, 2023 at 3:29 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> James, All,
>
> +Dominik, +Anders
>
> On 2023-01-06 18:04 -0700, James Hilliard spake thusly:
> > Migrate to cmake package infrastructure.
> >
> > Add new host-hiredis host-zstd dependencies.
> >
> > Add new ccache dependency exclusions to pkg-generic.
> >
> > Migrate HOST_CCACHE_PATCH_CONFIGURATION to handle updated
> > source files/format.
> >
> > License hashes changed due to migrating urls to https:
> > https://github.com/ccache/ccache/commit/a0f32f161f1b8b9c5d287ca8abe88e3fd1e940a2
> >
> > Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> > ---
> >  package/ccache/ccache.hash |  6 +++---
> >  package/ccache/ccache.mk   | 30 +++++++++++-------------------
>
> This misses a critical part: what prevents cmake from using ccache to
> build?
>
> Dominik and Anders got that correct in their patch:
>
>     +# we are built before ccache
>     +HOST_CMAKE_CONFIGURE_OPTS = \
>     +   $(HOST_CONFIGURE_OPTS) \
>     +   CC="$(HOSTCC_NOCCACHE)" \
>     +   GCC="$(HOSTCC_NOCCACHE)" \
>     +   CXX="$(HOSTCXX_NOCCACHE)"
>     +
>      define HOST_CMAKE_CONFIGURE_CMDS
>         (cd $(@D); \
>     -           $(HOST_CONFIGURE_OPTS) \
>     +           $(HOST_CMAKE_CONFIGURE_OPTS) \
>                 CFLAGS="$(HOST_CMAKE_CFLAGS)" \

Fixed in v3:
https://lore.kernel.org/buildroot/20230109001037.641422-1-james.hilliard1@gmail.com/

>
> (Arguably, CFLAGS assignment should have been moved into
> HOST_CMAKE_CONFIGURE_OPTS as well.)
>
> [--SNIP--]
> > diff --git a/package/ccache/ccache.mk b/package/ccache/ccache.mk
> > index 97aa8b1c75..0c208cb9e6 100644
> > --- a/package/ccache/ccache.mk
> > +++ b/package/ccache/ccache.mk
> > @@ -4,27 +4,19 @@
> >  #
> >  ################################################################################
> >
> > -CCACHE_VERSION = 3.7.12
> > +CCACHE_VERSION = 4.7.4
> >  CCACHE_SITE = https://github.com/ccache/ccache/releases/download/v$(CCACHE_VERSION)
> >  CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.xz
> >  CCACHE_LICENSE = GPL-3.0+, others
> >  CCACHE_LICENSE_FILES = LICENSE.adoc GPL-3.0.txt
> > -
> > -# Force ccache to use its internal zlib. The problem is that without
> > -# this, ccache would link against the zlib of the build system, but we
> > -# might build and install a different version of zlib in $(O)/host
> > -# afterwards, which ccache will pick up. This might break if there is
> > -# a version mismatch. A solution would be to add host-zlib has a
> > -# dependency of ccache, but it would require tuning the zlib .mk file
> > -# to use HOSTCC_NOCCACHE as the compiler. Instead, we take the easy
> > -# path: tell ccache to use its internal copy of zlib, so that ccache
> > -# has zero dependency besides the C library.
> > -HOST_CCACHE_CONF_OPTS += --with-bundled-zlib
> > +HOST_CCACHE_DEPENDENCIES = host-hiredis host-zstd
>
> Dominik and Anders made hiredis an optional dependency. Given that
> hiredis builds very, very fast (just about 3s here), I wonder if that
> really makes sense to make it optional; hiredis has no dependency that
> are not dependencies of ccache, so there is no hidden overhead.
>
> If hiredis support is compiled in, does that make it mandatory to use a
> hiredis server, or does ccache still uses local files by default?

Default is to use local files.

>
> If the latter, then I think a mandatory dependency is OK.

Yeah, it didn't seem like it made sense to make it optional since hiredis is a
small library.

>
> Dominik, Anders, did you make it a optional because of a previous review
> by Thomas asking so?
>
> [--SNIP--]
> > +HOST_CCACHE_CONF_OPTS += \
> > +     -DCMAKE_C_COMPILER_LAUNCHER="" \
> > +     -DCMAKE_CXX_COMPILER_LAUNCHER="" \
>
> Same as for hiredis: use -U.

Fixed in v3:
https://lore.kernel.org/buildroot/20230109001037.641422-2-james.hilliard1@gmail.com/

>
> [--SNIP--]
> > @@ -35,11 +27,11 @@ HOST_CCACHE_CONF_ENV = \
> >  #    the need to specify BR_CACHE_DIR when invoking ccache directly.
> >  #    CCache replaces "%s" with the home directory of the current user,
> >  #    So rewrite BR_CACHE_DIR to take that into consideration for SDK purpose
> > -HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,\%s/%,$(BR_CACHE_DIR))
> > +HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst $(HOME)/%,%,$(BR_CACHE_DIR))
>
> Err, this is probably bad and the comment above no longer matches the
> code.
>
> Did you meant using 'home_dir +' to replace '%s' (see Dominik and
> Anders' patch):

Well matching against the full 'home_dir +' doesn't seem to be needed,
we can just target the appended strings for replacement which is a bit
cleaner:
https://lore.kernel.org/buildroot/20230109001037.641422-2-james.hilliard1@gmail.com/

>
>     HOST_CCACHE_DEFAULT_CCACHE_DIR = $(patsubst \"$(HOME)/%,home_dir + \"/%,\"$(BR_CACHE_DIR)\")
>
> >  define HOST_CCACHE_PATCH_CONFIGURATION
> > -     sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/ccache.c
> > -     sed -i 's,"%s/.ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/conf.c
> > +     sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/src/Config.cpp
> > +     sed -i 's,".ccache","$(HOST_CCACHE_DEFAULT_CCACHE_DIR)",' $(@D)/src/Config.cpp
>
> This needs a bit of consolidation and further explanations, becasue
> ccache now has more locations where it looks for, and we do not want it
> to default to those:

I have them all being overridden now in my v3:
https://lore.kernel.org/buildroot/20230109001037.641422-2-james.hilliard1@gmail.com/

>
>     home_dir + "/.ccache"           -> the legacy location
>     home_dir + "/.cache/ccache"     -> new, XDG-based
>     home_dir + "/.config/ccache"    -> new, XDG-based
>
> Also, it seems that we need a patch to use BR_CACHE_DIR from the
> environment...

I cleaned this up a bit in my v3 to handle the cases from Dominik and
Anders' patch:
https://lore.kernel.org/buildroot/20230109001037.641422-2-james.hilliard1@gmail.com/

>
> [--SNIP--]
> > diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> > index f24e03a325..0863444221 100644
> > --- a/package/pkg-generic.mk
> > +++ b/package/pkg-generic.mk
> > @@ -792,7 +792,7 @@ $(2)_EXTRACT_DEPENDENCIES += \
> >  endif
> >
> >  ifeq ($$(BR2_CCACHE),y)
> > -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache,$(1)),)
> > +ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate host-ccache host-cmake host-hiredis host-pkgconf host-zstd,$(1)),)
>
> I've applied that part with just host-hiredis, host-pkgconf, and host-zstd,
> and left host-cmake out for now.
>
> Regards,
> Yann E. MORIN.
>
> >  $(2)_DEPENDENCIES += host-ccache
> >  endif
> >  endif
> > --
> > 2.34.1
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 4/4] package/ccache: bump to version 4.7.4

[Dominik Michael Rauh]

Yann, James, All,

On 08.01.23 23:29, Yann E. MORIN wrote:

> Dominik and Anders made hiredis an optional dependency. Given that
> hiredis builds very, very fast (just about 3s here), I wonder if that
> really makes sense to make it optional; hiredis has no dependency that
> are not dependencies of ccache, so there is no hidden overhead.
> 
> If hiredis support is compiled in, does that make it mandatory to use a
> hiredis server, or does ccache still uses local files by default?
> 
> If the latter, then I think a mandatory dependency is OK.
> 
> Dominik, Anders, did you make it a optional because of a previous review
> by Thomas asking so?

The decision to make hiredis support optional was only based on the fact 
that, well, it is purely optional :)

In hindsight I agree to enable the hiredis backend by default since
hiredis builds very quickly and does not pull in additional dependencies!

Maybe the more interesting question is whether it is really advisable to
still build ccache as part of Buildroot since, in contrast to hiredis,
building cmake, which is required by ccache, takes significantly longer 
and consumes much more resources.

On 07.08.22 14:39, Thomas Petazzoni wrote:

 > A possible alternative we could explore is to no longer build ccache
 > ourselves and rely on ccache being provided by the host distribution.
 > Of course, raises the question of which ccache version is available on
 > the host, and how to tweak its configuration to match the needs of
 > Buildroot.

Best regards,
Dominik

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 4/4] package/ccache: bump to version 4.7.4

[James Hilliard]

On Mon, Jan 9, 2023 at 12:21 PM Dominik Michael Rauh <dmrauh@posteo.de> wrote:
>
> Yann, James, All,
>
> On 08.01.23 23:29, Yann E. MORIN wrote:
>
> > Dominik and Anders made hiredis an optional dependency. Given that
> > hiredis builds very, very fast (just about 3s here), I wonder if that
> > really makes sense to make it optional; hiredis has no dependency that
> > are not dependencies of ccache, so there is no hidden overhead.
> >
> > If hiredis support is compiled in, does that make it mandatory to use a
> > hiredis server, or does ccache still uses local files by default?
> >
> > If the latter, then I think a mandatory dependency is OK.
> >
> > Dominik, Anders, did you make it a optional because of a previous review
> > by Thomas asking so?
>
> The decision to make hiredis support optional was only based on the fact
> that, well, it is purely optional :)
>
> In hindsight I agree to enable the hiredis backend by default since
> hiredis builds very quickly and does not pull in additional dependencies!
>
> Maybe the more interesting question is whether it is really advisable to
> still build ccache as part of Buildroot since, in contrast to hiredis,
> building cmake, which is required by ccache, takes significantly longer
> and consumes much more resources.

Well we use the prebuilt build host cmake when available so this
probably isn't that big an issue.

>
> On 07.08.22 14:39, Thomas Petazzoni wrote:
>
>  > A possible alternative we could explore is to no longer build ccache
>  > ourselves and rely on ccache being provided by the host distribution.
>  > Of course, raises the question of which ccache version is available on
>  > the host, and how to tweak its configuration to match the needs of
>  > Buildroot.
>
> Best regards,
> Dominik
>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/4] package/hiredis: enable host package

[Yann E. MORIN]

James, All,

There's this already pending, old-ish patch from Dominik and
Anders:
    https://patchwork.ozlabs.org/project/buildroot/patch/20220807093853.15579-1-dmrauh@posteo.de/

that tries to bump ccache.

I prefer your series, because everything is split as much as possible,
but I am looking at both patches to see if we can pick the best of the
two. And indeed, see below...

On 2023-01-06 18:04 -0700, James Hilliard spake thusly:
> The host-hiredis package is needed for the upcoming ccache update.
> 
> Note that we must disable ccache when building host-hiredis itself
> as host-ccache depends on host-hiredis.
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
>  package/hiredis/hiredis.mk | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/package/hiredis/hiredis.mk b/package/hiredis/hiredis.mk
> index f79b6c757a..f2a4836310 100644
> --- a/package/hiredis/hiredis.mk
> +++ b/package/hiredis/hiredis.mk
> @@ -22,4 +22,13 @@ else
>  HIREDIS_CONF_OPTS += -DENABLE_SSL=OFF
>  endif
>  
> +# We are a ccache dependency, so we can't use ccache
> +HOST_HIREDIS_CONF_OPTS += \

This is the first assignment, so no need to use append-assignment, jsut
use a simple assignemnt.

> +	-DCMAKE_C_COMPILER_LAUNCHER="" \
> +	-DCMAKE_CXX_COMPILER_LAUNCHER="" \

The patch from Dominik and Anders uses -U to "unset" the variables,
while you set them to empty. I think unsetting is semantically more
correct, and I checked: it works.

Also, the following options are not related to being build before
ccache, but the comment makes it confusing, so I prefer we split the
assignment when only parts of it is covered by a coment:

    HOST_HIREDIS_CONF_OPTS = \
        -DDISABLE_TESTS=ON \
        -DENABLE_SSL=OFF

    # Set CMAKE_BUILD_TYPE to Release or the libraries will be suffixed with "d"
    HOST_HIREDIS_CONF_OPTS += -DCMAKE_BUILD_TYPE=Release

    # We are a ccache dependency, so we can't use ccache; reset the
    # options set by the cmake infra
    HOST_HIREDIS_CONF_OPTS += \
        -UCMAKE_C_COMPILER_LAUNCHER \
        -UCMAKE_CXX_COMPILER_LAUNCHER

However, I wonder whether the -DCMAKE_BUILD_TYPE=Release makes sense.
Ideed, we need to set it for the target variant, because we can do a
debug build or not, and we set CMAKE_BUILD_TYPE in toolchainfile.cmake.
But for the host variant, we do not use toolchainfile.cmake, and we
never explicitly pass the build type on the command line either, so we
especially never set it to "Debug", and so the 'd' suffix should not be
added.

Ah, but CMAKE_BUILD_TYPE can come from the environment... Damn.

OK, so I did a few changes, and applied to master, thanks:

  - add Dominik and Anders in Cc to ack they provided inspiration
  - use -Ufoo instead of -Dfoo=""  (as per Dominik & Anders)
  - reorder CONF_OPTS assignments

Regards,
Yann E. MORIN.

> +	-DCMAKE_BUILD_TYPE=Release \
> +	-DDISABLE_TESTS=ON \
> +	-DENABLE_SSL=OFF
> +
>  $(eval $(cmake-package))
> +$(eval $(host-cmake-package))
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot