[PATCH 0/4] ccache.bbclass: Refactor it to make it more reliable

[PATCH 0/4] ccache.bbclass: Refactor it to make it more reliable

[Robert Yang]

Hi RP and Ross,

With this refactor, I think that we can even enable ccache by default, it is
useful for debugging recipes which require a lot of time to compile.

// Robert

The following changes since commit 8bc0d2f2197430723f8b2d0785169e48c883eedb:

  mdadm: Drop redundant patches and fix build with clang (2018-12-09 11:08:07 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/ccache
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ccache

Robert Yang (4):
  base.bbclass: Remove invalid ccache related code
  ccache.bbclass: Refactor it to make it more reliable
  apr/openssl10: Enable ccache for them
  cmake.bbclass/goarch.bbclass: Set CCACHE_DISABLE

 meta/classes/base.bbclass                          |  2 --
 meta/classes/ccache.bbclass                        | 25 +++++++++++++++++++---
 meta/classes/cmake.bbclass                         |  4 ++--
 meta/classes/goarch.bbclass                        |  4 ++++
 meta/conf/bitbake.conf                             | 21 +++++++++++++-----
 meta/conf/layer.conf                               |  1 -
 meta/lib/oe/utils.py                               |  3 +++
 .../openssl/openssl10_1.0.2q.bb                    |  3 ---
 meta/recipes-devtools/ccache/ccache.inc            |  2 ++
 meta/recipes-support/apr/apr_1.6.5.bb              |  1 -
 10 files changed, 49 insertions(+), 17 deletions(-)

-- 
2.10.2

[PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Robert Yang]

The code is in setup_hosttools_dir(), it searches ccache in PATH in reverse
order, I can't figure out what it is used for, maybe it is aimed to use host's
ccache rather than recipe-sysroot-native/usr/bin/ccache, but the later one
isn't in original PATH, so remove the code.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/base.bbclass | 2 --
 1 file changed, 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index e715ffa..36b9c15 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -122,8 +122,6 @@ def setup_hosttools_dir(dest, toolsvar, d, fatal=True):
         desttool = os.path.join(dest, tool)
         if not os.path.exists(desttool):
             srctool = bb.utils.which(path, tool, executable=True)
-            if "ccache" in srctool:
-                srctool = bb.utils.which(path, tool, executable=True, direction=1)
             if srctool:
                 os.symlink(srctool, desttool)
             else:
-- 
2.10.2

[PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

[Robert Yang]

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some
  hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
  build failures (m4-native failed at do_configure, and others will also be
  failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
  ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may
  cause unexpected problems and hard to debug. This is because ccache-native is
  in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
  rebuild target recipes, and then it would use hosttools/ccache, but the
  previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:
  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except
ccache-native's dependencies, host's cache won't be used any more. It is more
reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
can't check whether CCACHE is set or not in anonymous function since it is
always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
 meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
 meta/conf/layer.conf                    |  1 -
 meta/lib/oe/utils.py                    |  3 +++
 meta/recipes-devtools/ccache/ccache.inc |  2 ++
 5 files changed, 43 insertions(+), 9 deletions(-)

diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
index 9609020..f4eefce 100644
--- a/meta/classes/ccache.bbclass
+++ b/meta/classes/ccache.bbclass
@@ -1,4 +1,14 @@
-CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
+#
+# Usage:
+# - Enable ccache
+#   Add the following line to a conffile such as conf/local.conf:
+#   INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+#   Add the following line to the recipe if it can't be built with ccache:
+#   CCACHE_DISABLE = '1'
+#
+
 export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
 
 # We need to stop ccache considering the current directory or the
@@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
 # ${PV} or ${PR} change.
 export CCACHE_NOHASHDIR ?= "1"
 
-DEPENDS_append_class-target = " ccache-native"
-DEPENDS[vardepvalueexclude] = " ccache-native"
+python() {
+    """
+    Enable ccache for the recipe if is not a dependency of ccache-native
+    """
+    ccache_native_deps = d.getVar('CCACHE_NATIVE_DEPENDS') or ''
+    ccache_native_deps += 'ccache-native'
+    if not (d.getVar('PN') in ccache_native_deps.split() or \
+            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
+        d.appendVar('DEPENDS', ' ccache-native')
+        d.setVar('CCACHE', 'ccache ')
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6480062..cb1dba8 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -492,7 +492,7 @@ HOSTTOOLS += " \
 HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
 
 # Link to these if present
-HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
+HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
 
 # Temporary add few more detected in bitbake world
 HOSTTOOLS_NONFATAL += "join nl size yes zcat"
@@ -503,11 +503,22 @@ HOSTTOOLS_NONFATAL += "bzr"
 # Used by ssh fetcher
 HOSTTOOLS_NONFATAL += "scp"
 
+# This is used by both ccache.bbclass and ccahe-native recipe, these
+# recipes can't use ccache since ccache-native depends on them,
+# otherwise, there would be loop dependencies.
+CCACHE_NATIVE_DEPENDS ?= " \
+    gnu-config-native \
+    autoconf-native \
+    quilt-native \
+    zlib-native \
+    gettext-minimal-native \
+    texinfo-dummy-native \
+    m4-native \
+    libtool-native \
+    automake-native \
+    xz-native \
+"
 CCACHE ??= ""
-# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
-# autogen sets HOME=/dev/null so in certain situations builds can fail.
-# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
-export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"
 
 TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
 
diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
index c67dec4..4141d6c 100644
--- a/meta/conf/layer.conf
+++ b/meta/conf/layer.conf
@@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
   *->quilt-native \
   *->subversion-native \
   *->git-native \
-  *->ccache-native \
   *->icecc-create-env-native \
   gcc-cross-${TARGET_ARCH}->virtual/${TARGET_PREFIX}libc-for-gcc \
   gcc-cross-${TARGET_ARCH}->linux-libc-headers \
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
index 8a584d6..9c41705 100644
--- a/meta/lib/oe/utils.py
+++ b/meta/lib/oe/utils.py
@@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):
         return
 
     compiler = d.getVar("BUILD_CC")
+    # Get rid of ccache since it is not present when parsing.
+    if compiler.startswith('ccache '):
+        compiler = compiler[7:]
     try:
         env = os.environ.copy()
         env["PATH"] = d.getVar("PATH")
diff --git a/meta/recipes-devtools/ccache/ccache.inc b/meta/recipes-devtools/ccache/ccache.inc
index 6566328..259c196 100644
--- a/meta/recipes-devtools/ccache/ccache.inc
+++ b/meta/recipes-devtools/ccache/ccache.inc
@@ -9,6 +9,8 @@ LICENSE = "GPLv3+"
 
 DEPENDS = "zlib"
 
+DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
+
 SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz"
 
 inherit autotools
-- 
2.10.2

[PATCH 3/4] apr/openssl10: Enable ccache for them

[Robert Yang]

The work well now.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb | 3 ---
 meta/recipes-support/apr/apr_1.6.5.bb                 | 1 -
 2 files changed, 4 deletions(-)

diff --git a/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb b/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
index 6518dac..1986cf2 100644
--- a/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
+++ b/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
@@ -76,9 +76,6 @@ EXTRA_OEMAKE = "${@bb.utils.contains('PACKAGECONFIG', 'manpages', '', 'OE_DISABL
 
 export OE_LDFLAGS = "${LDFLAGS}"
 
-# openssl fails with ccache: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12810
-CCACHE = ""
-
 TERMIO ?= "-DTERMIO"
 TERMIO_libc-musl = "-DTERMIOS"
 
diff --git a/meta/recipes-support/apr/apr_1.6.5.bb b/meta/recipes-support/apr/apr_1.6.5.bb
index c2f5900..ed93a7e 100644
--- a/meta/recipes-support/apr/apr_1.6.5.bb
+++ b/meta/recipes-support/apr/apr_1.6.5.bb
@@ -40,7 +40,6 @@ CACHED_CONFIGUREVARS += "ac_cv_file__dev_zero=yes"
 # Otherwise libtool fails to compile apr-utils
 # x86_64-linux-libtool: compile: unable to infer tagged configuration
 # x86_64-linux-libtool:   error: specify a tag with '--tag'
-CCACHE = ""
 
 PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'ipv6', d)}"
 PACKAGECONFIG[ipv6] = "--enable-ipv6,--disable-ipv6,"
-- 
2.10.2

[PATCH 4/4] cmake.bbclass/goarch.bbclass: Set CCACHE_DISABLE

[Robert Yang]

They can't be built with ccache.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/cmake.bbclass  | 4 ++--
 meta/classes/goarch.bbclass | 4 ++++
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/cmake.bbclass b/meta/classes/cmake.bbclass
index b364d2b..12fc452 100644
--- a/meta/classes/cmake.bbclass
+++ b/meta/classes/cmake.bbclass
@@ -4,8 +4,8 @@ OECMAKE_SOURCEPATH ??= "${S}"
 DEPENDS_prepend = "cmake-native "
 B = "${WORKDIR}/build"
 
-# We need to unset CCACHE otherwise cmake gets too confused
-CCACHE = ""
+# We need to disable CCACHE otherwise cmake gets too confused
+CCACHE_DISABLE = "1"
 
 # What CMake generator to use.
 # The supported options are "Unix Makefiles" or "Ninja".
diff --git a/meta/classes/goarch.bbclass b/meta/classes/goarch.bbclass
index b2c94fa..1390f31 100644
--- a/meta/classes/goarch.bbclass
+++ b/meta/classes/goarch.bbclass
@@ -42,6 +42,10 @@ TUNE_CCARGS_remove = "-march=mips32r2"
 SECURITY_CFLAGS_mipsarch = "${SECURITY_NOPIE_CFLAGS}"
 SECURITY_NOPIE_CFLAGS ??= ""
 
+# go can't be built with ccache:
+# gcc: fatal error: no input files
+CCACHE_DISABLE = "1"
+
 def go_map_arch(a, d):
     import re
     if re.match('i.86', a):
-- 
2.10.2

Re: [PATCH 3/4] apr/openssl10: Enable ccache for them

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1883 bytes --]

On Sun, Dec 09, 2018 at 06:24:23PM -0800, Robert Yang wrote:
> The work well now.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb | 3 ---
>  meta/recipes-support/apr/apr_1.6.5.bb                 | 1 -
>  2 files changed, 4 deletions(-)
> 
> diff --git a/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb b/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
> index 6518dac..1986cf2 100644
> --- a/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
> +++ b/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
> @@ -76,9 +76,6 @@ EXTRA_OEMAKE = "${@bb.utils.contains('PACKAGECONFIG', 'manpages', '', 'OE_DISABL
>  
>  export OE_LDFLAGS = "${LDFLAGS}"
>  
> -# openssl fails with ccache: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12810
> -CCACHE = ""
> -
>  TERMIO ?= "-DTERMIO"
>  TERMIO_libc-musl = "-DTERMIOS"
>  
> diff --git a/meta/recipes-support/apr/apr_1.6.5.bb b/meta/recipes-support/apr/apr_1.6.5.bb
> index c2f5900..ed93a7e 100644
> --- a/meta/recipes-support/apr/apr_1.6.5.bb
> +++ b/meta/recipes-support/apr/apr_1.6.5.bb
> @@ -40,7 +40,6 @@ CACHED_CONFIGUREVARS += "ac_cv_file__dev_zero=yes"
>  # Otherwise libtool fails to compile apr-utils
>  # x86_64-linux-libtool: compile: unable to infer tagged configuration
>  # x86_64-linux-libtool:   error: specify a tag with '--tag'
> -CCACHE = ""

Remove the comment as well.

>  PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'ipv6', d)}"
>  PACKAGECONFIG[ipv6] = "--enable-ipv6,--disable-ipv6,"
> -- 
> 2.10.2
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

[Martin Hundebøll]

Hi Robert,

Just a small nit-pick below..

On 10/12/2018 03.24, Robert Yang wrote:
> The previous ccache.bbclass has the following problems:
> - It uses host's ccache for native recipes, but this may not work on some
>    hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>    build failures (m4-native failed at do_configure, and others will also be
>    failed if I disable CCACHE for m4-native)
> 
> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>    ccache-native, this may confuse user.
> 
> - The target recipes may use both host's ccache and ccache-native, this may
>    cause unexpected problems and hard to debug. This is because ccache-native is
>    in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>    rebuild target recipes, and then it would use hosttools/ccache, but the
>    previous ccache files were generated by ccache-native.
> 
> - Target recipes can't use ccache when no ccahe is installed on the host:
>    CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
> 
> After refactored:
> All types recipes (native, target and others) will use ccache-native except
> ccache-native's dependencies, host's cache won't be used any more. It is more
> reliable now, which will work everywhere when ccache-native can be built.
> 
> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
> can't check whether CCACHE is set or not in anonymous function since it is
> always set. Use CCACHE_DISABLE to disable it would be more clear.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
>   meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
>   meta/conf/layer.conf                    |  1 -
>   meta/lib/oe/utils.py                    |  3 +++
>   meta/recipes-devtools/ccache/ccache.inc |  2 ++
>   5 files changed, 43 insertions(+), 9 deletions(-)
> 

<snip>

> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
> index 8a584d6..9c41705 100644
> --- a/meta/lib/oe/utils.py
> +++ b/meta/lib/oe/utils.py
> @@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):
>           return
>   
>       compiler = d.getVar("BUILD_CC")
> +    # Get rid of ccache since it is not present when parsing.
> +    if compiler.startswith('ccache '):
> +        compiler = compiler[7:]

You can avoid the if-condition and hard-coded 7-value by doing:

   # Get rid of ccache since it is no present when parsing.
   compiler = d.getVar("BUILD_CC").lstrip("ccache-")

// Martin

>       try:
>           env = os.environ.copy()
>           env["PATH"] = d.getVar("PATH")
> diff --git a/meta/recipes-devtools/ccache/ccache.inc b/meta/recipes-devtools/ccache/ccache.inc
> index 6566328..259c196 100644
> --- a/meta/recipes-devtools/ccache/ccache.inc
> +++ b/meta/recipes-devtools/ccache/ccache.inc
> @@ -9,6 +9,8 @@ LICENSE = "GPLv3+"
>   
>   DEPENDS = "zlib"
>   
> +DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
> +
>   SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz"
>   
>   inherit autotools
> 

-- 
Kind regards,
Martin Hundebøll
Embedded Linux Consultant

+45 61 65 54 61
martin@geanix.com

Geanix IVS
https://geanix.com
DK39600706

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Richard Purdie]

On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
> The code is in setup_hosttools_dir(), it searches ccache in PATH in
> reverse order, I can't figure out what it is used for, maybe it is
> aimed to use host's ccache rather than recipe-sysroot-
> native/usr/bin/ccache, but the later one isn't in original PATH, so
> remove the code.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/classes/base.bbclass | 2 --
>  1 file changed, 2 deletions(-)

We should document it and I think its still needed.

Some distros have a "gcc" which automatically wraps ccache behind the
scenes. This code lets us find the real gcc and bypass ccache.

There would be oe-selftest failures on such a distro if we didn't do
that.

Cheers,

Richard

Re: [PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

[Richard Purdie]

On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
> The previous ccache.bbclass has the following problems:
> - It uses host's ccache for native recipes, but this may not work on some
>   hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>   build failures (m4-native failed at do_configure, and others will also be
>   failed if I disable CCACHE for m4-native)
> 
> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>   ccache-native, this may confuse user.
> 
> - The target recipes may use both host's ccache and ccache-native, this may
>   cause unexpected problems and hard to debug. This is because ccache-native is
>   in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>   rebuild target recipes, and then it would use hosttools/ccache, but the
>   previous ccache files were generated by ccache-native.
> 
> - Target recipes can't use ccache when no ccahe is installed on the host:
>   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
> 
> After refactored:
> All types recipes (native, target and others) will use ccache-native except
> ccache-native's dependencies, host's cache won't be used any more. It is more
> reliable now, which will work everywhere when ccache-native can be built.
> 
> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
> can't check whether CCACHE is set or not in anonymous function since it is
> always set. Use CCACHE_DISABLE to disable it would be more clear.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
>  meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
>  meta/conf/layer.conf                    |  1 -
>  meta/lib/oe/utils.py                    |  3 +++
>  meta/recipes-devtools/ccache/ccache.inc |  2 ++
>  5 files changed, 43 insertions(+), 9 deletions(-)
> 
> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
> index 9609020..f4eefce 100644
> --- a/meta/classes/ccache.bbclass
> +++ b/meta/classes/ccache.bbclass
> @@ -1,4 +1,14 @@
> -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
> +#
> +# Usage:
> +# - Enable ccache
> +#   Add the following line to a conffile such as conf/local.conf:
> +#   INHERIT += "ccache"
> +#
> +# - Disable ccache for a recipe
> +#   Add the following line to the recipe if it can't be built with ccache:
> +#   CCACHE_DISABLE = '1'
> +#
> +
>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>  
>  # We need to stop ccache considering the current directory or the
> @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>  # ${PV} or ${PR} change.
>  export CCACHE_NOHASHDIR ?= "1"
>  
> -DEPENDS_append_class-target = " ccache-native"
> -DEPENDS[vardepvalueexclude] = " ccache-native"
> +python() {
> +    """
> +    Enable ccache for the recipe if is not a dependency of ccache-native
> +    """
> +    ccache_native_deps = d.getVar('CCACHE_NATIVE_DEPENDS') or ''
> +    ccache_native_deps += 'ccache-native'
> +    if not (d.getVar('PN') in ccache_native_deps.split() or \
> +            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
> +        d.appendVar('DEPENDS', ' ccache-native')
> +        d.setVar('CCACHE', 'ccache ')
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 6480062..cb1dba8 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -492,7 +492,7 @@ HOSTTOOLS += " \
>  HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
>  
>  # Link to these if present
> -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
> +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
>  
>  # Temporary add few more detected in bitbake world
>  HOSTTOOLS_NONFATAL += "join nl size yes zcat"
> @@ -503,11 +503,22 @@ HOSTTOOLS_NONFATAL += "bzr"
>  # Used by ssh fetcher
>  HOSTTOOLS_NONFATAL += "scp"
>  
> +# This is used by both ccache.bbclass and ccahe-native recipe, these
> +# recipes can't use ccache since ccache-native depends on them,
> +# otherwise, there would be loop dependencies.
> +CCACHE_NATIVE_DEPENDS ?= " \
> +    gnu-config-native \
> +    autoconf-native \
> +    quilt-native \
> +    zlib-native \
> +    gettext-minimal-native \
> +    texinfo-dummy-native \
> +    m4-native \
> +    libtool-native \
> +    automake-native \
> +    xz-native \
> +"
>  CCACHE ??= ""
> -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
> -# autogen sets HOME=/dev/null so in certain situations builds can fail.
> -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
> -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"

Hardcoding a list of ccache-native dependencies is horrible and I do
not want to do that. 

Also, I want to stop the build up of unneeded 'stuff' in bitbake.conf.
We should start making better use of inc files.

Please also ensure that the oe-selftest ccache tests continue to work
after these changes. I haven't checked specifically here but we've had
a lot of trouble with that before.

Cheers,

Richard
 

Re: [PATCH 3/4] apr/openssl10: Enable ccache for them

[Robert Yang]

On 12/10/18 4:38 PM, Martin Jansa wrote:
> On Sun, Dec 09, 2018 at 06:24:23PM -0800, Robert Yang wrote:
>> The work well now.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb | 3 ---
>>   meta/recipes-support/apr/apr_1.6.5.bb                 | 1 -
>>   2 files changed, 4 deletions(-)
>>
>> diff --git a/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb b/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
>> index 6518dac..1986cf2 100644
>> --- a/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
>> +++ b/meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb
>> @@ -76,9 +76,6 @@ EXTRA_OEMAKE = "${@bb.utils.contains('PACKAGECONFIG', 'manpages', '', 'OE_DISABL
>>   
>>   export OE_LDFLAGS = "${LDFLAGS}"
>>   
>> -# openssl fails with ccache: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12810
>> -CCACHE = ""
>> -
>>   TERMIO ?= "-DTERMIO"
>>   TERMIO_libc-musl = "-DTERMIOS"
>>   
>> diff --git a/meta/recipes-support/apr/apr_1.6.5.bb b/meta/recipes-support/apr/apr_1.6.5.bb
>> index c2f5900..ed93a7e 100644
>> --- a/meta/recipes-support/apr/apr_1.6.5.bb
>> +++ b/meta/recipes-support/apr/apr_1.6.5.bb
>> @@ -40,7 +40,6 @@ CACHED_CONFIGUREVARS += "ac_cv_file__dev_zero=yes"
>>   # Otherwise libtool fails to compile apr-utils
>>   # x86_64-linux-libtool: compile: unable to infer tagged configuration
>>   # x86_64-linux-libtool:   error: specify a tag with '--tag'
>> -CCACHE = ""
> 
> Remove the comment as well.

Thanks, will fix it in V2.

// Robert

> 
>>   PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'ipv6', d)}"
>>   PACKAGECONFIG[ipv6] = "--enable-ipv6,--disable-ipv6,"
>> -- 
>> 2.10.2
>>
>> -- 
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 

Re: [PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

[Robert Yang]

On 12/10/18 4:55 PM, Martin Hundebøll wrote:
> Hi Robert,
> 
> Just a small nit-pick below..
> 
> On 10/12/2018 03.24, Robert Yang wrote:
>> The previous ccache.bbclass has the following problems:
>> - It uses host's ccache for native recipes, but this may not work on some
>>    hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>>    build failures (m4-native failed at do_configure, and others will also be
>>    failed if I disable CCACHE for m4-native)
>>
>> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>>    ccache-native, this may confuse user.
>>
>> - The target recipes may use both host's ccache and ccache-native, this may
>>    cause unexpected problems and hard to debug. This is because ccache-native is
>>    in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>>    rebuild target recipes, and then it would use hosttools/ccache, but the
>>    previous ccache files were generated by ccache-native.
>>
>> - Target recipes can't use ccache when no ccahe is installed on the host:
>>    CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>
>> After refactored:
>> All types recipes (native, target and others) will use ccache-native except
>> ccache-native's dependencies, host's cache won't be used any more. It is more
>> reliable now, which will work everywhere when ccache-native can be built.
>>
>> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
>> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
>> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
>> can't check whether CCACHE is set or not in anonymous function since it is
>> always set. Use CCACHE_DISABLE to disable it would be more clear.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
>>   meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
>>   meta/conf/layer.conf                    |  1 -
>>   meta/lib/oe/utils.py                    |  3 +++
>>   meta/recipes-devtools/ccache/ccache.inc |  2 ++
>>   5 files changed, 43 insertions(+), 9 deletions(-)
>>
> 
> <snip>
> 
>> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
>> index 8a584d6..9c41705 100644
>> --- a/meta/lib/oe/utils.py
>> +++ b/meta/lib/oe/utils.py
>> @@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):
>>           return
>>       compiler = d.getVar("BUILD_CC")
>> +    # Get rid of ccache since it is not present when parsing.
>> +    if compiler.startswith('ccache '):
>> +        compiler = compiler[7:]
> 
> You can avoid the if-condition and hard-coded 7-value by doing:
> 
>    # Get rid of ccache since it is no present when parsing.
>    compiler = d.getVar("BUILD_CC").lstrip("ccache-")

Thanks, will fix it in V2.

// Robert

> 
> // Martin
> 
>>       try:
>>           env = os.environ.copy()
>>           env["PATH"] = d.getVar("PATH")
>> diff --git a/meta/recipes-devtools/ccache/ccache.inc 
>> b/meta/recipes-devtools/ccache/ccache.inc
>> index 6566328..259c196 100644
>> --- a/meta/recipes-devtools/ccache/ccache.inc
>> +++ b/meta/recipes-devtools/ccache/ccache.inc
>> @@ -9,6 +9,8 @@ LICENSE = "GPLv3+"
>>   DEPENDS = "zlib"
>> +DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
>> +
>>   SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz"
>>   inherit autotools
>>
> 

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Robert Yang]

On 12/10/18 6:29 PM, Richard Purdie wrote:
> On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
>> The code is in setup_hosttools_dir(), it searches ccache in PATH in
>> reverse order, I can't figure out what it is used for, maybe it is
>> aimed to use host's ccache rather than recipe-sysroot-
>> native/usr/bin/ccache, but the later one isn't in original PATH, so
>> remove the code.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/classes/base.bbclass | 2 --
>>   1 file changed, 2 deletions(-)
> 
> We should document it and I think its still needed.
> 
> Some distros have a "gcc" which automatically wraps ccache behind the
> scenes. This code lets us find the real gcc and bypass ccache.

Ah, thanks for the explanations, I got it now. I think that we don't need the
code any more since we don't use host's ccache, and I have removed ccache from
HOSTTOOLS.

I will update commit message and send a V2.

// Robert

> 
> There would be oe-selftest failures on such a distro if we didn't do
> that.
> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

[Robert Yang]

On 12/11/18 12:16 AM, Richard Purdie wrote:
> On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
>> The previous ccache.bbclass has the following problems:
>> - It uses host's ccache for native recipes, but this may not work on some
>>    hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>>    build failures (m4-native failed at do_configure, and others will also be
>>    failed if I disable CCACHE for m4-native)
>>
>> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>>    ccache-native, this may confuse user.
>>
>> - The target recipes may use both host's ccache and ccache-native, this may
>>    cause unexpected problems and hard to debug. This is because ccache-native is
>>    in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>>    rebuild target recipes, and then it would use hosttools/ccache, but the
>>    previous ccache files were generated by ccache-native.
>>
>> - Target recipes can't use ccache when no ccahe is installed on the host:
>>    CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>
>> After refactored:
>> All types recipes (native, target and others) will use ccache-native except
>> ccache-native's dependencies, host's cache won't be used any more. It is more
>> reliable now, which will work everywhere when ccache-native can be built.
>>
>> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
>> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
>> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
>> can't check whether CCACHE is set or not in anonymous function since it is
>> always set. Use CCACHE_DISABLE to disable it would be more clear.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
>>   meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
>>   meta/conf/layer.conf                    |  1 -
>>   meta/lib/oe/utils.py                    |  3 +++
>>   meta/recipes-devtools/ccache/ccache.inc |  2 ++
>>   5 files changed, 43 insertions(+), 9 deletions(-)
>>
>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>> index 9609020..f4eefce 100644
>> --- a/meta/classes/ccache.bbclass
>> +++ b/meta/classes/ccache.bbclass
>> @@ -1,4 +1,14 @@
>> -CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>> +#
>> +# Usage:
>> +# - Enable ccache
>> +#   Add the following line to a conffile such as conf/local.conf:
>> +#   INHERIT += "ccache"
>> +#
>> +# - Disable ccache for a recipe
>> +#   Add the following line to the recipe if it can't be built with ccache:
>> +#   CCACHE_DISABLE = '1'
>> +#
>> +
>>   export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>>   
>>   # We need to stop ccache considering the current directory or the
>> @@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
>>   # ${PV} or ${PR} change.
>>   export CCACHE_NOHASHDIR ?= "1"
>>   
>> -DEPENDS_append_class-target = " ccache-native"
>> -DEPENDS[vardepvalueexclude] = " ccache-native"
>> +python() {
>> +    """
>> +    Enable ccache for the recipe if is not a dependency of ccache-native
>> +    """
>> +    ccache_native_deps = d.getVar('CCACHE_NATIVE_DEPENDS') or ''
>> +    ccache_native_deps += 'ccache-native'
>> +    if not (d.getVar('PN') in ccache_native_deps.split() or \
>> +            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
>> +        d.appendVar('DEPENDS', ' ccache-native')
>> +        d.setVar('CCACHE', 'ccache ')
>> +}
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index 6480062..cb1dba8 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -492,7 +492,7 @@ HOSTTOOLS += " \
>>   HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
>>   
>>   # Link to these if present
>> -HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
>> +HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
>>   
>>   # Temporary add few more detected in bitbake world
>>   HOSTTOOLS_NONFATAL += "join nl size yes zcat"
>> @@ -503,11 +503,22 @@ HOSTTOOLS_NONFATAL += "bzr"
>>   # Used by ssh fetcher
>>   HOSTTOOLS_NONFATAL += "scp"
>>   
>> +# This is used by both ccache.bbclass and ccahe-native recipe, these
>> +# recipes can't use ccache since ccache-native depends on them,
>> +# otherwise, there would be loop dependencies.
>> +CCACHE_NATIVE_DEPENDS ?= " \
>> +    gnu-config-native \
>> +    autoconf-native \
>> +    quilt-native \
>> +    zlib-native \
>> +    gettext-minimal-native \
>> +    texinfo-dummy-native \
>> +    m4-native \
>> +    libtool-native \
>> +    automake-native \
>> +    xz-native \
>> +"
>>   CCACHE ??= ""
>> -# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
>> -# autogen sets HOME=/dev/null so in certain situations builds can fail.
>> -# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
>> -export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"
> 
> Hardcoding a list of ccache-native dependencies is horrible and I do
> not want to do that.


Yes, I don't like this either:-). I've fixed it as:

* Remove all ccache-native's dependencies.
   ccache-native has two dependencies: zlib-native and autotools-native, we can
   use --with-bundled-zlib do get rid of zlib-native. And use
   INHIBIT_AUTOTOOLS_DEPS to get rid of autotools-native, and add a simple
   do_configure function:

do_configure_class-native() {
     oe_runconf
}

So that ccache-native can be depended by any recipe except quilt-native:

So now the code in ccache.bbclass is very simple:
python() {
     """
     Enable ccache for the recipe
     """
     pn = d.getVar('PN')
     if not pn in ('quilt-native', 'ccache-native'):
             d.appendVar('DEPENDS', ' ccache-native')
             d.setVar('CCACHE', 'ccache ')
}


We can also set PATCHTOOL = "patch" in ccache to make quilt-native can use
ccache too, but I don't think that we have too.

I will send a V2 after more testing.

// Robert


> 
> Also, I want to stop the build up of unneeded 'stuff' in bitbake.conf.
> We should start making better use of inc files.
> 
> Please also ensure that the oe-selftest ccache tests continue to work
> after these changes. I haven't checked specifically here but we've had
> a lot of trouble with that before.
> 
> Cheers,
> 
> Richard
>   
> 
> 

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[richard.purdie]

On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:
> 
> On 12/10/18 6:29 PM, Richard Purdie wrote:
> > On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
> > > The code is in setup_hosttools_dir(), it searches ccache in PATH
> > > in
> > > reverse order, I can't figure out what it is used for, maybe it
> > > is
> > > aimed to use host's ccache rather than recipe-sysroot-
> > > native/usr/bin/ccache, but the later one isn't in original PATH,
> > > so
> > > remove the code.
> > > 
> > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> > > ---
> > >   meta/classes/base.bbclass | 2 --
> > >   1 file changed, 2 deletions(-)
> > 
> > We should document it and I think its still needed.
> > 
> > Some distros have a "gcc" which automatically wraps ccache behind
> > the
> > scenes. This code lets us find the real gcc and bypass ccache.
> 
> Ah, thanks for the explanations, I got it now. I think that we don't
> need the
> code any more since we don't use host's ccache, and I have removed
> ccache from
> HOSTTOOLS.
> 
> I will update commit message and send a V2.

No, that isn't correct :(

The problem is that "gcc" can be linked directly to ccache by the host
distro. This means you end up using the host ccache just by calling
"gcc". This code is still needed if we want to control ccache and be
able to disable it and use ccache-native for determinism.

Cheers,

Richard

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Robert Yang]

Hi RP,

On 12/11/18 5:44 PM, richard.purdie@linuxfoundation.org wrote:
> On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:
>>
>> On 12/10/18 6:29 PM, Richard Purdie wrote:
>>> On Sun, 2018-12-09 at 18:24 -0800, Robert Yang wrote:
>>>> The code is in setup_hosttools_dir(), it searches ccache in PATH
>>>> in
>>>> reverse order, I can't figure out what it is used for, maybe it
>>>> is
>>>> aimed to use host's ccache rather than recipe-sysroot-
>>>> native/usr/bin/ccache, but the later one isn't in original PATH,
>>>> so
>>>> remove the code.
>>>>
>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>> ---
>>>>    meta/classes/base.bbclass | 2 --
>>>>    1 file changed, 2 deletions(-)
>>>
>>> We should document it and I think its still needed.
>>>
>>> Some distros have a "gcc" which automatically wraps ccache behind
>>> the
>>> scenes. This code lets us find the real gcc and bypass ccache.
>>
>> Ah, thanks for the explanations, I got it now. I think that we don't
>> need the
>> code any more since we don't use host's ccache, and I have removed
>> ccache from
>> HOSTTOOLS.
>>
>> I will update commit message and send a V2.
> 
> No, that isn't correct :(
> 
> The problem is that "gcc" can be linked directly to ccache by the host
> distro. This means you end up using the host ccache just by calling
> "gcc". This code is still needed if we want to control ccache and be
> able to disable it and use ccache-native for determinism.

I did a test:

$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
I got errors with/without the code:

$ bitbake -p
bb.BBHandledException
ERROR: Error running gcc  --version:

$ ls -l tmp/hosttools/gcc
tmp/hosttools/gcc -> /folk/lyang1/bin/gcc

$ ls -l /folk/lyang1/bin/gcc
/folk/lyang1/bin/gcc -> /usr/bin/ccache

It worked well if I run gcc --version manually:
$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
$ echo $?
0

There must be something wrong in such a case. I will fix this later.

Let's get back to he code:

             srctool = bb.utils.which(path, tool, executable=True)
             if "ccache" in srctool:
                 srctool = bb.utils.which(path, tool, executable=True, direction=1)
             if srctool:
                 os.symlink(srctool, desttool)


* If tool == "gcc", then srctool would be gcc, too.
* If tool == "ccache", then srctool would be ccache, too.

So I think that the code doesn't help when gcc links to ccache ?

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 
> 

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[richard.purdie]

On Tue, 2018-12-11 at 18:12 +0800, Robert Yang wrote:
> On 12/11/18 5:44 PM, richard.purdie@linuxfoundation.org wrote:
> > On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:
> > > On 12/10/18 6:29 PM, Richard Purdie wrote:
> > No, that isn't correct :(
> > 
> > The problem is that "gcc" can be linked directly to ccache by the
> > host
> > distro. This means you end up using the host ccache just by calling
> > "gcc". This code is still needed if we want to control ccache and
> > be
> > able to disable it and use ccache-native for determinism.
> 
> I did a test:
> 
> $ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
> I got errors with/without the code:
> 
> $ bitbake -p
> bb.BBHandledException
> ERROR: Error running gcc  --version:
> 
> $ ls -l tmp/hosttools/gcc
> tmp/hosttools/gcc -> /folk/lyang1/bin/gcc
> 
> $ ls -l /folk/lyang1/bin/gcc
> /folk/lyang1/bin/gcc -> /usr/bin/ccache
> 
> It worked well if I run gcc --version manually:
> $ gcc --version
> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
> Copyright (C) 2015 Free Software Foundation, Inc.
> $ echo $?
> 0
> 
> There must be something wrong in such a case. I will fix this later.
> 
> Let's get back to he code:
> 
>              srctool = bb.utils.which(path, tool, executable=True)
>              if "ccache" in srctool:
>                  srctool = bb.utils.which(path, tool,
> executable=True, direction=1)
>              if srctool:
>                  os.symlink(srctool, desttool)
> 
> 
> * If tool == "gcc", then srctool would be gcc, too.
> * If tool == "ccache", then srctool would be ccache, too.
> 
> So I think that the code doesn't help when gcc links to ccache ?

https://www.unix.com/man-page/suse/1/ccache/

When the distros we tested on did this they placed it in something like
bin/ccache/ so the code detects the "ccache" element of the path
returned by which() (something like /usr/bin/ccache/gcc but I don't
remember exactly).

Try using /folk/lyang1/ccache/ or /folk/lyang1/bin/ccache/ in your
example.

Cheers,

Richard

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Yang, Liezhi]

Sent from mobile phone

> 在 2018年12月11日，18:43，"richard.purdie@linuxfoundation.org" <richard.purdie@linuxfoundation.org> 写道：
> 
>> On Tue, 2018-12-11 at 18:12 +0800, Robert Yang wrote:
>>> On 12/11/18 5:44 PM, richard.purdie@linuxfoundation.org wrote:
>>>> On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:
>>>> On 12/10/18 6:29 PM, Richard Purdie wrote:
>>> No, that isn't correct :(
>>> 
>>> The problem is that "gcc" can be linked directly to ccache by the
>>> host
>>> distro. This means you end up using the host ccache just by calling
>>> "gcc". This code is still needed if we want to control ccache and
>>> be
>>> able to disable it and use ccache-native for determinism.
>> 
>> I did a test:
>> 
>> $ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
>> I got errors with/without the code:
>> 
>> $ bitbake -p
>> bb.BBHandledException
>> ERROR: Error running gcc  --version:
>> 
>> $ ls -l tmp/hosttools/gcc
>> tmp/hosttools/gcc -> /folk/lyang1/bin/gcc
>> 
>> $ ls -l /folk/lyang1/bin/gcc
>> /folk/lyang1/bin/gcc -> /usr/bin/ccache
>> 
>> It worked well if I run gcc --version manually:
>> $ gcc --version
>> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
>> Copyright (C) 2015 Free Software Foundation, Inc.
>> $ echo $?
>> 0
>> 
>> There must be something wrong in such a case. I will fix this later.
>> 
>> Let's get back to he code:
>> 
>>             srctool = bb.utils.which(path, tool, executable=True)
>>             if "ccache" in srctool:
>>                 srctool = bb.utils.which(path, tool,
>> executable=True, direction=1)
>>             if srctool:
>>                 os.symlink(srctool, desttool)
>> 
>> 
>> * If tool == "gcc", then srctool would be gcc, too.
>> * If tool == "ccache", then srctool would be ccache, too.
>> 
>> So I think that the code doesn't help when gcc links to ccache ?
> 
> https://www.unix.com/man-page/suse/1/ccache/
> 
> When the distros we tested on did this they placed it in something like
> bin/ccache/ so the code detects the "ccache" element of the path
> returned by which() (something like /usr/bin/ccache/gcc but I don't
> remember exactly).
> 
> Try using /folk/lyang1/ccache/ or /folk/lyang1/bin/ccache/ in your
> example.

Thanks, I will try it tomorrow.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 
> 

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Robert Yang]

On 12/11/18 6:42 PM, richard.purdie@linuxfoundation.org wrote:
> On Tue, 2018-12-11 at 18:12 +0800, Robert Yang wrote:
>> On 12/11/18 5:44 PM, richard.purdie@linuxfoundation.org wrote:
>>> On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:
>>>> On 12/10/18 6:29 PM, Richard Purdie wrote:
>>> No, that isn't correct :(
>>>
>>> The problem is that "gcc" can be linked directly to ccache by the
>>> host
>>> distro. This means you end up using the host ccache just by calling
>>> "gcc". This code is still needed if we want to control ccache and
>>> be
>>> able to disable it and use ccache-native for determinism.
>>
>> I did a test:
>>
>> $ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
>> I got errors with/without the code:
>>
>> $ bitbake -p
>> bb.BBHandledException
>> ERROR: Error running gcc  --version:
>>
>> $ ls -l tmp/hosttools/gcc
>> tmp/hosttools/gcc -> /folk/lyang1/bin/gcc
>>
>> $ ls -l /folk/lyang1/bin/gcc
>> /folk/lyang1/bin/gcc -> /usr/bin/ccache
>>
>> It worked well if I run gcc --version manually:
>> $ gcc --version
>> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
>> Copyright (C) 2015 Free Software Foundation, Inc.
>> $ echo $?
>> 0
>>
>> There must be something wrong in such a case. I will fix this later.
>>
>> Let's get back to he code:
>>
>>               srctool = bb.utils.which(path, tool, executable=True)
>>               if "ccache" in srctool:
>>                   srctool = bb.utils.which(path, tool,
>> executable=True, direction=1)
>>               if srctool:
>>                   os.symlink(srctool, desttool)
>>
>>
>> * If tool == "gcc", then srctool would be gcc, too.
>> * If tool == "ccache", then srctool would be ccache, too.
>>
>> So I think that the code doesn't help when gcc links to ccache ?
> 
> https://www.unix.com/man-page/suse/1/ccache/
> 
> When the distros we tested on did this they placed it in something like
> bin/ccache/ so the code detects the "ccache" element of the path
> returned by which() (something like /usr/bin/ccache/gcc but I don't
> remember exactly).
> 
> Try using /folk/lyang1/ccache/ or /folk/lyang1/bin/ccache/ in your
> example.

Ah, sorry, I mixed 'ccache in srctool' with 'ccache == srctool', yes, we do
need it. I will keep the original code, and add comments for it.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 
> 
> 

Re: [PATCH 2/4] ccache.bbclass: Refactor it to make it more reliable

[Robert Yang]

Hi Martin

On 12/10/18 4:55 PM, Martin Hundebøll wrote:
> Hi Robert,
> 
> Just a small nit-pick below..
> 
> On 10/12/2018 03.24, Robert Yang wrote:
>> The previous ccache.bbclass has the following problems:
>> - It uses host's ccache for native recipes, but this may not work on some
>>    hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always
>>    build failures (m4-native failed at do_configure, and others will also be
>>    failed if I disable CCACHE for m4-native)
>>
>> - native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses
>>    ccache-native, this may confuse user.
>>
>> - The target recipes may use both host's ccache and ccache-native, this may
>>    cause unexpected problems and hard to debug. This is because ccache-native is
>>    in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when
>>    rebuild target recipes, and then it would use hosttools/ccache, but the
>>    previous ccache files were generated by ccache-native.
>>
>> - Target recipes can't use ccache when no ccahe is installed on the host:
>>    CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>
>> After refactored:
>> All types recipes (native, target and others) will use ccache-native except
>> ccache-native's dependencies, host's cache won't be used any more. It is more
>> reliable now, which will work everywhere when ccache-native can be built.
>>
>> And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe
>> rather than "CCACHE = ''" since we set CCACHE in anonymous function, and
>> d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we
>> can't check whether CCACHE is set or not in anonymous function since it is
>> always set. Use CCACHE_DISABLE to disable it would be more clear.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
>>   meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
>>   meta/conf/layer.conf                    |  1 -
>>   meta/lib/oe/utils.py                    |  3 +++
>>   meta/recipes-devtools/ccache/ccache.inc |  2 ++
>>   5 files changed, 43 insertions(+), 9 deletions(-)
>>
> 
> <snip>
> 
>> diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py
>> index 8a584d6..9c41705 100644
>> --- a/meta/lib/oe/utils.py
>> +++ b/meta/lib/oe/utils.py
>> @@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):
>>           return
>>       compiler = d.getVar("BUILD_CC")
>> +    # Get rid of ccache since it is not present when parsing.
>> +    if compiler.startswith('ccache '):
>> +        compiler = compiler[7:]
> 
> You can avoid the if-condition and hard-coded 7-value by doing:
> 
>    # Get rid of ccache since it is no present when parsing.
>    compiler = d.getVar("BUILD_CC").lstrip("ccache-")

I'm afraid that this may not work, lstrip("ccache-") strips any characters
of 'ccache-' (c, a, h e -), which seems dangerous, and not what we want.
I've will revert it in V2's PULL.

// Robert

> 
> // Martin
> 
>>       try:
>>           env = os.environ.copy()
>>           env["PATH"] = d.getVar("PATH")
>> diff --git a/meta/recipes-devtools/ccache/ccache.inc 
>> b/meta/recipes-devtools/ccache/ccache.inc
>> index 6566328..259c196 100644
>> --- a/meta/recipes-devtools/ccache/ccache.inc
>> +++ b/meta/recipes-devtools/ccache/ccache.inc
>> @@ -9,6 +9,8 @@ LICENSE = "GPLv3+"
>>   DEPENDS = "zlib"
>> +DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
>> +
>>   SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz"
>>   inherit autotools
>>
> 

Re: [PATCH 1/4] base.bbclass: Remove invalid ccache related code

[Robert Yang]

On 12/11/18 6:12 PM, Robert Yang wrote:
> Hi RP,
> 
> On 12/11/18 5:44 PM, richard.purdie@linuxfoundation.org wrote:
>> On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:

[snip]

> I did a test:
> 
> $ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
> I got errors with/without the code:
> 
> $ bitbake -p
> bb.BBHandledException
> ERROR: Error running gcc  --version:

I've sent a patch to fix/explain this problem:

[OE-core] [PATCH 0/1] lib/oe/utils: Set stderr for host_gcc_version()


// Robert

> 
> $ ls -l tmp/hosttools/gcc
> tmp/hosttools/gcc -> /folk/lyang1/bin/gcc
> 
> $ ls -l /folk/lyang1/bin/gcc
> /folk/lyang1/bin/gcc -> /usr/bin/ccache
> 
> It worked well if I run gcc --version manually:
> $ gcc --version
> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
> Copyright (C) 2015 Free Software Foundation, Inc.
> $ echo $?
> 0
> 
> There must be something wrong in such a case. I will fix this later.
> 
> Let's get back to he code:
> 
>              srctool = bb.utils.which(path, tool, executable=True)
>              if "ccache" in srctool:
>                  srctool = bb.utils.which(path, tool, executable=True, direction=1)
>              if srctool:
>                  os.symlink(srctool, desttool)
> 
> 
> * If tool == "gcc", then srctool would be gcc, too.
> * If tool == "ccache", then srctool would be ccache, too.
> 
> So I think that the code doesn't help when gcc links to ccache ?
> 
> // Robert
> 
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>