[PATCH 1/5] allarch: Append to vardepsexclude, not overwrite

[PATCH 1/5] allarch: Append to vardepsexclude, not overwrite

[Richard Purdie]

These have values set elsewhere and this code was overwriting them leading
to odd signature issues. Append instead preserving the original values.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/allarch.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/allarch.bbclass b/meta/classes/allarch.bbclass
index a7ce024..51ba509c 100644
--- a/meta/classes/allarch.bbclass
+++ b/meta/classes/allarch.bbclass
@@ -43,8 +43,8 @@ python () {
         d.setVar("INHIBIT_PACKAGE_STRIP", "1")
 
         # These multilib values shouldn't change allarch packages so exclude them
-        d.setVarFlag("emit_pkgdata", "vardepsexclude", "MULTILIB_VARIANTS")
-        d.setVarFlag("write_specfile", "vardepsexclude", "MULTILIBS")
+        d.appendVarFlag("emit_pkgdata", "vardepsexclude", " MULTILIB_VARIANTS")
+        d.appendVarFlag("write_specfile", "vardepsexclude", " MULTILIBS")
     elif bb.data.inherits_class('packagegroup', d) and not bb.data.inherits_class('nativesdk', d):
         bb.error("Please ensure recipe %s sets PACKAGE_ARCH before inherit packagegroup" % d.getVar("FILE"))
 }
-- 
2.7.4

[PATCH 2/5] insane: Don't depend on OVERRIDES

[Richard Purdie]

In common with the other package handling functions, don't depend on the
value of OVERRIDES. This means when we change MACHINE, we don't have to
repackage everything.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/insane.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index fa932ef..f6fd7c7 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -973,6 +973,7 @@ def package_qa_check_rdepends(pkg, pkgdest, skip, taskdeps, packages, d):
                     error_msg = "%s contained in package %s requires %s, but no providers found in RDEPENDS_%s?" % \
                             (filerdepends[key].replace("_%s" % pkg, "").replace("@underscore@", "_"), pkg, key, pkg)
                     package_qa_handle_error("file-rdeps", error_msg, d)
+package_qa_check_rdepends[vardepsexclude] = "OVERRIDES"
 
 def package_qa_check_deps(pkg, pkgdest, skip, d):
 
-- 
2.7.4

[PATCH 3/5] utils: Exclude OVERRIDES from hashes in multilib functions

[Richard Purdie]

---
 meta/classes/utils.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/utils.bbclass b/meta/classes/utils.bbclass
index 96463ab..081e662 100644
--- a/meta/classes/utils.bbclass
+++ b/meta/classes/utils.bbclass
@@ -369,6 +369,7 @@ def get_multilib_datastore(variant, d):
     localdata.setVar("OVERRIDES", overrides)
     localdata.setVar("MLPREFIX", variant + "-")
     return localdata
+get_multilib_datastore[vardepsexclude] = "OVERRIDES"
 
 def all_multilib_tune_values(d, var, unique = True, need_split = True, delim = ' '):
     """Return a string of all ${var} in all multilib tune configuration"""
@@ -431,6 +432,7 @@ def all_multilib_tune_list(vars, d):
         values[v].append(localdata.getVar(v))
         values['ml'].append(item)
     return values
+all_multilib_tune_list[vardepsexclude] = "OVERRIDES"
 
 # If the user hasn't set up their name/email, set some defaults
 check_git_config() {
-- 
2.7.4

[PATCH 4/5] sstatetests: Use higher parallelism value

[Richard Purdie]

Since the processing code for signature generation is now threaded,
use higher thread values as examples in this code for better performance.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oeqa/selftest/cases/sstatetests.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
index 8d05e30..feadf3c 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -418,7 +418,7 @@ DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
 
         self.write_config("""
 TMPDIR = "${TOPDIR}/tmp-sstatesamehash"
-BB_NUMBER_THREADS = "1"
+BB_NUMBER_THREADS = "${@oe.utils.cpu_count()}"
 PARALLEL_MAKE = "-j 1"
 DL_DIR = "${TOPDIR}/download1"
 TIME = "111111"
@@ -431,7 +431,7 @@ http_proxy = ""
         bitbake("world meta-toolchain -S none")
         self.write_config("""
 TMPDIR = "${TOPDIR}/tmp-sstatesamehash2"
-BB_NUMBER_THREADS = "2"
+BB_NUMBER_THREADS = "${@oe.utils.cpu_count()-1}"
 PARALLEL_MAKE = "-j 2"
 DL_DIR = "${TOPDIR}/download2"
 TIME = "222222"
-- 
2.7.4

[PATCH 5/5] bitbake.conf: Don't exclude MACHINE/MACHINEOVERRIDES from hashes

[Richard Purdie]

A long time ago (6 years), this seemed like a good idea. The reality is
that OVERRIDES should not be being added to hashes and if it is, it likely
needs excluding in its own right. This was a nice workaround but we need
to fix the real underlying issues now. In some cases this means excluding
OVERRIDES from the variables dependency using the vardepsexclude flag however
caution is needed to ensure this is safe.

Variable values used to construct hashes are unexpanded but the values used
are computed after the application of OVERRIDES. The important detail is if
the end resulting unexpanded value changes, not the value of the OVERRIDES
used in the construction of that unexpanded value. This is why dependencies
on OVERRIDES itself shouldn't be in the hashes in general.

The recent DISTRO_FEATURES changes adding in override mappings for them
highlighted this issue. We have some good sstate tests which are effective
at highlighting where potential issues arrive with OVERRIDES contamination
(oe-selftest -r sstatetests.SStateTests).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/bitbake.conf | 2 --
 1 file changed, 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index bc438cc..5d5ddec 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -714,11 +714,9 @@ DISTRO_NAME ??= "OpenEmbedded"
 # This works for  functions as well, they are really just environment variables.
 # Default OVERRIDES to make compilation fail fast in case of build system misconfiguration.
 OVERRIDES = "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:build-${BUILD_OS}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}${DISTROFEATURESOVERRIDES}:${CLASSOVERRIDE}:forcevariable"
-OVERRIDES[vardepsexclude] = "MACHINEOVERRIDES"
 CLASSOVERRIDE ?= "class-target"
 DISTROOVERRIDES ?= "${@d.getVar('DISTRO') or ''}"
 MACHINEOVERRIDES ?= "${MACHINE}"
-MACHINEOVERRIDES[vardepsexclude] = "MACHINE"
 
 FILESOVERRIDES = "${TRANSLATED_TARGET_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}"
 
-- 
2.7.4

✗ patchtest: failure for "allarch: Append to vardepsexcl..." and 4 more

[Patchwork]

== Series Details ==

Series: "allarch: Append to vardepsexcl..." and 4 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/7211/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [3/5] utils: Exclude OVERRIDES from hashes in multilib functions
 Issue             Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fix    Sign off the patch (either manually or with "git commit --amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH 4/5] sstatetests: Use higher parallelism value

[Andre McCurdy]

On Tue, Jun 13, 2017 at 2:49 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Since the processing code for signature generation is now threaded,
> use higher thread values as examples in this code for better performance.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  meta/lib/oeqa/selftest/cases/sstatetests.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
> index 8d05e30..feadf3c 100644
> --- a/meta/lib/oeqa/selftest/cases/sstatetests.py
> +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
> @@ -418,7 +418,7 @@ DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
>
>          self.write_config("""
>  TMPDIR = "${TOPDIR}/tmp-sstatesamehash"
> -BB_NUMBER_THREADS = "1"
> +BB_NUMBER_THREADS = "${@oe.utils.cpu_count()}"
>  PARALLEL_MAKE = "-j 1"
>  DL_DIR = "${TOPDIR}/download1"
>  TIME = "111111"
> @@ -431,7 +431,7 @@ http_proxy = ""
>          bitbake("world meta-toolchain -S none")
>          self.write_config("""
>  TMPDIR = "${TOPDIR}/tmp-sstatesamehash2"
> -BB_NUMBER_THREADS = "2"
> +BB_NUMBER_THREADS = "${@oe.utils.cpu_count()-1}"

What happens here if I have one CPU core?

>  PARALLEL_MAKE = "-j 2"
>  DL_DIR = "${TOPDIR}/download2"
>  TIME = "222222"
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 4/5] sstatetests: Use higher parallelism value

[Richard Purdie]

On Tue, 2017-06-13 at 08:55 -0700, Andre McCurdy wrote:
> On Tue, Jun 13, 2017 at 2:49 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > Since the processing code for signature generation is now threaded,
> > use higher thread values as examples in this code for better
> > performance.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> >  meta/lib/oeqa/selftest/cases/sstatetests.py | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py
> > b/meta/lib/oeqa/selftest/cases/sstatetests.py
> > index 8d05e30..feadf3c 100644
> > --- a/meta/lib/oeqa/selftest/cases/sstatetests.py
> > +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
> > @@ -418,7 +418,7 @@ DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
> > 
> >          self.write_config("""
> >  TMPDIR = "${TOPDIR}/tmp-sstatesamehash"
> > -BB_NUMBER_THREADS = "1"
> > +BB_NUMBER_THREADS = "${@oe.utils.cpu_count()}"
> >  PARALLEL_MAKE = "-j 1"
> >  DL_DIR = "${TOPDIR}/download1"
> >  TIME = "111111"
> > @@ -431,7 +431,7 @@ http_proxy = ""
> >          bitbake("world meta-toolchain -S none")
> >          self.write_config("""
> >  TMPDIR = "${TOPDIR}/tmp-sstatesamehash2"
> > -BB_NUMBER_THREADS = "2"
> > +BB_NUMBER_THREADS = "${@oe.utils.cpu_count()-1}"
> What happens here if I have one CPU core?

I'll change it to a +1 :)

Cheers,

Richard