[PATCH 0/3] Fix problem with leftover files in tmp/sysroots-components

[PATCH 0/3] Fix problem with leftover files in tmp/sysroots-components

[Peter Kjellerstedt]

We have seen problems where sometimes files are leftover in
tmp/sysroots-components when changing PACKAGE_ARCH for a package. This
could then lead to those files being picked up instead of the real
files when the RSS for another package was created.

It turned out that this happened if the original architecture of the
package was one that was added using PACKAGE_EXTRA_ARCHS. Those extra
architectures were not handled in the sstate_eventhandler2() function
in sstate.bbclass that does the clean up of tmp/sysroots-components.

During the course of debugging this problem I also noticed that there
were two index files in tmp/sstate-control for machines that contain a
dash in their names. This turned out to be due to the use of the
stamp-extra-info task flag for some tasks, which were using ${MACHINE}
rather than ${MACHINE_ARCH} to specify the architecture. Since this
seemed inconsistent and confusing, I fixed that as well.

//Peter

The following changes since commit 312a5a93cac95fe62c56e374db728c825b272e64:

  layer.conf: Update LAYERSERIES rocko -> sumo (2018-04-06 11:39:49 +0100)

are available in the git repository at:

  git://push.yoctoproject.org/poky-contrib pkj/extra-archs-in-sstate

Peter Kjellerstedt (3):
  Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task
    flag
  sstate.bbclass: Add ${PACKAGE_EXTRA_ARCHS} to SSTATE_ARCHS
  license.bbclass: Minor simplification of get_deployed_dependencies()

 meta/classes/deploy.bbclass            | 2 +-
 meta/classes/image.bbclass             | 2 +-
 meta/classes/license.bbclass           | 4 +---
 meta/classes/package.bbclass           | 2 +-
 meta/classes/populate_sdk_base.bbclass | 2 +-
 meta/classes/populate_sdk_ext.bbclass  | 2 +-
 meta/classes/sstate.bbclass            | 5 +++--
 meta/recipes-core/meta/signing-keys.bb | 3 ++-
 8 files changed, 11 insertions(+), 11 deletions(-)

-- 
2.12.0

[PATCH 1/3] Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task flag

[Peter Kjellerstedt]

Without this change, there will be two sstate index files in
tmp/sstate-control for any machine that contains a dash in the
name.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/deploy.bbclass            | 2 +-
 meta/classes/image.bbclass             | 2 +-
 meta/classes/package.bbclass           | 2 +-
 meta/classes/populate_sdk_base.bbclass | 2 +-
 meta/classes/populate_sdk_ext.bbclass  | 2 +-
 meta/recipes-core/meta/signing-keys.bb | 3 ++-
 6 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/classes/deploy.bbclass b/meta/classes/deploy.bbclass
index 8ad07da015..6d52908783 100644
--- a/meta/classes/deploy.bbclass
+++ b/meta/classes/deploy.bbclass
@@ -8,4 +8,4 @@ python do_deploy_setscene () {
 }
 addtask do_deploy_setscene
 do_deploy[dirs] = "${DEPLOYDIR} ${B}"
-do_deploy[stamp-extra-info] = "${MACHINE}"
+do_deploy[stamp-extra-info] = "${MACHINE_ARCH}"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 1636eed6b0..2247b305da 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -289,7 +289,7 @@ SSTATETASKS += "do_image_complete"
 SSTATE_SKIP_CREATION_task-image-complete = '1'
 do_image_complete[sstate-inputdirs] = "${IMGDEPLOYDIR}"
 do_image_complete[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
-do_image_complete[stamp-extra-info] = "${MACHINE}"
+do_image_complete[stamp-extra-info] = "${MACHINE_ARCH}"
 addtask do_image_complete after do_image before do_build
 python do_image_complete_setscene () {
     sstate_setscene(d)
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 504c0232ef..9bba021efb 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -2234,7 +2234,7 @@ SSTATETASKS += "do_packagedata"
 do_packagedata[sstate-inputdirs] = "${PKGDESTWORK}"
 do_packagedata[sstate-outputdirs] = "${PKGDATA_DIR}"
 do_packagedata[sstate-lockfile-shared] = "${PACKAGELOCK}"
-do_packagedata[stamp-extra-info] = "${MACHINE}"
+do_packagedata[stamp-extra-info] = "${MACHINE_ARCH}"
 
 python do_packagedata_setscene () {
     sstate_setscene(d)
diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 77ec8aaec2..3560ef143c 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -156,7 +156,7 @@ SSTATE_SKIP_CREATION_task-populate-sdk = '1'
 do_populate_sdk[cleandirs] = "${SDKDEPLOYDIR}"
 do_populate_sdk[sstate-inputdirs] = "${SDKDEPLOYDIR}"
 do_populate_sdk[sstate-outputdirs] = "${SDK_DEPLOY}"
-do_populate_sdk[stamp-extra-info] = "${MACHINE}${SDKMACHINE}"
+do_populate_sdk[stamp-extra-info] = "${MACHINE_ARCH}${SDKMACHINE}"
 
 fakeroot create_sdk_files() {
 	cp ${COREBASE}/scripts/relocate_sdk.py ${SDK_OUTPUT}/${SDKPATH}/
diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 655375416b..e1bba49eaf 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -722,6 +722,6 @@ SSTATE_SKIP_CREATION_task-populate-sdk-ext = '1'
 do_populate_sdk_ext[cleandirs] = "${SDKEXTDEPLOYDIR}"
 do_populate_sdk_ext[sstate-inputdirs] = "${SDKEXTDEPLOYDIR}"
 do_populate_sdk_ext[sstate-outputdirs] = "${SDK_DEPLOY}"
-do_populate_sdk_ext[stamp-extra-info] = "${MACHINE}"
+do_populate_sdk_ext[stamp-extra-info] = "${MACHINE_ARCH}"
 
 addtask populate_sdk_ext after do_sdk_depends
diff --git a/meta/recipes-core/meta/signing-keys.bb b/meta/recipes-core/meta/signing-keys.bb
index 6387d90d47..1e1c7e3459 100644
--- a/meta/recipes-core/meta/signing-keys.bb
+++ b/meta/recipes-core/meta/signing-keys.bb
@@ -69,7 +69,8 @@ do_deploy () {
 do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_RPM}"
 # cleandirs should possibly be in deploy.bbclass but we need it
 do_deploy[cleandirs] = "${DEPLOYDIR}"
-# clear stamp-extra-info since MACHINE is normally put there by deploy.bbclass
+# clear stamp-extra-info since MACHINE_ARCH is normally put there by
+# deploy.bbclass
 do_deploy[stamp-extra-info] = ""
 addtask deploy after do_get_public_keys
 
-- 
2.12.0

[PATCH 2/3] sstate.bbclass: Add ${PACKAGE_EXTRA_ARCHS} to SSTATE_ARCHS

[Peter Kjellerstedt]

This makes sure files provided by packages that use any of the extra
architectures defined using ${PACKAGE_EXTRA_ARCHS} are cleaned from
tmp/sysroot-components when sstate_eventhandler2() executes.

Without this, changing a package from using one of the extra
architectures to another architecture would lead to files being
leftover in tmp/sysroot-components, which could later be picked up
when creating the RSS for another package rather than the files that
belonged to the updated package.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/sstate.bbclass | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 68089421f5..0b28850140 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -48,7 +48,8 @@ SSTATE_ARCHS = " \
     ${SDK_ARCH}_${PACKAGE_ARCH} \
     allarch \
     ${PACKAGE_ARCH} \
-    ${MACHINE}"
+    ${PACKAGE_EXTRA_ARCHS} \
+    ${MACHINE_ARCH}"
 
 SSTATE_MANMACH ?= "${SSTATE_PKGARCH}"
 
@@ -1026,7 +1027,7 @@ python sstate_eventhandler2() {
         with open(preservestampfile, 'r') as f:
             preservestamps = f.readlines()
     seen = []
-    for a in d.getVar("SSTATE_ARCHS").split():
+    for a in sorted(list(set(d.getVar("SSTATE_ARCHS").split()))):
         toremove = []
         i = d.expand("${SSTATE_MANIFESTS}/index-" + a)
         if not os.path.exists(i):
-- 
2.12.0

[PATCH 3/3] license.bbclass: Minor simplification of get_deployed_dependencies()

[Peter Kjellerstedt]

Since ${SSTATE_ARCHS} now contains ${PACKAGE_EXTRA_ARCHS} there is no
longer any need to add those extra architectures to the list of
architectures handled in get_deployed_dependencies().

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/license.bbclass | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index d353110464..9ac7e0bef8 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -226,9 +226,7 @@ def get_deployed_dependencies(d):
     # The manifest file name contains the arch. Because we are not running
     # in the recipe context it is necessary to check every arch used.
     sstate_manifest_dir = d.getVar("SSTATE_MANIFESTS")
-    sstate_archs = d.getVar("SSTATE_ARCHS")
-    extra_archs = d.getVar("PACKAGE_EXTRA_ARCHS")
-    archs = list(set(("%s %s" % (sstate_archs, extra_archs)).split()))
+    archs = list(set(d.getVar("SSTATE_ARCHS").split()))
     for dep in depends:
         # Some recipes have an arch on their own, so we try that first.
         special_arch = d.getVar("PACKAGE_ARCH_pn-%s" % dep)
-- 
2.12.0

✗ patchtest: failure for Fix problem with leftover files in tmp/sysroots-components

[Patchwork]

== Series Details ==

Series: Fix problem with leftover files in tmp/sysroots-components
Revision: 1
URL   : https://patchwork.openembedded.org/series/11706/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [1/3] Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task flag
 Issue             Shortlog does not follow expected format [test_shortlog_format] 
  Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for Fix problem with leftover files in tmp/sysroots-components

[Peter Kjellerstedt]

> -----Original Message-----
> From: Patchwork [mailto:patchwork@patchwork.openembedded.org]
> Sent: den 6 april 2018 20:34
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: ✗ patchtest: failure for Fix problem with leftover files in
> tmp/sysroots-components
> 
> == Series Details ==
> 
> Series: Fix problem with leftover files in tmp/sysroots-components
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/11706/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Patch            [1/3] Use ${MACHINE_ARCH} instead of ${MACHINE} for stamp-extra-info task flag
>  Issue             Shortlog does not follow expected format [test_shortlog_format]
>   Suggested fix    Commit shortlog (first line of commit message) should follow the format "<target>: <summary>"

Which do you prefer:
* Leave it as it is
* Prefix it with a generic "classes:" (which isn't all true as the last 
  hunk is a comment in a recipe about those classes)
* Split it in six almost identical commits

//Peter