[PATCH 0/2] Fix SPDX processing with AMD toolchain building

[PATCH 0/2] Fix SPDX processing with AMD toolchain building

[Mark Hatle]

AMD (Xilinx) toolchain building for our 'vitis' product line seems to trigger
two issues with the SPDX processing.  Specifically:

*) Baremetal toolchain elements may have directories listed in them, which
   triggers a hashing failure

*) The toolchains are built of multilib packages, which triggers a failure
   reported as 'No SPDX file found for package ....'


I think the first change is pretty obvious what needs to happen if directory
names can end up in the debugsrc list.  (Why they end up there, I have no
idea -- but this is all built from existing YP binutils, gcc, and newlib.)

The second patch it was tracked down that when multilibs are activated, the
system won't use the correct 'sstate_arch' to search for the files.  This
was resolved following a similar approach to the PACKAGE_ARCHs, but does
make the assumption that 'package.bbclass' is available in all instances.


Reproducing the build environment:

git clone https://git.yoctoproject.org/poky -b master
git clone https://github.com/Xilinx/meta-xilinx -b master-next
. ./poky oe-init-build-env
bitbake-layers add-layer ../meta-xilinx/meta-xilinx-core \
    ../meta-xilinx/meta-microblaze ../meta-xilinx/meta-xilinx-standalone \
    ../meta-xilinx/meta-vitis-tc

Build for each of the toolchains (really only aarch64-tc is needed to 
demonstrate the issues)

MACHINE=<machine> DISTRO=xilinx-standalone bitbake meta-xilinx-toolchain

Where machine is one of:
  aarch32-tc
  aarch64-tc
  arm-rm-tc
  microblaze-tc
  riscv-tc

(aarch64-tc has by far the fewest multilibs so is the fastest to parse and
build.)

Any questions, let me know!

(Also these changes were made available to https://git.yoctoproject.org/poky-contrib mgh/spdx-fixes)

Mark Hatle (2):
  create-sdpx-2.2.bbclass: Switch from exists to isfile checking
    debugsrc
  package.bbclass: Add ALL_MULTILIB_SSTATE_ARCHS

 meta/classes-global/package.bbclass  |  1 +
 meta/classes/create-spdx-2.2.bbclass | 15 ++++++++-------
 2 files changed, 9 insertions(+), 7 deletions(-)

-- 
1.8.3.1

[PATCH 1/2] create-sdpx-2.2.bbclass: Switch from exists to isfile checking debugsrc

[Mark Hatle]

From: Mark Hatle <mark.hatle@amd.com>

While debugsrc is almost always a file (or link), there are apparently
cases where a directory could be returned from the dwarfsrcfiles
processing.  When this happens, the hashing fails and an error results
when building the SPDX documents.

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 meta/classes/create-spdx-2.2.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass
index 239a95d..13d2aa1 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -223,7 +223,8 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag
                     debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '')
                 else:
                     debugsrc_path = search / debugsrc.lstrip("/")
-                if not debugsrc_path.exists():
+                # We can only hash files below, skip directories, links, etc.
+                if not os.path.isfile(debugsrc_path):
                     continue
 
                 file_sha256 = bb.utils.sha256_file(debugsrc_path)
-- 
1.8.3.1

[PATCH 2/2] package.bbclass: Add ALL_MULTILIB_SSTATE_ARCHS

[Mark Hatle]

From: Mark Hatle <mark.hatle@amd.com>

When create-spdx-2.2 class is processing documents, it needs to
find the document in a path that is related to the SSTATE_ARCH
when a packge is generated.  The SSTATE_ARCH can be affected by
multilib configurations, resulting is something like armv8a-mlib.

When the image (or SDK) is being generated and the components are
collected, the system has no knowledge of the multilib arch and
will fail to find it, such as:

  ERROR: meta-toolchain-1.0-r0 do_populate_sdk: No SPDX file found
   for package libilp32-libgcc-dbg,
   False sstate:libilp32-libgcc:armv8a-ilp32-mllibilp32-elf:14.1.0:r0:armv8a-ilp32:12:
   sstate:libilp32-libgcc::14.1.0:r0::12:

Adding in the new ALL_MULTILIB_SSTATE_ARCHS will provide a full
set of SSTATE_ARCHS including ones that contain the multilib
extension which will allow create-spdx-2.2 to correctly find the
document it is looking for.  This would also be valuable to any
other function doing a similar search through SSTATE_ARCH that may
have been extended with multilib configurations.

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 meta/classes-global/package.bbclass  |  1 +
 meta/classes/create-spdx-2.2.bbclass | 12 ++++++------
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/classes-global/package.bbclass b/meta/classes-global/package.bbclass
index 6cd8c01..5869fe9 100644
--- a/meta/classes-global/package.bbclass
+++ b/meta/classes-global/package.bbclass
@@ -55,6 +55,7 @@ PKGDEST = "${WORKDIR}/packages-split"
 LOCALE_SECTION ?= ''
 
 ALL_MULTILIB_PACKAGE_ARCHS = "${@all_multilib_tune_values(d, 'PACKAGE_ARCHS')}"
+ALL_MULTILIB_SSTATE_ARCHS = "${@all_multilib_tune_values(d, 'SSTATE_ARCHS')}"
 
 # rpm is used for the per-file dependency identification
 # dwarfsrcfiles is used to determine the list of debug source files
diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass
index 13d2aa1..d2cbec8 100644
--- a/meta/classes/create-spdx-2.2.bbclass
+++ b/meta/classes/create-spdx-2.2.bbclass
@@ -259,7 +259,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
     import oe.spdx
 
     deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     dep_recipes = []
@@ -304,7 +304,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
 
     return dep_recipes
 
-collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS"
+collect_dep_recipes[vardepsexclude] = "ALL_MULTILIB_SSTATE_ARCHS"
 
 def collect_dep_sources(d, dep_recipes):
     import oe.sbom
@@ -600,7 +600,7 @@ python do_create_runtime_spdx() {
 
     providers = collect_package_providers(d)
     pkg_arch = d.getVar("SSTATE_PKGARCH")
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     if not is_native:
@@ -706,7 +706,7 @@ python do_create_runtime_spdx() {
             oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d))
 }
 
-do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
+do_create_runtime_spdx[vardepsexclude] += "OVERRIDES ALL_MULTILIB_SSTATE_ARCHS"
 
 addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work
 SSTATETASKS += "do_create_runtime_spdx"
@@ -787,7 +787,7 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
     import bb.compress.zstd
 
     providers = collect_package_providers(d)
-    package_archs = d.getVar("SSTATE_ARCHS").split()
+    package_archs = d.getVar("ALL_MULTILIB_SSTATE_ARCHS").split()
     package_archs.reverse()
 
     creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
@@ -939,4 +939,4 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx
 
             tar.addfile(info, fileobj=index_str)
 
-combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS"
+combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS ALL_MULTILIB_SSTATE_ARCHS"
-- 
1.8.3.1