[PATCH 1/4] meta/lib/oe/sbom30.py: correct alias

[PATCH 1/4] meta/lib/oe/sbom30.py: correct alias

[Hongxu Jia]

After commit [spdx 3.0: Rework how SPDX aliases are linked] applied,
it added extra "/" to namespace, which causing the replacement of
UNIHASH missing a "/"

  http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-nativeUNIHASH/build/recipe

After applying this commit to remove "/" from namespace.

  http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-native/UNIHASH/build/recipe

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/sbom30.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
index 29cb9e45ad..08fea2aca3 100644
--- a/meta/lib/oe/sbom30.py
+++ b/meta/lib/oe/sbom30.py
@@ -267,7 +267,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
 
     def new_alias_id(self, obj, replace):
         unihash = self.d.getVar("BB_UNIHASH")
-        namespace = self.get_namespace() + "/"
+        namespace = self.get_namespace()
         if unihash not in obj._id:
             bb.warn(f"Unihash {unihash} not found in {obj._id}")
             return None
-- 
2.25.1

[PATCH 2/4] meta/lib/oe/spdx30_tasks.py: fix hash link conflict while multiconfig enabled

[Hongxu Jia]

While enabling multiconfig, build recipe (such as linux-libc-headers) in
different arch directories but generating duplicated hash link, and these
directories are listed in ${SPDX_MULTILIB_SSTATE_ARCHS}

In the following case, build recipe linux-libc-headers in multiconfig to
generate duplicated hash link in core2-32 and core2-64
$ echo 'MACHINE = qemux86-64' >> conf/local.conf
$ mkdir conf/multiconfig
$ echo 'MACHINE = "qemux86"' >> conf/multiconfig/qemux86.conf
$ echo 'BBMULTICONFIG = qemux86' >> conf/local.conf
$ bitbake linux-libc-headers mc:qemux86:linux-libc-headers
$ ls -al tmp/deploy/spdx/3.0.1/*/by-spdxid-hash/*/*.spdx.json |grep recipe-linux-libc-headers.spdx.json
tmp/deploy/spdx/3.0.1/core2-32/by-spdxid-hash/c5/c57f1b9dbf27d2c83c77ca49f971b31d6d6c5830434104c88a12c6e2eb687094.spdx.json -> ../../recipes/recipe-linux-libc-headers.spdx.json
tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-hash/c5/c57f1b9dbf27d2c83c77ca49f971b31d6d6c5830434104c88a12c6e2eb687094.spdx.json -> ../../recipes/recipe-linux-libc-headers.spdx.json

According to [1], and after [spdx 3.0: Rework how SPDX aliases are linked]
applied, SPDX 3.0 have a loop on ${SPDX_MULTILIB_SSTATE_ARCHS} to search for
the hash link to find jsonld file, return on first match

If hash link duplicated, the first match may not be right. Add ${SSTATE_PKGARCH}
to link prefix which could distinguish arch directories in spdxId prefix.

After applying this commit, the hash link of linux-libc-headers differs

ls -al tmp/deploy/spdx/3.0.1/*/by-spdxid-hash/*/*.spdx.json |grep recipe-linux-libc-headers.spdx.json
tmp/deploy/spdx/3.0.1/core2-32/by-spdxid-hash/4f/4f80c220686f345bce561f6d745165ee803e6390ba5e047f54cb28f4e23c167f.spdx.json -> ../../recipes/recipe-linux-libc-headers.spdx.json
tmp/deploy/spdx/3.0.1/core2-64/by-spdxid-hash/22/22db6f7b9cf02bbd3e64ffe2b9bceb17bf6f2f1366db1e72d8d82c337954b112.spdx.json -> ../../recipes/recipe-linux-libc-headers.spdx.json

[1] https://github.com/openembedded/openembedded-core/commit/f1499c36c1054fc90f7b7268cc95285f2eca72f7#diff-253805ec3eaba58c6c3967711921d5a2748974ba533c6d864d86a8e5ad4df4c4L923

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/sbom30.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
index 08fea2aca3..65603e69b7 100644
--- a/meta/lib/oe/sbom30.py
+++ b/meta/lib/oe/sbom30.py
@@ -786,7 +786,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
             OEIdAliasExtension(
                 alias=objset.new_alias_id(
                     document,
-                    OE_DOC_ALIAS_PREFIX + d.getVar("PN") + "/" + name + "/",
+                    OE_DOC_ALIAS_PREFIX + d.getVar("SSTATE_PKGARCH") + "/" + d.getVar("PN") + "/" + name + "/",
                 ),
             )
         )
-- 
2.25.1

[PATCH V2 3/4] gcc-cross.inc: add var-SSTATE_ARCHS_TUNEPKG to vardeps of do_create_spdx

[Hongxu Jia]

The recipe gcc-cross inherited cross.bbclass which had class-native
recipe behavior, but depends on class-target recipe linux-libc-headers.
It caused the tweaking of ${TUNE_PKGARCH} for 32bit arm did not trigger
rebuild of do_create_spdx in gcc-cross-arm

Due to commit [meta/lib/oe/spdx30_tasks.py: fix hash link conflict while
multiconfig enabled] applied, it added ${SSTATE_PKGARCH} to link prefix.
While tweaking ${TUNE_PKGARCH} for 32bit arm, use do_create_spdx from
sstate cache will caused ${SSTATE_PKGARCH} did not match

In the following case, the link hash has no change while machine was changed
$ echo 'MACHINE = "qemuarmv5"' >> conf/local.conf
$ bitbake gcc-cross-arm
$ grep linux-libc-headers tmp/deploy/spdx/3.0.1/x86_64/recipes/recipe-gcc-cross-arm.spdx.json
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/09523e44e7b6fe1aed03347e30191c00d93500ce6e1f453e8dfdfae98d325f3e/linux-libc-headers/UNIHASH/build/recipe",
$ echo 'MACHINE = "qemuarm"' >> conf/local.conf
$ bitbake gcc-cross-arm
$ grep linux-libc-headers tmp/deploy/spdx/3.0.1/x86_64/recipes/recipe-gcc-cross-arm.spdx.json
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/09523e44e7b6fe1aed03347e30191c00d93500ce6e1f453e8dfdfae98d325f3e/linux-libc-headers/UNIHASH/build/recipe",

Explicitly add var-SSTATE_ARCHS_TUNEPKG to vardeps of do_create_spdx to
trigger rebuild in this situation.

After applying this commit, the link hash has changed along with machine changed

$ echo 'MACHINE = "qemuarmv5"' >> conf/local.conf
$ bitbake gcc-cross-arm
$ grep linux-libc-headers tmp/deploy/spdx/3.0.1/x86_64/recipes/recipe-gcc-cross-arm.spdx.json
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/09523e44e7b6fe1aed03347e30191c00d93500ce6e1f453e8dfdfae98d325f3e/linux-libc-headers/UNIHASH/build/recipe",
$ echo 'MACHINE = "qemuarm"' >> conf/local.conf
$ bitbake gcc-cross-arm
$ grep linux-libc-headers tmp/deploy/spdx/3.0.1/x86_64/recipes/recipe-gcc-cross-arm.spdx.json
        "http://spdxdocs.org/openembedded-alias/by-doc-hash/087e29d3627a06f5377effa627b99907607f5af0fe8f6d116b4490650c404a55/linux-libc-headers/UNIHASH/build/recipe",

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/recipes-devtools/gcc/gcc-cross.inc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-devtools/gcc/gcc-cross.inc b/meta/recipes-devtools/gcc/gcc-cross.inc
index f85ccd5c23..f82228f55a 100644
--- a/meta/recipes-devtools/gcc/gcc-cross.inc
+++ b/meta/recipes-devtools/gcc/gcc-cross.inc
@@ -4,6 +4,7 @@ INHIBIT_DEFAULT_DEPS = "1"
 EXTRADEPENDS = ""
 DEPENDS = "virtual/${TARGET_PREFIX}binutils ${EXTRADEPENDS} ${NATIVEDEPS}"
 PROVIDES = "virtual/${TARGET_PREFIX}gcc virtual/${TARGET_PREFIX}g++"
+do_create_spdx[vardeps] += "SSTATE_ARCHS_TUNEPKG"
 python () {
     if d.getVar("TARGET_OS").startswith("linux"):
         d.setVar("EXTRADEPENDS", "linux-libc-headers")
-- 
2.25.1

[PATCH V3 4/4] meta/lib/oe/sbom30.py: fix alias in simplelicensing_customIdToUri not extracted

[Hongxu Jia]

After commit [spdx 3.0: Rework how SPDX aliases are linked] applied, it set
license_text_map with SPDX alias other than actual ID

The property of simplelicensing_customIdToUri is ListProp(ObjectProp(DictionaryEntry))),
and class DictionaryEntry has key and value, the property of value is StringProp other
than ObjectProp in which could not support to decode/extract SPDX alias with actual ID
in image jsonld file
-----------
      "simplelicensing_customIdToUri": [
        {
          "type": "DictionaryEntry",
          "key": "LicenseRef-PD",
          "value": "http://spdxdocs.org/openembedded-alias/by-doc-hash/d53e90e23b12c4ad640809a74a810e86f31c76cdbdf36487712d22a33d53362a/sqlite3-native/UNIHASH/license-text/PD"
        }
      ],
-----------

Add special code in the linking to manually go through all of the
simplelicensing_customIdToUri DictionaryEntry items and resolve
any aliases to actual objects
-----------
      "simplelicensing_customIdToUri": [
        {
          "type": "DictionaryEntry",
          "key": "LicenseRef-PD",
          "value": "http://spdx.org/spdxdocs/sqlite3-native-e5cc0672-d8dd-57e8-a2df-fe4615831fda/162c62b5b011cd3f82f413b3dae4d6d1542201552c964d5ce69fe170e0720b85/license-text/PD"
        }
      ]
-----------

Suggested-by: Joshua Watt <jpewhacker@gmail.com>
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/sbom30.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py
index 65603e69b7..3264cb2f7e 100644
--- a/meta/lib/oe/sbom30.py
+++ b/meta/lib/oe/sbom30.py
@@ -855,6 +855,18 @@ class ObjectSet(oe.spdx30.SHACLObjectSet):
         self.doc.import_ = sorted(imports.values(), key=lambda e: e.externalSpdxId)
         bb.debug(1, "Linking...")
         self.link()
+
+        # Manually go through all of the simplelicensing_customIdToUri DictionaryEntry
+        # items and resolve any aliases to actual objects.
+        for lic in self.foreach_type(oe.spdx30.simplelicensing_LicenseExpression):
+            for d in lic.simplelicensing_customIdToUri:
+                if d.value.startswith(OE_ALIAS_PREFIX):
+                    obj = self.find_by_id(d.value)
+                    if obj is not None:
+                        d.value = obj._id
+                    else:
+                        self.missing_ids.add(d.value)
+
         self.missing_ids -= set(imports.keys())
         return self.missing_ids
 
-- 
2.25.1

Re: [OE-core] [PATCH 1/4] meta/lib/oe/sbom30.py: correct alias

[Mathieu Dubois-Briand]

On Tue Dec 17, 2024 at 4:47 AM CET, hongxu via lists.openembedded.org wrote:
> After commit [spdx 3.0: Rework how SPDX aliases are linked] applied,
> it added extra "/" to namespace, which causing the replacement of
> UNIHASH missing a "/"
>
>   http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-nativeUNIHASH/build/recipe
>
> After applying this commit to remove "/" from namespace.
>
>   http://spdxdocs.org/openembedded-alias/by-doc-hash/0b308e4b9ad979f642d8787c61f76c31bdcad04837eeaaf8bc383f33f99bbeb8/flex-native/UNIHASH/build/recipe
>
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---

Hi Hongxu,

Sorry, but I still get the SPDX IDs were unable to be resolved warning.
E.g.:

WARNING: core-image-sato-sdk-1.0-r0 do_create_image_sbom_spdx: The following SPDX IDs were unable to be resolved:
  http://spdxdocs.org/openembedded-alias/by-doc-hash/82b2ddffbd624bcb26b4c93f8e7bf31e456ea81a1e5720354726d1fc7e433dd0/openssl/UNIHASH/package/openssl-bin
  http://spdxdocs.org/openembedded-alias/by-doc-hash/882a5d190518b194aea7380573eeee8a79f524131463308e414d0cb51f9a8099/openssl/UNIHASH/package/openssl
  http://spdxdocs.org/openembedded-alias/by-doc-hash/974e7ddc5b8f9307c770783e7d96e9e9fc65cd5b3f9cdd5a8578e4ed121dd8d1/initscripts/UNIHASH/build/recipe
  http://spdxdocs.org/openembedded-alias/by-doc-hash/f63cc948b407d32cfacbfc288d66c3fb64a6a611debfd7989d2cda55d7a957ca/openssh/UNIHASH/package/openssh

https://valkyrie.yoctoproject.org/#/builders/6/builds/668/steps/12/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/8/builds/666/steps/15/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/9/builds/646/steps/12/logs/stdio

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCH 1/4] meta/lib/oe/sbom30.py: correct alias

[hongxu]

[-- Attachment #1: Type: text/plain, Size: 4927 bytes --]

Hi Mathieu,

After apply all the 4 patches,
[PATCH 1/4] meta/lib/oe/sbom30.py: correct alias ( https://lists.openembedded.org/g/openembedded-core/message/208802?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3ACreated%2C%2C%2C20%2C2%2C20%2C110158019%2Cd%253D3&d=3 )
[PATCH 2/4] meta/lib/oe/spdx30_tasks.py: fix hash link conflict while multiconfig enabled ( https://lists.openembedded.org/g/openembedded-core/message/208798?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3ACreated%2C%2C%2C20%2C2%2C20%2C110158020%2Cd%253D3&d=3 )
[PATCH V2 3/4] gcc-cross.inc: add var-SSTATE_ARCHS_TUNEPKG to vardeps of do_create_spdx ( https://lists.openembedded.org/g/openembedded-core/message/208799?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3ACreated%2C%2C%2C20%2C2%2C20%2C110158021%2Cd%253D3&d=3 )
[PATCH V3 4/4] meta/lib/oe/sbom30.py: fix alias in simplelicensing_customIdToUri not extracted ( https://lists.openembedded.org/g/openembedded-core/message/208800?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3ACreated%2C%2C%2C20%2C2%2C20%2C110158022%2Cd%253D3&d=3 )

$ bitbake core-image-sato core-image-sato-sdk core-image-minimal core-image-sato:do_populate_sdk core-image-minimal:do_populate_sdk_ext core-image-sato:do_populate_sdk_ext

I could not re-produce the WARNING, f rom the  autobuilder log [1], I doubt the issue was caused by sstate cache between multiple builds with different poky sources

$ grep openssl stdio -n | grep spdx
2775:NOTE: Running setscene task 1619 of 7903 (virtual:nativesdk:/srv/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-connectivity/openssl/openssl_3.4.0.bb:do_create_packa
ge_spdx_setscene)
2805:NOTE: recipe nativesdk-openssl-3.4.0-r0: task do_create_package_spdx_setscene: Started
2823:NOTE: recipe nativesdk-openssl-3.4.0-r0: task do_create_package_spdx_setscene: Succeeded
6774:NOTE: Running setscene task 3110 of 7903 (/srv/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-connectivity/openssl/openssl_3.4.0.bb:do_create_package_spdx_setscene)
6799:NOTE: recipe openssl-3.4.0-r0: task do_create_package_spdx_setscene: Started
6823:NOTE: recipe openssl-3.4.0-r0: task do_create_package_spdx_setscene: Succeeded
7879:NOTE: Running setscene task 3505 of 7903 (virtual:native:/srv/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-connectivity/openssl/openssl_3.4.0.bb:do_create_package spdx_setscene)
7909:NOTE: recipe openssl-native-3.4.0-r0: task do_create_package_spdx_setscene: Started
7929:NOTE: recipe openssl-native-3.4.0-r0: task do_create_package_spdx_setscene: Succeeded
13460:NOTE: Running setscene task 5732 of 7903 (virtual:nativesdk:/srv/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-connectivity/openssl/openssl_3.4.0.bb:do_create_spd _setscene)
13494:NOTE: recipe nativesdk-openssl-3.4.0-r0: task do_create_spdx_setscene: Started
13512:NOTE: recipe nativesdk-openssl-3.4.0-r0: task do_create_spdx_setscene: Succeeded
15764:NOTE: Running setscene task 7051 of 7903 (/srv/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-connectivity/openssl/openssl_3.4.0.bb:do_create_spdx_setscene)
15788:NOTE: recipe openssl-3.4.0-r0: task do_create_spdx_setscene: Started
15814:NOTE: recipe openssl-3.4.0-r0: task do_create_spdx_setscene: Succeeded
16806:NOTE: Running setscene task 7666 of 7903 (virtual:native:/srv/pokybuild/yocto-worker/qemuarm64-armhost/build/meta/recipes-connectivity/openssl/openssl_3.4.0.bb:do_create_spdx_se
tscene)
16823:NOTE: recipe openssl-native-3.4.0-r0: task do_create_spdx_setscene: Started
16832:NOTE: recipe openssl-native-3.4.0-r0: task do_create_spdx_setscene: Succeeded
19470:  http://spdxdocs.org/openembedded-alias/by-doc-hash/82b2ddffbd624bcb26b4c93f8e7bf31e456ea81a1e5720354726d1fc7e433dd0/openssl/UNIHASH/package/openssl-bin
19471:  http://spdxdocs.org/openembedded-alias/by-doc-hash/882a5d190518b194aea7380573eeee8a79f524131463308e414d0cb51f9a8099/openssl/UNIHASH/package/openssl
19482:  http://spdxdocs.org/openembedded-alias/by-doc-hash/abb77695d73c96547fa2c4d5d658cd5c44a6ca7b4f0c07889cbf868d6c832bb1/openssl-native/UNIHASH/build/recipe
19866:  http://spdxdocs.org/openembedded-alias/by-doc-hash/abb77695d73c96547fa2c4d5d658cd5c44a6ca7b4f0c07889cbf868d6c832bb1/openssl-native/UNIHASH/build/recipe
19965:  http://spdxdocs.org/openembedded-alias/by-doc-hash/82b2ddffbd624bcb26b4c93f8e7bf31e456ea81a1e5720354726d1fc7e433dd0/openssl/UNIHASH/package/openssl-bin
19967:  http://spdxdocs.org/openembedded-alias/by-doc-hash/882a5d190518b194aea7380573eeee8a79f524131463308e414d0cb51f9a8099/openssl/UNIHASH/package/openssl
19982:  http://spdxdocs.org/openembedded-alias/by-doc-hash/abb77695d73c96547fa2c4d5d658cd5c44a6ca7b4f0c07889cbf868d6c832bb1/openssl-native/UNIHASH/build/recipe
20069:  http://spdxdocs.org/openembedded-alias/by-doc-hash/abb77695d73c96547fa2c4d5d658cd5c44a6ca7b4f0c07889cbf868d6c832bb1/openssl-native/UNIHASH/build/recipe

[1] https://valkyrie.yoctoproject.org/api/v2/logs/928428/raw

//Hongxu

[-- Attachment #2: Type: text/html, Size: 5833 bytes --]

Re: [OE-core] [PATCH 1/4] meta/lib/oe/sbom30.py: correct alias

[Mathieu Dubois-Briand]

[-- Attachment #1.1: Type: text/plain, Size: 1531 bytes --]

Hi Hongxu,

I'm back on this issue. So yesterday I tried again to pick these
patches, and got warnings again:

https://valkyrie.yoctoproject.org/#/builders/29/builds/682

If I relaunch the builds, sometimes they will success, sometimes I still
get the warnings. Something seems to be intermittent here.

I tried to reproduce locally, and again, I get the warnings in some
intermittent way.

My setup was:
- Clean build directory, empty sstate cache.
- The local.conf attached. This is mainly the configuration used for
  https://valkyrie.yoctoproject.org/#/builders/8/builds/698, with some
  minor changes (like disabling the shared sstate configuration).
- Using meta, meta-poky and meta-yocto-bsp at this revision:
  https://git.yoctoproject.org/poky-ci-archive/tag/?h=a-full-685
- Used: bitbake core-image-sato core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-full-cmdline core-image-sato:do_populate_sdk

Now here is how it went:
- On my very first build, I got some SPDX warnings.
- On my next build they were all gone.
- I tried to tweak a bit my config, cleaned my cache again and still got
  no warnings.
- Then after some various modifications, I switched back again to my
  very first configuration, and got again some SPDX warnings.

So I'm sorry, I cannot give you some exact instructions to reproduce
this issue, but still we have some intermittent warnings :(

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com


[-- Attachment #2: local.conf --]
[-- Type: text/plain, Size: 14170 bytes --]

#
# This file is your local configuration file and is where all local user settings
# are placed. The comments in this file give some guide to the options a new user
# to the system might want to change but pretty much any configuration option can
# be set in this file. More adventurous users can look at
# local.conf.sample.extended which contains other examples of configuration which
# can be placed in this file but new users likely won't need any of them
# initially. There's also site.conf.sample which contains examples of site specific
# information such as proxy server addresses.
#
# Lines starting with the '#' character are commented out and in some cases the
# default values are provided as comments to show people example syntax. Enabling
# the option is a question of removing the # character and making any change to the
# variable as required.

#
# Machine Selection
#
# You need to select a specific machine to target the build with. There are a selection
# of emulated machines available which can boot and run in the QEMU emulator:
#
#MACHINE ?= "qemuarm"
#MACHINE ?= "qemuarm64"
#MACHINE ?= "qemumips"
#MACHINE ?= "qemumips64"
#MACHINE ?= "qemuppc"
#MACHINE ?= "qemux86"
#MACHINE ?= "qemux86-64"
#
# There are also the following hardware board target machines included for 
# demonstration purposes:
#
#MACHINE ?= "beaglebone-yocto"
#MACHINE ?= "genericarm64"
#MACHINE ?= "genericx86"
#MACHINE ?= "genericx86-64"
#
# This sets the default machine to be qemux86-64 if no other machine is selected:
MACHINE ??= "qemux86-64"

# These are some of the more commonly used values. Looking at the files in the
# meta/conf/machine directory, or the conf/machine directory of any additional layers
# you add in will show all the available machines.

#
# Where to place downloads
#
# During a first build the system will download many different source code tarballs
# from various upstream projects. This can take a while, particularly if your network
# connection is slow. These are all stored in DL_DIR. When wiping and rebuilding you
# can preserve this directory to speed up this part of subsequent builds. This directory
# is safe to share between multiple builds on the same machine too.
#
# The default is a downloads directory under TOPDIR which is the build directory.
#
#DL_DIR ?= "${TOPDIR}/../build/downloads"

#
# Where to place shared-state files
#
# BitBake has the capability to accelerate builds based on previously built output.
# This is done using "shared state" files which can be thought of as cache objects
# and this option determines where those files are placed.
#
# You can wipe out TMPDIR leaving this directory intact and the build would regenerate
# from these files if no changes were made to the configuration. If changes were made
# to the configuration, only shared state files where the state was still valid would
# be used (done using checksums).
#
# The default is a sstate-cache directory under TOPDIR.
#
#SSTATE_DIR ?= "${TOPDIR}/sstate-cache"

#
# Where to place the build output
#
# This option specifies where the bulk of the building work should be done and
# where BitBake should place its temporary files and output. Keep in mind that
# this includes the extraction and compilation of many applications and the toolchain
# which can use Gigabytes of hard disk space.
#
# The default is a tmp directory under TOPDIR.
#
#TMPDIR = "${TOPDIR}/tmp"

#
# Default policy config
#
# The distribution setting controls which policy settings are used as defaults.
# The default value is fine for general Yocto project use, at least initially.
# Ultimately when creating custom policy, people will likely end up subclassing 
# these defaults.
#
DISTRO ?= "poky"
# As an example of a subclass there is a "bleeding" edge policy configuration
# where many versions are set to the absolute latest code from the upstream 
# source control systems. This is just mentioned here as an example, its not
# useful to most new users.
# DISTRO ?= "poky-bleeding"

#
# Package Management configuration
#
# This variable lists which packaging formats to enable. Multiple package backends
# can be enabled at once and the first item listed in the variable will be used
# to generate the root filesystems.
# Options are:
#  - 'package_deb' for debian style deb files
#  - 'package_ipk' for ipk files are used by opkg (a debian style embedded package manager)
#  - 'package_rpm' for rpm style packages
# E.g.: PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk"
# OE-Core defaults to ipkg, whilst Poky defaults to rpm:
# PACKAGE_CLASSES ?= "package_rpm"

#
# SDK target architecture
#
# This variable specifies the architecture to build SDK items for and means
# you can build the SDK packages for architectures other than the machine you are
# running the build on (i.e. building i686 packages on an x86_64 host).
# Supported values are i686, x86_64, aarch64
#SDKMACHINE ?= "i686"

#
# Extra image configuration defaults
#
# The EXTRA_IMAGE_FEATURES variable allows extra packages to be added to the generated
# images. Some of these options are added to certain image types automatically. Some
# of the features available are:
#  "dbg-pkgs"             - add -dbg packages for all installed packages
#                           (adds symbol information for debugging/profiling)
#  "src-pkgs"             - add -src packages for all installed packages
#                           (adds source code for debugging)
#  "dev-pkgs"             - add -dev packages for all installed packages
#                           (useful if you want to develop against libs in the image)
#  "ptest-pkgs"           - add -ptest packages for all ptest-enabled packages
#                           (useful if you want to run the package test suites)
#  "tools-sdk"            - add development tools (gcc, make, pkgconfig etc.)
#  "tools-debug"          - add debugging tools (gdb, strace)
#  "eclipse-debug"        - add Eclipse remote debugging support
#  "tools-profile"        - add profiling tools (oprofile, lttng, valgrind)
#  "tools-testapps"       - add useful testing tools (ts_print, aplay, arecord etc.)
#  "allow-empty-password" - allow users to have an empty password
#  "empty-root-password"  - the root user has no password set
#  "allow-root-login      - the root user can login
# There are other features that can be used here too, see
# meta/classes-recipe/image.bbclass and
# meta/classes-recipe/core-image.bbclass for more details.
# We default to allowing root login without a password for convenience.
EXTRA_IMAGE_FEATURES ?= "allow-empty-password empty-root-password allow-root-login"

#
# Additional image features
#
# The following is a list of additional classes to use when building images which
# enable extra features. Some available options which can be included in this variable
# are:
#   - 'buildstats' collect build statistics
USER_CLASSES ?= "buildstats"

#
# Runtime testing of images
#
# The build system can test booting virtual machine images under qemu (an emulator)
# after any root filesystems are created and run tests against those images. It can also
# run tests against any SDK that are built. To enable this uncomment these lines.
# See meta/classes-recipe/test{image,sdk}.bbclass for further details.
#IMAGE_CLASSES += "testimage testsdk"
#TESTIMAGE_AUTO:qemuall = "1"

#
# Interactive shell configuration
#
# Under certain circumstances the system may need input from you and to do this it
# can launch an interactive shell. It needs to do this since the build is
# multithreaded and needs to be able to handle the case where more than one parallel
# process may require the user's attention. The default is iterate over the available
# terminal types to find one that works.
#
# Examples of the occasions this may happen are when resolving patches which cannot
# be applied, to use the devshell or the kernel menuconfig
#
# Supported values are auto, gnome, xfce, rxvt, screen, konsole (KDE 3.x only), none
# Note: currently, Konsole support only works for KDE 3.x due to the way
# newer Konsole versions behave
#OE_TERMINAL = "auto"
# By default disable interactive patch resolution (tasks will just fail instead):
PATCHRESOLVE = "noop"

#
# Disk Space Monitoring during the build
#
# Monitor the disk space during the build. If there is less that 1GB of space or less
# than 100K inodes in any key build location (TMPDIR, DL_DIR, SSTATE_DIR), gracefully
# shutdown the build. If there is less than 100MB or 1K inodes, perform a hard halt
# of the build. The reason for this is that running completely out of space can corrupt
# files and damages the build in ways which may not be easily recoverable.
# It's necessary to monitor /tmp, if there is no space left the build will fail
# with very exotic errors.
BB_DISKMON_DIRS ??= "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    HALT,${TMPDIR},100M,1K \
    HALT,${DL_DIR},100M,1K \
    HALT,${SSTATE_DIR},100M,1K \
    HALT,/tmp,10M,1K"

#
# Shared-state files from other locations
#
# As mentioned above, shared state files are prebuilt cache data objects which can be
# used to accelerate build time. This variable can be used to configure the system
# to search other mirror locations for these objects before it builds the data itself.
#
# This can be a filesystem directory, or a remote url such as https or ftp. These
# would contain the sstate-cache results from previous builds (possibly from other
# machines). This variable works like fetcher MIRRORS/PREMIRRORS and points to the
# cache locations to check for the shared objects.
# NOTE: if the mirror uses the same structure as SSTATE_DIR, you need to add PATH
# at the end as shown in the examples below. This will be substituted with the
# correct path within the directory structure.
#SSTATE_MIRRORS ?= "\
#file://.* https://someserver.tld/share/sstate/PATH;downloadfilename=PATH \
#file://.* file:///some/local/dir/sstate/PATH"

#
# Yocto Project SState Mirror
#
# The Yocto Project has prebuilt artefacts available for its releases, you can enable
# use of these by uncommenting some of the following lines. This will mean the build uses
# the network to check for artefacts at the start of builds, which does slow it down
# initially but it will then speed up the builds by not having to build things if they are
# present in the cache. It assumes you can download something faster than you can build it
# which will depend on your network.
# Note: For this to work you also need hash-equivalence passthrough to the matching server
# There is a choice between our sstate server directly and a faster content delivery network
# (CDN) kindly provided by JSDelivr, uncomment one of the SSTATE_MIRRORS lines, not both.
# Using the CDN rather than the yoctoproject.org address is suggested/preferred.
#
#BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
#SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
#
###SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"


#
# Qemu configuration
#
# By default native qemu will build with a builtin VNC server where graphical output can be
# seen. The line below enables the SDL UI frontend too.
PACKAGECONFIG:append:pn-qemu-system-native = " sdl"
# By default libsdl2-native will be built, if you want to use your host's libSDL instead of 
# the minimal libsdl built by libsdl2-native then uncomment the ASSUME_PROVIDED line below.
#ASSUME_PROVIDED += "libsdl2-native"

# You can also enable the Gtk UI frontend, which takes somewhat longer to build, but adds
# a handy set of menus for controlling the emulator.
#PACKAGECONFIG:append:pn-qemu-system-native = " gtk+"

#
# Hash Equivalence
#
# Enable support for automatically running a local hash equivalence server and
# instruct bitbake to use a hash equivalence aware signature generator. Hash
# equivalence improves reuse of sstate by detecting when a given sstate
# artifact can be reused as equivalent, even if the current task hash doesn't
# match the one that generated the artifact.
#
# A shared hash equivalent server can be set with "<HOSTNAME>:<PORT>" format
#
#BB_HASHSERVE = "auto"
#BB_SIGNATURE_HANDLER = "OEEquivHash"

#
# Memory Resident Bitbake
#
# Bitbake's server component can stay in memory after the UI for the current command
# has completed. This means subsequent commands can run faster since there is no need
# for bitbake to reload cache files and so on. Number is in seconds, after which the
# server will shut down.
#
#BB_SERVER_TIMEOUT = "60"

# CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to
# track the version of this file when it was generated. This can safely be ignored if
# this doesn't mean anything to you.
CONF_VERSION = "2"
MACHINE = "qemuarm64"
DISTRO = "poky"
SDKMACHINE = "aarch64"
PACKAGE_CLASSES = "package_rpm package_deb package_ipk"
INHERIT += 'image-buildinfo'
IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'
QEMU_USE_KVM = 'True'
INHERIT += 'report-error'
PREMIRRORS = ''
BB_GENERATE_MIRROR_TARBALLS = '1'
BB_NUMBER_THREADS = '16'
BB_NUMBER_PARSE_THREADS = '16'
PARALLEL_MAKE = '-j 16 -l 75'
BB_PRESSURE_MAX_CPU = '20000'
BB_PRESSURE_MAX_IO = '20000'
XZ_MEMLIMIT = '5%'
XZ_THREADS = '8'
ZSTD_THREADS = '8'
BB_TASK_NICE_LEVEL = '5'
BB_TASK_NICE_LEVEL:task-testimage = '0'
BB_TASK_IONICE_LEVEL = '2.7'
BB_TASK_IONICE_LEVEL:task-testimage = '2.1'
IMAGE_CLASSES += 'testimage'
TEST_QEMUBOOT_TIMEOUT = '1500'
SANITY_TESTED_DISTROS = ''
SDK_EXT_TYPE = 'minimal'
SDK_INCLUDE_TOOLCHAIN = '1'
ESDK_LOCALCONF_REMOVE:append = 'BB_HASHSERVE'
BB_DISKMON_DIRS = 'STOPTASKS,${TMPDIR},1G,100K STOPTASKS,${DL_DIR},1G STOPTASKS,${SSTATE_DIR},1G STOPTASKS,/tmp,100M,30K HALT,${TMPDIR},100M,1K HALT,${DL_DIR},100M HALT,${SSTATE_DIR},100M HALT,/tmp,10M,1K'
BB_HEARTBEAT_EVENT = '60'
BB_LOG_HOST_STAT_ON_INTERVAL = '1'
BB_LOG_HOST_STAT_CMDS_INTERVAL = 'oe-time-dd-test.sh -c 100 -t 3'
BB_LOG_HOST_STAT_ON_FAILURE = '1'
BB_LOG_HOST_STAT_CMDS_FAILURE = 'oe-time-dd-test.sh -l'
BB_SERVER_TIMEOUT = '60'
BB_LOADFACTOR_MAX = '1.5'

Re: [OE-core] [PATCH 1/4] meta/lib/oe/sbom30.py: correct alias

[Hongxu Jia]

On 12/21/24 00:09, Mathieu Dubois-Briand wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
>
> Hi Hongxu,
>
> I'm back on this issue. So yesterday I tried again to pick these
> patches, and got warnings again:
>
> https://valkyrie.yoctoproject.org/#/builders/29/builds/682
>
> If I relaunch the builds, sometimes they will success, sometimes I still
> get the warnings. Something seems to be intermittent here.
>
> I tried to reproduce locally, and again, I get the warnings in some
> intermittent way.
>
> My setup was:
> - Clean build directory, empty sstate cache.
> - The local.conf attached. This is mainly the configuration used for
>    https://valkyrie.yoctoproject.org/#/builders/8/builds/698, with some
>    minor changes (like disabling the shared sstate configuration).
> - Using meta, meta-poky and meta-yocto-bsp at this revision:
>    https://git.yoctoproject.org/poky-ci-archive/tag/?h=a-full-685
> - Used: bitbake core-image-sato core-image-sato-sdk core-image-minimal core-image-minimal-dev core-image-full-cmdline core-image-sato:do_populate_sdk
>
> Now here is how it went:
> - On my very first build, I got some SPDX warnings.
> - On my next build they were all gone.
> - I tried to tweak a bit my config, cleaned my cache again and still got
>    no warnings.
> - Then after some various modifications, I switched back again to my
>    very first configuration, and got again some SPDX warnings.
>
> So I'm sorry, I cannot give you some exact instructions to reproduce
> this issue, but still we have some intermittent warnings :(
Hi Mathieu,

Thanks for the information you provided, it helps me to reproduce two issues

1. While hosts with different archs (such as x86-64, aarch64) sharing
same sstate cache, build image from existed sstate cache reported
SPDX ID missing warnings. Here is steps:

On x86-64 host, build image core-image-minimal for qemuarm64
$ echo 'MACHINE = "qemuarm64"' >> conf/local.conf
$ bitbake core-image-sato

On aarch64 host, sharing above sstate cache and build image
core-image-minimal for qemuarm64
$ echo 'MACHINE = "qemuarm64"' >> conf/local.conf
$ bitbake core-image-sato

2. Sharing same sstate cache for different machines

$ echo 'MACHINE = "qemux86-64"' >> conf/local.conf
$ bitbake core-image-minimal

$ echo 'MACHINE = "qemuarm64"' >> conf/local.conf
$ bitbake core-image-minimal

The issue was caused by the following two patches,

[OE-core] [PATCH 2/4] meta/lib/oe/spdx30_tasks.py: fix hash link 
conflict while multiconfig enabled
[OE-core] [PATCH V2 3/4] gcc-cross.inc: add var-SSTATE_ARCHS_TUNEPKG to 
vardeps of do_create_spdx

I will resend review and drop them

//Hongxu



> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
>