* [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information
@ 2026-02-20 15:40 Joshua Watt
2026-02-20 15:40 ` [OE-core][PATCH 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt
` (9 more replies)
0 siblings, 10 replies; 113+ messages in thread
From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw)
To: openembedded-core; +Cc: benjamin.robin, Joshua Watt
Changes the SPDX 3 output to include a "recipe" package that describe
static information available at parse time (without building). This is
primarily useful for gathering SPDX 3 VEX information about some or all
recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and
vex.bbclass.
Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for
helping work through this.
Joshua Watt (9):
llvm-project-source: Use allarch.bbclass
gcc-source: Use allarch.bbclass
spdx3: Add recipe SPDX data
spdx3: Add recipe SBoM task
spdx3: Add is-native property
spdx30: Include patch file information in VEX
spdx: De-duplicate CreationInfo
spdx: Ignore ASSUME_PROVIDED recipes
spdx_common: Check for dependent task in task flags
meta/classes-global/sstate.bbclass | 4 +-
meta/classes-global/staging.bbclass | 2 +-
.../create-spdx-image-3.0.bbclass | 4 +-
.../create-spdx-sdk-3.0.bbclass | 4 +-
meta/classes-recipe/kernel.bbclass | 2 +-
meta/classes-recipe/nospdx.bbclass | 1 +
meta/classes/create-spdx-2.2.bbclass | 12 +-
meta/classes/create-spdx-3.0.bbclass | 77 ++-
meta/classes/spdx-common.bbclass | 14 +-
meta/lib/oe/sbom30.py | 192 ++++---
meta/lib/oe/spdx30.py | 2 +-
meta/lib/oe/spdx30_tasks.py | 487 +++++++++++++-----
meta/lib/oe/spdx_common.py | 16 +-
.../meta/meta-world-recipe-sbom.bb | 26 +
.../clang/llvm-project-source.inc | 8 +-
meta/recipes-devtools/gcc/gcc-source.inc | 16 +-
16 files changed, 620 insertions(+), 247 deletions(-)
create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb
--
2.53.0
^ permalink raw reply [flat|nested] 113+ messages in thread* [OE-core][PATCH 1/9] llvm-project-source: Use allarch.bbclass 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 2/9] gcc-source: " Joshua Watt ` (8 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/meta/recipes-devtools/clang/llvm-project-source.inc b/meta/recipes-devtools/clang/llvm-project-source.inc index 13e54efbc2..6bb595b7bc 100644 --- a/meta/recipes-devtools/clang/llvm-project-source.inc +++ b/meta/recipes-devtools/clang/llvm-project-source.inc @@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 2/9] gcc-source: Use allarch.bbclass 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data Joshua Watt ` (7 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 265bcf4bef..3ac679b1a6 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 2/9] gcc-source: " Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-22 7:59 ` Mathieu Dubois-Briand 2026-02-20 15:40 ` [OE-core][PATCH 4/9] spdx3: Add recipe SBoM task Joshua Watt ` (6 subsequent siblings) 9 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- meta/classes-global/staging.bbclass | 2 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 13 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ 10 files changed, 343 insertions(+), 152 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2fd29d7323..95c44f404e 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-global/staging.bbclass b/meta/classes-global/staging.bbclass index 1008867a6c..b35bc9037f 100644 --- a/meta/classes-global/staging.bbclass +++ b/meta/classes-global/staging.bbclass @@ -654,7 +654,7 @@ addtask do_prepare_recipe_sysroot before do_configure after do_fetch python staging_taskhandler() { EXCLUDED_TASKS = ( "do_prepare_recipe_sysroot", - "do_create_spdx", + "do_create_recipe_spdx", ) bbtasks = e.tasklist for task in bbtasks: diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index f989b31c47..3a2c20dec2 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() { bb.error(f"Failed to parse kernel config file: {e}") build, build_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build + d, "builds", f"build-{pn}", oe.spdx30.build_Build ) kernel_build = build_objset.add_root( diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..f1ee0f9afd 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..cd70a07534 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,19 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} +# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source addtask do_create_spdx after \ + do_unpack \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +200,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +231,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..0c1fd09b6f 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -97,8 +98,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +107,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data 2026-02-20 15:40 ` [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data Joshua Watt @ 2026-02-22 7:59 ` Mathieu Dubois-Briand 0 siblings, 0 replies; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-02-22 7:59 UTC (permalink / raw) To: JPEWhacker, openembedded-core; +Cc: benjamin.robin On Fri Feb 20, 2026 at 4:40 PM CET, Joshua Watt via lists.openembedded.org wrote: > Adds a new package to the SPDX output that represents the recipe data > for a given recipe. Importantly, this data contains only things that can > be determined statically from only the recipe, so it doesn't require > fetching or building anything. This means that build time dependencies > and CVE information for recipes can be analyzed without needing to > actually do any builds. > > Sadly, license data cannot be included because NO_GENERIC_LICENSE means > that actual license text might only be available after do_fetch > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- Hi Joshua, It looks like this is causing some failures, at list for glib: ERROR: glib-2.0-1_2.86.4-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 619, function: extend_recipe_sysroot 0615: if "/bin/" in l or "/sbin/" in l: 0616: # defer /*bin/* files until last in case they need libs 0617: binfiles[l] = (targetdir, dest) 0618: else: *** 0619: staging_copyfile(l, targetdir, dest, postinsts, seendirs) 0620: 0621: # Handle deferred binfiles 0622: for l in binfiles: 0623: (targetdir, dest) = binfiles[l] File: '/srv/pokybuild/yocto-worker/musl-qemux86/build/layers/openembedded-core/meta/classes-global/staging.bbclass', lineno: 165, function: staging_copyfile 0161: os.symlink(linkto, dest) 0162: #bb.warn(c) 0163: else: 0164: try: *** 0165: os.link(c, dest) 0166: except OSError as err: 0167: if err.errno == errno.EXDEV: 0168: bb.utils.copyfile(c, dest) 0169: else: Exception: FileExistsError: [Errno 17] File exists: '/srv/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/sysroots-components/core2-32/glib-2.0/usr/include/glib-2.0/glib.h' -> '/srv/pokybuild/yocto-worker/musl-qemux86/build/build/tmp/work/core2-32-poky-linux-musl/glib-2.0/2.86.4/recipe-sysroot/usr/include/glib-2.0/glib.h' https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/3240 https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/3220 https://autobuilder.yoctoproject.org/valkyrie/#/builders/65/builds/3201 https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3296 And probably also this one: Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx ERROR: Command execution failed: 1 https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3219 Can you have a look? Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH 4/9] spdx3: Add recipe SBoM task 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (2 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 5/9] spdx3: Add is-native property Joshua Watt ` (5 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ``` Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 28 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++++ .../meta/meta-world-recipe-sbom.bb | 26 +++++++++++++++++ 4 files changed, 65 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index cd70a07534..e12c116486 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -240,6 +240,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 0c1fd09b6f..6f35dbf8f6 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..9a312a870d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("PN") + "-recipe-sbom" + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..60209fba7e --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,26 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + for p in world_target: + if p == self_pn or p in exclude: + continue + deps.append(p) +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 5/9] spdx3: Add is-native property 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (3 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 4/9] spdx3: Add recipe SBoM task Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 6/9] spdx30: Include patch file information in VEX Joshua Watt ` (4 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 9a312a870d..fff1ca6bea 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 6/9] spdx30: Include patch file information in VEX 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (4 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 5/9] spdx3: Add is-native property Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 7/9] spdx: De-duplicate CreationInfo Joshua Watt ` (3 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index fff1ca6bea..1c9346128c 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 7/9] spdx: De-duplicate CreationInfo 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (5 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 6/9] spdx30: Include patch file information in VEX Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 8/9] spdx: Ignore ASSUME_PROVIDED recipes Joshua Watt ` (2 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 8/9] spdx: Ignore ASSUME_PROVIDED recipes 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (6 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 7/9] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 9/9] spdx_common: Check for dependent task in task flags Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Ignore recipes in ASSUME_PROVIDED when generating SPDX dependencies, since these recipes are not actually built Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..03863ef362 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -104,9 +104,12 @@ def collect_direct_deps(d, dep_task): deps = set() + ignore = set(d.getVar("ASSUME_PROVIDED").split()) + ignore.add(pn) + for dep_name in this_dep.deps: dep_data = taskdepdata[dep_name] - if dep_data.taskname == dep_task and dep_data.pn != pn: + if dep_data.taskname == dep_task and dep_data.pn not in ignore: deps.add((dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps)) return sorted(deps) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH 9/9] spdx_common: Check for dependent task in task flags 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (7 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 8/9] spdx: Ignore ASSUME_PROVIDED recipes Joshua Watt @ 2026-02-20 15:40 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-20 15:40 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 03863ef362..9d39b5e9ac 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt ` (8 preceding siblings ...) 2026-02-20 15:40 ` [OE-core][PATCH 9/9] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt ` (9 more replies) 9 siblings, 10 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). Joshua Watt (8): llvm-project-source: Use allarch.bbclass gcc-source: Use allarch.bbclass spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 75 ++- meta/classes/spdx-common.bbclass | 16 +- meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 487 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + .../meta/meta-world-recipe-sbom.bb | 26 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 15 files changed, 615 insertions(+), 245 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb -- 2.53.0 ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 1/8] llvm-project-source: Use allarch.bbclass 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 2/8] gcc-source: " Joshua Watt ` (8 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/meta/recipes-devtools/clang/llvm-project-source.inc b/meta/recipes-devtools/clang/llvm-project-source.inc index 13e54efbc2..6bb595b7bc 100644 --- a/meta/recipes-devtools/clang/llvm-project-source.inc +++ b/meta/recipes-devtools/clang/llvm-project-source.inc @@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 2/8] gcc-source: Use allarch.bbclass 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 3/8] spdx3: Add recipe SPDX data Joshua Watt ` (7 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 265bcf4bef..3ac679b1a6 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 3/8] spdx3: Add recipe SPDX data 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 2/8] gcc-source: " Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 4/8] spdx3: Add recipe SBoM task Joshua Watt ` (6 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 49 ++- meta/classes/spdx-common.bbclass | 15 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ 9 files changed, 342 insertions(+), 151 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2fd29d7323..95c44f404e 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index f989b31c47..3a2c20dec2 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() { bb.error(f"Failed to parse kernel config file: {e}") build, build_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build + d, "builds", f"build-{pn}", oe.spdx30.build_Build ) kernel_build = build_objset.add_root( diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..f1ee0f9afd 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..61223ee0a5 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,17 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} addtask do_create_spdx after \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +198,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +229,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..2804c27b0b 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -72,6 +73,8 @@ def create_spdx_source_deps(d): # ourselves if oe.spdx_common.has_task(d, "do_unpack"): deps.append("%s:do_unpack" % pn) + if oe.spdx_common.has_task(d, "do_patch"): + deps.append("%s:do_patch" % pn) if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): @@ -97,8 +100,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +109,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 4/8] spdx3: Add recipe SBoM task 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (2 preceding siblings ...) 2026-02-24 23:00 ` [OE-core][PATCH v2 3/8] spdx3: Add recipe SPDX data Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 5/8] spdx3: Add is-native property Joshua Watt ` (5 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ``` Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 28 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++++ .../meta/meta-world-recipe-sbom.bb | 26 +++++++++++++++++ 4 files changed, 65 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 61223ee0a5..fdd0c60690 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -238,6 +238,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 2804c27b0b..1ec4877a6a 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..9a312a870d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("PN") + "-recipe-sbom" + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..60209fba7e --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,26 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + for p in world_target: + if p == self_pn or p in exclude: + continue + deps.append(p) +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 5/8] spdx3: Add is-native property 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (3 preceding siblings ...) 2026-02-24 23:00 ` [OE-core][PATCH v2 4/8] spdx3: Add recipe SBoM task Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 6/8] spdx30: Include patch file information in VEX Joshua Watt ` (4 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 9a312a870d..fff1ca6bea 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 6/8] spdx30: Include patch file information in VEX 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (4 preceding siblings ...) 2026-02-24 23:00 ` [OE-core][PATCH v2 5/8] spdx3: Add is-native property Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 7/8] spdx: De-duplicate CreationInfo Joshua Watt ` (3 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index fff1ca6bea..1c9346128c 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 7/8] spdx: De-duplicate CreationInfo 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (5 preceding siblings ...) 2026-02-24 23:00 ` [OE-core][PATCH v2 6/8] spdx30: Include patch file information in VEX Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 8/8] spdx_common: Check for dependent task in task flags Joshua Watt ` (2 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v2 8/8] spdx_common: Check for dependent task in task flags 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (6 preceding siblings ...) 2026-02-24 23:00 ` [OE-core][PATCH v2 7/8] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-02-24 23:00 ` Joshua Watt 2026-02-26 12:52 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-24 23:00 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..3aaf2a9c8b 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (7 preceding siblings ...) 2026-02-24 23:00 ` [OE-core][PATCH v2 8/8] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-02-26 12:52 ` Mathieu Dubois-Briand 2026-02-26 14:27 ` Benjamin Robin 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt 9 siblings, 1 reply; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-02-26 12:52 UTC (permalink / raw) To: JPEWhacker, openembedded-core; +Cc: benjamin.robin, ross.burton On Wed Feb 25, 2026 at 12:00 AM CET, Joshua Watt via lists.openembedded.org wrote: > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > Hi Joshua, Thanks for the new version, but I believe we still have some issues. A first one: ERROR: docbook-xml-dtd4-native-4.5-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated: ... File: '/srv/pokybuild/yocto-worker/genericx86/build/layers/openembedded-core/meta/lib/oe/spdx30_tasks.py', lineno: 89, function: add_license_text 0085: # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set 0086: filename = d.getVarFlag("NO_GENERIC_LICENSE", name) 0087: if filename: 0088: filename = d.expand("${S}/" + filename) *** 0089: with open(filename, errors="replace") as f: 0090: lic.simplelicensing_licenseText = f.read() 0091: return lic 0092: else: 0093: bb.fatal("Cannot find any text for license %s" % name) Exception: FileNotFoundError: [Errno 2] No such file or directory: '/srv/pokybuild/yocto-worker/genericx86/build/build/tmp/work/x86_64-linux/docbook-xml-dtd4-native/4.5/sources/LICENSE-OASIS' https://autobuilder.yoctoproject.org/valkyrie/#/builders/19/builds/3253 https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/3217 https://autobuilder.yoctoproject.org/valkyrie/#/builders/74/builds/3224 https://autobuilder.yoctoproject.org/valkyrie/#/builders/95/builds/3204 And a second one: Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx ERROR: Command execution failed: 1 https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086 https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253 Can you have a look at these? Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information 2026-02-26 12:52 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand @ 2026-02-26 14:27 ` Benjamin Robin 2026-02-26 15:09 ` Benjamin Robin 0 siblings, 1 reply; 113+ messages in thread From: Benjamin Robin @ 2026-02-26 14:27 UTC (permalink / raw) To: JPEWhacker, openembedded-core, Mathieu Dubois-Briand; +Cc: ross.burton On Thursday, February 26, 2026 at 1:52 PM, Mathieu Dubois-Briand wrote: > And a second one: > > Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx > ERROR: Command execution failed: 1 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253 For the second issue, it can be triggered by adding these lines in local.conf: MACHINE = "qemux86" OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32' With this configuration these 2 commands failed: - bitbake -c create_recipe_sbom meta-world-recipe-sbom - bitbake world -- Benjamin Robin, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information 2026-02-26 14:27 ` Benjamin Robin @ 2026-02-26 15:09 ` Benjamin Robin 2026-02-26 15:41 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Benjamin Robin @ 2026-02-26 15:09 UTC (permalink / raw) To: JPEWhacker, openembedded-core, Mathieu Dubois-Briand; +Cc: ross.burton On Thursday, February 26, 2026 at 3:27 PM, Benjamin Robin wrote: > On Thursday, February 26, 2026 at 1:52 PM, Mathieu Dubois-Briand wrote: > > And a second one: > > > > Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx > > ERROR: Command execution failed: 1 > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086 > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253 > > For the second issue, it can be triggered by adding these lines in local.conf: > > MACHINE = "qemux86" > OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32' > > With this configuration these 2 commands failed: > - bitbake -c create_recipe_sbom meta-world-recipe-sbom > - bitbake world I am proposing this fix for meta/recipes-core/meta/meta-world-recipe-sbom.bb: python calculate_extra_depends() { exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) exclude.add(self_pn) for variant in ('${MULTILIB_VARIANTS}' or "").split(): exclude.add(f"{variant}-{self_pn}") for p in world_target: if p in exclude: continue deps.append(p) } This fix solve the second issue (circular dependency). Best regards, -- Benjamin Robin, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information 2026-02-26 15:09 ` Benjamin Robin @ 2026-02-26 15:41 ` Joshua Watt 0 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 15:41 UTC (permalink / raw) To: Benjamin Robin; +Cc: openembedded-core, Mathieu Dubois-Briand, ross.burton On Thu, Feb 26, 2026 at 8:09 AM Benjamin Robin <benjamin.robin@bootlin.com> wrote: > > On Thursday, February 26, 2026 at 3:27 PM, Benjamin Robin wrote: > > On Thursday, February 26, 2026 at 1:52 PM, Mathieu Dubois-Briand wrote: > > > And a second one: > > > > > > Initialising tasks...ERROR: Task virtual:multilib:lib32:/srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx has circular dependency on /srv/pokybuild/yocto-worker/qemux86-world-alt/build/layers/openembedded-core/meta/recipes-core/meta/meta-world-recipe-sbom.bb:do_create_spdx > > > ERROR: Command execution failed: 1 > > > > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3086 > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/3253 > > > > For the second issue, it can be triggered by adding these lines in local.conf: > > > > MACHINE = "qemux86" > > OE_FRAGMENTS += 'core/yocto-autobuilder/multilib-x86-lib32' > > > > With this configuration these 2 commands failed: > > - bitbake -c create_recipe_sbom meta-world-recipe-sbom > > - bitbake world > > I am proposing this fix for meta/recipes-core/meta/meta-world-recipe-sbom.bb: > > python calculate_extra_depends() { > exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) > exclude.add(self_pn) > > for variant in ('${MULTILIB_VARIANTS}' or "").split(): > exclude.add(f"{variant}-{self_pn}") > > for p in world_target: > if p in exclude: > continue > deps.append(p) > } Thanks, I'll add this to my branch > > This fix solve the second issue (circular dependency). > > Best regards, > -- > Benjamin Robin, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > > > ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 0/8] Add SPDX 3 Recipe Information 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt ` (8 preceding siblings ...) 2026-02-26 12:52 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt ` (9 more replies) 9 siblings, 10 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run). Joshua Watt (8): llvm-project-source: Use allarch.bbclass gcc-source: Use allarch.bbclass spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 76 ++- meta/classes/spdx-common.bbclass | 16 +- meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 487 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + .../meta/meta-world-recipe-sbom.bb | 28 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 15 files changed, 618 insertions(+), 245 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb -- 2.53.0 ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 1/8] llvm-project-source: Use allarch.bbclass 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 2/8] gcc-source: " Joshua Watt ` (8 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/meta/recipes-devtools/clang/llvm-project-source.inc b/meta/recipes-devtools/clang/llvm-project-source.inc index 13e54efbc2..6bb595b7bc 100644 --- a/meta/recipes-devtools/clang/llvm-project-source.inc +++ b/meta/recipes-devtools/clang/llvm-project-source.inc @@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 2/8] gcc-source: Use allarch.bbclass 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 3/8] spdx3: Add recipe SPDX data Joshua Watt ` (7 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 265bcf4bef..3ac679b1a6 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 3/8] spdx3: Add recipe SPDX data 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 2/8] gcc-source: " Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 4/8] spdx3: Add recipe SBoM task Joshua Watt ` (6 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 50 ++- meta/classes/spdx-common.bbclass | 15 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ 9 files changed, 343 insertions(+), 151 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2fd29d7323..95c44f404e 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index f989b31c47..3a2c20dec2 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -904,7 +904,7 @@ python do_create_kernel_config_spdx() { bb.error(f"Failed to parse kernel config file: {e}") build, build_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", f"recipe-{pn}", oe.spdx30.build_Build + d, "builds", f"build-{pn}", oe.spdx30.build_Build ) kernel_build = build_objset.add_root( diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..f1ee0f9afd 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..c5f6462c5a 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,18 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} addtask do_create_spdx after \ + do_unpack \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +199,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +230,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..2804c27b0b 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -72,6 +73,8 @@ def create_spdx_source_deps(d): # ourselves if oe.spdx_common.has_task(d, "do_unpack"): deps.append("%s:do_unpack" % pn) + if oe.spdx_common.has_task(d, "do_patch"): + deps.append("%s:do_patch" % pn) if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): @@ -97,8 +100,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +109,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 4/8] spdx3: Add recipe SBoM task 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (2 preceding siblings ...) 2026-02-26 17:33 ` [OE-core][PATCH v3 3/8] spdx3: Add recipe SPDX data Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 5/8] spdx3: Add is-native property Joshua Watt ` (5 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ``` Co-authored-by: Benjamin Robin <benjamin.robin@bootlin.com> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 28 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++++ .../meta/meta-world-recipe-sbom.bb | 28 +++++++++++++++++++ 4 files changed, 67 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index c5f6462c5a..782aa2c52e 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -239,6 +239,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 2804c27b0b..1ec4877a6a 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..9a312a870d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("PN") + "-recipe-sbom" + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..5731e27c83 --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,28 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 5/8] spdx3: Add is-native property 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (3 preceding siblings ...) 2026-02-26 17:33 ` [OE-core][PATCH v3 4/8] spdx3: Add recipe SBoM task Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 6/8] spdx30: Include patch file information in VEX Joshua Watt ` (4 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 9a312a870d..fff1ca6bea 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 6/8] spdx30: Include patch file information in VEX 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (4 preceding siblings ...) 2026-02-26 17:33 ` [OE-core][PATCH v3 5/8] spdx3: Add is-native property Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 7/8] spdx: De-duplicate CreationInfo Joshua Watt ` (3 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index fff1ca6bea..1c9346128c 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 7/8] spdx: De-duplicate CreationInfo 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (5 preceding siblings ...) 2026-02-26 17:33 ` [OE-core][PATCH v3 6/8] spdx30: Include patch file information in VEX Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 8/8] spdx_common: Check for dependent task in task flags Joshua Watt ` (2 subsequent siblings) 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v3 8/8] spdx_common: Check for dependent task in task flags 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (6 preceding siblings ...) 2026-02-26 17:33 ` [OE-core][PATCH v3 7/8] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-02-26 17:33 ` Joshua Watt 2026-02-27 7:32 ` [OE-core][PATCH v3 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt 9 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-02-26 17:33 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..3aaf2a9c8b 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v3 0/8] Add SPDX 3 Recipe Information 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (7 preceding siblings ...) 2026-02-26 17:33 ` [OE-core][PATCH v3 8/8] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-02-27 7:32 ` Mathieu Dubois-Briand 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt 9 siblings, 0 replies; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-02-27 7:32 UTC (permalink / raw) To: JPEWhacker, openembedded-core; +Cc: benjamin.robin, ross.burton On Thu Feb 26, 2026 at 6:33 PM CET, Joshua Watt via lists.openembedded.org wrote: > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > > V2: Fixes a bug where do_populate_sysroot was running when it should not > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > incorrect (this is already handled by bitbake in the taskgraph, and > doesn't need to be manually removed). > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > dependency. meta-world-recipe-sbom also no longer runs in world builds, > as there's no reason to this. Finally, fixes a bug where > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > (because do_unpack was not run). > Hi Joshua, Thanks for new version, results looks way better overall, but we still have a few errors. I now have this issue on some builds: ERROR: nativesdk-sdk-provides-dummy-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-nativesdk-sdk-provides-dummy https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3335 https://autobuilder.yoctoproject.org/valkyrie/#/builders/45/builds/1105 https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3238 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3233 And some errors in oe-selftests: 2026-02-26 20:46:06,214 - oe-selftest - INFO - newlib.NewlibTest.test_newlib (subunit.RemotedTestCase) 2026-02-26 20:46:06,215 - oe-selftest - INFO - ... FAIL ... ERROR: gcc-cross-x86_64-15.2.0-r0 do_create_spdx: Could not find a builds SPDX document named build-linux-libc-headers ... ERROR: sysroot-test-1.0-r0 do_create_spdx: Could not find a builds SPDX document named build-sysroot-test-arch1 ... 2026-02-26 21:03:31,870 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase) 2026-02-26 21:03:31,870 - oe-selftest - INFO - ... FAIL ... 2026-02-26 21:20:09,006 - oe-selftest - INFO - spdx.SPDX30Check.test_custom_annotation_vars (subunit.RemotedTestCase) 2026-02-26 21:20:09,006 - oe-selftest - INFO - ... FAIL ... 2026-02-26 21:20:09,006 - oe-selftest - INFO - 4: 37/50 612/670 (18.90s) (0 failed) (spdx.SPDX30Check.test_custom_annotation_vars) 2026-02-26 21:20:09,006 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 306, in test_custom_annotation_vars objset = self.check_recipe_spdx( "base-files", ...<7 lines>... ), ) File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx return self.check_spdx_file(filename) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file self.assertExists(filename) ~~~~~~~~~~~~~~~~~^^^^^^^^^^ File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists raise self.failureException(msg) AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/deploy/spdx/3.0.1/qemux86_64/recipes/recipe-base-files.spdx.json' does not exist ... ERROR: gawk-native-5.3.2-r0 do_create_spdx: Applying patch '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/recipes-extended/gawk/gawk/0001-configure.ac-re-enable-disabled-printf-features.patch' on target directory '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/gawk-native/5.3.2/spdx/3.0.1/work/sources/gawk-5.3.2' CmdError('quilt --quiltrc /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/gawk-native/5.3.2/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout: stderr: /bin/sh: 1: quilt: not found ') ERROR: bzip2-native-1.0.8-r0 do_create_spdx: Applying patch '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/recipes-extended/bzip2/bzip2/0001-fix-bzip2-version-tmp-aaa-will-hang.patch' on target directory '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/bzip2-native/1.0.8/spdx/3.0.1/work/sources/bzip2-1.0.8' CmdError('quilt --quiltrc /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700993/tmp/work/x86_64-linux/bzip2-native/1.0.8/recipe-sysroot-native/etc/quiltrc push', 0, 'stdout: stderr: /bin/sh: 1: quilt: not found ') ... 2026-02-26 21:27:01,979 - oe-selftest - INFO - spdx.SPDX30Check.test_gcc_include_source (subunit.RemotedTestCase) 2026-02-26 21:27:01,980 - oe-selftest - INFO - ... FAIL ... 2026-02-26 21:27:26,620 - oe-selftest - INFO - spdx.SPDX30Check.test_kernel_config_spdx (subunit.RemotedTestCase) 2026-02-26 21:27:26,621 - oe-selftest - INFO - ... FAIL ... 2026-02-26 21:27:50,868 - oe-selftest - INFO - spdx.SPDX30Check.test_packageconfig_spdx (subunit.RemotedTestCase) 2026-02-26 21:27:50,868 - oe-selftest - INFO - ... FAIL ... Also, it looks like you did not add yourself as a maintainer of meta-world-recipe-sbom. https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3288 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3405 https://autobuilder.yoctoproject.org/valkyrie/#/builders/37/builds/3463 Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 0/9] Add SPDX 3 Recipe Information 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt ` (8 preceding siblings ...) 2026-02-27 7:32 ` [OE-core][PATCH v3 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt ` (11 more replies) 9 siblings, 12 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run). V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output. Joshua Watt (9): llvm-project-source: Use allarch.bbclass gcc-source: Use allarch.bbclass spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags spdx30: Skip install package CVE information meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 92 +++- meta/classes/spdx-common.bbclass | 22 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 488 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../meta/meta-world-recipe-sbom.bb | 29 ++ .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 17 files changed, 669 insertions(+), 260 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb -- 2.53.0 ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 1/9] llvm-project-source: Use allarch.bbclass 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 2/9] gcc-source: " Joshua Watt ` (10 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/meta/recipes-devtools/clang/llvm-project-source.inc b/meta/recipes-devtools/clang/llvm-project-source.inc index 13e54efbc2..6bb595b7bc 100644 --- a/meta/recipes-devtools/clang/llvm-project-source.inc +++ b/meta/recipes-devtools/clang/llvm-project-source.inc @@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 2/9] gcc-source: Use allarch.bbclass 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 3/9] spdx3: Add recipe SPDX data Joshua Watt ` (9 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 265bcf4bef..3ac679b1a6 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 3/9] spdx3: Add recipe SPDX data 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 2/9] gcc-source: " Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 4/9] spdx3: Add recipe SBoM task Joshua Watt ` (8 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2fd29d7323..95c44f404e 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index 22568d6e9c..6621a17f85 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -895,7 +895,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build: diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..3288cdf75a 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..672ca27cd0 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,19 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} addtask do_create_spdx after \ + do_unpack \ + do_patch \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +200,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +231,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..3c239a718b 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code @@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn) @@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 5830d7c087..759ca86b73 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", @@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """, @@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self): @@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}" @@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}") @@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """, -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 4/9] spdx3: Add recipe SBoM task 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (2 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 3/9] spdx3: Add recipe SPDX data Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 5/9] spdx3: Add is-native property Joshua Watt ` (7 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ``` Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 32 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 ++++++ meta/lib/oeqa/selftest/cases/spdx.py | 10 ++++++ .../meta/meta-world-recipe-sbom.bb | 29 +++++++++++++++++ 6 files changed, 83 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 672ca27cd0..c3ea95b8bc 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0" @@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3c239a718b..abf2332bee 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc index b5ab35d92a..5bea863798 100644 --- a/meta/conf/distro/include/maintainers.inc +++ b/meta/conf/distro/include/maintainers.inc @@ -532,6 +532,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <richard.purdie@linuxfoundation.org>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <JPEWhacker@gmail.com>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <yi.zhao@windriver.com>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <unassigned@yoctoproject.org>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <unassigned@yoctoproject.org>" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..b6c917045e 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 759ca86b73..efee0214fc 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + task="create_recipe_sbom", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..b47a3229c9 --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,29 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 5/9] spdx3: Add is-native property 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (3 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 4/9] spdx3: Add recipe SBoM task Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 6/9] spdx30: Include patch file information in VEX Joshua Watt ` (6 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index b6c917045e..a8fffbb085 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 6/9] spdx30: Include patch file information in VEX 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (4 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 5/9] spdx3: Add is-native property Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 7/9] spdx: De-duplicate CreationInfo Joshua Watt ` (5 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8fffbb085..aec47d4f81 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 7/9] spdx: De-duplicate CreationInfo 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (5 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 6/9] spdx30: Include patch file information in VEX Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 8/9] spdx_common: Check for dependent task in task flags Joshua Watt ` (4 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 8/9] spdx_common: Check for dependent task in task flags 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (6 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 7/9] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 9/9] spdx30: Skip install package CVE information Joshua Watt ` (3 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..3aaf2a9c8b 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v4 9/9] spdx30: Skip install package CVE information 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (7 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 8/9] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-03-03 0:43 ` Joshua Watt 2026-03-03 10:17 ` [OE-core][PATCH v4 0/9] Add SPDX 3 Recipe Information Antonin Godard ` (2 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-03 0:43 UTC (permalink / raw) To: openembedded-core; +Cc: benjamin.robin, ross.burton, Joshua Watt Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document. If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index c3ea95b8bc..88b7ef9f42 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)." +SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information." + SPDX_INCLUDE_TIMESTAMPS ?= "0" SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \ useful if you want to know when artifacts were produced and when builds \ diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index aec47d4f81..887fac813a 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -771,27 +771,28 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): + if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1": for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) + for cve in rel.to: + vex_statements[cve] = [] + vex_patches[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index efee0214fc..f1ea2694cf 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_package_vex(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf="""\ + SPDX_PACKAGE_INCLUDE_VEX = "1" + """, + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v4 0/9] Add SPDX 3 Recipe Information 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (8 preceding siblings ...) 2026-03-03 0:43 ` [OE-core][PATCH v4 9/9] spdx30: Skip install package CVE information Joshua Watt @ 2026-03-03 10:17 ` Antonin Godard 2026-03-03 14:08 ` Mathieu Dubois-Briand 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt 11 siblings, 0 replies; 113+ messages in thread From: Antonin Godard @ 2026-03-03 10:17 UTC (permalink / raw) To: JPEWhacker, openembedded-core; +Cc: benjamin.robin, ross.burton Hi, On Tue Mar 3, 2026 at 1:43 AM CET, Joshua Watt via lists.openembedded.org wrote: > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. Once merged, would it be possible to submit a docs patch to document the new variables in the variable glossary, and possibly update documentation/dev-manual/sbom.rst to document this new feature? Also, looking at the last patch, the CVE information removal seems worthy of a migration note in documentation/migration-guides/migration-6.0.rst? Antonin ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v4 0/9] Add SPDX 3 Recipe Information 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (9 preceding siblings ...) 2026-03-03 10:17 ` [OE-core][PATCH v4 0/9] Add SPDX 3 Recipe Information Antonin Godard @ 2026-03-03 14:08 ` Mathieu Dubois-Briand 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt 11 siblings, 0 replies; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-03-03 14:08 UTC (permalink / raw) To: JPEWhacker, openembedded-core; +Cc: benjamin.robin, ross.burton On Tue Mar 3, 2026 at 1:43 AM CET, Joshua Watt via lists.openembedded.org wrote: > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > > V2: Fixes a bug where do_populate_sysroot was running when it should not > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > incorrect (this is already handled by bitbake in the taskgraph, and > doesn't need to be manually removed). > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > dependency. meta-world-recipe-sbom also no longer runs in world builds, > as there's no reason to this. Finally, fixes a bug where > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > (because do_unpack was not run). > > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > information is linked to binary packages, or just recipes. Defaults to > "0" to significantly reduce the size of the SPDX output. > > Joshua Watt (9): > llvm-project-source: Use allarch.bbclass > gcc-source: Use allarch.bbclass > spdx3: Add recipe SPDX data > spdx3: Add recipe SBoM task > spdx3: Add is-native property > spdx30: Include patch file information in VEX > spdx: De-duplicate CreationInfo > spdx_common: Check for dependent task in task flags > spdx30: Skip install package CVE information > > meta/classes-global/sstate.bbclass | 4 +- > .../create-spdx-image-3.0.bbclass | 4 +- > .../create-spdx-sdk-3.0.bbclass | 4 +- > meta/classes-recipe/kernel.bbclass | 2 +- > meta/classes-recipe/nospdx.bbclass | 1 + > meta/classes/create-spdx-2.2.bbclass | 12 +- > meta/classes/create-spdx-3.0.bbclass | 92 +++- > meta/classes/spdx-common.bbclass | 22 +- > meta/conf/distro/include/maintainers.inc | 1 + > meta/lib/oe/sbom30.py | 192 ++++--- > meta/lib/oe/spdx30.py | 2 +- > meta/lib/oe/spdx30_tasks.py | 488 +++++++++++++----- > meta/lib/oe/spdx_common.py | 11 + > meta/lib/oeqa/selftest/cases/spdx.py | 41 +- > .../meta/meta-world-recipe-sbom.bb | 29 ++ > .../clang/llvm-project-source.inc | 8 +- > meta/recipes-devtools/gcc/gcc-source.inc | 16 +- > 17 files changed, 669 insertions(+), 260 deletions(-) > create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb Hi Joshua, Thanks for the new version, but it looks like one of the two errors is still present on several builds: ERROR: nativesdk-sdk-provides-dummy-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-nativesdk-sdk-provides-dummy https://autobuilder.yoctoproject.org/valkyrie/#/builders/16/builds/3310 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3262 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3282 https://autobuilder.yoctoproject.org/valkyrie/#/builders/40/builds/3267 ... Can you have a look at these? Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 00/13] Add SPDX 3 Recipe Information 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt ` (10 preceding siblings ...) 2026-03-03 14:08 ` Mathieu Dubois-Briand @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 01/13] llvm-project-source: Use allarch.bbclass Joshua Watt ` (14 more replies) 11 siblings, 15 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run). V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output. V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway Joshua Watt (13): llvm-project-source: Use allarch.bbclass gcc-source: Use allarch.bbclass spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags spdx30: Skip install package CVE information dummy-sdk-package: Disable SPDX spdx: Remove fatal errors for missing providers spdx3: Use common variable for vardeps glibc-testsuite: Do not generate SPDX meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 15 +- meta/classes/create-spdx-3.0.bbclass | 87 ++- meta/classes/spdx-common.bbclass | 22 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 192 ++++--- meta/lib/oe/spdx30.py | 2 +- meta/lib/oe/spdx30_tasks.py | 496 +++++++++++++----- meta/lib/oe/spdx_common.py | 11 + meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../glibc/glibc-testsuite_2.42.bb | 1 + meta/recipes-core/meta/dummy-sdk-package.inc | 1 + .../meta/meta-world-recipe-sbom.bb | 29 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- 19 files changed, 667 insertions(+), 270 deletions(-) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb -- 2.53.0 ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 01/13] llvm-project-source: Use allarch.bbclass 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 02/13] gcc-source: " Joshua Watt ` (13 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/meta/recipes-devtools/clang/llvm-project-source.inc b/meta/recipes-devtools/clang/llvm-project-source.inc index 13e54efbc2..6bb595b7bc 100644 --- a/meta/recipes-devtools/clang/llvm-project-source.inc +++ b/meta/recipes-devtools/clang/llvm-project-source.inc @@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 02/13] gcc-source: Use allarch.bbclass 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 01/13] llvm-project-source: Use allarch.bbclass Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 03/13] spdx3: Add recipe SPDX data Joshua Watt ` (12 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 265bcf4bef..3ac679b1a6 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 03/13] spdx3: Add recipe SPDX data 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 01/13] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 02/13] gcc-source: " Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 04/13] spdx3: Add recipe SBoM task Joshua Watt ` (11 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2fd29d7323..95c44f404e 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index 22568d6e9c..6621a17f85 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -895,7 +895,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build: diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..3288cdf75a 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..672ca27cd0 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,19 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} addtask do_create_spdx after \ + do_unpack \ + do_patch \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +200,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +231,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..3c239a718b 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code @@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn) @@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 5830d7c087..759ca86b73 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", @@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """, @@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self): @@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}" @@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}") @@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """, -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 04/13] spdx3: Add recipe SBoM task 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (2 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 03/13] spdx3: Add recipe SPDX data Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 05/13] spdx3: Add is-native property Joshua Watt ` (10 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ``` Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 32 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 ++++++ meta/lib/oeqa/selftest/cases/spdx.py | 10 ++++++ .../meta/meta-world-recipe-sbom.bb | 29 +++++++++++++++++ 6 files changed, 83 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 672ca27cd0..c3ea95b8bc 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0" @@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3c239a718b..abf2332bee 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc index b5ab35d92a..5bea863798 100644 --- a/meta/conf/distro/include/maintainers.inc +++ b/meta/conf/distro/include/maintainers.inc @@ -532,6 +532,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <richard.purdie@linuxfoundation.org>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <JPEWhacker@gmail.com>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <yi.zhao@windriver.com>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <unassigned@yoctoproject.org>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <unassigned@yoctoproject.org>" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..b6c917045e 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 759ca86b73..efee0214fc 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + task="create_recipe_sbom", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..b47a3229c9 --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,29 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 05/13] spdx3: Add is-native property 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (3 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 04/13] spdx3: Add recipe SBoM task Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 06/13] spdx30: Include patch file information in VEX Joshua Watt ` (9 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index b6c917045e..a8fffbb085 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 06/13] spdx30: Include patch file information in VEX 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (4 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 05/13] spdx3: Add is-native property Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 07/13] spdx: De-duplicate CreationInfo Joshua Watt ` (8 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8fffbb085..aec47d4f81 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 07/13] spdx: De-duplicate CreationInfo 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (5 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 06/13] spdx30: Include patch file information in VEX Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 08/13] spdx_common: Check for dependent task in task flags Joshua Watt ` (7 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 08/13] spdx_common: Check for dependent task in task flags 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (6 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 07/13] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 09/13] spdx30: Skip install package CVE information Joshua Watt ` (6 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..3aaf2a9c8b 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 09/13] spdx30: Skip install package CVE information 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (7 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 08/13] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 10/13] dummy-sdk-package: Disable SPDX Joshua Watt ` (5 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document. If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index c3ea95b8bc..88b7ef9f42 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)." +SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information." + SPDX_INCLUDE_TIMESTAMPS ?= "0" SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \ useful if you want to know when artifacts were produced and when builds \ diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index aec47d4f81..887fac813a 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -771,27 +771,28 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): + if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1": for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) + for cve in rel.to: + vex_statements[cve] = [] + vex_patches[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index efee0214fc..f1ea2694cf 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_package_vex(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf="""\ + SPDX_PACKAGE_INCLUDE_VEX = "1" + """, + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 10/13] dummy-sdk-package: Disable SPDX 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (8 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 09/13] spdx30: Skip install package CVE information Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 11/13] spdx: Remove fatal errors for missing providers Joshua Watt ` (4 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc index bf453cac9b..71e788b0b9 100644 --- a/meta/recipes-core/meta/dummy-sdk-package.inc +++ b/meta/recipes-core/meta/dummy-sdk-package.inc @@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 11/13] spdx: Remove fatal errors for missing providers 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (9 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 10/13] dummy-sdk-package: Disable SPDX Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 12/13] spdx3: Use common variable for vardeps Joshua Watt ` (3 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt When creating images and SDKs, do not error on missing providers. This allows recipes to use the `nospdx` inherit to prevent SPDX from being generated, but not result in an error when assembling the final image. Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-2.2.bbclass | 3 ++- meta/lib/oe/spdx30_tasks.py | 8 +------- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 3288cdf75a..aa39208eae 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx if packages: for name in sorted(packages.keys()): if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + bb.note("Unable to find SPDX provider for '%s'" % name) + continue pkg_name, pkg_hashfn = providers[name] diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 887fac813a..ba15d74278 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1251,11 +1251,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) providers = oe.spdx_common.collect_package_providers(d) build_deps = set() - missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - missing_providers.add(name) + bb.note(f"Unable to find SPDX provider for '{name}'") continue pkg_name, pkg_hashfn = providers[name] @@ -1274,11 +1273,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) - if missing_providers: - bb.fatal( - f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" - ) - if build_deps: objset.new_scoped_relationship( [build], -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 12/13] spdx3: Use common variable for vardeps 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (10 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 11/13] spdx: Remove fatal errors for missing providers Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 13/13] glibc-testsuite: Do not generate SPDX Joshua Watt ` (2 subsequent siblings) 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Instead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------ 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 88b7ef9f42..6df66c193b 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -174,6 +174,14 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " +SPDX3_VAR_DEPS = "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) @@ -185,13 +193,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" -do_create_recipe_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { sstate_setscene(d) @@ -222,13 +224,7 @@ do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " -do_create_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_spdx_setscene () { sstate_setscene(d) @@ -249,6 +245,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" +do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_package_spdx_setscene () { sstate_setscene(d) @@ -270,13 +267,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" -do_create_recipe_sbom[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_sbom_setscene () { sstate_setscene(d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v5 13/13] glibc-testsuite: Do not generate SPDX 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (11 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 12/13] spdx3: Use common variable for vardeps Joshua Watt @ 2026-03-04 16:44 ` Joshua Watt 2026-03-05 19:59 ` [OE-core][PATCH v5 00/13] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt 14 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-04 16:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt glibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-core/glibc/glibc-testsuite_2.42.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/glibc/glibc-testsuite_2.42.bb b/meta/recipes-core/glibc/glibc-testsuite_2.42.bb index 6477612feb..1a83a09802 100644 --- a/meta/recipes-core/glibc/glibc-testsuite_2.42.bb +++ b/meta/recipes-core/glibc/glibc-testsuite_2.42.bb @@ -32,6 +32,7 @@ do_check:append () { } inherit nopackages +inherit nospdx deltask do_stash_locale deltask do_install deltask do_populate_sysroot -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v5 00/13] Add SPDX 3 Recipe Information 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (12 preceding siblings ...) 2026-03-04 16:44 ` [OE-core][PATCH v5 13/13] glibc-testsuite: Do not generate SPDX Joshua Watt @ 2026-03-05 19:59 ` Mathieu Dubois-Briand 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt 14 siblings, 0 replies; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-03-05 19:59 UTC (permalink / raw) To: JPEWhacker, openembedded-core On Wed Mar 4, 2026 at 5:44 PM CET, Joshua Watt via lists.openembedded.org wrote: > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > > V2: Fixes a bug where do_populate_sysroot was running when it should not > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > incorrect (this is already handled by bitbake in the taskgraph, and > doesn't need to be manually removed). > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > dependency. meta-world-recipe-sbom also no longer runs in world builds, > as there's no reason to this. Finally, fixes a bug where > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > (because do_unpack was not run). > > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > information is linked to binary packages, or just recipes. Defaults to > "0" to significantly reduce the size of the SPDX output. > > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it > does funny things with its arch which prevents it from rebuilding SPDX > data properly, and no SPDX data is needed for it anyway > > Joshua Watt (13): > llvm-project-source: Use allarch.bbclass > gcc-source: Use allarch.bbclass > spdx3: Add recipe SPDX data > spdx3: Add recipe SBoM task > spdx3: Add is-native property > spdx30: Include patch file information in VEX > spdx: De-duplicate CreationInfo > spdx_common: Check for dependent task in task flags > spdx30: Skip install package CVE information > dummy-sdk-package: Disable SPDX > spdx: Remove fatal errors for missing providers > spdx3: Use common variable for vardeps > glibc-testsuite: Do not generate SPDX > > meta/classes-global/sstate.bbclass | 4 +- > .../create-spdx-image-3.0.bbclass | 4 +- > .../create-spdx-sdk-3.0.bbclass | 4 +- > meta/classes-recipe/kernel.bbclass | 2 +- > meta/classes-recipe/nospdx.bbclass | 1 + > meta/classes/create-spdx-2.2.bbclass | 15 +- > meta/classes/create-spdx-3.0.bbclass | 87 ++- > meta/classes/spdx-common.bbclass | 22 +- > meta/conf/distro/include/maintainers.inc | 1 + > meta/lib/oe/sbom30.py | 192 ++++--- > meta/lib/oe/spdx30.py | 2 +- > meta/lib/oe/spdx30_tasks.py | 496 +++++++++++++----- > meta/lib/oe/spdx_common.py | 11 + > meta/lib/oeqa/selftest/cases/spdx.py | 41 +- > .../glibc/glibc-testsuite_2.42.bb | 1 + > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > .../meta/meta-world-recipe-sbom.bb | 29 + > .../clang/llvm-project-source.inc | 8 +- > meta/recipes-devtools/gcc/gcc-source.inc | 16 +- > 19 files changed, 667 insertions(+), 270 deletions(-) > create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb Ok, we are almost there! We only have a selftest failure now: 2026-03-05 16:33:10,060 - oe-selftest - INFO - sysroot.SysrootTests.test_sysroot_cleanup (subunit.RemotedTestCase) 2026-03-05 16:33:10,061 - oe-selftest - INFO - ... FAIL ... ERROR: sysroot-test-1.0-r0 do_create_spdx: Could not find a builds SPDX document named build-sysroot-test-arch1 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3457 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3338 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3227 Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt ` (13 preceding siblings ...) 2026-03-05 19:59 ` [OE-core][PATCH v5 00/13] Add SPDX 3 Recipe Information Mathieu Dubois-Briand @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 01/15] llvm-project-source: Use allarch.bbclass Joshua Watt ` (15 more replies) 14 siblings, 16 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run). V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output. V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance Joshua Watt (15): llvm-project-source: Use allarch.bbclass gcc-source: Use allarch.bbclass spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags spdx30: Skip install package CVE information dummy-sdk-package: Disable SPDX spdx: Remove fatal errors for missing providers spdx3: Use common variable for vardeps glibc-testsuite: Do not generate SPDX spdx: Remove do_collect_spdx_deps task spdx: Update to latest bindings meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 2 +- meta/classes/create-spdx-2.2.bbclass | 33 +- meta/classes/create-spdx-3.0.bbclass | 92 +- meta/classes/spdx-common.bbclass | 34 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 239 +- meta/lib/oe/spdx30/__init__.py | 8 + meta/lib/oe/spdx30/__main__.py | 12 + meta/lib/oe/spdx30/cmd.py | 75 + meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++------- meta/lib/oe/spdx30/stub.pyi | 2544 +++++++ meta/lib/oe/spdx30_tasks.py | 512 +- meta/lib/oe/spdx_common.py | 78 +- meta/lib/oeqa/selftest/cases/spdx.py | 41 +- .../glibc/glibc-testsuite_2.42.bb | 1 + meta/recipes-core/meta/dummy-sdk-package.inc | 1 + .../meta/meta-world-recipe-sbom.bb | 29 + .../clang/llvm-project-source.inc | 8 +- meta/recipes-devtools/gcc/gcc-source.inc | 16 +- scripts/contrib/make-spdx-bindings.sh | 3 +- 24 files changed, 6922 insertions(+), 2756 deletions(-) create mode 100644 meta/lib/oe/spdx30/__init__.py create mode 100644 meta/lib/oe/spdx30/__main__.py create mode 100644 meta/lib/oe/spdx30/cmd.py rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%) create mode 100644 meta/lib/oe/spdx30/stub.pyi create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb -- 2.53.0 ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 01/15] llvm-project-source: Use allarch.bbclass 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 02/15] gcc-source: " Joshua Watt ` (14 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/clang/llvm-project-source.inc | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/meta/recipes-devtools/clang/llvm-project-source.inc b/meta/recipes-devtools/clang/llvm-project-source.inc index 13e54efbc2..6bb595b7bc 100644 --- a/meta/recipes-devtools/clang/llvm-project-source.inc +++ b/meta/recipes-devtools/clang/llvm-project-source.inc @@ -5,7 +5,7 @@ deltask do_populate_sysroot deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "llvm-project-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/llvm-project-source-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:llvm-project-source::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/llvm-project-source-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 02/15] gcc-source: Use allarch.bbclass 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 01/15] llvm-project-source: Use allarch.bbclass Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data Joshua Watt ` (13 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Converts the recipe to use allarch.bbclass. This is necessary because SSTATE_PKGARCH is set to "allarch" based on if allarch is inherited or not. If it is not, SSTATE_PKGARCH has the value "all", which means any data written out based on it cannot be found (because "all" is not in SSTATE_ARCHS) Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-devtools/gcc/gcc-source.inc | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/meta/recipes-devtools/gcc/gcc-source.inc b/meta/recipes-devtools/gcc/gcc-source.inc index 265bcf4bef..3ac679b1a6 100644 --- a/meta/recipes-devtools/gcc/gcc-source.inc +++ b/meta/recipes-devtools/gcc/gcc-source.inc @@ -1,11 +1,11 @@ -deltask do_configure -deltask do_compile -deltask do_install +deltask do_configure +deltask do_compile +deltask do_install deltask do_populate_sysroot -deltask do_populate_lic +deltask do_populate_lic RM_WORK_EXCLUDE += "${PN}" -inherit nopackages +inherit nopackages allarch PN = "gcc-source-${PV}" WORKDIR = "${TMPDIR}/work-shared/gcc-${PV}-${PR}" @@ -14,14 +14,8 @@ SSTATE_SWSPEC = "sstate:gcc::${PV}:${PR}::${SSTATE_VERSION}:" STAMP = "${STAMPS_DIR}/work-shared/gcc-${PV}-${PR}" STAMPCLEAN = "${STAMPS_DIR}/work-shared/gcc-${PV}-*" -INHIBIT_DEFAULT_DEPS = "1" DEPENDS = "" PACKAGES = "" -TARGET_ARCH = "allarch" -TARGET_AS_ARCH = "none" -TARGET_CC_ARCH = "none" -TARGET_LD_ARCH = "none" -TARGET_OS = "linux" baselib = "lib" PACKAGE_ARCH = "all" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 01/15] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 02/15] gcc-source: " Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-12 11:43 ` Richard Purdie 2026-03-10 18:38 ` [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task Joshua Watt ` (12 subsequent siblings) 15 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 2fd29d7323..95c44f404e 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -954,7 +954,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1116,7 +1116,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index d61cc82a4e..8fd4e1141a 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -882,7 +882,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build: diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..3288cdf75a 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..672ca27cd0 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,19 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} addtask do_create_spdx after \ + do_unpack \ + do_patch \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +200,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +231,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..3c239a718b 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code @@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn) @@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 5830d7c087..759ca86b73 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", @@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """, @@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self): @@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}" @@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}") @@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """, -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data 2026-03-10 18:38 ` [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data Joshua Watt @ 2026-03-12 11:43 ` Richard Purdie 2026-03-12 14:11 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Richard Purdie @ 2026-03-12 11:43 UTC (permalink / raw) To: JPEWhacker, openembedded-core On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > Adds a new package to the SPDX output that represents the recipe data > for a given recipe. Importantly, this data contains only things that can > be determined statically from only the recipe, so it doesn't require > fetching or building anything. This means that build time dependencies > and CVE information for recipes can be analyzed without needing to > actually do any builds. > > Sadly, license data cannot be included because NO_GENERIC_LICENSE means > that actual license text might only be available after do_fetch We talked about these patches on the review call. I'm a bit worried about the direction we're going from a few angles. The general theme is the complexity and increasingly seemingly tangled web we seem to be weaving and whether we're going to end up in a good place. Taking NO_GENERIC_LICENSE specifically, it may be we should mandate that such licenses are copied into the metadata, then we solve the license data problem that way? That would simplify some of the problems we're facing and reduce some set of the corner cases. This patch adds a new task into the task graph and I'm getting a bit worried about the number of them the SPDX class is adding. I appreciate there is a later patch removing one, which is nice though :) So, for this patch, could we just drop NO_GENERIC_LICENSE and how much code complexity improvement does that buy us? Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data 2026-03-12 11:43 ` Richard Purdie @ 2026-03-12 14:11 ` Joshua Watt 2026-03-12 17:50 ` Richard Purdie 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-12 14:11 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 5:43 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > Adds a new package to the SPDX output that represents the recipe data > > for a given recipe. Importantly, this data contains only things that can > > be determined statically from only the recipe, so it doesn't require > > fetching or building anything. This means that build time dependencies > > and CVE information for recipes can be analyzed without needing to > > actually do any builds. > > > > Sadly, license data cannot be included because NO_GENERIC_LICENSE means > > that actual license text might only be available after do_fetch > > We talked about these patches on the review call. I'm a bit worried > about the direction we're going from a few angles. > > The general theme is the complexity and increasingly seemingly tangled > web we seem to be weaving and whether we're going to end up in a good > place. > > Taking NO_GENERIC_LICENSE specifically, it may be we should mandate > that such licenses are copied into the metadata, then we solve the > license data problem that way? That would simplify some of the problems > we're facing and reduce some set of the corner cases. > > This patch adds a new task into the task graph and I'm getting a bit > worried about the number of them the SPDX class is adding. I appreciate > there is a later patch removing one, which is nice though :) With the removal of the vestigial task in this patch series, the task graph for SPDX is: do_create_recipe_spdx - > "static" information we can determine about the recipe just from the metadata (no fetching, compiling, etc.) do_create_spdx -> Information about what we built and how we built it. We obviously have to build to figure this part out (the definition of this didn't change in this patch series; it should really be called do_create_build_spdx, but it inherited the name from the SPDX 2 code, so I don't want to change it) do_create_runtime_spdx -> Information about runtime packages. This has to be a separate task because while the build graph is a DAG, the runtime graph is not. The definition of this didn't change in this patch series. Various SBoM assembly tasks: These are the tasks that take the individual SPDX files generated by the tasks above and link them into a complete document that ends up in DEPLOY_DIR. They are all identified by having "sbom" in the name (do_create_image_sbom_spdx) > > So, for this patch, could we just drop NO_GENERIC_LICENSE and how much > code complexity improvement does that buy us? I'm not clear what you mean by this. I'm not including any additional License information, because we don't have it. I didn't change any license handling in the SPDX code, and I didn't add any more, so if you're talking about simplifying the SPDX code by dropping NO_GENERIC_LICENSE, it gains you nothing here specifically. It might be nice to improve NO_GENERIC_LICENSE in general, but I don't think we can do that for 6.0. If we do that later, we might be able to add license information to the "recipe" level SPDX data. The comment in the commit messages was probably more of a gripe than useful information (It feels like we _should_ be able to get license data statically, but we can't). I'll just remove it. > > Cheers, > > Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data 2026-03-12 14:11 ` Joshua Watt @ 2026-03-12 17:50 ` Richard Purdie 0 siblings, 0 replies; 113+ messages in thread From: Richard Purdie @ 2026-03-12 17:50 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core On Thu, 2026-03-12 at 08:11 -0600, Joshua Watt wrote: > On Thu, Mar 12, 2026 at 5:43 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > Adds a new package to the SPDX output that represents the recipe data > > > for a given recipe. Importantly, this data contains only things that can > > > be determined statically from only the recipe, so it doesn't require > > > fetching or building anything. This means that build time dependencies > > > and CVE information for recipes can be analyzed without needing to > > > actually do any builds. > > > > > > Sadly, license data cannot be included because NO_GENERIC_LICENSE means > > > that actual license text might only be available after do_fetch > > > > We talked about these patches on the review call. I'm a bit worried > > about the direction we're going from a few angles. > > > > The general theme is the complexity and increasingly seemingly tangled > > web we seem to be weaving and whether we're going to end up in a good > > place. > > > > Taking NO_GENERIC_LICENSE specifically, it may be we should mandate > > that such licenses are copied into the metadata, then we solve the > > license data problem that way? That would simplify some of the problems > > we're facing and reduce some set of the corner cases. > > > > This patch adds a new task into the task graph and I'm getting a bit > > worried about the number of them the SPDX class is adding. I appreciate > > there is a later patch removing one, which is nice though :) > > With the removal of the vestigial task in this patch series, the task > graph for SPDX is: > > do_create_recipe_spdx - > "static" information we can determine about > the recipe just from the metadata (no fetching, compiling, etc.) > > do_create_spdx -> Information about what we built and how we built it. > We obviously have to build to figure this part out (the definition of > this didn't change in this patch series; it should really be called > do_create_build_spdx, but it inherited the name from the SPDX 2 code, > so I don't want to change it) > > do_create_runtime_spdx -> Information about runtime packages. This has > to be a separate task because while the build graph is a DAG, the > runtime graph is not. The definition of this didn't change in this > patch series. I'm not entirely sure why we couldn't collect both sets of information in one go in the same task, maybe inspecting BB_TASKDEPS instead of the tasks actual dependencies but that is getting distracted into other issues I guess. > Various SBoM assembly tasks: These are the tasks that take the > individual SPDX files generated by the tasks above and link them into > a complete document that ends up in DEPLOY_DIR. They are all > identified by having "sbom" in the name (do_create_image_sbom_spdx) > > > > So, for this patch, could we just drop NO_GENERIC_LICENSE and how much > > code complexity improvement does that buy us? > > I'm not clear what you mean by this. I'm not including any additional > License information, because we don't have it. I didn't change any > license handling in the SPDX code, and I didn't add any more, so if > you're talking about simplifying the SPDX code by dropping > NO_GENERIC_LICENSE, it gains you nothing here specifically. > > It might be nice to improve NO_GENERIC_LICENSE in general, but I don't > think we can do that for 6.0. If we do that later, we might be able to > add license information to the "recipe" level SPDX data. > > The comment in the commit messages was probably more of a gripe than > useful information (It feels like we _should_ be able to get license > data statically, but we can't). I'll just remove it. Ok, fair enough. I was more thinking that we could fix things so we could get that information. I think I was getting confused and thinking you were getting partial information. We should perhaps separate out the NO_GENERIC_LICENSE issue into a separate bug/issue to work on. Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (2 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-12 11:50 ` Richard Purdie 2026-03-10 18:38 ` [OE-core][PATCH v6 05/15] spdx3: Add is-native property Joshua Watt ` (11 subsequent siblings) 15 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake -c create_recipe_sbom meta-world-recipe-sbom ``` Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 32 +++++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 ++++++ meta/lib/oeqa/selftest/cases/spdx.py | 10 ++++++ .../meta/meta-world-recipe-sbom.bb | 29 +++++++++++++++++ 6 files changed, 83 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 672ca27cd0..c3ea95b8bc 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0" @@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3c239a718b..abf2332bee 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc index 3c7fc4974d..40e090d452 100644 --- a/meta/conf/distro/include/maintainers.inc +++ b/meta/conf/distro/include/maintainers.inc @@ -535,6 +535,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <richard.purdie@linuxfoundation.org>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <JPEWhacker@gmail.com>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <yi.zhao@windriver.com>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <unassigned@yoctoproject.org>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <unassigned@yoctoproject.org>" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..b6c917045e 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 759ca86b73..efee0214fc 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -151,6 +151,16 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + task="create_recipe_sbom", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..b47a3229c9 --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,29 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task 2026-03-10 18:38 ` [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task Joshua Watt @ 2026-03-12 11:50 ` Richard Purdie 2026-03-12 14:12 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Richard Purdie @ 2026-03-12 11:50 UTC (permalink / raw) To: JPEWhacker, openembedded-core On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > Adds a task that will create the complete recipe-level SBoM for a given > target recipe, following all dependencies. For example: > > ``` > bitbake -c create_recipe_sbom zstd > ``` > > Would produce the complete recipe SBoM for the zstd recipe, include all > build time dependencies (recursively). > > The complete SBoM for all (target) recipes can be built with: > > ``` > bitbake -c create_recipe_sbom meta-world-recipe-sbom > ``` > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> With this change I got a bit confused as it wasn't obvious which task you'd run just looking at the meta-world-recipe-sbom recipe. Should that recipe do something like: addtask create_recipe_sbom before do_build so that the recipe actually triggers the thing which it is there for? Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task 2026-03-12 11:50 ` Richard Purdie @ 2026-03-12 14:12 ` Joshua Watt 0 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-12 14:12 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 5:50 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > Adds a task that will create the complete recipe-level SBoM for a given > > target recipe, following all dependencies. For example: > > > > ``` > > bitbake -c create_recipe_sbom zstd > > ``` > > > > Would produce the complete recipe SBoM for the zstd recipe, include all > > build time dependencies (recursively). > > > > The complete SBoM for all (target) recipes can be built with: > > > > ``` > > bitbake -c create_recipe_sbom meta-world-recipe-sbom > > ``` > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > With this change I got a bit confused as it wasn't obvious which task > you'd run just looking at the meta-world-recipe-sbom recipe. > > Should that recipe do something like: > > addtask create_recipe_sbom before do_build > > so that the recipe actually triggers the thing which it is there for? Yes > > Cheers, > > Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 05/15] spdx3: Add is-native property 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (3 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 06/15] spdx30: Include patch file information in VEX Joshua Watt ` (10 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index b6c917045e..a8fffbb085 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 06/15] spdx30: Include patch file information in VEX 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (4 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 05/15] spdx3: Add is-native property Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 07/15] spdx: De-duplicate CreationInfo Joshua Watt ` (9 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8fffbb085..aec47d4f81 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 07/15] spdx: De-duplicate CreationInfo 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (5 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 06/15] spdx30: Include patch file information in VEX Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 08/15] spdx_common: Check for dependent task in task flags Joshua Watt ` (8 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 08/15] spdx_common: Check for dependent task in task flags 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (6 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 07/15] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information Joshua Watt ` (7 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..3aaf2a9c8b 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (7 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 08/15] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-12 11:55 ` Richard Purdie 2026-03-10 18:38 ` [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX Joshua Watt ` (6 subsequent siblings) 15 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Skips adding the install package CVE information by default. This information grows exponentially, since it ends up be N_CVES * N_PACKAGES. The CVE information for a given installed package can be determined by following the "generates" link between the install package and the recipe and looking at the CVE information for the recipe, meaning that the CVE information is only included once in the SPDX document. If users still need the legacy method of including CVE information for each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ 3 files changed, 43 insertions(+), 19 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index c3ea95b8bc..88b7ef9f42 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to including those already fixed upstream (warning: This can be large and \ slow)." +SPDX_PACKAGE_INCLUDE_VEX ?= "0" +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ + Normally, VEX information is only linked to the common recipe that `generates` the \ + binary packages, but setting this to '1' will cause it to also be linked into the \ + generated binary packages. This is off by default because linking the VEX data to \ + each package causes the SPDX output to grow very large, and the same information \ + can be determined by following the `generates` relationship back to the recipe. \ + Before recipe packages were introduced, this was the only way VEX data was \ + expressed; you may need to enable this if your downstream tools do not \ + understand how to trace back to the recipe to find VEX information." + SPDX_INCLUDE_TIMESTAMPS ?= "0" SPDX_INCLUDE_TIMESTAMPS[doc] = "Include time stamps in SPDX output. This is \ useful if you want to know when artifacts were produced and when builds \ diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index aec47d4f81..887fac813a 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -771,27 +771,28 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): + if (d.getVar("SPDX_PACKAGE_INCLUDE_VEX") or "") == "1": for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) + for cve in rel.to: + vex_statements[cve] = [] + vex_patches[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index efee0214fc..f1ea2694cf 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -429,3 +429,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): value, ["enabled", "disabled"], f"Unexpected PACKAGECONFIG value '{value}' for {key}" ) + + def test_package_vex(self): + objset = self.check_recipe_spdx( + "core-image-minimal", + "{DEPLOY_DIR_IMAGE}/core-image-minimal-{MACHINE}.rootfs.spdx.json", + extraconf="""\ + SPDX_PACKAGE_INCLUDE_VEX = "1" + """, + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information 2026-03-10 18:38 ` [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information Joshua Watt @ 2026-03-12 11:55 ` Richard Purdie 2026-03-12 14:15 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Richard Purdie @ 2026-03-12 11:55 UTC (permalink / raw) To: JPEWhacker, openembedded-core On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > Skips adding the install package CVE information by default. This > information grows exponentially, since it ends up be N_CVES * > N_PACKAGES. The CVE information for a given installed package can be > determined by following the "generates" link between the install package > and the recipe and looking at the CVE information for the recipe, > meaning that the CVE information is only included once in the SPDX > document. > > If users still need the legacy method of including CVE information for > each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ > meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- > meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ > 3 files changed, 43 insertions(+), 19 deletions(-) > > diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass > index c3ea95b8bc..88b7ef9f42 100644 > --- a/meta/classes/create-spdx-3.0.bbclass > +++ b/meta/classes/create-spdx-3.0.bbclass > @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to > including those already fixed upstream (warning: This can be large and \ > slow)." > > +SPDX_PACKAGE_INCLUDE_VEX ?= "0" > +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ > + Normally, VEX information is only linked to the common recipe that `generates` the \ > + binary packages, but setting this to '1' will cause it to also be linked into the \ > + generated binary packages. This is off by default because linking the VEX data to \ > + each package causes the SPDX output to grow very large, and the same information \ > + can be determined by following the `generates` relationship back to the recipe. \ > + Before recipe packages were introduced, this was the only way VEX data was \ > + expressed; you may need to enable this if your downstream tools do not \ > + understand how to trace back to the recipe to find VEX information." To me, removing this duplication and keeping the SPDX docs usable seems like a very sensible thing to do. Do we want/need to make it configurable? I appreciate some tools/usage may need fixing to work with this but adding configuration options like this makes it harder to use our code and also adds maintenance/testing overhead. I think I'm very much in favour of just changing and generating things like this unconditionally and if someone needs to work with it differently, they can post process the output. This goes back to my concern about the complexity of the code and configuration, I think we need to simplify and present fewer options to the user... Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information 2026-03-12 11:55 ` Richard Purdie @ 2026-03-12 14:15 ` Joshua Watt 2026-03-12 15:52 ` Richard Purdie 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-12 14:15 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 5:55 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > Skips adding the install package CVE information by default. This > > information grows exponentially, since it ends up be N_CVES * > > N_PACKAGES. The CVE information for a given installed package can be > > determined by following the "generates" link between the install package > > and the recipe and looking at the CVE information for the recipe, > > meaning that the CVE information is only included once in the SPDX > > document. > > > > If users still need the legacy method of including CVE information for > > each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > --- > > meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ > > meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- > > meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ > > 3 files changed, 43 insertions(+), 19 deletions(-) > > > > diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass > > index c3ea95b8bc..88b7ef9f42 100644 > > --- a/meta/classes/create-spdx-3.0.bbclass > > +++ b/meta/classes/create-spdx-3.0.bbclass > > @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to > > including those already fixed upstream (warning: This can be large and \ > > slow)." > > > > +SPDX_PACKAGE_INCLUDE_VEX ?= "0" > > +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ > > + Normally, VEX information is only linked to the common recipe that `generates` the \ > > + binary packages, but setting this to '1' will cause it to also be linked into the \ > > + generated binary packages. This is off by default because linking the VEX data to \ > > + each package causes the SPDX output to grow very large, and the same information \ > > + can be determined by following the `generates` relationship back to the recipe. \ > > + Before recipe packages were introduced, this was the only way VEX data was \ > > + expressed; you may need to enable this if your downstream tools do not \ > > + understand how to trace back to the recipe to find VEX information." > > To me, removing this duplication and keeping the SPDX docs usable seems > like a very sensible thing to do. Do we want/need to make it > configurable? > > I appreciate some tools/usage may need fixing to work with this but > adding configuration options like this makes it harder to use our code > and also adds maintenance/testing overhead. Maybe, but I'm very hesitant to break any existing SPDX based CVE workflows that people may have in a LTS release, which is the only reason I added the option. I'm fine to remove this after the LTS, IMHO it's just too close to LTS release to suddenly say "sorry this is all broken for you now" > > I think I'm very much in favour of just changing and generating things > like this unconditionally and if someone needs to work with it > differently, they can post process the output. > > This goes back to my concern about the complexity of the code and > configuration, I think we need to simplify and present fewer options to > the user... > > Cheers, > > Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information 2026-03-12 14:15 ` Joshua Watt @ 2026-03-12 15:52 ` Richard Purdie 2026-03-12 16:11 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Richard Purdie @ 2026-03-12 15:52 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core On Thu, 2026-03-12 at 08:15 -0600, Joshua Watt wrote: > On Thu, Mar 12, 2026 at 5:55 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > Skips adding the install package CVE information by default. This > > > information grows exponentially, since it ends up be N_CVES * > > > N_PACKAGES. The CVE information for a given installed package can be > > > determined by following the "generates" link between the install package > > > and the recipe and looking at the CVE information for the recipe, > > > meaning that the CVE information is only included once in the SPDX > > > document. > > > > > > If users still need the legacy method of including CVE information for > > > each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > --- > > > meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ > > > meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- > > > meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ > > > 3 files changed, 43 insertions(+), 19 deletions(-) > > > > > > diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass > > > index c3ea95b8bc..88b7ef9f42 100644 > > > --- a/meta/classes/create-spdx-3.0.bbclass > > > +++ b/meta/classes/create-spdx-3.0.bbclass > > > @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to > > > including those already fixed upstream (warning: This can be large and \ > > > slow)." > > > > > > +SPDX_PACKAGE_INCLUDE_VEX ?= "0" > > > +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ > > > + Normally, VEX information is only linked to the common recipe that `generates` the \ > > > + binary packages, but setting this to '1' will cause it to also be linked into the \ > > > + generated binary packages. This is off by default because linking the VEX data to \ > > > + each package causes the SPDX output to grow very large, and the same information \ > > > + can be determined by following the `generates` relationship back to the recipe. \ > > > + Before recipe packages were introduced, this was the only way VEX data was \ > > > + expressed; you may need to enable this if your downstream tools do not \ > > > + understand how to trace back to the recipe to find VEX information." > > > > To me, removing this duplication and keeping the SPDX docs usable seems > > like a very sensible thing to do. Do we want/need to make it > > configurable? > > > > I appreciate some tools/usage may need fixing to work with this but > > adding configuration options like this makes it harder to use our code > > and also adds maintenance/testing overhead. > > Maybe, but I'm very hesitant to break any existing SPDX based CVE > workflows that people may have in a LTS release, which is the only > reason I added the option. I'm fine to remove this after the LTS, IMHO > it's just too close to LTS release to suddenly say "sorry this is all > broken for you now" We're not yet at feature freeze and I'm getting very worried about all the combinations of things we're trying to support. Having so many user options is likely going to confuse users too . I can see both sides of this but I am leaning towards a cleaner implementation and having people adapt to it now rather than when we remove it later. Do we know of tools that are going to struggle with this change? I'm guessing people can adapt to the change relatively easily? Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information 2026-03-12 15:52 ` Richard Purdie @ 2026-03-12 16:11 ` Joshua Watt 0 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-12 16:11 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 9:52 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2026-03-12 at 08:15 -0600, Joshua Watt wrote: > > On Thu, Mar 12, 2026 at 5:55 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > > Skips adding the install package CVE information by default. This > > > > information grows exponentially, since it ends up be N_CVES * > > > > N_PACKAGES. The CVE information for a given installed package can be > > > > determined by following the "generates" link between the install package > > > > and the recipe and looking at the CVE information for the recipe, > > > > meaning that the CVE information is only included once in the SPDX > > > > document. > > > > > > > > If users still need the legacy method of including CVE information for > > > > each package, then then can set SPDX_PACKAGE_INCLUDE_VEX = "1" > > > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > > --- > > > > meta/classes/create-spdx-3.0.bbclass | 11 ++++++++ > > > > meta/lib/oe/spdx30_tasks.py | 39 ++++++++++++++-------------- > > > > meta/lib/oeqa/selftest/cases/spdx.py | 12 +++++++++ > > > > 3 files changed, 43 insertions(+), 19 deletions(-) > > > > > > > > diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass > > > > index c3ea95b8bc..88b7ef9f42 100644 > > > > --- a/meta/classes/create-spdx-3.0.bbclass > > > > +++ b/meta/classes/create-spdx-3.0.bbclass > > > > @@ -45,6 +45,17 @@ SPDX_INCLUDE_VEX[doc] = "Controls what VEX information is in the output. Set to > > > > including those already fixed upstream (warning: This can be large and \ > > > > slow)." > > > > > > > > +SPDX_PACKAGE_INCLUDE_VEX ?= "0" > > > > +SPDX_PACKAGE_INCLUDE_VEX[doc] = "Link VEX information to the binary package outputs. \ > > > > + Normally, VEX information is only linked to the common recipe that `generates` the \ > > > > + binary packages, but setting this to '1' will cause it to also be linked into the \ > > > > + generated binary packages. This is off by default because linking the VEX data to \ > > > > + each package causes the SPDX output to grow very large, and the same information \ > > > > + can be determined by following the `generates` relationship back to the recipe. \ > > > > + Before recipe packages were introduced, this was the only way VEX data was \ > > > > + expressed; you may need to enable this if your downstream tools do not \ > > > > + understand how to trace back to the recipe to find VEX information." > > > > > > To me, removing this duplication and keeping the SPDX docs usable seems > > > like a very sensible thing to do. Do we want/need to make it > > > configurable? > > > > > > I appreciate some tools/usage may need fixing to work with this but > > > adding configuration options like this makes it harder to use our code > > > and also adds maintenance/testing overhead. > > > > Maybe, but I'm very hesitant to break any existing SPDX based CVE > > workflows that people may have in a LTS release, which is the only > > reason I added the option. I'm fine to remove this after the LTS, IMHO > > it's just too close to LTS release to suddenly say "sorry this is all > > broken for you now" > > We're not yet at feature freeze and I'm getting very worried about all > the combinations of things we're trying to support. Having so many user > options is likely going to confuse users too . I can see both sides of > this but I am leaning towards a cleaner implementation and having > people adapt to it now rather than when we remove it later. > > Do we know of tools that are going to struggle with this change? I'm > guessing people can adapt to the change relatively easily? Any tool that didn't know to change would get a lot more CVE reports show up, since the VEX data saying how we fixed things (derived from CVE_STATUS and patches) would be "missing". The fix isn't terribly hard as long as you know to follow the relationship back to the recipe package (where all the VEX data now exists). > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (8 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-12 11:59 ` Richard Purdie 2026-03-10 18:38 ` [OE-core][PATCH v6 11/15] spdx: Remove fatal errors for missing providers Joshua Watt ` (5 subsequent siblings) 15 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt The dummy SDK packages do not need SPDX support, and since they play some games with allarch that cause problems, it's simplest to disable their SPDX output. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-core/meta/dummy-sdk-package.inc | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc index bf453cac9b..71e788b0b9 100644 --- a/meta/recipes-core/meta/dummy-sdk-package.inc +++ b/meta/recipes-core/meta/dummy-sdk-package.inc @@ -4,6 +4,7 @@ LICENSE = "MIT" PACKAGE_ARCH = "all" inherit allarch +inherit nospdx INHIBIT_DEFAULT_DEPS = "1" -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-10 18:38 ` [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX Joshua Watt @ 2026-03-12 11:59 ` Richard Purdie 2026-03-12 14:24 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Richard Purdie @ 2026-03-12 11:59 UTC (permalink / raw) To: JPEWhacker, openembedded-core On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > The dummy SDK packages do not need SPDX support, and since they play > some games with allarch that cause problems, it's simplest to disable > their SPDX output. > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > index bf453cac9b..71e788b0b9 100644 > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > @@ -4,6 +4,7 @@ LICENSE = "MIT" > PACKAGE_ARCH = "all" > > inherit allarch > +inherit nospdx > > INHIBIT_DEFAULT_DEPS = "1" I know why you did this, but after thinking about this, I'm worried and I'm not sure this is the right move. The SDK dummy packages are "odd" but they are actual package files that are generated and would be present in manifests and as such, if they're missing from the SPDX manifests and docs, this is going to raise questions. I appreciate they don't have content and their main use is their 'provides'. Taking a step back, we do support generic/multiple levels of package arch and since we support that, the SPDX code does need to handle that too. I'm therefore starting to worry that SPDX can't cope with the multiple package levels. Can you remind me what the issue is with the "all" arch here? Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-12 11:59 ` Richard Purdie @ 2026-03-12 14:24 ` Joshua Watt 2026-03-12 15:58 ` Richard Purdie 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-12 14:24 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 5:59 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > The dummy SDK packages do not need SPDX support, and since they play > > some games with allarch that cause problems, it's simplest to disable > > their SPDX output. > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > --- > > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > > index bf453cac9b..71e788b0b9 100644 > > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > > @@ -4,6 +4,7 @@ LICENSE = "MIT" > > PACKAGE_ARCH = "all" > > > > inherit allarch > > +inherit nospdx > > > > INHIBIT_DEFAULT_DEPS = "1" > > I know why you did this, but after thinking about this, I'm worried and > I'm not sure this is the right move. > > The SDK dummy packages are "odd" but they are actual package files that > are generated and would be present in manifests and as such, if they're > missing from the SPDX manifests and docs, this is going to raise > questions. I appreciate they don't have content and their main use is > their 'provides'. > > Taking a step back, we do support generic/multiple levels of package > arch and since we support that, the SPDX code does need to handle that > too. I'm therefore starting to worry that SPDX can't cope with the > multiple package levels. Can you remind me what the issue is with the > "all" arch here? The problem is specifically the anonymous python that does: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') Which seems specifically intended to bypass the allarch logic in a weird way. It's specifically doing this to put the package in a place no-one can find it (this includes SPDX); I didn't think it provided much value hence disabling it, but looking closer, I think we can do: SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}" in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it. > > Cheers, > > Richard > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-12 14:24 ` Joshua Watt @ 2026-03-12 15:58 ` Richard Purdie 2026-03-12 16:06 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Richard Purdie @ 2026-03-12 15:58 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core On Thu, 2026-03-12 at 08:24 -0600, Joshua Watt wrote: > On Thu, Mar 12, 2026 at 5:59 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > The dummy SDK packages do not need SPDX support, and since they play > > > some games with allarch that cause problems, it's simplest to disable > > > their SPDX output. > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > --- > > > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > > > 1 file changed, 1 insertion(+) > > > > > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > > > index bf453cac9b..71e788b0b9 100644 > > > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > > > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > > > @@ -4,6 +4,7 @@ LICENSE = "MIT" > > > PACKAGE_ARCH = "all" > > > > > > inherit allarch > > > +inherit nospdx > > > > > > INHIBIT_DEFAULT_DEPS = "1" > > > > I know why you did this, but after thinking about this, I'm worried and > > I'm not sure this is the right move. > > > > The SDK dummy packages are "odd" but they are actual package files that > > are generated and would be present in manifests and as such, if they're > > missing from the SPDX manifests and docs, this is going to raise > > questions. I appreciate they don't have content and their main use is > > their 'provides'. > > > > Taking a step back, we do support generic/multiple levels of package > > arch and since we support that, the SPDX code does need to handle that > > too. I'm therefore starting to worry that SPDX can't cope with the > > multiple package levels. Can you remind me what the issue is with the > > "all" arch here? > > The problem is specifically the anonymous python that does: > > d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > Which seems specifically intended to bypass the allarch logic in a > weird way. It's specifically doing this to put the package in a place > no-one can find it (this includes SPDX); I didn't think it provided > much value hence disabling it, but looking closer, I think we can do: > > SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}" > > in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it. That code was to put these packages into their own separate feed. Specifically: meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}" meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target" so there are separate package feeds being created and then the SDK code can select which package feeds it wants to use for a given configuration/recipe. That isn't to bypass allarch but just to customise the end package feed naming and allow different forms of SDK to be built by bringing in the different package combinations. My worry is that the SPDX code isn't going to handle the scenario where PACKAGE_ARCH is changed (for example for a specialist graphics stack) and that one of our general features will be lost. Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-12 15:58 ` Richard Purdie @ 2026-03-12 16:06 ` Joshua Watt 2026-03-12 16:43 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-12 16:06 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 9:58 AM Richard Purdie <richard.purdie@linuxfoundation.org> wrote: > > On Thu, 2026-03-12 at 08:24 -0600, Joshua Watt wrote: > > On Thu, Mar 12, 2026 at 5:59 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > > The dummy SDK packages do not need SPDX support, and since they play > > > > some games with allarch that cause problems, it's simplest to disable > > > > their SPDX output. > > > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > > --- > > > > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > > > > 1 file changed, 1 insertion(+) > > > > > > > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > index bf453cac9b..71e788b0b9 100644 > > > > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > > > > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > @@ -4,6 +4,7 @@ LICENSE = "MIT" > > > > PACKAGE_ARCH = "all" > > > > > > > > inherit allarch > > > > +inherit nospdx > > > > > > > > INHIBIT_DEFAULT_DEPS = "1" > > > > > > I know why you did this, but after thinking about this, I'm worried and > > > I'm not sure this is the right move. > > > > > > The SDK dummy packages are "odd" but they are actual package files that > > > are generated and would be present in manifests and as such, if they're > > > missing from the SPDX manifests and docs, this is going to raise > > > questions. I appreciate they don't have content and their main use is > > > their 'provides'. > > > > > > Taking a step back, we do support generic/multiple levels of package > > > arch and since we support that, the SPDX code does need to handle that > > > too. I'm therefore starting to worry that SPDX can't cope with the > > > multiple package levels. Can you remind me what the issue is with the > > > "all" arch here? > > > > The problem is specifically the anonymous python that does: > > > > d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > > > Which seems specifically intended to bypass the allarch logic in a > > weird way. It's specifically doing this to put the package in a place > > no-one can find it (this includes SPDX); I didn't think it provided > > much value hence disabling it, but looking closer, I think we can do: > > > > SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}" > > > > in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it. > > That code was to put these packages into their own separate feed. > Specifically: > > meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" > meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}" > meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target" > > so there are separate package feeds being created and then the SDK code > can select which package feeds it wants to use for a given > configuration/recipe. > > That isn't to bypass allarch but just to customise the end package feed > naming and allow different forms of SDK to be built by bringing in the > different package combinations. > > My worry is that the SPDX code isn't going to handle the scenario where > PACKAGE_ARCH is changed (for example for a specialist graphics stack) > and that one of our general features will be lost. Right, SPDX is (supposed) to do whatever sstate does, so if it works for sstate it works for SPDX. The problem isn't that this code does the arch changing (specifically), it's that it changes the arch to an arbitrary one that isn't part of the normal package search order (on purpose), and then expects everyone to manually add in that arch when they need the package (which, SPDX was _not_ doing but all of the other SDK code obviously is). I think my proposed fix is thus correct and I will test it which should cover this corner case. In general, I think most packages are handled correctly because again, SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS which is how I think other recipes would deal with a similar situation. However, the SDKs are different specifically because they expect SDK_PACKAGE_ARCH to be respected by anything dealing with the sdk generation code, which is what was missing in SPDX. > > Cheers, > > Richard > > > > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-12 16:06 ` Joshua Watt @ 2026-03-12 16:43 ` Joshua Watt 2026-03-12 18:02 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-12 16:43 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 10:06 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > On Thu, Mar 12, 2026 at 9:58 AM Richard Purdie > <richard.purdie@linuxfoundation.org> wrote: > > > > On Thu, 2026-03-12 at 08:24 -0600, Joshua Watt wrote: > > > On Thu, Mar 12, 2026 at 5:59 AM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > > > The dummy SDK packages do not need SPDX support, and since they play > > > > > some games with allarch that cause problems, it's simplest to disable > > > > > their SPDX output. > > > > > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > > > --- > > > > > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > > > > > 1 file changed, 1 insertion(+) > > > > > > > > > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > index bf453cac9b..71e788b0b9 100644 > > > > > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > @@ -4,6 +4,7 @@ LICENSE = "MIT" > > > > > PACKAGE_ARCH = "all" > > > > > > > > > > inherit allarch > > > > > +inherit nospdx > > > > > > > > > > INHIBIT_DEFAULT_DEPS = "1" > > > > > > > > I know why you did this, but after thinking about this, I'm worried and > > > > I'm not sure this is the right move. > > > > > > > > The SDK dummy packages are "odd" but they are actual package files that > > > > are generated and would be present in manifests and as such, if they're > > > > missing from the SPDX manifests and docs, this is going to raise > > > > questions. I appreciate they don't have content and their main use is > > > > their 'provides'. > > > > > > > > Taking a step back, we do support generic/multiple levels of package > > > > arch and since we support that, the SPDX code does need to handle that > > > > too. I'm therefore starting to worry that SPDX can't cope with the > > > > multiple package levels. Can you remind me what the issue is with the > > > > "all" arch here? > > > > > > The problem is specifically the anonymous python that does: > > > > > > d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > > > > > Which seems specifically intended to bypass the allarch logic in a > > > weird way. It's specifically doing this to put the package in a place > > > no-one can find it (this includes SPDX); I didn't think it provided > > > much value hence disabling it, but looking closer, I think we can do: > > > > > > SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}" > > > > > > in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it. > > > > That code was to put these packages into their own separate feed. > > Specifically: > > > > meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" > > meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}" > > meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target" > > > > so there are separate package feeds being created and then the SDK code > > can select which package feeds it wants to use for a given > > configuration/recipe. > > > > That isn't to bypass allarch but just to customise the end package feed > > naming and allow different forms of SDK to be built by bringing in the > > different package combinations. > > > > My worry is that the SPDX code isn't going to handle the scenario where > > PACKAGE_ARCH is changed (for example for a specialist graphics stack) > > and that one of our general features will be lost. > > Right, SPDX is (supposed) to do whatever sstate does, so if it works > for sstate it works for SPDX. The problem isn't that this code does > the arch changing (specifically), it's that it changes the arch to an > arbitrary one that isn't part of the normal package search order (on > purpose), and then expects everyone to manually add in that arch when > they need the package (which, SPDX was _not_ doing but all of the > other SDK code obviously is). I think my proposed fix is thus correct > and I will test it which should cover this corner case. > > In general, I think most packages are handled correctly because again, > SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS > which is how I think other recipes would deal with a similar > situation. However, the SDKs are different specifically because they > expect SDK_PACKAGE_ARCH to be respected by anything dealing with the > sdk generation code, which is what was missing in SPDX. Hang on, I think I'm getting this confused with a different problem..... > > > > > Cheers, > > > > Richard > > > > > > > > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-12 16:43 ` Joshua Watt @ 2026-03-12 18:02 ` Joshua Watt 2026-03-12 20:34 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-12 18:02 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 10:43 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > On Thu, Mar 12, 2026 at 10:06 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > > > On Thu, Mar 12, 2026 at 9:58 AM Richard Purdie > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > On Thu, 2026-03-12 at 08:24 -0600, Joshua Watt wrote: > > > > On Thu, Mar 12, 2026 at 5:59 AM Richard Purdie > > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > > > > The dummy SDK packages do not need SPDX support, and since they play > > > > > > some games with allarch that cause problems, it's simplest to disable > > > > > > their SPDX output. > > > > > > > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > > > > --- > > > > > > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > > > > > > 1 file changed, 1 insertion(+) > > > > > > > > > > > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > > index bf453cac9b..71e788b0b9 100644 > > > > > > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > > @@ -4,6 +4,7 @@ LICENSE = "MIT" > > > > > > PACKAGE_ARCH = "all" > > > > > > > > > > > > inherit allarch > > > > > > +inherit nospdx > > > > > > > > > > > > INHIBIT_DEFAULT_DEPS = "1" > > > > > > > > > > I know why you did this, but after thinking about this, I'm worried and > > > > > I'm not sure this is the right move. > > > > > > > > > > The SDK dummy packages are "odd" but they are actual package files that > > > > > are generated and would be present in manifests and as such, if they're > > > > > missing from the SPDX manifests and docs, this is going to raise > > > > > questions. I appreciate they don't have content and their main use is > > > > > their 'provides'. > > > > > > > > > > Taking a step back, we do support generic/multiple levels of package > > > > > arch and since we support that, the SPDX code does need to handle that > > > > > too. I'm therefore starting to worry that SPDX can't cope with the > > > > > multiple package levels. Can you remind me what the issue is with the > > > > > "all" arch here? > > > > > > > > The problem is specifically the anonymous python that does: > > > > > > > > d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > > > > > > > Which seems specifically intended to bypass the allarch logic in a > > > > weird way. It's specifically doing this to put the package in a place > > > > no-one can find it (this includes SPDX); I didn't think it provided > > > > much value hence disabling it, but looking closer, I think we can do: > > > > > > > > SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}" > > > > > > > > in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it. > > > > > > That code was to put these packages into their own separate feed. > > > Specifically: > > > > > > meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" > > > meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}" > > > meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > > meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target" > > > > > > so there are separate package feeds being created and then the SDK code > > > can select which package feeds it wants to use for a given > > > configuration/recipe. > > > > > > That isn't to bypass allarch but just to customise the end package feed > > > naming and allow different forms of SDK to be built by bringing in the > > > different package combinations. > > > > > > My worry is that the SPDX code isn't going to handle the scenario where > > > PACKAGE_ARCH is changed (for example for a specialist graphics stack) > > > and that one of our general features will be lost. > > > > Right, SPDX is (supposed) to do whatever sstate does, so if it works > > for sstate it works for SPDX. The problem isn't that this code does > > the arch changing (specifically), it's that it changes the arch to an > > arbitrary one that isn't part of the normal package search order (on > > purpose), and then expects everyone to manually add in that arch when > > they need the package (which, SPDX was _not_ doing but all of the > > other SDK code obviously is). I think my proposed fix is thus correct > > and I will test it which should cover this corner case. > > > > In general, I think most packages are handled correctly because again, > > SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS > > which is how I think other recipes would deal with a similar > > situation. However, the SDKs are different specifically because they > > expect SDK_PACKAGE_ARCH to be respected by anything dealing with the > > sdk generation code, which is what was missing in SPDX. > > Hang on, I think I'm getting this confused with a different problem..... Ok, I remember this one correctly now (who needs AI to hallucinate....). I think the correct thing to do here is to add: do_create_recipe_spdx[vardeps] += "PACKAGE_ARCH" because it's not rebuilding when this recipe changes PACKAGE_ARCH. The other SPDX tasks are because of an implicit dependency on PACKAGE_ARCH that re-triggers them. In the back of my mind I seem to recall we maybe didn't want to do this for some reason, but PACKAGE_ARCH isn't in BB_*_IGNORE_VARS so maybe I'm just mis-remembering. Anyway, if this fix seems correct I'll make it. > > > > > > > > > Cheers, > > > > > > Richard > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX 2026-03-12 18:02 ` Joshua Watt @ 2026-03-12 20:34 ` Joshua Watt 0 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-12 20:34 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core On Thu, Mar 12, 2026 at 12:02 PM Joshua Watt <jpewhacker@gmail.com> wrote: > > On Thu, Mar 12, 2026 at 10:43 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > > > On Thu, Mar 12, 2026 at 10:06 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > > > > > On Thu, Mar 12, 2026 at 9:58 AM Richard Purdie > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > On Thu, 2026-03-12 at 08:24 -0600, Joshua Watt wrote: > > > > > On Thu, Mar 12, 2026 at 5:59 AM Richard Purdie > > > > > <richard.purdie@linuxfoundation.org> wrote: > > > > > > > > > > > > On Tue, 2026-03-10 at 12:38 -0600, Joshua Watt via lists.openembedded.org wrote: > > > > > > > The dummy SDK packages do not need SPDX support, and since they play > > > > > > > some games with allarch that cause problems, it's simplest to disable > > > > > > > their SPDX output. > > > > > > > > > > > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > > > > > > --- > > > > > > > meta/recipes-core/meta/dummy-sdk-package.inc | 1 + > > > > > > > 1 file changed, 1 insertion(+) > > > > > > > > > > > > > > diff --git a/meta/recipes-core/meta/dummy-sdk-package.inc b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > > > index bf453cac9b..71e788b0b9 100644 > > > > > > > --- a/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > > > +++ b/meta/recipes-core/meta/dummy-sdk-package.inc > > > > > > > @@ -4,6 +4,7 @@ LICENSE = "MIT" > > > > > > > PACKAGE_ARCH = "all" > > > > > > > > > > > > > > inherit allarch > > > > > > > +inherit nospdx > > > > > > > > > > > > > > INHIBIT_DEFAULT_DEPS = "1" > > > > > > > > > > > > I know why you did this, but after thinking about this, I'm worried and > > > > > > I'm not sure this is the right move. > > > > > > > > > > > > The SDK dummy packages are "odd" but they are actual package files that > > > > > > are generated and would be present in manifests and as such, if they're > > > > > > missing from the SPDX manifests and docs, this is going to raise > > > > > > questions. I appreciate they don't have content and their main use is > > > > > > their 'provides'. > > > > > > > > > > > > Taking a step back, we do support generic/multiple levels of package > > > > > > arch and since we support that, the SPDX code does need to handle that > > > > > > too. I'm therefore starting to worry that SPDX can't cope with the > > > > > > multiple package levels. Can you remind me what the issue is with the > > > > > > "all" arch here? > > > > > > > > > > The problem is specifically the anonymous python that does: > > > > > > > > > > d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > > > > > > > > > Which seems specifically intended to bypass the allarch logic in a > > > > > weird way. It's specifically doing this to put the package in a place > > > > > no-one can find it (this includes SPDX); I didn't think it provided > > > > > much value hence disabling it, but looking closer, I think we can do: > > > > > > > > > > SSTATE_MULTILIB_SSTATE_ARCHS:append = " ${SDK_PACKAGE_ARCH}" > > > > > > > > > > in create-spdx-sdk-3.0.bbclass and that should fix it. I'll try it. > > > > > > > > That code was to put these packages into their own separate feed. > > > > Specifically: > > > > > > > > meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb:DUMMYARCH = "buildtools-dummy-${SDKPKGSUFFIX}" > > > > meta/recipes-core/meta/nativesdk-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-${SDKPKGSUFFIX}" > > > > meta/recipes-core/meta/dummy-sdk-package.inc: d.setVar('PACKAGE_ARCH', '${DUMMYARCH}') > > > > meta/recipes-core/meta/target-sdk-provides-dummy.bb:DUMMYARCH = "sdk-provides-dummy-target" > > > > > > > > so there are separate package feeds being created and then the SDK code > > > > can select which package feeds it wants to use for a given > > > > configuration/recipe. > > > > > > > > That isn't to bypass allarch but just to customise the end package feed > > > > naming and allow different forms of SDK to be built by bringing in the > > > > different package combinations. > > > > > > > > My worry is that the SPDX code isn't going to handle the scenario where > > > > PACKAGE_ARCH is changed (for example for a specialist graphics stack) > > > > and that one of our general features will be lost. > > > > > > Right, SPDX is (supposed) to do whatever sstate does, so if it works > > > for sstate it works for SPDX. The problem isn't that this code does > > > the arch changing (specifically), it's that it changes the arch to an > > > arbitrary one that isn't part of the normal package search order (on > > > purpose), and then expects everyone to manually add in that arch when > > > they need the package (which, SPDX was _not_ doing but all of the > > > other SDK code obviously is). I think my proposed fix is thus correct > > > and I will test it which should cover this corner case. > > > > > > In general, I think most packages are handled correctly because again, > > > SPDX does whatever sstate does, and SPDX even respects SSTATE_ARCHS > > > which is how I think other recipes would deal with a similar > > > situation. However, the SDKs are different specifically because they > > > expect SDK_PACKAGE_ARCH to be respected by anything dealing with the > > > sdk generation code, which is what was missing in SPDX. > > > > Hang on, I think I'm getting this confused with a different problem..... > > Ok, I remember this one correctly now (who needs AI to > hallucinate....). I think the correct thing to do here is to add: > > do_create_recipe_spdx[vardeps] += "PACKAGE_ARCH" > > because it's not rebuilding when this recipe changes PACKAGE_ARCH. The > other SPDX tasks are because of an implicit dependency on PACKAGE_ARCH > that re-triggers them. In the back of my mind I seem to recall we > maybe didn't want to do this for some reason, but PACKAGE_ARCH isn't > in BB_*_IGNORE_VARS so maybe I'm just mis-remembering. > > Anyway, if this fix seems correct I'll make it. After trying this it doesn't work either. I'll write down my analysis of why and maybe we can figure it out: First of all, the SPDX code writes out SPDX files based on their SSTATE_PKGARCH (which is why I say SPDX behaves like sstate). This was done back in 2023 [1]. SSTATE_PKGARCH is in BB_HASHEXCLUDE_COMMON, so changes in SSTATE_PKGARCH do not cause SPDX code to rebuild; I think this is intentional, but I'm not sure. nativesdk-sdk-provides-dummy sets `PACKAGE_ARCH = "all"` and inherits allarch, but in sstate.bbclass, there is this: elif bb.data.inherits_class('nativesdk', d): d.setVar('SSTATE_PKGARCH', d.expand("${SDK_ARCH}_${SDK_OS}")) So, SSTATE_PKGARCH is set to "${SDK_ARCH}_${SDK_OS}" After this, the anon python in dummy-sdk-package.inc set PACKAGE_ARCH to ${DUMMYARCH}, which in this case is always "sdk-provides-dummy-nativesdk" -- FWIW, this really wants to look like the culprit but actually is irrelevant to the problem. Now, the problem is that there is nothing to retrigger running do_create_recipe_spdx() when SDKMACHINE changes. This task (on purpose) has pretty minimal dependencies, but specifically nothing related to SDKMACHINE changing re-triggers this task. Thus consider the following command sequence which will trigger the bug (with a clean tmp dir): SDKMACHINE=x86_64 bitbake nativesdk-sdk-provides-dummy && SDKMACHINE=i686 bitbake -fc create_spdx nativesdk-provided-dummy In this case, the first command runs and puts the nativesdk-sdk-provides-dummy do_create_recipe_spdx files in "${SDK_ARCH}_${SDK_OS}" ("x86_64_linux"). The second command reruns do_create_spdx, but because SDKMACHINE doesn't factor into the taskhash of do_create_recipe_spdx, it does not rerun that task, assuming it's fine. Now, do_create_spdx cannot find the required SPDX file, because "x86_64_linux" is not in sstate search path (SSTATE_ARCHS) anymore. Maybe allarch.bbclass and nativesdk.bbclass just don't interact well (esp. when sstate.bbclass appears to be making some assumptions about those two)? I did some digging and there are not many other recipes that inherit both allarch and nativesdk, however the other one I found was ca-certificates, and indeed it has the same error when building nativesdk-ca-certificates and switching SDKMACHINE. I'm at a loss for where the actual bug is here. If it's true that SPDX should be using SSTATE_PKGARCH, I cannot see what it's doing wrong here (as I said, it more or less just copies what sstate does). I'm kinda surprised we don't have other sstate problems with this recipe, given all of this, but maybe sstate isn't hard failing on missing files. [1]: b2db10e966 ("create-spdx/sbom: Ensure files don't overlap between machines") > > > > > > > > > > > > > > > Cheers, > > > > > > > > Richard > > > > > > > > > > > > > > > > ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 11/15] spdx: Remove fatal errors for missing providers 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (9 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 12/15] spdx3: Use common variable for vardeps Joshua Watt ` (4 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt When creating images and SDKs, do not error on missing providers. This allows recipes to use the `nospdx` inherit to prevent SPDX from being generated, but not result in an error when assembling the final image. Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-2.2.bbclass | 3 ++- meta/lib/oe/spdx30_tasks.py | 8 +------- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 3288cdf75a..aa39208eae 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx if packages: for name in sorted(packages.keys()): if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + bb.note("Unable to find SPDX provider for '%s'" % name) + continue pkg_name, pkg_hashfn = providers[name] diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 887fac813a..ba15d74278 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1251,11 +1251,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) providers = oe.spdx_common.collect_package_providers(d) build_deps = set() - missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - missing_providers.add(name) + bb.note(f"Unable to find SPDX provider for '{name}'") continue pkg_name, pkg_hashfn = providers[name] @@ -1274,11 +1273,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) - if missing_providers: - bb.fatal( - f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" - ) - if build_deps: objset.new_scoped_relationship( [build], -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 12/15] spdx3: Use common variable for vardeps 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (10 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 11/15] spdx: Remove fatal errors for missing providers Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 13/15] glibc-testsuite: Do not generate SPDX Joshua Watt ` (3 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Instead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------ 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 88b7ef9f42..6df66c193b 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -174,6 +174,14 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " +SPDX3_VAR_DEPS = "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) @@ -185,13 +193,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" -do_create_recipe_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { sstate_setscene(d) @@ -222,13 +224,7 @@ do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " -do_create_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_spdx_setscene () { sstate_setscene(d) @@ -249,6 +245,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" +do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_package_spdx_setscene () { sstate_setscene(d) @@ -270,13 +267,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" -do_create_recipe_sbom[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_sbom_setscene () { sstate_setscene(d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 13/15] glibc-testsuite: Do not generate SPDX 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (11 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 12/15] spdx3: Use common variable for vardeps Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 14/15] spdx: Remove do_collect_spdx_deps task Joshua Watt ` (2 subsequent siblings) 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt glibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-core/glibc/glibc-testsuite_2.42.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/glibc/glibc-testsuite_2.42.bb b/meta/recipes-core/glibc/glibc-testsuite_2.42.bb index 6477612feb..1a83a09802 100644 --- a/meta/recipes-core/glibc/glibc-testsuite_2.42.bb +++ b/meta/recipes-core/glibc/glibc-testsuite_2.42.bb @@ -32,6 +32,7 @@ do_check:append () { } inherit nopackages +inherit nospdx deltask do_stash_locale deltask do_install deltask do_populate_sysroot -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v6 14/15] spdx: Remove do_collect_spdx_deps task 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (12 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 13/15] glibc-testsuite: Do not generate SPDX Joshua Watt @ 2026-03-10 18:38 ` Joshua Watt 2026-03-11 13:55 ` [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt 15 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-10 18:38 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Removes the do_collect_spdx_deps task. This task was added a long time ago, and appears to have been added due to a misunderstanding about how the task graph works. It is not necessary since tasks can directly call collect_direct_deps() with the appropriate task that they depend on to get their dependencies. This should fix several classes of SPDX bug where documents could not be found because the wrong deps were being looked for due to which tasks were re-run Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-recipe/nospdx.bbclass | 1 - meta/classes/create-spdx-2.2.bbclass | 22 +++++---- meta/classes/create-spdx-3.0.bbclass | 5 +- meta/classes/spdx-common.bbclass | 22 --------- meta/lib/oe/spdx30_tasks.py | 22 +++++---- meta/lib/oe/spdx_common.py | 69 +++++++++++++--------------- 6 files changed, 62 insertions(+), 79 deletions(-) diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index 90e14442ba..7c99fcd1ec 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -4,7 +4,6 @@ # SPDX-License-Identifier: MIT # -deltask do_collect_spdx_deps deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index aa39208eae..1c43156559 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -277,7 +277,7 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" -def collect_dep_recipes(d, doc, spdx_recipe): +def collect_dep_recipes(d, doc, spdx_recipe, direct_deps): import json from pathlib import Path import oe.sbom @@ -290,9 +290,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): dep_recipes = [] - deps = oe.spdx_common.get_spdx_deps(d) - - for dep in deps: + for dep in direct_deps: # If this dependency is not calculated in the taskhash skip it. # Otherwise, it can result in broken links since this task won't # rebuild and see the new SPDX ID if the dependency changes @@ -405,7 +403,7 @@ do_create_recipe_spdx() { : } do_create_recipe_spdx[noexec] = "1" -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx python do_create_spdx() { @@ -532,7 +530,8 @@ python do_create_spdx() { if archive is not None: recipe.packageFileName = str(recipe_archive.name) - dep_recipes = collect_dep_recipes(d, doc, recipe) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_recipes = collect_dep_recipes(d, doc, recipe, direct_deps) doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe)) @@ -603,7 +602,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -638,7 +637,9 @@ python do_create_runtime_spdx() { license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() @@ -760,6 +761,7 @@ addtask do_create_runtime_spdx_setscene do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_runtime_spdx[deptask] = "do_create_spdx" do_create_runtime_spdx[rdeptask] = "do_create_spdx" do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx" @@ -829,7 +831,9 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 6df66c193b..862e656b2c 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -186,13 +186,14 @@ python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) } -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx SSTATETASKS += "do_create_recipe_spdx" do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[deptask] += "do_create_recipe_spdx" do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { @@ -208,7 +209,6 @@ addtask do_create_spdx after \ do_unpack \ do_patch \ do_create_recipe_spdx \ - do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ before do_populate_sdk do_populate_sdk_ext do_build do_rm_work @@ -244,6 +244,7 @@ do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = "do_create_spdx" do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index abf2332bee..4b40cbf75c 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -87,28 +87,6 @@ def create_spdx_source_deps(d): return " ".join(deps) -python do_collect_spdx_deps() { - # This task calculates the build time dependencies of the recipe, and is - # required because while a task can deptask on itself, those dependencies - # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then - # downstream tasks read in the found dependencies when writing the actual - # SPDX document - import json - import oe.spdx_common - from pathlib import Path - - spdx_deps_file = Path(d.getVar("SPDXDEPS")) - - deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") - - with spdx_deps_file.open("w") as f: - json.dump(deps, f) -} -addtask do_collect_spdx_deps -do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" -do_collect_spdx_deps[dirs] = "${SPDXDIR}" - oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" oe.spdx_common.collect_direct_deps[vardeps] += "DEPENDS" oe.spdx_common.collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index ba15d74278..76f3338fb8 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -298,13 +298,11 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): - deps = oe.spdx_common.get_spdx_deps(d) - +def collect_dep_objsets(d, direct_deps, subdir, fn_prefix, obj_type, **attr_filter): dep_objsets = [] dep_objs = set() - for dep in deps: + for dep in direct_deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter @@ -551,8 +549,10 @@ def create_recipe_spdx(d): ) ) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") + dep_objsets, dep_recipes = collect_dep_objsets( - d, "static", "static-", oe.spdx30.software_Package + d, direct_deps, "static", "static-", oe.spdx30.software_Package ) if dep_recipes: @@ -753,8 +753,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_objsets, dep_builds = collect_dep_objsets( - d, "builds", "build-", oe.spdx30.build_Build + d, direct_deps, "builds", "build-", oe.spdx30.build_Build ) if dep_builds: @@ -1071,7 +1073,9 @@ def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") if get_is_native(d): @@ -1248,7 +1252,9 @@ def write_bitbake_spdx(d): def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None): import oe.sbom30 - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) build_deps = set() diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 3aaf2a9c8b..c0ef11f199 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -38,7 +38,7 @@ def extract_licenses(filename): def is_work_shared_spdx(d): - return '/work-shared/' in d.getVar('S') + return "/work-shared/" in d.getVar("S") def load_spdx_license_data(d): @@ -77,12 +77,15 @@ def process_sources(d): return True -@dataclass(frozen=True) +@dataclass(frozen=True, eq=True, order=True) class Dep(object): pn: str hashfn: str in_taskhash: bool + def to_tuple(self): + return (self.pn, self.hashfn, self.in_taskhash) + def collect_direct_deps(d, dep_task): """ @@ -105,7 +108,9 @@ def collect_direct_deps(d, dep_task): ) if not dep_task in depflags: - bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + bb.fatal( + f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}" + ) for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: @@ -118,25 +123,14 @@ def collect_direct_deps(d, dep_task): for dep_name in this_dep.deps: dep_data = taskdepdata[dep_name] if dep_data.taskname == dep_task and dep_data.pn != pn: - deps.add((dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps)) + deps.add( + Dep(dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps) + ) return sorted(deps) -def get_spdx_deps(d): - """ - Reads the SPDX dependencies JSON file and returns the data - """ - spdx_deps_file = Path(d.getVar("SPDXDEPS")) - - deps = [] - with spdx_deps_file.open("r") as f: - for d in json.load(f): - deps.append(Dep(*d)) - return deps - - -def collect_package_providers(d): +def collect_package_providers(d, direct_deps): """ Returns a dictionary where each RPROVIDES is mapped to the package that provides it @@ -145,16 +139,15 @@ def collect_package_providers(d): providers = {} - deps = collect_direct_deps(d, "do_create_spdx") - deps.append((d.getVar("PN"), d.getVar("BB_HASHFILENAME"), True)) + all_deps = direct_deps + [Dep(d.getVar("PN"), d.getVar("BB_HASHFILENAME"), True)] - for dep_pn, dep_hashfn, _ in deps: + for dep in all_deps: localdata = d - recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata) + recipe_data = oe.packagedata.read_pkgdata(dep.pn, localdata) if not recipe_data: localdata = bb.data.createCopy(d) localdata.setVar("PKGDATA_DIR", "${PKGDATA_DIR_SDK}") - recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata) + recipe_data = oe.packagedata.read_pkgdata(dep.pn, localdata) for pkg in recipe_data.get("PACKAGES", "").split(): pkg_data = oe.packagedata.read_subpkgdata_dict(pkg, localdata) @@ -171,7 +164,7 @@ def collect_package_providers(d): rprovides.add(pkg) for r in rprovides: - providers[r] = (pkg, dep_hashfn) + providers[r] = (pkg, dep.hashfn) return providers @@ -202,25 +195,21 @@ def get_patched_src(d): bb.build.exec_func("do_unpack", d) if d.getVar("SRC_URI") != "": - if bb.data.inherits_class('dos2unix', d): - bb.build.exec_func('do_convert_crlf_to_lf', d) + if bb.data.inherits_class("dos2unix", d): + bb.build.exec_func("do_convert_crlf_to_lf", d) bb.build.exec_func("do_patch", d) # Copy source from work-share to spdx_workdir if is_work_shared_spdx(d): - share_src = d.getVar('S') + share_src = d.getVar("S") d.setVar("WORKDIR", spdx_workdir) d.setVar("STAGING_DIR_NATIVE", spdx_sysroot_native) # Copy source to ${SPDXWORK}, same basename dir of ${S}; - src_dir = ( - spdx_workdir - + "/" - + os.path.basename(share_src) - ) + src_dir = spdx_workdir + "/" + os.path.basename(share_src) # For kernel souce, rename suffix dir 'kernel-source' # to ${BP} (${BPN}-${PV}) if bb.data.inherits_class("kernel", d): - src_dir = spdx_workdir + "/" + d.getVar('BP') + src_dir = spdx_workdir + "/" + d.getVar("BP") bb.note(f"copyhardlinktree {share_src} to {src_dir}") oe.path.copyhardlinktree(share_src, src_dir) @@ -233,7 +222,9 @@ def get_patched_src(d): def has_task(d, task): - return bool(d.getVarFlag(task, "task", False)) and not bool(d.getVarFlag(task, "noexec", False)) + return bool(d.getVarFlag(task, "task", False)) and not bool( + d.getVarFlag(task, "noexec", False) + ) def fetch_data_to_uri(fd, name): @@ -243,8 +234,8 @@ def fetch_data_to_uri(fd, name): uri = fd.type # crate: is not a valid URL. Use url field instead if exist - if uri == "crate" and hasattr(fd,"url"): - return fd.url + if uri == "crate" and hasattr(fd, "url"): + return fd.url # Map gitsm to git, since gitsm:// is not a valid URI protocol if uri == "gitsm": @@ -259,11 +250,13 @@ def fetch_data_to_uri(fd, name): return uri -def is_compiled_source (filename, compiled_sources, types): + +def is_compiled_source(filename, compiled_sources, types): """ Check if the file is a compiled file """ import os + # If we don't have compiled source, we assume all are compiled. if not compiled_sources: return True @@ -278,11 +271,13 @@ def is_compiled_source (filename, compiled_sources, types): # Check that the file is in the list return filename in compiled_sources + def get_compiled_sources(d): """ Get list of compiled sources from debug information and normalize the paths """ import itertools + source_info = oe.package.read_debugsources_info(d) if not source_info: bb.debug(1, "Do not have debugsources.list. Skipping") -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (13 preceding siblings ...) 2026-03-10 18:38 ` [OE-core][PATCH v6 14/15] spdx: Remove do_collect_spdx_deps task Joshua Watt @ 2026-03-11 13:55 ` Mathieu Dubois-Briand 2026-03-11 16:39 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt 15 siblings, 1 reply; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-03-11 13:55 UTC (permalink / raw) To: JPEWhacker, openembedded-core; +Cc: Stefano Tondo On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote: > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > > V2: Fixes a bug where do_populate_sysroot was running when it should not > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > incorrect (this is already handled by bitbake in the taskgraph, and > doesn't need to be manually removed). > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > dependency. meta-world-recipe-sbom also no longer runs in world builds, > as there's no reason to this. Finally, fixes a bug where > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > (because do_unpack was not run). > > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > information is linked to binary packages, or just recipes. Defaults to > "0" to significantly reduce the size of the SPDX output. > > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it > does funny things with its arch which prevents it from rebuilding SPDX > data properly, and no SPDX data is needed for it anyway > > V6: Fixes a bug where SPDX task would not correctly re-run when they > change, which would cause errors about missing SPDX document. Also > updates to the latest version of the SPDX bindings which improves > performance > Hi Joshua, Ok, we are almost there! I suspect it would work fine on master, but we have a fail on two tests that were recently added by Stefano, and were not merged so far. As both series might still evolve or get reviews, I will probably keep both in my branch, but some changes are needed if we want to merge both series. 2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-11 11:31:27,495 - oe-selftest - INFO - ... FAIL ... File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling objset = self.check_recipe_spdx( File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx return self.check_spdx_file(filename) File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file self.assertExists(filename) File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists raise self.failureException(msg) AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist ... 2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-11 12:39:25,603 - oe-selftest - INFO - ... FAIL ... 2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns) 2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns objset = self.check_recipe_spdx( File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx return self.check_spdx_file(filename) File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file self.assertExists(filename) File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists raise self.failureException(msg) AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270 For reference, this oe-core branch was used during the build: https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385 Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information 2026-03-11 13:55 ` [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information Mathieu Dubois-Briand @ 2026-03-11 16:39 ` Joshua Watt 2026-03-11 19:33 ` Mathieu Dubois-Briand 0 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-11 16:39 UTC (permalink / raw) To: Mathieu Dubois-Briand; +Cc: openembedded-core, Stefano Tondo On Wed, Mar 11, 2026 at 7:55 AM Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> wrote: > > On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote: > > Changes the SPDX 3 output to include a "recipe" package that describe > > static information available at parse time (without building). This is > > primarily useful for gathering SPDX 3 VEX information about some or all > > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > > vex.bbclass. > > > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > > helping work through this. > > > > V2: Fixes a bug where do_populate_sysroot was running when it should not > > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > > incorrect (this is already handled by bitbake in the taskgraph, and > > doesn't need to be manually removed). > > > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > > dependency. meta-world-recipe-sbom also no longer runs in world builds, > > as there's no reason to this. Finally, fixes a bug where > > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > > (because do_unpack was not run). > > > > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > > information is linked to binary packages, or just recipes. Defaults to > > "0" to significantly reduce the size of the SPDX output. > > > > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it > > does funny things with its arch which prevents it from rebuilding SPDX > > data properly, and no SPDX data is needed for it anyway > > > > V6: Fixes a bug where SPDX task would not correctly re-run when they > > change, which would cause errors about missing SPDX document. Also > > updates to the latest version of the SPDX bindings which improves > > performance > > > > Hi Joshua, > > Ok, we are almost there! > > I suspect it would work fine on master, but we have a fail on two tests > that were recently added by Stefano, and were not merged so far. > > As both series might still evolve or get reviews, I will probably keep > both in my branch, but some changes are needed if we want to merge both > series. This is actually semi-intentional. I renamed the "recipe-" SPDX files to "build-". The fix is simple, but it either needs to be applied to my changes or Stephanos, depending on the order. Do you have a preference? > > 2026-03-11 11:31:27,495 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) > 2026-03-11 11:31:27,495 - oe-selftest - INFO - ... FAIL > ... > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 451, in test_download_location_defensive_handling > objset = self.check_recipe_spdx( > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx > return self.check_spdx_file(filename) > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file > self.assertExists(filename) > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists > raise self.failureException(msg) > AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-m4.spdx.json' does not exist > ... > 2026-03-11 12:39:25,602 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) > 2026-03-11 12:39:25,603 - oe-selftest - INFO - ... FAIL > ... > 2026-03-11 12:39:25,611 - oe-selftest - INFO - 6: 45/55 656/681 (14.27s) (2 failed) (spdx.SPDX30Check.test_version_extraction_patterns) > 2026-03-11 12:39:25,611 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 479, in test_version_extraction_patterns > objset = self.check_recipe_spdx( > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 123, in check_recipe_spdx > return self.check_spdx_file(filename) > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 81, in check_spdx_file > self.assertExists(filename) > File "/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists > raise self.failureException(msg) > AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-armhost/build/build-st-1004290/tmp/deploy/spdx/3.0.1/cortexa57/recipes/recipe-tar.spdx.json' does not exist > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3499 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3380 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3270 > > For reference, this oe-core branch was used during the build: > https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3385 > > Thanks, > Mathieu > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information 2026-03-11 16:39 ` Joshua Watt @ 2026-03-11 19:33 ` Mathieu Dubois-Briand 2026-03-11 22:56 ` Joshua Watt 0 siblings, 1 reply; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-03-11 19:33 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core, Stefano Tondo On Wed Mar 11, 2026 at 5:39 PM CET, Joshua Watt wrote: > On Wed, Mar 11, 2026 at 7:55 AM Mathieu Dubois-Briand > <mathieu.dubois-briand@bootlin.com> wrote: >> >> On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote: >> > Changes the SPDX 3 output to include a "recipe" package that describe >> > static information available at parse time (without building). This is >> > primarily useful for gathering SPDX 3 VEX information about some or all >> > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and >> > vex.bbclass. >> > >> > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for >> > helping work through this. >> > >> > V2: Fixes a bug where do_populate_sysroot was running when it should not >> > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is >> > incorrect (this is already handled by bitbake in the taskgraph, and >> > doesn't need to be manually removed). >> > >> > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular >> > dependency. meta-world-recipe-sbom also no longer runs in world builds, >> > as there's no reason to this. Finally, fixes a bug where >> > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx >> > (because do_unpack was not run). >> > >> > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX >> > information is linked to binary packages, or just recipes. Defaults to >> > "0" to significantly reduce the size of the SPDX output. >> > >> > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it >> > does funny things with its arch which prevents it from rebuilding SPDX >> > data properly, and no SPDX data is needed for it anyway >> > >> > V6: Fixes a bug where SPDX task would not correctly re-run when they >> > change, which would cause errors about missing SPDX document. Also >> > updates to the latest version of the SPDX bindings which improves >> > performance >> > >> >> Hi Joshua, >> >> Ok, we are almost there! >> >> I suspect it would work fine on master, but we have a fail on two tests >> that were recently added by Stefano, and were not merged so far. >> >> As both series might still evolve or get reviews, I will probably keep >> both in my branch, but some changes are needed if we want to merge both >> series. > > This is actually semi-intentional. I renamed the "recipe-" SPDX files > to "build-". The fix is simple, but it either needs to be applied to > my changes or Stephanos, depending on the order. Do you have a > preference? > I do not have any particular preference, so please do as is easier for you. -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information 2026-03-11 19:33 ` Mathieu Dubois-Briand @ 2026-03-11 22:56 ` Joshua Watt 0 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-11 22:56 UTC (permalink / raw) To: Mathieu Dubois-Briand; +Cc: openembedded-core, Stefano Tondo On Wed, Mar 11, 2026 at 1:33 PM Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> wrote: > > On Wed Mar 11, 2026 at 5:39 PM CET, Joshua Watt wrote: > > On Wed, Mar 11, 2026 at 7:55 AM Mathieu Dubois-Briand > > <mathieu.dubois-briand@bootlin.com> wrote: > >> > >> On Tue Mar 10, 2026 at 7:38 PM CET, Joshua Watt via lists.openembedded.org wrote: > >> > Changes the SPDX 3 output to include a "recipe" package that describe > >> > static information available at parse time (without building). This is > >> > primarily useful for gathering SPDX 3 VEX information about some or all > >> > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > >> > vex.bbclass. > >> > > >> > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > >> > helping work through this. > >> > > >> > V2: Fixes a bug where do_populate_sysroot was running when it should not > >> > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > >> > incorrect (this is already handled by bitbake in the taskgraph, and > >> > doesn't need to be manually removed). > >> > > >> > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > >> > dependency. meta-world-recipe-sbom also no longer runs in world builds, > >> > as there's no reason to this. Finally, fixes a bug where > >> > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > >> > (because do_unpack was not run). > >> > > >> > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > >> > information is linked to binary packages, or just recipes. Defaults to > >> > "0" to significantly reduce the size of the SPDX output. > >> > > >> > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it > >> > does funny things with its arch which prevents it from rebuilding SPDX > >> > data properly, and no SPDX data is needed for it anyway > >> > > >> > V6: Fixes a bug where SPDX task would not correctly re-run when they > >> > change, which would cause errors about missing SPDX document. Also > >> > updates to the latest version of the SPDX bindings which improves > >> > performance > >> > > >> > >> Hi Joshua, > >> > >> Ok, we are almost there! > >> > >> I suspect it would work fine on master, but we have a fail on two tests > >> that were recently added by Stefano, and were not merged so far. > >> > >> As both series might still evolve or get reviews, I will probably keep > >> both in my branch, but some changes are needed if we want to merge both > >> series. > > > > This is actually semi-intentional. I renamed the "recipe-" SPDX files > > to "build-". The fix is simple, but it either needs to be applied to > > my changes or Stephanos, depending on the order. Do you have a > > preference? > > > > I do not have any particular preference, so please do as is easier for > you. Stefano has some feedback to address, so please take my patch series first and Stefano can fix it in the next revision. Thanks > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt ` (14 preceding siblings ...) 2026-03-11 13:55 ` [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information Mathieu Dubois-Briand @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 01/12] spdx3: Add recipe SPDX data Joshua Watt ` (11 more replies) 15 siblings, 12 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Changes the SPDX 3 output to include a "recipe" package that describe static information available at parse time (without building). This is primarily useful for gathering SPDX 3 VEX information about some or all recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and vex.bbclass. Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for helping work through this. V2: Fixes a bug where do_populate_sysroot was running when it should not be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is incorrect (this is already handled by bitbake in the taskgraph, and doesn't need to be manually removed). V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular dependency. meta-world-recipe-sbom also no longer runs in world builds, as there's no reason to this. Finally, fixes a bug where NO_GENERIC_LICENSE files would fail to be found in do_create_spdx (because do_unpack was not run). V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX information is linked to binary packages, or just recipes. Defaults to "0" to significantly reduce the size of the SPDX output. V5: Fixes dummy-sdk-packages to not generate SPDX output, since it does funny things with its arch which prevents it from rebuilding SPDX data properly, and no SPDX data is needed for it anyway V6: Fixes a bug where SPDX task would not correctly re-run when they change, which would cause errors about missing SPDX document. Also updates to the latest version of the SPDX bindings which improves performance V7: Makes meta-world-recipe-sbom create the world SBoM when run with a task (e.g. as part of do_build). Drops the removal of SPDX from dummy-sdk-packages as these have been fixed to work properly. Joshua Watt (12): spdx3: Add recipe SPDX data spdx3: Add recipe SBoM task spdx3: Add is-native property spdx30: Include patch file information in VEX spdx: De-duplicate CreationInfo spdx_common: Check for dependent task in task flags spdx30: Remove package VEX spdx: Remove fatal errors for missing providers spdx3: Use common variable for vardeps glibc-testsuite: Do not generate SPDX spdx: Remove do_collect_spdx_deps task spdx: Update to latest bindings meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 2 +- meta/classes/create-spdx-2.2.bbclass | 33 +- meta/classes/create-spdx-3.0.bbclass | 81 +- meta/classes/spdx-common.bbclass | 34 +- meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/sbom30.py | 239 +- meta/lib/oe/spdx30/__init__.py | 8 + meta/lib/oe/spdx30/__main__.py | 12 + meta/lib/oe/spdx30/cmd.py | 75 + meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++------- meta/lib/oe/spdx30/stub.pyi | 2544 +++++++ meta/lib/oe/spdx30_tasks.py | 459 +- meta/lib/oe/spdx_common.py | 78 +- meta/lib/oeqa/selftest/cases/spdx.py | 28 +- .../glibc/glibc-testsuite_2.43.bb | 1 + .../meta/meta-world-recipe-sbom.bb | 38 + scripts/contrib/make-spdx-bindings.sh | 3 +- 21 files changed, 6837 insertions(+), 2748 deletions(-) create mode 100644 meta/lib/oe/spdx30/__init__.py create mode 100644 meta/lib/oe/spdx30/__main__.py create mode 100644 meta/lib/oe/spdx30/cmd.py rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%) create mode 100644 meta/lib/oe/spdx30/stub.pyi create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb -- 2.53.0 ^ permalink raw reply [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 01/12] spdx3: Add recipe SPDX data 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 02/12] spdx3: Add recipe SBoM task Joshua Watt ` (10 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a new package to the SPDX output that represents the recipe data for a given recipe. Importantly, this data contains only things that can be determined statically from only the recipe, so it doesn't require fetching or building anything. This means that build time dependencies and CVE information for recipes can be analyzed without needing to actually do any builds. Sadly, license data cannot be included because NO_GENERIC_LICENSE means that actual license text might only be available after do_fetch Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-global/sstate.bbclass | 4 +- .../create-spdx-image-3.0.bbclass | 4 +- .../create-spdx-sdk-3.0.bbclass | 4 +- meta/classes-recipe/kernel.bbclass | 2 +- meta/classes-recipe/nospdx.bbclass | 1 + meta/classes/create-spdx-2.2.bbclass | 12 +- meta/classes/create-spdx-3.0.bbclass | 51 ++- meta/classes/spdx-common.bbclass | 21 +- meta/lib/oe/spdx30_tasks.py | 402 ++++++++++++------ meta/lib/oeqa/selftest/cases/spdx.py | 19 +- 10 files changed, 354 insertions(+), 166 deletions(-) diff --git a/meta/classes-global/sstate.bbclass b/meta/classes-global/sstate.bbclass index 1c3df0f544..88449d19c7 100644 --- a/meta/classes-global/sstate.bbclass +++ b/meta/classes-global/sstate.bbclass @@ -945,7 +945,7 @@ def sstate_checkhashes(sq_data, d, siginfo=False, currentcount=0, summary=True, extrapath = d.getVar("NATIVELSBSTRING") + "/" else: extrapath = "" - + tname = bb.runqueue.taskname_from_tid(task)[3:] if tname in ["fetch", "unpack", "patch", "populate_lic", "preconfigure"] and splithashfn[2]: @@ -1107,7 +1107,7 @@ def setscene_depvalid(task, taskdependees, notneeded, d, log=None): logit("Considering setscene task: %s" % (str(taskdependees[task])), log) - directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_deploy_archives"] + directtasks = ["do_populate_lic", "do_deploy_source_date_epoch", "do_shared_workdir", "do_stash_locale", "do_gcc_stash_builddir", "do_create_spdx", "do_create_recipe_spdx", "do_deploy_archives"] def isNativeCross(x): return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross") diff --git a/meta/classes-recipe/create-spdx-image-3.0.bbclass b/meta/classes-recipe/create-spdx-image-3.0.bbclass index 636ab14eb0..15a91e90e2 100644 --- a/meta/classes-recipe/create-spdx-image-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-image-3.0.bbclass @@ -34,7 +34,7 @@ addtask do_create_rootfs_spdx after do_rootfs before do_image SSTATETASKS += "do_create_rootfs_spdx" do_create_rootfs_spdx[sstate-inputdirs] = "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" -do_create_rootfs_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_rootfs_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_rootfs_spdx[cleandirs] += "${SPDXROOTFSDEPLOY}" do_create_rootfs_spdx[file-checksums] += "${SPDX3_DEP_FILES}" @@ -76,7 +76,7 @@ do_create_image_sbom_spdx[sstate-inputdirs] = "${SPDXIMAGEDEPLOYDIR}" do_create_image_sbom_spdx[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_image_sbom_spdx[stamp-extra-info] = "${MACHINE_ARCH}" do_create_image_sbom_spdx[cleandirs] = "${SPDXIMAGEDEPLOYDIR}" -do_create_image_sbom_spdx[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_create_image_sbom_spdx[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_create_image_sbom_spdx[file-checksums] += "${SPDX3_DEP_FILES}" python do_create_image_sbom_spdx_setscene() { diff --git a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass index e5f220cdfa..a4b8ed3bf9 100644 --- a/meta/classes-recipe/create-spdx-sdk-3.0.bbclass +++ b/meta/classes-recipe/create-spdx-sdk-3.0.bbclass @@ -5,14 +5,14 @@ # # SPDX SDK tasks -do_populate_sdk[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk[cleandirs] += "${SPDXSDKWORK}" do_populate_sdk[postfuncs] += "sdk_create_sbom" do_populate_sdk[file-checksums] += "${SPDX3_DEP_FILES}" POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_create_spdx" POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_create_spdx" -do_populate_sdk_ext[recrdeptask] += "do_create_spdx do_create_package_spdx" +do_populate_sdk_ext[recrdeptask] += "do_create_recipe_spdx do_create_spdx do_create_package_spdx" do_populate_sdk_ext[cleandirs] += "${SPDXSDKEXTWORK}" do_populate_sdk_ext[postfuncs] += "sdk_ext_create_sbom" do_populate_sdk_ext[file-checksums] += "${SPDX3_DEP_FILES}" diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass index 094c1148b6..2d8565bd55 100644 --- a/meta/classes-recipe/kernel.bbclass +++ b/meta/classes-recipe/kernel.bbclass @@ -883,7 +883,7 @@ do_create_spdx:append() { except Exception as e: bb.error(f"Failed to parse kernel config file: {e}") - path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "recipes", f"recipe-{pn}", deploydir=deploydir) + path = oe.sbom30.jsonld_arch_path(d, pkg_arch, "builds", f"build-{pn}", deploydir=deploydir) build_objset = oe.sbom30.load_jsonld(d, path, required=True) build = build_objset.find_root(oe.spdx30.build_Build) if not build: diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index b20e28218b..90e14442ba 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -5,6 +5,7 @@ # deltask do_collect_spdx_deps +deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime deltask do_create_package_spdx diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 65d10d86db..3288cdf75a 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -399,6 +399,15 @@ def get_license_list_version(license_data, d): return ".".join(license_data["licenseListVersion"].split(".")[:2]) +# This task is added for compatibility with tasks shared with SPDX 3, but +# doesn't do anything +do_create_recipe_spdx() { + : +} +do_create_recipe_spdx[noexec] = "1" +addtask do_create_recipe_spdx after do_collect_spdx_deps + + python do_create_spdx() { from datetime import datetime, timezone import oe.sbom @@ -594,7 +603,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -605,6 +614,7 @@ python do_create_spdx_setscene () { } addtask do_create_spdx_setscene +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index d4575d61c4..672ca27cd0 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -159,11 +159,18 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " -python do_create_spdx() { +python do_create_recipe_spdx() { import oe.spdx30_tasks - oe.spdx30_tasks.create_spdx(d) + oe.spdx30_tasks.create_recipe_spdx(d) } -do_create_spdx[vardeps] += "\ +addtask do_create_recipe_spdx after do_collect_spdx_deps + +SSTATETASKS += "do_create_recipe_spdx" +do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" +do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[vardeps] += "\ SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ SPDX_PACKAGE_ADDITIONAL_PURPOSE \ SPDX_PROFILES \ @@ -171,7 +178,19 @@ do_create_spdx[vardeps] += "\ SPDX_UUID_NAMESPACE \ " +python do_create_recipe_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_spdx_setscene + +python do_create_spdx() { + import oe.spdx30_tasks + oe.spdx30_tasks.create_spdx(d) +} addtask do_create_spdx after \ + do_unpack \ + do_patch \ + do_create_recipe_spdx \ do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ @@ -181,18 +200,25 @@ SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_spdx[file-checksums] += "${SPDX3_DEP_FILES}" - -python do_create_spdx_setscene () { - sstate_setscene(d) -} -addtask do_create_spdx_setscene - +do_create_spdx[deptask] += "do_create_spdx" do_create_spdx[dirs] = "${SPDXWORK}" do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}" do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " +do_create_spdx[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_spdx_setscene () { + sstate_setscene(d) +} +addtask do_create_spdx_setscene python do_create_package_spdx() { import oe.spdx30_tasks @@ -205,16 +231,15 @@ SSTATETASKS += "do_create_package_spdx" do_create_package_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[rdeptask] = "do_create_spdx" python do_create_package_spdx_setscene () { sstate_setscene(d) } addtask do_create_package_spdx_setscene -do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" -do_create_package_spdx[rdeptask] = "do_create_spdx" - python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3110230c9e..3c239a718b 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -23,6 +23,7 @@ SPDXDEPS = "${SPDXDIR}/deps.json" SPDX_TOOL_NAME ??= "oe-spdx-creator" SPDX_TOOL_VERSION ??= "1.0" +SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" SPDX_INCLUDE_SOURCES ??= "0" @@ -67,12 +68,6 @@ def create_spdx_source_deps(d): deps = [] if d.getVar("SPDX_INCLUDE_SOURCES") == "1": pn = d.getVar('PN') - # do_unpack is a hack for now; we only need it to get the - # dependencies do_unpack already has so we can extract the source - # ourselves - if oe.spdx_common.has_task(d, "do_unpack"): - deps.append("%s:do_unpack" % pn) - if oe.spdx_common.is_work_shared_spdx(d) and \ oe.spdx_common.process_sources(d): # For kernel source code @@ -84,8 +79,6 @@ def create_spdx_source_deps(d): # For gcc-source-${PV} source code if oe.spdx_common.has_task(d, "do_preconfigure"): deps.append("%s:do_preconfigure" % pn) - elif oe.spdx_common.has_task(d, "do_patch"): - deps.append("%s:do_patch" % pn) # For gcc-cross-x86_64 source code elif oe.spdx_common.has_task(d, "do_configure"): deps.append("%s:do_configure" % pn) @@ -97,8 +90,8 @@ python do_collect_spdx_deps() { # This task calculates the build time dependencies of the recipe, and is # required because while a task can deptask on itself, those dependencies # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_spdx and writes out the dependencies it finds, then - # do_create_spdx reads in the found dependencies when writing the actual + # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then + # downstream tasks read in the found dependencies when writing the actual # SPDX document import json import oe.spdx_common @@ -106,15 +99,13 @@ python do_collect_spdx_deps() { spdx_deps_file = Path(d.getVar("SPDXDEPS")) - deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") with spdx_deps_file.open("w") as f: json.dump(deps, f) } -# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_collect_spdx_deps after do_unpack -do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}" -do_collect_spdx_deps[deptask] = "do_create_spdx" +addtask do_collect_spdx_deps +do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" do_collect_spdx_deps[dirs] = "${SPDXDIR}" oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 99f2892dfb..a8b4525e3d 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -32,7 +32,9 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) -def add_license_expression(d, objset, license_expression, license_data): +def add_license_expression( + d, objset, license_expression, license_data, search_objsets=[] +): simple_license_text = {} license_text_map = {} license_ref_idx = 0 @@ -44,14 +46,15 @@ def add_license_expression(d, objset, license_expression, license_data): if name in simple_license_text: return simple_license_text[name] - lic = objset.find_filter( - oe.spdx30.simplelicensing_SimpleLicensingText, - name=name, - ) + for o in [objset] + search_objsets: + lic = o.find_filter( + oe.spdx30.simplelicensing_SimpleLicensingText, + name=name, + ) - if lic is not None: - simple_license_text[name] = lic - return lic + if lic is not None: + simple_license_text[name] = lic + return lic lic = objset.add( oe.spdx30.simplelicensing_SimpleLicensingText( @@ -178,7 +181,9 @@ def add_package_files( # Check if file is compiled if check_compiled_sources: - if not oe.spdx_common.is_compiled_source(filename, compiled_sources, types): + if not oe.spdx_common.is_compiled_source( + filename, compiled_sources, types + ): continue spdx_file = objset.new_file( @@ -293,17 +298,16 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, build): +def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): deps = oe.spdx_common.get_spdx_deps(d) dep_objsets = [] - dep_builds = set() + dep_objs = set() - dep_build_spdxids = set() for dep in deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) - dep_build, dep_objset = oe.sbom30.find_root_obj_in_jsonld( - d, "recipes", "recipe-" + dep.pn, oe.spdx30.build_Build + dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( + d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter ) # If the dependency is part of the taskhash, return it to be linked # against. Otherwise, it cannot be linked against because this recipe @@ -311,10 +315,10 @@ def collect_dep_objsets(d, build): if dep.in_taskhash: dep_objsets.append(dep_objset) - # The build _can_ be linked against (by alias) - dep_builds.add(dep_build) + # The object _can_ be linked against (by alias) + dep_objs.add(dep_obj) - return dep_objsets, dep_builds + return dep_objsets, dep_objs def index_sources_by_hash(sources, dest): @@ -423,9 +427,7 @@ def add_download_files(d, objset): if fd.method.supports_checksum(fd): # TODO Need something better than hard coding this for checksum_id in ["sha256", "sha1"]: - expected_checksum = getattr( - fd, "%s_expected" % checksum_id, None - ) + expected_checksum = getattr(fd, "%s_expected" % checksum_id, None) if expected_checksum is None: continue @@ -462,50 +464,96 @@ def set_purposes(d, element, *var_names, force_purposes=[]): ] -def create_spdx(d): - def set_var_field(var, obj, name, package=None): - val = None - if package: - val = d.getVar("%s:%s" % (var, package)) +def set_purls(spdx_package, purls): + if purls: + spdx_package.software_packageUrl = purls[0] - if not val: - val = d.getVar(var) + for p in sorted(set(purls)): + spdx_package.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, + identifier=p, + ) + ) - if val: - setattr(obj, name, val) + +def create_recipe_spdx(d): + deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + pn = d.getVar("PN") license_data = oe.spdx_common.load_spdx_license_data(d) - deploydir = Path(d.getVar("SPDXDEPLOY")) - deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) - spdx_workdir = Path(d.getVar("SPDXWORK")) - include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" - pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) include_vex = d.getVar("SPDX_INCLUDE_VEX") if not include_vex in ("none", "current", "all"): bb.fatal("SPDX_INCLUDE_VEX must be one of 'none', 'current', 'all'") - build_objset = oe.sbom30.ObjectSet.new_objset(d, "recipe-" + d.getVar("PN")) + recipe_objset = oe.sbom30.ObjectSet.new_objset(d, "static-" + pn) - build = build_objset.new_task_build("recipe", "recipe") - build_objset.set_element_alias(build) + recipe = recipe_objset.add_root( + oe.spdx30.software_Package( + _id=recipe_objset.new_spdxid("recipe", pn), + creationInfo=recipe_objset.doc.creationInfo, + name=d.getVar("PN"), + software_packageVersion=d.getVar("PV"), + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_sourceInfo=json.dumps( + { + "FILENAME": os.path.basename(d.getVar("FILE")), + "FILE_LAYERNAME": d.getVar("FILE_LAYERNAME"), + }, + separators=(",", ":"), + ), + ) + ) - build_objset.doc.rootElement.append(build) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) + + # TODO: This doesn't work before do_unpack because the license text has to + # be available for recipes with NO_GENERIC_LICENSE + # recipe_spdx_license = add_license_expression( + # d, + # recipe_objset, + # d.getVar("LICENSE"), + # license_data, + # ) + # recipe_objset.new_relationship( + # [recipe], + # oe.spdx30.RelationshipType.hasDeclaredLicense, + # [oe.sbom30.get_element_link_id(recipe_spdx_license)], + # ) + + if val := d.getVar("HOMEPAGE"): + recipe.software_homePage = val + + if val := d.getVar("SUMMARY"): + recipe.summary = val + + if val := d.getVar("DESCRIPTION"): + recipe.description = val + + for cpe_id in oe.cve_check.get_cpe_ids( + d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") + ): + recipe.externalIdentifier.append( + oe.spdx30.ExternalIdentifier( + externalIdentifierType=oe.spdx30.ExternalIdentifierType.cpe23, + identifier=cpe_id, + ) + ) - build_objset.set_is_native(is_native) + dep_objsets, dep_recipes = collect_dep_objsets( + d, "static", "static-", oe.spdx30.software_Package + ) - for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): - build_objset.new_annotation( - build, - "%s=%s" % (var, d.getVar(var)), - oe.spdx30.AnnotationType.other, + if dep_recipes: + recipe_objset.new_scoped_relationship( + [recipe], + oe.spdx30.RelationshipType.dependsOn, + oe.spdx30.LifecycleScopeType.build, + sorted(oe.sbom30.get_element_link_id(dep) for dep in dep_recipes), ) - build_inputs = set() - # Add CVEs cve_by_status = {} if include_vex != "none": @@ -514,7 +562,7 @@ def create_spdx(d): decoded_status = { "mapping": patched_cve["abbrev-status"], "detail": patched_cve["status"], - "description": patched_cve.get("justification", None) + "description": patched_cve.get("justification", None), } # If this CVE is fixed upstream, skip it unless all CVEs are @@ -531,8 +579,7 @@ def create_spdx(d): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue - spdx_cve = build_objset.new_cve_vuln(cve) - build_objset.set_element_alias(spdx_cve) + spdx_cve = recipe_objset.new_cve_vuln(cve) cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( spdx_cve, @@ -540,13 +587,118 @@ def create_spdx(d): decoded_status["description"], ) + all_cves = set() + for status, cves in cve_by_status.items(): + for cve, items in cves.items(): + spdx_cve, detail, description = items + spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) + + all_cves.add(spdx_cve) + + if status == "Patched": + recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + elif status == "Unpatched": + recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) + elif status == "Ignored": + spdx_vex = recipe_objset.new_vex_ignored_relationship( + [spdx_cve_id], + [recipe], + impact_statement=description, + ) + + vex_just_type = d.getVarFlag("CVE_CHECK_VEX_JUSTIFICATION", detail) + if vex_just_type: + if ( + vex_just_type + not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS + ): + bb.fatal( + f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" + ) + + for v in spdx_vex: + v.security_justificationType = ( + oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ + vex_just_type + ] + ) + + elif status == "Unknown": + bb.note(f"Skipping {cve} with status 'Unknown'") + else: + bb.fatal(f"Unknown {cve} status '{status}'") + + if all_cves: + recipe_objset.new_relationship( + [recipe], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted(list(all_cves)), + ) + + oe.sbom30.write_recipe_jsonld_doc(d, recipe_objset, "static", deploydir) + + +def load_recipe_spdx(d): + + return oe.sbom30.find_root_obj_in_jsonld( + d, + "static", + "static-" + d.getVar("PN"), + oe.spdx30.software_Package, + ) + + +def create_spdx(d): + def set_var_field(var, obj, name, package=None): + val = None + if package: + val = d.getVar("%s:%s" % (var, package)) + + if not val: + val = d.getVar(var) + + if val: + setattr(obj, name, val) + + license_data = oe.spdx_common.load_spdx_license_data(d) + + pn = d.getVar("PN") + deploydir = Path(d.getVar("SPDXDEPLOY")) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + spdx_workdir = Path(d.getVar("SPDXWORK")) + include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" + pkg_arch = d.getVar("SSTATE_PKGARCH") + is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( + "cross", d + ) + + recipe, recipe_objset = load_recipe_spdx(d) + + build_objset = oe.sbom30.ObjectSet.new_objset(d, "build-" + pn) + + build = build_objset.new_task_build("recipe", "recipe") + build_objset.set_element_alias(build) + + build_objset.doc.rootElement.append(build) + + build_objset.set_is_native(is_native) + + for var in (d.getVar("SPDX_CUSTOM_ANNOTATION_VARS") or "").split(): + build_objset.new_annotation( + build, + "%s=%s" % (var, d.getVar(var)), + oe.spdx30.AnnotationType.other, + ) + + build_inputs = set() + cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")) source_files = add_download_files(d, build_objset) build_inputs |= source_files recipe_spdx_license = add_license_expression( - d, build_objset, d.getVar("LICENSE"), license_data + d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset] ) build_objset.new_relationship( source_files, @@ -575,7 +727,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) - dep_objsets, dep_builds = collect_dep_objsets(d, build) + dep_objsets, dep_builds = collect_dep_objsets( + d, "builds", "build-", oe.spdx30.build_Build + ) + if dep_builds: build_objset.new_scoped_relationship( [build], @@ -587,6 +742,22 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} + # Collect all VEX statements from the recipe + vex_statements = {} + for rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, + ): + for cve in rel.to: + vex_statements[cve] = [] + + for cve in vex_statements.keys(): + for rel in recipe_objset.foreach_filter( + oe.spdx30.security_VexVulnAssessmentRelationship, + from_=cve, + ): + vex_statements[cve].append(rel) + # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -645,16 +816,7 @@ def create_spdx(d): or "" ).split() - if purls: - spdx_package.software_packageUrl = purls[0] - - for p in sorted(set(purls)): - spdx_package.externalIdentifier.append( - oe.spdx30.ExternalIdentifier( - externalIdentifierType=oe.spdx30.ExternalIdentifierType.packageUrl, - identifier=p, - ) - ) + set_purls(spdx_package, purls) pkg_objset.new_scoped_relationship( [oe.sbom30.get_element_link_id(build)], @@ -663,6 +825,13 @@ def create_spdx(d): [spdx_package], ) + pkg_objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(recipe)], + oe.spdx30.RelationshipType.generates, + oe.spdx30.LifecycleScopeType.build, + [spdx_package], + ) + for cpe_id in cpe_ids: spdx_package.externalIdentifier.append( oe.spdx30.ExternalIdentifier( @@ -696,7 +865,11 @@ def create_spdx(d): package_license = d.getVar("LICENSE:%s" % package) if package_license and package_license != d.getVar("LICENSE"): package_spdx_license = add_license_expression( - d, build_objset, package_license, license_data + d, + build_objset, + package_license, + license_data, + [recipe_objset], ) else: package_spdx_license = recipe_spdx_license @@ -721,58 +894,41 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # NOTE: CVE Elements live in the recipe collection - all_cves = set() - for status, cves in cve_by_status.items(): - for cve, items in cves.items(): - spdx_cve, detail, description = items - spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) - - all_cves.add(spdx_cve_id) + # Copy CVEs from recipe + if vex_statements: + pkg_objset.new_relationship( + [spdx_package], + oe.spdx30.RelationshipType.hasAssociatedVulnerability, + sorted( + oe.sbom30.get_element_link_id(cve) + for cve in vex_statements.keys() + ), + ) - if status == "Patched": + for cve, vexes in vex_statements.items(): + for vex in vexes: + if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: pkg_objset.new_vex_patched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Unpatched": + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( - [spdx_cve_id], [spdx_package] + [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) - elif status == "Ignored": + elif ( + vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect + ): spdx_vex = pkg_objset.new_vex_ignored_relationship( - [spdx_cve_id], + [oe.sbom30.get_element_link_id(cve)], [spdx_package], - impact_statement=description, + impact_statement=vex.security_impactStatement, ) - vex_just_type = d.getVarFlag( - "CVE_CHECK_VEX_JUSTIFICATION", detail - ) - if vex_just_type: - if ( - vex_just_type - not in oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS - ): - bb.fatal( - f"Unknown vex justification '{vex_just_type}', detail '{detail}', for ignored {cve}" - ) - + if vex.security_justificationType: for v in spdx_vex: - v.security_justificationType = oe.spdx30.security_VexJustificationType.NAMED_INDIVIDUALS[ - vex_just_type - ] - - elif status == "Unknown": - bb.note(f"Skipping {cve} with status 'Unknown'") - else: - bb.fatal(f"Unknown {cve} status '{status}'") - - if all_cves: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted(list(all_cves)), - ) + v.security_justificationType = ( + vex.security_justificationType + ) bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( @@ -851,14 +1007,15 @@ def create_spdx(d): status = "enabled" if feature in enabled else "disabled" build.build_parameter.append( oe.spdx30.DictionaryEntry( - key=f"PACKAGECONFIG:{feature}", - value=status + key=f"PACKAGECONFIG:{feature}", value=status ) ) - bb.note(f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled") + bb.note( + f"Added PACKAGECONFIG entries: {len(enabled)} enabled, {len(disabled)} disabled" + ) - oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "recipes", deploydir) + oe.sbom30.write_recipe_jsonld_doc(d, build_objset, "builds", deploydir) def create_package_spdx(d): @@ -1197,17 +1354,17 @@ def create_image_spdx(d): image_path = image_deploy_dir / image_filename if os.path.isdir(image_path): a = add_package_files( - d, - objset, - image_path, - lambda file_counter: objset.new_spdxid( - "imagefile", str(file_counter) - ), - lambda filepath: [], - license_data=None, - ignore_dirs=[], - ignore_top_level_dirs=[], - archive=None, + d, + objset, + image_path, + lambda file_counter: objset.new_spdxid( + "imagefile", str(file_counter) + ), + lambda filepath: [], + license_data=None, + ignore_dirs=[], + ignore_top_level_dirs=[], + archive=None, ) artifacts.extend(a) else: @@ -1234,7 +1391,6 @@ def create_image_spdx(d): set_timestamp_now(d, a, "builtTime") - if artifacts: objset.new_scoped_relationship( [image_build], diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 5830d7c087..759ca86b73 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -141,6 +141,11 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): SPDX_CLASS = "create-spdx-3.0" def test_base_files(self): + self.check_recipe_spdx( + "base-files", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/static/static-base-files.spdx.json", + task="create_recipe_spdx", + ) self.check_recipe_spdx( "base-files", "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", @@ -149,7 +154,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-gcc.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-gcc.spdx.json", extraconf="""\ SPDX_INCLUDE_SOURCES = "1" """, @@ -162,12 +167,12 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): if software_file.name == filename: found = True self.logger.info( - f"The spdxId of {filename} in recipe-gcc.spdx.json is {software_file.spdxId}" + f"The spdxId of {filename} in build-gcc.spdx.json is {software_file.spdxId}" ) break self.assertTrue( - found, f"Not found source file {filename} in recipe-gcc.spdx.json\n" + found, f"Not found source file {filename} in build-gcc.spdx.json\n" ) def test_core_image_minimal(self): @@ -305,7 +310,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): # This will fail with NameError if new_annotation() is called incorrectly objset = self.check_recipe_spdx( "base-files", - "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/recipes/recipe-base-files.spdx.json", + "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/builds/build-base-files.spdx.json", extraconf=textwrap.dedent( f"""\ ANNOTATION1 = "{ANNOTATION_VAR1}" @@ -360,8 +365,8 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_kernel_config_spdx(self): kernel_recipe = get_bb_var("PREFERRED_PROVIDER_virtual/kernel") - spdx_file = f"recipe-{kernel_recipe}.spdx.json" - spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/recipes/{spdx_file}" + spdx_file = f"build-{kernel_recipe}.spdx.json" + spdx_path = f"{{DEPLOY_DIR_SPDX}}/{{SSTATE_PKGARCH}}/builds/{spdx_file}" # Make sure kernel is configured first bitbake(f"-c configure {kernel_recipe}") @@ -392,7 +397,7 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): def test_packageconfig_spdx(self): objset = self.check_recipe_spdx( "tar", - "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/recipes/recipe-tar.spdx.json", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-tar.spdx.json", extraconf="""\ SPDX_INCLUDE_PACKAGECONFIG = "1" """, -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 02/12] spdx3: Add recipe SBoM task 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 01/12] spdx3: Add recipe SPDX data Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 03/12] spdx3: Add is-native property Joshua Watt ` (9 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a task that will create the complete recipe-level SBoM for a given target recipe, following all dependencies. For example: ``` bitbake -c create_recipe_sbom zstd ``` Would produce the complete recipe SBoM for the zstd recipe, include all build time dependencies (recursively). The complete SBoM for all (target) recipes can be built with: ``` bitbake meta-world-recipe-sbom ``` Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 32 ++++++++++++++++ meta/classes/spdx-common.bbclass | 1 + meta/conf/distro/include/maintainers.inc | 1 + meta/lib/oe/spdx30_tasks.py | 10 +++++ meta/lib/oeqa/selftest/cases/spdx.py | 9 +++++ .../meta/meta-world-recipe-sbom.bb | 38 +++++++++++++++++++ 6 files changed, 91 insertions(+) create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 672ca27cd0..c3ea95b8bc 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -142,6 +142,10 @@ SPDX_PACKAGE_URLS[doc] = "A space separated list of Package URLs (purls) for \ Override this variable to replace the default, otherwise append or prepend \ to add additional purls." +SPDX_RECIPE_SBOM_NAME ?= "${PN}-recipe-sbom" +SPDX_RECIPE_SBOM_NAME[doc] = "The name of output recipe SBoM when using \ + create_recipe_sbom" + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0" @@ -240,6 +244,34 @@ python do_create_package_spdx_setscene () { } addtask do_create_package_spdx_setscene +addtask do_create_recipe_sbom after create_recipe_spdx +python do_create_recipe_sbom() { + import oe.spdx30_tasks + from pathlib import Path + deploydir = Path(d.getVar("SPDXRECIPESBOMDEPLOY")) + oe.spdx30_tasks.create_recipe_sbom(d, deploydir) +} + +SSTATETASKS += "do_create_recipe_sbom" +do_create_recipe_sbom[recrdeptask] = "do_create_recipe_spdx" +do_create_recipe_sbom[nostamp] = "1" +do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" +do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" +do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" +do_create_recipe_sbom[vardeps] += "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + +python do_create_recipe_sbom_setscene () { + sstate_setscene(d) +} +addtask do_create_recipe_sbom_setscene + python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 3c239a718b..abf2332bee 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -25,6 +25,7 @@ SPDX_TOOL_VERSION ??= "1.0" SPDXRECIPEDEPLOY = "${SPDXDIR}/recipe-deploy" SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy" +SPDXRECIPESBOMDEPLOY = "${SPDXDIR}/recipes-bom-deploy" SPDX_INCLUDE_SOURCES ??= "0" SPDX_INCLUDE_COMPILED_SOURCES ??= "0" diff --git a/meta/conf/distro/include/maintainers.inc b/meta/conf/distro/include/maintainers.inc index c7a646a643..86a11867a1 100644 --- a/meta/conf/distro/include/maintainers.inc +++ b/meta/conf/distro/include/maintainers.inc @@ -534,6 +534,7 @@ RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <richard.purdie@linuxfo RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-toolchain = "Richard Purdie <richard.purdie@linuxfoundation.org>" RECIPE_MAINTAINER:pn-meta-world-pkgdata = "Richard Purdie <richard.purdie@linuxfoundation.org>" +RECIPE_MAINTAINER:pn-meta-world-recipe-sbom = "Joshua Watt <JPEWhacker@gmail.com>" RECIPE_MAINTAINER:pn-mingetty = "Yi Zhao <yi.zhao@windriver.com>" RECIPE_MAINTAINER:pn-mini-x-session = "Unassigned <unassigned@yoctoproject.org>" RECIPE_MAINTAINER:pn-minicom = "Unassigned <unassigned@yoctoproject.org>" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8b4525e3d..b6c917045e 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1564,3 +1564,13 @@ def create_sdk_sbom(d, sdk_deploydir, spdx_work_dir, toolchain_outputname): oe.sbom30.write_jsonld_doc( d, objset, sdk_deploydir / (toolchain_outputname + ".spdx.json") ) + + +def create_recipe_sbom(d, deploydir): + sbom_name = d.getVar("SPDX_RECIPE_SBOM_NAME") + + recipe, recipe_objset = load_recipe_spdx(d) + + objset, sbom = oe.sbom30.create_sbom(d, sbom_name, [recipe], [recipe_objset]) + + oe.sbom30.write_jsonld_doc(d, objset, deploydir / (sbom_name + ".spdx.json")) diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py index 759ca86b73..af1144c1e5 100644 --- a/meta/lib/oeqa/selftest/cases/spdx.py +++ b/meta/lib/oeqa/selftest/cases/spdx.py @@ -151,6 +151,15 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): "{DEPLOY_DIR_SPDX}/{MACHINE_ARCH}/packages/package-base-files.spdx.json", ) + def test_world_sbom(self): + objset = self.check_recipe_spdx( + "meta-world-recipe-sbom", + "{DEPLOY_DIR_IMAGE}/world-recipe-sbom.spdx.json", + ) + + # Document should be fully linked + self.check_objset_missing_ids(objset) + def test_gcc_include_source(self): objset = self.check_recipe_spdx( "gcc", diff --git a/meta/recipes-core/meta/meta-world-recipe-sbom.bb b/meta/recipes-core/meta/meta-world-recipe-sbom.bb new file mode 100644 index 0000000000..841fb8ea16 --- /dev/null +++ b/meta/recipes-core/meta/meta-world-recipe-sbom.bb @@ -0,0 +1,38 @@ +SUMMARY = "Generates a combined SBoM for all world recipes" +LICENSE = "MIT" + +INHIBIT_DEFAULT_DEPS = "1" + +PACKAGE_ARCH = "${MACHINE_ARCH}" + +inherit nopackages +deltask do_fetch +deltask do_unpack +deltask do_patch +deltask do_configure +deltask do_compile +deltask do_install + +do_prepare_recipe_sysroot[deptask] = "" + +WORLD_SBOM_EXCLUDE ?= "" + +EXCLUDE_FROM_WORLD = "1" +SPDX_RECIPE_SBOM_NAME = "world-recipe-sbom" + +python calculate_extra_depends() { + exclude = set('${WORLD_SBOM_EXCLUDE}'.split()) + exclude |= set(f"{v}-{self_pn}" for v in '${MULTILIB_VARIANTS}'.split()) + exclude.add(self_pn) + + deps.extend(p for p in world_target if p not in exclude) +} + +python() { + # Ensure that do_create_recipe_sbom is the only dependency of do_build, + # since the sole purpose of this recipe is to produce the world recipe SBoM + d.setVarFlag("do_build", "deps", ["do_create_recipe_sbom"]) + d.setVarFlag("do_build", "deptask", "") + d.setVarFlag("do_build", "rdeptask", "") + d.setVarFlag("do_build", "recrdeptask", "") +} -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 03/12] spdx3: Add is-native property 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 01/12] spdx3: Add recipe SPDX data Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 02/12] spdx3: Add recipe SBoM task Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 04/12] spdx30: Include patch file information in VEX Joshua Watt ` (8 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Adds a custom is-native property to the recipe package to indicate if it is a native recipe Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 20 ++++++++++++++++++++ meta/lib/oe/spdx30_tasks.py | 18 +++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 227ac51877..50a72fce39 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -118,6 +118,26 @@ class OEDocumentExtension(oe.spdx30.extension_Extension): ) +@oe.spdx30.register(OE_SPDX_BASE + "recipe-extension") +class OERecipeExtension(oe.spdx30.extension_Extension): + """ + This extension is added to recipe software_Packages to indicate various + useful bits of information about the recipe + """ + + CLOSED = True + + @classmethod + def _register_props(cls): + super()._register_props() + cls._add_property( + "is_native", + oe.spdx30.BooleanProp(), + OE_SPDX_BASE + "is-native", + max_count=1, + ) + + def spdxid_hash(*items): h = hashlib.md5() for i in items: diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index b6c917045e..a8fffbb085 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,10 @@ def set_purls(spdx_package, purls): ) +def get_is_native(d): + return bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d) + + def create_recipe_spdx(d): deploydir = Path(d.getVar("SPDXRECIPEDEPLOY")) deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) @@ -507,6 +511,11 @@ def create_recipe_spdx(d): ) ) + if get_is_native(d): + ext = oe.sbom30.OERecipeExtension() + ext.is_native = True + recipe.extension.append(ext) + set_purls(recipe, (d.getVar("SPDX_PACKAGE_URLS") or "").split()) # TODO: This doesn't work before do_unpack because the license text has to @@ -668,9 +677,7 @@ def create_spdx(d): spdx_workdir = Path(d.getVar("SPDXWORK")) include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1" pkg_arch = d.getVar("SSTATE_PKGARCH") - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) + is_native = get_is_native(d) recipe, recipe_objset = load_recipe_spdx(d) @@ -1021,14 +1028,11 @@ def create_spdx(d): def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class( - "cross", d - ) providers = oe.spdx_common.collect_package_providers(d) pkg_arch = d.getVar("SSTATE_PKGARCH") - if is_native: + if get_is_native(d): return bb.build.exec_func("read_subpackage_metadata", d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 04/12] spdx30: Include patch file information in VEX 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (2 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 03/12] spdx3: Add is-native property Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 05/12] spdx: De-duplicate CreationInfo Joshua Watt ` (7 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Modifies the SPDX VEX output to include the patches that fix a particular vulnerability. This is done by adding a `patchedBy` relationship from the `VexFixedVulnAssessmentRelationship` to the `File` that provides the fix. If the file can be located without fetching (e.g. is a file:// in SRC_URI), the checksum will be included. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 60 ++++++++++++++------------- meta/lib/oe/spdx30_tasks.py | 81 ++++++++++++++++++++++++++++--------- 2 files changed, 92 insertions(+), 49 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 50a72fce39..21f084dc16 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -620,37 +620,38 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): - sha256_hash = bb.utils.sha256_file(path) + def new_file(self, _id, name, path, *, purposes=[], hashfile=True): + if hashfile: + sha256_hash = bb.utils.sha256_file(path) - for f in self.by_sha256_hash.get(sha256_hash, []): - if not isinstance(f, oe.spdx30.software_File): - continue + for f in self.by_sha256_hash.get(sha256_hash, []): + if not isinstance(f, oe.spdx30.software_File): + continue - if purposes: - new_primary = purposes[0] - new_additional = [] + if purposes: + new_primary = purposes[0] + new_additional = [] - if f.software_primaryPurpose: - new_additional.append(f.software_primaryPurpose) - new_additional.extend(f.software_additionalPurpose) + if f.software_primaryPurpose: + new_additional.append(f.software_primaryPurpose) + new_additional.extend(f.software_additionalPurpose) - new_additional = sorted( - list(set(p for p in new_additional if p != new_primary)) - ) + new_additional = sorted( + list(set(p for p in new_additional if p != new_primary)) + ) - f.software_primaryPurpose = new_primary - f.software_additionalPurpose = new_additional + f.software_primaryPurpose = new_primary + f.software_additionalPurpose = new_additional - if f.name != name: - for e in f.extension: - if isinstance(e, OEFileNameAliasExtension): - e.aliases.append(name) - break - else: - f.extension.append(OEFileNameAliasExtension(aliases=[name])) + if f.name != name: + for e in f.extension: + if isinstance(e, OEFileNameAliasExtension): + e.aliases.append(name) + break + else: + f.extension.append(OEFileNameAliasExtension(aliases=[name])) - return f + return f spdx_file = oe.spdx30.software_File( _id=_id, @@ -661,12 +662,13 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] - spdx_file.verifiedUsing.append( - oe.spdx30.Hash( - algorithm=oe.spdx30.HashAlgorithm.sha256, - hashValue=sha256_hash, + if hashfile: + spdx_file.verifiedUsing.append( + oe.spdx30.Hash( + algorithm=oe.spdx30.HashAlgorithm.sha256, + hashValue=sha256_hash, + ) ) - ) return self.add(spdx_file) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index a8fffbb085..aec47d4f81 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -568,44 +568,63 @@ def create_recipe_spdx(d): if include_vex != "none": patched_cves = oe.cve_check.get_patched_cves(d) for cve, patched_cve in patched_cves.items(): - decoded_status = { - "mapping": patched_cve["abbrev-status"], - "detail": patched_cve["status"], - "description": patched_cve.get("justification", None), - } + mapping = patched_cve["abbrev-status"] + detail = patched_cve["status"] + description = patched_cve.get("justification", None) + resources = patched_cve.get("resource", []) # If this CVE is fixed upstream, skip it unless all CVEs are # specified. - if ( - include_vex != "all" - and "detail" in decoded_status - and decoded_status["detail"] - in ( - "fixed-version", - "cpe-stable-backport", - ) + if include_vex != "all" and detail in ( + "fixed-version", + "cpe-stable-backport", ): bb.debug(1, "Skipping %s since it is already fixed upstream" % cve) continue spdx_cve = recipe_objset.new_cve_vuln(cve) - cve_by_status.setdefault(decoded_status["mapping"], {})[cve] = ( + cve_by_status.setdefault(mapping, {})[cve] = ( spdx_cve, - decoded_status["detail"], - decoded_status["description"], + detail, + description, + resources, ) all_cves = set() for status, cves in cve_by_status.items(): for cve, items in cves.items(): - spdx_cve, detail, description = items + spdx_cve, detail, description, resources = items spdx_cve_id = oe.sbom30.get_element_link_id(spdx_cve) all_cves.add(spdx_cve) if status == "Patched": - recipe_objset.new_vex_patched_relationship([spdx_cve_id], [recipe]) + spdx_vex = recipe_objset.new_vex_patched_relationship( + [spdx_cve_id], [recipe] + ) + patches = [] + for idx, filepath in enumerate(resources): + patches.append( + recipe_objset.new_file( + recipe_objset.new_spdxid( + "patch", str(idx), os.path.basename(filepath) + ), + os.path.basename(filepath), + filepath, + purposes=[oe.spdx30.software_SoftwarePurpose.patch], + hashfile=os.path.isfile(filepath), + ) + ) + + if patches: + recipe_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + patches, + ) + elif status == "Unpatched": recipe_objset.new_vex_unpatched_relationship([spdx_cve_id], [recipe]) elif status == "Ignored": @@ -751,12 +770,14 @@ def create_spdx(d): # Collect all VEX statements from the recipe vex_statements = {} + vex_patches = {} for rel in recipe_objset.foreach_filter( oe.spdx30.Relationship, relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, ): for cve in rel.to: vex_statements[cve] = [] + vex_patches[cve] = [] for cve in vex_statements.keys(): for rel in recipe_objset.foreach_filter( @@ -764,6 +785,13 @@ def create_spdx(d): from_=cve, ): vex_statements[cve].append(rel) + if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: + for patch_rel in recipe_objset.foreach_filter( + oe.spdx30.Relationship, + relationshipType=oe.spdx30.RelationshipType.patchedBy, + from_=rel, + ): + vex_patches[cve].extend(patch_rel.to) # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task @@ -889,7 +917,9 @@ def create_spdx(d): # Add concluded license relationship if manually set # Only add when license analysis has been explicitly performed - concluded_license_str = d.getVar("SPDX_CONCLUDED_LICENSE:%s" % package) or d.getVar("SPDX_CONCLUDED_LICENSE") + concluded_license_str = d.getVar( + "SPDX_CONCLUDED_LICENSE:%s" % package + ) or d.getVar("SPDX_CONCLUDED_LICENSE") if concluded_license_str: concluded_spdx_license = add_license_expression( d, build_objset, concluded_license_str, license_data @@ -915,9 +945,20 @@ def create_spdx(d): for cve, vexes in vex_statements.items(): for vex in vexes: if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - pkg_objset.new_vex_patched_relationship( + spdx_vex = pkg_objset.new_vex_patched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] ) + if vex_patches[cve]: + pkg_objset.new_scoped_relationship( + spdx_vex, + oe.spdx30.RelationshipType.patchedBy, + oe.spdx30.LifecycleScopeType.build, + [ + oe.sbom30.get_element_link_id(p) + for p in vex_patches[cve] + ], + ) + elif vex.relationshipType == oe.spdx30.RelationshipType.affects: pkg_objset.new_vex_unpatched_relationship( [oe.sbom30.get_element_link_id(cve)], [spdx_package] -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 05/12] spdx: De-duplicate CreationInfo 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (3 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 04/12] spdx30: Include patch file information in VEX Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 06/12] spdx_common: Check for dependent task in task flags Joshua Watt ` (6 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt De-duplicates CreationInfo objects that are identical (except for ID) when writing out an SBoM. This significantly reduces the number of CreationInfo objects that end up in the final document. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/sbom30.py | 112 ++++++++++++++++++++++++++++++------------ meta/lib/oe/spdx30.py | 2 +- 2 files changed, 81 insertions(+), 33 deletions(-) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 21f084dc16..55a2863d2d 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -14,6 +14,7 @@ import uuid import os import oe.spdx_common from datetime import datetime, timezone +from contextlib import contextmanager OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" @@ -191,6 +192,25 @@ def to_list(l): return l +class Dedup(object): + def __init__(self, objset): + self.unique = set() + self.dedup = {} + self.objset = objset + + def find_duplicates(self, cmp, typ, **kwargs): + for o in self.objset.foreach_filter(typ, **kwargs): + for u in self.unique: + if cmp(u, o): + self.dedup[o] = u + break + else: + self.unique.add(o) + + def get(self, o): + return self.dedup.get(o, o) + + class ObjectSet(oe.spdx30.SHACLObjectSet): def __init__(self, d): super().__init__() @@ -895,6 +915,45 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): self.missing_ids -= set(imports.keys()) return self.missing_ids + @contextmanager + def deduplicate(self): + d = Dedup(self) + + yield d + + visited = set() + + def visit(o, path): + if isinstance(o, oe.spdx30.SHACLObject): + if o in visited: + return False + visited.add(o) + + for k in o: + v = o[k] + if isinstance(v, oe.spdx30.SHACLObject): + o[k] = d.get(v) + + elif isinstance(o, oe.spdx30.ListProxy): + for idx, v in enumerate(o): + if isinstance(v, oe.spdx30.SHACLObject): + o[idx] = d.get(v) + + return True + + if d.dedup: + for o in self.objects: + o.walk(visit) + + for k, v in d.dedup.items(): + bb.debug( + 1, + f"Removing duplicate {k.__class__.__name__} {k._id or id(k)} -> {v._id or id(v)}", + ) + self.objects.discard(k) + + self.create_index() + def load_jsonld(d, path, required=False): deserializer = oe.spdx30.JSONLDDeserializer() @@ -1080,39 +1139,28 @@ def create_sbom(d, name, root_elements, add_objectsets=[]): # SBoM should be the only root element of the document objset.doc.rootElement = [sbom] - # De-duplicate licenses - unique = set() - dedup = {} - for lic in objset.foreach_type(oe.spdx30.simplelicensing_LicenseExpression): - for u in unique: - if ( - u.simplelicensing_licenseExpression - == lic.simplelicensing_licenseExpression - and u.simplelicensing_licenseListVersion - == lic.simplelicensing_licenseListVersion - ): - dedup[lic] = u - break - else: - unique.add(lic) - - if dedup: - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasDeclaredLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] - - for rel in objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasConcludedLicense, - ): - rel.to = [dedup.get(to, to) for to in rel.to] + def cmp_license_expression(a, b): + return ( + a.simplelicensing_licenseExpression == b.simplelicensing_licenseExpression + and a.simplelicensing_licenseListVersion + == b.simplelicensing_licenseListVersion + ) - for k, v in dedup.items(): - bb.debug(1, f"Removing duplicate License {k._id} -> {v._id}") - objset.objects.remove(k) + def cmp_creation_info(a, b): + data_a = {k: a[k] for k in a} + data_b = {k: b[k] for k in b} + data_a["@id"] = "" + data_b["@id"] = "" + return data_a == data_b + + with objset.deduplicate() as dedup: + # De-duplicate licenses + dedup.find_duplicates( + cmp_license_expression, + oe.spdx30.simplelicensing_LicenseExpression, + ) - objset.create_index() + # Deduplicate creation info + dedup.find_duplicates(cmp_creation_info, oe.spdx30.CreationInfo) return objset, sbom diff --git a/meta/lib/oe/spdx30.py b/meta/lib/oe/spdx30.py index cd97eebd18..1f58402ffc 100644 --- a/meta/lib/oe/spdx30.py +++ b/meta/lib/oe/spdx30.py @@ -701,7 +701,7 @@ class SHACLObject(object): self.__dict__["_obj_data"][iri] = prop.init() def __iter__(self): - return self._OBJ_PROPERTIES.keys() + return iter(self._OBJ_PROPERTIES.keys()) def walk(self, callback, path=None): """ -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 06/12] spdx_common: Check for dependent task in task flags 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (4 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 05/12] spdx: De-duplicate CreationInfo Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 07/12] spdx30: Remove package VEX Joshua Watt ` (5 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Checks that the task being used to detect dependencies is present in at least one dependency task flag of the current task. This helps prevent errors where the wrong task is specified and never found. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx_common.py | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 72c24180d5..3aaf2a9c8b 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -96,6 +96,17 @@ def collect_direct_deps(d, dep_task): taskdepdata = d.getVar("BB_TASKDEPDATA", False) + # Check that the task is listed one of the task dependency flags of the + # current task + depflags = ( + set((d.getVarFlag(current_task, "deptask") or "").split()) + | set((d.getVarFlag(current_task, "rdeptask") or "").split()) + | set((d.getVarFlag(current_task, "recrdeptask") or "").split()) + ) + + if not dep_task in depflags: + bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: break -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 07/12] spdx30: Remove package VEX 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (5 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 06/12] spdx_common: Check for dependent task in task flags Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 08/12] spdx: Remove fatal errors for missing providers Joshua Watt ` (4 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Removes VEX statements from packages. These are no longer necessary since the VEX data is now attached to the recipes, which significantly reduces the duplication of the data, and thus the size of the SPDX output files. Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/lib/oe/spdx30_tasks.py | 72 ------------------------------------- 1 file changed, 72 deletions(-) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index aec47d4f81..5b651900c4 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -768,31 +768,6 @@ def create_spdx(d): debug_source_ids = set() source_hash_cache = {} - # Collect all VEX statements from the recipe - vex_statements = {} - vex_patches = {} - for rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.hasAssociatedVulnerability, - ): - for cve in rel.to: - vex_statements[cve] = [] - vex_patches[cve] = [] - - for cve in vex_statements.keys(): - for rel in recipe_objset.foreach_filter( - oe.spdx30.security_VexVulnAssessmentRelationship, - from_=cve, - ): - vex_statements[cve].append(rel) - if rel.relationshipType == oe.spdx30.RelationshipType.fixedIn: - for patch_rel in recipe_objset.foreach_filter( - oe.spdx30.Relationship, - relationshipType=oe.spdx30.RelationshipType.patchedBy, - from_=rel, - ): - vex_patches[cve].extend(patch_rel.to) - # Write out the package SPDX data now. It is not complete as we cannot # write the runtime data, so write it to a staging area and a later task # will write out the final collection @@ -931,53 +906,6 @@ def create_spdx(d): [oe.sbom30.get_element_link_id(concluded_spdx_license)], ) - # Copy CVEs from recipe - if vex_statements: - pkg_objset.new_relationship( - [spdx_package], - oe.spdx30.RelationshipType.hasAssociatedVulnerability, - sorted( - oe.sbom30.get_element_link_id(cve) - for cve in vex_statements.keys() - ), - ) - - for cve, vexes in vex_statements.items(): - for vex in vexes: - if vex.relationshipType == oe.spdx30.RelationshipType.fixedIn: - spdx_vex = pkg_objset.new_vex_patched_relationship( - [oe.sbom30.get_element_link_id(cve)], [spdx_package] - ) - if vex_patches[cve]: - pkg_objset.new_scoped_relationship( - spdx_vex, - oe.spdx30.RelationshipType.patchedBy, - oe.spdx30.LifecycleScopeType.build, - [ - oe.sbom30.get_element_link_id(p) - for p in vex_patches[cve] - ], - ) - - elif vex.relationshipType == oe.spdx30.RelationshipType.affects: - pkg_objset.new_vex_unpatched_relationship( - [oe.sbom30.get_element_link_id(cve)], [spdx_package] - ) - elif ( - vex.relationshipType == oe.spdx30.RelationshipType.doesNotAffect - ): - spdx_vex = pkg_objset.new_vex_ignored_relationship( - [oe.sbom30.get_element_link_id(cve)], - [spdx_package], - impact_statement=vex.security_impactStatement, - ) - - if vex.security_justificationType: - for v in spdx_vex: - v.security_justificationType = ( - vex.security_justificationType - ) - bb.debug(1, "Adding package files to SPDX for package %s" % pkg_name) package_files = add_package_files( d, -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 08/12] spdx: Remove fatal errors for missing providers 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (6 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 07/12] spdx30: Remove package VEX Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 09/12] spdx3: Use common variable for vardeps Joshua Watt ` (3 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt When creating images and SDKs, do not error on missing providers. This allows recipes to use the `nospdx` inherit to prevent SPDX from being generated, but not result in an error when assembling the final image. Note that runtime packages generation already ignored missing providers, so this is changing image and SDK generation to match Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-2.2.bbclass | 3 ++- meta/lib/oe/spdx30_tasks.py | 8 +------- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index 3288cdf75a..aa39208eae 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -858,7 +858,8 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx if packages: for name in sorted(packages.keys()): if name not in providers: - bb.fatal("Unable to find SPDX provider for '%s'" % name) + bb.note("Unable to find SPDX provider for '%s'" % name) + continue pkg_name, pkg_hashfn = providers[name] diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 5b651900c4..c4af191974 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1178,11 +1178,10 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) providers = oe.spdx_common.collect_package_providers(d) build_deps = set() - missing_providers = set() for name in sorted(packages.keys()): if name not in providers: - missing_providers.add(name) + bb.note(f"Unable to find SPDX provider for '{name}'") continue pkg_name, pkg_hashfn = providers[name] @@ -1201,11 +1200,6 @@ def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None) for h, f in pkg_objset.by_sha256_hash.items(): files_by_hash.setdefault(h, set()).update(f) - if missing_providers: - bb.fatal( - f"Unable to find SPDX provider(s) for: {', '.join(sorted(missing_providers))}" - ) - if build_deps: objset.new_scoped_relationship( [build], -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 09/12] spdx3: Use common variable for vardeps 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (7 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 08/12] spdx: Remove fatal errors for missing providers Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 10/12] glibc-testsuite: Do not generate SPDX Joshua Watt ` (2 subsequent siblings) 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Instead of repeating the vardeps for each SPDX task with the necessary variables, use a common variable to make it easier to manage Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/create-spdx-3.0.bbclass | 33 ++++++++++------------------ 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index c3ea95b8bc..869bbd472f 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -163,6 +163,14 @@ SPDX3_DEP_FILES = "\ ${SPDX_LICENSES}:True \ " +SPDX3_VAR_DEPS = "\ + SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ + SPDX_PACKAGE_ADDITIONAL_PURPOSE \ + SPDX_PROFILES \ + SPDX_NAMESPACE_PREFIX \ + SPDX_UUID_NAMESPACE \ + " + python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) @@ -174,13 +182,7 @@ do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" -do_create_recipe_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { sstate_setscene(d) @@ -211,13 +213,7 @@ do_create_spdx[depends] += " \ ${PATCHDEPENDENCY} \ ${@create_spdx_source_deps(d)} \ " -do_create_spdx[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_spdx_setscene () { sstate_setscene(d) @@ -238,6 +234,7 @@ do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[rdeptask] = "do_create_spdx" +do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_package_spdx_setscene () { sstate_setscene(d) @@ -259,13 +256,7 @@ do_create_recipe_sbom[sstate-inputdirs] = "${SPDXRECIPESBOMDEPLOY}" do_create_recipe_sbom[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}" do_create_recipe_sbom[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_sbom[cleandirs] = "${SPDXRECIPESBOMDEPLOY}" -do_create_recipe_sbom[vardeps] += "\ - SPDX_INCLUDE_BITBAKE_PARENT_BUILD \ - SPDX_PACKAGE_ADDITIONAL_PURPOSE \ - SPDX_PROFILES \ - SPDX_NAMESPACE_PREFIX \ - SPDX_UUID_NAMESPACE \ - " +do_create_recipe_sbom[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_sbom_setscene () { sstate_setscene(d) -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 10/12] glibc-testsuite: Do not generate SPDX 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (8 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 09/12] spdx3: Use common variable for vardeps Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 11/12] spdx: Remove do_collect_spdx_deps task Joshua Watt 2026-03-18 13:49 ` [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information Joshua Watt 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt glibc-testsuite does not run on target or factor into the build supply chain, since its purpose is run tests in Qemu at build time Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/recipes-core/glibc/glibc-testsuite_2.43.bb | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/recipes-core/glibc/glibc-testsuite_2.43.bb b/meta/recipes-core/glibc/glibc-testsuite_2.43.bb index 28af6961c3..899955adfb 100644 --- a/meta/recipes-core/glibc/glibc-testsuite_2.43.bb +++ b/meta/recipes-core/glibc/glibc-testsuite_2.43.bb @@ -64,6 +64,7 @@ do_check:append () { } inherit nopackages +inherit nospdx deltask do_stash_locale deltask do_install deltask do_populate_sysroot -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* [OE-core][PATCH v7 11/12] spdx: Remove do_collect_spdx_deps task 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (9 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 10/12] glibc-testsuite: Do not generate SPDX Joshua Watt @ 2026-03-18 13:44 ` Joshua Watt 2026-03-18 13:49 ` [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information Joshua Watt 11 siblings, 0 replies; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:44 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Removes the do_collect_spdx_deps task. This task was added a long time ago, and appears to have been added due to a misunderstanding about how the task graph works. It is not necessary since tasks can directly call collect_direct_deps() with the appropriate task that they depend on to get their dependencies. This should fix several classes of SPDX bug where documents could not be found because the wrong deps were being looked for due to which tasks were re-run Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes-recipe/nospdx.bbclass | 1 - meta/classes/create-spdx-2.2.bbclass | 22 +++++---- meta/classes/create-spdx-3.0.bbclass | 5 +- meta/classes/spdx-common.bbclass | 22 --------- meta/lib/oe/spdx30_tasks.py | 22 +++++---- meta/lib/oe/spdx_common.py | 69 +++++++++++++--------------- 6 files changed, 62 insertions(+), 79 deletions(-) diff --git a/meta/classes-recipe/nospdx.bbclass b/meta/classes-recipe/nospdx.bbclass index 90e14442ba..7c99fcd1ec 100644 --- a/meta/classes-recipe/nospdx.bbclass +++ b/meta/classes-recipe/nospdx.bbclass @@ -4,7 +4,6 @@ # SPDX-License-Identifier: MIT # -deltask do_collect_spdx_deps deltask do_create_recipe_spdx deltask do_create_spdx deltask do_create_spdx_runtime diff --git a/meta/classes/create-spdx-2.2.bbclass b/meta/classes/create-spdx-2.2.bbclass index aa39208eae..1c43156559 100644 --- a/meta/classes/create-spdx-2.2.bbclass +++ b/meta/classes/create-spdx-2.2.bbclass @@ -277,7 +277,7 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR" -def collect_dep_recipes(d, doc, spdx_recipe): +def collect_dep_recipes(d, doc, spdx_recipe, direct_deps): import json from pathlib import Path import oe.sbom @@ -290,9 +290,7 @@ def collect_dep_recipes(d, doc, spdx_recipe): dep_recipes = [] - deps = oe.spdx_common.get_spdx_deps(d) - - for dep in deps: + for dep in direct_deps: # If this dependency is not calculated in the taskhash skip it. # Otherwise, it can result in broken links since this task won't # rebuild and see the new SPDX ID if the dependency changes @@ -405,7 +403,7 @@ do_create_recipe_spdx() { : } do_create_recipe_spdx[noexec] = "1" -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx python do_create_spdx() { @@ -532,7 +530,8 @@ python do_create_spdx() { if archive is not None: recipe.packageFileName = str(recipe_archive.name) - dep_recipes = collect_dep_recipes(d, doc, recipe) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_recipes = collect_dep_recipes(d, doc, recipe, direct_deps) doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d)) dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe)) @@ -603,7 +602,7 @@ python do_create_spdx() { } do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS" # NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source -addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch do_collect_spdx_deps before do_populate_sdk do_build do_rm_work +addtask do_create_spdx after do_create_recipe_spdx do_package do_packagedata do_unpack do_patch before do_populate_sdk do_build do_rm_work SSTATETASKS += "do_create_spdx" do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}" @@ -638,7 +637,9 @@ python do_create_runtime_spdx() { license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() @@ -760,6 +761,7 @@ addtask do_create_runtime_spdx_setscene do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_runtime_spdx[deptask] = "do_create_spdx" do_create_runtime_spdx[rdeptask] = "do_create_spdx" do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx" @@ -829,7 +831,9 @@ def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx license_data = oe.spdx_common.load_spdx_license_data(d) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) package_archs = d.getVar("SPDX_MULTILIB_SSTATE_ARCHS").split() package_archs.reverse() diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 869bbd472f..c3deb22598 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -175,13 +175,14 @@ python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) } -addtask do_create_recipe_spdx after do_collect_spdx_deps +addtask do_create_recipe_spdx SSTATETASKS += "do_create_recipe_spdx" do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}" do_create_recipe_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_recipe_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_recipe_spdx[cleandirs] = "${SPDXRECIPEDEPLOY}" +do_create_recipe_spdx[deptask] += "do_create_recipe_spdx" do_create_recipe_spdx[vardeps] += "${SPDX3_VAR_DEPS}" python do_create_recipe_spdx_setscene () { @@ -197,7 +198,6 @@ addtask do_create_spdx after \ do_unpack \ do_patch \ do_create_recipe_spdx \ - do_collect_spdx_deps \ do_deploy_source_date_epoch \ do_populate_sysroot do_package do_packagedata \ before do_populate_sdk do_populate_sdk_ext do_build do_rm_work @@ -233,6 +233,7 @@ do_create_package_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}" do_create_package_spdx[file-checksums] += "${SPDX3_DEP_FILES}" do_create_package_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}" do_create_package_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}" +do_create_package_spdx[deptask] = "do_create_spdx" do_create_package_spdx[rdeptask] = "do_create_spdx" do_create_package_spdx[vardeps] += "${SPDX3_VAR_DEPS}" diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index abf2332bee..4b40cbf75c 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -87,28 +87,6 @@ def create_spdx_source_deps(d): return " ".join(deps) -python do_collect_spdx_deps() { - # This task calculates the build time dependencies of the recipe, and is - # required because while a task can deptask on itself, those dependencies - # do not show up in BB_TASKDEPDATA. To work around that, this task does the - # deptask on do_create_recipe_spdx and writes out the dependencies it finds, then - # downstream tasks read in the found dependencies when writing the actual - # SPDX document - import json - import oe.spdx_common - from pathlib import Path - - spdx_deps_file = Path(d.getVar("SPDXDEPS")) - - deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") - - with spdx_deps_file.open("w") as f: - json.dump(deps, f) -} -addtask do_collect_spdx_deps -do_collect_spdx_deps[deptask] = "do_create_recipe_spdx" -do_collect_spdx_deps[dirs] = "${SPDXDIR}" - oe.spdx_common.collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA" oe.spdx_common.collect_direct_deps[vardeps] += "DEPENDS" oe.spdx_common.collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA" diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index c4af191974..353d783fa2 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -298,13 +298,11 @@ def get_package_sources_from_debug( return dep_source_files -def collect_dep_objsets(d, subdir, fn_prefix, obj_type, **attr_filter): - deps = oe.spdx_common.get_spdx_deps(d) - +def collect_dep_objsets(d, direct_deps, subdir, fn_prefix, obj_type, **attr_filter): dep_objsets = [] dep_objs = set() - for dep in deps: + for dep in direct_deps: bb.debug(1, "Fetching SPDX for dependency %s" % (dep.pn)) dep_obj, dep_objset = oe.sbom30.find_root_obj_in_jsonld( d, subdir, fn_prefix + dep.pn, obj_type, **attr_filter @@ -551,8 +549,10 @@ def create_recipe_spdx(d): ) ) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_recipe_spdx") + dep_objsets, dep_recipes = collect_dep_objsets( - d, "static", "static-", oe.spdx30.software_Package + d, direct_deps, "static", "static-", oe.spdx30.software_Package ) if dep_recipes: @@ -753,8 +753,10 @@ def create_spdx(d): build_inputs |= files index_sources_by_hash(files, dep_sources) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + dep_objsets, dep_builds = collect_dep_objsets( - d, "builds", "build-", oe.spdx30.build_Build + d, direct_deps, "builds", "build-", oe.spdx30.build_Build ) if dep_builds: @@ -998,7 +1000,9 @@ def create_package_spdx(d): deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) deploydir = Path(d.getVar("SPDXRUNTIMEDEPLOY")) - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) pkg_arch = d.getVar("SSTATE_PKGARCH") if get_is_native(d): @@ -1175,7 +1179,9 @@ def write_bitbake_spdx(d): def collect_build_package_inputs(d, objset, build, packages, files_by_hash=None): import oe.sbom30 - providers = oe.spdx_common.collect_package_providers(d) + direct_deps = oe.spdx_common.collect_direct_deps(d, "do_create_spdx") + + providers = oe.spdx_common.collect_package_providers(d, direct_deps) build_deps = set() diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py index 3aaf2a9c8b..c0ef11f199 100644 --- a/meta/lib/oe/spdx_common.py +++ b/meta/lib/oe/spdx_common.py @@ -38,7 +38,7 @@ def extract_licenses(filename): def is_work_shared_spdx(d): - return '/work-shared/' in d.getVar('S') + return "/work-shared/" in d.getVar("S") def load_spdx_license_data(d): @@ -77,12 +77,15 @@ def process_sources(d): return True -@dataclass(frozen=True) +@dataclass(frozen=True, eq=True, order=True) class Dep(object): pn: str hashfn: str in_taskhash: bool + def to_tuple(self): + return (self.pn, self.hashfn, self.in_taskhash) + def collect_direct_deps(d, dep_task): """ @@ -105,7 +108,9 @@ def collect_direct_deps(d, dep_task): ) if not dep_task in depflags: - bb.fatal(f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}") + bb.fatal( + f"Task {dep_task} was not found in any dependency flag of {pn}:{current_task}" + ) for this_dep in taskdepdata.values(): if this_dep[0] == pn and this_dep[1] == current_task: @@ -118,25 +123,14 @@ def collect_direct_deps(d, dep_task): for dep_name in this_dep.deps: dep_data = taskdepdata[dep_name] if dep_data.taskname == dep_task and dep_data.pn != pn: - deps.add((dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps)) + deps.add( + Dep(dep_data.pn, dep_data.hashfn, dep_name in this_dep.taskhash_deps) + ) return sorted(deps) -def get_spdx_deps(d): - """ - Reads the SPDX dependencies JSON file and returns the data - """ - spdx_deps_file = Path(d.getVar("SPDXDEPS")) - - deps = [] - with spdx_deps_file.open("r") as f: - for d in json.load(f): - deps.append(Dep(*d)) - return deps - - -def collect_package_providers(d): +def collect_package_providers(d, direct_deps): """ Returns a dictionary where each RPROVIDES is mapped to the package that provides it @@ -145,16 +139,15 @@ def collect_package_providers(d): providers = {} - deps = collect_direct_deps(d, "do_create_spdx") - deps.append((d.getVar("PN"), d.getVar("BB_HASHFILENAME"), True)) + all_deps = direct_deps + [Dep(d.getVar("PN"), d.getVar("BB_HASHFILENAME"), True)] - for dep_pn, dep_hashfn, _ in deps: + for dep in all_deps: localdata = d - recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata) + recipe_data = oe.packagedata.read_pkgdata(dep.pn, localdata) if not recipe_data: localdata = bb.data.createCopy(d) localdata.setVar("PKGDATA_DIR", "${PKGDATA_DIR_SDK}") - recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata) + recipe_data = oe.packagedata.read_pkgdata(dep.pn, localdata) for pkg in recipe_data.get("PACKAGES", "").split(): pkg_data = oe.packagedata.read_subpkgdata_dict(pkg, localdata) @@ -171,7 +164,7 @@ def collect_package_providers(d): rprovides.add(pkg) for r in rprovides: - providers[r] = (pkg, dep_hashfn) + providers[r] = (pkg, dep.hashfn) return providers @@ -202,25 +195,21 @@ def get_patched_src(d): bb.build.exec_func("do_unpack", d) if d.getVar("SRC_URI") != "": - if bb.data.inherits_class('dos2unix', d): - bb.build.exec_func('do_convert_crlf_to_lf', d) + if bb.data.inherits_class("dos2unix", d): + bb.build.exec_func("do_convert_crlf_to_lf", d) bb.build.exec_func("do_patch", d) # Copy source from work-share to spdx_workdir if is_work_shared_spdx(d): - share_src = d.getVar('S') + share_src = d.getVar("S") d.setVar("WORKDIR", spdx_workdir) d.setVar("STAGING_DIR_NATIVE", spdx_sysroot_native) # Copy source to ${SPDXWORK}, same basename dir of ${S}; - src_dir = ( - spdx_workdir - + "/" - + os.path.basename(share_src) - ) + src_dir = spdx_workdir + "/" + os.path.basename(share_src) # For kernel souce, rename suffix dir 'kernel-source' # to ${BP} (${BPN}-${PV}) if bb.data.inherits_class("kernel", d): - src_dir = spdx_workdir + "/" + d.getVar('BP') + src_dir = spdx_workdir + "/" + d.getVar("BP") bb.note(f"copyhardlinktree {share_src} to {src_dir}") oe.path.copyhardlinktree(share_src, src_dir) @@ -233,7 +222,9 @@ def get_patched_src(d): def has_task(d, task): - return bool(d.getVarFlag(task, "task", False)) and not bool(d.getVarFlag(task, "noexec", False)) + return bool(d.getVarFlag(task, "task", False)) and not bool( + d.getVarFlag(task, "noexec", False) + ) def fetch_data_to_uri(fd, name): @@ -243,8 +234,8 @@ def fetch_data_to_uri(fd, name): uri = fd.type # crate: is not a valid URL. Use url field instead if exist - if uri == "crate" and hasattr(fd,"url"): - return fd.url + if uri == "crate" and hasattr(fd, "url"): + return fd.url # Map gitsm to git, since gitsm:// is not a valid URI protocol if uri == "gitsm": @@ -259,11 +250,13 @@ def fetch_data_to_uri(fd, name): return uri -def is_compiled_source (filename, compiled_sources, types): + +def is_compiled_source(filename, compiled_sources, types): """ Check if the file is a compiled file """ import os + # If we don't have compiled source, we assume all are compiled. if not compiled_sources: return True @@ -278,11 +271,13 @@ def is_compiled_source (filename, compiled_sources, types): # Check that the file is in the list return filename in compiled_sources + def get_compiled_sources(d): """ Get list of compiled sources from debug information and normalize the paths """ import itertools + source_info = oe.package.read_debugsources_info(d) if not source_info: bb.debug(1, "Do not have debugsources.list. Skipping") -- 2.53.0 ^ permalink raw reply related [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt ` (10 preceding siblings ...) 2026-03-18 13:44 ` [OE-core][PATCH v7 11/12] spdx: Remove do_collect_spdx_deps task Joshua Watt @ 2026-03-18 13:49 ` Joshua Watt 2026-03-19 7:07 ` Mathieu Dubois-Briand 11 siblings, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-18 13:49 UTC (permalink / raw) To: openembedded-core, Mathieu Dubois-Briand, Stefano Tondo Mathieu, Here is my latest SPDX patch set (finally!). I'm sorry for the delay and the mix up with Stefano's patches. I believe that you can now stack Stefanos patches on top of mine and everything should be OK Thank you, Joshua On Wed, Mar 18, 2026 at 7:46 AM Joshua Watt <jpewhacker@gmail.com> wrote: > > Changes the SPDX 3 output to include a "recipe" package that describe > static information available at parse time (without building). This is > primarily useful for gathering SPDX 3 VEX information about some or all > recipes, enabling SPDX 3 to be used in place of cve_check.bbclass and > vex.bbclass. > > Special thanks to Benjamin Robin <benjamin.robin@bootlin.com> for > helping work through this. > > V2: Fixes a bug where do_populate_sysroot was running when it should not > be. Drops the patch to ignore ASSUME_PROVIDES recipes, since this is > incorrect (this is already handled by bitbake in the taskgraph, and > doesn't need to be manually removed). > > V3: Fixes a bug where meta-world-recipe-sbom was reporting a circular > dependency. meta-world-recipe-sbom also no longer runs in world builds, > as there's no reason to this. Finally, fixes a bug where > NO_GENERIC_LICENSE files would fail to be found in do_create_spdx > (because do_unpack was not run). > > V4: Fixes test cases. Adds SPDX_PACKAGE_INCLUDE_VEX to control if VEX > information is linked to binary packages, or just recipes. Defaults to > "0" to significantly reduce the size of the SPDX output. > > V5: Fixes dummy-sdk-packages to not generate SPDX output, since it > does funny things with its arch which prevents it from rebuilding SPDX > data properly, and no SPDX data is needed for it anyway > > V6: Fixes a bug where SPDX task would not correctly re-run when they > change, which would cause errors about missing SPDX document. Also > updates to the latest version of the SPDX bindings which improves > performance > > V7: Makes meta-world-recipe-sbom create the world SBoM when run with a > task (e.g. as part of do_build). Drops the removal of SPDX from > dummy-sdk-packages as these have been fixed to work properly. > > Joshua Watt (12): > spdx3: Add recipe SPDX data > spdx3: Add recipe SBoM task > spdx3: Add is-native property > spdx30: Include patch file information in VEX > spdx: De-duplicate CreationInfo > spdx_common: Check for dependent task in task flags > spdx30: Remove package VEX > spdx: Remove fatal errors for missing providers > spdx3: Use common variable for vardeps > glibc-testsuite: Do not generate SPDX > spdx: Remove do_collect_spdx_deps task > spdx: Update to latest bindings > > meta/classes-global/sstate.bbclass | 4 +- > .../create-spdx-image-3.0.bbclass | 4 +- > .../create-spdx-sdk-3.0.bbclass | 4 +- > meta/classes-recipe/kernel.bbclass | 2 +- > meta/classes-recipe/nospdx.bbclass | 2 +- > meta/classes/create-spdx-2.2.bbclass | 33 +- > meta/classes/create-spdx-3.0.bbclass | 81 +- > meta/classes/spdx-common.bbclass | 34 +- > meta/conf/distro/include/maintainers.inc | 1 + > meta/lib/oe/sbom30.py | 239 +- > meta/lib/oe/spdx30/__init__.py | 8 + > meta/lib/oe/spdx30/__main__.py | 12 + > meta/lib/oe/spdx30/cmd.py | 75 + > meta/lib/oe/{spdx30.py => spdx30/model.py} | 5935 ++++++++++------- > meta/lib/oe/spdx30/stub.pyi | 2544 +++++++ > meta/lib/oe/spdx30_tasks.py | 459 +- > meta/lib/oe/spdx_common.py | 78 +- > meta/lib/oeqa/selftest/cases/spdx.py | 28 +- > .../glibc/glibc-testsuite_2.43.bb | 1 + > .../meta/meta-world-recipe-sbom.bb | 38 + > scripts/contrib/make-spdx-bindings.sh | 3 +- > 21 files changed, 6837 insertions(+), 2748 deletions(-) > create mode 100644 meta/lib/oe/spdx30/__init__.py > create mode 100644 meta/lib/oe/spdx30/__main__.py > create mode 100644 meta/lib/oe/spdx30/cmd.py > rename meta/lib/oe/{spdx30.py => spdx30/model.py} (52%) > create mode 100644 meta/lib/oe/spdx30/stub.pyi > create mode 100644 meta/recipes-core/meta/meta-world-recipe-sbom.bb > > -- > 2.53.0 > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information 2026-03-18 13:49 ` [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information Joshua Watt @ 2026-03-19 7:07 ` Mathieu Dubois-Briand 2026-03-19 12:02 ` Mathieu Dubois-Briand 2026-03-19 21:55 ` Joshua Watt 0 siblings, 2 replies; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-03-19 7:07 UTC (permalink / raw) To: Joshua Watt, openembedded-core, Stefano Tondo On Wed Mar 18, 2026 at 2:49 PM CET, Joshua Watt wrote: > Mathieu, > > Here is my latest SPDX patch set (finally!). I'm sorry for the delay > and the mix up with Stefano's patches. I believe that you can now > stack Stefanos patches on top of mine and everything should be OK > > Thank you, > Joshua > Thanks! I did apply this series, but got some conflicts. I did try to solve them, but I might have solved them wrong. For reference, the result is here: https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456 So with this, I've got a few issues. First an error appearing in various builds: ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405 https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 And still some selftest failures: 2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL ... 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL ... 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568 I will probably launch a build with this series alone a bit later today, to distinguish what comes from this series and what comes from the combination. Thank, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information 2026-03-19 7:07 ` Mathieu Dubois-Briand @ 2026-03-19 12:02 ` Mathieu Dubois-Briand 2026-03-19 21:55 ` Joshua Watt 1 sibling, 0 replies; 113+ messages in thread From: Mathieu Dubois-Briand @ 2026-03-19 12:02 UTC (permalink / raw) To: Joshua Watt, openembedded-core, Stefano Tondo On Thu Mar 19, 2026 at 8:07 AM CET, Mathieu Dubois-Briand wrote: > On Wed Mar 18, 2026 at 2:49 PM CET, Joshua Watt wrote: >> Mathieu, >> >> Here is my latest SPDX patch set (finally!). I'm sorry for the delay >> and the mix up with Stefano's patches. I believe that you can now >> stack Stefanos patches on top of mine and everything should be OK >> >> Thank you, >> Joshua >> > > Thanks! > > I did apply this series, but got some conflicts. I did try to solve > them, but I might have solved them wrong. > > For reference, the result is here: > > https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456 > > So with this, I've got a few issues. First an error appearing in various > builds: > > ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > > And still some selftest failures: > > 2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) > 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL > ... > 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) > 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL > ... > 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) > 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568 > > I will probably launch a build with this series alone a bit later today, > to distinguish what comes from this series and what comes from the > combination. > > Thank, > Mathieu So I did launch some build with this series applied on top of master, and we basically have the same errors: https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3410 https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3389 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3467 The last one is still building right now, but we do see some errors already. Thanks, Mathieu -- Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information 2026-03-19 7:07 ` Mathieu Dubois-Briand 2026-03-19 12:02 ` Mathieu Dubois-Briand @ 2026-03-19 21:55 ` Joshua Watt 2026-03-19 22:14 ` Richard Purdie 1 sibling, 1 reply; 113+ messages in thread From: Joshua Watt @ 2026-03-19 21:55 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core Richard, These errors are caused because those recipes do: PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}" Which is not in SSTATE_ARCHS, and therefore the required SPDX files cannot be found. It appears this was done circa 2016: 1dbc6ec4ca ("buildtools/uninative-tarball: Fix deployment overlap issues") , but I'm not sure it was correct before that. IDK what the fix should be; Maybe SSTATE_PKGARCH should always be part of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be manually added? On Thu, Mar 19, 2026 at 1:07 AM Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com> wrote: > > On Wed Mar 18, 2026 at 2:49 PM CET, Joshua Watt wrote: > > Mathieu, > > > > Here is my latest SPDX patch set (finally!). I'm sorry for the delay > > and the mix up with Stefano's patches. I believe that you can now > > stack Stefanos patches on top of mine and everything should be OK > > > > Thank you, > > Joshua > > > > Thanks! > > I did apply this series, but got some conflicts. I did try to solve > them, but I might have solved them wrong. > > For reference, the result is here: > > https://git.yoctoproject.org/poky-ci-archive/log/?h=oecore/autobuilder.yoctoproject.org/valkyrie/a-full-3456 > > So with this, I've got a few issues. First an error appearing in various > builds: > > ERROR: buildtools-tarball-1.0-r0 do_create_spdx: Could not find a static SPDX document named static-buildtools-tarball > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/36/builds/3405 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/2/builds/3422 > > And still some selftest failures: > > 2026-03-18 21:56:34,082 - oe-selftest - INFO - runtime_test.TestExport.test_testexport_sdk (subunit.RemotedTestCase) > 2026-03-18 21:56:34,083 - oe-selftest - INFO - ... FAIL > ... > 2026-03-18 22:25:15,422 - oe-selftest - INFO - spdx.SPDX30Check.test_download_location_defensive_handling (subunit.RemotedTestCase) > 2026-03-18 22:25:15,422 - oe-selftest - INFO - ... FAIL > ... > 2026-03-18 23:06:54,297 - oe-selftest - INFO - spdx.SPDX30Check.test_version_extraction_patterns (subunit.RemotedTestCase) > 2026-03-18 23:06:54,297 - oe-selftest - INFO - ... FAIL > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3462 > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3568 > > I will probably launch a build with this series alone a bit later today, > to distinguish what comes from this series and what comes from the > combination. > > Thank, > Mathieu > > -- > Mathieu Dubois-Briand, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com > ^ permalink raw reply [flat|nested] 113+ messages in thread
* Re: [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information 2026-03-19 21:55 ` Joshua Watt @ 2026-03-19 22:14 ` Richard Purdie 0 siblings, 0 replies; 113+ messages in thread From: Richard Purdie @ 2026-03-19 22:14 UTC (permalink / raw) To: Joshua Watt; +Cc: openembedded-core Hi, On Thu, 2026-03-19 at 15:55 -0600, Joshua Watt wrote: > These errors are caused because those recipes do: > > PACKAGE_ARCH = "${SDK_ARCH}_${SDK_OS}" > > Which is not in SSTATE_ARCHS, and therefore the required SPDX files > cannot be found. It appears this was done circa 2016: 1dbc6ec4ca > ("buildtools/uninative-tarball: Fix deployment overlap issues") , but > I'm not sure it was correct before that. > > IDK what the fix should be; Maybe SSTATE_PKGARCH should always be > part of SSTATE_ARCHS? Or maybe ${SDK_ARCH}_${SDK_OS} needs to be > manually added? Sorry, this was my fault then, broken by https://git.openembedded.org/openembedded-core/commit/?id=834efe5eeaa7edae27d54a382ab864ef8f924b2d I think they need changing to ${SDK_ARCH}-${SDKPKGSUFFIX}. I'll try a patch. Cheers, Richard ^ permalink raw reply [flat|nested] 113+ messages in thread
end of thread, other threads:[~2026-03-19 22:14 UTC | newest] Thread overview: 113+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-02-20 15:40 [OE-core][PATCH 0/9] Add SPDX 3 Recipe Information Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 2/9] gcc-source: " Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 3/9] spdx3: Add recipe SPDX data Joshua Watt 2026-02-22 7:59 ` Mathieu Dubois-Briand 2026-02-20 15:40 ` [OE-core][PATCH 4/9] spdx3: Add recipe SBoM task Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 5/9] spdx3: Add is-native property Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 6/9] spdx30: Include patch file information in VEX Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 7/9] spdx: De-duplicate CreationInfo Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 8/9] spdx: Ignore ASSUME_PROVIDED recipes Joshua Watt 2026-02-20 15:40 ` [OE-core][PATCH 9/9] spdx_common: Check for dependent task in task flags Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 2/8] gcc-source: " Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 3/8] spdx3: Add recipe SPDX data Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 4/8] spdx3: Add recipe SBoM task Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 5/8] spdx3: Add is-native property Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 6/8] spdx30: Include patch file information in VEX Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 7/8] spdx: De-duplicate CreationInfo Joshua Watt 2026-02-24 23:00 ` [OE-core][PATCH v2 8/8] spdx_common: Check for dependent task in task flags Joshua Watt 2026-02-26 12:52 ` [OE-core][PATCH v2 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-02-26 14:27 ` Benjamin Robin 2026-02-26 15:09 ` Benjamin Robin 2026-02-26 15:41 ` Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 " Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 1/8] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 2/8] gcc-source: " Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 3/8] spdx3: Add recipe SPDX data Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 4/8] spdx3: Add recipe SBoM task Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 5/8] spdx3: Add is-native property Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 6/8] spdx30: Include patch file information in VEX Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 7/8] spdx: De-duplicate CreationInfo Joshua Watt 2026-02-26 17:33 ` [OE-core][PATCH v3 8/8] spdx_common: Check for dependent task in task flags Joshua Watt 2026-02-27 7:32 ` [OE-core][PATCH v3 0/8] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-03-03 0:43 ` [OE-core][PATCH v4 0/9] " Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 1/9] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 2/9] gcc-source: " Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 3/9] spdx3: Add recipe SPDX data Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 4/9] spdx3: Add recipe SBoM task Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 5/9] spdx3: Add is-native property Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 6/9] spdx30: Include patch file information in VEX Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 7/9] spdx: De-duplicate CreationInfo Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 8/9] spdx_common: Check for dependent task in task flags Joshua Watt 2026-03-03 0:43 ` [OE-core][PATCH v4 9/9] spdx30: Skip install package CVE information Joshua Watt 2026-03-03 10:17 ` [OE-core][PATCH v4 0/9] Add SPDX 3 Recipe Information Antonin Godard 2026-03-03 14:08 ` Mathieu Dubois-Briand 2026-03-04 16:44 ` [OE-core][PATCH v5 00/13] " Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 01/13] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 02/13] gcc-source: " Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 03/13] spdx3: Add recipe SPDX data Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 04/13] spdx3: Add recipe SBoM task Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 05/13] spdx3: Add is-native property Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 06/13] spdx30: Include patch file information in VEX Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 07/13] spdx: De-duplicate CreationInfo Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 08/13] spdx_common: Check for dependent task in task flags Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 09/13] spdx30: Skip install package CVE information Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 10/13] dummy-sdk-package: Disable SPDX Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 11/13] spdx: Remove fatal errors for missing providers Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 12/13] spdx3: Use common variable for vardeps Joshua Watt 2026-03-04 16:44 ` [OE-core][PATCH v5 13/13] glibc-testsuite: Do not generate SPDX Joshua Watt 2026-03-05 19:59 ` [OE-core][PATCH v5 00/13] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-03-10 18:38 ` [OE-core][PATCH v6 00/15] " Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 01/15] llvm-project-source: Use allarch.bbclass Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 02/15] gcc-source: " Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 03/15] spdx3: Add recipe SPDX data Joshua Watt 2026-03-12 11:43 ` Richard Purdie 2026-03-12 14:11 ` Joshua Watt 2026-03-12 17:50 ` Richard Purdie 2026-03-10 18:38 ` [OE-core][PATCH v6 04/15] spdx3: Add recipe SBoM task Joshua Watt 2026-03-12 11:50 ` Richard Purdie 2026-03-12 14:12 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 05/15] spdx3: Add is-native property Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 06/15] spdx30: Include patch file information in VEX Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 07/15] spdx: De-duplicate CreationInfo Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 08/15] spdx_common: Check for dependent task in task flags Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 09/15] spdx30: Skip install package CVE information Joshua Watt 2026-03-12 11:55 ` Richard Purdie 2026-03-12 14:15 ` Joshua Watt 2026-03-12 15:52 ` Richard Purdie 2026-03-12 16:11 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 10/15] dummy-sdk-package: Disable SPDX Joshua Watt 2026-03-12 11:59 ` Richard Purdie 2026-03-12 14:24 ` Joshua Watt 2026-03-12 15:58 ` Richard Purdie 2026-03-12 16:06 ` Joshua Watt 2026-03-12 16:43 ` Joshua Watt 2026-03-12 18:02 ` Joshua Watt 2026-03-12 20:34 ` Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 11/15] spdx: Remove fatal errors for missing providers Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 12/15] spdx3: Use common variable for vardeps Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 13/15] glibc-testsuite: Do not generate SPDX Joshua Watt 2026-03-10 18:38 ` [OE-core][PATCH v6 14/15] spdx: Remove do_collect_spdx_deps task Joshua Watt 2026-03-11 13:55 ` [OE-core][PATCH v6 00/15] Add SPDX 3 Recipe Information Mathieu Dubois-Briand 2026-03-11 16:39 ` Joshua Watt 2026-03-11 19:33 ` Mathieu Dubois-Briand 2026-03-11 22:56 ` Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 00/12] " Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 01/12] spdx3: Add recipe SPDX data Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 02/12] spdx3: Add recipe SBoM task Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 03/12] spdx3: Add is-native property Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 04/12] spdx30: Include patch file information in VEX Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 05/12] spdx: De-duplicate CreationInfo Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 06/12] spdx_common: Check for dependent task in task flags Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 07/12] spdx30: Remove package VEX Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 08/12] spdx: Remove fatal errors for missing providers Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 09/12] spdx3: Use common variable for vardeps Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 10/12] glibc-testsuite: Do not generate SPDX Joshua Watt 2026-03-18 13:44 ` [OE-core][PATCH v7 11/12] spdx: Remove do_collect_spdx_deps task Joshua Watt 2026-03-18 13:49 ` [OE-core][PATCH v7 00/12] Add SPDX 3 Recipe Information Joshua Watt 2026-03-19 7:07 ` Mathieu Dubois-Briand 2026-03-19 12:02 ` Mathieu Dubois-Briand 2026-03-19 21:55 ` Joshua Watt 2026-03-19 22:14 ` Richard Purdie
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox