* [PATCH v2 0/4] spdx30: identity and purls of dependency downloads
@ 2026-11-18 23:44 Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 1/4] spdx30: Take name, version and PURL of downloads from the fetcher Stefano Tondo
` (4 more replies)
0 siblings, 5 replies; 9+ messages in thread
From: Stefano Tondo @ 2026-11-18 23:44 UTC (permalink / raw)
To: openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Mathieu Dubois-Briand, stefano.tondo.ext
Hi,
This series implements the parts of the RFC "spdx30: name, version
and purl for dependency downloads" [1] that we agreed on. It needs
the bitbake series "fetch: package identity of crate, gomod and npm
URLs" [2].
1. The SPDX of a crate, gomod or npm download gets the name, the
version and the purl from the fetcher.
2. For Cargo recipes, a new do_cargo_metadata task writes the crate
dependency graph for the target. The build links to each crate
download with a runtime, build or test scope. A crate that is not
used on the target gets an annotation.
3. Each output package gets the purls of its source downloads as
external identifiers, as Joshua proposed: a statically linked
dependency is part of the binary. A purl is left out only when the
cargo graph proves that the crate is not in the target. The
software_packageUrl of the package does not change.
4. Selftests with python3-bcrypt.
For python3-bcrypt, the output package gets the purls of 25 of its
42 crates. The other 17 crates are not built into the target.
Two points for review:
- All output packages of a recipe get the purls, also -dbg and -src.
I have no proof per package, so I followed the rule "keep the purl
unless there is proof".
- Go recipes have no filter yet. go version -m can give the proof per
binary, and I can add it in a follow-up.
Changes in v2:
- Patch 2/4: the walk over the cargo graph expands each workspace
member only once. In v1 it did not end for python3-uv-build, where
the member uv-preview is a dev dependency of itself. do_create_spdx
used memory until the OOM killer stopped it. Thanks to Mathieu for
the report [3].
- Patch 2/4: add unit tests for the scopes, with member cycles.
- Rebased on master.
Tests: oe-selftest -r oelib.cargo_scopes spdx.SPDX30Check passes on
master (22 tests) and on master-next (22 tests). With v2,
do_create_spdx of python3-uv-build and python3-uv-build-native takes
less than 2 seconds and about 100 MB, with uv-build 0.13.7 and
0.13.8. For the other cargo recipes in OE-Core, the scopes are the
same as in v1.
[1] https://lore.kernel.org/openembedded-core/20261105122145.543282-1-stondo@gmail.com/
[2] https://lore.kernel.org/bitbake-devel/20261118092831.201831-1-stondo@gmail.com/
[3] https://lore.kernel.org/openembedded-core/DMYFTUY2A6RH.1WI2PDCNCPOJY@bootlin.com/
Stefano Tondo (4):
spdx30: Take name, version and PURL of downloads from the fetcher
spdx30: Record the lifecycle scope of crate dependencies
spdx30: Add the PURLs of the source downloads to the output packages
oeqa/selftest: spdx: Test crate identity, scope and output PURLs
meta/classes-recipe/cargo.bbclass | 17 +++
meta/lib/oe/spdx30_tasks.py | 128 +++++++++++++++++-
.../oeqa/selftest/cases/oelib/cargo_scopes.py | 73 ++++++++++
meta/lib/oeqa/selftest/cases/spdx.py | 122 +++++++++++++++++
4 files changed, 339 insertions(+), 1 deletion(-)
create mode 100644 meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py
--
2.56.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v2 1/4] spdx30: Take name, version and PURL of downloads from the fetcher
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
@ 2026-11-18 23:44 ` Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 2/4] spdx30: Record the lifecycle scope of crate dependencies Stefano Tondo
` (3 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: Stefano Tondo @ 2026-11-18 23:44 UTC (permalink / raw)
To: openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Mathieu Dubois-Briand, stefano.tondo.ext
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
The fetchers of package registries (crate, gomod, npm) know the name
and the version of each package they download. Today the SPDX of such
a download only has the file name, so an SBOM scanner cannot match
the dependency.
Ask the fetcher with package_identity() and set the name, the version
and the PURL of the download. This needs package_identity() in
bitbake.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
meta/lib/oe/spdx30_tasks.py | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 6eaa74ce3d..58fe45ec7e 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -463,6 +463,19 @@ def enrich_source_package(d, dl, fd, file_name, primary_purpose):
)
+def set_package_identity(d, dl, fd):
+ """Set name, version and PURL of a download that names a package.
+
+ Fetchers of package registries (crate, gomod, npm) know which package
+ a URL downloads, see FetchMethod.package_identity() in bitbake.
+ """
+ identity = fd.method.package_identity(fd, d)
+ if identity:
+ dl.name = identity["name"]
+ dl.software_packageVersion = identity["version"]
+ dl.software_packageUrl = identity["purl"]
+
+
def add_download_files(d, objset):
inputs = set()
release_date = get_release_date(d)
@@ -528,6 +541,7 @@ def add_download_files(d, objset):
)
enrich_source_package(d, dl, fd, file_name, primary_purpose)
+ set_package_identity(d, dl, fd)
if release_date is not None:
dl.releaseTime = release_date
--
2.56.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 2/4] spdx30: Record the lifecycle scope of crate dependencies
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 1/4] spdx30: Take name, version and PURL of downloads from the fetcher Stefano Tondo
@ 2026-11-18 23:44 ` Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 3/4] spdx30: Add the PURLs of the source downloads to the output packages Stefano Tondo
` (2 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: Stefano Tondo @ 2026-11-18 23:44 UTC (permalink / raw)
To: openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Mathieu Dubois-Briand, stefano.tondo.ext
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
A Cargo.lock file lists the crates for all targets, and some crates
only run at build time (build scripts, proc-macros and the crates
they use). The fetcher downloads all of them, but only some end up in
the binary.
Add a do_cargo_metadata task that writes the dependency graph for the
target with cargo metadata --filter-platform. The create_spdx task
reads this graph. It links the build to each crate download with a
scoped dependsOn relationship: runtime, build or test. A crate that is
not used on the target gets an annotation instead.
The walk expands each workspace member only once. Members can depend
on each other in a cycle, and a member can be a dev dependency of
itself, as uv-preview is in python3-uv-build.
Add unit tests for the scopes, with member cycles.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
meta/classes-recipe/cargo.bbclass | 17 ++++
meta/lib/oe/spdx30_tasks.py | 90 +++++++++++++++++++
.../oeqa/selftest/cases/oelib/cargo_scopes.py | 73 +++++++++++++++
3 files changed, 180 insertions(+)
create mode 100644 meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py
diff --git a/meta/classes-recipe/cargo.bbclass b/meta/classes-recipe/cargo.bbclass
index 586071988a..b3e82f9d5b 100644
--- a/meta/classes-recipe/cargo.bbclass
+++ b/meta/classes-recipe/cargo.bbclass
@@ -26,6 +26,23 @@ cargo_do_compile () {
"${CARGO}" build ${CARGO_BUILD_FLAGS} ${PACKAGECONFIG_CONFARGS} "$@"
}
+# The crate dependency graph for the target, with the crates that are not
+# used on it removed. create-spdx uses it to give each crate download a
+# lifecycle scope.
+CARGO_METADATA_FILE = "${WORKDIR}/cargo-metadata.json"
+
+do_cargo_metadata () {
+ rm -f "${CARGO_METADATA_FILE}"
+ # Some recipes inherit cargo without being a cargo project
+ if [ ! -f "${CARGO_MANIFEST_PATH}" ]; then
+ return
+ fi
+ "${CARGO}" metadata --format-version 1 --offline --locked \
+ --manifest-path "${CARGO_MANIFEST_PATH}" \
+ --filter-platform "${RUST_HOST_SYS}" > "${CARGO_METADATA_FILE}"
+}
+addtask cargo_metadata after do_configure before do_compile do_create_spdx
+
cargo_do_install () {
local have_installed=false
for tgt in "${B}/target/${CARGO_TARGET_SUBDIR}/"*; do
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 58fe45ec7e..7aae73ed0d 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -476,6 +476,94 @@ def set_package_identity(d, dl, fd):
dl.software_packageUrl = identity["purl"]
+def get_cargo_scopes(d):
+ """Return {(crate, version): scope} from CARGO_METADATA_FILE.
+
+ The file is the output of cargo metadata --filter-platform for the
+ target, so crates used only on other targets are absent. Walks the
+ dependency graph from the workspace members: normal dependencies are
+ runtime, build dependencies and proc-macro crates (and everything they
+ pull in) are build, dev dependencies of the members are test.
+ """
+ path = d.getVar("CARGO_METADATA_FILE")
+ if not path or not os.path.exists(path):
+ return None
+
+ with open(path) as f:
+ metadata = json.load(f)
+
+ packages = {p["id"]: p for p in metadata["packages"]}
+ nodes = {n["id"]: n for n in metadata["resolve"]["nodes"]}
+ members = set(metadata["workspace_members"])
+ rank = {"test": 1, "build": 2, "runtime": 3}
+
+ def is_proc_macro(pkg_id):
+ return any("proc-macro" in t["kind"] for t in packages[pkg_id]["targets"])
+
+ scopes = {}
+ expanded_members = set()
+ stack = [(m, "runtime") for m in members]
+ while stack:
+ pkg_id, scope = stack.pop()
+ if pkg_id in members:
+ # A member is a root with the runtime scope. Expand it only
+ # once: members can depend on each other in a cycle, and a
+ # member can even be a dev dependency of itself.
+ if pkg_id in expanded_members:
+ continue
+ expanded_members.add(pkg_id)
+ scope = "runtime"
+ else:
+ if scope == "runtime" and is_proc_macro(pkg_id):
+ scope = "build"
+ if rank.get(scopes.get(pkg_id), 0) >= rank[scope]:
+ continue
+ scopes[pkg_id] = scope
+ for dep in nodes[pkg_id]["deps"]:
+ for kind in (k["kind"] for k in dep["dep_kinds"]):
+ if kind == "dev":
+ if pkg_id in members:
+ stack.append((dep["pkg"], "test"))
+ elif kind == "build":
+ stack.append((dep["pkg"], "test" if scope == "test" else "build"))
+ else:
+ stack.append((dep["pkg"], scope))
+
+ return {
+ (packages[i]["name"], packages[i]["version"]): s for i, s in scopes.items()
+ }
+
+
+def add_dependency_scopes(d, objset, build, source_files):
+ """Give crate downloads a lifecycle scope, see get_cargo_scopes()."""
+ scopes = get_cargo_scopes(d)
+ if scopes is None:
+ return
+
+ by_scope = {}
+ for dl in source_files:
+ if not (getattr(dl, "software_packageUrl", None) or "").startswith("pkg:cargo/"):
+ continue
+ scope = scopes.get((dl.name, dl.software_packageVersion))
+ if scope:
+ by_scope.setdefault(scope, []).append(dl)
+ else:
+ objset.new_annotation(
+ dl,
+ "%s %s is fetched but not built: not used on target %s"
+ % (dl.name, dl.software_packageVersion, d.getVar("RUST_HOST_SYS")),
+ oe.spdx30.AnnotationType.other,
+ )
+
+ for scope, dls in sorted(by_scope.items()):
+ objset.new_scoped_relationship(
+ [build],
+ oe.spdx30.RelationshipType.dependsOn,
+ getattr(oe.spdx30.LifecycleScopeType, scope),
+ sorted(oe.sbom30.get_element_link_id(dl) for dl in dls),
+ )
+
+
def add_download_files(d, objset):
inputs = set()
release_date = get_release_date(d)
@@ -978,6 +1066,8 @@ def create_spdx(d):
source_files = add_download_files(d, build_objset)
build_inputs |= source_files
+ if bb.data.inherits_class("cargo", d):
+ add_dependency_scopes(d, build_objset, build, source_files)
recipe_spdx_license = add_license_expression(
d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset]
diff --git a/meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py b/meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py
new file mode 100644
index 0000000000..6273346668
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py
@@ -0,0 +1,73 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import json
+import os
+import tempfile
+from unittest.case import TestCase
+
+import bb.data
+import oe.spdx30_tasks
+
+
+def package(name, proc_macro=False):
+ kind = ["proc-macro"] if proc_macro else ["lib"]
+ return {"id": name, "name": name, "version": "1.0.0", "targets": [{"kind": kind}]}
+
+
+def dep(name, *kinds):
+ return {"pkg": name, "dep_kinds": [{"kind": kind} for kind in kinds or (None,)]}
+
+
+class TestCargoScopes(TestCase):
+ def get_scopes(self, members, packages, deps):
+ metadata = {
+ "packages": packages,
+ "workspace_members": members,
+ "resolve": {"nodes": [{"id": p["id"], "deps": deps.get(p["id"], [])} for p in packages]},
+ }
+ with tempfile.TemporaryDirectory() as tmpdir:
+ path = os.path.join(tmpdir, "cargo-metadata.json")
+ with open(path, "w") as f:
+ json.dump(metadata, f)
+ d = bb.data.init()
+ d.setVar("CARGO_METADATA_FILE", path)
+ scopes = oe.spdx30_tasks.get_cargo_scopes(d)
+ return {name: scope for (name, version), scope in scopes.items()}
+
+ def test_scopes(self):
+ packages = [package(n) for n in ("app", "runtime-dep", "build-dep", "build-dep-dep",
+ "dev-dep", "shared")]
+ packages.append(package("derive", proc_macro=True))
+ deps = {
+ "app": [dep("runtime-dep"), dep("build-dep", "build"), dep("dev-dep", "dev"),
+ dep("derive"), dep("shared", "dev")],
+ "build-dep": [dep("build-dep-dep")],
+ "runtime-dep": [dep("shared")],
+ }
+ self.assertEqual(self.get_scopes(["app"], packages, deps), {
+ "runtime-dep": "runtime",
+ "shared": "runtime",
+ "build-dep": "build",
+ "build-dep-dep": "build",
+ "derive": "build",
+ "dev-dep": "test",
+ })
+
+ def test_member_cycles(self):
+ # Workspace members that depend on each other in a cycle, and a member
+ # that is a dev dependency of itself, as in uv. Each member is a root
+ # and is expanded once.
+ packages = [package(n) for n in ("member-a", "member-b", "member-c", "crate-a", "crate-c")]
+ deps = {
+ "member-a": [dep("member-b"), dep("member-a", "dev"), dep("crate-a")],
+ "member-b": [dep("member-c"), dep("member-a", "dev")],
+ "member-c": [dep("member-a"), dep("member-c", None, "dev"), dep("crate-c", "dev")],
+ }
+ self.assertEqual(self.get_scopes(["member-a", "member-b", "member-c"], packages, deps), {
+ "crate-a": "runtime",
+ "crate-c": "test",
+ })
--
2.56.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 3/4] spdx30: Add the PURLs of the source downloads to the output packages
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 1/4] spdx30: Take name, version and PURL of downloads from the fetcher Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 2/4] spdx30: Record the lifecycle scope of crate dependencies Stefano Tondo
@ 2026-11-18 23:44 ` Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs Stefano Tondo
2026-11-19 12:34 ` [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Mathieu Dubois-Briand
4 siblings, 0 replies; 9+ messages in thread
From: Stefano Tondo @ 2026-11-18 23:44 UTC (permalink / raw)
To: openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Mathieu Dubois-Briand, stefano.tondo.ext
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
Statically linked code, for example a Rust crate, is part of the
binary, but the SPDX of the output package only names the recipe.
A vulnerability scanner that checks the packages of an image does
not find the dependency.
Add the PURLs of the source downloads to the external identifiers of
each output package. Leave out the crates that the cargo dependency
graph proves are not built into the target. The software_packageUrl
of the package stays the first PURL in SPDX_PACKAGE_URLS.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
meta/lib/oe/spdx30_tasks.py | 30 ++++++++++++++++++++++++++----
1 file changed, 26 insertions(+), 4 deletions(-)
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 7aae73ed0d..49494a5007 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -535,16 +535,24 @@ def get_cargo_scopes(d):
def add_dependency_scopes(d, objset, build, source_files):
- """Give crate downloads a lifecycle scope, see get_cargo_scopes()."""
+ """Give crate downloads a lifecycle scope, see get_cargo_scopes().
+
+ Returns the crate downloads that are not built into the target: crates
+ that are not used on it, build scripts, proc-macros and the crates
+ they use, and test-only crates.
+ """
scopes = get_cargo_scopes(d)
if scopes is None:
- return
+ return set()
by_scope = {}
+ not_in_target = set()
for dl in source_files:
if not (getattr(dl, "software_packageUrl", None) or "").startswith("pkg:cargo/"):
continue
scope = scopes.get((dl.name, dl.software_packageVersion))
+ if scope != "runtime":
+ not_in_target.add(dl)
if scope:
by_scope.setdefault(scope, []).append(dl)
else:
@@ -563,6 +571,8 @@ def add_dependency_scopes(d, objset, build, source_files):
sorted(oe.sbom30.get_element_link_id(dl) for dl in dls),
)
+ return not_in_target
+
def add_download_files(d, objset):
inputs = set()
@@ -1066,8 +1076,20 @@ def create_spdx(d):
source_files = add_download_files(d, build_objset)
build_inputs |= source_files
+
+ # The output packages inherit the PURLs of the downloads they are built
+ # from, as the code of statically linked dependencies is in them. Only
+ # the downloads known not to be built into the target are left out.
+ not_in_target = set()
if bb.data.inherits_class("cargo", d):
- add_dependency_scopes(d, build_objset, build, source_files)
+ not_in_target = add_dependency_scopes(d, build_objset, build, source_files)
+ dependency_purls = sorted(
+ {
+ dl.software_packageUrl
+ for dl in source_files
+ if getattr(dl, "software_packageUrl", None) and dl not in not_in_target
+ }
+ )
recipe_spdx_license = add_license_expression(
d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset]
@@ -1180,7 +1202,7 @@ def create_spdx(d):
d.getVar("SPDX_PACKAGE_URLS:%s" % package)
or d.getVar("SPDX_PACKAGE_URLS")
or ""
- ).split()
+ ).split() + dependency_purls
set_purls(spdx_package, purls)
--
2.56.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
` (2 preceding siblings ...)
2026-11-18 23:44 ` [PATCH v2 3/4] spdx30: Add the PURLs of the source downloads to the output packages Stefano Tondo
@ 2026-11-18 23:44 ` Stefano Tondo
2026-11-19 10:03 ` Paul Barker
2026-11-19 12:34 ` [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Mathieu Dubois-Briand
4 siblings, 1 reply; 9+ messages in thread
From: Stefano Tondo @ 2026-11-18 23:44 UTC (permalink / raw)
To: openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Mathieu Dubois-Briand, stefano.tondo.ext
From: Stefano Tondo <stefano.tondo.ext@siemens.com>
python3-bcrypt fetches 42 crates. The new tests check three things:
- Each crate download has its name, version and PURL.
- Each crate download has a lifecycle scope, or an annotation that it
is not built.
- The output package gets the PURLs of the runtime crates only.
The tests name only crates that do not change on recipe upgrades.
blowfish is the cipher of bcrypt. Only the pyo3 proc-macros use syn,
at build time. The other checks do not use crate names.
Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
meta/lib/oeqa/selftest/cases/spdx.py | 122 +++++++++++++++++++++++++++
1 file changed, 122 insertions(+)
diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py
index ce9a8af4f6..d75c6146c7 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -9,6 +9,7 @@ import hashlib
from datetime import datetime, timezone
from oeqa.selftest.case import OESelftestTestCase
from oeqa.utils.commands import bitbake, get_bb_var, get_bb_vars
+import oe.purl
import oe.spdx30
@@ -543,3 +544,124 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
# Document should be fully linked
self.check_objset_missing_ids(objset)
+
+ def test_crate_package_identity(self):
+ """Test that crate downloads carry name, version and a cargo PURL."""
+ objset = self.check_recipe_spdx(
+ "python3-bcrypt",
+ "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-python3-bcrypt.spdx.json",
+ )
+
+ prefix = "https://static.crates.io/crates/"
+ crates = 0
+ for pkg in objset.foreach_type(oe.spdx30.software_Package):
+ location = pkg.software_downloadLocation or ""
+ if not location.startswith(prefix):
+ continue
+ name, version, _ = location[len(prefix):].split("/")
+ self.assertEqual(pkg.name, name)
+ self.assertEqual(pkg.software_packageVersion, version)
+ self.assertEqual(
+ pkg.software_packageUrl,
+ f"pkg:cargo/{oe.purl.quote(name)}@{oe.purl.quote(version)}",
+ )
+ crates += 1
+
+ self.assertGreater(crates, 0, "no crate downloads found")
+
+ def test_crate_dependency_scope(self):
+ """Test that crate downloads get a lifecycle scope for the target.
+
+ Every crate download is either the target of one scoped dependsOn
+ relationship or annotated as not built. python3-bcrypt fetches crates
+ for other platforms (WebAssembly, UEFI), which are never built for a
+ Linux target. blowfish is the cipher of bcrypt, and syn only runs at
+ build time in proc-macros, so these names do not change on upgrades.
+ """
+ objset = self.check_recipe_spdx(
+ "python3-bcrypt",
+ "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-python3-bcrypt.spdx.json",
+ )
+
+ crates = {
+ pkg._id: pkg.name
+ for pkg in objset.foreach_type(oe.spdx30.software_Package)
+ if (pkg.software_packageUrl or "").startswith("pkg:cargo/")
+ and (pkg.software_downloadLocation or "").startswith("https://static.crates.io/")
+ }
+
+ scope_of = {}
+ for rel in objset.foreach_type(oe.spdx30.LifecycleScopedRelationship):
+ if rel.relationshipType != oe.spdx30.RelationshipType.dependsOn:
+ continue
+ for to in rel.to:
+ to_id = to if isinstance(to, str) else to._id
+ if to_id in crates:
+ self.assertNotIn(crates[to_id], scope_of)
+ scope_of[crates[to_id]] = rel.scope
+
+ not_built = set()
+ for annotation in objset.foreach_type(oe.spdx30.Annotation):
+ subject = annotation.subject
+ subject_id = subject if isinstance(subject, str) else subject._id
+ if subject_id in crates and "not built" in annotation.statement:
+ not_built.add(crates[subject_id])
+
+ self.assertEqual(set(scope_of) | not_built, set(crates.values()))
+ self.assertFalse(set(scope_of) & not_built)
+ self.assertTrue(not_built, "no crate for another platform found")
+ self.assertEqual(scope_of["syn"], oe.spdx30.LifecycleScopeType.build)
+ self.assertEqual(scope_of["blowfish"], oe.spdx30.LifecycleScopeType.runtime)
+
+ def test_crate_purls_in_output_package(self):
+ """Test that the output package inherits the PURLs of its crates.
+
+ The package gets the PURLs of the crates with the runtime scope in
+ the build SPDX, and no other crate PURLs. syn only runs at build
+ time in proc-macros, so it is not in the binary.
+ """
+ objset = self.check_recipe_spdx(
+ "python3-bcrypt",
+ "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/packages/package-python3-bcrypt.spdx.json",
+ )
+
+ package = next(
+ pkg
+ for pkg in objset.foreach_type(oe.spdx30.software_Package)
+ if pkg.name == "python3-bcrypt"
+ )
+ purls = {
+ e.identifier
+ for e in package.externalIdentifier
+ if e.externalIdentifierType == oe.spdx30.ExternalIdentifierType.packageUrl
+ }
+ crates = {p[len("pkg:cargo/"):].split("@")[0] for p in purls if p.startswith("pkg:cargo/")}
+
+ # The package's own PURL stays first
+ own_purls = get_bb_var("SPDX_PACKAGE_URLS", "python3-bcrypt").split()
+ self.assertEqual(package.software_packageUrl, own_purls[0])
+ self.assertTrue(set(own_purls) <= purls)
+ self.assertIn("blowfish", crates)
+ self.assertNotIn("syn", crates)
+
+ spdx_vars = get_bb_vars(["DEPLOY_DIR_SPDX", "SSTATE_PKGARCH"], "python3-bcrypt")
+ build_objset = self.check_spdx_file(
+ "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-python3-bcrypt.spdx.json".format(**spdx_vars)
+ )
+ downloads = {
+ pkg._id: pkg.software_packageUrl
+ for pkg in build_objset.foreach_type(oe.spdx30.software_Package)
+ if (pkg.software_packageUrl or "").startswith("pkg:cargo/")
+ }
+ runtime = set()
+ for rel in build_objset.foreach_type(oe.spdx30.LifecycleScopedRelationship):
+ if rel.scope != oe.spdx30.LifecycleScopeType.runtime:
+ continue
+ for to in rel.to:
+ to_id = to if isinstance(to, str) else to._id
+ if to_id in downloads:
+ runtime.add(downloads[to_id])
+
+ self.assertTrue(runtime, "no crate with the runtime scope found")
+ cargo_purls = {p for p in purls if p.startswith("pkg:cargo/")}
+ self.assertEqual(cargo_purls - set(own_purls), runtime)
--
2.56.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs
2026-11-18 23:44 ` [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs Stefano Tondo
@ 2026-11-19 10:03 ` Paul Barker
2026-11-19 10:30 ` Stefano Tondo
0 siblings, 1 reply; 9+ messages in thread
From: Paul Barker @ 2026-11-19 10:03 UTC (permalink / raw)
To: Stefano Tondo, openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
Mathieu Dubois-Briand, stefano.tondo.ext
On Wed Nov 18, 2026 at 11:44 PM GMT, Stefano Tondo wrote:
> From: Stefano Tondo <stefano.tondo.ext@siemens.com>
>
> python3-bcrypt fetches 42 crates. The new tests check three things:
>
> - Each crate download has its name, version and PURL.
> - Each crate download has a lifecycle scope, or an annotation that it
> is not built.
> - The output package gets the PURLs of the runtime crates only.
>
> The tests name only crates that do not change on recipe upgrades.
> blowfish is the cipher of bcrypt. Only the pyo3 proc-macros use syn,
> at build time. The other checks do not use crate names.
Hi Stefano,
A couple of quick questions about this test case.
Does it use a fixed version of the python3-bcrypt recipe or just
whatever version is currently in OE-Core?
If it tracks the version used in OE-Core, is it likely to start failing
when python3-bcrypt adds or removes dependencies in the future?
Best regards,
--
Paul Barker
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs
2026-11-19 10:03 ` Paul Barker
@ 2026-11-19 10:30 ` Stefano Tondo
0 siblings, 0 replies; 9+ messages in thread
From: Stefano Tondo @ 2026-11-19 10:30 UTC (permalink / raw)
To: Paul Barker
Cc: openembedded-core, Joshua Watt, Alexander Kanavin,
Stefan Herbrechtsmeier, Mathieu Dubois-Briand, stefano.tondo.ext
On Thu Nov 19, 2026 at 10:03 AM GMT, Paul Barker wrote:
> Does it use a fixed version of the python3-bcrypt recipe or just
> whatever version is currently in OE-Core?
Hi Paul,
It uses the version that is currently in OE-Core. There is no copy of
the recipe in meta-selftest.
> If it tracks the version used in OE-Core, is it likely to start failing
> when python3-bcrypt adds or removes dependencies in the future?
I do not expect it to fail. I wrote the tests for this case. They do
not contain crate versions or crate counts. The 42 crates are only in the
commit message. The tests get the expected values from the build:
- The name, version and PURL of each crate must agree with its
crates.io download URL.
- Each crate must have exactly one lifecycle scope, or an annotation
that it is not built.
- The cargo PURLs of the output package must be equal to the set of
crates that have the runtime scope in the build SPDX.
When the recipe adds or removes a dependency, these sets change together.
The tests use three facts about the dependencies of python3-bcrypt:
- blowfish has the runtime scope. It is the cipher of bcrypt.
- syn has the build scope. The pyo3 proc-macros use it.
- At least one crate is for another platform, so it is not built.
Today there are five: r-efi, wasi, wasip2 and wit-bindgen come
from getrandom, and portable-atomic comes from pyo3.
Only a large change in bcrypt or pyo3 can make the first two false.
The third is less certain, but getrandom supports WebAssembly and UEFI,
and it is unlikely to stop. If this check fails, the message shows the
fact that changed. Then we can use a different crate or remove the
check.
Best regards,
Stefano
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 0/4] spdx30: identity and purls of dependency downloads
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
` (3 preceding siblings ...)
2026-11-18 23:44 ` [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs Stefano Tondo
@ 2026-11-19 12:34 ` Mathieu Dubois-Briand
2026-11-19 13:35 ` Stefano Tondo
4 siblings, 1 reply; 9+ messages in thread
From: Mathieu Dubois-Briand @ 2026-11-19 12:34 UTC (permalink / raw)
To: Stefano Tondo, openembedded-core
Cc: Joshua Watt, Alexander Kanavin, Stefan Herbrechtsmeier,
stefano.tondo.ext
On Thu Nov 19, 2026 at 12:44 AM CET, Stefano Tondo wrote:
> Hi,
>
> This series implements the parts of the RFC "spdx30: name, version
> and purl for dependency downloads" [1] that we agreed on. It needs
> the bitbake series "fetch: package identity of crate, gomod and npm
> URLs" [2].
>
> 1. The SPDX of a crate, gomod or npm download gets the name, the
> version and the purl from the fetcher.
> 2. For Cargo recipes, a new do_cargo_metadata task writes the crate
> dependency graph for the target. The build links to each crate
> download with a runtime, build or test scope. A crate that is not
> used on the target gets an annotation.
> 3. Each output package gets the purls of its source downloads as
> external identifiers, as Joshua proposed: a statically linked
> dependency is part of the binary. A purl is left out only when the
> cargo graph proves that the crate is not in the target. The
> software_packageUrl of the package does not change.
> 4. Selftests with python3-bcrypt.
>
> For python3-bcrypt, the output package gets the purls of 25 of its
> 42 crates. The other 17 crates are not built into the target.
>
> Two points for review:
>
> - All output packages of a recipe get the purls, also -dbg and -src.
> I have no proof per package, so I followed the rule "keep the purl
> unless there is proof".
> - Go recipes have no filter yet. go version -m can give the proof per
> binary, and I can add it in a follow-up.
>
Hi Stefano,
Thanks for the v2, it does look better in term of memory.
Yet I can see this failure:
ERROR: zip-native-3.0-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
...
ERROR: zip-native-3.0-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_create_spdx(d)
0003:
File: '/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/classes/create-spdx-3.0.bbclass', lineno: 264, function: do_create_spdx
0260:addtask do_create_recipe_spdx_setscene
0261:
0262:python do_create_spdx() {
0263: import oe.spdx30_tasks
*** 0264: oe.spdx30_tasks.create_spdx(d)
0265:}
0266:addtask do_create_spdx after \
0267: do_unpack \
0268: do_patch \
File: '/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/lib/oe/spdx30_tasks.py', lineno: 1077, function: create_spdx
1073: build_inputs = set()
1074:
1075: cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION"))
1076:
*** 1077: source_files = add_download_files(d, build_objset)
1078: build_inputs |= source_files
1079:
1080: # The output packages inherit the PURLs of the downloads they are built
1081: # from, as the code of statically linked dependencies is in them. Only
File: '/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/lib/oe/spdx30_tasks.py', lineno: 642, function: add_download_files
0638: )
0639: )
0640:
0641: enrich_source_package(d, dl, fd, file_name, primary_purpose)
*** 0642: set_package_identity(d, dl, fd)
0643:
0644: if release_date is not None:
0645: dl.releaseTime = release_date
0646:
File: '/srv/pokybuild/yocto-worker/qemux86-world/build/layers/openembedded-core/meta/lib/oe/spdx30_tasks.py', lineno: 472, function: set_package_identity
0468:
0469: Fetchers of package registries (crate, gomod, npm) know which package
0470: a URL downloads, see FetchMethod.package_identity() in bitbake.
0471: """
*** 0472: identity = fd.method.package_identity(fd, d)
0473: if identity:
0474: dl.name = identity["name"]
0475: dl.software_packageVersion = identity["version"]
0476: dl.software_packageUrl = identity["purl"]
Exception: AttributeError: 'Wget' object has no attribute 'package_identity'
ERROR: Logfile of failure stored in: /srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/gstreamer1.0-plugins-rs/0.15.4/temp/log.do_compile.1234281
Log data follows:
| DEBUG: Executing shell function do_compile
| NOTE: Using rust targets from /srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/gstreamer1.0-plugins-rs/0.15.4/rust-targets/
| Error: CliError { error: Some(no matching package named `gstreamer` found
| location searched: directory source `/srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/gstreamer1.0-plugins-rs/0.15.4/sources/cargo_home/bitbake` (which is replacing registry `crates-io`)
| required by package `gst-plugin-tutorial v0.15.4 (/srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/gstreamer1.0-plugins-rs/0.15.4/sources/gstreamer1.0-plugins-rs-0.15.4/tutorial)`
| help: packages with similar names: bitreader
|
| note: offline mode (via `--offline`) can sometimes cause surprising resolution failures
| help: if this error is too confusing you may wish to retry without `--offline`
|
| Stack backtrace:
| 0: <unknown>
| 1: <unknown>
| 2: <unknown>
| 3: <unknown>
| 4: <unknown>
| 5: <unknown>
| 6: <unknown>
| 7: <unknown>
| 8: <unknown>
| 9: <unknown>
| 10: <unknown>
| 11: <unknown>
| 12: <unknown>
| 13: <unknown>
| 14: __libc_start_call_main
| 15: __libc_start_main@GLIBC_2.2.5
| 16: <unknown>), exit_code: 101 }
| WARNING: exit code 1 from a shell command.
And something similar later with lib32-lsb-release:
ERROR: lib32-lsb-release-1.4-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/59/builds/5072
Can you have a look at the issue?
Thanks,
Mathieu
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 0/4] spdx30: identity and purls of dependency downloads
2026-11-19 12:34 ` [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Mathieu Dubois-Briand
@ 2026-11-19 13:35 ` Stefano Tondo
0 siblings, 0 replies; 9+ messages in thread
From: Stefano Tondo @ 2026-11-19 13:35 UTC (permalink / raw)
To: Mathieu Dubois-Briand
Cc: openembedded-core, Joshua Watt, Alexander Kanavin,
Stefan Herbrechtsmeier, Jaipaul Cheernam, stefano.tondo.ext
On Thu Nov 19, 2026 at 1:34 PM CET, Mathieu Dubois-Briand wrote:
> Thanks for the v2, it does look better in term of memory.
>
> Yet I can see this failure:
>
> ERROR: zip-native-3.0-r0 do_create_spdx: Error executing a python function in exec_func_python() autogenerated:
[...]
> Exception: AttributeError: 'Wget' object has no attribute 'package_identity'
Hi Mathieu,
Thanks for the test.
This series needs the bitbake series "fetch: package identity of
crate, gomod and npm URLs" [1]. It adds FetchMethod.package_identity().
The bitbake of this build (b6d163ca65 on mathieu/master-next) does not
have it. The two bitbake patches apply cleanly on b6d163ca65. Can you
add them for the next build? The lib32-lsb-release failure has the
same cause.
> ERROR: Logfile of failure stored in: /srv/pokybuild/yocto-worker/qemux86-world/build/build/tmp/work/x86-64-v3-poky-linux/gstreamer1.0-plugins-rs/0.15.4/temp/log.do_compile.1234281
[...]
> | Error: CliError { error: Some(no matching package named `gstreamer` found
I think this failure comes from "gstreamer1.0-plugins-rs: Upgrade
0.15.3 -> 0.15.4", and not from this series. This series does not
change do_compile.
At the 0.15.4 tag, Cargo.lock takes gstreamer, gstreamer-base,
gstreamer-audio and gstreamer-video from the gstreamer-rs git
repository, and not from crates.io. The new
gstreamer1.0-plugins-rs-crates.inc lists only the crates.io crates, so
these crates are missing: it has 1048 entries, 0.15.3 had 1098. No
SRC_URI entry fetches them from git, so the offline build does not
find them. I added Jaipaul to Cc.
[1] https://lore.kernel.org/bitbake-devel/20261118092831.201831-1-stondo@gmail.com/
Best regards,
Stefano
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-11-19 13:35 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 1/4] spdx30: Take name, version and PURL of downloads from the fetcher Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 2/4] spdx30: Record the lifecycle scope of crate dependencies Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 3/4] spdx30: Add the PURLs of the source downloads to the output packages Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs Stefano Tondo
2026-11-19 10:03 ` Paul Barker
2026-11-19 10:30 ` Stefano Tondo
2026-11-19 12:34 ` [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Mathieu Dubois-Briand
2026-11-19 13:35 ` Stefano Tondo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox