All of lore.kernel.org
 help / color / mirror / Atom feed
From: Stefano Tondo <stondo@gmail.com>
To: openembedded-core@lists.openembedded.org
Cc: Joshua Watt <JPEWhacker@gmail.com>,
	Alexander Kanavin <alex.kanavin@gmail.com>,
	Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>,
	Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>,
	stefano.tondo.ext@siemens.com
Subject: [PATCH v2 2/4] spdx30: Record the lifecycle scope of crate dependencies
Date: Thu, 19 Nov 2026 00:44:23 +0100	[thread overview]
Message-ID: <20261118234425.3681880-3-stondo@gmail.com> (raw)
In-Reply-To: <20261118234425.3681880-1-stondo@gmail.com>

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

A Cargo.lock file lists the crates for all targets, and some crates
only run at build time (build scripts, proc-macros and the crates
they use). The fetcher downloads all of them, but only some end up in
the binary.

Add a do_cargo_metadata task that writes the dependency graph for the
target with cargo metadata --filter-platform. The create_spdx task
reads this graph. It links the build to each crate download with a
scoped dependsOn relationship: runtime, build or test. A crate that is
not used on the target gets an annotation instead.

The walk expands each workspace member only once. Members can depend
on each other in a cycle, and a member can be a dev dependency of
itself, as uv-preview is in python3-uv-build.

Add unit tests for the scopes, with member cycles.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/classes-recipe/cargo.bbclass             | 17 ++++
 meta/lib/oe/spdx30_tasks.py                   | 90 +++++++++++++++++++
 .../oeqa/selftest/cases/oelib/cargo_scopes.py | 73 +++++++++++++++
 3 files changed, 180 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py

diff --git a/meta/classes-recipe/cargo.bbclass b/meta/classes-recipe/cargo.bbclass
index 586071988a..b3e82f9d5b 100644
--- a/meta/classes-recipe/cargo.bbclass
+++ b/meta/classes-recipe/cargo.bbclass
@@ -26,6 +26,23 @@ cargo_do_compile () {
 	"${CARGO}" build ${CARGO_BUILD_FLAGS} ${PACKAGECONFIG_CONFARGS} "$@"
 }
 
+# The crate dependency graph for the target, with the crates that are not
+# used on it removed. create-spdx uses it to give each crate download a
+# lifecycle scope.
+CARGO_METADATA_FILE = "${WORKDIR}/cargo-metadata.json"
+
+do_cargo_metadata () {
+	rm -f "${CARGO_METADATA_FILE}"
+	# Some recipes inherit cargo without being a cargo project
+	if [ ! -f "${CARGO_MANIFEST_PATH}" ]; then
+		return
+	fi
+	"${CARGO}" metadata --format-version 1 --offline --locked \
+		--manifest-path "${CARGO_MANIFEST_PATH}" \
+		--filter-platform "${RUST_HOST_SYS}" > "${CARGO_METADATA_FILE}"
+}
+addtask cargo_metadata after do_configure before do_compile do_create_spdx
+
 cargo_do_install () {
 	local have_installed=false
 	for tgt in "${B}/target/${CARGO_TARGET_SUBDIR}/"*; do
diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 58fe45ec7e..7aae73ed0d 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -476,6 +476,94 @@ def set_package_identity(d, dl, fd):
         dl.software_packageUrl = identity["purl"]
 
 
+def get_cargo_scopes(d):
+    """Return {(crate, version): scope} from CARGO_METADATA_FILE.
+
+    The file is the output of cargo metadata --filter-platform for the
+    target, so crates used only on other targets are absent. Walks the
+    dependency graph from the workspace members: normal dependencies are
+    runtime, build dependencies and proc-macro crates (and everything they
+    pull in) are build, dev dependencies of the members are test.
+    """
+    path = d.getVar("CARGO_METADATA_FILE")
+    if not path or not os.path.exists(path):
+        return None
+
+    with open(path) as f:
+        metadata = json.load(f)
+
+    packages = {p["id"]: p for p in metadata["packages"]}
+    nodes = {n["id"]: n for n in metadata["resolve"]["nodes"]}
+    members = set(metadata["workspace_members"])
+    rank = {"test": 1, "build": 2, "runtime": 3}
+
+    def is_proc_macro(pkg_id):
+        return any("proc-macro" in t["kind"] for t in packages[pkg_id]["targets"])
+
+    scopes = {}
+    expanded_members = set()
+    stack = [(m, "runtime") for m in members]
+    while stack:
+        pkg_id, scope = stack.pop()
+        if pkg_id in members:
+            # A member is a root with the runtime scope. Expand it only
+            # once: members can depend on each other in a cycle, and a
+            # member can even be a dev dependency of itself.
+            if pkg_id in expanded_members:
+                continue
+            expanded_members.add(pkg_id)
+            scope = "runtime"
+        else:
+            if scope == "runtime" and is_proc_macro(pkg_id):
+                scope = "build"
+            if rank.get(scopes.get(pkg_id), 0) >= rank[scope]:
+                continue
+            scopes[pkg_id] = scope
+        for dep in nodes[pkg_id]["deps"]:
+            for kind in (k["kind"] for k in dep["dep_kinds"]):
+                if kind == "dev":
+                    if pkg_id in members:
+                        stack.append((dep["pkg"], "test"))
+                elif kind == "build":
+                    stack.append((dep["pkg"], "test" if scope == "test" else "build"))
+                else:
+                    stack.append((dep["pkg"], scope))
+
+    return {
+        (packages[i]["name"], packages[i]["version"]): s for i, s in scopes.items()
+    }
+
+
+def add_dependency_scopes(d, objset, build, source_files):
+    """Give crate downloads a lifecycle scope, see get_cargo_scopes()."""
+    scopes = get_cargo_scopes(d)
+    if scopes is None:
+        return
+
+    by_scope = {}
+    for dl in source_files:
+        if not (getattr(dl, "software_packageUrl", None) or "").startswith("pkg:cargo/"):
+            continue
+        scope = scopes.get((dl.name, dl.software_packageVersion))
+        if scope:
+            by_scope.setdefault(scope, []).append(dl)
+        else:
+            objset.new_annotation(
+                dl,
+                "%s %s is fetched but not built: not used on target %s"
+                % (dl.name, dl.software_packageVersion, d.getVar("RUST_HOST_SYS")),
+                oe.spdx30.AnnotationType.other,
+            )
+
+    for scope, dls in sorted(by_scope.items()):
+        objset.new_scoped_relationship(
+            [build],
+            oe.spdx30.RelationshipType.dependsOn,
+            getattr(oe.spdx30.LifecycleScopeType, scope),
+            sorted(oe.sbom30.get_element_link_id(dl) for dl in dls),
+        )
+
+
 def add_download_files(d, objset):
     inputs = set()
     release_date = get_release_date(d)
@@ -978,6 +1066,8 @@ def create_spdx(d):
 
     source_files = add_download_files(d, build_objset)
     build_inputs |= source_files
+    if bb.data.inherits_class("cargo", d):
+        add_dependency_scopes(d, build_objset, build, source_files)
 
     recipe_spdx_license = add_license_expression(
         d, build_objset, d.getVar("LICENSE"), license_data, [recipe_objset]
diff --git a/meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py b/meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py
new file mode 100644
index 0000000000..6273346668
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/oelib/cargo_scopes.py
@@ -0,0 +1,73 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import json
+import os
+import tempfile
+from unittest.case import TestCase
+
+import bb.data
+import oe.spdx30_tasks
+
+
+def package(name, proc_macro=False):
+    kind = ["proc-macro"] if proc_macro else ["lib"]
+    return {"id": name, "name": name, "version": "1.0.0", "targets": [{"kind": kind}]}
+
+
+def dep(name, *kinds):
+    return {"pkg": name, "dep_kinds": [{"kind": kind} for kind in kinds or (None,)]}
+
+
+class TestCargoScopes(TestCase):
+    def get_scopes(self, members, packages, deps):
+        metadata = {
+            "packages": packages,
+            "workspace_members": members,
+            "resolve": {"nodes": [{"id": p["id"], "deps": deps.get(p["id"], [])} for p in packages]},
+        }
+        with tempfile.TemporaryDirectory() as tmpdir:
+            path = os.path.join(tmpdir, "cargo-metadata.json")
+            with open(path, "w") as f:
+                json.dump(metadata, f)
+            d = bb.data.init()
+            d.setVar("CARGO_METADATA_FILE", path)
+            scopes = oe.spdx30_tasks.get_cargo_scopes(d)
+        return {name: scope for (name, version), scope in scopes.items()}
+
+    def test_scopes(self):
+        packages = [package(n) for n in ("app", "runtime-dep", "build-dep", "build-dep-dep",
+                                         "dev-dep", "shared")]
+        packages.append(package("derive", proc_macro=True))
+        deps = {
+            "app": [dep("runtime-dep"), dep("build-dep", "build"), dep("dev-dep", "dev"),
+                    dep("derive"), dep("shared", "dev")],
+            "build-dep": [dep("build-dep-dep")],
+            "runtime-dep": [dep("shared")],
+        }
+        self.assertEqual(self.get_scopes(["app"], packages, deps), {
+            "runtime-dep": "runtime",
+            "shared": "runtime",
+            "build-dep": "build",
+            "build-dep-dep": "build",
+            "derive": "build",
+            "dev-dep": "test",
+        })
+
+    def test_member_cycles(self):
+        # Workspace members that depend on each other in a cycle, and a member
+        # that is a dev dependency of itself, as in uv. Each member is a root
+        # and is expanded once.
+        packages = [package(n) for n in ("member-a", "member-b", "member-c", "crate-a", "crate-c")]
+        deps = {
+            "member-a": [dep("member-b"), dep("member-a", "dev"), dep("crate-a")],
+            "member-b": [dep("member-c"), dep("member-a", "dev")],
+            "member-c": [dep("member-a"), dep("member-c", None, "dev"), dep("crate-c", "dev")],
+        }
+        self.assertEqual(self.get_scopes(["member-a", "member-b", "member-c"], packages, deps), {
+            "crate-a": "runtime",
+            "crate-c": "test",
+        })
-- 
2.56.1


  parent reply	other threads:[~2026-11-18 23:44 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-18 23:44 [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 1/4] spdx30: Take name, version and PURL of downloads from the fetcher Stefano Tondo
2026-11-18 23:44 ` Stefano Tondo [this message]
2026-11-18 23:44 ` [PATCH v2 3/4] spdx30: Add the PURLs of the source downloads to the output packages Stefano Tondo
2026-11-18 23:44 ` [PATCH v2 4/4] oeqa/selftest: spdx: Test crate identity, scope and output PURLs Stefano Tondo
2026-11-19 10:03   ` Paul Barker
2026-11-19 10:30     ` Stefano Tondo
2026-11-19 12:34 ` [PATCH v2 0/4] spdx30: identity and purls of dependency downloads Mathieu Dubois-Briand
2026-11-19 13:35   ` Stefano Tondo
2026-11-19 16:37     ` Mathieu Dubois-Briand
     [not found]     ` <18E97F9E4135A022.1484727@lists.openembedded.org>
2026-11-19 16:43       ` [OE-core] " Mathieu Dubois-Briand

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261118234425.3681880-3-stondo@gmail.com \
    --to=stondo@gmail.com \
    --cc=JPEWhacker@gmail.com \
    --cc=alex.kanavin@gmail.com \
    --cc=mathieu.dubois-briand@bootlin.com \
    --cc=openembedded-core@lists.openembedded.org \
    --cc=stefan.herbrechtsmeier@weidmueller.com \
    --cc=stefano.tondo.ext@siemens.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.