* [PATCH 0/2] Add meta layer information to spdx
@ 2026-09-14 11:44 Daniel Wagenknecht
2026-09-14 11:44 ` [PATCH 1/2] spdx: add entries for meta-layers Daniel Wagenknecht
` (2 more replies)
0 siblings, 3 replies; 16+ messages in thread
From: Daniel Wagenknecht @ 2026-09-14 11:44 UTC (permalink / raw)
To: openembedded-core; +Cc: Daniel Wagenknecht
Hello,
as discussed before here
https://lists.openembedded.org/g/openembedded-core/message/226131
and here
https://lists.openembedded.org/g/openembedded-core/message/225774
it would be good to include information about the layers used in a build in the
generated spdx documents.
My primary goal is to track where all the configuration files added via file
protocol in `SRC_URI` originate from, but the approach could be extended to
track recipes, classes, machine config etc.
These patches are meant as a basis to discuss the approach and should be
considered draft status.
The approach is to generate a `layers.spdx.json` on each build invocation
that lists each layer used in the build as a `softwarePackage` with version
and source information as well as - if can be reasonably deducted - the
downloadLocation. Each source file added via file protocol is then linked to the
containing meta layer via a `contains` relationship.
Open Questions:
- is the overall architecture reasonable/correct?
- referring to layers is inconsistent across tooling. E.g. the core layer is
either named `core` as defined by `BBFILE_COLLECTIONS` or `meta` as that is
the layer directory in the openembedded-core repository.
I chose the latter since that is what the utility function
`oe.buildcfg.get_layer_revisions` returns. The package URL (PURL)
implementation uses the other naming scheme though. Any thoughts on which
should be chosen and why?
- I do not understand the alias linking yet. With the current implementation
I see many hash mismatches like
WARNING: gcc-runtime-15.3.0-r0 do_create_spdx: Element http://spdx.org/spdxdocs/layers-7ad0d8d7-57f9-5c3c-9411-49227dd50610/layers/layer/meta
has alias http://spdx.org/spdxdocs/UNIHASH-7ad0d8d7-57f9-5c3c-9411-49227dd50610/UNIHASH/layer/meta, but it should have prefix
http://spdxdocs.org/openembedded-alias/by-doc-hash/e80f1d61f4749e663e80a4dfb25ebb1127b5f50600d3065de7ed91030942a1ef/
I've currently got some time available to work on this to get a proper solution
integrated into openembedded-core.
Best Wishes
Daniel Wagenknecht
Daniel Wagenknecht (2):
spdx: add entries for meta-layers
spdx: link files to the meta-layer providing them
meta/classes/create-spdx-3.0.bbclass | 1 +
meta/classes/spdx-common.bbclass | 3 +
meta/lib/oe/sbom30.py | 12 +++
meta/lib/oe/spdx30_tasks.py | 118 ++++++++++++++++++++++++++-
4 files changed, 130 insertions(+), 4 deletions(-)
--
2.54.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [PATCH 1/2] spdx: add entries for meta-layers 2026-09-14 11:44 [PATCH 0/2] Add meta layer information to spdx Daniel Wagenknecht @ 2026-09-14 11:44 ` Daniel Wagenknecht 2026-09-14 11:44 ` [PATCH 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht 2026-09-15 8:02 ` [OE-core] [PATCH 0/2] Add meta layer information to spdx Antonin Godard 2 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-09-14 11:44 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht This adds support for determining the revision and downloadLocation of meta-layers and adds them to the OE generated SPDX document. This downloadLocation will result in a NOASSERTION entry if - the meta-layer is not in a git repo - the git repo is dirty - the git remote does not have a valid URL - the git remote does not contain the used revision as far as can be eluded from the local repo Variables to configure the behavior: - SPDX_LAYER_REMOTE (default: origin): select the name of the git remote to use for layers. Set to an empty string to always return NOASSERTION - SPDX_LAYER_REMOTE_layername: override SPDX_LAYER_REMOTE for a specific layer - SPDX_REQUIRE_LAYER_ASSERTION: set to true to fail a build if determining downloadLocation fails for any layer Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- meta/classes/create-spdx-3.0.bbclass | 1 + meta/classes/spdx-common.bbclass | 3 ++ meta/lib/oe/sbom30.py | 12 +++++ meta/lib/oe/spdx30_tasks.py | 74 ++++++++++++++++++++++++++++ 4 files changed, 90 insertions(+) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 56fd01fd533..1ef6b15ac22 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -285,6 +285,7 @@ addtask do_create_recipe_sbom_setscene python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() + oe.spdx30_tasks.write_layers_spdx(d) oe.spdx30_tasks.write_bitbake_spdx(d) } diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 40701730a69..b64537dfb48 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -89,6 +89,9 @@ SPDX_FILE_EXCLUDE_PATTERNS[doc] = "Space-separated list of Python regular \ (no filtering). Example: \ SPDX_FILE_EXCLUDE_PATTERNS = '\\.patch$ \\.diff$ /test/ \\.pyc$ \\.o$'" +SPDX_REQUIRE_LAYER_ASSERTION ??= "0" +SPDX_LAYER_REMOTE ??= "origin" + python () { from oe.cve_check import extend_cve_status extend_cve_status(d) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index 0f1f9281ad3..d2a3155b2b1 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -752,6 +752,16 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): return bb_objset + def import_layers_objset(self): + deploy_dir_spdx = Path(self.d.getVar("DEPLOY_DIR_SPDX")) + layers_objset = load_jsonld( + self.d, deploy_dir_spdx / "layers.spdx.json", required=True + ) + self.doc.import_.extend(layers_objset.doc.import_) + self.update(layers_objset.objects) + + return layers_objset + def import_bitbake_build(self): def find_bitbake_build(objset): return objset.find_filter( @@ -793,6 +803,8 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): [build], ) + layers = self.import_layers_objset() + if self.d.getVar("SPDX_INCLUDE_BUILD_VARIABLES") == "1": for varname in sorted(self.d.keys()): if varname.startswith("__"): diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 1821dd7de4a..0a8f4b0e1a0 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1220,6 +1220,80 @@ def create_package_spdx(d): oe.sbom30.write_recipe_jsonld_doc(d, common_objset, "common-package", deploydir) +def git_remote_has_rev(path, remote, rev): + try: + branchlist, _ = bb.process.run(f"git branch --remotes --list '{remote}/*' --contains {rev}", cwd=path) + except bb.process.ExecutionError: + return False + return len(branchlist) > 0 + +def get_layer_downloadLocation(d, l_path, l_name, l_branch, l_rev, l_ismodified): + from urllib.parse import urlparse, urlunparse + + repo = oe.buildcfg.get_metadata_git_toplevel(l_path) + + if not repo or not l_name or l_ismodified: + return "NOASSERTION" + + remoteName = d.getVar(f"SPDX_LAYER_REMOTE_{l_name}") or d.getVar(f"SPDX_LAYER_REMOTE") + if not remoteName: + return "NOASSERTION" + + # parse and reconstruct the url to drop potentially present username and password + remoteUrl = urlparse(oe.buildcfg.get_metadata_git_remote_url(repo, remoteName)) + if not remoteUrl or remoteUrl.scheme not in ["https", "http", "git", "ssh"]: + return "NOASSERTION" + remoteUrl = urlunparse(remoteUrl) + + # revision = d.getVar(f"SPDX_LAYER_CLAIM_REV_UNCHECKED_{l_name}") + # if not revision: + revision = l_rev + if revision == "<unknown>" or not git_remote_has_rev(repo, remoteName, revision): + return "NOASSERTION" + + relpath = os.path.relpath(l_path, repo) + + return f"git+{remoteUrl}@{revision}{'#' + relpath if relpath else ''}" + + +def write_layers_spdx(d): + d.setVar("PN", "layers") + d.setVar("BB_TASKHASH", "layers") + oe.spdx_common.load_spdx_license_data(d) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + + layer_objset = oe.sbom30.ObjectSet.new_objset(d, "layers") + + layers = oe.buildcfg.get_layer_revisions(d) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + downloadLocation = get_layer_downloadLocation(d, l_path, l_name, l_branch, l_rev, l_ismodified) + + + if oe.utils.vartrue("SPDX_REQUIRE_LAYER_ASSERTION", True, False, d) and downloadLocation == "NOASSERTION": + bb.fatal(f"Layer {l_name} does not have a valid downloadLocation") + layer = oe.spdx30.software_Package( + _id=layer_objset.new_spdxid("layer", l_name), + creationInfo=layer_objset.doc.creationInfo, + name=l_name, + software_packageVersion=l_rev, + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_downloadLocation=downloadLocation, + software_sourceInfo=json.dumps( + { + "branch": l_branch, + "is-modified": (len(l_ismodified) > 0), + }, + separators=(",", ":"), + ), + ) + + layer_objset.add_root(layer) + layer_objset.set_element_alias(layer) + + oe.sbom30.write_jsonld_doc(d, layer_objset, deploy_dir_spdx / "layers.spdx.json") + + + def write_bitbake_spdx(d): # Set PN to "bitbake" so that SPDX IDs can be generated d.setVar("PN", "bitbake") -- 2.54.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 2/2] spdx: link files to the meta-layer providing them 2026-09-14 11:44 [PATCH 0/2] Add meta layer information to spdx Daniel Wagenknecht 2026-09-14 11:44 ` [PATCH 1/2] spdx: add entries for meta-layers Daniel Wagenknecht @ 2026-09-14 11:44 ` Daniel Wagenknecht 2026-09-15 8:02 ` [OE-core] [PATCH 0/2] Add meta layer information to spdx Antonin Godard 2 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-09-14 11:44 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht Many recipes ship files that originate from a meta-layer. Link each file added via the `file` protocol to the spdx entry of the meta-layer containing that file. Some files might not originate from a meta-layer (`file` protocol supports absolute paths), fail the build if SPDX_REQUIRE_LAYER_ASSERTION is set to true. Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- meta/lib/oe/spdx30_tasks.py | 44 +++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 0a8f4b0e1a0..f782ea1cadf 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -497,6 +497,7 @@ def add_download_files(d, objset): primary_purpose = oe.spdx30.software_SoftwarePurpose.source if fd.type == "file": + (layer_path, layer_objset) = get_layer_info(d, fd.localpath) if os.path.isdir(fd.localpath): walk_idx = 1 for root, dirs, files in os.walk(fd.localpath, onerror=walk_error): @@ -512,23 +513,36 @@ def add_download_files(d, objset): objset.new_spdxid( "source", str(download_idx + 1), str(walk_idx) ), - os.path.join( - file_name, os.path.relpath(f_path, fd.localpath) - ), + os.path.relpath(f_path, layer_path), f_path, purposes=[primary_purpose], ) + if layer_objset is not None: + objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(layer_objset)], + oe.spdx30.RelationshipType.contains, + oe.spdx30.LifecycleScopeType.build, + [file], + ) + inputs.add(file) walk_idx += 1 else: file = objset.new_file( objset.new_spdxid("source", str(download_idx + 1)), - file_name, + os.path.relpath(fd.localpath, layer_path), fd.localpath, purposes=[primary_purpose], ) + if layer_objset is not None: + objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(layer_objset)], + oe.spdx30.RelationshipType.contains, + oe.spdx30.LifecycleScopeType.build, + [file], + ) inputs.add(file) else: @@ -803,6 +817,28 @@ def load_recipe_spdx(d): oe.spdx30.software_Package, ) +def get_layer_info(d, path): + layer = None + path = Path(path) + + layers = oe.buildcfg.get_layer_revisions(d) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + l_path = Path(l_path) + if l_path in path.parents: + layer = l_name + break + + if not l_path: + if oe.utils.vartrue("SPDX_REQUIRE_LAYER_ASSERTION", True, False, d): + bb.fatal(f"Path {path} does not originate from a meta-layer!") + return ("/", None) + + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + objset = oe.sbom30.load_jsonld(d, deploy_dir_spdx / "layers.spdx.json", required=True) + spdx_obj = objset.find_root(oe.spdx30.software_Package, name=l_name) + + return (l_path, spdx_obj) + def create_spdx(d): def set_var_field(var, obj, name, package=None): -- 2.54.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [OE-core] [PATCH 0/2] Add meta layer information to spdx 2026-09-14 11:44 [PATCH 0/2] Add meta layer information to spdx Daniel Wagenknecht 2026-09-14 11:44 ` [PATCH 1/2] spdx: add entries for meta-layers Daniel Wagenknecht 2026-09-14 11:44 ` [PATCH 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht @ 2026-09-15 8:02 ` Antonin Godard 2026-09-15 13:04 ` [PATCH v2 " Daniel Wagenknecht ` (2 more replies) 2 siblings, 3 replies; 16+ messages in thread From: Antonin Godard @ 2026-09-15 8:02 UTC (permalink / raw) To: dwagenknecht, openembedded-core On Mon Sep 14, 2026 at 1:44 PM CEST, Daniel Wagenknecht via lists.openembedded.org wrote: > Hello, > > as discussed before here > https://lists.openembedded.org/g/openembedded-core/message/226131 > and here > https://lists.openembedded.org/g/openembedded-core/message/225774 > > it would be good to include information about the layers used in a build in the > generated spdx documents. > > My primary goal is to track where all the configuration files added via file > protocol in `SRC_URI` originate from, but the approach could be extended to > track recipes, classes, machine config etc. > > These patches are meant as a basis to discuss the approach and should be > considered draft status. > > The approach is to generate a `layers.spdx.json` on each build invocation > that lists each layer used in the build as a `softwarePackage` with version > and source information as well as - if can be reasonably deducted - the > downloadLocation. Each source file added via file protocol is then linked to the > containing meta layer via a `contains` relationship. > > Open Questions: > - is the overall architecture reasonable/correct? > - referring to layers is inconsistent across tooling. E.g. the core layer is > either named `core` as defined by `BBFILE_COLLECTIONS` or `meta` as that is > the layer directory in the openembedded-core repository. > I chose the latter since that is what the utility function > `oe.buildcfg.get_layer_revisions` returns. The package URL (PURL) > implementation uses the other naming scheme though. Any thoughts on which > should be chosen and why? > - I do not understand the alias linking yet. With the current implementation > I see many hash mismatches like > WARNING: gcc-runtime-15.3.0-r0 do_create_spdx: Element http://spdx.org/spdxdocs/layers-7ad0d8d7-57f9-5c3c-9411-49227dd50610/layers/layer/meta > has alias http://spdx.org/spdxdocs/UNIHASH-7ad0d8d7-57f9-5c3c-9411-49227dd50610/UNIHASH/layer/meta, but it should have prefix > http://spdxdocs.org/openembedded-alias/by-doc-hash/e80f1d61f4749e663e80a4dfb25ebb1127b5f50600d3065de7ed91030942a1ef/ > > I've currently got some time available to work on this to get a proper solution > integrated into openembedded-core. This series is causing the following issue on our Autobuilder: ERROR: No SPDX document named /srv/pokybuild/yocto-worker/qemuarm-alt/build/build/tmp/deploy/spdx/3.0.1/bitbake.spdx.json found See full log here: https://autobuilder.yoctoproject.org/valkyrie/#/builders/74/builds/4598 Can you look into it? Thanks, Antonin ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 0/2] Add meta layer information to spdx 2026-09-15 8:02 ` [OE-core] [PATCH 0/2] Add meta layer information to spdx Antonin Godard @ 2026-09-15 13:04 ` Daniel Wagenknecht 2026-09-16 7:08 ` [OE-core] " Antonin Godard 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht 2026-09-16 5:39 ` [PATCH v2 1/2] spdx: add entries for meta-layers Daniel Wagenknecht 2026-09-16 5:39 ` [PATCH v2 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht 2 siblings, 2 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-09-15 13:04 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht v2 of the patch series to fix the build failures on the autobuilder reported by Antonin Godard. The questions from the initial patch series are still open and additionally the final document seems to be deployed from sstate cache even if inputs (layer revisions) change. Please continue to consider this as a draft for discussion only! v2: - fix build failures on clean build (layers.spdx.json created before bitbake.spdx.json) - rebase onto current master branch - remove commented out parts for overwriting the layer commit id Daniel Wagenknecht (2): spdx: add entries for meta-layers spdx: link files to the meta-layer providing them meta/classes/create-spdx-3.0.bbclass | 1 + meta/classes/spdx-common.bbclass | 3 + meta/lib/oe/sbom30.py | 12 +++ meta/lib/oe/spdx30_tasks.py | 116 ++++++++++++++++++++++++++- 4 files changed, 128 insertions(+), 4 deletions(-) -- 2.54.0 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OE-core] [PATCH v2 0/2] Add meta layer information to spdx 2026-09-15 13:04 ` [PATCH v2 " Daniel Wagenknecht @ 2026-09-16 7:08 ` Antonin Godard 2026-09-16 8:45 ` Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht 1 sibling, 1 reply; 16+ messages in thread From: Antonin Godard @ 2026-09-16 7:08 UTC (permalink / raw) To: dwagenknecht, openembedded-core Hi, On Tue Sep 15, 2026 at 3:04 PM CEST, Daniel Wagenknecht via lists.openembedded.org wrote: > Please continue to consider this as a draft for discussion only! It is useful to mark these kinds of series with "RFC", if you don't consider it ready for merging. I must admit though I skipped over your mention of this being a draft in your cover letter. Antonin ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OE-core] [PATCH v2 0/2] Add meta layer information to spdx 2026-09-16 7:08 ` [OE-core] " Antonin Godard @ 2026-09-16 8:45 ` Daniel Wagenknecht 0 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-09-16 8:45 UTC (permalink / raw) To: antonin.godard; +Cc: openembedded-core Hello Antonin, thanks for the reply. I agree, some more prominent mark of this being a draft would have been helpful. OE does not have any defined prefixes / markers right? At least I could not find any in the OE contributor guide. To me RFC is more of a formal process, but I'll include a [DRAFT] marker for future revisions of this patch series as long as they are not ready for merging yet. Daniel On Wed, 2026-09-16 at 09:08 +0200, Antonin Godard via lists.openembedded.org wrote: > Hi, > > On Tue Sep 15, 2026 at 3:04 PM CEST, Daniel Wagenknecht via > lists.openembedded.org wrote: > > Please continue to consider this as a draft for discussion only! > > It is useful to mark these kinds of series with "RFC", if you don't > consider it > ready for merging. I must admit though I skipped over your mention of > this being > a draft in your cover letter. > > Antonin > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#245930): > https://lists.openembedded.org/g/openembedded-core/message/245930 > Mute This Topic: https://lists.openembedded.org/mt/121260101/5939739 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: > https://lists.openembedded.org/g/openembedded-core/unsub [ > dwagenknecht@emlix.com] > -=-=-=-=-=-=-=-=-=-=-=- ^ permalink raw reply [flat|nested] 16+ messages in thread
* [DRAFT PATCH v3 0/2] Add meta layer information to spdx 2026-09-15 13:04 ` [PATCH v2 " Daniel Wagenknecht 2026-09-16 7:08 ` [OE-core] " Antonin Godard @ 2026-11-06 11:54 ` Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers Daniel Wagenknecht ` (3 more replies) 1 sibling, 4 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-11-06 11:54 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht v3 of the patch series looking for feedback on the overall architecture. I worked out the problem in alias linking, so a major blocker is out of the way. Please continue to consider this as a draft for discussion only! During discussion on IRC some questions were raised especially regarding the interaction with sstate caching. The layer information is only part of `create_spdx`, not `create_recipe_spdx`. It is thus only added to the final image spdx documents, not to the individual recipe or packagage spdx document. Images and their spdx documents are always rebuild, so with this approach no additional rebuilds occur even if a layers revision changes. My still open questions are: * is the overall architecture reasonable/correct? * referring to layers is inconsistent across tooling. E.g. the core layer is either named `core` as defined by `BBFILE_COLLECTIONS` or `meta` as that is the layer directory in the openembedded-core repository. I chose the latter since that is what the utility function `oe.buildcfg.get_layer_revisions` returns. The package URL (PURL) implementation uses the other naming scheme though. Any thoughts on which should be chosen and why? v3: * rebase onto current master branch * fixed the alias linking v2: * fix build failures on clean build (layers.spdx.json created before bitbake.spdx.json) * rebase onto current master branch * remove commented out parts for overwriting the layer commit id Daniel Wagenknecht (2): spdx: add entries for meta-layers spdx: link files to the meta-layer providing them meta/classes/create-spdx-3.0.bbclass | 1 + meta/classes/spdx-common.bbclass | 3 + meta/lib/oe/sbom30.py | 12 +++ meta/lib/oe/spdx30_tasks.py | 118 ++++++++++++++++++++++++++- 4 files changed, 130 insertions(+), 4 deletions(-) -- 2.54.0 ^ permalink raw reply [flat|nested] 16+ messages in thread
* [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht @ 2026-11-06 11:54 ` Daniel Wagenknecht 2026-11-06 12:20 ` Paul Barker 2026-11-06 11:54 ` [DRAFT PATCH v3 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht ` (2 subsequent siblings) 3 siblings, 1 reply; 16+ messages in thread From: Daniel Wagenknecht @ 2026-11-06 11:54 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht This adds support for determining the revision and downloadLocation of meta-layers and adds them to the OE generated SPDX document. This downloadLocation will result in a NOASSERTION entry if * the meta-layer is not in a git repo * the git repo is dirty * the git remote does not have a valid URL * the git remote does not contain the used revision as far as can be eluded from the local repo Variables to configure the behavior: * SPDX_LAYER_REMOTE (default: origin): select the name of the git remote to use for layers. Set to an empty string to always return NOASSERTION * SPDX_LAYER_REMOTE_layername: override SPDX_LAYER_REMOTE for a specific layer * SPDX_REQUIRE_LAYER_ASSERTION: set to true to fail a build if determining downloadLocation fails for any layer Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- meta/classes/create-spdx-3.0.bbclass | 1 + meta/classes/spdx-common.bbclass | 3 ++ meta/lib/oe/sbom30.py | 12 +++++ meta/lib/oe/spdx30_tasks.py | 74 ++++++++++++++++++++++++++++ 4 files changed, 90 insertions(+) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 6d47aa13016..507486cfd19 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -288,6 +288,7 @@ python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() oe.spdx30_tasks.write_bitbake_spdx(d) + oe.spdx30_tasks.write_layers_spdx(d) } addhandler spdx30_build_started_handler diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 40701730a69..b64537dfb48 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -89,6 +89,9 @@ SPDX_FILE_EXCLUDE_PATTERNS[doc] = "Space-separated list of Python regular \ (no filtering). Example: \ SPDX_FILE_EXCLUDE_PATTERNS = '\\.patch$ \\.diff$ /test/ \\.pyc$ \\.o$'" +SPDX_REQUIRE_LAYER_ASSERTION ??= "0" +SPDX_LAYER_REMOTE ??= "origin" + python () { from oe.cve_check import extend_cve_status extend_cve_status(d) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index e279cfc82bf..bde3ff1f553 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -816,6 +816,16 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): return bb_objset + def import_layers_objset(self): + deploy_dir_spdx = Path(self.d.getVar("DEPLOY_DIR_SPDX")) + layers_objset = load_jsonld( + self.d, deploy_dir_spdx / "layers.spdx.json", required=True + ) + self.doc.import_.extend(layers_objset.doc.import_) + self.update(layers_objset.objects) + + return layers_objset + def import_bitbake_build(self): def find_bitbake_build(objset): return objset.find_filter( @@ -857,6 +867,8 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): [build], ) + layers = self.import_layers_objset() + if self.d.getVar("SPDX_INCLUDE_BUILD_VARIABLES") == "1": for varname in sorted(self.d.keys()): if varname.startswith("__"): diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 646ec7d0743..31d964d8ce8 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1232,6 +1232,80 @@ def create_package_spdx(d): oe.sbom30.write_recipe_jsonld_doc(d, common_objset, "common-package", deploydir) +def git_remote_has_rev(path, remote, rev): + try: + branchlist, _ = bb.process.run(f"git branch --remotes --list '{remote}/*' --contains {rev}", cwd=path) + except bb.process.ExecutionError: + return False + return len(branchlist) > 0 + +def get_layer_downloadLocation(d, l_path, l_name, l_branch, l_rev, l_ismodified): + from urllib.parse import urlparse, urlunparse + + repo = oe.buildcfg.get_metadata_git_toplevel(l_path) + + if not repo or not l_name or l_ismodified: + return "NOASSERTION" + + remoteName = d.getVar(f"SPDX_LAYER_REMOTE_{l_name}") or d.getVar(f"SPDX_LAYER_REMOTE") + if not remoteName: + return "NOASSERTION" + + # parse and reconstruct the url to drop potentially present username and password + remoteUrl = urlparse(oe.buildcfg.get_metadata_git_remote_url(repo, remoteName)) + if not remoteUrl or remoteUrl.scheme not in ["https", "http", "git", "ssh"]: + return "NOASSERTION" + remoteUrl = urlunparse(remoteUrl) + + revision = l_rev + if revision == "<unknown>" or not git_remote_has_rev(repo, remoteName, revision): + return "NOASSERTION" + + relpath = os.path.relpath(l_path, repo) + + return f"git+{remoteUrl}@{revision}{'#' + relpath if relpath else ''}" + + +def write_layers_spdx(d): + d.setVar("PN", "layers") + d.setVar("BB_TASKHASH", "layers") + oe.spdx_common.load_spdx_license_data(d) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + + layer_objset = oe.sbom30.ObjectSet.new_objset(d, "layers") + + layers = oe.buildcfg.get_layer_revisions(d) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + downloadLocation = get_layer_downloadLocation(d, l_path, l_name, l_branch, l_rev, l_ismodified) + + + if oe.utils.vartrue("SPDX_REQUIRE_LAYER_ASSERTION", True, False, d) and downloadLocation == "NOASSERTION": + bb.fatal(f"Layer {l_name} does not have a valid downloadLocation") + layer = oe.spdx30.software_Package( + _id=layer_objset.new_spdxid("layer", l_name), + creationInfo=layer_objset.doc.creationInfo, + name=l_name, + software_packageVersion=l_rev, + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_downloadLocation=downloadLocation, + software_sourceInfo=json.dumps( + { + "branch": l_branch, + "is-modified": (len(l_ismodified) > 0), + }, + separators=(",", ":"), + ), + ) + + layer_objset.add_root(layer) + + for obj in layer_objset.foreach_type(oe.spdx30.Element): + obj.extension.append(oe.sbom30.OEIdAliasExtension()) + + oe.sbom30.write_jsonld_doc(d, layer_objset, deploy_dir_spdx / "layers.spdx.json") + + + def write_bitbake_spdx(d): # Set PN to "bitbake" so that SPDX IDs can be generated d.setVar("PN", "bitbake") -- 2.54.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers 2026-11-06 11:54 ` [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers Daniel Wagenknecht @ 2026-11-06 12:20 ` Paul Barker 2026-11-06 13:37 ` Daniel Wagenknecht 0 siblings, 1 reply; 16+ messages in thread From: Paul Barker @ 2026-11-06 12:20 UTC (permalink / raw) To: Daniel Wagenknecht, openembedded-core On Fri Nov 6, 2026 at 11:54 AM GMT, Daniel Wagenknecht wrote: > This adds support for determining the revision and downloadLocation of > meta-layers and adds them to the OE generated SPDX document. > > This downloadLocation will result in a NOASSERTION entry if > * the meta-layer is not in a git repo > * the git repo is dirty > * the git remote does not have a valid URL > * the git remote does not contain the used revision as far as can be > eluded from the local repo > > Variables to configure the behavior: > * SPDX_LAYER_REMOTE (default: origin): select the name of the git remote > to use for layers. Set to an empty string to always return NOASSERTION > * SPDX_LAYER_REMOTE_layername: override SPDX_LAYER_REMOTE for a specific > layer > * SPDX_REQUIRE_LAYER_ASSERTION: set to true to fail a build if > determining downloadLocation fails for any layer > > Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> Hi Daniel, I don't think downloadLocation should be populated by default, it may refer to an internal network location for an organisations' private repositories and leaking those URLs may not be desired. This probably needs to be opt-in. Best regards, -- Paul Barker ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers 2026-11-06 12:20 ` Paul Barker @ 2026-11-06 13:37 ` Daniel Wagenknecht 0 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-11-06 13:37 UTC (permalink / raw) To: Paul Barker, openembedded-core Hello Paul, On Fri, 2026-11-06 at 12:20 +0000, Paul Barker wrote: > I don't think downloadLocation should be populated by default, it may > refer to an internal network location for an organisations' private > repositories I already wondered about this when first implementing the approach. By SPDX spec it is OK to have a private URL here (SPDX 2 documentation explicitely noted private URLs being allowed for this property, in SPDX 3 I did not find such a clear statement but also nothing contradicting it) > and leaking those URLs may not be desired. by being implicit (parsed from the current repo checkout) it's different to private servers explicitely mentioned in SRC_URI, so I agree it may not be desired and considered an information leak. > > This probably needs to be opt-in. That would be OK with me, I'll make it opt-in in a next revision. Best Wishes Daniel ^ permalink raw reply [flat|nested] 16+ messages in thread
* [DRAFT PATCH v3 2/2] spdx: link files to the meta-layer providing them 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers Daniel Wagenknecht @ 2026-11-06 11:54 ` Daniel Wagenknecht 2026-11-06 12:23 ` [DRAFT PATCH v3 0/2] Add meta layer information to spdx Paul Barker 2026-11-10 23:32 ` [OE-core] " Joshua Watt 3 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-11-06 11:54 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht Many recipes ship files that originate from a meta-layer. Link each file added via the `file` protocol to the spdx entry of the meta-layer containing that file. Some files might not originate from a meta-layer (`file` protocol supports absolute paths), fail the build if SPDX_REQUIRE_LAYER_ASSERTION is set to true. Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- meta/lib/oe/spdx30_tasks.py | 44 +++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 31d964d8ce8..4bdbbcfab97 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -477,6 +477,7 @@ def add_download_files(d, objset): primary_purpose = oe.spdx30.software_SoftwarePurpose.source if fd.type == "file": + (layer_path, layer_objset) = get_layer_info(d, fd.localpath) if os.path.isdir(fd.localpath): walk_idx = 1 for root, dirs, files in os.walk(fd.localpath, onerror=walk_error): @@ -492,23 +493,36 @@ def add_download_files(d, objset): objset.new_spdxid( "source", str(download_idx + 1), str(walk_idx) ), - os.path.join( - file_name, os.path.relpath(f_path, fd.localpath) - ), + os.path.relpath(f_path, layer_path), f_path, purposes=[primary_purpose], ) + if layer_objset is not None: + objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(layer_objset)], + oe.spdx30.RelationshipType.contains, + oe.spdx30.LifecycleScopeType.build, + [file], + ) + inputs.add(file) walk_idx += 1 else: file = objset.new_file( objset.new_spdxid("source", str(download_idx + 1)), - file_name, + os.path.relpath(fd.localpath, layer_path), fd.localpath, purposes=[primary_purpose], ) + if layer_objset is not None: + objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(layer_objset)], + oe.spdx30.RelationshipType.contains, + oe.spdx30.LifecycleScopeType.build, + [file], + ) inputs.add(file) else: @@ -801,6 +815,28 @@ def load_recipe_spdx(d): oe.spdx30.software_Package, ) +def get_layer_info(d, path): + layer = None + path = Path(path) + + layers = oe.buildcfg.get_layer_revisions(d) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + l_path = Path(l_path) + if l_path in path.parents: + layer = l_name + break + + if not l_path: + if oe.utils.vartrue("SPDX_REQUIRE_LAYER_ASSERTION", True, False, d): + bb.fatal(f"Path {path} does not originate from a meta-layer!") + return ("/", None) + + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + objset = oe.sbom30.load_jsonld(d, deploy_dir_spdx / "layers.spdx.json", required=True) + spdx_obj = objset.find_root(oe.spdx30.software_Package, name=l_name) + + return (l_path, spdx_obj) + def create_spdx(d): def set_var_field(var, obj, name, package=None): -- 2.54.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [DRAFT PATCH v3 0/2] Add meta layer information to spdx 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht @ 2026-11-06 12:23 ` Paul Barker 2026-11-10 23:32 ` [OE-core] " Joshua Watt 3 siblings, 0 replies; 16+ messages in thread From: Paul Barker @ 2026-11-06 12:23 UTC (permalink / raw) To: Daniel Wagenknecht, openembedded-core On Fri Nov 6, 2026 at 11:54 AM GMT, Daniel Wagenknecht wrote: > v3 of the patch series looking for feedback on the overall architecture. I > worked out the problem in alias linking, so a major blocker is out of the way. > > Please continue to consider this as a draft for discussion only! > > During discussion on IRC some questions were raised especially regarding the > interaction with sstate caching. > The layer information is only part of `create_spdx`, not `create_recipe_spdx`. > It is thus only added to the final image spdx documents, not to the individual > recipe or packagage spdx document. Images and their spdx documents are always > rebuild, so with this approach no additional rebuilds occur even if a layers > revision changes. > > My still open questions are: > * is the overall architecture reasonable/correct? > * referring to layers is inconsistent across tooling. E.g. the core layer is > either named `core` as defined by `BBFILE_COLLECTIONS` or `meta` as that is > the layer directory in the openembedded-core repository. > I chose the latter since that is what the utility function > `oe.buildcfg.get_layer_revisions` returns. The package URL (PURL) > implementation uses the other naming scheme though. Any thoughts on which > should be chosen and why? Hi Daniel, It's worth thinking about how this will interact with the proposed changes to how layer.conf will work [1] [2]. That should hopefully improve consistency in how layers are referred to across tools, but may also impact other parts of these patches. [1]: https://lists.openembedded.org/g/openembedded-architecture/topic/layers_and_layer_conf_time/121884200 [2]: https://lists.openembedded.org/g/openembedded-architecture/topic/layer_changes_poc_and_rfc/122113105 Best regards, -- Paul Barker ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [OE-core] [DRAFT PATCH v3 0/2] Add meta layer information to spdx 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht ` (2 preceding siblings ...) 2026-11-06 12:23 ` [DRAFT PATCH v3 0/2] Add meta layer information to spdx Paul Barker @ 2026-11-10 23:32 ` Joshua Watt 3 siblings, 0 replies; 16+ messages in thread From: Joshua Watt @ 2026-11-10 23:32 UTC (permalink / raw) To: dwagenknecht; +Cc: openembedded-core [-- Attachment #1: Type: text/plain, Size: 3362 bytes --] On Fri, Nov 6, 2026 at 4:55 AM Daniel Wagenknecht via openembedded-core <dwagenknecht=emlix.com@lists.openembedded.org> wrote: > v3 of the patch series looking for feedback on the overall architecture. I > worked out the problem in alias linking, so a major blocker is out of the > way. > > Please continue to consider this as a draft for discussion only! > > During discussion on IRC some questions were raised especially regarding > the > interaction with sstate caching. > The layer information is only part of `create_spdx`, not > `create_recipe_spdx`. > It is thus only added to the final image spdx documents, not to the > individual > recipe or packagage spdx document. Images and their spdx documents are > always > rebuild, so with this approach no additional rebuilds occur even if a > layers > revision changes. > > My still open questions are: > * is the overall architecture reasonable/correct? > Overall, this is approach seems reasonable to me. It's similar to how I've thought about doing it in the past. > * referring to layers is inconsistent across tooling. E.g. the core layer > is > either named `core` as defined by `BBFILE_COLLECTIONS` or `meta` as that > is > the layer directory in the openembedded-core repository. > I chose the latter since that is what the utility function > `oe.buildcfg.get_layer_revisions` returns. The package URL (PURL) > implementation uses the other naming scheme though. Any thoughts on which > should be chosen and why? This is very tricky. In an ideal world, we could use a "git" PURL as the official identifier: https://github.com/package-url/purl-spec/blob/main/docs/types/definitions/git-definition.md However, I'm not sure that we can reliably use the actual git URL of the layer to construct the PURL, since mirroring, caching, etc. are pretty common. Maybe there should be a per-layer variable that specifies the PURL(s) to use for the layer and would thus survive mirroring, caching, etc.? It would also present the option for a PURL to be something other than just a "git" type (e.g. it could be "github", "bitbucket", or other future PURL types). > > > v3: > * rebase onto current master branch > * fixed the alias linking > > v2: > * fix build failures on clean build (layers.spdx.json created before > bitbake.spdx.json) > * rebase onto current master branch > * remove commented out parts for overwriting the layer commit id > > Daniel Wagenknecht (2): > spdx: add entries for meta-layers > spdx: link files to the meta-layer providing them > > meta/classes/create-spdx-3.0.bbclass | 1 + > meta/classes/spdx-common.bbclass | 3 + > meta/lib/oe/sbom30.py | 12 +++ > meta/lib/oe/spdx30_tasks.py | 118 ++++++++++++++++++++++++++- > 4 files changed, 130 insertions(+), 4 deletions(-) > > -- > 2.54.0 > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > View/Reply Online (#249634): > https://lists.openembedded.org/g/openembedded-core/message/249634 > Mute This Topic: https://lists.openembedded.org/mt/122125585/3616693 > Group Owner: openembedded-core+owner@lists.openembedded.org > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ > JPEWhacker@gmail.com] > -=-=-=-=-=-=-=-=-=-=-=- > > [-- Attachment #2: Type: text/html, Size: 4923 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 1/2] spdx: add entries for meta-layers 2026-09-15 8:02 ` [OE-core] [PATCH 0/2] Add meta layer information to spdx Antonin Godard 2026-09-15 13:04 ` [PATCH v2 " Daniel Wagenknecht @ 2026-09-16 5:39 ` Daniel Wagenknecht 2026-09-16 5:39 ` [PATCH v2 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht 2 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-09-16 5:39 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht This adds support for determining the revision and downloadLocation of meta-layers and adds them to the OE generated SPDX document. This downloadLocation will result in a NOASSERTION entry if - the meta-layer is not in a git repo - the git repo is dirty - the git remote does not have a valid URL - the git remote does not contain the used revision as far as can be eluded from the local repo Variables to configure the behavior: - SPDX_LAYER_REMOTE (default: origin): select the name of the git remote to use for layers. Set to an empty string to always return NOASSERTION - SPDX_LAYER_REMOTE_layername: override SPDX_LAYER_REMOTE for a specific layer - SPDX_REQUIRE_LAYER_ASSERTION: set to true to fail a build if determining downloadLocation fails for any layer Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- meta/classes/create-spdx-3.0.bbclass | 1 + meta/classes/spdx-common.bbclass | 3 ++ meta/lib/oe/sbom30.py | 12 +++++ meta/lib/oe/spdx30_tasks.py | 72 ++++++++++++++++++++++++++++ 4 files changed, 88 insertions(+) diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass index 56fd01fd533..9e4f737f85d 100644 --- a/meta/classes/create-spdx-3.0.bbclass +++ b/meta/classes/create-spdx-3.0.bbclass @@ -286,6 +286,7 @@ python spdx30_build_started_handler () { import oe.spdx30_tasks d = e.data.createCopy() oe.spdx30_tasks.write_bitbake_spdx(d) + oe.spdx30_tasks.write_layers_spdx(d) } addhandler spdx30_build_started_handler diff --git a/meta/classes/spdx-common.bbclass b/meta/classes/spdx-common.bbclass index 40701730a69..b64537dfb48 100644 --- a/meta/classes/spdx-common.bbclass +++ b/meta/classes/spdx-common.bbclass @@ -89,6 +89,9 @@ SPDX_FILE_EXCLUDE_PATTERNS[doc] = "Space-separated list of Python regular \ (no filtering). Example: \ SPDX_FILE_EXCLUDE_PATTERNS = '\\.patch$ \\.diff$ /test/ \\.pyc$ \\.o$'" +SPDX_REQUIRE_LAYER_ASSERTION ??= "0" +SPDX_LAYER_REMOTE ??= "origin" + python () { from oe.cve_check import extend_cve_status extend_cve_status(d) diff --git a/meta/lib/oe/sbom30.py b/meta/lib/oe/sbom30.py index e02382c3cc7..46dae429ffb 100644 --- a/meta/lib/oe/sbom30.py +++ b/meta/lib/oe/sbom30.py @@ -762,6 +762,16 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): return bb_objset + def import_layers_objset(self): + deploy_dir_spdx = Path(self.d.getVar("DEPLOY_DIR_SPDX")) + layers_objset = load_jsonld( + self.d, deploy_dir_spdx / "layers.spdx.json", required=True + ) + self.doc.import_.extend(layers_objset.doc.import_) + self.update(layers_objset.objects) + + return layers_objset + def import_bitbake_build(self): def find_bitbake_build(objset): return objset.find_filter( @@ -803,6 +813,8 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): [build], ) + layers = self.import_layers_objset() + if self.d.getVar("SPDX_INCLUDE_BUILD_VARIABLES") == "1": for varname in sorted(self.d.keys()): if varname.startswith("__"): diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 9978ae731ca..9094a6cbd64 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -1205,6 +1205,78 @@ def create_package_spdx(d): oe.sbom30.write_recipe_jsonld_doc(d, common_objset, "common-package", deploydir) +def git_remote_has_rev(path, remote, rev): + try: + branchlist, _ = bb.process.run(f"git branch --remotes --list '{remote}/*' --contains {rev}", cwd=path) + except bb.process.ExecutionError: + return False + return len(branchlist) > 0 + +def get_layer_downloadLocation(d, l_path, l_name, l_branch, l_rev, l_ismodified): + from urllib.parse import urlparse, urlunparse + + repo = oe.buildcfg.get_metadata_git_toplevel(l_path) + + if not repo or not l_name or l_ismodified: + return "NOASSERTION" + + remoteName = d.getVar(f"SPDX_LAYER_REMOTE_{l_name}") or d.getVar(f"SPDX_LAYER_REMOTE") + if not remoteName: + return "NOASSERTION" + + # parse and reconstruct the url to drop potentially present username and password + remoteUrl = urlparse(oe.buildcfg.get_metadata_git_remote_url(repo, remoteName)) + if not remoteUrl or remoteUrl.scheme not in ["https", "http", "git", "ssh"]: + return "NOASSERTION" + remoteUrl = urlunparse(remoteUrl) + + revision = l_rev + if revision == "<unknown>" or not git_remote_has_rev(repo, remoteName, revision): + return "NOASSERTION" + + relpath = os.path.relpath(l_path, repo) + + return f"git+{remoteUrl}@{revision}{'#' + relpath if relpath else ''}" + + +def write_layers_spdx(d): + d.setVar("PN", "layers") + d.setVar("BB_TASKHASH", "layers") + oe.spdx_common.load_spdx_license_data(d) + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + + layer_objset = oe.sbom30.ObjectSet.new_objset(d, "layers") + + layers = oe.buildcfg.get_layer_revisions(d) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + downloadLocation = get_layer_downloadLocation(d, l_path, l_name, l_branch, l_rev, l_ismodified) + + + if oe.utils.vartrue("SPDX_REQUIRE_LAYER_ASSERTION", True, False, d) and downloadLocation == "NOASSERTION": + bb.fatal(f"Layer {l_name} does not have a valid downloadLocation") + layer = oe.spdx30.software_Package( + _id=layer_objset.new_spdxid("layer", l_name), + creationInfo=layer_objset.doc.creationInfo, + name=l_name, + software_packageVersion=l_rev, + software_primaryPurpose=oe.spdx30.software_SoftwarePurpose.specification, + software_downloadLocation=downloadLocation, + software_sourceInfo=json.dumps( + { + "branch": l_branch, + "is-modified": (len(l_ismodified) > 0), + }, + separators=(",", ":"), + ), + ) + + layer_objset.add_root(layer) + layer_objset.set_element_alias(layer) + + oe.sbom30.write_jsonld_doc(d, layer_objset, deploy_dir_spdx / "layers.spdx.json") + + + def write_bitbake_spdx(d): # Set PN to "bitbake" so that SPDX IDs can be generated d.setVar("PN", "bitbake") -- 2.54.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 2/2] spdx: link files to the meta-layer providing them 2026-09-15 8:02 ` [OE-core] [PATCH 0/2] Add meta layer information to spdx Antonin Godard 2026-09-15 13:04 ` [PATCH v2 " Daniel Wagenknecht 2026-09-16 5:39 ` [PATCH v2 1/2] spdx: add entries for meta-layers Daniel Wagenknecht @ 2026-09-16 5:39 ` Daniel Wagenknecht 2 siblings, 0 replies; 16+ messages in thread From: Daniel Wagenknecht @ 2026-09-16 5:39 UTC (permalink / raw) To: openembedded-core; +Cc: Daniel Wagenknecht Many recipes ship files that originate from a meta-layer. Link each file added via the `file` protocol to the spdx entry of the meta-layer containing that file. Some files might not originate from a meta-layer (`file` protocol supports absolute paths), fail the build if SPDX_REQUIRE_LAYER_ASSERTION is set to true. Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com> --- meta/lib/oe/spdx30_tasks.py | 44 +++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-) diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py index 9094a6cbd64..6b1308c5586 100644 --- a/meta/lib/oe/spdx30_tasks.py +++ b/meta/lib/oe/spdx30_tasks.py @@ -453,6 +453,7 @@ def add_download_files(d, objset): primary_purpose = oe.spdx30.software_SoftwarePurpose.source if fd.type == "file": + (layer_path, layer_objset) = get_layer_info(d, fd.localpath) if os.path.isdir(fd.localpath): walk_idx = 1 for root, dirs, files in os.walk(fd.localpath, onerror=walk_error): @@ -468,23 +469,36 @@ def add_download_files(d, objset): objset.new_spdxid( "source", str(download_idx + 1), str(walk_idx) ), - os.path.join( - file_name, os.path.relpath(f_path, fd.localpath) - ), + os.path.relpath(f_path, layer_path), f_path, purposes=[primary_purpose], ) + if layer_objset is not None: + objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(layer_objset)], + oe.spdx30.RelationshipType.contains, + oe.spdx30.LifecycleScopeType.build, + [file], + ) + inputs.add(file) walk_idx += 1 else: file = objset.new_file( objset.new_spdxid("source", str(download_idx + 1)), - file_name, + os.path.relpath(fd.localpath, layer_path), fd.localpath, purposes=[primary_purpose], ) + if layer_objset is not None: + objset.new_scoped_relationship( + [oe.sbom30.get_element_link_id(layer_objset)], + oe.spdx30.RelationshipType.contains, + oe.spdx30.LifecycleScopeType.build, + [file], + ) inputs.add(file) else: @@ -774,6 +788,28 @@ def load_recipe_spdx(d): oe.spdx30.software_Package, ) +def get_layer_info(d, path): + layer = None + path = Path(path) + + layers = oe.buildcfg.get_layer_revisions(d) + for (l_path, l_name, l_branch, l_rev, l_ismodified) in layers: + l_path = Path(l_path) + if l_path in path.parents: + layer = l_name + break + + if not l_path: + if oe.utils.vartrue("SPDX_REQUIRE_LAYER_ASSERTION", True, False, d): + bb.fatal(f"Path {path} does not originate from a meta-layer!") + return ("/", None) + + deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX")) + objset = oe.sbom30.load_jsonld(d, deploy_dir_spdx / "layers.spdx.json", required=True) + spdx_obj = objset.find_root(oe.spdx30.software_Package, name=l_name) + + return (l_path, spdx_obj) + def create_spdx(d): def set_var_field(var, obj, name, package=None): -- 2.54.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-11-10 23:32 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-09-14 11:44 [PATCH 0/2] Add meta layer information to spdx Daniel Wagenknecht 2026-09-14 11:44 ` [PATCH 1/2] spdx: add entries for meta-layers Daniel Wagenknecht 2026-09-14 11:44 ` [PATCH 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht 2026-09-15 8:02 ` [OE-core] [PATCH 0/2] Add meta layer information to spdx Antonin Godard 2026-09-15 13:04 ` [PATCH v2 " Daniel Wagenknecht 2026-09-16 7:08 ` [OE-core] " Antonin Godard 2026-09-16 8:45 ` Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 " Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 1/2] spdx: add entries for meta-layers Daniel Wagenknecht 2026-11-06 12:20 ` Paul Barker 2026-11-06 13:37 ` Daniel Wagenknecht 2026-11-06 11:54 ` [DRAFT PATCH v3 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht 2026-11-06 12:23 ` [DRAFT PATCH v3 0/2] Add meta layer information to spdx Paul Barker 2026-11-10 23:32 ` [OE-core] " Joshua Watt 2026-09-16 5:39 ` [PATCH v2 1/2] spdx: add entries for meta-layers Daniel Wagenknecht 2026-09-16 5:39 ` [PATCH v2 2/2] spdx: link files to the meta-layer providing them Daniel Wagenknecht
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox