* [RFC][OE-core 0/7] SPDX3 Proof-of-Concept
@ 2023-10-26 10:48 Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 1/7] create-spdx-3.0: copy 2.2 class Marta Rybczynska
` (8 more replies)
0 siblings, 9 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Marta Rybczynska
This patch-set adds a proof-of-concept implementation of the upcoming
SPDX3 standard to the SBOM generation of the Yocto Project/OpenEmbedded.
The current code delivers an equivalent of what is produced for SPDX2.2.
The standard has not been released yet, and there is some specification
work in progress still. Our questions and open points are available
in the README.SPDX3 file.
Also, this first RFC delivery will be followed by another one with
SPDX assembly and the Licensing profile.
Louis Rannou (5):
create-spdx-3.0: copy 2.2 class
oe/spdx: extend spdx.py objects
oe/sbom: change the write_doc to prepare for spdx3
create-spdx-3.0: SPDX3 objects as classes
oe/sbom: search into json
Marta Rybczynska (1):
README.SPDX3: add file
Samantha Jalabert (1):
create-spdx-3.0: support for recipe spdx creation
README.SPDX3 | 42 ++
meta/classes/create-spdx-3.0.bbclass | 878 +++++++++++++++++++++++++++
meta/classes/create-spdx.bbclass | 2 +-
meta/lib/oe/sbom.py | 37 +-
meta/lib/oe/spdx.py | 30 +-
meta/lib/oe/spdx3.py | 385 ++++++++++++
6 files changed, 1364 insertions(+), 10 deletions(-)
create mode 100644 README.SPDX3
create mode 100644 meta/classes/create-spdx-3.0.bbclass
create mode 100644 meta/lib/oe/spdx3.py
--
2.42.0
^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC][OE-core 1/7] create-spdx-3.0: copy 2.2 class
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 2/7] oe/spdx: extend spdx.py objects Marta Rybczynska
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Louis Rannou, Marta Rybczynska
From: Louis Rannou <louis.rannou@syslinbit.com>
Initialize the work on SPDX 3 with a copy of the SPDX 2.2. Change default to
SPDX 3.
Signed-off-by: Louis Rannou <louis.rannou@syslinbit.com>
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
---
meta/classes/create-spdx-3.0.bbclass | 1158 ++++++++++++++++++++++++++
meta/classes/create-spdx.bbclass | 2 +-
2 files changed, 1159 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/create-spdx-3.0.bbclass
diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass
new file mode 100644
index 0000000000..b0aef80db1
--- /dev/null
+++ b/meta/classes/create-spdx-3.0.bbclass
@@ -0,0 +1,1158 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+DEPLOY_DIR_SPDX ??= "${DEPLOY_DIR}/spdx"
+
+# The product name that the CVE database uses. Defaults to BPN, but may need to
+# be overriden per recipe (for example tiff.bb sets CVE_PRODUCT=libtiff).
+CVE_PRODUCT ??= "${BPN}"
+CVE_VERSION ??= "${PV}"
+
+SPDXDIR ??= "${WORKDIR}/spdx"
+SPDXDEPLOY = "${SPDXDIR}/deploy"
+SPDXWORK = "${SPDXDIR}/work"
+SPDXIMAGEWORK = "${SPDXDIR}/image-work"
+SPDXSDKWORK = "${SPDXDIR}/sdk-work"
+SPDXDEPS = "${SPDXDIR}/deps.json"
+
+SPDX_TOOL_NAME ??= "oe-spdx-creator"
+SPDX_TOOL_VERSION ??= "1.0"
+
+SPDXRUNTIMEDEPLOY = "${SPDXDIR}/runtime-deploy"
+
+SPDX_INCLUDE_SOURCES ??= "0"
+SPDX_ARCHIVE_SOURCES ??= "0"
+SPDX_ARCHIVE_PACKAGED ??= "0"
+
+SPDX_UUID_NAMESPACE ??= "sbom.openembedded.org"
+SPDX_NAMESPACE_PREFIX ??= "http://spdx.org/spdxdoc"
+SPDX_PRETTY ??= "0"
+
+SPDX_LICENSES ??= "${COREBASE}/meta/files/spdx-licenses.json"
+
+SPDX_CUSTOM_ANNOTATION_VARS ??= ""
+
+SPDX_ORG ??= "OpenEmbedded ()"
+SPDX_SUPPLIER ??= "Organization: ${SPDX_ORG}"
+SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created from \
+ this recipe. For SPDX documents create using this class during the build, this \
+ is the contact information for the person or organization who is doing the \
+ build."
+
+def extract_licenses(filename):
+ import re
+
+ lic_regex = re.compile(rb'^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$', re.MULTILINE)
+
+ try:
+ with open(filename, 'rb') as f:
+ size = min(15000, os.stat(filename).st_size)
+ txt = f.read(size)
+ licenses = re.findall(lic_regex, txt)
+ if licenses:
+ ascii_licenses = [lic.decode('ascii') for lic in licenses]
+ return ascii_licenses
+ except Exception as e:
+ bb.warn(f"Exception reading {filename}: {e}")
+ return None
+
+def get_doc_namespace(d, doc):
+ import uuid
+ namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS, d.getVar("SPDX_UUID_NAMESPACE"))
+ return "%s/%s-%s" % (d.getVar("SPDX_NAMESPACE_PREFIX"), doc.name, str(uuid.uuid5(namespace_uuid, doc.name)))
+
+def create_annotation(d, comment):
+ from datetime import datetime, timezone
+
+ creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+ annotation = oe.spdx.SPDXAnnotation()
+ annotation.annotationDate = creation_time
+ annotation.annotationType = "OTHER"
+ annotation.annotator = "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION"))
+ annotation.comment = comment
+ return annotation
+
+def recipe_spdx_is_native(d, recipe):
+ return any(a.annotationType == "OTHER" and
+ a.annotator == "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION")) and
+ a.comment == "isNative" for a in recipe.annotations)
+
+def is_work_shared_spdx(d):
+ return bb.data.inherits_class('kernel', d) or ('work-shared' in d.getVar('WORKDIR'))
+
+def get_json_indent(d):
+ if d.getVar("SPDX_PRETTY") == "1":
+ return 2
+ return None
+
+python() {
+ import json
+ if d.getVar("SPDX_LICENSE_DATA"):
+ return
+
+ with open(d.getVar("SPDX_LICENSES"), "r") as f:
+ data = json.load(f)
+ # Transform the license array to a dictionary
+ data["licenses"] = {l["licenseId"]: l for l in data["licenses"]}
+ d.setVar("SPDX_LICENSE_DATA", data)
+}
+
+def convert_license_to_spdx(lic, document, d, existing={}):
+ from pathlib import Path
+ import oe.spdx
+
+ license_data = d.getVar("SPDX_LICENSE_DATA")
+ extracted = {}
+
+ def add_extracted_license(ident, name):
+ nonlocal document
+
+ if name in extracted:
+ return
+
+ extracted_info = oe.spdx.SPDXExtractedLicensingInfo()
+ extracted_info.name = name
+ extracted_info.licenseId = ident
+ extracted_info.extractedText = None
+
+ if name == "PD":
+ # Special-case this.
+ extracted_info.extractedText = "Software released to the public domain"
+ else:
+ # Seach for the license in COMMON_LICENSE_DIR and LICENSE_PATH
+ for directory in [d.getVar('COMMON_LICENSE_DIR')] + (d.getVar('LICENSE_PATH') or '').split():
+ try:
+ with (Path(directory) / name).open(errors="replace") as f:
+ extracted_info.extractedText = f.read()
+ break
+ except FileNotFoundError:
+ pass
+ if extracted_info.extractedText is None:
+ # If it's not SPDX or PD, then NO_GENERIC_LICENSE must be set
+ filename = d.getVarFlag('NO_GENERIC_LICENSE', name)
+ if filename:
+ filename = d.expand("${S}/" + filename)
+ with open(filename, errors="replace") as f:
+ extracted_info.extractedText = f.read()
+ else:
+ bb.fatal("Cannot find any text for license %s" % name)
+
+ extracted[name] = extracted_info
+ document.hasExtractedLicensingInfos.append(extracted_info)
+
+ def convert(l):
+ if l == "(" or l == ")":
+ return l
+
+ if l == "&":
+ return "AND"
+
+ if l == "|":
+ return "OR"
+
+ if l == "CLOSED":
+ return "NONE"
+
+ spdx_license = d.getVarFlag("SPDXLICENSEMAP", l) or l
+ if spdx_license in license_data["licenses"]:
+ return spdx_license
+
+ try:
+ spdx_license = existing[l]
+ except KeyError:
+ spdx_license = "LicenseRef-" + l
+ add_extracted_license(spdx_license, l)
+
+ return spdx_license
+
+ lic_split = lic.replace("(", " ( ").replace(")", " ) ").replace("|", " | ").replace("&", " & ").split()
+
+ return ' '.join(convert(l) for l in lic_split)
+
+def process_sources(d):
+ pn = d.getVar('PN')
+ assume_provided = (d.getVar("ASSUME_PROVIDED") or "").split()
+ if pn in assume_provided:
+ for p in d.getVar("PROVIDES").split():
+ if p != pn:
+ pn = p
+ break
+
+ # glibc-locale: do_fetch, do_unpack and do_patch tasks have been deleted,
+ # so avoid archiving source here.
+ if pn.startswith('glibc-locale'):
+ return False
+ if d.getVar('PN') == "libtool-cross":
+ return False
+ if d.getVar('PN') == "libgcc-initial":
+ return False
+ if d.getVar('PN') == "shadow-sysroot":
+ return False
+
+ # We just archive gcc-source for all the gcc related recipes
+ if d.getVar('BPN') in ['gcc', 'libgcc']:
+ bb.debug(1, 'spdx: There is bug in scan of %s is, do nothing' % pn)
+ return False
+
+ return True
+
+
+def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archive=None, ignore_dirs=[], ignore_top_level_dirs=[]):
+ from pathlib import Path
+ import oe.spdx
+ import hashlib
+
+ source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
+ if source_date_epoch:
+ source_date_epoch = int(source_date_epoch)
+
+ sha1s = []
+ spdx_files = []
+
+ file_counter = 1
+ for subdir, dirs, files in os.walk(topdir):
+ dirs[:] = [d for d in dirs if d not in ignore_dirs]
+ if subdir == str(topdir):
+ dirs[:] = [d for d in dirs if d not in ignore_top_level_dirs]
+
+ for file in files:
+ filepath = Path(subdir) / file
+ filename = str(filepath.relative_to(topdir))
+
+ if not filepath.is_symlink() and filepath.is_file():
+ spdx_file = oe.spdx.SPDXFile()
+ spdx_file.SPDXID = get_spdxid(file_counter)
+ for t in get_types(filepath):
+ spdx_file.fileTypes.append(t)
+ spdx_file.fileName = filename
+
+ if archive is not None:
+ with filepath.open("rb") as f:
+ info = archive.gettarinfo(fileobj=f)
+ info.name = filename
+ info.uid = 0
+ info.gid = 0
+ info.uname = "root"
+ info.gname = "root"
+
+ if source_date_epoch is not None and info.mtime > source_date_epoch:
+ info.mtime = source_date_epoch
+
+ archive.addfile(info, f)
+
+ sha1 = bb.utils.sha1_file(filepath)
+ sha1s.append(sha1)
+ spdx_file.checksums.append(oe.spdx.SPDXChecksum(
+ algorithm="SHA1",
+ checksumValue=sha1,
+ ))
+ spdx_file.checksums.append(oe.spdx.SPDXChecksum(
+ algorithm="SHA256",
+ checksumValue=bb.utils.sha256_file(filepath),
+ ))
+
+ if "SOURCE" in spdx_file.fileTypes:
+ extracted_lics = extract_licenses(filepath)
+ if extracted_lics:
+ spdx_file.licenseInfoInFiles = extracted_lics
+
+ doc.files.append(spdx_file)
+ doc.add_relationship(spdx_pkg, "CONTAINS", spdx_file)
+ spdx_pkg.hasFiles.append(spdx_file.SPDXID)
+
+ spdx_files.append(spdx_file)
+
+ file_counter += 1
+
+ sha1s.sort()
+ verifier = hashlib.sha1()
+ for v in sha1s:
+ verifier.update(v.encode("utf-8"))
+ spdx_pkg.packageVerificationCode.packageVerificationCodeValue = verifier.hexdigest()
+
+ return spdx_files
+
+
+def add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources):
+ from pathlib import Path
+ import hashlib
+ import oe.packagedata
+ import oe.spdx
+
+ debug_search_paths = [
+ Path(d.getVar('PKGD')),
+ Path(d.getVar('STAGING_DIR_TARGET')),
+ Path(d.getVar('STAGING_DIR_NATIVE')),
+ Path(d.getVar('STAGING_KERNEL_DIR')),
+ ]
+
+ pkg_data = oe.packagedata.read_subpkgdata_extended(package, d)
+
+ if pkg_data is None:
+ return
+
+ for file_path, file_data in pkg_data["files_info"].items():
+ if not "debugsrc" in file_data:
+ continue
+
+ for pkg_file in package_files:
+ if file_path.lstrip("/") == pkg_file.fileName.lstrip("/"):
+ break
+ else:
+ bb.fatal("No package file found for %s in %s; SPDX found: %s" % (str(file_path), package,
+ " ".join(p.fileName for p in package_files)))
+ continue
+
+ for debugsrc in file_data["debugsrc"]:
+ ref_id = "NOASSERTION"
+ for search in debug_search_paths:
+ if debugsrc.startswith("/usr/src/kernel"):
+ debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '')
+ else:
+ debugsrc_path = search / debugsrc.lstrip("/")
+ if not debugsrc_path.exists():
+ continue
+
+ file_sha256 = bb.utils.sha256_file(debugsrc_path)
+
+ if file_sha256 in sources:
+ source_file = sources[file_sha256]
+
+ doc_ref = package_doc.find_external_document_ref(source_file.doc.documentNamespace)
+ if doc_ref is None:
+ doc_ref = oe.spdx.SPDXExternalDocumentRef()
+ doc_ref.externalDocumentId = "DocumentRef-dependency-" + source_file.doc.name
+ doc_ref.spdxDocument = source_file.doc.documentNamespace
+ doc_ref.checksum.algorithm = "SHA1"
+ doc_ref.checksum.checksumValue = source_file.doc_sha1
+ package_doc.externalDocumentRefs.append(doc_ref)
+
+ ref_id = "%s:%s" % (doc_ref.externalDocumentId, source_file.file.SPDXID)
+ else:
+ bb.debug(1, "Debug source %s with SHA256 %s not found in any dependency" % (str(debugsrc_path), file_sha256))
+ break
+ else:
+ bb.debug(1, "Debug source %s not found" % debugsrc)
+
+ package_doc.add_relationship(pkg_file, "GENERATED_FROM", ref_id, comment=debugsrc)
+
+add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
+
+def collect_dep_recipes(d, doc, spdx_recipe):
+ import json
+ from pathlib import Path
+ import oe.sbom
+ import oe.spdx
+
+ deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
+ spdx_deps_file = Path(d.getVar("SPDXDEPS"))
+ package_archs = d.getVar("SSTATE_ARCHS").split()
+ package_archs.reverse()
+
+ dep_recipes = []
+
+ with spdx_deps_file.open("r") as f:
+ deps = json.load(f)
+
+ for dep_pn, dep_hashfn in deps:
+ dep_recipe_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "recipe-" + dep_pn, dep_hashfn)
+ if not dep_recipe_path:
+ bb.fatal("Cannot find any SPDX file for recipe %s, %s" % (dep_pn, dep_hashfn))
+
+ spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path)
+
+ for pkg in spdx_dep_doc.packages:
+ if pkg.name == dep_pn:
+ spdx_dep_recipe = pkg
+ break
+ else:
+ continue
+
+ dep_recipes.append(oe.sbom.DepRecipe(spdx_dep_doc, spdx_dep_sha1, spdx_dep_recipe))
+
+ dep_recipe_ref = oe.spdx.SPDXExternalDocumentRef()
+ dep_recipe_ref.externalDocumentId = "DocumentRef-dependency-" + spdx_dep_doc.name
+ dep_recipe_ref.spdxDocument = spdx_dep_doc.documentNamespace
+ dep_recipe_ref.checksum.algorithm = "SHA1"
+ dep_recipe_ref.checksum.checksumValue = spdx_dep_sha1
+
+ doc.externalDocumentRefs.append(dep_recipe_ref)
+
+ doc.add_relationship(
+ "%s:%s" % (dep_recipe_ref.externalDocumentId, spdx_dep_recipe.SPDXID),
+ "BUILD_DEPENDENCY_OF",
+ spdx_recipe
+ )
+
+ return dep_recipes
+
+collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS"
+
+def collect_dep_sources(d, dep_recipes):
+ import oe.sbom
+
+ sources = {}
+ for dep in dep_recipes:
+ # Don't collect sources from native recipes as they
+ # match non-native sources also.
+ if recipe_spdx_is_native(d, dep.recipe):
+ continue
+ recipe_files = set(dep.recipe.hasFiles)
+
+ for spdx_file in dep.doc.files:
+ if spdx_file.SPDXID not in recipe_files:
+ continue
+
+ if "SOURCE" in spdx_file.fileTypes:
+ for checksum in spdx_file.checksums:
+ if checksum.algorithm == "SHA256":
+ sources[checksum.checksumValue] = oe.sbom.DepSource(dep.doc, dep.doc_sha1, dep.recipe, spdx_file)
+ break
+
+ return sources
+
+def add_download_packages(d, doc, recipe):
+ import os.path
+ from bb.fetch2 import decodeurl, CHECKSUM_LIST
+ import bb.process
+ import oe.spdx
+ import oe.sbom
+
+ for download_idx, src_uri in enumerate(d.getVar('SRC_URI').split()):
+ f = bb.fetch2.FetchData(src_uri, d)
+
+ for name in f.names:
+ package = oe.spdx.SPDXPackage()
+ package.name = "%s-source-%d" % (d.getVar("PN"), download_idx + 1)
+ package.SPDXID = oe.sbom.get_download_spdxid(d, download_idx + 1)
+
+ if f.type == "file":
+ continue
+
+ uri = f.type
+ proto = getattr(f, "proto", None)
+ if proto is not None:
+ uri = uri + "+" + proto
+ uri = uri + "://" + f.host + f.path
+
+ if f.method.supports_srcrev():
+ uri = uri + "@" + f.revisions[name]
+
+ if f.method.supports_checksum(f):
+ for checksum_id in CHECKSUM_LIST:
+ if checksum_id.upper() not in oe.spdx.SPDXPackage.ALLOWED_CHECKSUMS:
+ continue
+
+ expected_checksum = getattr(f, "%s_expected" % checksum_id)
+ if expected_checksum is None:
+ continue
+
+ c = oe.spdx.SPDXChecksum()
+ c.algorithm = checksum_id.upper()
+ c.checksumValue = expected_checksum
+ package.checksums.append(c)
+
+ package.downloadLocation = uri
+ doc.packages.append(package)
+ doc.add_relationship(doc, "DESCRIBES", package)
+ # In the future, we might be able to do more fancy dependencies,
+ # but this should be sufficient for now
+ doc.add_relationship(package, "BUILD_DEPENDENCY_OF", recipe)
+
+def collect_direct_deps(d, dep_task):
+ current_task = "do_" + d.getVar("BB_CURRENTTASK")
+ pn = d.getVar("PN")
+
+ taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+
+ for this_dep in taskdepdata.values():
+ if this_dep[0] == pn and this_dep[1] == current_task:
+ break
+ else:
+ bb.fatal(f"Unable to find this {pn}:{current_task} in taskdepdata")
+
+ deps = set()
+ for dep_name in this_dep[3]:
+ dep_data = taskdepdata[dep_name]
+ if dep_data[1] == dep_task and dep_data[0] != pn:
+ deps.add((dep_data[0], dep_data[7]))
+
+ return sorted(deps)
+
+collect_direct_deps[vardepsexclude] += "BB_TASKDEPDATA"
+collect_direct_deps[vardeps] += "DEPENDS"
+
+python do_collect_spdx_deps() {
+ # This task calculates the build time dependencies of the recipe, and is
+ # required because while a task can deptask on itself, those dependencies
+ # do not show up in BB_TASKDEPDATA. To work around that, this task does the
+ # deptask on do_create_spdx and writes out the dependencies it finds, then
+ # do_create_spdx reads in the found dependencies when writing the actual
+ # SPDX document
+ import json
+ from pathlib import Path
+
+ spdx_deps_file = Path(d.getVar("SPDXDEPS"))
+
+ deps = collect_direct_deps(d, "do_create_spdx")
+
+ with spdx_deps_file.open("w") as f:
+ json.dump(deps, f)
+}
+# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
+addtask do_collect_spdx_deps after do_unpack
+do_collect_spdx_deps[depends] += "${PATCHDEPENDENCY}"
+do_collect_spdx_deps[deptask] = "do_create_spdx"
+do_collect_spdx_deps[dirs] = "${SPDXDIR}"
+
+python do_create_spdx() {
+ from datetime import datetime, timezone
+ import oe.sbom
+ import oe.spdx
+ import uuid
+ from pathlib import Path
+ from contextlib import contextmanager
+ import oe.cve_check
+
+ @contextmanager
+ def optional_tarfile(name, guard, mode="w"):
+ import tarfile
+ import bb.compress.zstd
+
+ num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+
+ if guard:
+ name.parent.mkdir(parents=True, exist_ok=True)
+ with bb.compress.zstd.open(name, mode=mode + "b", num_threads=num_threads) as f:
+ with tarfile.open(fileobj=f, mode=mode + "|") as tf:
+ yield tf
+ else:
+ yield None
+
+
+ deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
+ spdx_workdir = Path(d.getVar("SPDXWORK"))
+ include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
+ archive_sources = d.getVar("SPDX_ARCHIVE_SOURCES") == "1"
+ archive_packaged = d.getVar("SPDX_ARCHIVE_PACKAGED") == "1"
+ pkg_arch = d.getVar("SSTATE_PKGARCH")
+
+ creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+ doc = oe.spdx.SPDXDocument()
+
+ doc.name = "recipe-" + d.getVar("PN")
+ doc.documentNamespace = get_doc_namespace(d, doc)
+ doc.creationInfo.created = creation_time
+ doc.creationInfo.comment = "This document was created by analyzing recipe files during the build."
+ doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+ doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
+ doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
+ doc.creationInfo.creators.append("Person: N/A ()")
+
+ recipe = oe.spdx.SPDXPackage()
+ recipe.name = d.getVar("PN")
+ recipe.versionInfo = d.getVar("PV")
+ recipe.SPDXID = oe.sbom.get_recipe_spdxid(d)
+ recipe.supplier = d.getVar("SPDX_SUPPLIER")
+ if bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d):
+ recipe.annotations.append(create_annotation(d, "isNative"))
+
+ homepage = d.getVar("HOMEPAGE")
+ if homepage:
+ recipe.homepage = homepage
+
+ license = d.getVar("LICENSE")
+ if license:
+ recipe.licenseDeclared = convert_license_to_spdx(license, doc, d)
+
+ summary = d.getVar("SUMMARY")
+ if summary:
+ recipe.summary = summary
+
+ description = d.getVar("DESCRIPTION")
+ if description:
+ recipe.description = description
+
+ if d.getVar("SPDX_CUSTOM_ANNOTATION_VARS"):
+ for var in d.getVar('SPDX_CUSTOM_ANNOTATION_VARS').split():
+ recipe.annotations.append(create_annotation(d, var + "=" + d.getVar(var)))
+
+ # Some CVEs may be patched during the build process without incrementing the version number,
+ # so querying for CVEs based on the CPE id can lead to false positives. To account for this,
+ # save the CVEs fixed by patches to source information field in the SPDX.
+ patched_cves = oe.cve_check.get_patched_cves(d)
+ patched_cves = list(patched_cves)
+ patched_cves = ' '.join(patched_cves)
+ if patched_cves:
+ recipe.sourceInfo = "CVEs fixed: " + patched_cves
+
+ cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION"))
+ if cpe_ids:
+ for cpe_id in cpe_ids:
+ cpe = oe.spdx.SPDXExternalReference()
+ cpe.referenceCategory = "SECURITY"
+ cpe.referenceType = "http://spdx.org/rdf/references/cpe23Type"
+ cpe.referenceLocator = cpe_id
+ recipe.externalRefs.append(cpe)
+
+ doc.packages.append(recipe)
+ doc.add_relationship(doc, "DESCRIBES", recipe)
+
+ add_download_packages(d, doc, recipe)
+
+ if process_sources(d) and include_sources:
+ recipe_archive = deploy_dir_spdx / "recipes" / (doc.name + ".tar.zst")
+ with optional_tarfile(recipe_archive, archive_sources) as archive:
+ spdx_get_src(d)
+
+ add_package_files(
+ d,
+ doc,
+ recipe,
+ spdx_workdir,
+ lambda file_counter: "SPDXRef-SourceFile-%s-%d" % (d.getVar("PN"), file_counter),
+ lambda filepath: ["SOURCE"],
+ ignore_dirs=[".git"],
+ ignore_top_level_dirs=["temp"],
+ archive=archive,
+ )
+
+ if archive is not None:
+ recipe.packageFileName = str(recipe_archive.name)
+
+ dep_recipes = collect_dep_recipes(d, doc, recipe)
+
+ doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d))
+ dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
+
+ recipe_ref = oe.spdx.SPDXExternalDocumentRef()
+ recipe_ref.externalDocumentId = "DocumentRef-recipe-" + recipe.name
+ recipe_ref.spdxDocument = doc.documentNamespace
+ recipe_ref.checksum.algorithm = "SHA1"
+ recipe_ref.checksum.checksumValue = doc_sha1
+
+ sources = collect_dep_sources(d, dep_recipes)
+ found_licenses = {license.name:recipe_ref.externalDocumentId + ":" + license.licenseId for license in doc.hasExtractedLicensingInfos}
+
+ if not recipe_spdx_is_native(d, recipe):
+ bb.build.exec_func("read_subpackage_metadata", d)
+
+ pkgdest = Path(d.getVar("PKGDEST"))
+ for package in d.getVar("PACKAGES").split():
+ if not oe.packagedata.packaged(package, d):
+ continue
+
+ package_doc = oe.spdx.SPDXDocument()
+ pkg_name = d.getVar("PKG:%s" % package) or package
+ package_doc.name = pkg_name
+ package_doc.documentNamespace = get_doc_namespace(d, package_doc)
+ package_doc.creationInfo.created = creation_time
+ package_doc.creationInfo.comment = "This document was created by analyzing packages created during the build."
+ package_doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+ package_doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
+ package_doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
+ package_doc.creationInfo.creators.append("Person: N/A ()")
+ package_doc.externalDocumentRefs.append(recipe_ref)
+
+ package_license = d.getVar("LICENSE:%s" % package) or d.getVar("LICENSE")
+
+ spdx_package = oe.spdx.SPDXPackage()
+
+ spdx_package.SPDXID = oe.sbom.get_package_spdxid(pkg_name)
+ spdx_package.name = pkg_name
+ spdx_package.versionInfo = d.getVar("PV")
+ spdx_package.licenseDeclared = convert_license_to_spdx(package_license, package_doc, d, found_licenses)
+ spdx_package.supplier = d.getVar("SPDX_SUPPLIER")
+
+ package_doc.packages.append(spdx_package)
+
+ package_doc.add_relationship(spdx_package, "GENERATED_FROM", "%s:%s" % (recipe_ref.externalDocumentId, recipe.SPDXID))
+ package_doc.add_relationship(package_doc, "DESCRIBES", spdx_package)
+
+ package_archive = deploy_dir_spdx / "packages" / (package_doc.name + ".tar.zst")
+ with optional_tarfile(package_archive, archive_packaged) as archive:
+ package_files = add_package_files(
+ d,
+ package_doc,
+ spdx_package,
+ pkgdest / package,
+ lambda file_counter: oe.sbom.get_packaged_file_spdxid(pkg_name, file_counter),
+ lambda filepath: ["BINARY"],
+ ignore_top_level_dirs=['CONTROL', 'DEBIAN'],
+ archive=archive,
+ )
+
+ if archive is not None:
+ spdx_package.packageFileName = str(package_archive.name)
+
+ add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources)
+
+ oe.sbom.write_doc(d, package_doc, pkg_arch, "packages", indent=get_json_indent(d))
+}
+do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
+# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
+addtask do_create_spdx after do_package do_packagedata do_unpack do_collect_spdx_deps before do_populate_sdk do_build do_rm_work
+
+SSTATETASKS += "do_create_spdx"
+do_create_spdx[sstate-inputdirs] = "${SPDXDEPLOY}"
+do_create_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
+
+python do_create_spdx_setscene () {
+ sstate_setscene(d)
+}
+addtask do_create_spdx_setscene
+
+do_create_spdx[dirs] = "${SPDXWORK}"
+do_create_spdx[cleandirs] = "${SPDXDEPLOY} ${SPDXWORK}"
+do_create_spdx[depends] += "${PATCHDEPENDENCY}"
+
+def collect_package_providers(d):
+ from pathlib import Path
+ import oe.sbom
+ import oe.spdx
+ import json
+
+ deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
+
+ providers = {}
+
+ deps = collect_direct_deps(d, "do_create_spdx")
+ deps.append((d.getVar("PN"), d.getVar("BB_HASHFILENAME")))
+
+ for dep_pn, dep_hashfn in deps:
+ localdata = d
+ recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata)
+ if not recipe_data:
+ localdata = bb.data.createCopy(d)
+ localdata.setVar("PKGDATA_DIR", "${PKGDATA_DIR_SDK}")
+ recipe_data = oe.packagedata.read_pkgdata(dep_pn, localdata)
+
+ for pkg in recipe_data.get("PACKAGES", "").split():
+
+ pkg_data = oe.packagedata.read_subpkgdata_dict(pkg, localdata)
+ rprovides = set(n for n, _ in bb.utils.explode_dep_versions2(pkg_data.get("RPROVIDES", "")).items())
+ rprovides.add(pkg)
+
+ if "PKG" in pkg_data:
+ pkg = pkg_data["PKG"]
+ rprovides.add(pkg)
+
+ for r in rprovides:
+ providers[r] = (pkg, dep_hashfn)
+
+ return providers
+
+collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA"
+
+python do_create_runtime_spdx() {
+ from datetime import datetime, timezone
+ import oe.sbom
+ import oe.spdx
+ import oe.packagedata
+ from pathlib import Path
+
+ deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
+ spdx_deploy = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
+ is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
+
+ creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+ providers = collect_package_providers(d)
+ pkg_arch = d.getVar("SSTATE_PKGARCH")
+ package_archs = d.getVar("SSTATE_ARCHS").split()
+ package_archs.reverse()
+
+ if not is_native:
+ bb.build.exec_func("read_subpackage_metadata", d)
+
+ dep_package_cache = {}
+
+ pkgdest = Path(d.getVar("PKGDEST"))
+ for package in d.getVar("PACKAGES").split():
+ localdata = bb.data.createCopy(d)
+ pkg_name = d.getVar("PKG:%s" % package) or package
+ localdata.setVar("PKG", pkg_name)
+ localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + package)
+
+ if not oe.packagedata.packaged(package, localdata):
+ continue
+
+ pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, pkg_arch, "packages")
+
+ package_doc, package_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
+
+ for p in package_doc.packages:
+ if p.name == pkg_name:
+ spdx_package = p
+ break
+ else:
+ bb.fatal("Package '%s' not found in %s" % (pkg_name, pkg_spdx_path))
+
+ runtime_doc = oe.spdx.SPDXDocument()
+ runtime_doc.name = "runtime-" + pkg_name
+ runtime_doc.documentNamespace = get_doc_namespace(localdata, runtime_doc)
+ runtime_doc.creationInfo.created = creation_time
+ runtime_doc.creationInfo.comment = "This document was created by analyzing package runtime dependencies."
+ runtime_doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+ runtime_doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
+ runtime_doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
+ runtime_doc.creationInfo.creators.append("Person: N/A ()")
+
+ package_ref = oe.spdx.SPDXExternalDocumentRef()
+ package_ref.externalDocumentId = "DocumentRef-package-" + package
+ package_ref.spdxDocument = package_doc.documentNamespace
+ package_ref.checksum.algorithm = "SHA1"
+ package_ref.checksum.checksumValue = package_doc_sha1
+
+ runtime_doc.externalDocumentRefs.append(package_ref)
+
+ runtime_doc.add_relationship(
+ runtime_doc.SPDXID,
+ "AMENDS",
+ "%s:%s" % (package_ref.externalDocumentId, package_doc.SPDXID)
+ )
+
+ deps = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "")
+ seen_deps = set()
+ for dep, _ in deps.items():
+ if dep in seen_deps:
+ continue
+
+ if dep not in providers:
+ continue
+
+ (dep, dep_hashfn) = providers[dep]
+
+ if not oe.packagedata.packaged(dep, localdata):
+ continue
+
+ dep_pkg_data = oe.packagedata.read_subpkgdata_dict(dep, d)
+ dep_pkg = dep_pkg_data["PKG"]
+
+ if dep in dep_package_cache:
+ (dep_spdx_package, dep_package_ref) = dep_package_cache[dep]
+ else:
+ dep_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, dep_pkg, dep_hashfn)
+ if not dep_path:
+ bb.fatal("No SPDX file found for package %s, %s" % (dep_pkg, dep_hashfn))
+
+ spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_path)
+
+ for pkg in spdx_dep_doc.packages:
+ if pkg.name == dep_pkg:
+ dep_spdx_package = pkg
+ break
+ else:
+ bb.fatal("Package '%s' not found in %s" % (dep_pkg, dep_path))
+
+ dep_package_ref = oe.spdx.SPDXExternalDocumentRef()
+ dep_package_ref.externalDocumentId = "DocumentRef-runtime-dependency-" + spdx_dep_doc.name
+ dep_package_ref.spdxDocument = spdx_dep_doc.documentNamespace
+ dep_package_ref.checksum.algorithm = "SHA1"
+ dep_package_ref.checksum.checksumValue = spdx_dep_sha1
+
+ dep_package_cache[dep] = (dep_spdx_package, dep_package_ref)
+
+ runtime_doc.externalDocumentRefs.append(dep_package_ref)
+
+ runtime_doc.add_relationship(
+ "%s:%s" % (dep_package_ref.externalDocumentId, dep_spdx_package.SPDXID),
+ "RUNTIME_DEPENDENCY_OF",
+ "%s:%s" % (package_ref.externalDocumentId, spdx_package.SPDXID)
+ )
+ seen_deps.add(dep)
+
+ oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d))
+}
+
+do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
+
+addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work
+SSTATETASKS += "do_create_runtime_spdx"
+do_create_runtime_spdx[sstate-inputdirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_runtime_spdx[sstate-outputdirs] = "${DEPLOY_DIR_SPDX}"
+
+python do_create_runtime_spdx_setscene () {
+ sstate_setscene(d)
+}
+addtask do_create_runtime_spdx_setscene
+
+do_create_runtime_spdx[dirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_runtime_spdx[cleandirs] = "${SPDXRUNTIMEDEPLOY}"
+do_create_runtime_spdx[rdeptask] = "do_create_spdx"
+
+def spdx_get_src(d):
+ """
+ save patched source of the recipe in SPDX_WORKDIR.
+ """
+ import shutil
+ spdx_workdir = d.getVar('SPDXWORK')
+ spdx_sysroot_native = d.getVar('STAGING_DIR_NATIVE')
+ pn = d.getVar('PN')
+
+ workdir = d.getVar("WORKDIR")
+
+ try:
+ # The kernel class functions require it to be on work-shared, so we dont change WORKDIR
+ if not is_work_shared_spdx(d):
+ # Change the WORKDIR to make do_unpack do_patch run in another dir.
+ d.setVar('WORKDIR', spdx_workdir)
+ # Restore the original path to recipe's native sysroot (it's relative to WORKDIR).
+ d.setVar('STAGING_DIR_NATIVE', spdx_sysroot_native)
+
+ # The changed 'WORKDIR' also caused 'B' changed, create dir 'B' for the
+ # possibly requiring of the following tasks (such as some recipes's
+ # do_patch required 'B' existed).
+ bb.utils.mkdirhier(d.getVar('B'))
+
+ bb.build.exec_func('do_unpack', d)
+ # Copy source of kernel to spdx_workdir
+ if is_work_shared_spdx(d):
+ share_src = d.getVar('WORKDIR')
+ d.setVar('WORKDIR', spdx_workdir)
+ d.setVar('STAGING_DIR_NATIVE', spdx_sysroot_native)
+ src_dir = spdx_workdir + "/" + d.getVar('PN')+ "-" + d.getVar('PV') + "-" + d.getVar('PR')
+ bb.utils.mkdirhier(src_dir)
+ if bb.data.inherits_class('kernel',d):
+ share_src = d.getVar('STAGING_KERNEL_DIR')
+ cmd_copy_share = "cp -rf " + share_src + "/* " + src_dir + "/"
+ cmd_copy_shared_res = os.popen(cmd_copy_share).read()
+ bb.note("cmd_copy_shared_result = " + cmd_copy_shared_res)
+
+ git_path = src_dir + "/.git"
+ if os.path.exists(git_path):
+ shutils.rmtree(git_path)
+
+ # Make sure gcc and kernel sources are patched only once
+ if not (d.getVar('SRC_URI') == "" or is_work_shared_spdx(d)):
+ bb.build.exec_func('do_patch', d)
+
+ # Some userland has no source.
+ if not os.path.exists( spdx_workdir ):
+ bb.utils.mkdirhier(spdx_workdir)
+ finally:
+ d.setVar("WORKDIR", workdir)
+
+spdx_get_src[vardepsexclude] += "STAGING_KERNEL_DIR"
+
+do_rootfs[recrdeptask] += "do_create_spdx do_create_runtime_spdx"
+do_rootfs[cleandirs] += "${SPDXIMAGEWORK}"
+
+ROOTFS_POSTUNINSTALL_COMMAND =+ "image_combine_spdx"
+
+do_populate_sdk[recrdeptask] += "do_create_spdx do_create_runtime_spdx"
+do_populate_sdk[cleandirs] += "${SPDXSDKWORK}"
+POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_combine_spdx"
+POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_combine_spdx"
+
+python image_combine_spdx() {
+ import os
+ import oe.sbom
+ from pathlib import Path
+ from oe.rootfs import image_list_installed_packages
+
+ image_name = d.getVar("IMAGE_NAME")
+ image_link_name = d.getVar("IMAGE_LINK_NAME")
+ imgdeploydir = Path(d.getVar("IMGDEPLOYDIR"))
+ img_spdxid = oe.sbom.get_image_spdxid(image_name)
+ packages = image_list_installed_packages(d)
+
+ combine_spdx(d, image_name, imgdeploydir, img_spdxid, packages, Path(d.getVar("SPDXIMAGEWORK")))
+
+ def make_image_link(target_path, suffix):
+ if image_link_name:
+ link = imgdeploydir / (image_link_name + suffix)
+ if link != target_path:
+ link.symlink_to(os.path.relpath(target_path, link.parent))
+
+ spdx_tar_path = imgdeploydir / (image_name + ".spdx.tar.zst")
+ make_image_link(spdx_tar_path, ".spdx.tar.zst")
+}
+
+python sdk_host_combine_spdx() {
+ sdk_combine_spdx(d, "host")
+}
+
+python sdk_target_combine_spdx() {
+ sdk_combine_spdx(d, "target")
+}
+
+def sdk_combine_spdx(d, sdk_type):
+ import oe.sbom
+ from pathlib import Path
+ from oe.sdk import sdk_list_installed_packages
+
+ sdk_name = d.getVar("TOOLCHAIN_OUTPUTNAME") + "-" + sdk_type
+ sdk_deploydir = Path(d.getVar("SDKDEPLOYDIR"))
+ sdk_spdxid = oe.sbom.get_sdk_spdxid(sdk_name)
+ sdk_packages = sdk_list_installed_packages(d, sdk_type == "target")
+ combine_spdx(d, sdk_name, sdk_deploydir, sdk_spdxid, sdk_packages, Path(d.getVar('SPDXSDKWORK')))
+
+def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx_workdir):
+ import os
+ import oe.spdx
+ import oe.sbom
+ import io
+ import json
+ from datetime import timezone, datetime
+ from pathlib import Path
+ import tarfile
+ import bb.compress.zstd
+
+ providers = collect_package_providers(d)
+ package_archs = d.getVar("SSTATE_ARCHS").split()
+ package_archs.reverse()
+
+ creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+ deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
+ source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
+
+ doc = oe.spdx.SPDXDocument()
+ doc.name = rootfs_name
+ doc.documentNamespace = get_doc_namespace(d, doc)
+ doc.creationInfo.created = creation_time
+ doc.creationInfo.comment = "This document was created by analyzing the source of the Yocto recipe during the build."
+ doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+ doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
+ doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
+ doc.creationInfo.creators.append("Person: N/A ()")
+
+ image = oe.spdx.SPDXPackage()
+ image.name = d.getVar("PN")
+ image.versionInfo = d.getVar("PV")
+ image.SPDXID = rootfs_spdxid
+ image.supplier = d.getVar("SPDX_SUPPLIER")
+
+ doc.packages.append(image)
+
+ for name in sorted(packages.keys()):
+ if name not in providers:
+ bb.fatal("Unable to find SPDX provider for '%s'" % name)
+
+ pkg_name, pkg_hashfn = providers[name]
+
+ pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn)
+ if not pkg_spdx_path:
+ bb.fatal("No SPDX file found for package %s, %s" % (pkg_name, pkg_hashfn))
+
+ pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
+
+ for p in pkg_doc.packages:
+ if p.name == name:
+ pkg_ref = oe.spdx.SPDXExternalDocumentRef()
+ pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name
+ pkg_ref.spdxDocument = pkg_doc.documentNamespace
+ pkg_ref.checksum.algorithm = "SHA1"
+ pkg_ref.checksum.checksumValue = pkg_doc_sha1
+
+ doc.externalDocumentRefs.append(pkg_ref)
+ doc.add_relationship(image, "CONTAINS", "%s:%s" % (pkg_ref.externalDocumentId, p.SPDXID))
+ break
+ else:
+ bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path))
+
+ runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn)
+ if not runtime_spdx_path:
+ bb.fatal("No runtime SPDX document found for %s, %s" % (name, pkg_hashfn))
+
+ runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path)
+
+ runtime_ref = oe.spdx.SPDXExternalDocumentRef()
+ runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name
+ runtime_ref.spdxDocument = runtime_doc.documentNamespace
+ runtime_ref.checksum.algorithm = "SHA1"
+ runtime_ref.checksum.checksumValue = runtime_doc_sha1
+
+ # "OTHER" isn't ideal here, but I can't find a relationship that makes sense
+ doc.externalDocumentRefs.append(runtime_ref)
+ doc.add_relationship(
+ image,
+ "OTHER",
+ "%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID),
+ comment="Runtime dependencies for %s" % name
+ )
+
+ image_spdx_path = spdx_workdir / (rootfs_name + ".spdx.json")
+
+ with image_spdx_path.open("wb") as f:
+ doc.to_json(f, sort_keys=True, indent=get_json_indent(d))
+
+ num_threads = int(d.getVar("BB_NUMBER_THREADS"))
+
+ visited_docs = set()
+
+ index = {"documents": []}
+
+ spdx_tar_path = rootfs_deploydir / (rootfs_name + ".spdx.tar.zst")
+ with bb.compress.zstd.open(spdx_tar_path, "w", num_threads=num_threads) as f:
+ with tarfile.open(fileobj=f, mode="w|") as tar:
+ def collect_spdx_document(path):
+ nonlocal tar
+ nonlocal deploy_dir_spdx
+ nonlocal source_date_epoch
+ nonlocal index
+
+ if path in visited_docs:
+ return
+
+ visited_docs.add(path)
+
+ with path.open("rb") as f:
+ doc, sha1 = oe.sbom.read_doc(f)
+ f.seek(0)
+
+ if doc.documentNamespace in visited_docs:
+ return
+
+ bb.note("Adding SPDX document %s" % path)
+ visited_docs.add(doc.documentNamespace)
+ info = tar.gettarinfo(fileobj=f)
+
+ info.name = doc.name + ".spdx.json"
+ info.uid = 0
+ info.gid = 0
+ info.uname = "root"
+ info.gname = "root"
+
+ if source_date_epoch is not None and info.mtime > int(source_date_epoch):
+ info.mtime = int(source_date_epoch)
+
+ tar.addfile(info, f)
+
+ index["documents"].append({
+ "filename": info.name,
+ "documentNamespace": doc.documentNamespace,
+ "sha1": sha1,
+ })
+
+ for ref in doc.externalDocumentRefs:
+ ref_path = oe.sbom.doc_find_by_namespace(deploy_dir_spdx, package_archs, ref.spdxDocument)
+ if not ref_path:
+ bb.fatal("Cannot find any SPDX file for document %s" % ref.spdxDocument)
+ collect_spdx_document(ref_path)
+
+ collect_spdx_document(image_spdx_path)
+
+ index["documents"].sort(key=lambda x: x["filename"])
+
+ index_str = io.BytesIO(json.dumps(
+ index,
+ sort_keys=True,
+ indent=get_json_indent(d),
+ ).encode("utf-8"))
+
+ info = tarfile.TarInfo()
+ info.name = "index.json"
+ info.size = len(index_str.getvalue())
+ info.uid = 0
+ info.gid = 0
+ info.uname = "root"
+ info.gname = "root"
+
+ tar.addfile(info, fileobj=index_str)
+
+combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS"
diff --git a/meta/classes/create-spdx.bbclass b/meta/classes/create-spdx.bbclass
index 19c6c0ff0b..b604973ae0 100644
--- a/meta/classes/create-spdx.bbclass
+++ b/meta/classes/create-spdx.bbclass
@@ -5,4 +5,4 @@
#
# Include this class when you don't care what version of SPDX you get; it will
# be updated to the latest stable version that is supported
-inherit create-spdx-2.2
+inherit create-spdx-3.0
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC][OE-core 2/7] oe/spdx: extend spdx.py objects
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 1/7] create-spdx-3.0: copy 2.2 class Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 3/7] oe/sbom: change the write_doc to prepare for spdx3 Marta Rybczynska
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Louis Rannou
From: Louis Rannou <louis.rannou@syslinbit.com>
Extend objects used to build the spdx scheme:
- add support for inheritance
- hide all attributes starting by _spdx
- add methods to list properties and item pairs
- improve the serializer to match the spdx3 scheme
Signed-off-by: Louis Rannou <louis.rannou@syslinbit.com>
---
meta/lib/oe/sbom.py | 2 +-
meta/lib/oe/spdx.py | 30 +++++++++++++++++++++++-------
2 files changed, 24 insertions(+), 8 deletions(-)
diff --git a/meta/lib/oe/sbom.py b/meta/lib/oe/sbom.py
index fd4b6895d8..824839378a 100644
--- a/meta/lib/oe/sbom.py
+++ b/meta/lib/oe/sbom.py
@@ -77,7 +77,7 @@ def write_doc(d, spdx_doc, arch, subdir, spdx_deploy=None, indent=None):
dest = doc_path(spdx_deploy, spdx_doc.name, arch, subdir)
dest.parent.mkdir(exist_ok=True, parents=True)
with dest.open("wb") as f:
- doc_sha1 = spdx_doc.to_json(f, sort_keys=True, indent=indent)
+ doc_sha1 = spdx_doc.to_json(f, sort_keys=False, indent=indent)
l = _doc_path_by_namespace(spdx_deploy, arch, spdx_doc.documentNamespace)
l.parent.mkdir(exist_ok=True, parents=True)
diff --git a/meta/lib/oe/spdx.py b/meta/lib/oe/spdx.py
index 7aaf2af5ed..97b9e011ad 100644
--- a/meta/lib/oe/spdx.py
+++ b/meta/lib/oe/spdx.py
@@ -145,9 +145,13 @@ class MetaSPDXObject(type):
def __new__(mcls, name, bases, attrs):
attrs["_properties"] = {}
- for key in attrs.keys():
- if isinstance(attrs[key], _Property):
- prop = attrs[key]
+ at = {}
+ for basecls in bases:
+ at.update(basecls._properties)
+ at.update(attrs)
+ for key in at.keys():
+ if isinstance(at[key], _Property):
+ prop = at[key]
attrs["_properties"][key] = prop
prop.set_property(attrs, key)
@@ -166,15 +170,27 @@ class SPDXObject(metaclass=MetaSPDXObject):
if name in d:
self._spdx[name] = prop.init(d[name])
- def serializer(self):
- return self._spdx
-
def __setattr__(self, name, value):
- if name in self._properties or name == "_spdx":
+ # All attributes must be in _properties or are hidden variables which
+ # must be prefixed with _spdx
+ if name in self._properties or name[:len("_spdx")] == "_spdx":
super().__setattr__(name, value)
return
raise KeyError("%r is not a valid SPDX property" % name)
+ def properties(self):
+ return self._properties.keys()
+
+ def items(self):
+ return self._properties.items()
+
+ def serializer(self, rootElement):
+ main = {"type": self.__class__.__name__[len("SPDX3"):]}
+ for (key, value) in self._spdx.items():
+ if key[0] == '_':
+ key = key[1:]
+ main.update({key: value})
+ return main
#
# These are the SPDX objects implemented from the spec. The *only* properties
# that can be added to these objects are ones directly specified in the SPDX
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC][OE-core 3/7] oe/sbom: change the write_doc to prepare for spdx3
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 1/7] create-spdx-3.0: copy 2.2 class Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 2/7] oe/spdx: extend spdx.py objects Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 4/7] create-spdx-3.0: SPDX3 objects as classes Marta Rybczynska
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core
Cc: richard.purdie, Louis Rannou, Marta Rybczynska, Samantha Jalabert
From: Louis Rannou <louis.rannou@syslinbit.com>
This changes the prototype of write_doc as the SPDX3 documentation does not
specify yet which is the root element.
Signed-off-by: Louis Rannou <louis.rannou@syslinbit.com>
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
---
meta/lib/oe/sbom.py | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/meta/lib/oe/sbom.py b/meta/lib/oe/sbom.py
index 824839378a..28db9cf719 100644
--- a/meta/lib/oe/sbom.py
+++ b/meta/lib/oe/sbom.py
@@ -68,7 +68,8 @@ def doc_path(spdx_deploy, doc_name, arch, subdir):
return spdx_deploy / arch / subdir / (doc_name + ".spdx.json")
-def write_doc(d, spdx_doc, arch, subdir, spdx_deploy=None, indent=None):
+# WARNING: Changed for SPDX3
+def write_doc(d, spdx_graph, spdx_doc, arch, subdir, spdx_deploy=None, indent=None):
from pathlib import Path
if spdx_deploy is None:
@@ -77,7 +78,7 @@ def write_doc(d, spdx_doc, arch, subdir, spdx_deploy=None, indent=None):
dest = doc_path(spdx_deploy, spdx_doc.name, arch, subdir)
dest.parent.mkdir(exist_ok=True, parents=True)
with dest.open("wb") as f:
- doc_sha1 = spdx_doc.to_json(f, sort_keys=False, indent=indent)
+ doc_sha1 = spdx_graph.to_json(f, sort_keys=False, indent=indent)
l = _doc_path_by_namespace(spdx_deploy, arch, spdx_doc.documentNamespace)
l.parent.mkdir(exist_ok=True, parents=True)
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC][OE-core 4/7] create-spdx-3.0: SPDX3 objects as classes
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
` (2 preceding siblings ...)
2023-10-26 10:48 ` [RFC][OE-core 3/7] oe/sbom: change the write_doc to prepare for spdx3 Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 5/7] oe/sbom: search into json Marta Rybczynska
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Louis Rannou, Samantha Jalabert
From: Louis Rannou <louis.rannou@syslinbit.com>
Create SPDX3 objects that classes as they are described in the SPDX3 model.
Signed-off-by: Louis Rannou <louis.rannou@syslinbit.com>
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
---
meta/lib/oe/spdx3.py | 385 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 385 insertions(+)
create mode 100644 meta/lib/oe/spdx3.py
diff --git a/meta/lib/oe/spdx3.py b/meta/lib/oe/spdx3.py
new file mode 100644
index 0000000000..ecbe999258
--- /dev/null
+++ b/meta/lib/oe/spdx3.py
@@ -0,0 +1,385 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+
+#
+# This library is intended to set the data types for the SPDX3 specification. It
+# is not intended to encode any particular OE specific behaviors, see the
+# sbom.py for that.
+#
+
+from oe.spdx import _String, _StringList, _Object, _ObjectList
+from oe.spdx import SPDXObject
+
+import json
+import hashlib
+
+class SPDX3Tool(SPDXObject):
+ pass
+
+class SPDX3Agent(SPDXObject):
+ pass
+
+#
+# Profile: Core - Enumerations
+#
+SPDX3HashAlgorithm = [
+ "blake2b256",
+ "blake2b384",
+ "blake2b512",
+ "blake3",
+ "crystalsKyber",
+ "crystalsDilithium",
+ "falcon",
+ "md2",
+ "md4",
+ "md5",
+ "md6",
+ "other",
+ "sha1",
+ "sha224",
+ "sha256",
+ "sha3_224",
+ "sha3_256",
+ "sha3_384",
+ "sha3_512",
+ "sha384",
+ "sha512",
+ "spdxPvcSha1",
+ "spdxPvcSha256",
+ "sphincsPlus"
+]
+
+#
+# Profile: Core - Datatypes
+#
+
+class SPDX3IntegrityMethod(SPDXObject):
+ comment = _String()
+
+class SPDX3Hash(SPDX3IntegrityMethod):
+ hashValue = _String()
+ algorithm = _String()
+
+#
+# Profile: Core - Classes
+#
+class SPDX3CreationInfo(SPDXObject):
+ specVersion = _String(default="3.0.0")
+ created = _String()
+ createdBy = _ObjectList(SPDX3Agent)
+ profile = _StringList(default=["core", "software"]) # TODO: not in creationInfo in spec
+ createdUsing = _ObjectList(SPDX3Tool)
+ dataLicense = _String(default="CC0-1.0")
+
+ def serializer(self):
+ """
+ Serialize a creationInfo element.
+ createdBy and createdUsing are only stored with their spdxId.
+ other attributes are ordinary serialized
+ """
+ main = {"type": self.__class__.__name__[len("SPDX3"):],
+ "createdBy": []}
+
+ main["createdBy"] = [c.spdxId for c in self._spdx["createdBy"]]
+ if "createdUsing" in self._spdx and len(self._spdx["createdUsing"]):
+ main["createdUsing"] = [c.spdxId for c in self._spdx["createdUsing"]]
+
+ for (key, value) in self._spdx.items():
+ if not key in ["createdBy", "createdUsing"]:
+ main.update({key: value})
+ return main
+
+class SPDX3ExternalMap(SPDXObject):
+ externalId = _String()
+ verifiedUsing = _ObjectList(SPDX3IntegrityMethod)
+ definingDocument = _String()
+
+class SPDX3Element(SPDXObject):
+ spdxId = _String(default="SPDXRef-DOCUMENT")
+ name = _String()
+ summary = _String()
+ description = _String()
+ creationInfo = _String()
+ verifiedUsing = _ObjectList(SPDX3IntegrityMethod)
+# packages = _ObjectList(SPDXPackage)
+# files = _ObjectList(SPDXFile)
+# relationships = _ObjectList(SPDXRelationship)
+# externalDocumentRefs = _ObjectList(SPDXExternalDocumentRef)
+# hasExtractedLicensingInfos = _ObjectList(SPDXExtractedLicensingInfo)
+
+ def serializer(self, rootElement):
+ """
+ Default serialization of an Element
+ creationInfo is moved to the root and refered with its id
+ context and element defined in ElementCollection and Bundle are ignored
+ Element objects are ignored
+ other attributes are ordinary serialized
+ """
+ main = {"type": self.__class__.__name__[len("SPDX3"):]}
+
+ for (key, value) in self._spdx.items():
+ if key == "creationInfo":
+ _id = rootElement.creationinfo(value)
+ main["creationInfo"] = _id
+ elif key not in ["context", "element"] \
+ and not isinstance(value, SPDX3Element):
+ if key[0] == '_':
+ main.update({key[1:]: value})
+ else:
+ main.update({key: value})
+ return main
+
+ def add_relationship(self, _from, relationship, _to):
+ if isinstance(_from, SPDX3Element):
+ from_spdxid = _from.spdxId
+ else:
+ from_spdxid = _from
+
+ if isinstance(_to, SPDX3Element):
+ to_spdxid = _to.spdxId
+ else:
+ to_spdxid = _to
+
+ for element in self.element:
+ if isinstance(element, SPDX3Relationship) \
+ and element._from == from_spdxid \
+ and element.relationshipType == relationship:
+ element.to.append(to_spdxid)
+ return element.spdxId
+
+ r = SPDX3Relationship(
+ _from=from_spdxid,
+ relationshipType=relationship,
+ to = [to_spdxid]
+ )
+
+ self.element.append(r)
+ return r.spdxId
+
+ def find_external_map(self, sourceDocumentNamespace):
+ for i in self.imports:
+ if i.definingDocument == sourceDocumentNamespace:
+ return i
+
+class SPDX3Relationship(SPDX3Element):
+ spdxId = _String(default="SPDXRef-Relationship") # TODO: increment id
+ _from = _String()
+ to = _StringList()
+ relationshipType = _String()
+
+class SPDX3Annotation(SPDX3Element):
+ spdxId = _String(default="SPDXRef-Annotation") # TODO: increment id
+ annotationType = _String()
+ statement = _String()
+ subject = _String()
+
+class SPDX3Agent(SPDX3Element):
+ pass
+
+class SPDX3Person(SPDX3Agent):
+ pass
+
+class SPDX3Organization(SPDX3Agent):
+ pass
+
+class SPDX3Tool(SPDX3Element):
+ pass
+
+class SPDX3Artifact(SPDX3Element):
+ suppliedBy = _ObjectList(SPDX3Agent)
+
+class SPDX3ElementCollection(SPDX3Element):
+ element = _ObjectList(SPDX3Element)
+ imports = _ObjectList(SPDX3ExternalMap)
+
+class SPDX3Bundle(SPDX3ElementCollection):
+ context = _String(default="")
+
+class SPDX3SpdxDocument(SPDX3Bundle):
+ documentNamespace = _String() # TODO: where is this definition ?
+ creationInfo = _Object(SPDX3CreationInfo)
+
+ _spdxcounter = 1
+
+ def __init__(self):
+ self._spdxcreationinfo = {}
+ super().__init__()
+
+ def creationinfo(self, c):
+ """
+ Look for a creationInfo in the dictionary. If it does not exist,
+ create a unique id and append it if it does not exist.
+ Return the id.
+ """
+ for (_id, info) in self._spdxcreationinfo.items():
+ if c == info:
+ return _id
+ _id = "_:CreationInfo{}".format(SPDX3SpdxDocument._spdxcounter)
+ SPDX3SpdxDocument._spdxcounter += 1
+ self._spdxcreationinfo[_id] = c
+ return _id
+
+ def serializer(self, rootElement):
+ """
+ Serialize a SpdxDocument element.
+ context has a specific serialization
+ attributes of type Element are moved to root
+ attributes are are ordinary serialized (context and element are ignored)
+ all elements are moved to root
+ """
+ chunk = {"@context": [self.context, {}]}
+ root = super().serializer(rootElement)
+ chunk["@graph"] = []
+
+ body = []
+ for (_, value) in self._spdx.items():
+ if isinstance(value, SPDX3Element):
+ body.append(value.serializer(rootElement))
+
+ if len(self.element):
+ root["element"] = []
+ for e in self.element:
+ root["element"].append(e.spdxId)
+ body.append(e.serializer(rootElement))
+
+ for (_id, c) in self._spdxcreationinfo.items():
+ cser = {"@id": _id}
+ cser.update(c.serializer())
+ chunk["@graph"].append(cser)
+
+ chunk["@graph"].append(root)
+ chunk["@graph"] = chunk["@graph"] + body
+
+ return chunk
+
+ def to_json(self, f, *, sort_keys=False, indent=None, separators=None):
+ class Encoder(json.JSONEncoder):
+ def __init__(self, rootElement=None, **kwargs):
+ self.rootElement = rootElement
+ super(Encoder, self).__init__(**kwargs)
+
+ def default(self, o):
+ if isinstance(o, SPDX3SpdxDocument):
+ return o.serializer(self.rootElement)
+ elif isinstance(o, SPDXObject):
+ root = o.serializer(self.rootElement)
+ return root
+
+ return super().default(o)
+
+ sha1 = hashlib.sha1()
+ for chunk in Encoder(
+ rootElement=self,
+ sort_keys=sort_keys,
+ indent=indent,
+ separators=separators,
+ ).iterencode(self):
+ chunk = chunk.encode("utf-8")
+ f.write(chunk)
+ sha1.update(chunk)
+
+ return sha1.hexdigest()
+
+ @classmethod
+ def from_json(cls, f, attributes=[]):
+ """
+ Look into a json file for all objects of given type. Return the document
+ element and a dictionary of required objects.
+ """
+ class Decoder(json.JSONDecoder):
+ def __init__(self, *args, **kwargs):
+ super().__init__(object_hook=self.object_hook, *args, **kwargs)
+
+ def object_hook(self, d):
+ if 'type' in d.keys():
+ if d['type'] in attributes or d['type'] == 'SpdxDocument':
+ return d
+ if '@graph' in d.keys():
+ spdxDocument = None
+ attr = {a: [] for a in attributes}
+ for p in d['@graph']:
+ if p is not None:
+ if p['type'] == 'SpdxDocument':
+ spdxDocument = p
+ else:
+ attr[p['type']].append(p)
+ return (spdxDocument, attr)
+
+ return json.load(f, cls=Decoder)
+
+#
+# Profile: Software - Datatypes
+#
+SPDX3SoftwarePurpose = [
+ "application",
+ "archive",
+ "bom",
+ "configuration",
+ "container",
+ "data",
+ "device",
+ "documentation",
+ "executable",
+ "file",
+ "firmware",
+ "framework",
+ "install",
+ "library",
+ "module",
+ "operatingSystem",
+ "patch",
+ "source",
+ "other"
+]
+
+class SPDX3SoftwareArtifact(SPDX3Artifact):
+ primaryPurpose = _String()
+ additionalPurpose = _StringList()
+
+class SPDX3Package(SPDX3SoftwareArtifact):
+ packageVersion = _String()
+ homePage = _String()
+ downloadLocation = _String()
+
+class SPDX3File(SPDX3SoftwareArtifact):
+ pass
+
+#
+# OpenEmbedded base class
+#
+class SPDX3Graph(SPDXObject):
+ # TODO: rework: graph should only have a list of objects and more
+ # intelligence in to_json
+ package = _Object(SPDX3Package)
+ creationInfo = _Object(SPDX3CreationInfo)
+ doc = _Object(SPDX3SpdxDocument)
+ tool = _Object(SPDX3Tool)
+ organization = _Object(SPDX3Organization)
+ person = _Object(SPDX3Person)
+
+ def __init__(self, **d):
+ super().__init__(**d)
+
+
+ def to_json(self, f, *, sort_keys=False, indent=None, separators=None):
+ class Encoder(json.JSONEncoder):
+ def default(self, o):
+ if isinstance(o, SPDXObject):
+ return o.serializer()
+
+ return super().default(o)
+
+ sha1 = hashlib.sha1()
+ for chunk in Encoder(
+ sort_keys=sort_keys,
+ indent=indent,
+ separators=separators,
+ ).iterencode(self):
+ chunk = chunk.encode("utf-8")
+ f.write(chunk)
+ sha1.update(chunk)
+
+ return sha1.hexdigest()
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC][OE-core 5/7] oe/sbom: search into json
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
` (3 preceding siblings ...)
2023-10-26 10:48 ` [RFC][OE-core 4/7] create-spdx-3.0: SPDX3 objects as classes Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 6/7] README.SPDX3: add file Marta Rybczynska
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Louis Rannou
From: Louis Rannou <louis.rannou@syslinbit.com>
Create a function that search into a json-ld instead of completely loading it.
Signed-off-by: Louis Rannou <louis.rannou@syslinbit.com>
---
meta/lib/oe/sbom.py | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)
diff --git a/meta/lib/oe/sbom.py b/meta/lib/oe/sbom.py
index 28db9cf719..21333c0a84 100644
--- a/meta/lib/oe/sbom.py
+++ b/meta/lib/oe/sbom.py
@@ -119,3 +119,35 @@ def read_doc(fn):
doc = oe.spdx.SPDXDocument.from_json(f)
return (doc, sha1.hexdigest())
+
+
+def search_doc(fn, attr_types=None):
+ """
+ Look for all attributes in the given dictionary. Return the document
+ element, a dictionary of the required attributes and the sha1 of the file.
+ """
+ import hashlib
+ import oe.spdx3
+ import io
+ import contextlib
+
+ @contextlib.contextmanager
+ def get_file():
+ if isinstance(fn, io.IOBase):
+ yield fn
+ else:
+ with fn.open("rb") as f:
+ yield f
+
+ with get_file() as f:
+ sha1 = hashlib.sha1()
+ while True:
+ chunk = f.read(4096)
+ if not chunk:
+ break
+ sha1.update(chunk)
+
+ f.seek(0)
+ doc, attributes = oe.spdx3.SPDX3SpdxDocument.from_json(f, attr_types or [])
+
+ return (doc, attributes, sha1.hexdigest())
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC][OE-core 6/7] README.SPDX3: add file
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
` (4 preceding siblings ...)
2023-10-26 10:48 ` [RFC][OE-core 5/7] oe/sbom: search into json Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 7/7] create-spdx-3.0: support for recipe spdx creation Marta Rybczynska
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Marta Rybczynska
Add a specific readme for SPDX3 with open questions and other notes
related to the PoC.
Signed-off-by: Marta Rybczynska <marta.rybczynska@syslinbit.com>
---
README.SPDX3 | 42 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
create mode 100644 README.SPDX3
diff --git a/README.SPDX3 b/README.SPDX3
new file mode 100644
index 0000000000..57f98756ab
--- /dev/null
+++ b/README.SPDX3
@@ -0,0 +1,42 @@
+This repository contains the Proof-of-Concept code for SPDX3 support
+in the Yocto Project.
+
+What does the code include:
+* The SPDX3 generation with JSON-LD serialization, still using .json extension
+* Implementations of the core, and software profiles
+
+Here are the known limitations:
+* At the time of writing this code, the SPDX3 specification is still undergoing
+ changes. Especially, the root element has not been yet decided. Because of
+ that, the code might require changes when the final specification is
+ released.
+
+* Some parts of the SPDX3 require clarifications. Current issues:
+ - Software.Package.homepage is sometiemes also called homePage: need to
+ confirm spelling
+ - Core.Relationship.from needs special care in Python as it conflicts
+ with a built-in
+ - should suppliedBy be serialized by an array or as a single string?
+ - In examples, SpdxDocument has an attribute namespace. It does not in the
+ documentation
+ - what is the equivalent of the documentNamespace that was in 2.2?
+
+* SPDX3 introduces modular model, where content depends on the profile used.
+ The configuration of profiles to generate needs to be reworked. Today,
+ generation is gated by variables shared with SPDX2.2 code like
+ SPDX_INCLUDE_SOURCES. In SPDX3 it could be done by enabling specific
+ profiles and variables like SPDX3_ENABLE_LICENSING or SPDX3_ENABLE_SECURITY.
+
+* The implementation includes data similar to the YP SPDX 2.2 content. SPDX 3.0
+ has additional profiles and fields that did not exist in the earier version.
+ The project needs a discussion on what is useful to include in the YP SPDX.
+ Additional profiles and classes might be implemented to carry that data.
+
+* The security profile implementation has been prototyped. However, some part
+ of the needed data is necessary from the cve-check database (for example:
+ CVSS). Obtaining the information is possible, but will require dependency on
+ the cve-check to download the database, then refactoring of the cve-check
+ database accesses so that they can be done from other classes while keeping
+ correct locks. Also, VulnAssessmentRelationship requires classification
+ of fixes as "Fixed", "NotAffected", while YP cve-check has only one category
+ for both. At the moment of writing this, there is a patch on the ML.
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [RFC][OE-core 7/7] create-spdx-3.0: support for recipe spdx creation
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
` (5 preceding siblings ...)
2023-10-26 10:48 ` [RFC][OE-core 6/7] README.SPDX3: add file Marta Rybczynska
@ 2023-10-26 10:48 ` Marta Rybczynska
2023-10-27 7:37 ` [OE-core] [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Luca Ceresoli
[not found] ` <1791E6C172CE81BE.2594@lists.openembedded.org>
8 siblings, 0 replies; 10+ messages in thread
From: Marta Rybczynska @ 2023-10-26 10:48 UTC (permalink / raw)
To: openembedded-core; +Cc: richard.purdie, Samantha Jalabert
From: Samantha Jalabert <samantha.jalabert@syslinbit.com>
Change functions and tasks to match the SPDX 3 model.
Signed-off-by: Samantha Jalabert <samantha.jalabert@syslinbit.com>
---
meta/classes/create-spdx-3.0.bbclass | 728 +++++++++------------------
1 file changed, 224 insertions(+), 504 deletions(-)
diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass
index b0aef80db1..ffe34969a8 100644
--- a/meta/classes/create-spdx-3.0.bbclass
+++ b/meta/classes/create-spdx-3.0.bbclass
@@ -11,7 +11,7 @@ DEPLOY_DIR_SPDX ??= "${DEPLOY_DIR}/spdx"
CVE_PRODUCT ??= "${BPN}"
CVE_VERSION ??= "${PV}"
-SPDXDIR ??= "${WORKDIR}/spdx"
+SPDXDIR ??= "${WORKDIR}/spdx-3.0"
SPDXDEPLOY = "${SPDXDIR}/deploy"
SPDXWORK = "${SPDXDIR}/work"
SPDXIMAGEWORK = "${SPDXDIR}/image-work"
@@ -64,21 +64,74 @@ def get_doc_namespace(d, doc):
namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS, d.getVar("SPDX_UUID_NAMESPACE"))
return "%s/%s-%s" % (d.getVar("SPDX_NAMESPACE_PREFIX"), doc.name, str(uuid.uuid5(namespace_uuid, doc.name)))
-def create_annotation(d, comment):
+def generate_creationInfo(d, document):
+ """
+ Generate the creationInfo and its elements for a document
+ """
from datetime import datetime, timezone
+ import oe.spdx3
creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
- annotation = oe.spdx.SPDXAnnotation()
- annotation.annotationDate = creation_time
- annotation.annotationType = "OTHER"
- annotation.annotator = "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION"))
- annotation.comment = comment
- return annotation
+
+ document.creationInfo = oe.spdx3.SPDX3CreationInfo()
+ document.creationInfo.specVersion = "3.0.0"
+ document.creationInfo.created = creation_time
+ document.creationInfo.dataLicense = "https://spdx.org/licenses/CC0-1.0"
+
+ tool = oe.spdx3.SPDX3Tool()
+ tool.name = "OpenEmbedded Core create-spdx.bbclass"
+ tool.spdxId = "spdx-" + d.getVar("PN") + ":SPDXRef-Actor-" + tool.name.replace(" ", "")
+ tool.creationInfo = document.creationInfo
+ document.element.append(tool)
+ document.creationInfo.createdUsing.append(tool)
+
+ organization = oe.spdx3.SPDX3Organization()
+ organization.name = d.getVar("SPDX_ORG")
+ organization.spdxId = "spdx-" + d.getVar("PN") + ":SPDXRef-Actor-" + organization.name.replace(" ", "")
+ organization.creationInfo = document.creationInfo
+ document.element.append(organization)
+ document.creationInfo.createdBy.append(organization)
+
+ person = oe.spdx3.SPDX3Person()
+ person.name = "Person: N/A ()"
+ person.spdxId = "spdx-" + d.getVar("PN") + ":SPDXRef-Actor-" + person.name.replace(" ", "")
+ document.creationInfo.createdBy.append(person)
+ document.element.append(person)
+
+def get_supplier(d, doc=None):
+ """
+ Get the supplier of a document or create it.
+ """
+ import oe.spdx3
+
+ supplier = d.getVar("SPDX_SUPPLIER")
+ agentName = supplier.split(": ")[1]
+ agentType = supplier.split(": ")[0]
+
+ if doc:
+ for element in doc.element:
+ if(isinstance(element, oe.spdx3.SPDX3Agent) and element.name == agentName):
+ return element
+
+ if(agentType == "Organization"):
+ agent = oe.spdx3.SPDX3Organization()
+ elif(agentType == "Person"):
+ agent = oe.spdx3.SPDX3Person()
+ else:
+ raise KeyError("%r is not a valid SPDX agent type" % agentType)
+
+ agent.name = agentName
+ agent.spdxId = "spdx-" + d.getVar("PN") + ":SPDXRef-Actor-" + agent.name.replace(" ", "")
+ agent.creationInfo = doc.creationInfo
+
+ return agent
def recipe_spdx_is_native(d, recipe):
- return any(a.annotationType == "OTHER" and
- a.annotator == "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION")) and
- a.comment == "isNative" for a in recipe.annotations)
+ return False
+# TODO: find a better way to mark native recipes
+# return any(a.annotationType == "OTHER" and
+# a.annotator == "Tool: %s - %s" % (d.getVar("SPDX_TOOL_NAME"), d.getVar("SPDX_TOOL_VERSION")) and
+# a.comment == "isNative" for a in recipe.annotations)
def is_work_shared_spdx(d):
return bb.data.inherits_class('kernel', d) or ('work-shared' in d.getVar('WORKDIR'))
@@ -113,7 +166,7 @@ def convert_license_to_spdx(lic, document, d, existing={}):
if name in extracted:
return
- extracted_info = oe.spdx.SPDXExtractedLicensingInfo()
+ extracted_info = oe.spdx.SPDX3ExtractedLicensingInfo()
extracted_info.name = name
extracted_info.licenseId = ident
extracted_info.extractedText = None
@@ -202,8 +255,7 @@ def process_sources(d):
def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archive=None, ignore_dirs=[], ignore_top_level_dirs=[]):
from pathlib import Path
- import oe.spdx
- import hashlib
+ import oe.spdx3
source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
if source_date_epoch:
@@ -223,11 +275,18 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
filename = str(filepath.relative_to(topdir))
if not filepath.is_symlink() and filepath.is_file():
- spdx_file = oe.spdx.SPDXFile()
- spdx_file.SPDXID = get_spdxid(file_counter)
- for t in get_types(filepath):
- spdx_file.fileTypes.append(t)
- spdx_file.fileName = filename
+ spdx_file = oe.spdx3.SPDX3File()
+ spdx_file.name = filename
+ spdx_file.spdxId = get_spdxid(file_counter)
+ spdx_file.primaryPurpose = None
+ spdx_file.additionalPurpose = []
+ types = get_types(filepath)
+ for t in types:
+ if t in oe.spdx3.SPDX3SoftwarePurpose:
+ if spdx_file.primaryPurpose == None:
+ spdx_file.primaryPurpose = t
+ else:
+ spdx_file.additionalPurpose.append(t)
if archive is not None:
with filepath.open("rb") as f:
@@ -245,42 +304,37 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv
sha1 = bb.utils.sha1_file(filepath)
sha1s.append(sha1)
- spdx_file.checksums.append(oe.spdx.SPDXChecksum(
- algorithm="SHA1",
- checksumValue=sha1,
- ))
- spdx_file.checksums.append(oe.spdx.SPDXChecksum(
- algorithm="SHA256",
- checksumValue=bb.utils.sha256_file(filepath),
- ))
-
- if "SOURCE" in spdx_file.fileTypes:
- extracted_lics = extract_licenses(filepath)
- if extracted_lics:
- spdx_file.licenseInfoInFiles = extracted_lics
-
- doc.files.append(spdx_file)
- doc.add_relationship(spdx_pkg, "CONTAINS", spdx_file)
- spdx_pkg.hasFiles.append(spdx_file.SPDXID)
- spdx_files.append(spdx_file)
+ hashSha1 = oe.spdx3.SPDX3Hash()
+ hashSha1.algorithm = "sha1"
+ hashSha1.hashValue = sha1
+ spdx_file.verifiedUsing.append(hashSha1)
- file_counter += 1
+ hashSha256 = oe.spdx3.SPDX3Hash()
+ hashSha256.algorithm = "sha256"
+ hashSha256.hashValue = bb.utils.sha256_file(filepath)
+ spdx_file.verifiedUsing.append(hashSha256)
+
+ # TODO: Rework when License Profile implemented
+ #if "SOURCE" in spdx_file.fileTypes:
+ # extracted_lics = extract_licenses(filepath)
+ # if extracted_lics:
+ # spdx_file.licenseInfoInFiles = extracted_lics
- sha1s.sort()
- verifier = hashlib.sha1()
- for v in sha1s:
- verifier.update(v.encode("utf-8"))
- spdx_pkg.packageVerificationCode.packageVerificationCodeValue = verifier.hexdigest()
+ doc.element.append(spdx_file)
+
+ doc.add_relationship(spdx_pkg, "contains", spdx_file)
+
+ spdx_files.append(spdx_file)
+ file_counter += 1
return spdx_files
def add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources):
from pathlib import Path
- import hashlib
import oe.packagedata
- import oe.spdx
+ import oe.spdx3
debug_search_paths = [
Path(d.getVar('PKGD')),
@@ -299,15 +353,15 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag
continue
for pkg_file in package_files:
- if file_path.lstrip("/") == pkg_file.fileName.lstrip("/"):
+ if file_path.lstrip("/") == pkg_file.name.lstrip("/"):
break
else:
bb.fatal("No package file found for %s in %s; SPDX found: %s" % (str(file_path), package,
- " ".join(p.fileName for p in package_files)))
+ " ".join(p.name for p in package_files)))
continue
for debugsrc in file_data["debugsrc"]:
- ref_id = "NOASSERTION"
+ ref_id = None
for search in debug_search_paths:
if debugsrc.startswith("/usr/src/kernel"):
debugsrc_path = search / debugsrc.replace('/usr/src/kernel/', '')
@@ -320,24 +374,32 @@ def add_package_sources_from_debug(d, package_doc, spdx_package, package, packag
if file_sha256 in sources:
source_file = sources[file_sha256]
-
- doc_ref = package_doc.find_external_document_ref(source_file.doc.documentNamespace)
+ doc_ref = package_doc.find_external_map(source_file.doc.documentNamespace)
if doc_ref is None:
- doc_ref = oe.spdx.SPDXExternalDocumentRef()
- doc_ref.externalDocumentId = "DocumentRef-dependency-" + source_file.doc.name
- doc_ref.spdxDocument = source_file.doc.documentNamespace
- doc_ref.checksum.algorithm = "SHA1"
- doc_ref.checksum.checksumValue = source_file.doc_sha1
- package_doc.externalDocumentRefs.append(doc_ref)
-
- ref_id = "%s:%s" % (doc_ref.externalDocumentId, source_file.file.SPDXID)
+ doc_ref = oe.spdx3.SPDX3ExternalMap()
+ doc_ref.externalId = "DocumentRef-dependency-" + source_file.doc.name
+ doc_ref.verifiedUsing = oe.spdx3.SPDX3Hash()
+ doc_ref.verifiedUsing.algorithm = "sha1"
+ doc_ref.verifiedUsing.hashValue = source_file.doc_sha1
+ doc_ref.definingDocument = source_file.doc.documentNamespace
+
+ package_doc.imports.append(doc_ref)
+
+ ref_id = "%s:%s" % (doc_ref.externalId, source_file.file.spdxId)
else:
bb.debug(1, "Debug source %s with SHA256 %s not found in any dependency" % (str(debugsrc_path), file_sha256))
break
else:
bb.debug(1, "Debug source %s not found" % debugsrc)
- package_doc.add_relationship(pkg_file, "GENERATED_FROM", ref_id, comment=debugsrc)
+ relation_id = package_doc.add_relationship(ref_id, "generates", pkg_file)
+ comment = oe.spdx3.SPDX3Annotation()
+ comment.subject = relation_id
+ comment.annotationType = "other"
+ comment.statement = "debugsrc"
+ package_doc.element.append(comment)
+
+ return
add_package_sources_from_debug[vardepsexclude] += "STAGING_KERNEL_DIR"
@@ -345,7 +407,7 @@ def collect_dep_recipes(d, doc, spdx_recipe):
import json
from pathlib import Path
import oe.sbom
- import oe.spdx
+ import oe.spdx3
deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
spdx_deps_file = Path(d.getVar("SPDXDEPS"))
@@ -362,10 +424,10 @@ def collect_dep_recipes(d, doc, spdx_recipe):
if not dep_recipe_path:
bb.fatal("Cannot find any SPDX file for recipe %s, %s" % (dep_pn, dep_hashfn))
- spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_recipe_path)
+ spdx_dep_doc, spdx_dep_pkg, spdx_dep_sha1 = oe.sbom.search_doc(dep_recipe_path, ["Package"])
- for pkg in spdx_dep_doc.packages:
- if pkg.name == dep_pn:
+ for pkg in spdx_dep_pkg['Package']:
+ if pkg["name"] == dep_pn:
spdx_dep_recipe = pkg
break
else:
@@ -373,19 +435,15 @@ def collect_dep_recipes(d, doc, spdx_recipe):
dep_recipes.append(oe.sbom.DepRecipe(spdx_dep_doc, spdx_dep_sha1, spdx_dep_recipe))
- dep_recipe_ref = oe.spdx.SPDXExternalDocumentRef()
- dep_recipe_ref.externalDocumentId = "DocumentRef-dependency-" + spdx_dep_doc.name
- dep_recipe_ref.spdxDocument = spdx_dep_doc.documentNamespace
- dep_recipe_ref.checksum.algorithm = "SHA1"
- dep_recipe_ref.checksum.checksumValue = spdx_dep_sha1
-
- doc.externalDocumentRefs.append(dep_recipe_ref)
+ dep_recipe_ref = oe.spdx3.SPDX3ExternalMap()
+ dep_recipe_ref.externalId = "DocumentRef-%s" % spdx_dep_doc["name"]
+ hashSha1 = oe.spdx3.SPDX3Hash()
+ hashSha1.algorithm = "sha1"
+ hashSha1.hashValue = spdx_dep_sha1
+ dep_recipe_ref.verifiedUsing.append(hashSha1)
- doc.add_relationship(
- "%s:%s" % (dep_recipe_ref.externalDocumentId, spdx_dep_recipe.SPDXID),
- "BUILD_DEPENDENCY_OF",
- spdx_recipe
- )
+ doc.imports.append(dep_recipe_ref)
+ doc.add_relationship("%s:%s" % (dep_recipe_ref.externalId, spdx_dep_recipe["spdxId"]), "buildDependency", spdx_recipe)
return dep_recipes
@@ -393,24 +451,35 @@ collect_dep_recipes[vardepsexclude] = "SSTATE_ARCHS"
def collect_dep_sources(d, dep_recipes):
import oe.sbom
+ import oe.spdx3
sources = {}
for dep in dep_recipes:
# Don't collect sources from native recipes as they
# match non-native sources also.
- if recipe_spdx_is_native(d, dep.recipe):
- continue
- recipe_files = set(dep.recipe.hasFiles)
-
- for spdx_file in dep.doc.files:
- if spdx_file.SPDXID not in recipe_files:
- continue
+ if hasattr(dep.doc, "element"):
+ for element in dep.doc.element:
+ if isinstance(element, oe.spdx3.SPDX3Annotation) \
+ and element.subject == dep.recipe.spdxId \
+ and element.statement == "isNative":
+ continue
- if "SOURCE" in spdx_file.fileTypes:
- for checksum in spdx_file.checksums:
- if checksum.algorithm == "SHA256":
- sources[checksum.checksumValue] = oe.sbom.DepSource(dep.doc, dep.doc_sha1, dep.recipe, spdx_file)
- break
+ recipe_files = []
+
+ if hasattr(dep.doc, "element"):
+ for element in dep.doc.element:
+ if isinstance(element, oe.spdx3.SPDX3Relationship) and element._from == dep.recipe.spdxId and element.relationshipType == "contains":
+ recipe_files = element.to
+
+ for element in dep.doc.element:
+ if isinstance(element, oe.spdx3.SPDX3File) \
+ and element.spdxId not in recipe_files \
+ and (element.primaryPurpose == "source" or "source" in element.additionalPurpose):
+ for checksum in element.verifiedUsing:
+ if algorithm in checksum.properties() \
+ and checksum.algorithm == "sha256":
+ sources[checksum.hashValue] = oe.sbom.DepSource(dep.doc, dep.doc_sha1, dep.recipe, spdx_file)
+ break
return sources
@@ -418,16 +487,16 @@ def add_download_packages(d, doc, recipe):
import os.path
from bb.fetch2 import decodeurl, CHECKSUM_LIST
import bb.process
- import oe.spdx
+ import oe.spdx3
import oe.sbom
for download_idx, src_uri in enumerate(d.getVar('SRC_URI').split()):
f = bb.fetch2.FetchData(src_uri, d)
for name in f.names:
- package = oe.spdx.SPDXPackage()
+ package = oe.spdx3.SPDX3Package()
package.name = "%s-source-%d" % (d.getVar("PN"), download_idx + 1)
- package.SPDXID = oe.sbom.get_download_spdxid(d, download_idx + 1)
+ package.spdxId = oe.sbom.get_download_spdxid(d, download_idx + 1)
if f.type == "file":
continue
@@ -443,42 +512,28 @@ def add_download_packages(d, doc, recipe):
if f.method.supports_checksum(f):
for checksum_id in CHECKSUM_LIST:
- if checksum_id.upper() not in oe.spdx.SPDXPackage.ALLOWED_CHECKSUMS:
+ if checksum_id not in oe.spdx3.SPDX3HashAlgorithm:
continue
expected_checksum = getattr(f, "%s_expected" % checksum_id)
if expected_checksum is None:
continue
- c = oe.spdx.SPDXChecksum()
+ c = oe.spdx3.SPDX3Hash()
c.algorithm = checksum_id.upper()
- c.checksumValue = expected_checksum
- package.checksums.append(c)
+ c.hashValue = expected_checksum
+ package.verifiedUsing.append(c)
package.downloadLocation = uri
- doc.packages.append(package)
- doc.add_relationship(doc, "DESCRIBES", package)
- # In the future, we might be able to do more fancy dependencies,
- # but this should be sufficient for now
- doc.add_relationship(package, "BUILD_DEPENDENCY_OF", recipe)
+ doc.element.append(package)
-def collect_direct_deps(d, dep_task):
- current_task = "do_" + d.getVar("BB_CURRENTTASK")
- pn = d.getVar("PN")
+ doc.add_relationship(doc, "describes", package)
+ doc.add_relationship(package, "buildDependency", recipe)
- taskdepdata = d.getVar("BB_TASKDEPDATA", False)
- for this_dep in taskdepdata.values():
- if this_dep[0] == pn and this_dep[1] == current_task:
- break
- else:
- bb.fatal(f"Unable to find this {pn}:{current_task} in taskdepdata")
+def collect_direct_deps(d, dep_task):
deps = set()
- for dep_name in this_dep[3]:
- dep_data = taskdepdata[dep_name]
- if dep_data[1] == dep_task and dep_data[0] != pn:
- deps.add((dep_data[0], dep_data[7]))
return sorted(deps)
@@ -509,9 +564,8 @@ do_collect_spdx_deps[deptask] = "do_create_spdx"
do_collect_spdx_deps[dirs] = "${SPDXDIR}"
python do_create_spdx() {
- from datetime import datetime, timezone
import oe.sbom
- import oe.spdx
+ import oe.spdx3
import uuid
from pathlib import Path
from contextlib import contextmanager
@@ -538,36 +592,34 @@ python do_create_spdx() {
include_sources = d.getVar("SPDX_INCLUDE_SOURCES") == "1"
archive_sources = d.getVar("SPDX_ARCHIVE_SOURCES") == "1"
archive_packaged = d.getVar("SPDX_ARCHIVE_PACKAGED") == "1"
- pkg_arch = d.getVar("SSTATE_PKGARCH")
-
- creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
- doc = oe.spdx.SPDXDocument()
+ doc = oe.spdx3.SPDX3SpdxDocument()
doc.name = "recipe-" + d.getVar("PN")
doc.documentNamespace = get_doc_namespace(d, doc)
- doc.creationInfo.created = creation_time
- doc.creationInfo.comment = "This document was created by analyzing recipe files during the build."
- doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
- doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
- doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
- doc.creationInfo.creators.append("Person: N/A ()")
-
- recipe = oe.spdx.SPDXPackage()
+ generate_creationInfo(d, doc)
+
+ recipe = oe.spdx3.SPDX3Package()
+ recipe.spdxId = oe.sbom.get_recipe_spdxid(d)
recipe.name = d.getVar("PN")
- recipe.versionInfo = d.getVar("PV")
- recipe.SPDXID = oe.sbom.get_recipe_spdxid(d)
- recipe.supplier = d.getVar("SPDX_SUPPLIER")
+ recipe.packageVersion = d.getVar("PV")
+ recipe.suppliedBy.append(get_supplier(d, doc))
+
if bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d):
- recipe.annotations.append(create_annotation(d, "isNative"))
+ comment = oe.spdx3.SPDX3Annotation()
+ comment.annotationType = "other"
+ comment.subject = recipe.spdxId
+ comment.statement = "isNative"
+
+ doc.element.append(comment)
homepage = d.getVar("HOMEPAGE")
if homepage:
- recipe.homepage = homepage
-
- license = d.getVar("LICENSE")
- if license:
- recipe.licenseDeclared = convert_license_to_spdx(license, doc, d)
+ recipe.homePage = homepage
+# TODO: Rework when License Profile implemented
+# license = d.getVar("LICENSE")
+# if license:
+# recipe.licenseDeclared = convert_license_to_spdx(license, doc, d)
summary = d.getVar("SUMMARY")
if summary:
@@ -581,26 +633,11 @@ python do_create_spdx() {
for var in d.getVar('SPDX_CUSTOM_ANNOTATION_VARS').split():
recipe.annotations.append(create_annotation(d, var + "=" + d.getVar(var)))
- # Some CVEs may be patched during the build process without incrementing the version number,
- # so querying for CVEs based on the CPE id can lead to false positives. To account for this,
- # save the CVEs fixed by patches to source information field in the SPDX.
- patched_cves = oe.cve_check.get_patched_cves(d)
- patched_cves = list(patched_cves)
- patched_cves = ' '.join(patched_cves)
- if patched_cves:
- recipe.sourceInfo = "CVEs fixed: " + patched_cves
-
- cpe_ids = oe.cve_check.get_cpe_ids(d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION"))
- if cpe_ids:
- for cpe_id in cpe_ids:
- cpe = oe.spdx.SPDXExternalReference()
- cpe.referenceCategory = "SECURITY"
- cpe.referenceType = "http://spdx.org/rdf/references/cpe23Type"
- cpe.referenceLocator = cpe_id
- recipe.externalRefs.append(cpe)
-
- doc.packages.append(recipe)
- doc.add_relationship(doc, "DESCRIBES", recipe)
+ # TODO: CVE handling
+
+ doc.element.append(recipe)
+
+ doc.add_relationship(doc, "describes", recipe)
add_download_packages(d, doc, recipe)
@@ -615,7 +652,7 @@ python do_create_spdx() {
recipe,
spdx_workdir,
lambda file_counter: "SPDXRef-SourceFile-%s-%d" % (d.getVar("PN"), file_counter),
- lambda filepath: ["SOURCE"],
+ lambda filepath: ["source"],
ignore_dirs=[".git"],
ignore_top_level_dirs=["temp"],
archive=archive,
@@ -626,17 +663,13 @@ python do_create_spdx() {
dep_recipes = collect_dep_recipes(d, doc, recipe)
- doc_sha1 = oe.sbom.write_doc(d, doc, pkg_arch, "recipes", indent=get_json_indent(d))
+ doc_sha1 = oe.sbom.write_doc(d, doc, doc, d.getVar("SSTATE_PKGARCH"), "recipes", indent=get_json_indent(d))
dep_recipes.append(oe.sbom.DepRecipe(doc, doc_sha1, recipe))
- recipe_ref = oe.spdx.SPDXExternalDocumentRef()
- recipe_ref.externalDocumentId = "DocumentRef-recipe-" + recipe.name
- recipe_ref.spdxDocument = doc.documentNamespace
- recipe_ref.checksum.algorithm = "SHA1"
- recipe_ref.checksum.checksumValue = doc_sha1
+ #TODO: references
sources = collect_dep_sources(d, dep_recipes)
- found_licenses = {license.name:recipe_ref.externalDocumentId + ":" + license.licenseId for license in doc.hasExtractedLicensingInfos}
+# found_licenses = {license.name:recipe_ref.externalDocumentId + ":" + license.licenseId for license in doc.hasExtractedLicensingInfos}
if not recipe_spdx_is_native(d, recipe):
bb.build.exec_func("read_subpackage_metadata", d)
@@ -646,42 +679,41 @@ python do_create_spdx() {
if not oe.packagedata.packaged(package, d):
continue
- package_doc = oe.spdx.SPDXDocument()
+ doc = oe.spdx3.SPDX3SpdxDocument()
pkg_name = d.getVar("PKG:%s" % package) or package
- package_doc.name = pkg_name
- package_doc.documentNamespace = get_doc_namespace(d, package_doc)
- package_doc.creationInfo.created = creation_time
- package_doc.creationInfo.comment = "This document was created by analyzing packages created during the build."
- package_doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
- package_doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
- package_doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
- package_doc.creationInfo.creators.append("Person: N/A ()")
- package_doc.externalDocumentRefs.append(recipe_ref)
+ doc.name = pkg_name
+ doc.documentNamespace = get_doc_namespace(d, doc)
+ generate_creationInfo(d, doc)
+
+ # TODO: Rework when License Profile implemented
+ # package_doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
+ # package_doc.externalDocumentRefs.append(recipe_ref)
package_license = d.getVar("LICENSE:%s" % package) or d.getVar("LICENSE")
- spdx_package = oe.spdx.SPDXPackage()
+ spdx_package = oe.spdx3.SPDX3Package()
- spdx_package.SPDXID = oe.sbom.get_package_spdxid(pkg_name)
+ spdx_package.spdxId = oe.sbom.get_package_spdxid(pkg_name)
spdx_package.name = pkg_name
- spdx_package.versionInfo = d.getVar("PV")
- spdx_package.licenseDeclared = convert_license_to_spdx(package_license, package_doc, d, found_licenses)
- spdx_package.supplier = d.getVar("SPDX_SUPPLIER")
+ spdx_package.packageVersion = d.getVar("PV")
+ # TODO: Rework when License Profile implemented
+ #spdx_package.licenseDeclared = convert_license_to_spdx(package_license, package_doc, d, found_licenses)
+ spdx_package.suppliedBy = [ d.getVar("SPDX_SUPPLIER") ]
- package_doc.packages.append(spdx_package)
+ doc.element.append(spdx_package)
- package_doc.add_relationship(spdx_package, "GENERATED_FROM", "%s:%s" % (recipe_ref.externalDocumentId, recipe.SPDXID))
- package_doc.add_relationship(package_doc, "DESCRIBES", spdx_package)
+ doc.add_relationship(recipe, "generates", spdx_package)
+ doc.add_relationship(doc, "describes", spdx_package)
- package_archive = deploy_dir_spdx / "packages" / (package_doc.name + ".tar.zst")
+ package_archive = deploy_dir_spdx / "packages" / (doc.name + ".tar.zst")
with optional_tarfile(package_archive, archive_packaged) as archive:
package_files = add_package_files(
d,
- package_doc,
+ doc,
spdx_package,
pkgdest / package,
lambda file_counter: oe.sbom.get_packaged_file_spdxid(pkg_name, file_counter),
- lambda filepath: ["BINARY"],
+ lambda filepath: ["executable"],
ignore_top_level_dirs=['CONTROL', 'DEBIAN'],
archive=archive,
)
@@ -689,9 +721,9 @@ python do_create_spdx() {
if archive is not None:
spdx_package.packageFileName = str(package_archive.name)
- add_package_sources_from_debug(d, package_doc, spdx_package, package, package_files, sources)
+ add_package_sources_from_debug(d, doc, spdx_package, package, package_files, sources)
- oe.sbom.write_doc(d, package_doc, pkg_arch, "packages", indent=get_json_indent(d))
+ oe.sbom.write_doc(d, doc, doc, d.getVar("SSTATE_PKGARCH"), "packages", indent=get_json_indent(d))
}
do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS"
# NOTE: depending on do_unpack is a hack that is necessary to get it's dependencies for archive the source
@@ -749,127 +781,11 @@ def collect_package_providers(d):
collect_package_providers[vardepsexclude] += "BB_TASKDEPDATA"
python do_create_runtime_spdx() {
- from datetime import datetime, timezone
- import oe.sbom
- import oe.spdx
- import oe.packagedata
- from pathlib import Path
-
- deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
- spdx_deploy = Path(d.getVar("SPDXRUNTIMEDEPLOY"))
- is_native = bb.data.inherits_class("native", d) or bb.data.inherits_class("cross", d)
-
- creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
-
- providers = collect_package_providers(d)
- pkg_arch = d.getVar("SSTATE_PKGARCH")
- package_archs = d.getVar("SSTATE_ARCHS").split()
- package_archs.reverse()
-
- if not is_native:
- bb.build.exec_func("read_subpackage_metadata", d)
-
- dep_package_cache = {}
-
- pkgdest = Path(d.getVar("PKGDEST"))
- for package in d.getVar("PACKAGES").split():
- localdata = bb.data.createCopy(d)
- pkg_name = d.getVar("PKG:%s" % package) or package
- localdata.setVar("PKG", pkg_name)
- localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + package)
-
- if not oe.packagedata.packaged(package, localdata):
- continue
-
- pkg_spdx_path = oe.sbom.doc_path(deploy_dir_spdx, pkg_name, pkg_arch, "packages")
-
- package_doc, package_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
-
- for p in package_doc.packages:
- if p.name == pkg_name:
- spdx_package = p
- break
- else:
- bb.fatal("Package '%s' not found in %s" % (pkg_name, pkg_spdx_path))
-
- runtime_doc = oe.spdx.SPDXDocument()
- runtime_doc.name = "runtime-" + pkg_name
- runtime_doc.documentNamespace = get_doc_namespace(localdata, runtime_doc)
- runtime_doc.creationInfo.created = creation_time
- runtime_doc.creationInfo.comment = "This document was created by analyzing package runtime dependencies."
- runtime_doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
- runtime_doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
- runtime_doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
- runtime_doc.creationInfo.creators.append("Person: N/A ()")
-
- package_ref = oe.spdx.SPDXExternalDocumentRef()
- package_ref.externalDocumentId = "DocumentRef-package-" + package
- package_ref.spdxDocument = package_doc.documentNamespace
- package_ref.checksum.algorithm = "SHA1"
- package_ref.checksum.checksumValue = package_doc_sha1
-
- runtime_doc.externalDocumentRefs.append(package_ref)
-
- runtime_doc.add_relationship(
- runtime_doc.SPDXID,
- "AMENDS",
- "%s:%s" % (package_ref.externalDocumentId, package_doc.SPDXID)
- )
-
- deps = bb.utils.explode_dep_versions2(localdata.getVar("RDEPENDS") or "")
- seen_deps = set()
- for dep, _ in deps.items():
- if dep in seen_deps:
- continue
-
- if dep not in providers:
- continue
-
- (dep, dep_hashfn) = providers[dep]
-
- if not oe.packagedata.packaged(dep, localdata):
- continue
-
- dep_pkg_data = oe.packagedata.read_subpkgdata_dict(dep, d)
- dep_pkg = dep_pkg_data["PKG"]
-
- if dep in dep_package_cache:
- (dep_spdx_package, dep_package_ref) = dep_package_cache[dep]
- else:
- dep_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, dep_pkg, dep_hashfn)
- if not dep_path:
- bb.fatal("No SPDX file found for package %s, %s" % (dep_pkg, dep_hashfn))
-
- spdx_dep_doc, spdx_dep_sha1 = oe.sbom.read_doc(dep_path)
-
- for pkg in spdx_dep_doc.packages:
- if pkg.name == dep_pkg:
- dep_spdx_package = pkg
- break
- else:
- bb.fatal("Package '%s' not found in %s" % (dep_pkg, dep_path))
-
- dep_package_ref = oe.spdx.SPDXExternalDocumentRef()
- dep_package_ref.externalDocumentId = "DocumentRef-runtime-dependency-" + spdx_dep_doc.name
- dep_package_ref.spdxDocument = spdx_dep_doc.documentNamespace
- dep_package_ref.checksum.algorithm = "SHA1"
- dep_package_ref.checksum.checksumValue = spdx_dep_sha1
-
- dep_package_cache[dep] = (dep_spdx_package, dep_package_ref)
-
- runtime_doc.externalDocumentRefs.append(dep_package_ref)
-
- runtime_doc.add_relationship(
- "%s:%s" % (dep_package_ref.externalDocumentId, dep_spdx_package.SPDXID),
- "RUNTIME_DEPENDENCY_OF",
- "%s:%s" % (package_ref.externalDocumentId, spdx_package.SPDXID)
- )
- seen_deps.add(dep)
-
- oe.sbom.write_doc(d, runtime_doc, pkg_arch, "runtime", spdx_deploy, indent=get_json_indent(d))
+ # TODO: implement for SPDX3
+ return
}
-do_create_runtime_spdx[vardepsexclude] += "OVERRIDES SSTATE_ARCHS"
+do_create_runtime_spdx[vardepsexclude] += "OVERRIDES"
addtask do_create_runtime_spdx after do_create_spdx before do_build do_rm_work
SSTATETASKS += "do_create_runtime_spdx"
@@ -950,209 +866,13 @@ POPULATE_SDK_POST_HOST_COMMAND:append:task-populate-sdk = " sdk_host_combine_spd
POPULATE_SDK_POST_TARGET_COMMAND:append:task-populate-sdk = " sdk_target_combine_spdx"
python image_combine_spdx() {
- import os
- import oe.sbom
- from pathlib import Path
- from oe.rootfs import image_list_installed_packages
-
- image_name = d.getVar("IMAGE_NAME")
- image_link_name = d.getVar("IMAGE_LINK_NAME")
- imgdeploydir = Path(d.getVar("IMGDEPLOYDIR"))
- img_spdxid = oe.sbom.get_image_spdxid(image_name)
- packages = image_list_installed_packages(d)
-
- combine_spdx(d, image_name, imgdeploydir, img_spdxid, packages, Path(d.getVar("SPDXIMAGEWORK")))
-
- def make_image_link(target_path, suffix):
- if image_link_name:
- link = imgdeploydir / (image_link_name + suffix)
- if link != target_path:
- link.symlink_to(os.path.relpath(target_path, link.parent))
-
- spdx_tar_path = imgdeploydir / (image_name + ".spdx.tar.zst")
- make_image_link(spdx_tar_path, ".spdx.tar.zst")
+ return
}
python sdk_host_combine_spdx() {
- sdk_combine_spdx(d, "host")
+ return
}
python sdk_target_combine_spdx() {
- sdk_combine_spdx(d, "target")
+ return
}
-
-def sdk_combine_spdx(d, sdk_type):
- import oe.sbom
- from pathlib import Path
- from oe.sdk import sdk_list_installed_packages
-
- sdk_name = d.getVar("TOOLCHAIN_OUTPUTNAME") + "-" + sdk_type
- sdk_deploydir = Path(d.getVar("SDKDEPLOYDIR"))
- sdk_spdxid = oe.sbom.get_sdk_spdxid(sdk_name)
- sdk_packages = sdk_list_installed_packages(d, sdk_type == "target")
- combine_spdx(d, sdk_name, sdk_deploydir, sdk_spdxid, sdk_packages, Path(d.getVar('SPDXSDKWORK')))
-
-def combine_spdx(d, rootfs_name, rootfs_deploydir, rootfs_spdxid, packages, spdx_workdir):
- import os
- import oe.spdx
- import oe.sbom
- import io
- import json
- from datetime import timezone, datetime
- from pathlib import Path
- import tarfile
- import bb.compress.zstd
-
- providers = collect_package_providers(d)
- package_archs = d.getVar("SSTATE_ARCHS").split()
- package_archs.reverse()
-
- creation_time = datetime.now(tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
- deploy_dir_spdx = Path(d.getVar("DEPLOY_DIR_SPDX"))
- source_date_epoch = d.getVar("SOURCE_DATE_EPOCH")
-
- doc = oe.spdx.SPDXDocument()
- doc.name = rootfs_name
- doc.documentNamespace = get_doc_namespace(d, doc)
- doc.creationInfo.created = creation_time
- doc.creationInfo.comment = "This document was created by analyzing the source of the Yocto recipe during the build."
- doc.creationInfo.licenseListVersion = d.getVar("SPDX_LICENSE_DATA")["licenseListVersion"]
- doc.creationInfo.creators.append("Tool: OpenEmbedded Core create-spdx.bbclass")
- doc.creationInfo.creators.append("Organization: %s" % d.getVar("SPDX_ORG"))
- doc.creationInfo.creators.append("Person: N/A ()")
-
- image = oe.spdx.SPDXPackage()
- image.name = d.getVar("PN")
- image.versionInfo = d.getVar("PV")
- image.SPDXID = rootfs_spdxid
- image.supplier = d.getVar("SPDX_SUPPLIER")
-
- doc.packages.append(image)
-
- for name in sorted(packages.keys()):
- if name not in providers:
- bb.fatal("Unable to find SPDX provider for '%s'" % name)
-
- pkg_name, pkg_hashfn = providers[name]
-
- pkg_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, pkg_name, pkg_hashfn)
- if not pkg_spdx_path:
- bb.fatal("No SPDX file found for package %s, %s" % (pkg_name, pkg_hashfn))
-
- pkg_doc, pkg_doc_sha1 = oe.sbom.read_doc(pkg_spdx_path)
-
- for p in pkg_doc.packages:
- if p.name == name:
- pkg_ref = oe.spdx.SPDXExternalDocumentRef()
- pkg_ref.externalDocumentId = "DocumentRef-%s" % pkg_doc.name
- pkg_ref.spdxDocument = pkg_doc.documentNamespace
- pkg_ref.checksum.algorithm = "SHA1"
- pkg_ref.checksum.checksumValue = pkg_doc_sha1
-
- doc.externalDocumentRefs.append(pkg_ref)
- doc.add_relationship(image, "CONTAINS", "%s:%s" % (pkg_ref.externalDocumentId, p.SPDXID))
- break
- else:
- bb.fatal("Unable to find package with name '%s' in SPDX file %s" % (name, pkg_spdx_path))
-
- runtime_spdx_path = oe.sbom.doc_find_by_hashfn(deploy_dir_spdx, package_archs, "runtime-" + name, pkg_hashfn)
- if not runtime_spdx_path:
- bb.fatal("No runtime SPDX document found for %s, %s" % (name, pkg_hashfn))
-
- runtime_doc, runtime_doc_sha1 = oe.sbom.read_doc(runtime_spdx_path)
-
- runtime_ref = oe.spdx.SPDXExternalDocumentRef()
- runtime_ref.externalDocumentId = "DocumentRef-%s" % runtime_doc.name
- runtime_ref.spdxDocument = runtime_doc.documentNamespace
- runtime_ref.checksum.algorithm = "SHA1"
- runtime_ref.checksum.checksumValue = runtime_doc_sha1
-
- # "OTHER" isn't ideal here, but I can't find a relationship that makes sense
- doc.externalDocumentRefs.append(runtime_ref)
- doc.add_relationship(
- image,
- "OTHER",
- "%s:%s" % (runtime_ref.externalDocumentId, runtime_doc.SPDXID),
- comment="Runtime dependencies for %s" % name
- )
-
- image_spdx_path = spdx_workdir / (rootfs_name + ".spdx.json")
-
- with image_spdx_path.open("wb") as f:
- doc.to_json(f, sort_keys=True, indent=get_json_indent(d))
-
- num_threads = int(d.getVar("BB_NUMBER_THREADS"))
-
- visited_docs = set()
-
- index = {"documents": []}
-
- spdx_tar_path = rootfs_deploydir / (rootfs_name + ".spdx.tar.zst")
- with bb.compress.zstd.open(spdx_tar_path, "w", num_threads=num_threads) as f:
- with tarfile.open(fileobj=f, mode="w|") as tar:
- def collect_spdx_document(path):
- nonlocal tar
- nonlocal deploy_dir_spdx
- nonlocal source_date_epoch
- nonlocal index
-
- if path in visited_docs:
- return
-
- visited_docs.add(path)
-
- with path.open("rb") as f:
- doc, sha1 = oe.sbom.read_doc(f)
- f.seek(0)
-
- if doc.documentNamespace in visited_docs:
- return
-
- bb.note("Adding SPDX document %s" % path)
- visited_docs.add(doc.documentNamespace)
- info = tar.gettarinfo(fileobj=f)
-
- info.name = doc.name + ".spdx.json"
- info.uid = 0
- info.gid = 0
- info.uname = "root"
- info.gname = "root"
-
- if source_date_epoch is not None and info.mtime > int(source_date_epoch):
- info.mtime = int(source_date_epoch)
-
- tar.addfile(info, f)
-
- index["documents"].append({
- "filename": info.name,
- "documentNamespace": doc.documentNamespace,
- "sha1": sha1,
- })
-
- for ref in doc.externalDocumentRefs:
- ref_path = oe.sbom.doc_find_by_namespace(deploy_dir_spdx, package_archs, ref.spdxDocument)
- if not ref_path:
- bb.fatal("Cannot find any SPDX file for document %s" % ref.spdxDocument)
- collect_spdx_document(ref_path)
-
- collect_spdx_document(image_spdx_path)
-
- index["documents"].sort(key=lambda x: x["filename"])
-
- index_str = io.BytesIO(json.dumps(
- index,
- sort_keys=True,
- indent=get_json_indent(d),
- ).encode("utf-8"))
-
- info = tarfile.TarInfo()
- info.name = "index.json"
- info.size = len(index_str.getvalue())
- info.uid = 0
- info.gid = 0
- info.uname = "root"
- info.gname = "root"
-
- tar.addfile(info, fileobj=index_str)
-
-combine_spdx[vardepsexclude] += "BB_NUMBER_THREADS SSTATE_ARCHS"
--
2.42.0
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [OE-core] [RFC][OE-core 0/7] SPDX3 Proof-of-Concept
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
` (6 preceding siblings ...)
2023-10-26 10:48 ` [RFC][OE-core 7/7] create-spdx-3.0: support for recipe spdx creation Marta Rybczynska
@ 2023-10-27 7:37 ` Luca Ceresoli
[not found] ` <1791E6C172CE81BE.2594@lists.openembedded.org>
8 siblings, 0 replies; 10+ messages in thread
From: Luca Ceresoli @ 2023-10-27 7:37 UTC (permalink / raw)
To: Marta Rybczynska; +Cc: openembedded-core, richard.purdie, Marta Rybczynska
Hello Marta,
On Thu, 26 Oct 2023 12:48:40 +0200
"Marta Rybczynska" <rybczynska@gmail.com> wrote:
> This patch-set adds a proof-of-concept implementation of the upcoming
> SPDX3 standard to the SBOM generation of the Yocto Project/OpenEmbedded.
>
> The current code delivers an equivalent of what is produced for SPDX2.2.
> The standard has not been released yet, and there is some specification
> work in progress still. Our questions and open points are available
> in the README.SPDX3 file.
>
> Also, this first RFC delivery will be followed by another one with
> SPDX assembly and the Licensing profile.
I ran this series on the autobuilders and it's triggering:
2023-10-26 22:30:42,532 - oe-selftest - INFO - 2: 27/33 214/548 (53.63s) (0 failed) (spdx.SPDXCheck.test_spdx_base_files)
2023-10-26 22:30:42,532 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/spdx.py", line 54, in test_spdx_base_files
self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files")
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/spdx.py", line 51, in check_recipe_spdx
result = check_spdx_json(full_file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/spdx.py", line 44, in check_spdx_json
self.assertNotEqual(report["SPDXID"], None)
~~~~~~^^^^^^^^^^
KeyError: 'SPDXID'
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5969/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5969/steps/14/logs/stdio
Luca
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OE-core] [RFC][OE-core 0/7] SPDX3 Proof-of-Concept
[not found] ` <1791E6C172CE81BE.2594@lists.openembedded.org>
@ 2023-10-27 7:40 ` Luca Ceresoli
0 siblings, 0 replies; 10+ messages in thread
From: Luca Ceresoli @ 2023-10-27 7:40 UTC (permalink / raw)
To: Luca Ceresoli via lists.openembedded.org
Cc: luca.ceresoli, Marta Rybczynska, openembedded-core,
richard.purdie, Marta Rybczynska
On Fri, 27 Oct 2023 09:37:28 +0200
"Luca Ceresoli via lists.openembedded.org"
<luca.ceresoli=bootlin.com@lists.openembedded.org> wrote:
> Hello Marta,
>
> On Thu, 26 Oct 2023 12:48:40 +0200
> "Marta Rybczynska" <rybczynska@gmail.com> wrote:
>
> > This patch-set adds a proof-of-concept implementation of the upcoming
> > SPDX3 standard to the SBOM generation of the Yocto Project/OpenEmbedded.
> >
> > The current code delivers an equivalent of what is produced for SPDX2.2.
> > The standard has not been released yet, and there is some specification
> > work in progress still. Our questions and open points are available
> > in the README.SPDX3 file.
> >
> > Also, this first RFC delivery will be followed by another one with
> > SPDX assembly and the Licensing profile.
>
> I ran this series on the autobuilders and it's triggering:
>
> 2023-10-26 22:30:42,532 - oe-selftest - INFO - 2: 27/33 214/548 (53.63s) (0 failed) (spdx.SPDXCheck.test_spdx_base_files)
> 2023-10-26 22:30:42,532 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
> File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/spdx.py", line 54, in test_spdx_base_files
> self.check_recipe_spdx("packages", "base-files.spdx.json", "base-files")
> File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/spdx.py", line 51, in check_recipe_spdx
> result = check_spdx_json(full_file_path)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/lib/oeqa/selftest/cases/spdx.py", line 44, in check_spdx_json
> self.assertNotEqual(report["SPDXID"], None)
> ~~~~~~^^^^^^^^^^
> KeyError: 'SPDXID'
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5969/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5969/steps/14/logs/stdio
And also, noticed after sending the previous e-mail:
ERROR: libtool-native-2.4.7-r0 do_create_spdx: Recipe libtool-native is trying to create package libltdl which was already written by recipe libtool. This will cause corruption, please resolve this and only provide the package from one recipe or the other or only build one of the recipes.
ERROR: quilt-native-0.67-r0 do_create_spdx: Recipe quilt-native is trying to create package guards which was already written by recipe quilt. This will cause corruption, please resolve this and only provide the package from one recipe or the other or only build one of the recipes.
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/7390/steps/16/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/5639/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/96/builds/5527/steps/12/logs/stdio
Luca
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2023-10-27 7:40 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-26 10:48 [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 1/7] create-spdx-3.0: copy 2.2 class Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 2/7] oe/spdx: extend spdx.py objects Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 3/7] oe/sbom: change the write_doc to prepare for spdx3 Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 4/7] create-spdx-3.0: SPDX3 objects as classes Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 5/7] oe/sbom: search into json Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 6/7] README.SPDX3: add file Marta Rybczynska
2023-10-26 10:48 ` [RFC][OE-core 7/7] create-spdx-3.0: support for recipe spdx creation Marta Rybczynska
2023-10-27 7:37 ` [OE-core] [RFC][OE-core 0/7] SPDX3 Proof-of-Concept Luca Ceresoli
[not found] ` <1791E6C172CE81BE.2594@lists.openembedded.org>
2023-10-27 7:40 ` Luca Ceresoli
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox