[PATCH 0/3] spdx30: Fix two bugs in the source download enrichment

[PATCH 0/3] spdx30: Fix two bugs in the source download enrichment

[Stefano Tondo]

Two fixes for the source download enrichment added in 9f1825e74d
("spdx30: Add Git version and PURL to source downloads"):

- _enrich_source_package() and _generate_git_purl() start with an
  underscore, so bitbake leaves them out of the do_create_spdx
  signature, and SPDX_GIT_PURL_MAPPINGS with them. Changing the
  mappings restored the previous SPDX from sstate.

- gitsm downloads got no version and no PURL, because only fd.type
  "git" was handled.

Patch 1 changes the do_create_spdx signature of every recipe once.

Patch 3 adds one selftest per fix. Each fails on master without its
fix. With the series, all SPDX30Check selftests pass.

Stefano Tondo (3):
  spdx30: Make source download enrichment part of the task signature
  spdx30: Enrich gitsm source downloads like git ones
  oeqa/selftest: Add tests for source download enrichment

 meta/lib/oe/spdx30_tasks.py          | 10 +++---
 meta/lib/oeqa/selftest/cases/spdx.py | 47 ++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+), 5 deletions(-)

-- 
2.56.0

[PATCH 1/3] spdx30: Make source download enrichment part of the task signature

[Stefano Tondo]

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

bitbake does not track module functions whose names start with an
underscore when it computes task signatures. _enrich_source_package()
and _generate_git_purl() were therefore not part of the do_create_spdx
signature, and neither was SPDX_GIT_PURL_MAPPINGS, which
_generate_git_purl() reads. Changing the mappings or this code did not
rerun do_create_spdx, and the previous SPDX was restored from sstate.

Rename both functions so that bitbake tracks them. This changes the
do_create_spdx signature of every recipe once.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oe/spdx30_tasks.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 646ec7d074..803a9e05a9 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -358,7 +358,7 @@ def collect_dep_sources(dep_objsets, dest):
             index_sources_by_hash(e.to, dest)
 
 
-def _generate_git_purl(d, download_location, srcrev):
+def generate_git_purl(d, download_location, srcrev):
     """Generate a Package URL for a Git source from its download location.
 
     Parses the Git URL to identify the hosting service and generates the
@@ -416,7 +416,7 @@ def _generate_git_purl(d, download_location, srcrev):
     return None
 
 
-def _enrich_source_package(d, dl, fd, file_name, primary_purpose):
+def enrich_source_package(d, dl, fd, file_name, primary_purpose):
     """Enrich a Git source download package with version, PURL, and external refs.
 
     For Git sources, extracts the full SHA-1 from SRCREV as the version,
@@ -435,7 +435,7 @@ def _enrich_source_package(d, dl, fd, file_name, primary_purpose):
         # Generate PURL for Git hosting services
         download_location = getattr(dl, "software_downloadLocation", None)
         if version and download_location:
-            purl = _generate_git_purl(d, download_location, version)
+            purl = generate_git_purl(d, download_location, version)
 
     if version:
         dl.software_packageVersion = version
@@ -524,7 +524,7 @@ def add_download_files(d, objset):
                 )
             )
 
-            _enrich_source_package(d, dl, fd, file_name, primary_purpose)
+            enrich_source_package(d, dl, fd, file_name, primary_purpose)
 
             if release_date is not None:
                 dl.releaseTime = release_date
-- 
2.56.0

[PATCH 2/3] spdx30: Enrich gitsm source downloads like git ones

[Stefano Tondo]

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

The source download enrichment only handled fd.type "git", so gitsm
downloads got the VCS external reference but neither the SRCREV as
version nor the PURL. The gitsm fetcher is a git fetcher with
submodule support and carries the same revision, so treat it the same.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oe/spdx30_tasks.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/spdx30_tasks.py b/meta/lib/oe/spdx30_tasks.py
index 803a9e05a9..7f3280a763 100644
--- a/meta/lib/oe/spdx30_tasks.py
+++ b/meta/lib/oe/spdx30_tasks.py
@@ -426,7 +426,7 @@ def enrich_source_package(d, dl, fd, file_name, primary_purpose):
     version = None
     purl = None
 
-    if fd.type == "git":
+    if fd.type in ("git", "gitsm"):
         # Use full SHA-1 from fd.revision
         srcrev = getattr(fd, "revision", None)
         if srcrev and srcrev not in {"${AUTOREV}", "AUTOINC", "INVALID"}:
-- 
2.56.0

[PATCH 3/3] oeqa/selftest: Add tests for source download enrichment

[Stefano Tondo]

From: Stefano Tondo <stefano.tondo.ext@siemens.com>

test_gitsm_source_version builds git-submodule-test, which uses gitsm,
and checks that its source download carries the SRCREV as version.

test_git_purl_mappings_rerun_create_spdx builds fastfloat twice, first
with the default mapping and then with github.com mapped to another
PURL type, and checks that the second SPDX carries the new PURL. Before
the enrichment became part of the task signature, the second build
restored the first SPDX from sstate.

Signed-off-by: Stefano Tondo <stefano.tondo.ext@siemens.com>
---
 meta/lib/oeqa/selftest/cases/spdx.py | 47 ++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/spdx.py b/meta/lib/oeqa/selftest/cases/spdx.py
index c900558b44..e5d6715aa9 100644
--- a/meta/lib/oeqa/selftest/cases/spdx.py
+++ b/meta/lib/oeqa/selftest/cases/spdx.py
@@ -445,6 +445,53 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase):
                 f"Version '{version}' for package '{name}' should contain digits"
             )
 
+    def test_gitsm_source_version(self):
+        """Test that gitsm source downloads are enriched like git ones.
+
+        git-submodule-test is fetched with gitsm://, so its source download
+        must carry the SRCREV as version.
+        """
+        objset = self.check_recipe_spdx(
+            "git-submodule-test",
+            "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-git-submodule-test.spdx.json",
+        )
+
+        srcrev = get_bb_var("SRCREV", "git-submodule-test")
+        for pkg in objset.foreach_type(oe.spdx30.software_Package):
+            location = pkg.software_downloadLocation or ""
+            if location.startswith("git+https://git.yoctoproject.org/git-submodule-test"):
+                self.assertEqual(pkg.software_packageVersion, srcrev)
+                break
+        else:
+            self.fail("git-submodule-test gitsm source download not found")
+
+    def test_git_purl_mappings_rerun_create_spdx(self):
+        """Test that SPDX_GIT_PURL_MAPPINGS is part of the do_create_spdx signature.
+
+        fastfloat is fetched from github.com. Build it with the default
+        mapping, then map github.com to another PURL type: do_create_spdx
+        must run again instead of restoring the first result from sstate.
+        """
+        spdx_path = "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-fastfloat.spdx.json"
+        location_prefix = "git+https://github.com/fastfloat/fast_float"
+        srcrev = get_bb_var("SRCREV", "fastfloat")
+
+        def source_purl(objset):
+            for pkg in objset.foreach_type(oe.spdx30.software_Package):
+                if (pkg.software_downloadLocation or "").startswith(location_prefix):
+                    return pkg.software_packageUrl
+            self.fail("fastfloat git source download not found")
+
+        objset = self.check_recipe_spdx("fastfloat", spdx_path)
+        self.assertEqual(source_purl(objset), f"pkg:github/fastfloat/fast_float@{srcrev}")
+
+        objset = self.check_recipe_spdx(
+            "fastfloat",
+            spdx_path,
+            extraconf='SPDX_GIT_PURL_MAPPINGS = "github.com:pkg:selftest"\n',
+        )
+        self.assertEqual(source_purl(objset), f"pkg:selftest/fastfloat/fast_float@{srcrev}")
+
     def test_release_date_source_date_epoch(self):
         """releaseTime should be derived from SDE_FILE via epochfile_read().
 
-- 
2.56.0