[OE-core][PATCH] lib: Fix dependencies on SPDX code

[OE-core][PATCH] lib: Fix dependencies on SPDX code

[Joshua Watt]

The SPDX library code was being ignored from taskhash calculations due
to accidentally being omitted from BBIMPORTS. This meant that changes in
the code or dependent variables would not cause the task to rebuild
correctly.

In order to add spdx_common, convert the `Dep` object from a named tuple
to a frozen dataclass. These function more or less equivalently, but the
bitbake code parser cannot handle named tuples.

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/lib/oe/__init__.py    | 2 +-
 meta/lib/oe/spdx_common.py | 8 ++++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/__init__.py b/meta/lib/oe/__init__.py
index d7604812834..3179a3f3d2c 100644
--- a/meta/lib/oe/__init__.py
+++ b/meta/lib/oe/__init__.py
@@ -11,4 +11,4 @@ __path__ = extend_path(__path__, __name__)
 # processed correctly (e.g. qa)
 BBIMPORTS = ["qa", "data", "path", "utils", "types", "package", "packagedata", \
              "packagegroup", "sstatesig", "lsb", "cachedpath", "license", \
-             "reproducible", "rust", "buildcfg", "go"]
+             "reproducible", "rust", "buildcfg", "go", "spdx30_tasks", "spdx_common"]
diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
index 23a17271d6c..e1b26edaaf6 100644
--- a/meta/lib/oe/spdx_common.py
+++ b/meta/lib/oe/spdx_common.py
@@ -12,7 +12,7 @@ import re
 import shutil
 
 from pathlib import Path
-
+from dataclasses import dataclass
 
 LIC_REGEX = re.compile(
     rb"^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$",
@@ -77,7 +77,11 @@ def process_sources(d):
     return True
 
 
-Dep = collections.namedtuple("Dep", ["pn", "hashfn", "in_taskhash"])
+@dataclass(frozen=True)
+class Dep(object):
+    pn: str
+    hashfn: str
+    in_taskhash: bool
 
 
 def collect_direct_deps(d, dep_task):
-- 
2.47.1

Re: [OE-core][PATCH] lib: Fix dependencies on SPDX code

[Mathieu Dubois-Briand]

On Fri Mar 7, 2025 at 4:43 PM CET, Joshua Watt via lists.openembedded.org wrote:
> The SPDX library code was being ignored from taskhash calculations due
> to accidentally being omitted from BBIMPORTS. This meant that changes in
> the code or dependent variables would not cause the task to rebuild
> correctly.
>
> In order to add spdx_common, convert the `Dep` object from a named tuple
> to a frozen dataclass. These function more or less equivalently, but the
> bitbake code parser cannot handle named tuples.
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---

Hi Joshua,

I believe this is triggering some issue on the autobuilder:

2025-03-10 09:45:19,401 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 482, in test_sstate_allarch_samesigs
    self.sstate_common_samesigs(configA, configB, allarch=True)
  File "/srv/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/sstatetests.py", line 213, in sstate_common_samesigs
    self.assertEqual(files1, files2)
  File "/usr/lib/python3.12/unittest/case.py", line 885, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.12/unittest/case.py", line 1184, in assertDictEqual
    self.fail(self._formatMessage(msg, standardMsg))
  File "/usr/lib/python3.12/unittest/case.py", line 715, in fail
    raise self.failureException(msg)
AssertionError: {'nat[215727 chars]x': 'd8ed538990588f9d37804e867be98c9b4809de8f2[88942 chars]d1a'} != {'nat[215727 chars]x': '481bab8ce0c89a635d3ec3f3c4a28c0b5a67b2c23[88942 chars]d1a'}

https://autobuilder.yoctoproject.org/valkyrie/api/v2/logs/1715394/raw_inline

Can you have a look at this error please?


-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[OE-core][PATCH v2] lib: Fix dependencies on SPDX code

[Joshua Watt]

The SPDX library code was being ignored from taskhash calculations due
to accidentally being omitted from BBIMPORTS. This meant that changes in
the code or dependent variables would not cause the task to rebuild
correctly.

In order to add spdx_common, convert the `Dep` object from a named tuple
to a frozen dataclass. These function more or less equivalently, but the
bitbake code parser cannot handle named tuples.

Finally, the vardepsexclude that used to be present on the recipe tasks
needs to be moved to the python code in order for the variables to be
correctly ignored. Several unused exclusions were removed

Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
 meta/classes/create-spdx-3.0.bbclass | 3 +--
 meta/lib/oe/__init__.py              | 2 +-
 meta/lib/oe/spdx_common.py           | 8 ++++++--
 3 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/meta/classes/create-spdx-3.0.bbclass b/meta/classes/create-spdx-3.0.bbclass
index 25f3aa5f433..b4a5156e709 100644
--- a/meta/classes/create-spdx-3.0.bbclass
+++ b/meta/classes/create-spdx-3.0.bbclass
@@ -134,7 +134,6 @@ python do_create_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_spdx(d)
 }
-do_create_spdx[vardepsexclude] += "BB_NUMBER_THREADS SPDX_BUILD_HOST"
 do_create_spdx[vardeps] += "\
     SPDX_INCLUDE_BITBAKE_PARENT_BUILD \
     SPDX_PACKAGE_ADDITIONAL_PURPOSE \
@@ -170,7 +169,7 @@ python do_create_package_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_package_spdx(d)
 }
-do_create_package_spdx[vardepsexclude] += "OVERRIDES SPDX_MULTILIB_SSTATE_ARCHS"
+oe.spdx30_tasks.create_package_spdx[vardepsexclude] = "OVERRIDES"
 
 addtask do_create_package_spdx after do_create_spdx before do_build do_rm_work
 SSTATETASKS += "do_create_package_spdx"
diff --git a/meta/lib/oe/__init__.py b/meta/lib/oe/__init__.py
index d7604812834..3179a3f3d2c 100644
--- a/meta/lib/oe/__init__.py
+++ b/meta/lib/oe/__init__.py
@@ -11,4 +11,4 @@ __path__ = extend_path(__path__, __name__)
 # processed correctly (e.g. qa)
 BBIMPORTS = ["qa", "data", "path", "utils", "types", "package", "packagedata", \
              "packagegroup", "sstatesig", "lsb", "cachedpath", "license", \
-             "reproducible", "rust", "buildcfg", "go"]
+             "reproducible", "rust", "buildcfg", "go", "spdx30_tasks", "spdx_common"]
diff --git a/meta/lib/oe/spdx_common.py b/meta/lib/oe/spdx_common.py
index 23a17271d6c..e1b26edaaf6 100644
--- a/meta/lib/oe/spdx_common.py
+++ b/meta/lib/oe/spdx_common.py
@@ -12,7 +12,7 @@ import re
 import shutil
 
 from pathlib import Path
-
+from dataclasses import dataclass
 
 LIC_REGEX = re.compile(
     rb"^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$",
@@ -77,7 +77,11 @@ def process_sources(d):
     return True
 
 
-Dep = collections.namedtuple("Dep", ["pn", "hashfn", "in_taskhash"])
+@dataclass(frozen=True)
+class Dep(object):
+    pn: str
+    hashfn: str
+    in_taskhash: bool
 
 
 def collect_direct_deps(d, dep_task):
-- 
2.47.1