[bitbake][scarthgap][2.8][PATCH 0/2] Patch review

[bitbake][scarthgap][2.8][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of changes for 2.8/scarthgap and have comments back by
end of day Monday, July 21

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7150

The following changes since commit 8714a02e13477a9d97858b3642e05f28247454b5:

  fetch2/wget: Fix failure path for files that are empty or don't exist (2024-06-14 08:48:23 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (2):
  codeparser/data: Ensure module function contents changing is accounted
    for
  codeparser: Skip non-local functions for module dependencies

 lib/bb/codeparser.py | 33 ++++++++++++++++++++++-----------
 lib/bb/data.py       |  2 +-
 2 files changed, 23 insertions(+), 12 deletions(-)

-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 1/2] codeparser/data: Ensure module function contents changing is accounted for

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Currently, if a pylib function changes contents, the taskhash remains
unchanged since we assume the functions have stable output. This is
probably a poor assumption so take the code of the function into account
in the taskhashes. This avoids certain frustrating build failures we've
been seeing in automated testing.

To make this work we have to add an extra entry to the python code parsing
cache so that we can store the hashed function contents for efficiency as
in the python module case, that isn't used as the key to the cache.

The cache version changes since we're adding data to the cache.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b2c3438ebe62793ebabe2c282534893908d520b4)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/codeparser.py | 28 +++++++++++++++++-----------
 lib/bb/data.py       |  2 +-
 2 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
index 2e8b7ced3..0b2890cf8 100644
--- a/lib/bb/codeparser.py
+++ b/lib/bb/codeparser.py
@@ -82,14 +82,14 @@ def add_module_functions(fn, functions, namespace):
             if e in functions:
                 execs.remove(e)
                 execs.add(namespace + "." + e)
-        modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy()]
+        modulecode_deps[name] = [parser.references.copy(), execs, parser.var_execs.copy(), parser.contains.copy(), parser.extra]
         #bb.warn("%s: %s\nRefs:%s Execs: %s %s %s" % (name, fn, parser.references, parser.execs, parser.var_execs, parser.contains))
 
 def update_module_dependencies(d):
     for mod in modulecode_deps:
         excludes = set((d.getVarFlag(mod, "vardepsexclude") or "").split())
         if excludes:
-            modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3]]
+            modulecode_deps[mod] = [modulecode_deps[mod][0] - excludes, modulecode_deps[mod][1] - excludes, modulecode_deps[mod][2] - excludes, modulecode_deps[mod][3], modulecode_deps[mod][4]]
 
 # A custom getstate/setstate using tuples is actually worth 15% cachesize by
 # avoiding duplication of the attribute names!
@@ -112,21 +112,22 @@ class SetCache(object):
 codecache = SetCache()
 
 class pythonCacheLine(object):
-    def __init__(self, refs, execs, contains):
+    def __init__(self, refs, execs, contains, extra):
         self.refs = codecache.internSet(refs)
         self.execs = codecache.internSet(execs)
         self.contains = {}
         for c in contains:
             self.contains[c] = codecache.internSet(contains[c])
+        self.extra = extra
 
     def __getstate__(self):
-        return (self.refs, self.execs, self.contains)
+        return (self.refs, self.execs, self.contains, self.extra)
 
     def __setstate__(self, state):
-        (refs, execs, contains) = state
-        self.__init__(refs, execs, contains)
+        (refs, execs, contains, extra) = state
+        self.__init__(refs, execs, contains, extra)
     def __hash__(self):
-        l = (hash(self.refs), hash(self.execs))
+        l = (hash(self.refs), hash(self.execs), hash(self.extra))
         for c in sorted(self.contains.keys()):
             l = l + (c, hash(self.contains[c]))
         return hash(l)
@@ -155,7 +156,7 @@ class CodeParserCache(MultiProcessCache):
     # so that an existing cache gets invalidated. Additionally you'll need
     # to increment __cache_version__ in cache.py in order to ensure that old
     # recipe caches don't trigger "Taskhash mismatch" errors.
-    CACHE_VERSION = 11
+    CACHE_VERSION = 12
 
     def __init__(self):
         MultiProcessCache.__init__(self)
@@ -169,8 +170,8 @@ class CodeParserCache(MultiProcessCache):
         self.pythoncachelines = {}
         self.shellcachelines = {}
 
-    def newPythonCacheLine(self, refs, execs, contains):
-        cacheline = pythonCacheLine(refs, execs, contains)
+    def newPythonCacheLine(self, refs, execs, contains, extra):
+        cacheline = pythonCacheLine(refs, execs, contains, extra)
         h = hash(cacheline)
         if h in self.pythoncachelines:
             return self.pythoncachelines[h]
@@ -338,6 +339,7 @@ class PythonParser():
             self.contains = {}
             for i in codeparsercache.pythoncache[h].contains:
                 self.contains[i] = set(codeparsercache.pythoncache[h].contains[i])
+            self.extra = codeparsercache.pythoncache[h].extra
             return
 
         if h in codeparsercache.pythoncacheextras:
@@ -346,6 +348,7 @@ class PythonParser():
             self.contains = {}
             for i in codeparsercache.pythoncacheextras[h].contains:
                 self.contains[i] = set(codeparsercache.pythoncacheextras[h].contains[i])
+            self.extra = codeparsercache.pythoncacheextras[h].extra
             return
 
         if fixedhash and not node:
@@ -364,8 +367,11 @@ class PythonParser():
                 self.visit_Call(n)
 
         self.execs.update(self.var_execs)
+        self.extra = None
+        if fixedhash:
+            self.extra = bbhash(str(node))
 
-        codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains)
+        codeparsercache.pythoncacheextras[h] = codeparsercache.newPythonCacheLine(self.references, self.execs, self.contains, self.extra)
 
 class ShellParser():
     def __init__(self, name, log):
diff --git a/lib/bb/data.py b/lib/bb/data.py
index 505f42950..f672a8445 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -293,7 +293,7 @@ def build_dependencies(key, keys, mod_funcs, shelldeps, varflagsexcl, ignored_va
         if key in mod_funcs:
             exclusions = set()
             moddep = bb.codeparser.modulecode_deps[key]
-            value = handle_contains("", moddep[3], exclusions, d)
+            value = handle_contains(moddep[4], moddep[3], exclusions, d)
             return frozenset((moddep[0] | keys & moddep[1]) - ignored_vars), value
 
         if key[-1] == ']':
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 2/2] codeparser: Skip non-local functions for module dependencies

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If modules do something like "from glob import glob" then we end up
checksumming the glob code. That leads to bugs as the code can change
between different python versions for example, leading to checksum
instability.

We should ignore functions not from the current file as implemented
by this change.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1e6f862864539d6f6a0bea3e4479e0dd40ff3091)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/codeparser.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
index 0b2890cf8..1001ca190 100644
--- a/lib/bb/codeparser.py
+++ b/lib/bb/codeparser.py
@@ -72,6 +72,11 @@ def add_module_functions(fn, functions, namespace):
             parser.parse_python(None, filename=fn, lineno=1, fixedhash=fixedhash+f)
             #bb.warn("Cached %s" % f)
         except KeyError:
+            targetfn = inspect.getsourcefile(functions[f])
+            if fn != targetfn:
+                # Skip references to other modules outside this file
+                #bb.warn("Skipping %s" % name)
+                continue
             lines, lineno = inspect.getsourcelines(functions[f])
             src = "".join(lines)
             parser.parse_python(src, filename=fn, lineno=lineno, fixedhash=fixedhash+f)
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of changes for 2.18/scarthgap and have comments back
by end of day Friday, September 12

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2347

The following changes since commit 982645110a19ebb94d519926a4e14c8a2a205cfd:

  utils: Optimise signal/sigmask performance (2025-07-31 09:21:27 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Bin Lan (1):
  bb/fetch2/__init__.py: remove a DeprecationWarning in uri_replace()

Joshua Watt (1):
  Use a "fork" multiprocessing context

 lib/bb/__init__.py                 | 28 ++++++++++++++++++++++++++++
 lib/bb/asyncrpc/serv.py            |  2 +-
 lib/bb/cooker.py                   |  2 +-
 lib/bb/fetch2/__init__.py          |  2 +-
 lib/bb/server/process.py           |  2 +-
 lib/bb/tests/support/httpserver.py |  4 ++--
 lib/bb/utils.py                    |  4 +---
 lib/hashserv/tests.py              |  2 +-
 8 files changed, 36 insertions(+), 10 deletions(-)

-- 
2.43.0

[bitbake][scarthgap][2.8][PATCH 0/2] Patch review

[Steve Sakoman]

Please review this set of changes for 2.8/scarthgap and have comments back by
end of day Thursday, November 6

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2676

The following changes since commit 1c9ec1ffde75809de34c10d3ec2b40d84d258cb4:

  bitbake: Bump version to 2.8.1 (2025-09-17 23:22:33 +0100)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Bin Lan (1):
  bb/fetch2/__init__.py: remove a DeprecationWarning in uri_replace()

Philippe-Alexandre Mathieu (1):
  fetch2/wget: Keep query parameters in URL during checkstatus

 lib/bb/fetch2/__init__.py | 2 +-
 lib/bb/fetch2/wget.py     | 7 +++++--
 2 files changed, 6 insertions(+), 3 deletions(-)

-- 
2.43.0

[bitbake][scarthgap][2.8][PATCH 0/2] Patch review

[Yoann Congal]

Please review this set of changes for scarthgap and have comments back by
end of day Wednesday, February 11.

Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3183

The following changes since commit 8dcf084522b9c66a6639b5f117f554fde9b6b45a:

  fetch2/wget: Keep query parameters in URL during checkstatus (2025-11-12 15:53:31 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

for you to fetch changes up to 1b923cdd50449aaa41152b72eac8669f27ec7ec2:

  knotty: Make sure getTerminalColumns() returns two integers (2026-02-09 01:22:13 +0100)

----------------------------------------------------------------

Enrico Scholz (1):
  knotty: fix TIOCGWINSZ call for Python 3.14 and later

Zoltan Boszormenyi (1):
  knotty: Make sure getTerminalColumns() returns two integers

 lib/bb/ui/knotty.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)