[bitbake][wrynose][2.18][PATCH 0/4] Patch review

[bitbake][wrynose][2.18][PATCH 0/4] Patch review

[Yoann Congal]

Please review this set of changes for wrynose and have comments back by
end of day Friday, September 11.

Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/4697

The following changes since commit 0ad6c1c34a5e07a5f8dd66ab248c1e7b37b69fa9:

  doc: bitbake-user-manual-environment-setup: use pip to install bitbake-setup (2026-08-20 11:15:51 +0100)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.18-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.18-nut

for you to fetch changes up to f0df79b6b56f6a69067adf974978d9405077f246:

  tests/fetch: Swap bitbake github mirror for YP one (2026-09-08 14:13:56 +0200)

----------------------------------------------------------------

Alejandro Mery (1):
  data: Return a list from exported_vars()

Richard Purdie (3):
  fetch/gitsm: Store the original url data to fix relative gitsm paths
  tests/fetch: Swap github grpc repo for our own small test repo
  tests/fetch: Swap bitbake github mirror for YP one

 bin/bitbake-worker        |  6 ++++--
 lib/bb/data.py            | 10 +++++++++-
 lib/bb/fetch2/__init__.py |  1 +
 lib/bb/fetch2/gitsm.py    |  5 ++++-
 lib/bb/tests/data.py      | 28 ++++++++++++++++++++++++++++
 lib/bb/tests/fetch.py     | 16 ++++++++--------
 6 files changed, 54 insertions(+), 12 deletions(-)

[bitbake][wrynose][2.18][PATCH 1/4] data: Return a list from exported_vars()

[Yoann Congal]

From: Alejandro Mery <amery@apptly.co>

exported_vars() was a generator, so each value expanded when the result
was iterated rather than when it was called. bitbake-worker binds it
before bb.utils.empty_environment() and iterates it afterwards, so any
expansion deferred until iteration ran against the wiped environment
rather than the one bitbake started with.

The corruption is not uniform. The worker writes each variable into
os.environ as it goes, so only the first value expands against a fully
empty environment and the rest see it partially rebuilt, in
exported_keys() order.

Recipes whose exported variables expand a command during that loop hit
this. A gitver-style PV such as "${@get_git_pv(d, ...)}" runs git while
PATH is empty, so the git wrapper on PATH is bypassed and the real git
runs directly. Under pseudo this fakes uid 0 against a repository owned
by the real user, and git aborts with "detected dubious ownership",
failing do_package intermittently (only on reparse, when the value is
re-expanded rather than served from cache).

Build and return the list in exported_vars() itself, so the expansion is
complete before it returns and no caller has to know it was ever
deferred. A value that fails to expand now warns at that point rather
than during iteration, and a result nothing consumes is expanded anyway;
neither is a change for the one caller, which consumes all of it. Update
the comment at the worker's call site, which documented the generator
that is now gone, and add tests for the expansion timing and for the
warning that moved with it.

Signed-off-by: Alejandro Mery <amery@apptly.co>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c870f5bd96ad02efc02c58133c27bdae3a300e3e)
Signed-off-by: Alejandro Mery <amery@apptly.co>
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
[YC: removed patch history from the commit message]
---
 bin/bitbake-worker   |  6 ++++--
 lib/bb/data.py       | 10 +++++++++-
 lib/bb/tests/data.py | 28 ++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+), 3 deletions(-)

diff --git a/bin/bitbake-worker b/bin/bitbake-worker
index aa14ef191..ae21f277f 100755
--- a/bin/bitbake-worker
+++ b/bin/bitbake-worker
@@ -288,8 +288,10 @@ def fork_off_task(cfg, data, databuilder, workerdata, extraconfigdata, runtask):
                     else:
                         logger.debug("Skipping disable network for %s since %s is not a local uid." % (taskname, uid))
 
-                # exported_vars() returns a generator which *cannot* be passed to os.environ.update() 
-                # successfully. We also need to unset anything from the environment which shouldn't be there 
+                # exported_vars() expands each value, so it must be called before
+                # empty_environment() below: an expansion that shells out needs the
+                # environment bitbake was started with. We also need to unset
+                # anything from the environment which shouldn't be there
                 exports = bb.data.exported_vars(the_data)
 
                 bb.utils.empty_environment()
diff --git a/lib/bb/data.py b/lib/bb/data.py
index b12972c03..7c01e6bf8 100644
--- a/lib/bb/data.py
+++ b/lib/bb/data.py
@@ -193,6 +193,12 @@ def exported_keys(d):
                                       not bb.utils.to_boolean(d.getVarFlag(key, 'unexport')))
 
 def exported_vars(d):
+    """Return the exported variables as a list of (key, value) pairs.
+
+    Every value is expanded before returning, so a caller that changes the
+    environment afterwards still gets what the expansion saw.
+    """
+    exported = []
     k = list(exported_keys(d))
     for key in k:
         try:
@@ -202,7 +208,9 @@ def exported_vars(d):
             continue
 
         if value is not None:
-            yield key, str(value)
+            exported.append((key, str(value)))
+
+    return exported
 
 def emit_func(func, o=sys.__stdout__, d = init()):
     """Emits all items in the data store in a format such that it can be sourced by a shell."""
diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py
index fd690a9e2..a83c71e2c 100644
--- a/lib/bb/tests/data.py
+++ b/lib/bb/tests/data.py
@@ -723,3 +723,31 @@ class EmitVar(unittest.TestCase):
         self.assertEqual(self.get_output(out), ['bad_chars="a\\"b \\',
                                                 'c\\`d \\',
                                                 'e\\$f"'])
+
+class ExportedVars(unittest.TestCase):
+    def test_expanded_before_returning(self):
+        # Called while the variable is set, read once it is gone: the value
+        # has to be the one from the call, not from the read.
+        d = bb.data.init()
+        d.setVar("TESTVAR", "${@os.environ.get('BB_TEST_EXPORT', 'gone')}")
+        d.setVarFlag("TESTVAR", "export", "1")
+
+        with bb.utils.environment(BB_TEST_EXPORT="present"):
+            exported = bb.data.exported_vars(d)
+
+        self.assertEqual(dict(exported), {"TESTVAR": "present"})
+        self.assertIsInstance(exported, list)
+
+    def test_unexpandable_value_warns_and_is_skipped(self):
+        # The warning belongs to the call, not to a later iteration.
+        d = bb.data.init()
+        d.setVar("TESTVAR", "value")
+        d.setVarFlag("TESTVAR", "export", "1")
+        d.setVar("TESTBROKEN", "${@int('not a number')}")
+        d.setVarFlag("TESTBROKEN", "export", "1")
+
+        with LogRecord() as logs:
+            exported = bb.data.exported_vars(d)
+
+        self.assertEqual(dict(exported), {"TESTVAR": "value"})
+        self.assertTrue(logContains("Unable to export ${TESTBROKEN}", logs))

[bitbake][wrynose][2.18][PATCH 2/4] fetch/gitsm: Store the original url data to fix relative gitsm paths

[Yoann Congal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If the gitsm:// url contains a git submodule with a relative "../" path
and the fetcher is configured to use a premirror, it will fail as the
host name encoded in the original url is lost and the correct paths
can't be found.

Save the original url data in the new data so that it can be consulted to
fix this kind of edge case.

The test case change that follows this provides a test of this functionality.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 58872250f72f5760b4126552bdec5497f5a1a1f4)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/fetch2/__init__.py | 1 +
 lib/bb/fetch2/gitsm.py    | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index e0494b3cb..b9373d2a9 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1052,6 +1052,7 @@ def build_mirroruris(origud, mirrors, ld):
                     newud = FetchData(newuri, ld)
                     newud.ignore_checksums = True
                     newud.setup_localpath(ld)
+                    newud.origud = ud
                 except bb.fetch2.BBFetchException as e:
                     logger.debug("Mirror fetch failure for url %s (original url: %s)" % (newuri, origud.url))
                     logger.debug(str(e))
diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 5869e1b99..8b4ec6209 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -88,7 +88,10 @@ class GitSM(Git):
 
             # Convert relative to absolute uri based on parent uri
             if  uris[m].startswith('..') or uris[m].startswith('./'):
-                newud = copy.copy(ud)
+                if hasattr(ud, "origud"):
+                    newud = copy.copy(ud.origud)
+                else:
+                    newud = copy.copy(ud)
                 newud.path = os.path.normpath(os.path.join(newud.path, uris[m]))
                 uris[m] = Git._get_repo_url(self, newud)
 

[bitbake][wrynose][2.18][PATCH 3/4] tests/fetch: Swap github grpc repo for our own small test repo

[Yoann Congal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The grpc repo is 1GB in size and makes a poor choice of test repo. With github
rate limiting, it is proving probematic.

Swap if for our own gitsm test repo which is smaller but can serve the purposes
of the test.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a64b3157d74f883002318cc46c70bfa1c3609110)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index c071401e6..845741060 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1199,7 +1199,7 @@ class FetcherNetworkTest(FetcherTest):
     def test_git_submodule_restricted_network_premirrors(self):
         # this test is to ensure that premirrors will be tried in restricted network
         # that is, BB_ALLOWED_NETWORKS does not contain the domain the url uses
-        url = "gitsm://github.com/grpc/grpc.git;protocol=https;name=grpc;branch=v1.60.x;rev=0ef13a7555dbaadd4633399242524129eef5e231"
+        url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=38e61644af90dccd73c03ed3acaed98c8dda9294;branch=master;protocol=https"
         # create a download directory to be used as premirror later
         tempdir = tempfile.mkdtemp(prefix="bitbake-fetch-")
         dl_premirror = os.path.join(tempdir, "download-premirror")

[bitbake][wrynose][2.18][PATCH 4/4] tests/fetch: Swap bitbake github mirror for YP one

[Yoann Congal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We need a reliable mirror of the bitbake repo and github is no longer that. Swap
to use the bitbake-contrib repo which carrys enough bitbake data to work for the
tests we need.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f311ff38b4bf79675f677f73f1d390723b1c4490)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/tests/fetch.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 845741060..137505343 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -575,8 +575,8 @@ class GitDownloadDirectoryNamingTest(FetcherTest):
         super(GitDownloadDirectoryNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
         self.recipe_dir = "git.openembedded.org.bitbake"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
-        self.mirror_dir = "github.com.openembedded.bitbake.git"
+        self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+        self.mirror_dir = "git.openembedded.org.bitbake-contrib.git"
 
         self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
 
@@ -623,8 +623,8 @@ class TarballNamingTest(FetcherTest):
         super(TarballNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
         self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
-        self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz"
+        self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+        self.mirror_tarball = "git2_git.openembedded.org.bitbake-contrib.git.tar.gz"
 
         self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
         self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
@@ -657,8 +657,8 @@ class GitShallowTarballNamingTest(FetcherTest):
         super(GitShallowTarballNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
         self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
-        self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz"
+        self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+        self.mirror_tarball = "gitshallow_git.openembedded.org.bitbake-contrib.git_82ea737-1_master.tar.gz"
 
         self.d.setVar('BB_GIT_SHALLOW', '1')
         self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
@@ -2406,7 +2406,7 @@ class GitShallowTest(FetcherTest):
 
     @skipIfNoNetwork()
     def test_bitbake(self):
-        self.git('remote add --mirror=fetch origin https://github.com/openembedded/bitbake', cwd=self.srcdir)
+        self.git('remote add --mirror=fetch origin https://git.openembedded.org/bitbake-contrib', cwd=self.srcdir)
         self.git('config core.bare true', cwd=self.srcdir)
         self.git('fetch', cwd=self.srcdir)