[bitbake][scarthgap][2.8][PATCH 0/4] Patch review

[bitbake][scarthgap][2.8][PATCH 0/4] Patch review

[Steve Sakoman]

Please review this set of changes for scarthgap/2.8 and have comments back
by end of day Friday, November 1

Passed a-full on autobuilder:

https://valkyrie.yoctoproject.org/#/builders/29/builds/370

The following changes since commit 270c0c8bdc802c04fd3021c082d58cf8e8d68d82:

  asyncrpc: Use client timeout for websocket open timeout (2024-10-14 15:44:00 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (2):
  tests/fetch: Use our own mirror of sysprof to decouple from gnome
    gitlab
  tests/fetch: Use our own mirror of mobile-broadband-provider to
    decouple from gnome gitlab

Robert Yang (2):
  gitsm: Add call_process_submodules() to remove duplicated code
  gitsm: Remove downloads/tmpdir when failed

 lib/bb/fetch2/gitsm.py | 44 +++++++++++++++---------------------------
 lib/bb/tests/fetch.py  |  4 ++--
 2 files changed, 18 insertions(+), 30 deletions(-)

-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 0/4] Patch review

[Steve Sakoman]

Please review this set of changes for 2.8/scarthgap and have comments back
by end of day Friday, March 7

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/1127

The following changes since commit 4e443aeab9096b41c9e5ba41cd21027ecaa20285:

  bitbake-diffsigs: fix handling when finding only a single sigfile (2025-03-05 14:24:43 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (4):
  utils: Print information about lock issue before exiting
  utils: Tweak lock_timeout logic
  utils: Add signal blocking for lock_timeout
  event/utils: Avoid deadlock from lock_timeout() and recursive events

 lib/bb/event.py | 10 +++++++++-
 lib/bb/utils.py | 20 +++++++++++++++++++-
 2 files changed, 28 insertions(+), 2 deletions(-)

-- 
2.43.0

[bitbake][scarthgap][2.8][PATCH 0/4] Patch review

[Yoann Congal]

Please review this set of changes for 2.8/scarthgap and have comments back by
end of day Monday, March 23.

Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3457

The following changes since commit a0a5ce49f28d886b1dac173842642e69517b44e3:

  COW: Fix hardcoded magic numbers and work with python 3.13 (2026-03-16 09:24:26 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

for you to fetch changes up to d3b4c352dd33fca90cd31649eda054b884478739:

  fetch2: Fix LFS object checkout in submodules (2026-03-19 00:14:50 +0100)

----------------------------------------------------------------

Philip Lorenz (2):
  fetch2: Fix incorrect lfs parametrization for submodules
  fetch2: Fix LFS object checkout in submodules

Robert Yang (1):
  gitsm: Add clean function

Ross Burton (1):
  fetch2: don't try to preserve all attributes when unpacking files

 lib/bb/fetch2/__init__.py |  2 +-
 lib/bb/fetch2/gitsm.py    | 26 +++++++++++++++++++-------
 2 files changed, 20 insertions(+), 8 deletions(-)

[bitbake][scarthgap][2.8][PATCH 0/4] Patch review

[Jeremy Rosen]

(Acting as LTS maintainer in training, process has been reviewed by
Yoann Congal)

Please review this set of changes for scarthgap and have comments back by 
end of day Tuesday, June 16.                                                                                                                                                                                                              

please not that patch 0001-fetch2-wget-handle-HTTP-308-Permanent-Redirect.patch 
was added to the patchset to enable backport of
0002-fetch2-wget-limit-auth-on-checkstatus-redirects.patch

Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3980

The following changes since commit a4daa14312d659333984aa1ae58ddfe0c96392f1:

  hashserv: validate unihash values (2026-06-06 15:52:19 +0200)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

for you to fetch changes up to 0880963fea4d91a034e4a6e007d23f98658ab986:

  data: fix issue with varflag exclusion (2026-06-12 11:39:20 +0200)

----------------------------------------------------------------

Anders Heimer (2):
  fetch2/wget: limit auth on checkstatus redirects
  tests/fetch: cover checkstatus redirect auth handling

Marcio Henriques (1):
  data: fix issue with varflag exclusion

Ross Burton (1):
  fetch2/wget: handle HTTP 308 Permanent Redirect

 lib/bb/data.py        |  2 +-
 lib/bb/fetch2/wget.py | 65 +++++++++++++++++++++++++++++++++++++++----
 lib/bb/tests/fetch.py | 62 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 123 insertions(+), 6 deletions(-)

-- 
2.53.0

[bitbake][scarthgap][2.8][PATCH 0/4] Patch review

[Yoann Congal]

Please review this set of changes for scarthgap and have comments back by
end of day Tuesday, September 15.

Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/29/builds/4718
* oe-selftest-fedora fails with 16415 – AB-INT: github infrastructure issues
  I still need to backport fixes from master to wrynose, and then,
  scarthgap.
* qemuarm64-ptest failed with 16267 – [scarthgap] AB-INT PTEST: python3 failure (test_wrong_cert_tls13)
  retried in https://autobuilder.yoctoproject.org/valkyrie/?#/builders/61/builds/4516


The following changes since commit cff3be6f664f8f07a40727ff63eeeb24d6f5e00b:

  utils: Add NFS EEXISTS/isdir failure workaround (2026-07-19 18:02:05 +0200)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

for you to fetch changes up to cd50c3481e32afff995a143c3459cfa5ae25abd5:

  tests/fetch: Swap bitbake github mirror for YP one (2026-09-09 17:57:33 +0200)

----------------------------------------------------------------

Richard Purdie (3):
  fetch/gitsm: Store the original url data to fix relative gitsm paths
  tests/fetch: Swap github grpc repo for our own small test repo
  tests/fetch: Swap bitbake github mirror for YP one

Sebastian Muxel (1):
  fetch2/gcp: treat GatewayTimeout as fetch failure

 lib/bb/fetch2/__init__.py |  1 +
 lib/bb/fetch2/gcp.py      | 24 ++++++++++++++++++++++--
 lib/bb/fetch2/gitsm.py    |  5 ++++-
 lib/bb/tests/fetch.py     | 16 ++++++++--------
 4 files changed, 35 insertions(+), 11 deletions(-)

[bitbake][scarthgap][2.8][PATCH 1/4] fetch2/gcp: treat GatewayTimeout as fetch failure

[Yoann Congal]

From: Sebastian Muxel <smuxel@snap.com>

blob.exists() and blob.download_to_filename() can raise
google.api_core.exceptions.GatewayTimeout after the GCS client's own
retries. Uncaught, that escapes as a hard error instead of a normal
fetch/checkstatus failure (and blocks mirror fallback on download).

Catch GatewayTimeout in checkstatus() and download(), log a warning so
the timeout is visible, and raise FetchError.

(cherry-picked from commit 251f01e9afa1dcb9a49f8a31981e698017a43754)

AI-Generated: Cursor with Grok 4.5
Signed-off-by: Sebastian Muxel <smuxel@snap.com>
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Mathieu Dubois-Briand <mathieu.dubois-briand@bootlin.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/fetch2/gcp.py | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/lib/bb/fetch2/gcp.py b/lib/bb/fetch2/gcp.py
index 2ee9ed219..c3cbc948f 100644
--- a/lib/bb/fetch2/gcp.py
+++ b/lib/bb/fetch2/gcp.py
@@ -57,7 +57,7 @@ class GCP(FetchMethod):
         Fetch urls using the GCP API.
         Assumes localpath was called first.
         """
-        from google.api_core.exceptions import NotFound
+        from google.api_core.exceptions import GatewayTimeout, NotFound
         logger.debug2(f"Trying to download gs://{ud.host}{ud.path} to {ud.localpath}")
         if self.gcp_client is None:
             self.get_gcp_client()
@@ -71,6 +71,13 @@ class GCP(FetchMethod):
             blob.download_to_filename(ud.localpath)
         except NotFound:
             raise FetchError("The GCP API threw a NotFound exception")
+        except GatewayTimeout as e:
+            # The GCS client already retries GatewayTimeout internally.
+            # Raise FetchError so mirror fallback can proceed.
+            logger.warning(
+                f"GCP API GatewayTimeout while downloading gs://{ud.host}{ud.path}: {e}"
+            )
+            raise FetchError(f"Transient GCP API GatewayTimeout for gs://{ud.host}{ud.path}")
 
         # Additional sanity checks copied from the wget class (although there
         # are no known issues which mean these are required, treat the GCP API
@@ -88,6 +95,8 @@ class GCP(FetchMethod):
         """
         Check the status of a URL.
         """
+        from google.api_core.exceptions import GatewayTimeout
+
         logger.debug2(f"Checking status of gs://{ud.host}{ud.path}")
         if self.gcp_client is None:
             self.get_gcp_client()
@@ -96,7 +105,18 @@ class GCP(FetchMethod):
 
         # Path sometimes has leading slash, so strip it
         path = ud.path.lstrip("/")
-        if self.gcp_client.bucket(ud.host).blob(path).exists() == False:
+        try:
+            exists = self.gcp_client.bucket(ud.host).blob(path).exists()
+        except GatewayTimeout as e:
+            # The GCS client already retries GatewayTimeout internally.
+            # Surface a normal checkstatus failure and warn so the timeout
+            # is visible to operators.
+            logger.warning(
+                f"GCP API GatewayTimeout while checking gs://{ud.host}{ud.path}; treating as unavailable: {e}"
+            )
+            raise FetchError(f"Transient GCP API GatewayTimeout for gs://{ud.host}{ud.path}")
+
+        if exists == False:
             raise FetchError(f"The GCP API reported that gs://{ud.host}{ud.path} does not exist")
         else:
             return True

[bitbake][scarthgap][2.8][PATCH 2/4] fetch/gitsm: Store the original url data to fix relative gitsm paths

[Yoann Congal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If the gitsm:// url contains a git submodule with a relative "../" path
and the fetcher is configured to use a premirror, it will fail as the
host name encoded in the original url is lost and the correct paths
can't be found.

Save the original url data in the new data so that it can be consulted to
fix this kind of edge case.

The test case change that follows this provides a test of this functionality.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 58872250f72f5760b4126552bdec5497f5a1a1f4)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/fetch2/__init__.py | 1 +
 lib/bb/fetch2/gitsm.py    | 5 ++++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index f2835397a..e89f5be2a 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1011,6 +1011,7 @@ def build_mirroruris(origud, mirrors, ld):
                     newud = FetchData(newuri, ld)
                     newud.ignore_checksums = True
                     newud.setup_localpath(ld)
+                    newud.origud = ud
                 except bb.fetch2.BBFetchException as e:
                     logger.debug("Mirror fetch failure for url %s (original url: %s)" % (newuri, origud.url))
                     logger.debug(str(e))
diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index ef1905333..e866edeb8 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -89,7 +89,10 @@ class GitSM(Git):
 
                 # Convert relative to absolute uri based on parent uri
                 if  uris[m].startswith('..') or uris[m].startswith('./'):
-                    newud = copy.copy(ud)
+                    if hasattr(ud, "origud"):
+                        newud = copy.copy(ud.origud)
+                    else:
+                        newud = copy.copy(ud)
                     newud.path = os.path.normpath(os.path.join(newud.path, uris[m]))
                     uris[m] = Git._get_repo_url(self, newud)
 

[bitbake][scarthgap][2.8][PATCH 3/4] tests/fetch: Swap github grpc repo for our own small test repo

[Yoann Congal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The grpc repo is 1GB in size and makes a poor choice of test repo. With github
rate limiting, it is proving probematic.

Swap if for our own gitsm test repo which is smaller but can serve the purposes
of the test.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a64b3157d74f883002318cc46c70bfa1c3609110)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/tests/fetch.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index a658b89a8..3ed98593d 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1193,7 +1193,7 @@ class FetcherNetworkTest(FetcherTest):
     def test_git_submodule_restricted_network_premirrors(self):
         # this test is to ensure that premirrors will be tried in restricted network
         # that is, BB_ALLOWED_NETWORKS does not contain the domain the url uses
-        url = "gitsm://github.com/grpc/grpc.git;protocol=https;name=grpc;branch=v1.60.x;rev=0ef13a7555dbaadd4633399242524129eef5e231"
+        url = "gitsm://git.yoctoproject.org/git-submodule-test;branch=master;rev=38e61644af90dccd73c03ed3acaed98c8dda9294;branch=master;protocol=https"
         # create a download directory to be used as premirror later
         tempdir = tempfile.mkdtemp(prefix="bitbake-fetch-")
         dl_premirror = os.path.join(tempdir, "download-premirror")

[bitbake][scarthgap][2.8][PATCH 4/4] tests/fetch: Swap bitbake github mirror for YP one

[Yoann Congal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We need a reliable mirror of the bitbake repo and github is no longer that. Swap
to use the bitbake-contrib repo which carrys enough bitbake data to work for the
tests we need.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f311ff38b4bf79675f677f73f1d390723b1c4490)
Signed-off-by: Yoann Congal <yoann.congal@smile.fr>
---
 lib/bb/tests/fetch.py | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 3ed98593d..3e2c6f8c1 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -568,8 +568,8 @@ class GitDownloadDirectoryNamingTest(FetcherTest):
         super(GitDownloadDirectoryNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
         self.recipe_dir = "git.openembedded.org.bitbake"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
-        self.mirror_dir = "github.com.openembedded.bitbake.git"
+        self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+        self.mirror_dir = "git.openembedded.org.bitbake-contrib.git"
 
         self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
 
@@ -616,8 +616,8 @@ class TarballNamingTest(FetcherTest):
         super(TarballNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
         self.recipe_tarball = "git2_git.openembedded.org.bitbake.tar.gz"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
-        self.mirror_tarball = "git2_github.com.openembedded.bitbake.git.tar.gz"
+        self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+        self.mirror_tarball = "git2_git.openembedded.org.bitbake-contrib.git.tar.gz"
 
         self.d.setVar('BB_GENERATE_MIRROR_TARBALLS', '1')
         self.d.setVar('SRCREV', '82ea737a0b42a8b53e11c9cde141e9e9c0bd8c40')
@@ -650,8 +650,8 @@ class GitShallowTarballNamingTest(FetcherTest):
         super(GitShallowTarballNamingTest, self).setUp()
         self.recipe_url = "git://git.openembedded.org/bitbake;branch=master;protocol=https"
         self.recipe_tarball = "gitshallow_git.openembedded.org.bitbake_82ea737-1_master.tar.gz"
-        self.mirror_url = "git://github.com/openembedded/bitbake.git;protocol=https;branch=master"
-        self.mirror_tarball = "gitshallow_github.com.openembedded.bitbake.git_82ea737-1_master.tar.gz"
+        self.mirror_url = "git://git.openembedded.org/bitbake-contrib.git;protocol=https;branch=master"
+        self.mirror_tarball = "gitshallow_git.openembedded.org.bitbake-contrib.git_82ea737-1_master.tar.gz"
 
         self.d.setVar('BB_GIT_SHALLOW', '1')
         self.d.setVar('BB_GENERATE_SHALLOW_TARBALLS', '1')
@@ -2367,7 +2367,7 @@ class GitShallowTest(FetcherTest):
 
     @skipIfNoNetwork()
     def test_bitbake(self):
-        self.git('remote add --mirror=fetch origin https://github.com/openembedded/bitbake', cwd=self.srcdir)
+        self.git('remote add --mirror=fetch origin https://git.openembedded.org/bitbake-contrib', cwd=self.srcdir)
         self.git('config core.bare true', cwd=self.srcdir)
         self.git('fetch', cwd=self.srcdir)