[RFC PATCH 0/6] fetch2: add support for implicit urls

[RFC PATCH 0/6] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

The patch series add support for implicit URLs inside the fetcher. The
implicit URLs could be defined inside a source like a version control
system (git submodule) or a lock file (package-lock.json, cargo.lock or
go.sum). The integration of implicit URLs beside explicit URLs
simplifies the fetcher classes and avoid bugs because of iterations
between the Fetch and FetchMethod classes.

The series remove most methods inside the gitsm fetcher and only leaves
the parsing of the git submodules and the unpack functionality. It
allows the gitsm fetcher to use the premirror only feature. The current
implementation leads to problems because the download of the git
submodules is triggered via the download method which is called deeply
inside the fetcher code.


Stefan Herbrechtsmeier (6):
  fetch2: rename u to url in Fetch class
  fetch2: call functions within loops of Fetch class
  fetch2: add helper to get urldata in Fetch class
  fetch2: add support for implicit urls
  fetch2: gitsm: use implicit urls feature
  tests: fetch: add test case for gitsm implicit local paths

 lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++-----------
 lib/bb/fetch2/gitsm.py    |  46 ++------------
 lib/bb/tests/fetch.py     |  12 ++++
 3 files changed, 109 insertions(+), 77 deletions(-)

-- 
2.39.5

[RFC PATCH 1/6] fetch2: rename u to url in Fetch class

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Rename the varialbe u to url in the Fetch class to clarify its meaning.
Avoid using short variable names over long ranges and simplify
subsequent changes.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 lib/bb/fetch2/__init__.py | 42 +++++++++++++++++++--------------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index de301543a..bc9808da1 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1841,8 +1841,8 @@ class Fetch(object):
         """
         local = []
 
-        for u in self.urls:
-            ud = self.ud[u]
+        for url in self.urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
 
@@ -1859,8 +1859,8 @@ class Fetch(object):
         premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
 
         checksum_missing_messages = []
-        for u in urls:
-            ud = self.ud[u]
+        for url in urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
             done = False
@@ -1882,7 +1882,7 @@ class Fetch(object):
                             # contents mismatch the fetcher can still try upstream and mirrors
                             m.update_donestamp(ud, self.d)
                         except ChecksumError as e:
-                            logger.warning("Checksum failure encountered with premirror download of %s - will attempt other sources." % u)
+                            logger.warning("Checksum failure encountered with premirror download of %s - will attempt other sources." % url)
                             logger.debug(str(e))
                             done = False
 
@@ -1914,14 +1914,14 @@ class Fetch(object):
 
                     except BBFetchException as e:
                         if isinstance(e, ChecksumError):
-                            logger.warning("Checksum failure encountered with download of %s - will attempt other sources if available" % u)
+                            logger.warning("Checksum failure encountered with download of %s - will attempt other sources if available" % url)
                             logger.debug(str(e))
                             if os.path.exists(ud.localpath):
                                 rename_bad_checksum(ud, e.checksum)
                         elif isinstance(e, NoChecksumError):
                             raise
                         else:
-                            logger.warning('Failed to fetch URL %s, attempting MIRRORS if available' % u)
+                            logger.warning('Failed to fetch URL %s, attempting MIRRORS if available' % url)
                             logger.debug(str(e))
                         firsterr = e
                         # Remove any incomplete fetch
@@ -1934,13 +1934,13 @@ class Fetch(object):
                 if not done or not m.done(ud, d):
                     if firsterr:
                         logger.error(str(firsterr))
-                    raise FetchError("Unable to fetch URL from any source.", u)
+                    raise FetchError("Unable to fetch URL from any source.", url)
 
                 m.update_donestamp(ud, d)
 
             except IOError as e:
                 if e.errno in [errno.ESTALE]:
-                    logger.error("Stale Error Observed %s." % u)
+                    logger.error("Stale Error Observed %s." % url)
                     raise ChecksumError("Stale Error Detected")
 
             except BBFetchException as e:
@@ -1949,7 +1949,7 @@ class Fetch(object):
                     checksum_missing_messages.append(message)
                     continue
                 elif isinstance(e, ChecksumError):
-                    logger.error("Checksum failure fetching %s" % u)
+                    logger.error("Checksum failure fetching %s" % url)
                 raise
 
             finally:
@@ -1971,24 +1971,24 @@ class Fetch(object):
         if not urls:
             urls = self.urls
 
-        for u in urls:
-            ud = self.ud[u]
+        for url in urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
-            logger.debug("Testing URL %s", u)
-            # First try checking uri, u, from PREMIRRORS
+            logger.debug("Testing URL %s", url)
+            # First try checking uri, url, from PREMIRRORS
             mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
             ret = m.try_mirrors(self, ud, self.d, mirrors, True)
             if not ret:
-                # Next try checking from the original uri, u
+                # Next try checking from the original uri, url
                 ret = m.checkstatus(self, ud, self.d)
                 if not ret:
-                    # Finally, try checking uri, u, from MIRRORS
+                    # Finally, try checking uri, url, from MIRRORS
                     mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
                     ret = m.try_mirrors(self, ud, self.d, mirrors, True)
 
             if not ret:
-                raise FetchError("URL doesn't work", u)
+                raise FetchError("URL doesn't work", url)
 
     def unpack(self, root, urls=None):
         """
@@ -2000,16 +2000,16 @@ class Fetch(object):
 
         unpack_tracer.start(root, self.ud, self.d)
 
-        for u in urls:
-            ud = self.ud[u]
+        for url in urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
 
             if ud.lockfile:
                 lf = bb.utils.lockfile(ud.lockfile)
 
-            unpack_tracer.start_url(u)
+            unpack_tracer.start_url(url)
             ud.method.unpack(ud, root, self.d)
-            unpack_tracer.finish_url(u)
+            unpack_tracer.finish_url(url)
 
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
-- 
2.39.5

[RFC PATCH 2/6] fetch2: call functions within loops of Fetch class

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Call functions within the for all URLs loops of the Fetch class to
simplify subsequent changes.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 lib/bb/fetch2/__init__.py | 41 ++++++++++++++++++++++++++++-----------
 1 file changed, 30 insertions(+), 11 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index bc9808da1..80f1a0c08 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1841,11 +1841,14 @@ class Fetch(object):
         """
         local = []
 
-        for url in self.urls:
+        def localpath(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
 
+        for url in self.urls:
+            localpath(url)
+
         return local
 
     def download(self, urls=None):
@@ -1859,7 +1862,7 @@ class Fetch(object):
         premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
 
         checksum_missing_messages = []
-        for url in urls:
+        def download(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
@@ -1947,14 +1950,18 @@ class Fetch(object):
                 if isinstance(e, NoChecksumError):
                     (message, _) = e.args
                     checksum_missing_messages.append(message)
-                    continue
-                elif isinstance(e, ChecksumError):
-                    logger.error("Checksum failure fetching %s" % url)
-                raise
+                else:
+                    if isinstance(e, ChecksumError):
+                        logger.error("Checksum failure fetching %s" % url)
+                    raise
 
             finally:
                 if ud.lockfile:
                     bb.utils.unlockfile(lf)
+
+        for url in urls:
+            download(url)
+
         if checksum_missing_messages:
             logger.error("Missing SRC_URI checksum, please add those to the recipe: \n%s", "\n".join(checksum_missing_messages))
             raise BBFetchException("There was some missing checksums in the recipe")
@@ -1971,7 +1978,7 @@ class Fetch(object):
         if not urls:
             urls = self.urls
 
-        for url in urls:
+        def checkstatus(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
@@ -1990,6 +1997,9 @@ class Fetch(object):
             if not ret:
                 raise FetchError("URL doesn't work", url)
 
+        for url in urls:
+            checkstatus(url)
+
     def unpack(self, root, urls=None):
         """
         Unpack urls to root
@@ -2000,7 +2010,7 @@ class Fetch(object):
 
         unpack_tracer.start(root, self.ud, self.d)
 
-        for url in urls:
+        def unpack(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
 
@@ -2014,6 +2024,9 @@ class Fetch(object):
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
 
+        for url in urls:
+            unpack(url)
+
         unpack_tracer.complete()
 
     def clean(self, urls=None):
@@ -2024,14 +2037,14 @@ class Fetch(object):
         if not urls:
             urls = self.urls
 
-        for url in urls:
+        def clean(url):
             if url not in self.ud:
                 self.ud[url] = FetchData(url, self.d)
             ud = self.ud[url]
             ud.setup_localpath(self.d)
 
             if not ud.localfile and ud.localpath is None:
-                continue
+                return
 
             if ud.lockfile:
                 lf = bb.utils.lockfile(ud.lockfile)
@@ -2043,6 +2056,9 @@ class Fetch(object):
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
 
+        for url in urls:
+            clean(url)
+
     def expanded_urldata(self, urls=None):
         """
         Get an expanded list of FetchData objects covering both the given
@@ -2054,11 +2070,14 @@ class Fetch(object):
             urls = self.urls
 
         urldata = []
-        for url in urls:
+        def expand_urldata(url):
             ud = self.ud[url]
             urldata.append(ud)
             urldata += ud.method.implicit_urldata(ud, self.d)
 
+        for url in urls:
+            expand_urldata(url)
+
         return urldata
 
 class FetchConnectionCache(object):
-- 
2.39.5

[RFC PATCH 3/6] fetch2: add helper to get urldata in Fetch class

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Add a private helper function to get the urldata object of a given URL.
Create the urldate object if it doesn't exist inside the urldata object
list of the fetch object.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 lib/bb/fetch2/__init__.py | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 80f1a0c08..6da75008b 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1829,11 +1829,9 @@ class Fetch(object):
             urldata_cache[key] = self.ud
 
     def localpath(self, url):
-        if url not in self.urls:
-            self.ud[url] = FetchData(url, self.d)
-
-        self.ud[url].setup_localpath(self.d)
-        return self.ud[url].localpath
+        ud = self._get_urldata(url)
+        ud.setup_localpath(self.d)
+        return ud.localpath
 
     def localpaths(self):
         """
@@ -1842,7 +1840,7 @@ class Fetch(object):
         local = []
 
         def localpath(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
 
@@ -1863,7 +1861,7 @@ class Fetch(object):
 
         checksum_missing_messages = []
         def download(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             m = ud.method
             done = False
@@ -1979,7 +1977,7 @@ class Fetch(object):
             urls = self.urls
 
         def checkstatus(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             m = ud.method
             logger.debug("Testing URL %s", url)
@@ -2011,7 +2009,7 @@ class Fetch(object):
         unpack_tracer.start(root, self.ud, self.d)
 
         def unpack(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
 
             if ud.lockfile:
@@ -2038,9 +2036,7 @@ class Fetch(object):
             urls = self.urls
 
         def clean(url):
-            if url not in self.ud:
-                self.ud[url] = FetchData(url, self.d)
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
 
             if not ud.localfile and ud.localpath is None:
@@ -2071,7 +2067,7 @@ class Fetch(object):
 
         urldata = []
         def expand_urldata(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             urldata.append(ud)
             urldata += ud.method.implicit_urldata(ud, self.d)
 
@@ -2080,6 +2076,12 @@ class Fetch(object):
 
         return urldata
 
+    def _get_urldata(self, url):
+        if url not in self.ud:
+            self.ud[url] = FetchData(url, self.d)
+            self.ud[url].unpack_tracer = unpack_tracer
+        return self.ud[url]
+
 class FetchConnectionCache(object):
     """
         A class which represents an container for socket connections.
-- 
2.39.5

[RFC PATCH 4/6] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Add support for implicit URLs which are defined inside another source
like a version control system (git submodule) or an embedded lock file
(package-lock.json, cargo.lock or go.sum). The implicit URLs are
extracted from the source and thereby only available after the download
of the explicit source. The integration of implicit URLs beside explicit
URLs simplifies the fetcher classes and avoid bugs because of iterations
between the Fetch and FetchMethod classes.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 lib/bb/fetch2/__init__.py | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 6da75008b..e7bbdac14 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1312,6 +1312,7 @@ class FetchData(object):
         if not self.pswd and "pswd" in self.parm:
             self.pswd = self.parm["pswd"]
         self.setup = False
+        self.implicit_urls = None
 
         def configure_checksum(checksum_id):
             checksum_plain_name = "%ssum" % checksum_id
@@ -1409,6 +1410,11 @@ class FetchData(object):
 
         return d.getVar("SRCDATE") or d.getVar("DATE")
 
+    def setup_implicit_urls(self, d):
+        if self.implicit_urls is None:
+            self.setup_localpath(d)
+            self.implicit_urls = self.method.implicit_urls(self, d)
+
 class FetchMethod(object):
     """Base class for 'fetch'ing data"""
 
@@ -1727,6 +1733,9 @@ class FetchMethod(object):
         """
         return []
 
+    def implicit_urls(self, ud, d):
+        return []
+
 
 class DummyUnpackTracer(object):
     """
@@ -1843,6 +1852,9 @@ class Fetch(object):
             ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                localpath(u)
 
         for url in self.urls:
             localpath(url)
@@ -1960,6 +1972,16 @@ class Fetch(object):
         for url in urls:
             download(url)
 
+        def download_implicit(url):
+            ud = self._get_urldata(url)
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                download(u)
+                download_implicit(u)
+
+        for url in urls:
+            download_implicit(url)
+
         if checksum_missing_messages:
             logger.error("Missing SRC_URI checksum, please add those to the recipe: \n%s", "\n".join(checksum_missing_messages))
             raise BBFetchException("There was some missing checksums in the recipe")
@@ -2022,6 +2044,12 @@ class Fetch(object):
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
 
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                unpack(u)
+            if hasattr(ud.method, "postunpack"):
+                ud.method.postunpack(ud, root, self.d)
+
         for url in urls:
             unpack(url)
 
@@ -2038,6 +2066,10 @@ class Fetch(object):
         def clean(url):
             ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
+            if os.path.exists(ud.localpath):
+                ud.setup_implicit_urls(self.d)
+                for u in ud.implicit_urls:
+                    clean(u)
 
             if not ud.localfile and ud.localpath is None:
                 return
@@ -2070,6 +2102,9 @@ class Fetch(object):
             ud = self._get_urldata(url)
             urldata.append(ud)
             urldata += ud.method.implicit_urldata(ud, self.d)
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                expand_urldata(u)
 
         for url in urls:
             expand_urldata(url)
-- 
2.39.5

[RFC PATCH 5/6] fetch2: gitsm: use implicit urls feature

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Return the submodule URLs via implicit_urls function and remove the
manual handling of the submodule urls.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

 lib/bb/fetch2/gitsm.py | 46 +++++-------------------------------------
 1 file changed, 5 insertions(+), 41 deletions(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 5ddc81f86..90fe92f74 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -24,7 +24,6 @@ import tempfile
 from   bb.fetch2.git import Git
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
-from   bb.fetch2 import Fetch
 
 class GitSM(Git):
     def supports(self, ud, d):
@@ -141,9 +140,9 @@ class GitSM(Git):
 
             urls.append(url)
 
-        return Fetch(urls, d, cache=False) if urls else None
+        return urls
 
-    def call_process_submodules(self, ud, d):
+    def implicit_urls(self, ud, d):
         # If we're using a shallow mirror tarball it needs to be
         # unpacked temporarily so that we can examine the .gitmodules file
         # Unpack even when ud.clonedir is not available,
@@ -157,27 +156,6 @@ class GitSM(Git):
         else:
             raise bb.fetch2.FetchError("Submodule source not available.")
 
-    def need_update(self, ud, d):
-        if Git.need_update(self, ud, d):
-            return True
-
-        need_update = False
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            for url in fetch.urls:
-                urldata = fetch.ud[url]
-                if urldata.method.need_update(urldata, d):
-                    need_update = True
-
-        return need_update
-
-    def download(self, ud, d):
-        Git.download(self, ud, d)
-
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            fetch.download()
-
     def unpack(self, ud, destdir, d):
         fulldestdir = self.destdir(ud, destdir, d)
 
@@ -204,25 +182,11 @@ class GitSM(Git):
                 logger.error("Unable to set git config core.bare to false for %s" % fulldestdir)
                 raise
 
-        fetch = self.process_submodules(ud, fulldestdir, d)
-        if fetch:
-            fetch.unpack(destdir)
-
-        if not ud.bareclone and fetch:
+    def postunpack(self, ud, destdir, d):
+        fulldestdir = self.destdir(ud, destdir, d)
+        if not ud.bareclone:
             cmdprefix = ""
             # Avoid LFS smudging (replacing the LFS pointers with the actual content) when LFS shouldn't be used but git-lfs is installed.
             if not self._need_lfs(ud):
                 cmdprefix = "GIT_LFS_SKIP_SMUDGE=1 "
             runfetchcmd("%s%s submodule update --recursive --no-fetch" % (cmdprefix, ud.basecmd), d, quiet=True, workdir=fulldestdir)
-    def clean(self, ud, d):
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            fetch.clean()
-        Git.clean(self, ud, d)
-
-    def implicit_urldata(self, ud, d):
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            return fetch.expanded_urldata()
-
-        return []
-- 
2.39.5

[RFC PATCH 6/6] tests: fetch: add test case for gitsm implicit local paths

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

 lib/bb/tests/fetch.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 6067b02d2..e181a9a2a 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -3818,3 +3818,15 @@ class LocalPathsTest(FetcherTest):
         }
         fetcher = bb.fetch2.Fetch(localpaths.keys(), self.d)
         self.assertEqual(fetcher.localpaths(), list(localpaths.values()))
+
+    def test_git_submodule_CLI11(self):
+        url = "gitsm://github.com/CLIUtils/CLI11;protocol=https;rev=bd4dc911847d0cde7a6b41dfa626a85aab213baf;branch=main"
+        localpaths = [
+            f"{self.dldir}/git2/github.com.CLIUtils.CLI11",
+            f"{self.dldir}/git2/github.com.google.googletest.git",
+            f"{self.dldir}/git2/github.com.arsenm.sanitizers-cmake",
+            f"{self.dldir}/git2/github.com.nlohmann.json.git"
+        ]
+        fetcher = bb.fetch.Fetch([url], self.d)
+        fetcher.download()
+        self.assertEqual(fetcher.localpaths(), localpaths)
-- 
2.39.5

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Mathieu Dubois-Briand]

On Tue Sep 2, 2025 at 8:55 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> The patch series add support for implicit URLs inside the fetcher. The
> implicit URLs could be defined inside a source like a version control
> system (git submodule) or a lock file (package-lock.json, cargo.lock or
> go.sum). The integration of implicit URLs beside explicit URLs
> simplifies the fetcher classes and avoid bugs because of iterations
> between the Fetch and FetchMethod classes.
>
> The series remove most methods inside the gitsm fetcher and only leaves
> the parsing of the git submodules and the unpack functionality. It
> allows the gitsm fetcher to use the premirror only feature. The current
> implementation leads to problems because the download of the git
> submodules is triggered via the download method which is called deeply
> inside the fetcher code.
>
>
> Stefan Herbrechtsmeier (6):
>   fetch2: rename u to url in Fetch class
>   fetch2: call functions within loops of Fetch class
>   fetch2: add helper to get urldata in Fetch class
>   fetch2: add support for implicit urls
>   fetch2: gitsm: use implicit urls feature
>   tests: fetch: add test case for gitsm implicit local paths
>
>  lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++-----------
>  lib/bb/fetch2/gitsm.py    |  46 ++------------
>  lib/bb/tests/fetch.py     |  12 ++++
>  3 files changed, 109 insertions(+), 77 deletions(-)

Hi Stefan,

I know it is just an RFC so far, but I did launch a build on the
autobuilder. It was mostly correct but fails a selftest:

ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Error executing a python function in exec_func_python() autogenerated:
...
File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 2102, function: expand_urldata
...
Exception: UnboundLocalError: cannot access local variable 'urldata' where it is not associated with a value
...
2025-09-03 06:21:46,098 - oe-selftest - INFO - archiver.Archiver.test_archiver_mode_mirror_gitsm (subunit.RemotedTestCase)
2025-09-03 06:21:46,099 - oe-selftest - INFO -  ... FAIL

And a similar error with
archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow.

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

Hi Mathieu,

Thank you for testing. Until now I only run the bitbake tests. I will 
look into the error and will add a test case to bitbake to catch the 
problem early.

Thanks
   Stefan

Am 04.09.2025 um 08:00 schrieb Mathieu Dubois-Briand:
> On Tue Sep 2, 2025 at 8:55 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>
>> The patch series add support for implicit URLs inside the fetcher. The
>> implicit URLs could be defined inside a source like a version control
>> system (git submodule) or a lock file (package-lock.json, cargo.lock or
>> go.sum). The integration of implicit URLs beside explicit URLs
>> simplifies the fetcher classes and avoid bugs because of iterations
>> between the Fetch and FetchMethod classes.
>>
>> The series remove most methods inside the gitsm fetcher and only leaves
>> the parsing of the git submodules and the unpack functionality. It
>> allows the gitsm fetcher to use the premirror only feature. The current
>> implementation leads to problems because the download of the git
>> submodules is triggered via the download method which is called deeply
>> inside the fetcher code.
>>
>>
>> Stefan Herbrechtsmeier (6):
>>    fetch2: rename u to url in Fetch class
>>    fetch2: call functions within loops of Fetch class
>>    fetch2: add helper to get urldata in Fetch class
>>    fetch2: add support for implicit urls
>>    fetch2: gitsm: use implicit urls feature
>>    tests: fetch: add test case for gitsm implicit local paths
>>
>>   lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++-----------
>>   lib/bb/fetch2/gitsm.py    |  46 ++------------
>>   lib/bb/tests/fetch.py     |  12 ++++
>>   3 files changed, 109 insertions(+), 77 deletions(-)
> Hi Stefan,
>
> I know it is just an RFC so far, but I did launch a build on the
> autobuilder. It was mostly correct but fails a selftest:
>
> ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Error executing a python function in exec_func_python() autogenerated:
> ...
> File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 2102, function: expand_urldata
> ...
> Exception: UnboundLocalError: cannot access local variable 'urldata' where it is not associated with a value
> ...
> 2025-09-03 06:21:46,098 - oe-selftest - INFO - archiver.Archiver.test_archiver_mode_mirror_gitsm (subunit.RemotedTestCase)
> 2025-09-03 06:21:46,099 - oe-selftest - INFO -  ... FAIL
>
> And a similar error with
> archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow.
>
> Thanks,
> Mathieu
>

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

Am 04.09.2025 um 08:00 schrieb Mathieu Dubois-Briand:
> On Tue Sep 2, 2025 at 8:55 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>
>> The patch series add support for implicit URLs inside the fetcher. The
>> implicit URLs could be defined inside a source like a version control
>> system (git submodule) or a lock file (package-lock.json, cargo.lock or
>> go.sum). The integration of implicit URLs beside explicit URLs
>> simplifies the fetcher classes and avoid bugs because of iterations
>> between the Fetch and FetchMethod classes.
>>
>> The series remove most methods inside the gitsm fetcher and only leaves
>> the parsing of the git submodules and the unpack functionality. It
>> allows the gitsm fetcher to use the premirror only feature. The current
>> implementation leads to problems because the download of the git
>> submodules is triggered via the download method which is called deeply
>> inside the fetcher code.
>>
>>
>> Stefan Herbrechtsmeier (6):
>>    fetch2: rename u to url in Fetch class
>>    fetch2: call functions within loops of Fetch class
>>    fetch2: add helper to get urldata in Fetch class
>>    fetch2: add support for implicit urls
>>    fetch2: gitsm: use implicit urls feature
>>    tests: fetch: add test case for gitsm implicit local paths
>>
>>   lib/bb/fetch2/__init__.py | 128 +++++++++++++++++++++++++++-----------
>>   lib/bb/fetch2/gitsm.py    |  46 ++------------
>>   lib/bb/tests/fetch.py     |  12 ++++
>>   3 files changed, 109 insertions(+), 77 deletions(-)
> I know it is just an RFC so far, but I did launch a build on the
> autobuilder. It was mostly correct but fails a selftest:
>
> ERROR: git-submodule-test-1.0-r0 do_ar_mirror: Error executing a python function in exec_func_python() autogenerated:
> ...
> File: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 2102, function: expand_urldata
> ...
> Exception: UnboundLocalError: cannot access local variable 'urldata' where it is not associated with a value
> ...
> 2025-09-03 06:21:46,098 - oe-selftest - INFO - archiver.Archiver.test_archiver_mode_mirror_gitsm (subunit.RemotedTestCase)
> 2025-09-03 06:21:46,099 - oe-selftest - INFO -  ... FAIL
>
> And a similar error with
> archiver.Archiver.test_archiver_mode_mirror_gitsm_shallow.

I've fixed the issue and added a BitBake test case. Perhaps you could 
test the second version of the two patch series.

Regards
   Stefan

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Richard Purdie]

On Tue, 2025-09-02 at 08:55 +0200, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> The patch series add support for implicit URLs inside the fetcher. The
> implicit URLs could be defined inside a source like a version control
> system (git submodule) or a lock file (package-lock.json, cargo.lock or
> go.sum). The integration of implicit URLs beside explicit URLs
> simplifies the fetcher classes and avoid bugs because of iterations
> between the Fetch and FetchMethod classes.
> 
> The series remove most methods inside the gitsm fetcher and only leaves
> the parsing of the git submodules and the unpack functionality. It
> allows the gitsm fetcher to use the premirror only feature. The current
> implementation leads to problems because the download of the git
> submodules is triggered via the download method which is called deeply
> inside the fetcher code.

We had the discussion a while back and the conclusion seemed to be that
implict urls were disliked by a significant number of people as the
were too unclear about what was going on behind the scenes and also
made things like software manifests harder. There was a strong
preference for metadata helpers and explicit lists of components which
we have for crates/rust and now for go too.

It feels like this series is moving us back to the other direction. Is
that correct and if so, what has changed in the approach since the last
discussion?

Cheers,

Richard

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 3103 bytes --]

Am 07.09.2025 um 17:52 schrieb Richard Purdie via lists.openembedded.org:
> On Tue, 2025-09-02 at 08:55 +0200, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
>> The patch series add support for implicit URLs inside the fetcher. The
>> implicit URLs could be defined inside a source like a version control
>> system (git submodule) or a lock file (package-lock.json, cargo.lock or
>> go.sum). The integration of implicit URLs beside explicit URLs
>> simplifies the fetcher classes and avoid bugs because of iterations
>> between the Fetch and FetchMethod classes.
>>
>> The series remove most methods inside the gitsm fetcher and only leaves
>> the parsing of the git submodules and the unpack functionality. It
>> allows the gitsm fetcher to use the premirror only feature. The current
>> implementation leads to problems because the download of the git
>> submodules is triggered via the download method which is called deeply
>> inside the fetcher code.
> We had the discussion a while back and the conclusion seemed to be that
> implict urls were disliked by a significant number of people as the
> were too unclear about what was going on behind the scenes and also
> made things like software manifests harder.

It looks like I miss some discussion and especially the conclusion.

I assume you mean recipes by software manifests.

>   There was a strong
> preference for metadata helpers and explicit lists of components which
> we have for crates/rust and now for go too.

Does this mean the npmsw and gitsm fetchers are obsolete and should be 
replace by a metadata helpers to fix open issues?

> It feels like this series is moving us back to the other direction. Is
> that correct and if so, what has changed in the approach since the last
> discussion?

Do you mean the response to my last RFC? In this case it wasn't clear to 
me that the project is against implicit URLs and that the npmsw and 
gitsm fetcher are the wrong direction. This series is only a cleanup of 
the existing functionally. The gitsm fetcher uses implicit URLs but 
doesn't work correct because of the misuses of the download code. It is 
useless to start the discussion again. The project doesn't like implicit 
URLs. It prefers a special task as a replacement for the separate 
recipetool. It decides against the on-the-fly parse inside the fetcher. 
How should I proceed? I have working code which parse the cargo.lock, 
go.sum and package-lock.json files and only use Git and Wget fetchers. 
The code uses the vendor feature of the package managers to create a 
patchable folder of the sources. It simplifies the npm class and add 
additional classes to build packages. The code integrates valid URLs 
inside the SBOM and creates components with name and version per 
dependency. Furthermore, I have rework the gitsm fetcher to hopefully 
fix some open issues. I can convert my cargo, go, npm and gitsm parser 
into metadata helpers but this is useless if the project dislike python 
functions to generate URLs like the pypi class or prefer package manager 
specific code inside the fetcher. Regard Stefan

[-- Attachment #2: Type: text/html, Size: 4030 bytes --]

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Richard Purdie]

On Mon, 2025-09-08 at 11:20 +0200, Stefan Herbrechtsmeier wrote:
>  Am 07.09.2025 um 17:52 schrieb Richard Purdie via lists.openembedded.org:
>  On Tue, 2025-09-02 at 08:55 +0200, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> > > The patch series add support for implicit URLs inside the fetcher. The
> > > implicit URLs could be defined inside a source like a version control
> > > system (git submodule) or a lock file (package-lock.json, cargo.lock or
> > > go.sum). The integration of implicit URLs beside explicit URLs
> > > simplifies the fetcher classes and avoid bugs because of iterations
> > > between the Fetch and FetchMethod classes.
> > > 
> > > The series remove most methods inside the gitsm fetcher and only leaves
> > > the parsing of the git submodules and the unpack functionality. It
> > > allows the gitsm fetcher to use the premirror only feature. The current
> > > implementation leads to problems because the download of the git
> > > submodules is triggered via the download method which is called deeply
> > > inside the fetcher code.

> >  We had the discussion a while back and the conclusion seemed to be that
> > implict urls were disliked by a significant number of people as the
> > were too unclear about what was going on behind the scenes and also
> > made things like software manifests harder.
> >   
> > 
> 
> It looks like I miss some discussion and especially the conclusion.
>  
> 
>  I assume you mean recipes by software manifests.

No, I did mean software manifests. If the urls are explict, it makes
generating manifests of the sources being used more obvious for people
to understand. Yes, there are programmatic ways of doing it with
implicit urls but people don't like them.

By conclusions, I was taking that as the outcome of the last set of
discussions but there was a lot of different emails and it was hard to
follow. Perhaps i got the conclusion wrong, I don't know.

> >  There was a strong
> > preference for metadata helpers and explicit lists of components
> > which
> > we have for crates/rust and now for go too.
> >  
>  
> Does this mean the npmsw and gitsm fetchers are obsolete and should
> be replace by a metadata helpers to fix open issues?

I really don't know about npmsw. I don't use it and I don't really
follow development there. I'm don't know much about the current set of
issues it may have.

With gitsm, I think that is generally accepted by people and I don't
see a strong reason to change it at present. I would be interested in a
clear summary of what the known issues are (e.g. the premirror issue
you mentioned).

> > It feels like this series is moving us back to the other direction.
> > Is
> > that correct and if so, what has changed in the approach since the
> > last
> > discussion?
> >  
>  
> Do you mean the response to my last RFC? In this case it wasn't clear
> to me that the project is against implicit URLs and that the npmsw
> and gitsm fetcher are the wrong direction.

I'm trying to read the "mood" of our developer community and right now,
it feels like putting a lot of complexity hidden in the fetcher isn't
what people want to see as they don't understand it and can't "see"
what is going on. During the discussions, I think we identified some
key fundamental issues with implict urls for some fetch types too.

There is some hard work needs to be done in trying to summarise those
discussions and writing down the "results" so that we don't have to
redo this every time a new patch series comes along. By that, I mean a
non-emotive list of the current advantages, disdtantages and known bugs
of the current approach and any proposed alternatives we might choose.
 It perhaps falls to me as the developer lead for bitbake to try and do
it but I'd very much welcome help from anyone else in trying to do it
as I simply don't have the mental bandwidth to try and do that for this
topic right now (due to e.g. bitbake-setup). I certainly don't have to
be the one who does it. I appreciate it isn't a fun task though.


> This series is only a cleanup of the existing functionally. The gitsm
> fetcher uses implicit URLs but doesn't work correct because of the
> misuses of the download code.

I'm worried about where the series is trying to take the project and
codebase though, hence the questions about intent.


> It is useless to start the discussion again. The project doesn't like
> implicit URLs. It prefers a special task as a replacement for the
> separate recipetool. It decides against the on-the-fly parse inside
> the fetcher.

The developers using the project feel much happier with that approach,
yes.

> How should I proceed? I have working code which parse the cargo.lock,
> go.sum and package-lock.json files and only use Git and Wget
> fetchers. The code uses the vendor feature of the package managers to
> create a patchable folder of the sources. It simplifies the npm class
> and add additional classes to build packages. The code integrates
> valid URLs inside the SBOM and creates components with name and
> version per dependency. Furthermore, I have rework the gitsm fetcher
> to hopefully fix some open issues. I can convert my cargo, go, npm
> and gitsm parser into metadata helpers but this is useless if the
> project dislike python functions to generate URLs like the pypi class
> or prefer package manager specific code inside the fetcher.

So you are proposing we drop the crate and gomod fetchers in favour of
implict urls?

I'd suggest sharing a branch of your changes so that others can see and
understand the implications and hopefully experiment a bit, see if it
can convince some people that implict urls are the way forward.

Cheers,

Richard

Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 8571 bytes --]

Am 08.09.2025 um 12:26 schrieb Richard Purdie:
> On Mon, 2025-09-08 at 11:20 +0200, Stefan Herbrechtsmeier wrote:
>>   Am 07.09.2025 um 17:52 schrieb Richard Purdie via lists.openembedded.org:
>>   On Tue, 2025-09-02 at 08:55 +0200, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
>>>> The patch series add support for implicit URLs inside the fetcher. The
>>>> implicit URLs could be defined inside a source like a version control
>>>> system (git submodule) or a lock file (package-lock.json, cargo.lock or
>>>> go.sum). The integration of implicit URLs beside explicit URLs
>>>> simplifies the fetcher classes and avoid bugs because of iterations
>>>> between the Fetch and FetchMethod classes.
>>>>
>>>> The series remove most methods inside the gitsm fetcher and only leaves
>>>> the parsing of the git submodules and the unpack functionality. It
>>>> allows the gitsm fetcher to use the premirror only feature. The current
>>>> implementation leads to problems because the download of the git
>>>> submodules is triggered via the download method which is called deeply
>>>> inside the fetcher code.
>>>   We had the discussion a while back and the conclusion seemed to be that
>>> implict urls were disliked by a significant number of people as the
>>> were too unclear about what was going on behind the scenes and also
>>> made things like software manifests harder.
>>>    
>>>
>> It looks like I miss some discussion and especially the conclusion.
>>   
>>
>>   I assume you mean recipes by software manifests.
> No, I did mean software manifests. If the urls are explict, it makes
> generating manifests of the sources being used more obvious for people
> to understand.
Do you think the following is really helpful for a software manifest?

npm://registry.npmjs.org/;package=xyz;version=1.2.3
gomod://xyz;version=1.2.3
crate://crates.io/xyz/1.2.3

I assume software manifests needs the real URLs and a common style for 
the name and version.

How do we solve the problem if we keep the gitsm fetcher and doesn't 
propose an alternative solution?

> Yes, there are programmatic ways of doing it with
> implicit urls but people don't like them.

Based on your reaction regarding the gitsm fetcher I would say a lot of 
people like the implicit URLs. But without details it is impossible to 
understand the real problem. Maybe a task to generate a generic include 
file or a file in the work directory with explicit and implicit URLs 
solve their problems.

> By conclusions, I was taking that as the outcome of the last set of
> discussions but there was a lot of different emails and it was hard to
> follow. Perhaps i got the conclusion wrong, I don't know.

My understand was that implicit URLs have drawbacks but nothing we 
couldn't overcome.

>>>   There was a strong
>>> preference for metadata helpers and explicit lists of components
>>> which
>>> we have for crates/rust and now for go too.
>>>   
>>   
>> Does this mean the npmsw and gitsm fetchers are obsolete and should
>> be replace by a metadata helpers to fix open issues?
> I really don't know about npmsw. I don't use it and I don't really
> follow development there. I'm don't know much about the current set of
> issues it may have.
>
> With gitsm, I think that is generally accepted by people and I don't
> see a strong reason to change it at present.

Doesn't you say that people don't like implicit URLs and that they are 
problematical for software manifests?

>   I would be interested in a
> clear summary of what the known issues are (e.g. the premirror issue
> you mentioned).
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15875

The combination of BB_FETCH_PREMIRRORONLY and gitsm doesn’t work because 
the download of the submodules is handled inside the try_mirror_url 
function after the enable of BB_NO_NETWORK. The current code has a 
recurrence between the download method of the Fetch class and the 
download function of the FetchMethod class.

The need_update method is used to trigger a download of the submodules 
and the extract_urldata function is only acceptable after a download. It 
doesn't matter if bitbake support one or more fetchers with implicit 
URLs. Any user needs to handle it anyway. Either we should deprecate 
implicit URLs to simply the code or we should support it inside the 
Fetch class.

>>> It feels like this series is moving us back to the other direction.
>>> Is
>>> that correct and if so, what has changed in the approach since the
>>> last
>>> discussion?
>>>   
>>   
>> Do you mean the response to my last RFC? In this case it wasn't clear
>> to me that the project is against implicit URLs and that the npmsw
>> and gitsm fetcher are the wrong direction.
> I'm trying to read the "mood" of our developer community and right now,
> it feels like putting a lot of complexity hidden in the fetcher isn't
> what people want to see as they don't understand it and can't "see"
> what is going on. During the discussions, I think we identified some
> key fundamental issues with implict urls for some fetch types too.
That's the reason for this series. It adds native support for implicit 
URLs and doesn't misuse the need_update function. The implicit URL 
fetchers only need to implement the implicit_urls function.

> There is some hard work needs to be done in trying to summarise those
> discussions and writing down the "results" so that we don't have to
> redo this every time a new patch series comes along. By that, I mean a
> non-emotive list of the current advantages, disdtantages and known bugs
> of the current approach and any proposed alternatives we might choose.
>   It perhaps falls to me as the developer lead for bitbake to try and do
> it but I'd very much welcome help from anyone else in trying to do it
> as I simply don't have the mental bandwidth to try and do that for this
> topic right now (due to e.g. bitbake-setup). I certainly don't have to
> be the one who does it. I appreciate it isn't a fun task though.

What is the current approach? Do you mean implicit vs explicit URLs?

>> This series is only a cleanup of the existing functionally. The gitsm
>> fetcher uses implicit URLs but doesn't work correct because of the
>> misuses of the download code.
> I'm worried about where the series is trying to take the project and
> codebase though, hence the questions about intent.

I only rework the code and make features explicit. The fetchers already 
use implicit URLs but now it is clear that the localpaths method doesn't 
return the local paths of the implicit URLs. The implicit URLs are 
handled by the download method direct and not deeply inside the download 
method via the download method of the fetcher. A recursion inside the 
git submodules shouldn't be a problem because the outer download is 
finished before the inner download starts.

>> It is useless to start the discussion again. The project doesn't like
>> implicit URLs. It prefers a special task as a replacement for the
>> separate recipetool. It decides against the on-the-fly parse inside
>> the fetcher.
> The developers using the project feel much happier with that approach,
> yes.

I wonder why these developers prefer a single gitsm SRC_URI entry 
instead of a list of git entries because they lose the fine grain 
control in this case.

>> How should I proceed? I have working code which parse the cargo.lock,
>> go.sum and package-lock.json files and only use Git and Wget
>> fetchers. The code uses the vendor feature of the package managers to
>> create a patchable folder of the sources. It simplifies the npm class
>> and add additional classes to build packages. The code integrates
>> valid URLs inside the SBOM and creates components with name and
>> version per dependency. Furthermore, I have rework the gitsm fetcher
>> to hopefully fix some open issues. I can convert my cargo, go, npm
>> and gitsm parser into metadata helpers but this is useless if the
>> project dislike python functions to generate URLs like the pypi class
>> or prefer package manager specific code inside the fetcher.
> So you are proposing we drop the crate and gomod fetchers in favour of
> implict urls?
No. These changes are mutual exclusivity. In my case the crate and gomod 
fetcher aren't needed.

> I'd suggest sharing a branch of your changes so that others can see and
> understand the implications and hopefully experiment a bit, see if it
> can convince some people that implict urls are the way forward.
I will do but need some time because I focus on the rework of the gitsm 
fetcher.

Regards
   Stefan

[-- Attachment #2: Type: text/html, Size: 12364 bytes --]