[RFC PATCH v2 0/7] fetch2: add support for implicit urls

[RFC PATCH v2 0/7] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

The patch series add support for implicit URLs inside the fetcher. The
implicit URLs could be defined inside a source like a version control
system (git submodule) or a lock file (package-lock.json, cargo.lock or
go.sum). The integration of implicit URLs beside explicit URLs
simplifies the fetcher classes and avoid bugs because of iterations
between the Fetch and FetchMethod classes.

The series remove most methods inside the gitsm fetcher and only leaves
the parsing of the git submodules and the unpack functionality. It
allows the gitsm fetcher to use the premirror only feature. The current
implementation leads to problems because the download of the git
submodules is triggered via the download method which is called deeply
inside the fetcher code.

Changes in v2:
- Move test case for gitsm local paths into FetcherLocalTest class
- Remove implicit urls from localpaths
- Add commit to test the expanded_urldata function
- Fix UnboundLocalError of urldata in expand_urldata function
- Remove implicit URLs from localpaths for backward compatibility

Stefan Herbrechtsmeier (7):
  tests: fetch: add test case for gitsm implicit local paths
  tests: fetch: add test cases for expanded_urldata
  fetch2: rename u to url in Fetch class
  fetch2: call functions within loops of Fetch class
  fetch2: add helper to get urldata in Fetch class
  fetch2: add support for implicit urls
  fetch2: gitsm: use implicit urls feature

 lib/bb/fetch2/__init__.py | 127 +++++++++++++++++++++++++++-----------
 lib/bb/fetch2/gitsm.py    |  46 ++------------
 lib/bb/tests/fetch.py     |  59 ++++++++++++++++++
 3 files changed, 154 insertions(+), 78 deletions(-)

-- 
2.39.5

[RFC PATCH v2 1/7] tests: fetch: add test case for gitsm implicit local paths

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

Changes in v2:
- Move test case for gitsm local paths into FetcherLocalTest class
- Remove implicit urls from localpaths

 lib/bb/tests/fetch.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index dcbe60783..feaba235b 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -900,6 +900,21 @@ class FetcherLocalTest(FetcherTest):
         fetcher = bb.fetch2.Fetch(localpaths.keys(), self.d)
         self.assertEqual(fetcher.localpaths(), list(localpaths.values()))
 
+    def test_git_submodule_localpaths(self):
+        urls = [
+            "gitsm://git.yoctoproject.org/git-submodule-test;protocol=https"
+                ";branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee"
+        ]
+        filenames = [
+            "git.yoctoproject.org.git-submodule-test"
+        ]
+        fetcher = bb.fetch.Fetch(urls, self.d)
+        localpaths = fetcher.localpaths()
+        self.assertEqual(len(localpaths), len(filenames))
+        for localpath, filename in zip(localpaths, filenames):
+            l = os.path.join(self.dldir, "git2", filename)
+            self.assertEqual(localpath, l)
+
 class FetcherNoNetworkTest(FetcherTest):
     def setUp(self):
         super().setUp()
-- 
2.39.5

[RFC PATCH v2 2/7] tests: fetch: add test cases for expanded_urldata

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Add test cases for the expanded_urldata function. The test cases
contains plain tests for local, wget and git fetcher and an extended
test for the gitsm fetcher.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

Changes in v2:
- Add commit to test the expanded_urldata function

 lib/bb/tests/fetch.py | 44 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index feaba235b..806870198 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -915,6 +915,20 @@ class FetcherLocalTest(FetcherTest):
             l = os.path.join(self.dldir, "git2", filename)
             self.assertEqual(localpath, l)
 
+    def test_expanded_urldata(self):
+        urls = [
+            "file://archive.tar.gz",
+            "https://downloads.yoctoproject.org/releases/bitbake/bitbake-1.0.tar.gz",
+            "git://git.openembedded.org/bitbake;branch=master;protocol=https"
+                ";rev=270a05b0b4ba0959fe0624d2a4885d7b70426da5"
+        ]
+        fetcher = bb.fetch.Fetch(urls, self.d)
+        urldata = fetcher.expanded_urldata()
+
+        self.assertEqual(len(urldata), len(urls))
+        for ud, url in zip(urldata, urls):
+            self.assertEqual(ud.url, url)
+
 class FetcherNoNetworkTest(FetcherTest):
     def setUp(self):
         super().setUp()
@@ -1316,6 +1330,36 @@ class FetcherNetworkTest(FetcherTest):
             with self.assertRaises(bb.fetch2.FetchError):
                 fetcher.download()
 
+    @skipIfNoNetwork()
+    def test_git_submodule_expanded_urldata(self):
+        expected_urldata = [
+            (
+                "gitsm://git.yoctoproject.org/git-submodule-test;protocol=https"
+                    ";branch=master;rev=a2885dd7d25380d23627e7544b7bbb55014b16ee",
+                "a2885dd7d25380d23627e7544b7bbb55014b16ee"
+            ), (
+                "gitsm://git.openembedded.org/bitbake;protocol=git",
+                "c39b99792547b642570ea5152070e7396e812390"
+            ), (
+                "gitsm://git.yoctoproject.org/bitbake-gitsm-test1;protocol=git",
+                "120f4c731cdc4a24cd29e44c4c02c4c658d8bc0e"
+            ), (
+                "gitsm://git.openembedded.org/bitbake;protocol=git",
+                "52a144a7daa94b2bd239d582cb71d1f03119918f"
+            ), (
+                "gitsm://git.yoctoproject.org/bitbake-gitsm-test2;protocol=git",
+                "f66699ed76c2b21ebe2a94ee23450415d2c7dee4"
+            )
+        ]
+        fetcher = bb.fetch.Fetch([expected_urldata[0][0]], self.d)
+        fetcher.download()
+        urldata = fetcher.expanded_urldata()
+
+        self.assertEqual(len(urldata), len(expected_urldata))
+        for ud, (url, rev) in zip(urldata, expected_urldata):
+            self.assertEqual(ud.url[:len(url)], url)
+            self.assertEqual(ud.revision, rev)
+
 class SVNTest(FetcherTest):
     def skipIfNoSvn():
         if not shutil.which("svn"):
-- 
2.39.5

[RFC PATCH v2 3/7] fetch2: rename u to url in Fetch class

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Rename the varialbe u to url in the Fetch class to clarify its meaning.
Avoid using short variable names over long ranges and simplify
subsequent changes.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

(no changes since v1)

 lib/bb/fetch2/__init__.py | 42 +++++++++++++++++++--------------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index de301543a..bc9808da1 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1841,8 +1841,8 @@ class Fetch(object):
         """
         local = []
 
-        for u in self.urls:
-            ud = self.ud[u]
+        for url in self.urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
 
@@ -1859,8 +1859,8 @@ class Fetch(object):
         premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
 
         checksum_missing_messages = []
-        for u in urls:
-            ud = self.ud[u]
+        for url in urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
             done = False
@@ -1882,7 +1882,7 @@ class Fetch(object):
                             # contents mismatch the fetcher can still try upstream and mirrors
                             m.update_donestamp(ud, self.d)
                         except ChecksumError as e:
-                            logger.warning("Checksum failure encountered with premirror download of %s - will attempt other sources." % u)
+                            logger.warning("Checksum failure encountered with premirror download of %s - will attempt other sources." % url)
                             logger.debug(str(e))
                             done = False
 
@@ -1914,14 +1914,14 @@ class Fetch(object):
 
                     except BBFetchException as e:
                         if isinstance(e, ChecksumError):
-                            logger.warning("Checksum failure encountered with download of %s - will attempt other sources if available" % u)
+                            logger.warning("Checksum failure encountered with download of %s - will attempt other sources if available" % url)
                             logger.debug(str(e))
                             if os.path.exists(ud.localpath):
                                 rename_bad_checksum(ud, e.checksum)
                         elif isinstance(e, NoChecksumError):
                             raise
                         else:
-                            logger.warning('Failed to fetch URL %s, attempting MIRRORS if available' % u)
+                            logger.warning('Failed to fetch URL %s, attempting MIRRORS if available' % url)
                             logger.debug(str(e))
                         firsterr = e
                         # Remove any incomplete fetch
@@ -1934,13 +1934,13 @@ class Fetch(object):
                 if not done or not m.done(ud, d):
                     if firsterr:
                         logger.error(str(firsterr))
-                    raise FetchError("Unable to fetch URL from any source.", u)
+                    raise FetchError("Unable to fetch URL from any source.", url)
 
                 m.update_donestamp(ud, d)
 
             except IOError as e:
                 if e.errno in [errno.ESTALE]:
-                    logger.error("Stale Error Observed %s." % u)
+                    logger.error("Stale Error Observed %s." % url)
                     raise ChecksumError("Stale Error Detected")
 
             except BBFetchException as e:
@@ -1949,7 +1949,7 @@ class Fetch(object):
                     checksum_missing_messages.append(message)
                     continue
                 elif isinstance(e, ChecksumError):
-                    logger.error("Checksum failure fetching %s" % u)
+                    logger.error("Checksum failure fetching %s" % url)
                 raise
 
             finally:
@@ -1971,24 +1971,24 @@ class Fetch(object):
         if not urls:
             urls = self.urls
 
-        for u in urls:
-            ud = self.ud[u]
+        for url in urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
-            logger.debug("Testing URL %s", u)
-            # First try checking uri, u, from PREMIRRORS
+            logger.debug("Testing URL %s", url)
+            # First try checking uri, url, from PREMIRRORS
             mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
             ret = m.try_mirrors(self, ud, self.d, mirrors, True)
             if not ret:
-                # Next try checking from the original uri, u
+                # Next try checking from the original uri, url
                 ret = m.checkstatus(self, ud, self.d)
                 if not ret:
-                    # Finally, try checking uri, u, from MIRRORS
+                    # Finally, try checking uri, url, from MIRRORS
                     mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
                     ret = m.try_mirrors(self, ud, self.d, mirrors, True)
 
             if not ret:
-                raise FetchError("URL doesn't work", u)
+                raise FetchError("URL doesn't work", url)
 
     def unpack(self, root, urls=None):
         """
@@ -2000,16 +2000,16 @@ class Fetch(object):
 
         unpack_tracer.start(root, self.ud, self.d)
 
-        for u in urls:
-            ud = self.ud[u]
+        for url in urls:
+            ud = self.ud[url]
             ud.setup_localpath(self.d)
 
             if ud.lockfile:
                 lf = bb.utils.lockfile(ud.lockfile)
 
-            unpack_tracer.start_url(u)
+            unpack_tracer.start_url(url)
             ud.method.unpack(ud, root, self.d)
-            unpack_tracer.finish_url(u)
+            unpack_tracer.finish_url(url)
 
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
-- 
2.39.5

[RFC PATCH v2 4/7] fetch2: call functions within loops of Fetch class

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Call functions within the for all URLs loops of the Fetch class to
simplify subsequent changes.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

Changes in v2:
- Fix UnboundLocalError of urldata in expand_urldata function

 lib/bb/fetch2/__init__.py | 43 ++++++++++++++++++++++++++++-----------
 1 file changed, 31 insertions(+), 12 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index bc9808da1..3ff08f817 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1841,11 +1841,14 @@ class Fetch(object):
         """
         local = []
 
-        for url in self.urls:
+        def localpath(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
 
+        for url in self.urls:
+            localpath(url)
+
         return local
 
     def download(self, urls=None):
@@ -1859,7 +1862,7 @@ class Fetch(object):
         premirroronly = bb.utils.to_boolean(self.d.getVar("BB_FETCH_PREMIRRORONLY"))
 
         checksum_missing_messages = []
-        for url in urls:
+        def download(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
@@ -1947,14 +1950,18 @@ class Fetch(object):
                 if isinstance(e, NoChecksumError):
                     (message, _) = e.args
                     checksum_missing_messages.append(message)
-                    continue
-                elif isinstance(e, ChecksumError):
-                    logger.error("Checksum failure fetching %s" % url)
-                raise
+                else:
+                    if isinstance(e, ChecksumError):
+                        logger.error("Checksum failure fetching %s" % url)
+                    raise
 
             finally:
                 if ud.lockfile:
                     bb.utils.unlockfile(lf)
+
+        for url in urls:
+            download(url)
+
         if checksum_missing_messages:
             logger.error("Missing SRC_URI checksum, please add those to the recipe: \n%s", "\n".join(checksum_missing_messages))
             raise BBFetchException("There was some missing checksums in the recipe")
@@ -1971,7 +1978,7 @@ class Fetch(object):
         if not urls:
             urls = self.urls
 
-        for url in urls:
+        def checkstatus(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
             m = ud.method
@@ -1990,6 +1997,9 @@ class Fetch(object):
             if not ret:
                 raise FetchError("URL doesn't work", url)
 
+        for url in urls:
+            checkstatus(url)
+
     def unpack(self, root, urls=None):
         """
         Unpack urls to root
@@ -2000,7 +2010,7 @@ class Fetch(object):
 
         unpack_tracer.start(root, self.ud, self.d)
 
-        for url in urls:
+        def unpack(url):
             ud = self.ud[url]
             ud.setup_localpath(self.d)
 
@@ -2014,6 +2024,9 @@ class Fetch(object):
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
 
+        for url in urls:
+            unpack(url)
+
         unpack_tracer.complete()
 
     def clean(self, urls=None):
@@ -2024,14 +2037,14 @@ class Fetch(object):
         if not urls:
             urls = self.urls
 
-        for url in urls:
+        def clean(url):
             if url not in self.ud:
                 self.ud[url] = FetchData(url, self.d)
             ud = self.ud[url]
             ud.setup_localpath(self.d)
 
             if not ud.localfile and ud.localpath is None:
-                continue
+                return
 
             if ud.lockfile:
                 lf = bb.utils.lockfile(ud.lockfile)
@@ -2043,6 +2056,9 @@ class Fetch(object):
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
 
+        for url in urls:
+            clean(url)
+
     def expanded_urldata(self, urls=None):
         """
         Get an expanded list of FetchData objects covering both the given
@@ -2054,10 +2070,13 @@ class Fetch(object):
             urls = self.urls
 
         urldata = []
-        for url in urls:
+        def expand_urldata(url):
             ud = self.ud[url]
             urldata.append(ud)
-            urldata += ud.method.implicit_urldata(ud, self.d)
+            urldata.extend(ud.method.implicit_urldata(ud, self.d))
+
+        for url in urls:
+            expand_urldata(url)
 
         return urldata
 
-- 
2.39.5

[RFC PATCH v2 5/7] fetch2: add helper to get urldata in Fetch class

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Add a private helper function to get the urldata object of a given URL.
Create the urldate object if it doesn't exist inside the urldata object
list of the fetch object.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
---

(no changes since v1)

 lib/bb/fetch2/__init__.py | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 3ff08f817..c4d7cc033 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1829,11 +1829,9 @@ class Fetch(object):
             urldata_cache[key] = self.ud
 
     def localpath(self, url):
-        if url not in self.urls:
-            self.ud[url] = FetchData(url, self.d)
-
-        self.ud[url].setup_localpath(self.d)
-        return self.ud[url].localpath
+        ud = self._get_urldata(url)
+        ud.setup_localpath(self.d)
+        return ud.localpath
 
     def localpaths(self):
         """
@@ -1842,7 +1840,7 @@ class Fetch(object):
         local = []
 
         def localpath(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             local.append(ud.localpath)
 
@@ -1863,7 +1861,7 @@ class Fetch(object):
 
         checksum_missing_messages = []
         def download(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             m = ud.method
             done = False
@@ -1979,7 +1977,7 @@ class Fetch(object):
             urls = self.urls
 
         def checkstatus(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
             m = ud.method
             logger.debug("Testing URL %s", url)
@@ -2011,7 +2009,7 @@ class Fetch(object):
         unpack_tracer.start(root, self.ud, self.d)
 
         def unpack(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
 
             if ud.lockfile:
@@ -2038,9 +2036,7 @@ class Fetch(object):
             urls = self.urls
 
         def clean(url):
-            if url not in self.ud:
-                self.ud[url] = FetchData(url, self.d)
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
 
             if not ud.localfile and ud.localpath is None:
@@ -2071,7 +2067,7 @@ class Fetch(object):
 
         urldata = []
         def expand_urldata(url):
-            ud = self.ud[url]
+            ud = self._get_urldata(url)
             urldata.append(ud)
             urldata.extend(ud.method.implicit_urldata(ud, self.d))
 
@@ -2080,6 +2076,12 @@ class Fetch(object):
 
         return urldata
 
+    def _get_urldata(self, url):
+        if url not in self.ud:
+            self.ud[url] = FetchData(url, self.d)
+            self.ud[url].unpack_tracer = unpack_tracer
+        return self.ud[url]
+
 class FetchConnectionCache(object):
     """
         A class which represents an container for socket connections.
-- 
2.39.5

[RFC PATCH v2 6/7] fetch2: add support for implicit urls

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Add support for implicit URLs which are defined inside another source
like a version control system (git submodule) or an embedded lock file
(package-lock.json, cargo.lock or go.sum). The implicit URLs are
extracted from the source and thereby only available after the download
of the explicit source. The integration of implicit URLs beside explicit
URLs simplifies the fetcher classes and avoid bugs because of iterations
between the Fetch and FetchMethod classes.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

Changes in v2:
- Remove implicit URLs from localpaths for backward compatibility

 lib/bb/fetch2/__init__.py | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index c4d7cc033..1bf16b7e6 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -1312,6 +1312,7 @@ class FetchData(object):
         if not self.pswd and "pswd" in self.parm:
             self.pswd = self.parm["pswd"]
         self.setup = False
+        self.implicit_urls = None
 
         def configure_checksum(checksum_id):
             checksum_plain_name = "%ssum" % checksum_id
@@ -1409,6 +1410,11 @@ class FetchData(object):
 
         return d.getVar("SRCDATE") or d.getVar("DATE")
 
+    def setup_implicit_urls(self, d):
+        if self.implicit_urls is None:
+            self.setup_localpath(d)
+            self.implicit_urls = self.method.implicit_urls(self, d)
+
 class FetchMethod(object):
     """Base class for 'fetch'ing data"""
 
@@ -1727,6 +1733,9 @@ class FetchMethod(object):
         """
         return []
 
+    def implicit_urls(self, ud, d):
+        return []
+
 
 class DummyUnpackTracer(object):
     """
@@ -1960,6 +1969,16 @@ class Fetch(object):
         for url in urls:
             download(url)
 
+        def download_implicit(url):
+            ud = self._get_urldata(url)
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                download(u)
+                download_implicit(u)
+
+        for url in urls:
+            download_implicit(url)
+
         if checksum_missing_messages:
             logger.error("Missing SRC_URI checksum, please add those to the recipe: \n%s", "\n".join(checksum_missing_messages))
             raise BBFetchException("There was some missing checksums in the recipe")
@@ -2022,6 +2041,12 @@ class Fetch(object):
             if ud.lockfile:
                 bb.utils.unlockfile(lf)
 
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                unpack(u)
+            if hasattr(ud.method, "postunpack"):
+                ud.method.postunpack(ud, root, self.d)
+
         for url in urls:
             unpack(url)
 
@@ -2038,6 +2063,10 @@ class Fetch(object):
         def clean(url):
             ud = self._get_urldata(url)
             ud.setup_localpath(self.d)
+            if os.path.exists(ud.localpath):
+                ud.setup_implicit_urls(self.d)
+                for u in ud.implicit_urls:
+                    clean(u)
 
             if not ud.localfile and ud.localpath is None:
                 return
@@ -2070,6 +2099,9 @@ class Fetch(object):
             ud = self._get_urldata(url)
             urldata.append(ud)
             urldata.extend(ud.method.implicit_urldata(ud, self.d))
+            ud.setup_implicit_urls(self.d)
+            for u in ud.implicit_urls:
+                expand_urldata(u)
 
         for url in urls:
             expand_urldata(url)
-- 
2.39.5

[RFC PATCH v2 7/7] fetch2: gitsm: use implicit urls feature

[Stefan Herbrechtsmeier]

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Return the submodule URLs via implicit_urls function and remove the
manual handling of the submodule urls.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

(no changes since v1)

 lib/bb/fetch2/gitsm.py | 46 +++++-------------------------------------
 1 file changed, 5 insertions(+), 41 deletions(-)

diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index 5ddc81f86..90fe92f74 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -24,7 +24,6 @@ import tempfile
 from   bb.fetch2.git import Git
 from   bb.fetch2 import runfetchcmd
 from   bb.fetch2 import logger
-from   bb.fetch2 import Fetch
 
 class GitSM(Git):
     def supports(self, ud, d):
@@ -141,9 +140,9 @@ class GitSM(Git):
 
             urls.append(url)
 
-        return Fetch(urls, d, cache=False) if urls else None
+        return urls
 
-    def call_process_submodules(self, ud, d):
+    def implicit_urls(self, ud, d):
         # If we're using a shallow mirror tarball it needs to be
         # unpacked temporarily so that we can examine the .gitmodules file
         # Unpack even when ud.clonedir is not available,
@@ -157,27 +156,6 @@ class GitSM(Git):
         else:
             raise bb.fetch2.FetchError("Submodule source not available.")
 
-    def need_update(self, ud, d):
-        if Git.need_update(self, ud, d):
-            return True
-
-        need_update = False
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            for url in fetch.urls:
-                urldata = fetch.ud[url]
-                if urldata.method.need_update(urldata, d):
-                    need_update = True
-
-        return need_update
-
-    def download(self, ud, d):
-        Git.download(self, ud, d)
-
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            fetch.download()
-
     def unpack(self, ud, destdir, d):
         fulldestdir = self.destdir(ud, destdir, d)
 
@@ -204,25 +182,11 @@ class GitSM(Git):
                 logger.error("Unable to set git config core.bare to false for %s" % fulldestdir)
                 raise
 
-        fetch = self.process_submodules(ud, fulldestdir, d)
-        if fetch:
-            fetch.unpack(destdir)
-
-        if not ud.bareclone and fetch:
+    def postunpack(self, ud, destdir, d):
+        fulldestdir = self.destdir(ud, destdir, d)
+        if not ud.bareclone:
             cmdprefix = ""
             # Avoid LFS smudging (replacing the LFS pointers with the actual content) when LFS shouldn't be used but git-lfs is installed.
             if not self._need_lfs(ud):
                 cmdprefix = "GIT_LFS_SKIP_SMUDGE=1 "
             runfetchcmd("%s%s submodule update --recursive --no-fetch" % (cmdprefix, ud.basecmd), d, quiet=True, workdir=fulldestdir)
-    def clean(self, ud, d):
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            fetch.clean()
-        Git.clean(self, ud, d)
-
-    def implicit_urldata(self, ud, d):
-        fetch = self.call_process_submodules(ud, d)
-        if fetch:
-            return fetch.expanded_urldata()
-
-        return []
-- 
2.39.5

Re: [bitbake-devel] [RFC PATCH v2 0/7] fetch2: add support for implicit urls

[Mathieu Dubois-Briand]

On Fri Sep 5, 2025 at 8:44 AM CEST, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> The patch series add support for implicit URLs inside the fetcher. The
> implicit URLs could be defined inside a source like a version control
> system (git submodule) or a lock file (package-lock.json, cargo.lock or
> go.sum). The integration of implicit URLs beside explicit URLs
> simplifies the fetcher classes and avoid bugs because of iterations
> between the Fetch and FetchMethod classes.
>
> The series remove most methods inside the gitsm fetcher and only leaves
> the parsing of the git submodules and the unpack functionality. It
> allows the gitsm fetcher to use the premirror only feature. The current
> implementation leads to problems because the download of the git
> submodules is triggered via the download method which is called deeply
> inside the fetcher code.
>
> Changes in v2:
> - Move test case for gitsm local paths into FetcherLocalTest class
> - Remove implicit urls from localpaths
> - Add commit to test the expanded_urldata function
> - Fix UnboundLocalError of urldata in expand_urldata function
> - Remove implicit URLs from localpaths for backward compatibility
>
> Stefan Herbrechtsmeier (7):
>   tests: fetch: add test case for gitsm implicit local paths
>   tests: fetch: add test cases for expanded_urldata
>   fetch2: rename u to url in Fetch class
>   fetch2: call functions within loops of Fetch class
>   fetch2: add helper to get urldata in Fetch class
>   fetch2: add support for implicit urls
>   fetch2: gitsm: use implicit urls feature
>
>  lib/bb/fetch2/__init__.py | 127 +++++++++++++++++++++++++++-----------
>  lib/bb/fetch2/gitsm.py    |  46 ++------------
>  lib/bb/tests/fetch.py     |  59 ++++++++++++++++++
>  3 files changed, 154 insertions(+), 78 deletions(-)

Thanks,

I took a bit of time to test this new version, no issue on the
autobuilder.

Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com