[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[Steve Sakoman]

Please review this set of changes for scarthgap/2.8 and have comments back by
end of day Wednesday, June 19

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7047

The following changes since commit 0791e66a47185d6f202af6be0d39e36a8e41850a:

  tests/fetch: Tweak test to match upstream repo url change (2024-06-10 05:57:30 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (2):
  tests/fetch: Tweak to work on Fedora40
  fetch2/wget: Fix failure path for files that are empty or don't exist

Rudolf J Streif (1):
  fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

 lib/bb/fetch2/wget.py | 25 +++++++++++++------------
 lib/bb/tests/fetch.py |  6 +++---
 2 files changed, 16 insertions(+), 15 deletions(-)

-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 1/3] fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

[Steve Sakoman]

From: Rudolf J Streif <rudolf.streif@ibeeto.com>

Some distributions (namely Fedora Core 40) have started replacing
wget with wget2. There are some changes to wget2 that make it
incompatible with wget:

1. ftp/ftps is not supported anymore
2. progress 'dot' is not yet supported
3. Relative paths in -P and -O are not correctly dealt with

Item 1: Is already dealt with since Scarthgap by only adding the
option --passive-ftp when the URL specifies ftp/sftp. While that
won't help if ftp/sftp is actually required it at least does
not break http/https downloads.

Item 2: While not supported it at least does not break the operation.

Item 3: If there are relative path components in -P or -O then wget2
only deals with them correctly if there is one, and only one, relative
path component at the beginning of the path:

-P ./downloads     works
-P ../downloads    works
-P ../../downloads does not work
-P ./../downloads  does not work
-P /home/user/downloads/../downloads does not work

In cases where there are more than one relative path component at
the beginning of the path and/or one or more reltaive path
component somewhere in the middle or end of the path, wget2 aborts
with the message Internal error: Unexpected relative path: '<path>')

Such can happen if DL_DIR includes relative path components e.g.
DL_DIR = "${TOPDIR}/../../downloads".

This patch canonicalizes DL_DIR before it is passed to wget.

Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/wget.py | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index fbfa6938a..b41f53a9e 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -108,7 +108,8 @@ class Wget(FetchMethod):
 
         fetchcmd = self.basecmd
 
-        localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp"
+        dldir = os.path.realpath(d.getVar("DL_DIR"))
+        localpath = os.path.join(dldir, ud.localfile) + ".tmp"
         bb.utils.mkdirhier(os.path.dirname(localpath))
         fetchcmd += " -O %s" % shlex.quote(localpath)
 
@@ -128,9 +129,9 @@ class Wget(FetchMethod):
         uri = ud.url.split(";")[0]
         if os.path.exists(ud.localpath):
             # file exists, but we didnt complete it.. trying again..
-            fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
+            fetchcmd += " -c -P " + dldir + " '" + uri + "'"
         else:
-            fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri)
+            fetchcmd += " -P " + dldir + " '" + uri + "'"
 
         self._runwget(ud, d, fetchcmd, False)
 
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 2/3] tests/fetch: Tweak to work on Fedora40

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.

Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/tests/fetch.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index 8a91f51d3..33cc9bcac 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1511,7 +1511,7 @@ class FetchLatestVersionTest(FetcherTest):
 
     def test_wget_latest_versionstring(self):
         testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata"
-        server = HTTPService(testdata)
+        server = HTTPService(testdata, host="127.0.0.1")
         server.start()
         port = server.port
         try:
@@ -1519,10 +1519,10 @@ class FetchLatestVersionTest(FetcherTest):
                 self.d.setVar("PN", k[0])
                 checkuri = ""
                 if k[2]:
-                    checkuri = "http://localhost:%s/" % port + k[2]
+                    checkuri = "http://127.0.0.1:%s/" % port + k[2]
                 self.d.setVar("UPSTREAM_CHECK_URI", checkuri)
                 self.d.setVar("UPSTREAM_CHECK_REGEX", k[3])
-                url = "http://localhost:%s/" % port + k[1]
+                url = "http://127.0.0.1:%s/" % port + k[1]
                 ud = bb.fetch2.FetchData(url, self.d)
                 pupver = ud.method.latest_versionstring(ud, self.d)
                 verstring = pupver[0]
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 3/3] fetch2/wget: Fix failure path for files that are empty or don't exist

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

When we intercepted the file download to a temp file, we broke the
exist/size checks which need to happen before the rename. Correct
the ordering.

For some reason, python 3.12 exposes this problem in the selftests
differently to previous versions.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit c56bd9a9280378bc64c6a7fe6d7b70847e0b9e6d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/fetch2/wget.py | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index b41f53a9e..2e9211763 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -135,6 +135,15 @@ class Wget(FetchMethod):
 
         self._runwget(ud, d, fetchcmd, False)
 
+        # Sanity check since wget can pretend it succeed when it didn't
+        # Also, this used to happen if sourceforge sent us to the mirror page
+        if not os.path.exists(localpath):
+            raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, localpath), uri)
+
+        if os.path.getsize(localpath) == 0:
+            os.remove(localpath)
+            raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri)
+
         # Try and verify any checksum now, meaning if it isn't correct, we don't remove the
         # original file, which might be a race (imagine two recipes referencing the same
         # source, one with an incorrect checksum)
@@ -144,15 +153,6 @@ class Wget(FetchMethod):
         # Our lock prevents multiple writers but mirroring code may grab incomplete files
         os.rename(localpath, localpath[:-4])
 
-        # Sanity check since wget can pretend it succeed when it didn't
-        # Also, this used to happen if sourceforge sent us to the mirror page
-        if not os.path.exists(ud.localpath):
-            raise FetchError("The fetch command returned success for url %s but %s doesn't exist?!" % (uri, ud.localpath), uri)
-
-        if os.path.getsize(ud.localpath) == 0:
-            os.remove(ud.localpath)
-            raise FetchError("The fetch of %s resulted in a zero size file?! Deleting and failing since this isn't right." % (uri), uri)
-
         return True
 
     def checkstatus(self, fetch, ud, d, try_again=True):
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[Steve Sakoman]

Please review this set of changes for scarthgap/2.8 and have comments back
by end of day Friday, December 6

Passed a-full on autobuilder:

https://valkyrie.yoctoproject.org/#/builders/29/builds/582

The following changes since commit 6c2641f7a9e92c1b82e306f59ddd3c1249c52cbf:

  fetch2: use persist_data context managers (2024-11-28 13:59:58 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (3):
  runqueue: Fix performance of multiconfigs with large overlap
  runqueue: Optimise setscene loop processing
  runqueue: Fix scenetask processing performance issue

 lib/bb/runqueue.py | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[Steve Sakoman]

Please reviwe this set of changes for 2.8/scarthgap and have comments back
by end of day Thursday, January 23

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/856

The following changes since commit f40a3a477d5241b697bf2fb030dd804c1ff5839f:

  runqueue: Fix scenetask processing performance issue (2024-12-03 06:34:38 -0800)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Chris Laplante (3):
  ui/knotty: print log paths for failed tasks in summary
  ui/knotty: respect NO_COLOR & check for tty; rename print_hyperlink =>
    format_hyperlink
  cooker: Make cooker 'skiplist' per-multiconfig/mc

 lib/bb/command.py     | 21 ++++++++++++++++++---
 lib/bb/cooker.py      | 11 ++++++-----
 lib/bb/tinfoil.py     | 16 ++++++++++++----
 lib/bb/ui/knotty.py   | 20 +++++++++++++++++---
 lib/bblayers/query.py | 14 +++++++-------
 5 files changed, 60 insertions(+), 22 deletions(-)

-- 
2.43.0