* [bitbake][kirkstone][2.0][PATCH 1/4] wget: Make wget --passive-ftp option conditional on ftp/ftps
2024-10-05 13:44 [bitbake][kirkstone][2.0][PATCH 0/4] Patch review Steve Sakoman
@ 2024-10-05 13:44 ` Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 2/4] tests/fetch: Tweak to work on Fedora40 Steve Sakoman
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Steve Sakoman @ 2024-10-05 13:44 UTC (permalink / raw)
To: bitbake-devel
From: Rob Woolley <rob.woolley@windriver.com>
Fedora 40 introduces wget2 as a drop-in replacement for wget. This
rewrite does not currently have support for FTP. This causes
the wget fetcher to fail complaining about an unrecognized option.
Making --passive-ftp conditional based on the protocol used in
the SRC_URI limits the scope of the problem. It also gives us
an opportunity to build the older wget as a host tool.
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f10e630fd7561746d835a4378e8777e78f56e44a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/wget.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index b3a3de571..9bcb0d4b7 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -88,7 +88,10 @@ class Wget(FetchMethod):
if not ud.localfile:
ud.localfile = d.expand(urllib.parse.unquote(ud.host + ud.path).replace("/", "."))
- self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30 --passive-ftp"
+ self.basecmd = d.getVar("FETCHCMD_wget") or "/usr/bin/env wget -t 2 -T 30"
+
+ if ud.type == 'ftp' or ud.type == 'ftps':
+ self.basecmd += " --passive-ftp"
if not self.check_certs(d):
self.basecmd += " --no-check-certificate"
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [bitbake][kirkstone][2.0][PATCH 2/4] tests/fetch: Tweak to work on Fedora40
2024-10-05 13:44 [bitbake][kirkstone][2.0][PATCH 0/4] Patch review Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 1/4] wget: Make wget --passive-ftp option conditional on ftp/ftps Steve Sakoman
@ 2024-10-05 13:44 ` Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 3/4] fetch/wget: Move files into place atomically Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 4/4] fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility Steve Sakoman
3 siblings, 0 replies; 6+ messages in thread
From: Steve Sakoman @ 2024-10-05 13:44 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
On Fedora40, "localhost" sometimes resolves to ::1 and sometimes to 127.0.0.1
and python only binds to one of the addresses, leading to test failures.
Use 127.0.0.1 explicitly to avoid problems of the name resolution, we're trying
to test things other than the host networking.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9adc6da42618f41bf0d6b558d62b2f3c13bedd61)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/tests/fetch.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index a7d9d5e88..efe5479a6 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -1417,7 +1417,7 @@ class FetchLatestVersionTest(FetcherTest):
def test_wget_latest_versionstring(self):
testdata = os.path.dirname(os.path.abspath(__file__)) + "/fetch-testdata"
- server = HTTPService(testdata)
+ server = HTTPService(testdata, host="127.0.0.1")
server.start()
port = server.port
try:
@@ -1425,10 +1425,10 @@ class FetchLatestVersionTest(FetcherTest):
self.d.setVar("PN", k[0])
checkuri = ""
if k[2]:
- checkuri = "http://localhost:%s/" % port + k[2]
+ checkuri = "http://127.0.0.1:%s/" % port + k[2]
self.d.setVar("UPSTREAM_CHECK_URI", checkuri)
self.d.setVar("UPSTREAM_CHECK_REGEX", k[3])
- url = "http://localhost:%s/" % port + k[1]
+ url = "http://127.0.0.1:%s/" % port + k[1]
ud = bb.fetch2.FetchData(url, self.d)
pupver = ud.method.latest_versionstring(ud, self.d)
verstring = pupver[0]
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [bitbake][kirkstone][2.0][PATCH 3/4] fetch/wget: Move files into place atomically
2024-10-05 13:44 [bitbake][kirkstone][2.0][PATCH 0/4] Patch review Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 1/4] wget: Make wget --passive-ftp option conditional on ftp/ftps Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 2/4] tests/fetch: Tweak to work on Fedora40 Steve Sakoman
@ 2024-10-05 13:44 ` Steve Sakoman
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 4/4] fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility Steve Sakoman
3 siblings, 0 replies; 6+ messages in thread
From: Steve Sakoman @ 2024-10-05 13:44 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cd7cce4cf4be5c742d29671169354fe84220b47a)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/wget.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index 9bcb0d4b7..708f3a42e 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -109,10 +109,9 @@ class Wget(FetchMethod):
fetchcmd = self.basecmd
- if 'downloadfilename' in ud.parm:
- localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile)
- bb.utils.mkdirhier(os.path.dirname(localpath))
- fetchcmd += " -O %s" % shlex.quote(localpath)
+ localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp"
+ bb.utils.mkdirhier(os.path.dirname(localpath))
+ fetchcmd += " -O %s" % shlex.quote(localpath)
if ud.user and ud.pswd:
fetchcmd += " --auth-no-challenge"
@@ -136,6 +135,10 @@ class Wget(FetchMethod):
self._runwget(ud, d, fetchcmd, False)
+ # Remove the ".tmp" and move the file into position atomically
+ # Our lock prevents multiple writers but mirroring code may grab incomplete files
+ os.rename(localpath, localpath[:-4])
+
# Sanity check since wget can pretend it succeed when it didn't
# Also, this used to happen if sourceforge sent us to the mirror page
if not os.path.exists(ud.localpath):
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [bitbake][kirkstone][2.0][PATCH 4/4] fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility
2024-10-05 13:44 [bitbake][kirkstone][2.0][PATCH 0/4] Patch review Steve Sakoman
` (2 preceding siblings ...)
2024-10-05 13:44 ` [bitbake][kirkstone][2.0][PATCH 3/4] fetch/wget: Move files into place atomically Steve Sakoman
@ 2024-10-05 13:44 ` Steve Sakoman
3 siblings, 0 replies; 6+ messages in thread
From: Steve Sakoman @ 2024-10-05 13:44 UTC (permalink / raw)
To: bitbake-devel
From: Rudolf J Streif <rudolf.streif@ibeeto.com>
Some distributions (namely Fedora Core 40) have started replacing
wget with wget2. There are some changes to wget2 that make it
incompatible with wget:
1. ftp/ftps is not supported anymore
2. progress 'dot' is not yet supported
3. Relative paths in -P and -O are not correctly dealt with
Item 1: Is already dealt with since Scarthgap by only adding the
option --passive-ftp when the URL specifies ftp/sftp. While that
won't help if ftp/sftp is actually required it at least does
not break http/https downloads.
Item 2: While not supported it at least does not break the operation.
Item 3: If there are relative path components in -P or -O then wget2
only deals with them correctly if there is one, and only one, relative
path component at the beginning of the path:
-P ./downloads works
-P ../downloads works
-P ../../downloads does not work
-P ./../downloads does not work
-P /home/user/downloads/../downloads does not work
In cases where there are more than one relative path component at
the beginning of the path and/or one or more reltaive path
component somewhere in the middle or end of the path, wget2 aborts
with the message Internal error: Unexpected relative path: '<path>')
Such can happen if DL_DIR includes relative path components e.g.
DL_DIR = "${TOPDIR}/../../downloads".
This patch canonicalizes DL_DIR before it is passed to wget.
Signed-off-by: Rudolf J Streif <rudolf.streif@ibeeto.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3e4208952b086adc510e78c1c5f9cf4550d79dc9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
(cherry picked from commit 47678142e26bb76d1351886060deff5e75039bc9)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/wget.py | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index 708f3a42e..3849c8a5a 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -109,7 +109,8 @@ class Wget(FetchMethod):
fetchcmd = self.basecmd
- localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile) + ".tmp"
+ dldir = os.path.realpath(d.getVar("DL_DIR"))
+ localpath = os.path.join(dldir, ud.localfile) + ".tmp"
bb.utils.mkdirhier(os.path.dirname(localpath))
fetchcmd += " -O %s" % shlex.quote(localpath)
@@ -129,9 +130,9 @@ class Wget(FetchMethod):
uri = ud.url.split(";")[0]
if os.path.exists(ud.localpath):
# file exists, but we didnt complete it.. trying again..
- fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
+ fetchcmd += " -c -P " + dldir + " '" + uri + "'"
else:
- fetchcmd += d.expand(" -P ${DL_DIR} '%s'" % uri)
+ fetchcmd += " -P " + dldir + " '" + uri + "'"
self._runwget(ud, d, fetchcmd, False)
--
2.34.1
^ permalink raw reply related [flat|nested] 6+ messages in thread