* [Buildroot] [PATCH 1/2 v2] support/download: introduce curl backend for FTP transfers
2024-06-02 18:23 [Buildroot] [PATCH 0/2 v2] support/download: fix running on hosts with wget2 (branch yem/dl-curl) Yann E. MORIN
@ 2024-06-02 18:23 ` Yann E. MORIN
2024-07-12 12:57 ` Thomas Petazzoni via buildroot
2024-06-02 18:23 ` [Buildroot] [PATCH 2/2 v2] utils/genrandconfig: do not check certificates with curl Yann E. MORIN
1 sibling, 1 reply; 5+ messages in thread
From: Yann E. MORIN @ 2024-06-02 18:23 UTC (permalink / raw)
To: buildroot; +Cc: Yann E. MORIN
Recent versions of wget, starting with wget 2.0, aka wget2 thereafter,
no longer support FTP (nor FTPS, aka FTP-over-SSL). wget2 is packaged in
Fedora 40, recently released; F40 does not even have the old wget
available in its repository anymore.
Introduce cURL as a download backend, that we use for FTP and FPTS
protocols.
Note that the -q flag does not means being quiet; it means that a curlrc
file should not be parsed. The long option is --disable, which meaning
is not much more obivous than the short -q. It also has to be the first
option on the command line.
Since we no longer use WGET to retrieve FTP-hosted files, we can drop
the --passive-ftp option, which is what was causing wget2 to abort in
error.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
Notes: the pre-existing shellcheck errors in the dl-wrapper were not
fixed; this is not urgent, so is left for another series.
---
Changes v1 ->v2;
- drop --passive-ftp for BR2_WGET
---
Config.in | 6 ++++-
docs/manual/prerequisite.adoc | 1 +
package/pkg-download.mk | 1 +
package/pkg-generic.mk | 2 ++
support/download/curl | 45 +++++++++++++++++++++++++++++++++++
support/download/dl-wrapper | 1 +
utils/genrandconfig | 2 +-
7 files changed, 56 insertions(+), 2 deletions(-)
create mode 100755 support/download/curl
diff --git a/Config.in b/Config.in
index b5a94325c4..e0257ba3e8 100644
--- a/Config.in
+++ b/Config.in
@@ -103,9 +103,13 @@ menu "Build options"
menu "Commands"
+config BR2_CURL
+ string "Curl command"
+ default "curl -q --ftp-pasv --retry 3"
+
config BR2_WGET
string "Wget command"
- default "wget --passive-ftp -nd -t 3"
+ default "wget -nd -t 3"
config BR2_SVN
string "Subversion (svn) command"
diff --git a/docs/manual/prerequisite.adoc b/docs/manual/prerequisite.adoc
index 262a5153f5..846a7482ac 100644
--- a/docs/manual/prerequisite.adoc
+++ b/docs/manual/prerequisite.adoc
@@ -75,6 +75,7 @@ packages using any of these methods, you will need to install the
corresponding tool on the host system:
+
** +bazaar+
+** +curl+
** +cvs+
** +git+
** +mercurial+
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 4be45c9d12..455443c164 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -8,6 +8,7 @@
################################################################################
# Download method commands
+export CURL := $(call qstrip,$(BR2_CURL))
export WGET := $(call qstrip,$(BR2_WGET))
export SVN := $(call qstrip,$(BR2_SVN))
export CVS := $(call qstrip,$(BR2_CVS))
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index a2749320c3..e1c16b7343 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -1253,6 +1253,8 @@ else ifeq ($$($(2)_SITE_METHOD),hg)
DL_TOOLS_DEPENDENCIES += hg
else ifeq ($$($(2)_SITE_METHOD),cvs)
DL_TOOLS_DEPENDENCIES += cvs
+else ifneq ($(filter ftp ftps,$$($(2)_SITE_METHOD)),)
+DL_TOOLS_DEPENDENCIES += curl
endif # SITE_METHOD
# cargo/go vendoring (may) need git
diff --git a/support/download/curl b/support/download/curl
new file mode 100755
index 0000000000..bea4485a6c
--- /dev/null
+++ b/support/download/curl
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+# We want to catch any unexpected failure, and exit immediately
+set -e
+
+# Download helper for curl, to be called from the download wrapper script
+#
+# Options:
+# -q Be quiet.
+# -o FILE Save into file FILE.
+# -f FILENAME The filename of the tarball to get at URL
+# -u URL Download file at URL.
+#
+# Environment:
+# CURL : the curl command to call
+
+quiet=
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
+ case "${OPT}" in
+ q) quiet=-s;;
+ o) output="${OPTARG}";;
+ f) filename="${OPTARG}";;
+ u) url="${OPTARG}";;
+ :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
+ \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+ esac
+done
+
+shift $((OPTIND-1)) # Get rid of our options
+
+# Caller needs to single-quote its arguments to prevent them from
+# being expanded a second time (in case there are spaces in them)
+_curl() {
+ if [ -z "${quiet}" ]; then
+ printf '%s ' "${CURL}" "${@}"; printf '\n'
+ fi
+ _plain_curl "$@"
+}
+# Note: please keep command below aligned with what is printed above
+_plain_curl() {
+ # shellcheck disable=SC2086 # We want splitting
+ eval ${CURL} "${@}"
+}
+
+_curl ${quiet} "${@}" --output "'${output}'" "'${url}/${filename}'"
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 35428faeef..069b2c1c21 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -91,6 +91,7 @@ main() {
backend="${backend_urlencode%|*}"
case "${backend}" in
git|svn|cvs|bzr|file|scp|hg|sftp) ;;
+ ftp|ftps) backend="curl" ;;
*) backend="wget" ;;
esac
uri=${uri#*+}
diff --git a/utils/genrandconfig b/utils/genrandconfig
index b838dda34d..b7d277105f 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -688,7 +688,7 @@ async def gen_config(args):
configlines += minimalf.readlines()
# Allow hosts with old certificates to download over https
- configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n")
+ configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
# Per-package folder
if randint(0, 15) == 0:
--
2.45.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 5+ messages in thread* [Buildroot] [PATCH 2/2 v2] utils/genrandconfig: do not check certificates with curl
2024-06-02 18:23 [Buildroot] [PATCH 0/2 v2] support/download: fix running on hosts with wget2 (branch yem/dl-curl) Yann E. MORIN
2024-06-02 18:23 ` [Buildroot] [PATCH 1/2 v2] support/download: introduce curl backend for FTP transfers Yann E. MORIN
@ 2024-06-02 18:23 ` Yann E. MORIN
2024-07-12 12:57 ` Thomas Petazzoni via buildroot
1 sibling, 1 reply; 5+ messages in thread
From: Yann E. MORIN @ 2024-06-02 18:23 UTC (permalink / raw)
To: buildroot; +Cc: Yann E. MORIN
genrandconfig is used in autobuilders, and some autobuilders are running
on old distributions that are lacking the most recent CAs, causing build
failures because package sources can't be retrieved.
Do for the curl backend what we already did a while back for the wget
backend, with commit 0866a280e40a (utils/genrandconfig: use
--no-check-certificate in wget by default); in curl, the equivalent
would be --insecure, and applies to the ftps transport.
The integrity of the downloads are validated against our bundled hashes
so there is no risk of corruption of the downloaded files. The only
issue would be that an MITM could inspect the transaction, the same way
as for the wget --no-check-certificate in 0866a280e40a, but this is not
considered a high-level issue (we're anyway talking FTPS here, that's a
legacy protocol that has other issues).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
Note: this is totally untested, because FTPS is not widespread and no
known package was available via FTPS. This patch can probably be dropped.
---
utils/genrandconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/utils/genrandconfig b/utils/genrandconfig
index b7d277105f..a7b012ba9b 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -689,6 +689,7 @@ async def gen_config(args):
# Allow hosts with old certificates to download over https
configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
+ configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n")
# Per-package folder
if randint(0, 15) == 0:
--
2.45.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 5+ messages in thread