[Buildroot] [PATCH 0/2 v2] support/download: fix running on hosts with wget2 (branch yem/dl-curl)

[Buildroot] [PATCH 0/2 v2] support/download: fix running on hosts with wget2 (branch yem/dl-curl)

[Yann E. MORIN]

Hello All!

Recently, Fedora 40 was released, which uses wget2 instead of the
"original" wget (aka wget1). wget2 is almost a drop-in replacement
for wget1, except it no longer supports FTP and WARC. While WARC was
unused in Buildroot, FTP is still used by a few packges, some of
which only available via FTP.

Switch to using curl for FTP downloads.

We could have switched to curl for http/s downlads too, but it is
possible that existing packages (esp. in br2-external trees) use
wget1 options, like we used to in the now-dropped amd-catalyst
package for example. So we decided to only convert FTP downloads
over to curl, and keep the rest unchanged, to minise the impact.

Changes v1 -> v2:
  - drop --passive-ftp from BR2_WGET

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (2):
      support/download: introduce curl backend for FTP transfers
      utils/genrandconfig: do not check certificates with curl

 Config.in                     |  6 +++++-
 docs/manual/prerequisite.adoc |  1 +
 package/pkg-download.mk       |  1 +
 package/pkg-generic.mk        |  2 ++
 support/download/curl         | 45 +++++++++++++++++++++++++++++++++++++++++++
 support/download/dl-wrapper   |  1 +
 utils/genrandconfig           |  3 ++-
 7 files changed, 57 insertions(+), 2 deletions(-)
 create mode 100755 support/download/curl

--
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/2 v2] support/download: introduce curl backend for FTP transfers

[Yann E. MORIN]

Recent versions of wget, starting with wget 2.0, aka wget2 thereafter,
no longer support FTP (nor FTPS, aka FTP-over-SSL). wget2 is packaged in
Fedora 40, recently released; F40 does not even have the old wget
available in its repository anymore.

Introduce cURL as a download backend, that we use for FTP and FPTS
protocols.

Note that the -q flag does not means being quiet; it means that a curlrc
file should not be parsed. The long option is --disable, which meaning
is not much more obivous than the short -q. It also has to be the first
option on the command line.

Since we no longer use WGET to retrieve FTP-hosted files, we can drop
the --passive-ftp option, which is what was causing wget2 to abort in
error.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

---
Notes: the pre-existing shellcheck errors in the dl-wrapper were not
fixed; this is not urgent, so is left for another series.

---
Changes v1 ->v2;
  - drop --passive-ftp for BR2_WGET
---
 Config.in                     |  6 ++++-
 docs/manual/prerequisite.adoc |  1 +
 package/pkg-download.mk       |  1 +
 package/pkg-generic.mk        |  2 ++
 support/download/curl         | 45 +++++++++++++++++++++++++++++++++++
 support/download/dl-wrapper   |  1 +
 utils/genrandconfig           |  2 +-
 7 files changed, 56 insertions(+), 2 deletions(-)
 create mode 100755 support/download/curl

diff --git a/Config.in b/Config.in
index b5a94325c4..e0257ba3e8 100644
--- a/Config.in
+++ b/Config.in
@@ -103,9 +103,13 @@ menu "Build options"
 
 menu "Commands"
 
+config BR2_CURL
+	string "Curl command"
+	default "curl -q --ftp-pasv --retry 3"
+
 config BR2_WGET
 	string "Wget command"
-	default "wget --passive-ftp -nd -t 3"
+	default "wget -nd -t 3"
 
 config BR2_SVN
 	string "Subversion (svn) command"
diff --git a/docs/manual/prerequisite.adoc b/docs/manual/prerequisite.adoc
index 262a5153f5..846a7482ac 100644
--- a/docs/manual/prerequisite.adoc
+++ b/docs/manual/prerequisite.adoc
@@ -75,6 +75,7 @@ packages using any of these methods, you will need to install the
 corresponding tool on the host system:
 +
 ** +bazaar+
+** +curl+
 ** +cvs+
 ** +git+
 ** +mercurial+
diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index 4be45c9d12..455443c164 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -8,6 +8,7 @@
 ################################################################################
 
 # Download method commands
+export CURL := $(call qstrip,$(BR2_CURL))
 export WGET := $(call qstrip,$(BR2_WGET))
 export SVN := $(call qstrip,$(BR2_SVN))
 export CVS := $(call qstrip,$(BR2_CVS))
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index a2749320c3..e1c16b7343 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -1253,6 +1253,8 @@ else ifeq ($$($(2)_SITE_METHOD),hg)
 DL_TOOLS_DEPENDENCIES += hg
 else ifeq ($$($(2)_SITE_METHOD),cvs)
 DL_TOOLS_DEPENDENCIES += cvs
+else ifneq ($(filter ftp ftps,$$($(2)_SITE_METHOD)),)
+DL_TOOLS_DEPENDENCIES += curl
 endif # SITE_METHOD
 
 # cargo/go vendoring (may) need git
diff --git a/support/download/curl b/support/download/curl
new file mode 100755
index 0000000000..bea4485a6c
--- /dev/null
+++ b/support/download/curl
@@ -0,0 +1,45 @@
+#!/usr/bin/env bash
+
+# We want to catch any unexpected failure, and exit immediately
+set -e
+
+# Download helper for curl, to be called from the download wrapper script
+#
+# Options:
+#   -q          Be quiet.
+#   -o FILE     Save into file FILE.
+#   -f FILENAME The filename of the tarball to get at URL
+#   -u URL      Download file at URL.
+#
+# Environment:
+#   CURL     : the curl command to call
+
+quiet=
+while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
+    case "${OPT}" in
+    q)  quiet=-s;;
+    o)  output="${OPTARG}";;
+    f)  filename="${OPTARG}";;
+    u)  url="${OPTARG}";;
+    :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
+    \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
+    esac
+done
+
+shift $((OPTIND-1)) # Get rid of our options
+
+# Caller needs to single-quote its arguments to prevent them from
+# being expanded a second time (in case there are spaces in them)
+_curl() {
+    if [ -z "${quiet}" ]; then
+        printf '%s ' "${CURL}" "${@}"; printf '\n'
+    fi
+    _plain_curl "$@"
+}
+# Note: please keep command below aligned with what is printed above
+_plain_curl() {
+    # shellcheck disable=SC2086  # We want splitting
+    eval ${CURL} "${@}"
+}
+
+_curl ${quiet} "${@}" --output "'${output}'" "'${url}/${filename}'"
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index 35428faeef..069b2c1c21 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -91,6 +91,7 @@ main() {
         backend="${backend_urlencode%|*}"
         case "${backend}" in
             git|svn|cvs|bzr|file|scp|hg|sftp) ;;
+            ftp|ftps) backend="curl" ;;
             *) backend="wget" ;;
         esac
         uri=${uri#*+}
diff --git a/utils/genrandconfig b/utils/genrandconfig
index b838dda34d..b7d277105f 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -688,7 +688,7 @@ async def gen_config(args):
         configlines += minimalf.readlines()
 
     # Allow hosts with old certificates to download over https
-    configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n")
+    configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
 
     # Per-package folder
     if randint(0, 15) == 0:
-- 
2.45.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2 v2] utils/genrandconfig: do not check certificates with curl

[Yann E. MORIN]

genrandconfig is used in autobuilders, and some autobuilders are running
on old distributions that are lacking the most recent CAs, causing build
failures because package sources can't be retrieved.

Do for the curl backend what we already did a while back for the wget
backend, with commit 0866a280e40a (utils/genrandconfig: use
--no-check-certificate in wget by default); in curl, the equivalent
would be --insecure, and applies to the ftps transport.

The integrity of the downloads are validated against our bundled hashes
so there is no risk of corruption of the downloaded files. The only
issue would be that an MITM could inspect the transaction, the same way
as for the wget --no-check-certificate in 0866a280e40a, but this is not
considered a high-level issue (we're anyway talking FTPS here, that's a
legacy protocol that has other issues).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>

---
Note: this is totally untested, because FTPS is not widespread and no
known package was available via FTPS. This patch can probably be dropped.
---
 utils/genrandconfig | 1 +
 1 file changed, 1 insertion(+)

diff --git a/utils/genrandconfig b/utils/genrandconfig
index b7d277105f..a7b012ba9b 100755
--- a/utils/genrandconfig
+++ b/utils/genrandconfig
@@ -689,6 +689,7 @@ async def gen_config(args):
 
     # Allow hosts with old certificates to download over https
     configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
+    configlines.append("BR2_CURL=\"curl --ftp-pasv --retry 3 --insecure\"\n")
 
     # Per-package folder
     if randint(0, 15) == 0:
-- 
2.45.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2 v2] support/download: introduce curl backend for FTP transfers

[Thomas Petazzoni via buildroot]

Hello Yann,

On Sun,  2 Jun 2024 20:23:07 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> diff --git a/Config.in b/Config.in
> index b5a94325c4..e0257ba3e8 100644
> --- a/Config.in
> +++ b/Config.in
> @@ -103,9 +103,13 @@ menu "Build options"
>  
>  menu "Commands"
>  
> +config BR2_CURL
> +	string "Curl command"
> +	default "curl -q --ftp-pasv --retry 3"
> +
>  config BR2_WGET
>  	string "Wget command"
> -	default "wget --passive-ftp -nd -t 3"
> +	default "wget -nd -t 3"

This change was not really related to introducing curl support... and
in fact Peter already applied 1a61c1d9b1919120883b689c3767925bd91d3028
doing such a change.

> diff --git a/utils/genrandconfig b/utils/genrandconfig
> index b838dda34d..b7d277105f 100755
> --- a/utils/genrandconfig
> +++ b/utils/genrandconfig
> @@ -688,7 +688,7 @@ async def gen_config(args):
>          configlines += minimalf.readlines()
>  
>      # Allow hosts with old certificates to download over https
> -    configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"\n")
> +    configlines.append("BR2_WGET=\"wget -nd -t 3 --no-check-certificate\"\n")
>  
>      # Per-package folder
>      if randint(0, 15) == 0:

Also, this was not directly related to the introduction of curl
support, so I split this out into a separate commit.

Applied, thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2 v2] utils/genrandconfig: do not check certificates with curl

[Thomas Petazzoni via buildroot]

On Sun,  2 Jun 2024 20:23:08 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> genrandconfig is used in autobuilders, and some autobuilders are running
> on old distributions that are lacking the most recent CAs, causing build
> failures because package sources can't be retrieved.
> 
> Do for the curl backend what we already did a while back for the wget
> backend, with commit 0866a280e40a (utils/genrandconfig: use
> --no-check-certificate in wget by default); in curl, the equivalent
> would be --insecure, and applies to the ftps transport.
> 
> The integrity of the downloads are validated against our bundled hashes
> so there is no risk of corruption of the downloaded files. The only
> issue would be that an MITM could inspect the transaction, the same way
> as for the wget --no-check-certificate in 0866a280e40a, but this is not
> considered a high-level issue (we're anyway talking FTPS here, that's a
> legacy protocol that has other issues).
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> 
> ---
> Note: this is totally untested, because FTPS is not widespread and no
> known package was available via FTPS. This patch can probably be dropped.
> ---
>  utils/genrandconfig | 1 +
>  1 file changed, 1 insertion(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot