[Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O

[Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O

[Romain Naour via buildroot]

As reported by Julien Olivain, in some cases the "wget -O output ..."
does not get honored when GNU Wget2 is used as drop-in replacement for
GNU Wget.

For example, qt6* packages archives are failing (due to multipart
downloads: HTTP response 206):

  [Files: 1  Bytes: 46.00M [388.39KB/s] Redirects: 1  Todo: 0  Errors: 184

  sha256sum: [...]/build/.qtbase-everywhere-src-6.8.1.tar.xz.bFXkzc/output: No such file or directory
  ERROR: while checking hashes from package/qt6/qt6base/qt6base.hash
  ERROR: qtbase-everywhere-src-6.8.1.tar.xz has wrong sha256 hash:
  ERROR: expected: 40b14562ef3bd779bc0e0418ea2ae08fa28235f8ea6e8c0cb3bce1d6ad58dcaf
  ERROR: got     :
  ERROR: Incomplete download, or man-in-the-middle (MITM) attack

Fedora 40 is one of the first Linux distribution providing GNU Wget2 as
wget [1]. Some incompatibility issues with Wget1.x are discovered with
early Wget2 versions.

This is the case here with Wget2 2.2.0 and Metalink handling that
ignore -O option [2][3].

The issue was not noticed until now since the download infrastructure
uses a mirror (s.b.o) as fallback.

Since wget -O is part of our download infrastructure since 2012.05 [4],
ensure wget2 is at least 2.2.1 or any newer version that contains the
-O fix [5].

[1] https://fedoraproject.org/wiki/Changes/Wget2asWget
[2] https://gitlab.com/gnuwget/wget2/-/issues/685
[3] https://gitlab.com/gnuwget/wget2/-/blob/0651b0a447e4d6dd3f100800d0d93db90f2bd2fb/docs/wget2.md#-o---output-documentfile
[4] https://gitlab.com/buildroot.org/buildroot/-/commit/cf2486bf317e4bbf88c801fb96183ba62be78cc8
[5] https://gitlab.com/gnuwget/wget2/-/commit/15c06c2f2a5f5eaa962893beec97a5211743ec05

Reported-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
---
 support/dependencies/dependencies.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
index fb0defd0c9..e49831a3fd 100755
--- a/support/dependencies/dependencies.sh
+++ b/support/dependencies/dependencies.sh
@@ -184,6 +184,21 @@ for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
 			exit 1
 		fi
 	fi
+
+	if test $prog = "wget" ; then
+		WGET_VERSION="$(wget --version | head -n1 | cut -d ' ' -f3)"
+		WGET_MAJOR="$(echo "${WGET_VERSION}" | cut -d . -f 1)"
+		WGET_MINOR="$(echo "${WGET_VERSION}" | cut -d . -f 2)"
+		WGET_PATCH="$(echo "${WGET_VERSION}" | cut -d . -f 3)"
+
+		# Wget2 <= 2.2.0 issue with -O : https://gitlab.com/gnuwget/wget2/-/issues/685
+		if [ "${WGET_MAJOR}" -eq 2 ] && \
+			! [ "${WGET_MAJOR}" -eq 2 -a "${WGET_MINOR}" -ge 2 -a "${WGET_PATCH}" -gt 0 ] ; then
+			echo
+			echo "You have GNU Wget2 '${WGET_VERSION}' installed. GNU Wget2 > 2.2.0 is is required"
+			exit 1;
+		fi
+	fi
 done
 
 if test "${missing_progs}" = "yes" ; then
-- 
2.47.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O

[David Laight]

On Sat, 11 Jan 2025 17:28:41 +0100
Romain Naour via buildroot <buildroot@buildroot.org> wrote:

> As reported by Julien Olivain, in some cases the "wget -O output ..."
> does not get honored when GNU Wget2 is used as drop-in replacement for
> GNU Wget.
> 
> For example, qt6* packages archives are failing (due to multipart
> downloads: HTTP response 206):
> 
>   [Files: 1  Bytes: 46.00M [388.39KB/s] Redirects: 1  Todo: 0  Errors: 184
> 
>   sha256sum: [...]/build/.qtbase-everywhere-src-6.8.1.tar.xz.bFXkzc/output: No such file or directory
>   ERROR: while checking hashes from package/qt6/qt6base/qt6base.hash
>   ERROR: qtbase-everywhere-src-6.8.1.tar.xz has wrong sha256 hash:
>   ERROR: expected: 40b14562ef3bd779bc0e0418ea2ae08fa28235f8ea6e8c0cb3bce1d6ad58dcaf
>   ERROR: got     :
>   ERROR: Incomplete download, or man-in-the-middle (MITM) attack
> 
> Fedora 40 is one of the first Linux distribution providing GNU Wget2 as
> wget [1]. Some incompatibility issues with Wget1.x are discovered with
> early Wget2 versions.
> 
> This is the case here with Wget2 2.2.0 and Metalink handling that
> ignore -O option [2][3].
> 
> The issue was not noticed until now since the download infrastructure
> uses a mirror (s.b.o) as fallback.
> 
> Since wget -O is part of our download infrastructure since 2012.05 [4],
> ensure wget2 is at least 2.2.1 or any newer version that contains the
> -O fix [5].
> 
> [1] https://fedoraproject.org/wiki/Changes/Wget2asWget
> [2] https://gitlab.com/gnuwget/wget2/-/issues/685
> [3] https://gitlab.com/gnuwget/wget2/-/blob/0651b0a447e4d6dd3f100800d0d93db90f2bd2fb/docs/wget2.md#-o---output-documentfile
> [4] https://gitlab.com/buildroot.org/buildroot/-/commit/cf2486bf317e4bbf88c801fb96183ba62be78cc8
> [5] https://gitlab.com/gnuwget/wget2/-/commit/15c06c2f2a5f5eaa962893beec97a5211743ec05
> 
> Reported-by: Julien Olivain <ju.o@free.fr>
> Signed-off-by: Romain Naour <romain.naour@smile.fr>
> ---
>  support/dependencies/dependencies.sh | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
> index fb0defd0c9..e49831a3fd 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -184,6 +184,21 @@ for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
>  			exit 1
>  		fi
>  	fi
> +
> +	if test $prog = "wget" ; then
> +		WGET_VERSION="$(wget --version | head -n1 | cut -d ' ' -f3)"
> +		WGET_MAJOR="$(echo "${WGET_VERSION}" | cut -d . -f 1)"
> +		WGET_MINOR="$(echo "${WGET_VERSION}" | cut -d . -f 2)"
> +		WGET_PATCH="$(echo "${WGET_VERSION}" | cut -d . -f 3)"

How about:
		IFS=".$IFS"
		set $(wget --version)
		WGET_MAJOR=$3
		WGET_MINOR=$4
		WGET_PATCH=$5
		IFS="${IFS#.}"

Saves all those fork+exec.

> +
> +		# Wget2 <= 2.2.0 issue with -O : https://gitlab.com/gnuwget/wget2/-/issues/685
> +		if [ "${WGET_MAJOR}" -eq 2 ] && \
> +			! [ "${WGET_MAJOR}" -eq 2 -a "${WGET_MINOR}" -ge 2 -a "${WGET_PATCH}" -gt 0 ] ; then

Doesn't that also fail for 2.3.0 ?
I think this is ok:
		if [ $WGET_MAJOR = 2 && $WGET_MINOR$WGET_PATCH -le 20 ] ; then
any two digit minor/patch will fail the test.
(And there can't be an separators - so quotes aren't needed.)

	David


> +			echo
> +			echo "You have GNU Wget2 '${WGET_VERSION}' installed. GNU Wget2 > 2.2.0 is is required"
> +			exit 1;
> +		fi
> +	fi
>  done
>  
>  if test "${missing_progs}" = "yes" ; then

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O

[Romain Naour via buildroot]

Hello David, All,

Le 11/01/2025 à 18:18, David Laight a écrit :
> On Sat, 11 Jan 2025 17:28:41 +0100
> Romain Naour via buildroot <buildroot@buildroot.org> wrote:
> 
>> As reported by Julien Olivain, in some cases the "wget -O output ..."
>> does not get honored when GNU Wget2 is used as drop-in replacement for
>> GNU Wget.
>>
>> For example, qt6* packages archives are failing (due to multipart
>> downloads: HTTP response 206):
>>
>>   [Files: 1  Bytes: 46.00M [388.39KB/s] Redirects: 1  Todo: 0  Errors: 184
>>
>>   sha256sum: [...]/build/.qtbase-everywhere-src-6.8.1.tar.xz.bFXkzc/output: No such file or directory
>>   ERROR: while checking hashes from package/qt6/qt6base/qt6base.hash
>>   ERROR: qtbase-everywhere-src-6.8.1.tar.xz has wrong sha256 hash:
>>   ERROR: expected: 40b14562ef3bd779bc0e0418ea2ae08fa28235f8ea6e8c0cb3bce1d6ad58dcaf
>>   ERROR: got     :
>>   ERROR: Incomplete download, or man-in-the-middle (MITM) attack
>>
>> Fedora 40 is one of the first Linux distribution providing GNU Wget2 as
>> wget [1]. Some incompatibility issues with Wget1.x are discovered with
>> early Wget2 versions.
>>
>> This is the case here with Wget2 2.2.0 and Metalink handling that
>> ignore -O option [2][3].
>>
>> The issue was not noticed until now since the download infrastructure
>> uses a mirror (s.b.o) as fallback.
>>
>> Since wget -O is part of our download infrastructure since 2012.05 [4],
>> ensure wget2 is at least 2.2.1 or any newer version that contains the
>> -O fix [5].
>>
>> [1] https://fedoraproject.org/wiki/Changes/Wget2asWget
>> [2] https://gitlab.com/gnuwget/wget2/-/issues/685
>> [3] https://gitlab.com/gnuwget/wget2/-/blob/0651b0a447e4d6dd3f100800d0d93db90f2bd2fb/docs/wget2.md#-o---output-documentfile
>> [4] https://gitlab.com/buildroot.org/buildroot/-/commit/cf2486bf317e4bbf88c801fb96183ba62be78cc8
>> [5] https://gitlab.com/gnuwget/wget2/-/commit/15c06c2f2a5f5eaa962893beec97a5211743ec05
>>
>> Reported-by: Julien Olivain <ju.o@free.fr>
>> Signed-off-by: Romain Naour <romain.naour@smile.fr>
>> ---
>>  support/dependencies/dependencies.sh | 15 +++++++++++++++
>>  1 file changed, 15 insertions(+)
>>
>> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
>> index fb0defd0c9..e49831a3fd 100755
>> --- a/support/dependencies/dependencies.sh
>> +++ b/support/dependencies/dependencies.sh
>> @@ -184,6 +184,21 @@ for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
>>  			exit 1
>>  		fi
>>  	fi
>> +
>> +	if test $prog = "wget" ; then
>> +		WGET_VERSION="$(wget --version | head -n1 | cut -d ' ' -f3)"
>> +		WGET_MAJOR="$(echo "${WGET_VERSION}" | cut -d . -f 1)"
>> +		WGET_MINOR="$(echo "${WGET_VERSION}" | cut -d . -f 2)"
>> +		WGET_PATCH="$(echo "${WGET_VERSION}" | cut -d . -f 3)"
> 
> How about:
> 		IFS=".$IFS"
> 		set $(wget --version)
> 		WGET_MAJOR=$3
> 		WGET_MINOR=$4
> 		WGET_PATCH=$5
> 		IFS="${IFS#.}"
> 
> Saves all those fork+exec.

This script may also requires some rework to avoid such fork+exec.
I wanted to keep similar "style".

> 
>> +
>> +		# Wget2 <= 2.2.0 issue with -O : https://gitlab.com/gnuwget/wget2/-/issues/685
>> +		if [ "${WGET_MAJOR}" -eq 2 ] && \
>> +			! [ "${WGET_MAJOR}" -eq 2 -a "${WGET_MINOR}" -ge 2 -a "${WGET_PATCH}" -gt 0 ] ; then
> 
> Doesn't that also fail for 2.3.0 ?
> I think this is ok:
> 		if [ $WGET_MAJOR = 2 && $WGET_MINOR$WGET_PATCH -le 20 ] ; then
> any two digit minor/patch will fail the test.
> (And there can't be an separators - so quotes aren't needed.)

Thanks for spotting this mistake!

I was wondering if it's allowed by shellcheck but I noticed this script (among
others) is not checked by shellcheck.

.checkpackageignore:
support/dependencies/dependencies.sh Shellcheck

At least it would requires a "shellcheck disable=SC2086".

Thoughts?

Best regards,
Romain

> 
> 	David
> 
> 
>> +			echo
>> +			echo "You have GNU Wget2 '${WGET_VERSION}' installed. GNU Wget2 > 2.2.0 is is required"
>> +			exit 1;
>> +		fi
>> +	fi
>>  done
>>  
>>  if test "${missing_progs}" = "yes" ; then
> 

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot