All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O
@ 2025-01-11 16:28 Romain Naour via buildroot
  2025-01-11 17:18 ` David Laight
  0 siblings, 1 reply; 3+ messages in thread
From: Romain Naour via buildroot @ 2025-01-11 16:28 UTC (permalink / raw)
  To: buildroot; +Cc: Romain Naour, Julien Olivain

As reported by Julien Olivain, in some cases the "wget -O output ..."
does not get honored when GNU Wget2 is used as drop-in replacement for
GNU Wget.

For example, qt6* packages archives are failing (due to multipart
downloads: HTTP response 206):

  [Files: 1  Bytes: 46.00M [388.39KB/s] Redirects: 1  Todo: 0  Errors: 184

  sha256sum: [...]/build/.qtbase-everywhere-src-6.8.1.tar.xz.bFXkzc/output: No such file or directory
  ERROR: while checking hashes from package/qt6/qt6base/qt6base.hash
  ERROR: qtbase-everywhere-src-6.8.1.tar.xz has wrong sha256 hash:
  ERROR: expected: 40b14562ef3bd779bc0e0418ea2ae08fa28235f8ea6e8c0cb3bce1d6ad58dcaf
  ERROR: got     :
  ERROR: Incomplete download, or man-in-the-middle (MITM) attack

Fedora 40 is one of the first Linux distribution providing GNU Wget2 as
wget [1]. Some incompatibility issues with Wget1.x are discovered with
early Wget2 versions.

This is the case here with Wget2 2.2.0 and Metalink handling that
ignore -O option [2][3].

The issue was not noticed until now since the download infrastructure
uses a mirror (s.b.o) as fallback.

Since wget -O is part of our download infrastructure since 2012.05 [4],
ensure wget2 is at least 2.2.1 or any newer version that contains the
-O fix [5].

[1] https://fedoraproject.org/wiki/Changes/Wget2asWget
[2] https://gitlab.com/gnuwget/wget2/-/issues/685
[3] https://gitlab.com/gnuwget/wget2/-/blob/0651b0a447e4d6dd3f100800d0d93db90f2bd2fb/docs/wget2.md#-o---output-documentfile
[4] https://gitlab.com/buildroot.org/buildroot/-/commit/cf2486bf317e4bbf88c801fb96183ba62be78cc8
[5] https://gitlab.com/gnuwget/wget2/-/commit/15c06c2f2a5f5eaa962893beec97a5211743ec05

Reported-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
---
 support/dependencies/dependencies.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
index fb0defd0c9..e49831a3fd 100755
--- a/support/dependencies/dependencies.sh
+++ b/support/dependencies/dependencies.sh
@@ -184,6 +184,21 @@ for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
 			exit 1
 		fi
 	fi
+
+	if test $prog = "wget" ; then
+		WGET_VERSION="$(wget --version | head -n1 | cut -d ' ' -f3)"
+		WGET_MAJOR="$(echo "${WGET_VERSION}" | cut -d . -f 1)"
+		WGET_MINOR="$(echo "${WGET_VERSION}" | cut -d . -f 2)"
+		WGET_PATCH="$(echo "${WGET_VERSION}" | cut -d . -f 3)"
+
+		# Wget2 <= 2.2.0 issue with -O : https://gitlab.com/gnuwget/wget2/-/issues/685
+		if [ "${WGET_MAJOR}" -eq 2 ] && \
+			! [ "${WGET_MAJOR}" -eq 2 -a "${WGET_MINOR}" -ge 2 -a "${WGET_PATCH}" -gt 0 ] ; then
+			echo
+			echo "You have GNU Wget2 '${WGET_VERSION}' installed. GNU Wget2 > 2.2.0 is is required"
+			exit 1;
+		fi
+	fi
 done
 
 if test "${missing_progs}" = "yes" ; then
-- 
2.47.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply related	[flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O
  2025-01-11 16:28 [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O Romain Naour via buildroot
@ 2025-01-11 17:18 ` David Laight
  2025-01-13 13:18   ` Romain Naour via buildroot
  0 siblings, 1 reply; 3+ messages in thread
From: David Laight @ 2025-01-11 17:18 UTC (permalink / raw)
  To: Romain Naour via buildroot; +Cc: Romain Naour, Julien Olivain

On Sat, 11 Jan 2025 17:28:41 +0100
Romain Naour via buildroot <buildroot@buildroot.org> wrote:

> As reported by Julien Olivain, in some cases the "wget -O output ..."
> does not get honored when GNU Wget2 is used as drop-in replacement for
> GNU Wget.
> 
> For example, qt6* packages archives are failing (due to multipart
> downloads: HTTP response 206):
> 
>   [Files: 1  Bytes: 46.00M [388.39KB/s] Redirects: 1  Todo: 0  Errors: 184
> 
>   sha256sum: [...]/build/.qtbase-everywhere-src-6.8.1.tar.xz.bFXkzc/output: No such file or directory
>   ERROR: while checking hashes from package/qt6/qt6base/qt6base.hash
>   ERROR: qtbase-everywhere-src-6.8.1.tar.xz has wrong sha256 hash:
>   ERROR: expected: 40b14562ef3bd779bc0e0418ea2ae08fa28235f8ea6e8c0cb3bce1d6ad58dcaf
>   ERROR: got     :
>   ERROR: Incomplete download, or man-in-the-middle (MITM) attack
> 
> Fedora 40 is one of the first Linux distribution providing GNU Wget2 as
> wget [1]. Some incompatibility issues with Wget1.x are discovered with
> early Wget2 versions.
> 
> This is the case here with Wget2 2.2.0 and Metalink handling that
> ignore -O option [2][3].
> 
> The issue was not noticed until now since the download infrastructure
> uses a mirror (s.b.o) as fallback.
> 
> Since wget -O is part of our download infrastructure since 2012.05 [4],
> ensure wget2 is at least 2.2.1 or any newer version that contains the
> -O fix [5].
> 
> [1] https://fedoraproject.org/wiki/Changes/Wget2asWget
> [2] https://gitlab.com/gnuwget/wget2/-/issues/685
> [3] https://gitlab.com/gnuwget/wget2/-/blob/0651b0a447e4d6dd3f100800d0d93db90f2bd2fb/docs/wget2.md#-o---output-documentfile
> [4] https://gitlab.com/buildroot.org/buildroot/-/commit/cf2486bf317e4bbf88c801fb96183ba62be78cc8
> [5] https://gitlab.com/gnuwget/wget2/-/commit/15c06c2f2a5f5eaa962893beec97a5211743ec05
> 
> Reported-by: Julien Olivain <ju.o@free.fr>
> Signed-off-by: Romain Naour <romain.naour@smile.fr>
> ---
>  support/dependencies/dependencies.sh | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
> index fb0defd0c9..e49831a3fd 100755
> --- a/support/dependencies/dependencies.sh
> +++ b/support/dependencies/dependencies.sh
> @@ -184,6 +184,21 @@ for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
>  			exit 1
>  		fi
>  	fi
> +
> +	if test $prog = "wget" ; then
> +		WGET_VERSION="$(wget --version | head -n1 | cut -d ' ' -f3)"
> +		WGET_MAJOR="$(echo "${WGET_VERSION}" | cut -d . -f 1)"
> +		WGET_MINOR="$(echo "${WGET_VERSION}" | cut -d . -f 2)"
> +		WGET_PATCH="$(echo "${WGET_VERSION}" | cut -d . -f 3)"

How about:
		IFS=".$IFS"
		set $(wget --version)
		WGET_MAJOR=$3
		WGET_MINOR=$4
		WGET_PATCH=$5
		IFS="${IFS#.}"

Saves all those fork+exec.

> +
> +		# Wget2 <= 2.2.0 issue with -O : https://gitlab.com/gnuwget/wget2/-/issues/685
> +		if [ "${WGET_MAJOR}" -eq 2 ] && \
> +			! [ "${WGET_MAJOR}" -eq 2 -a "${WGET_MINOR}" -ge 2 -a "${WGET_PATCH}" -gt 0 ] ; then

Doesn't that also fail for 2.3.0 ?
I think this is ok:
		if [ $WGET_MAJOR = 2 && $WGET_MINOR$WGET_PATCH -le 20 ] ; then
any two digit minor/patch will fail the test.
(And there can't be an separators - so quotes aren't needed.)

	David


> +			echo
> +			echo "You have GNU Wget2 '${WGET_VERSION}' installed. GNU Wget2 > 2.2.0 is is required"
> +			exit 1;
> +		fi
> +	fi
>  done
>  
>  if test "${missing_progs}" = "yes" ; then

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O
  2025-01-11 17:18 ` David Laight
@ 2025-01-13 13:18   ` Romain Naour via buildroot
  0 siblings, 0 replies; 3+ messages in thread
From: Romain Naour via buildroot @ 2025-01-13 13:18 UTC (permalink / raw)
  To: David Laight, Romain Naour via buildroot; +Cc: Julien Olivain

Hello David, All,

Le 11/01/2025 à 18:18, David Laight a écrit :
> On Sat, 11 Jan 2025 17:28:41 +0100
> Romain Naour via buildroot <buildroot@buildroot.org> wrote:
> 
>> As reported by Julien Olivain, in some cases the "wget -O output ..."
>> does not get honored when GNU Wget2 is used as drop-in replacement for
>> GNU Wget.
>>
>> For example, qt6* packages archives are failing (due to multipart
>> downloads: HTTP response 206):
>>
>>   [Files: 1  Bytes: 46.00M [388.39KB/s] Redirects: 1  Todo: 0  Errors: 184
>>
>>   sha256sum: [...]/build/.qtbase-everywhere-src-6.8.1.tar.xz.bFXkzc/output: No such file or directory
>>   ERROR: while checking hashes from package/qt6/qt6base/qt6base.hash
>>   ERROR: qtbase-everywhere-src-6.8.1.tar.xz has wrong sha256 hash:
>>   ERROR: expected: 40b14562ef3bd779bc0e0418ea2ae08fa28235f8ea6e8c0cb3bce1d6ad58dcaf
>>   ERROR: got     :
>>   ERROR: Incomplete download, or man-in-the-middle (MITM) attack
>>
>> Fedora 40 is one of the first Linux distribution providing GNU Wget2 as
>> wget [1]. Some incompatibility issues with Wget1.x are discovered with
>> early Wget2 versions.
>>
>> This is the case here with Wget2 2.2.0 and Metalink handling that
>> ignore -O option [2][3].
>>
>> The issue was not noticed until now since the download infrastructure
>> uses a mirror (s.b.o) as fallback.
>>
>> Since wget -O is part of our download infrastructure since 2012.05 [4],
>> ensure wget2 is at least 2.2.1 or any newer version that contains the
>> -O fix [5].
>>
>> [1] https://fedoraproject.org/wiki/Changes/Wget2asWget
>> [2] https://gitlab.com/gnuwget/wget2/-/issues/685
>> [3] https://gitlab.com/gnuwget/wget2/-/blob/0651b0a447e4d6dd3f100800d0d93db90f2bd2fb/docs/wget2.md#-o---output-documentfile
>> [4] https://gitlab.com/buildroot.org/buildroot/-/commit/cf2486bf317e4bbf88c801fb96183ba62be78cc8
>> [5] https://gitlab.com/gnuwget/wget2/-/commit/15c06c2f2a5f5eaa962893beec97a5211743ec05
>>
>> Reported-by: Julien Olivain <ju.o@free.fr>
>> Signed-off-by: Romain Naour <romain.naour@smile.fr>
>> ---
>>  support/dependencies/dependencies.sh | 15 +++++++++++++++
>>  1 file changed, 15 insertions(+)
>>
>> diff --git a/support/dependencies/dependencies.sh b/support/dependencies/dependencies.sh
>> index fb0defd0c9..e49831a3fd 100755
>> --- a/support/dependencies/dependencies.sh
>> +++ b/support/dependencies/dependencies.sh
>> @@ -184,6 +184,21 @@ for prog in perl tar wget cpio unzip rsync bc cmp find xargs ${DL_TOOLS} ; do
>>  			exit 1
>>  		fi
>>  	fi
>> +
>> +	if test $prog = "wget" ; then
>> +		WGET_VERSION="$(wget --version | head -n1 | cut -d ' ' -f3)"
>> +		WGET_MAJOR="$(echo "${WGET_VERSION}" | cut -d . -f 1)"
>> +		WGET_MINOR="$(echo "${WGET_VERSION}" | cut -d . -f 2)"
>> +		WGET_PATCH="$(echo "${WGET_VERSION}" | cut -d . -f 3)"
> 
> How about:
> 		IFS=".$IFS"
> 		set $(wget --version)
> 		WGET_MAJOR=$3
> 		WGET_MINOR=$4
> 		WGET_PATCH=$5
> 		IFS="${IFS#.}"
> 
> Saves all those fork+exec.

This script may also requires some rework to avoid such fork+exec.
I wanted to keep similar "style".

> 
>> +
>> +		# Wget2 <= 2.2.0 issue with -O : https://gitlab.com/gnuwget/wget2/-/issues/685
>> +		if [ "${WGET_MAJOR}" -eq 2 ] && \
>> +			! [ "${WGET_MAJOR}" -eq 2 -a "${WGET_MINOR}" -ge 2 -a "${WGET_PATCH}" -gt 0 ] ; then
> 
> Doesn't that also fail for 2.3.0 ?
> I think this is ok:
> 		if [ $WGET_MAJOR = 2 && $WGET_MINOR$WGET_PATCH -le 20 ] ; then
> any two digit minor/patch will fail the test.
> (And there can't be an separators - so quotes aren't needed.)

Thanks for spotting this mistake!

I was wondering if it's allowed by shellcheck but I noticed this script (among
others) is not checked by shellcheck.

.checkpackageignore:
support/dependencies/dependencies.sh Shellcheck

At least it would requires a "shellcheck disable=SC2086".

Thoughts?

Best regards,
Romain

> 
> 	David
> 
> 
>> +			echo
>> +			echo "You have GNU Wget2 '${WGET_VERSION}' installed. GNU Wget2 > 2.2.0 is is required"
>> +			exit 1;
>> +		fi
>> +	fi
>>  done
>>  
>>  if test "${missing_progs}" = "yes" ; then
> 

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

^ permalink raw reply	[flat|nested] 3+ messages in thread
end of thread, other threads:[~2025-01-13 13:18 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-01-11 16:28 [Buildroot] [PATCH] support/dependencies/dependencies.sh: ensure wget2 is > 2.2.0 for metalink support with -O Romain Naour via buildroot
2025-01-11 17:18 ` David Laight
2025-01-13 13:18   ` Romain Naour via buildroot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.