[PATCH] fetch2/wget: include paramters in URL

[PATCH] fetch2/wget: include paramters in URL

[Patrick Williams]

The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
 lib/bb/fetch2/wget.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..d41d7408 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -99,7 +99,13 @@ class Wget(FetchMethod):
         if ud.user and ud.pswd:
             fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
 
-        uri = ud.url.split(";")[0]
+        # Join the URL components together.
+        urls = ud.url.split(";")
+        if len(urls) == 1:
+            uri = urls[0]
+        else:
+            uri = urls[0] + "?" + "&".join(urls[1:])
+
         if os.path.exists(ud.localpath):
             # file exists, but we didnt complete it.. trying again..
             fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
-- 
2.26.2

Re: [PATCH] fetch2/wget: include paramters in URL

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1506 bytes --]

On Tue, Dec 29, 2020 at 09:13:16AM -0600, Patrick Williams wrote:
> The wget fetcher currently silently drops all parameters and has
> been that way since it was originally introduced in fetch via 080eb8764.
> This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
> additional parameters.
> 
> Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
> ---
>  lib/bb/fetch2/wget.py | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e6d9f528..d41d7408 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -99,7 +99,13 @@ class Wget(FetchMethod):
>          if ud.user and ud.pswd:
>              fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
>  
> -        uri = ud.url.split(";")[0]
> +        # Join the URL components together.
> +        urls = ud.url.split(";")
> +        if len(urls) == 1:
> +            uri = urls[0]
> +        else:
> +            uri = urls[0] + "?" + "&".join(urls[1:])
> +
>          if os.path.exists(ud.localpath):
>              # file exists, but we didnt complete it.. trying again..
>              fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
> -- 
> 2.26.2
> 

This causes some problems that I need to work out more.  In my downloads
directory I ended up with files like:

    'libxml2-2.9.10.tar.gz?name=libtar'
    'libxml2-2.9.10.tar.gz?name=libtar.1'

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

[PATCH v2] fetch2/wget: include parameters in URL

[Patrick Williams]

The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
 lib/bb/fetch2/wget.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..d9432c5b 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -99,7 +99,14 @@ class Wget(FetchMethod):
         if ud.user and ud.pswd:
             fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
 
-        uri = ud.url.split(";")[0]
+        # Join the URL components together.
+        urls = ud.url.split(";")
+        if len(urls) == 1:
+            uri = urls[0]
+        else:
+            uri = urls[0] + "?" + "&".join(urls[1:])
+            fetchcmd += " -O '%s'" % urls[0].split("/")[-1]
+
         if os.path.exists(ud.localpath):
             # file exists, but we didnt complete it.. trying again..
             fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
-- 
2.26.2

[PATCH v3] fetch2/wget: include parameters in URL

[Patrick Williams]

The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
 lib/bb/fetch2/wget.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..80baf04a 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -99,7 +99,14 @@ class Wget(FetchMethod):
         if ud.user and ud.pswd:
             fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
 
-        uri = ud.url.split(";")[0]
+        # Join the URL components together.
+        urls = ud.url.split(";")
+        if len(urls) == 1:
+            uri = urls[0]
+        else:
+            uri = urls[0] + "?" + "&".join(urls[1:])
+            fetchcmd += " -O '%s/%s'" % (d.getVar("DL_DIR"), urls[0].split("/")[-1])
+
         if os.path.exists(ud.localpath):
             # file exists, but we didnt complete it.. trying again..
             fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
-- 
2.26.2

Re: [PATCH v3] fetch2/wget: include parameters in URL

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]

On Tue, Dec 29, 2020 at 03:57:46PM -0600, Patrick Williams wrote:
> The wget fetcher currently silently drops all parameters and has
> been that way since it was originally introduced in fetch via 080eb8764.
> This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
> additional parameters.
> 
> Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
> ---
>  lib/bb/fetch2/wget.py | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e6d9f528..80baf04a 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -99,7 +99,14 @@ class Wget(FetchMethod):
>          if ud.user and ud.pswd:
>              fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
>  
> -        uri = ud.url.split(";")[0]
> +        # Join the URL components together.
> +        urls = ud.url.split(";")
> +        if len(urls) == 1:
> +            uri = urls[0]
> +        else:
> +            uri = urls[0] + "?" + "&".join(urls[1:])
> +            fetchcmd += " -O '%s/%s'" % (d.getVar("DL_DIR"), urls[0].split("/")[-1])
> +

I've now tested this with a `bitbake <image> --runall=fetch` for one of
my machine images and was successful.  The `-O` above feels a little
fragile but it is required for the uninative blob because that ends up
in a `uninative/<checksum>` subdirectory.

In uninative.bbclass there is:
            localdata.setVar('DL_DIR', tarballdir)

The `-P ${DL_DIR}` below seems to not take effect if `-O` is given.

>          if os.path.exists(ud.localpath):
>              # file exists, but we didnt complete it.. trying again..
>              fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
> -- 
> 2.26.2
> 

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]

[PATCH v4] fetch2/wget: include parameters in URL

[Patrick Williams]

The wget fetcher currently silently drops all parameters and has
been that way since it was originally introduced in fetch via 080eb8764.
This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
additional parameters.

Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
---
 lib/bb/fetch2/wget.py | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
index e6d9f528..0019f10c 100644
--- a/lib/bb/fetch2/wget.py
+++ b/lib/bb/fetch2/wget.py
@@ -91,7 +91,14 @@ class Wget(FetchMethod):
 
         fetchcmd = self.basecmd
 
-        if 'downloadfilename' in ud.parm:
+        # Join the URL components together.
+        url_parts = ud.url.split(";")
+        if len(url_parts) == 1:
+            uri = url_parts[0]
+        else:
+            uri = url_parts[0] + "?" + "&".join(url_parts[1:])
+
+        if 'downloadfilename' in ud.parm or len(url_parts) != 1:
             localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile)
             bb.utils.mkdirhier(os.path.dirname(localpath))
             fetchcmd += " -O %s" % shlex.quote(localpath)
@@ -99,7 +106,6 @@ class Wget(FetchMethod):
         if ud.user and ud.pswd:
             fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
 
-        uri = ud.url.split(";")[0]
         if os.path.exists(ud.localpath):
             # file exists, but we didnt complete it.. trying again..
             fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
@@ -441,7 +447,7 @@ class Wget(FetchMethod):
                     valid = 1
                 elif self._vercmp(version, newver) < 0:
                     version = newver
-                
+
         pupver = re.sub('_', '.', version[1])
 
         bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
-- 
2.26.2

Re: [PATCH v4] fetch2/wget: include parameters in URL

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 2407 bytes --]

On Wed, Dec 30, 2020 at 04:03:23PM -0600, Patrick Williams wrote:
> The wget fetcher currently silently drops all parameters and has
> been that way since it was originally introduced in fetch via 080eb8764.
> This prevents use in SOURCE_MIRROR_URL and SRC_URI which require
> additional parameters.
> 
> Signed-off-by: Patrick Williams <patrick@stwcx.xyz>
> ---
>  lib/bb/fetch2/wget.py | 12 +++++++++---
>  1 file changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/lib/bb/fetch2/wget.py b/lib/bb/fetch2/wget.py
> index e6d9f528..0019f10c 100644
> --- a/lib/bb/fetch2/wget.py
> +++ b/lib/bb/fetch2/wget.py
> @@ -91,7 +91,14 @@ class Wget(FetchMethod):
>  
>          fetchcmd = self.basecmd
>  
> -        if 'downloadfilename' in ud.parm:
> +        # Join the URL components together.
> +        url_parts = ud.url.split(";")
> +        if len(url_parts) == 1:
> +            uri = url_parts[0]
> +        else:
> +            uri = url_parts[0] + "?" + "&".join(url_parts[1:])
> +
> +        if 'downloadfilename' in ud.parm or len(url_parts) != 1:
>              localpath = os.path.join(d.getVar("DL_DIR"), ud.localfile)
>              bb.utils.mkdirhier(os.path.dirname(localpath))
>              fetchcmd += " -O %s" % shlex.quote(localpath)
> @@ -99,7 +106,6 @@ class Wget(FetchMethod):
>          if ud.user and ud.pswd:
>              fetchcmd += " --user=%s --password=%s --auth-no-challenge" % (ud.user, ud.pswd)
>  
> -        uri = ud.url.split(";")[0]

I ran into one more edge case trying to re-download a file from my
internal mirror service: netbase_6.1~bpo10+1.tar.xz .  The URL
contains a URL-quoted version of the filename.  Switched the code
to reuse the 'downloadfilename' logic earlier in the function, which
is safer anyhow because it uses shlex, etc.

>          if os.path.exists(ud.localpath):
>              # file exists, but we didnt complete it.. trying again..
>              fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
> @@ -441,7 +447,7 @@ class Wget(FetchMethod):
>                      valid = 1
>                  elif self._vercmp(version, newver) < 0:
>                      version = newver
> -                
> +
>          pupver = re.sub('_', '.', version[1])
>  
>          bb.debug(3, "*** %s -> UpstreamVersion = %s (CurrentVersion = %s)" %
> -- 
> 2.26.2
> 

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 849 bytes --]