[PATCH] wget.py: support for semi-colons in URL

[PATCH] wget.py: support for semi-colons in URL

[Juro Bystricky]

Some URLs contain semi-colons. For example, some GIT repositories
support downloading snapshots using URL such as:

"http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"

Currently there is no way to construct SRC_URI with such URLs.
Bitbake uses semi-colons in SRC_URI as delimiters for various parameters.

This patch allows using of semi-colons in URLs: ';;' (double semi-colon)
in SRC_URI is processed  as a single semi-colon that is a part of a URL name.
For example, the above URL would become:

SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47"

Note that it is not possible to determine the name of the downloaded file
from URL alone. The onus to specify the proper name is on the user,
via the parameter 'downloadfilename':

SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=myfile.tar.gz"

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 bitbake/lib/bb/fetch2/wget.py | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/wget.py b/bitbake/lib/bb/fetch2/wget.py
index 425b6b9..a623b83 100644
--- a/bitbake/lib/bb/fetch2/wget.py
+++ b/bitbake/lib/bb/fetch2/wget.py
@@ -80,7 +80,21 @@ class Wget(FetchMethod):
             bb.utils.mkdirhier(os.path.dirname(dldir + os.sep + ud.localfile))
             fetchcmd += " -O " + dldir + os.sep + ud.localfile
 
-        uri = ud.url.split(";")[0]
+        # Process any ";;" in SRC_URI, these are escape sequences for ";" in URL
+        if ';;' in ud.url:
+           # If URL contains ';' the name of the downloaded file cannot be derived reliably
+           # (or derived at all) using URL string alone.
+           # Therefore we insist the file name is specified explicitly by the user.
+           if not 'downloadfilename' in ud.parm:
+               raise FetchError("Could not determine the name of the downloaded file. Please specify explicitly by using SRC_URI option 'downloadfilename'")
+
+           # Convert ";;" into something unique (not containing ';')
+           uri = ud.url.replace(";;","%3B%3B");
+           uri = uri.split(";")[0]
+           uri = uri.replace("%3B%3B",";");
+        else:
+           uri = ud.url.split(";")[0]
+
         if os.path.exists(ud.localpath):
             # file exists, but we didnt complete it.. trying again..
             fetchcmd += d.expand(" -c -P ${DL_DIR} '%s'" % uri)
-- 
1.9.1

Re: [PATCH] wget.py: support for semi-colons in URL

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1415 bytes --]

On Sat, Jul 25, 2015 at 11:32 AM, Juro Bystricky <juro.bystricky@intel.com>
wrote:

> Some URLs contain semi-colons. For example, some GIT repositories
> support downloading snapshots using URL such as:
>
> "http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
>
> Currently there is no way to construct SRC_URI with such URLs.
> Bitbake uses semi-colons in SRC_URI as delimiters for various parameters.
>
> This patch allows using of semi-colons in URLs: ';;' (double semi-colon)
> in SRC_URI is processed  as a single semi-colon that is a part of a URL
> name.
> For example, the above URL would become:
>
> SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47"
>
> Note that it is not possible to determine the name of the downloaded file
> from URL alone. The onus to specify the proper name is on the user,
> via the parameter 'downloadfilename':
>
> SRC_URI = "
> http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=myfile.tar.gz
> "
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
>

In my opinion, if we’re going to support this, we should really support it
in a generic way, in urldecode/urlencode & the URL class, and if needed
leverage that in wget.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2316 bytes --]

Re: [PATCH] wget.py: support for semi-colons in URL

[Bystricky, Juro]

[-- Attachment #1: Type: text/plain, Size: 2711 bytes --]

I agree that a generic solution is preferable. I also I think it is only prudent to support all valid URLs.
The situation is complicated by the fact that at present it may not be possible to determine how to interpret the ‘;’ in SRC_URI.
It can be part of the URL or it can be a SRC_URI delimiter/separator.
As I don’t suppose changing the delimiter character from semi-colon to something else (i.e. an unsafe URL character)
is an option, some other changes to the SRC_URI syntax are needed, for example:



1.       Some form of escape sequence (i.e. ‘;;’ means treat as ‘;’) in URL, i.e.

SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=abc.tar"



2.       Quoted string implementation, using one of the “unsafe” URL characters such as ‘`’, to delimit  the URL

SRC_URI = "’ http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47`;downloadfilename=abc.tar"



3.       Something else ?

Thanks

Juro



From: kergoth@gmail.com [mailto:kergoth@gmail.com] On Behalf Of Christopher Larson
Sent: Saturday, July 25, 2015 12:15 PM
To: Bystricky, Juro
Cc: bitbake-devel@lists.openembedded.org; Purdie, Richard
Subject: Re: [bitbake-devel] [PATCH] wget.py: support for semi-colons in URL


On Sat, Jul 25, 2015 at 11:32 AM, Juro Bystricky <juro.bystricky@intel.com<mailto:juro.bystricky@intel.com>> wrote:
Some URLs contain semi-colons. For example, some GIT repositories
support downloading snapshots using URL such as:

"http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"

Currently there is no way to construct SRC_URI with such URLs.
Bitbake uses semi-colons in SRC_URI as delimiters for various parameters.

This patch allows using of semi-colons in URLs: ';;' (double semi-colon)
in SRC_URI is processed  as a single semi-colon that is a part of a URL name.
For example, the above URL would become:

SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47"

Note that it is not possible to determine the name of the downloaded file
from URL alone. The onus to specify the proper name is on the user,
via the parameter 'downloadfilename':

SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=myfile.tar.gz"

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com<mailto:juro.bystricky@intel.com>>

In my opinion, if we’re going to support this, we should really support it in a generic way, in urldecode/urlencode & the URL class, and if needed leverage that in wget.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 11900 bytes --]

Re: [PATCH] wget.py: support for semi-colons in URL

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1786 bytes --]

On Mon, Jul 27, 2015 at 9:10 AM, Bystricky, Juro <juro.bystricky@intel.com>
wrote:

> I agree that a generic solution is preferable. I also I think it is only
> prudent to support all valid URLs.
>
> The situation is complicated by the fact that at present it may not be
> possible to determine how to interpret the ‘;’ in SRC_URI.
>
> It can be part of the URL or it can be a SRC_URI delimiter/separator.
>
> As I don’t suppose changing the delimiter character from semi-colon to
> something else (i.e. an unsafe URL character)
>
> is an option, some other changes to the SRC_URI syntax are needed, for
> example:
>
>
>
> 1.       Some form of escape sequence (i.e. ‘;;’ means treat as ‘;’) in
> URL, i.e.
>
> SRC_URI = "
> http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=abc.tar
> "
>

This is probably the only sane bet to retain compatibility, at least as far
as I can tell, but perhaps someone else has an idea.

2.       Quoted string implementation, using one of the “unsafe” URL
> characters such as ‘`’, to delimit  the URL
>
> SRC_URI = "’
> http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47`;downloadfilename=abc.tar
> <http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47;downloadfilename=abc.tar>
> "
>

Clearly this would be ideal, and is what we should have done, but it’s a
bit late now :) It’s not the only questionable choice we made in the URL
handling back when we wrote the original fetcher (e.g. file:// rather than
file:, but afaik one should only use the former if there’s a netloc/host).
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3643 bytes --]

Re: [PATCH] wget.py: support for semi-colons in URL

[Richard Purdie]

On Mon, 2015-07-27 at 10:58 -0700, Christopher Larson wrote:
> 
> On Mon, Jul 27, 2015 at 9:10 AM, Bystricky, Juro
> <juro.bystricky@intel.com> wrote:
>         I agree that a generic solution is preferable. I also I think
>         it is only prudent to support all valid URLs.
>         
>         The situation is complicated by the fact that at present it
>         may not be possible to determine how to interpret the ‘;’ in
>         SRC_URI.
>         
>         It can be part of the URL or it can be a SRC_URI
>         delimiter/separator.
>         
>         As I don’t suppose changing the delimiter character from
>         semi-colon to something else (i.e. an unsafe URL character)
>         
>         is an option, some other changes to the SRC_URI syntax are
>         needed, for example:
>         
>          
>         
>         1.      Some form of escape sequence (i.e. ‘;;’ means treat as
>         ‘;’) in URL, i.e.
>         
>         SRC_URI =
>         "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=abc.tar"
>         
> 
> 
> This is probably the only sane bet to retain compatibility, at least
> as far as I can tell, but perhaps someone else has an idea.
>
>         2.      Quoted string implementation, using one of the
>         “unsafe” URL characters such as ‘`’, to delimit  the URL
>
>         SRC_URI = "’
>         http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47`;downloadfilename=abc.tar"
>
> Clearly this would be ideal, and is what we should have done, but it’s
> a bit late now :) It’s not the only questionable choice we made in the
> URL handling back when we wrote the original fetcher (e.g. file://
> rather than file:, but afaik one should only use the former if there’s
> a netloc/host).

Yes, horse has bolted now. A bit like a lot of bitbake, hindsight is
great! :}.

There is an alternative I've been mulling over the weekend:

http://abc123.com/git/?p=gcc/gcc.git;extraparams=`;a=snapshot;h=a5dd47`;downloadfilename=abc.tar"
or
http://abc123.com/git/?p=gcc/gcc.git;extraparams=a=snapshot%3Bh=a5dd47;downloadfilename=abc.tar"

(or some variant of that, pick your favourite quote characters and ;
placement is for discussion). We'd then just pull out extraparams and
unescape it somehow. Each fetcher could then deal with it as
appropriate, or its just a wget fetcher specific option.

Just thinking out loud really...

Cheers,

Richard

Re: [PATCH] wget.py: support for semi-colons in URL

[Peter Urbanec]

I have not examined the code, but could the URL use the standard URL 
encoding mechanism to represent ; as %3B ?

The idea being that the initial "Bitbake URL" is parsed / split using 
the original string with semicolons and the URL escaped semicolon is 
passed through to a lower layer that uses that part of the string as an 
"Internet URL"

Cheers,
	Peter


On 28/07/15 02:10, Bystricky, Juro wrote:
> I agree that a generic solution is preferable. I also I think it is only
> prudent to support all valid URLs.
>
> The situation is complicated by the fact that at present it may not be
> possible to determine how to interpret the ‘;’ in SRC_URI.
>
> It can be part of the URL or it can be a SRC_URI delimiter/separator.
>
> As I don’t suppose changing the delimiter character from semi-colon to
> something else (i.e. an unsafe URL character)
>
> is an option, some other changes to the SRC_URI syntax are needed, for
> example:
>
> 1.Some form of escape sequence (i.e. ‘;;’ means treat as ‘;’) in URL, i.e.
>
> SRC_URI =
> "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=abc.tar"
>
> 2.Quoted string implementation, using one of the “unsafe” URL characters
> such as ‘`’, to delimit  the URL
>
> SRC_URI = "’
> http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47`;downloadfilename=abc.tar"
>
> 3.Something else ?
>
> Thanks
>
> Juro
>
> *From:*kergoth@gmail.com [mailto:kergoth@gmail.com] *On Behalf Of
> *Christopher Larson
> *Sent:* Saturday, July 25, 2015 12:15 PM
> *To:* Bystricky, Juro
> *Cc:* bitbake-devel@lists.openembedded.org; Purdie, Richard
> *Subject:* Re: [bitbake-devel] [PATCH] wget.py: support for semi-colons
> in URL
>
> On Sat, Jul 25, 2015 at 11:32 AM, Juro Bystricky
> <juro.bystricky@intel.com <mailto:juro.bystricky@intel.com>> wrote:
>
>     Some URLs contain semi-colons. For example, some GIT repositories
>     support downloading snapshots using URL such as:
>
>     "http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
>
>     Currently there is no way to construct SRC_URI with such URLs.
>     Bitbake uses semi-colons in SRC_URI as delimiters for various
>     parameters.
>
>     This patch allows using of semi-colons in URLs: ';;' (double semi-colon)
>     in SRC_URI is processed  as a single semi-colon that is a part of a
>     URL name.
>     For example, the above URL would become:
>
>     SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47"
>
>     Note that it is not possible to determine the name of the downloaded
>     file
>     from URL alone. The onus to specify the proper name is on the user,
>     via the parameter 'downloadfilename':
>
>     SRC_URI =
>     "http://abc123.com/git/?p=gcc/gcc.git;;a=snapshot;;h=a5dd47;downloadfilename=myfile.tar.gz"
>
>     Signed-off-by: Juro Bystricky <juro.bystricky@intel.com
>     <mailto:juro.bystricky@intel.com>>
>
>
> In my opinion, if we’re going to support this, we should really support
> it in a generic way, in urldecode/urlencode & the URL class, and if
> needed leverage that in wget.
> --
>
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
>
>
>

Re: [PATCH] wget.py: support for semi-colons in URL

[Olof Johansson]

Excerpts from Peter Urbanec's message of 2015-07-28 07:47:17 +0200:
> I have not examined the code, but could the URL use the standard URL 
> encoding mechanism to represent ; as %3B ?
> 
> The idea being that the initial "Bitbake URL" is parsed / split using 
> the original string with semicolons and the URL escaped semicolon is 
> passed through to a lower layer that uses that part of the string as an 
> "Internet URL"

URI encoded enitities today are handled correctly in my opinion. If I
have a filename including a ;, I don't want it to be parsed as part of
the URL, neither by bitbake nor the remote server --- that's where URI
encoding comes in.

By hijacking URI encoding to mean "part of the URI, not parsed by
bitbake", you will suprise users that expect it to mean "URI data, not
URI control character" if I understand your proposal correctly.

That could be solved by double encoding, but I don't think that would be
worth it, all occurences of URI encoding in SRC_URI today would have to
be updated. E.g. %20 -> %2520.

-- 
olof johansson

Re: [PATCH] wget.py: support for semi-colons in URL

[Olof Johansson]

Excerpts from Juro Bystricky's message of 2015-07-25 20:32:43 +0200:
> Some URLs contain semi-colons. For example, some GIT repositories
> support downloading snapshots using URL such as:
> 
> "http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
> 
> Currently there is no way to construct SRC_URI with such URLs.
> Bitbake uses semi-colons in SRC_URI as delimiters for various parameters.

W3C _recommends_ [1] that HTTP URL consumers treat ; in query strings as
an &:

> We recommend that HTTP server implementors, and in particular, CGI
> implementors support the use of ";" in place of "&" to save authors
> the trouble of escaping "&" characters in this manner.

This means that

  http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47

and

  http://abc123.com/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47

should be equivalent. I don't know how well spread this support is, or
if it works in your case, but it's a workaround if it does at least.

1: http://www.w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.2.2

-- 
olof johansson

Re: [PATCH] wget.py: support for semi-colons in URL

[Bystricky, Juro]

I tested your suggestion (replacing ";" with "&" in the SRC_URI ) and in my case it did work properly.
So even if this may not solve everything, I believe this should be properly documented somewhere
as this seems to be currently the only way to use SRC_URI with semi-colons in URL.

Note that one still needs to specify the downloaded file name manually, i.e.

SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47; downloadfilename=myfile.bz2"

Thanks

Juro Bystricky


> -----Original Message-----
> From: Olof Johansson [mailto:olof.johansson@axis.com]
> Sent: Tuesday, July 28, 2015 4:01 AM
> To: Bystricky, Juro; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH] wget.py: support for semi-colons in
> URL
> 
> Excerpts from Juro Bystricky's message of 2015-07-25 20:32:43 +0200:
> > Some URLs contain semi-colons. For example, some GIT repositories
> > support downloading snapshots using URL such as:
> >
> > "http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47"
> >
> > Currently there is no way to construct SRC_URI with such URLs.
> > Bitbake uses semi-colons in SRC_URI as delimiters for various parameters.
> 
> W3C _recommends_ [1] that HTTP URL consumers treat ; in query strings as
> an &:
> 
> > We recommend that HTTP server implementors, and in particular, CGI
> > implementors support the use of ";" in place of "&" to save authors
> > the trouble of escaping "&" characters in this manner.
> 
> This means that
> 
>   http://abc123.com/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47
> 
> and
> 
>   http://abc123.com/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47
> 
> should be equivalent. I don't know how well spread this support is, or if it
> works in your case, but it's a workaround if it does at least.
> 
> 1: http://www.w3.org/TR/1999/REC-html401-
> 19991224/appendix/notes.html#h-B.2.2
> 
> --
> olof johansson

Re: [PATCH] wget.py: support for semi-colons in URL

[Richard Purdie]

On Tue, 2015-07-28 at 16:06 +0000, Bystricky, Juro wrote:
> I tested your suggestion (replacing ";" with "&" in the SRC_URI ) and in my case it did work properly.
> So even if this may not solve everything, I believe this should be properly documented somewhere
> as this seems to be currently the only way to use SRC_URI with semi-colons in URL.
> 
> Note that one still needs to specify the downloaded file name manually, i.e.
> 
> SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47; downloadfilename=myfile.bz2"

A patch adding some comments about this to wget.py would be nice, as
would a patch documenting it in the bitbake manual...

Cheers,

Richard

Re: [PATCH] wget.py: support for semi-colons in URL

[Bystricky, Juro]

OK, I will send in  a new patch for bitbake manual. 

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Tuesday, July 28, 2015 12:38 PM
> To: Bystricky, Juro
> Cc: Olof Johansson; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH] wget.py: support for semi-colons in
> URL
> 
> On Tue, 2015-07-28 at 16:06 +0000, Bystricky, Juro wrote:
> > I tested your suggestion (replacing ";" with "&" in the SRC_URI ) and in my
> case it did work properly.
> > So even if this may not solve everything, I believe this should be
> > properly documented somewhere as this seems to be currently the only
> way to use SRC_URI with semi-colons in URL.
> >
> > Note that one still needs to specify the downloaded file name manually,
> i.e.
> >
> > SRC_URI = "http://abc123.com/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;
> downloadfilename=myfile.bz2"
> 
> A patch adding some comments about this to wget.py would be nice, as
> would a patch documenting it in the bitbake manual...
> 
> Cheers,
> 
> Richard