[PATCH] fetch2: Set maxsplit to match expected variables

[PATCH] fetch2: Set maxsplit to match expected variables

[dit.kozmaj]

From: Dit Kozmaj <dit.kozmaj@kynetics.com>

Set the maxsplit value to match the expected number of variables.
This also avoids an unnecessary split as the parameters are in the form 'key=value'
and the 'value' could contain the '=' character.

Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
---
 bitbake/lib/bb/fetch2/__init__.py | 2 +-
 bitbake/lib/bb/tests/fetch.py     | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 3e6555bd67..79b289eaec 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -388,7 +388,7 @@ def decodeurl(url):
             if s:
                 if not '=' in s:
                     raise MalformedUrl(url, "The URL: '%s' is invalid: parameter %s does not specify a value (missing '=')" % (url, s))
-                s1, s2 = s.split('=')
+                s1, s2 = s.split('=', 1)
                 p[s1] = s2
 
     return type, host, urllib.parse.unquote(path), user, pswd, p
diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 61dd5cccaf..a261ad69d8 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -1197,6 +1197,7 @@ class URLHandle(unittest.TestCase):
        "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', 'anonymous', collections.OrderedDict([('tag', 'V0-99-81'), ('module', 'familiar/dist/ipkg')])),
        "git://git.openembedded.org/bitbake;branch=@foo" : ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'}),
        "file://somelocation;someparam=1": ('file', '', 'somelocation', '', '', {'someparam': '1'}),
+       "https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" : ('https', 'somesite.com', '/somerepo.git', '', '', {'user': 'anyUser:idtoken=1234'}),
     }
     # we require a pathname to encodeurl but users can still pass such urls to 
     # decodeurl and we need to handle them
-- 
2.25.1

Re: [bitbake-devel] [PATCH] fetch2: Set maxsplit to match expected variables

[Alexandre Belloni]

Hello,

On 25/07/2023 13:47:10+0000, Dit Kozmaj wrote:
> From: Dit Kozmaj <dit.kozmaj@kynetics.com>
> 
> Set the maxsplit value to match the expected number of variables.
> This also avoids an unnecessary split as the parameters are in the form 'key=value'
> and the 'value' could contain the '=' character.
> 
> Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
> ---
>  bitbake/lib/bb/fetch2/__init__.py | 2 +-
>  bitbake/lib/bb/tests/fetch.py     | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
> 

This doesn't apply one master, which branch did you use? Can you rebase?

> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index 3e6555bd67..79b289eaec 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -388,7 +388,7 @@ def decodeurl(url):
>              if s:
>                  if not '=' in s:
>                      raise MalformedUrl(url, "The URL: '%s' is invalid: parameter %s does not specify a value (missing '=')" % (url, s))
> -                s1, s2 = s.split('=')
> +                s1, s2 = s.split('=', 1)
>                  p[s1] = s2
>  
>      return type, host, urllib.parse.unquote(path), user, pswd, p
> diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
> index 61dd5cccaf..a261ad69d8 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -1197,6 +1197,7 @@ class URLHandle(unittest.TestCase):
>         "cvs://anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg" : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', 'anonymous', collections.OrderedDict([('tag', 'V0-99-81'), ('module', 'familiar/dist/ipkg')])),
>         "git://git.openembedded.org/bitbake;branch=@foo" : ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'}),
>         "file://somelocation;someparam=1": ('file', '', 'somelocation', '', '', {'someparam': '1'}),
> +       "https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" : ('https', 'somesite.com', '/somerepo.git', '', '', {'user': 'anyUser:idtoken=1234'}),
>      }
>      # we require a pathname to encodeurl but users can still pass such urls to 
>      # decodeurl and we need to handle them
> -- 
> 2.25.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14887): https://lists.openembedded.org/g/bitbake-devel/message/14887
> Mute This Topic: https://lists.openembedded.org/mt/100350479/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [bitbake-devel] [PATCH] fetch2: Set maxsplit to match expected variables

[Dit Kozmaj]

[-- Attachment #1: Type: text/plain, Size: 3347 bytes --]

Hello, I was using the Dunfell branch.
I forget to move to the master branch to apply the patch. Sorry about that.
However I send a new Patch v2 that applies on the master branch and
obviously test it.

Thanks.



On Tue, Jul 25, 2023 at 6:39 PM Alexandre Belloni <
alexandre.belloni@bootlin.com> wrote:

> Hello,
>
> On 25/07/2023 13:47:10+0000, Dit Kozmaj wrote:
> > From: Dit Kozmaj <dit.kozmaj@kynetics.com>
> >
> > Set the maxsplit value to match the expected number of variables.
> > This also avoids an unnecessary split as the parameters are in the form
> 'key=value'
> > and the 'value' could contain the '=' character.
> >
> > Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
> > ---
> >  bitbake/lib/bb/fetch2/__init__.py | 2 +-
> >  bitbake/lib/bb/tests/fetch.py     | 1 +
> >  2 files changed, 2 insertions(+), 1 deletion(-)
> >
>
> This doesn't apply one master, which branch did you use? Can you rebase?
>
> > diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py
> > index 3e6555bd67..79b289eaec 100644
> > --- a/bitbake/lib/bb/fetch2/__init__.py
> > +++ b/bitbake/lib/bb/fetch2/__init__.py
> > @@ -388,7 +388,7 @@ def decodeurl(url):
> >              if s:
> >                  if not '=' in s:
> >                      raise MalformedUrl(url, "The URL: '%s' is invalid:
> parameter %s does not specify a value (missing '=')" % (url, s))
> > -                s1, s2 = s.split('=')
> > +                s1, s2 = s.split('=', 1)
> >                  p[s1] = s2
> >
> >      return type, host, urllib.parse.unquote(path), user, pswd, p
> > diff --git a/bitbake/lib/bb/tests/fetch.py
> b/bitbake/lib/bb/tests/fetch.py
> > index 61dd5cccaf..a261ad69d8 100644
> > --- a/bitbake/lib/bb/tests/fetch.py
> > +++ b/bitbake/lib/bb/tests/fetch.py
> > @@ -1197,6 +1197,7 @@ class URLHandle(unittest.TestCase):
> >         "cvs://
> anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg"
> : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', 'anonymous',
> collections.OrderedDict([('tag', 'V0-99-81'), ('module',
> 'familiar/dist/ipkg')])),
> >         "git://git.openembedded.org/bitbake;branch=@foo" : ('git', '
> git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'}),
> >         "file://somelocation;someparam=1": ('file', '', 'somelocation',
> '', '', {'someparam': '1'}),
> > +       "https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" :
> ('https', 'somesite.com', '/somerepo.git', '', '', {'user':
> 'anyUser:idtoken=1234'}),
> >      }
> >      # we require a pathname to encodeurl but users can still pass such
> urls to
> >      # decodeurl and we need to handle them
> > --
> > 2.25.1
> >
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#14887):
> https://lists.openembedded.org/g/bitbake-devel/message/14887
> > Mute This Topic: https://lists.openembedded.org/mt/100350479/3617179
> > Group Owner: bitbake-devel+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

[-- Attachment #2: Type: text/html, Size: 5604 bytes --]

Re: [bitbake-devel] [PATCH] fetch2: Set maxsplit to match expected variables

[Dit Kozmaj]

[-- Attachment #1: Type: text/plain, Size: 3773 bytes --]

Bitbucket added a new feature that provides limited access to a specific
repository using a Repository Access Tokens. This token comes in the form
token-auth:key.

https://support.atlassian.com/bitbucket-cloud/docs/repository-access-tokens/
https://support.atlassian.com/bitbucket-cloud/docs/using-access-tokens/#Include-the-Repository-Access-Token-in-the-URL

The token can be used in the user parameter of SRC_URI as this parameter is
in the form user=userid:password, and works correctly excluding the cases
when the password contains a "=" symbol.
The error comes from the split logic, which doesn't manage the case when
the value contains the symbol "=".
The proposed change guarantees that only two variables will be returned,
splitted in "key" and "value" and the value can contain, especially in
cases when it is a password, the "=" character.
As requested, we also add a test case , run the test with bitbake-selftest
bb.tests.fetch. The result is successful.



On Tue, Jul 25, 2023 at 3:47 PM Dit Kozmaj via lists.openembedded.org
<dit.kozmaj=kynetics.com@lists.openembedded.org> wrote:

> From: Dit Kozmaj <dit.kozmaj@kynetics.com>
>
> Set the maxsplit value to match the expected number of variables.
> This also avoids an unnecessary split as the parameters are in the form
> 'key=value'
> and the 'value' could contain the '=' character.
>
> Signed-off-by: Dit Kozmaj <dit.kozmaj@kynetics.com>
> ---
>  bitbake/lib/bb/fetch2/__init__.py | 2 +-
>  bitbake/lib/bb/tests/fetch.py     | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/bitbake/lib/bb/fetch2/__init__.py
> b/bitbake/lib/bb/fetch2/__init__.py
> index 3e6555bd67..79b289eaec 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -388,7 +388,7 @@ def decodeurl(url):
>              if s:
>                  if not '=' in s:
>                      raise MalformedUrl(url, "The URL: '%s' is invalid:
> parameter %s does not specify a value (missing '=')" % (url, s))
> -                s1, s2 = s.split('=')
> +                s1, s2 = s.split('=', 1)
>                  p[s1] = s2
>
>      return type, host, urllib.parse.unquote(path), user, pswd, p
> diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
> index 61dd5cccaf..a261ad69d8 100644
> --- a/bitbake/lib/bb/tests/fetch.py
> +++ b/bitbake/lib/bb/tests/fetch.py
> @@ -1197,6 +1197,7 @@ class URLHandle(unittest.TestCase):
>         "cvs://
> anoncvs:anonymous@cvs.handhelds.org/cvs;tag=V0-99-81;module=familiar/dist/ipkg"
> : ('cvs', 'cvs.handhelds.org', '/cvs', 'anoncvs', 'anonymous',
> collections.OrderedDict([('tag', 'V0-99-81'), ('module',
> 'familiar/dist/ipkg')])),
>         "git://git.openembedded.org/bitbake;branch=@foo" : ('git', '
> git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'}),
>         "file://somelocation;someparam=1": ('file', '', 'somelocation',
> '', '', {'someparam': '1'}),
> +       "https://somesite.com/somerepo.git;user=anyUser:idtoken=1234" :
> ('https', 'somesite.com', '/somerepo.git', '', '', {'user':
> 'anyUser:idtoken=1234'}),
>      }
>      # we require a pathname to encodeurl but users can still pass such
> urls to
>      # decodeurl and we need to handle them
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14887):
> https://lists.openembedded.org/g/bitbake-devel/message/14887
> Mute This Topic: https://lists.openembedded.org/mt/100350479/7542774
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> dit.kozmaj@kynetics.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 6188 bytes --]