Setting SRC_URI via anonymous Python fails in OE 2.1

Setting SRC_URI via anonymous Python fails in OE 2.1

[Andre McCurdy]

The following works in OE 2.0, but fails in 2.1 and master:

  python () {
      d.setVar("SRC_URI_FOO", "http://foo.com/foo.tgz")
  }

  SRC_URI = "${SRC_URI_FOO}"


Is it a regression or was something like that never intended to work?

Re: Setting SRC_URI via anonymous Python fails in OE 2.1

[Paul Eggleton]

Hi Andre,

On Tue, 28 Jun 2016 15:53:24 Andre McCurdy wrote:
> The following works in OE 2.0, but fails in 2.1 and master:
> 
>   python () {
>       d.setVar("SRC_URI_FOO", "http://foo.com/foo.tgz")
>   }
> 
>   SRC_URI = "${SRC_URI_FOO}"
> 
> 
> Is it a regression or was something like that never intended to work?

You didn't mention how it fails. However, I don't see any reason why we 
wouldn't want this to work, so if it does no longer work that seems like a bug 
to me.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Setting SRC_URI via anonymous Python fails in OE 2.1

[Andre McCurdy]

On Tue, Jun 28, 2016 at 7:15 PM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> Hi Andre,
>
> On Tue, 28 Jun 2016 15:53:24 Andre McCurdy wrote:
>> The following works in OE 2.0, but fails in 2.1 and master:
>>
>>   python () {
>>       d.setVar("SRC_URI_FOO", "http://foo.com/foo.tgz")
>>   }
>>
>>   SRC_URI = "${SRC_URI_FOO}"
>>
>> Is it a regression or was something like that never intended to work?
>
> You didn't mention how it fails. However, I don't see any reason why we
> wouldn't want this to work, so if it does no longer work that seems like a bug
> to me.

The problem is that the intermediate variable doesn't seem to get
expanded, so SRC_URI ends up containing the string "${SRC_URI_FOO}".

ERROR: /.../footest_0.1.bb: Error executing a python function in <code>:

The stack trace of python calls that resulted in this exception/failure was:
File: '<code>', lineno: 1, function: <module>
 *** 0001:__anon_651__home_andre_rdk_meta_classes_base_bbclass(d)
     0002:__anon_1315__home_andre_rdk_meta_classes_insane_bbclass(d)
     0003:__anon_242__home_andre_rdk_meta_classes_package_bbclass(d)
     0004:__anon_747__home_andre_rdk_meta_classes_package_rpm_bbclass(d)
     0005:__anon_25__home_andre_rdk_meta_classes_debian_bbclass(d)

File: '/.../meta/classes/base.bbclass', lineno: 552, function:
__anon_651__home_andre_rdk_meta_classes_base_bbclass
     0548:    needsrcrev = False
     0549:    srcuri = d.getVar('SRC_URI', True)
     0550:    for uri in srcuri.split():
     0551:        (scheme, _ , path) = bb.fetch.decodeurl(uri)[:3]
 *** 0552:
     0553:        # HTTP/FTP use the wget fetcher
     0554:        if scheme in ("http", "https", "ftp"):
     0555:            d.appendVarFlag('do_fetch', 'depends', '
wget-native:do_populate_sysroot')
     0556:

File: '/.../bitbake/lib/bb/fetch2/__init__.py', lineno: 360, function: decodeurl
     0356:    """
     0357:
     0358:    m =
re.compile('(?P<type>[^:]*)://((?P<user>[^/]+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url)
     0359:    if not m:
 *** 0360:        raise MalformedUrl(url)
     0361:
     0362:    type = m.group('type')
     0363:    location = m.group('location')
     0364:    if not location:

Exception: bb.fetch2.MalformedUrl: The URL: '${SRC_URI_FOO}' is
invalid and cannot be interpreted

ERROR: Failed to parse recipe: /.../footest_0.1.bb


> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre

Re: Setting SRC_URI via anonymous Python fails in OE 2.1

[Richard Purdie]

On Wed, 2016-06-29 at 00:40 -0700, Andre McCurdy wrote:
> On Tue, Jun 28, 2016 at 7:15 PM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
> > Hi Andre,
> > 
> > On Tue, 28 Jun 2016 15:53:24 Andre McCurdy wrote:
> > > The following works in OE 2.0, but fails in 2.1 and master:
> > > 
> > >   python () {
> > >       d.setVar("SRC_URI_FOO", "http://foo.com/foo.tgz")
> > >   }
> > > 
> > >   SRC_URI = "${SRC_URI_FOO}"
> > > 
> > > Is it a regression or was something like that never intended to
> > > work?
> > 
> > You didn't mention how it fails. However, I don't see any reason
> > why we
> > wouldn't want this to work, so if it does no longer work that seems
> > like a bug
> > to me.
> 
> The problem is that the intermediate variable doesn't seem to get
> expanded, so SRC_URI ends up containing the string "${SRC_URI_FOO}".
> 
> ERROR: /.../footest_0.1.bb: Error executing a python function in
> <code>:
> 
> The stack trace of python calls that resulted in this
> exception/failure was:
> File: '<code>', lineno: 1, function: <module>
>  *** 0001:__anon_651__home_andre_rdk_meta_classes_base_bbclass(d)
>      0002:__anon_1315__home_andre_rdk_meta_classes_insane_bbclass(d)
>      0003:__anon_242__home_andre_rdk_meta_classes_package_bbclass(d)
>     
>  0004:__anon_747__home_andre_rdk_meta_classes_package_rpm_bbclass(d)
>      0005:__anon_25__home_andre_rdk_meta_classes_debian_bbclass(d)
> 
> File: '/.../meta/classes/base.bbclass', lineno: 552, function:
> __anon_651__home_andre_rdk_meta_classes_base_bbclass
>      0548:    needsrcrev = False
>      0549:    srcuri = d.getVar('SRC_URI', True)
>      0550:    for uri in srcuri.split():
>      0551:        (scheme, _ , path) = bb.fetch.decodeurl(uri)[:3]
>  *** 0552:
>      0553:        # HTTP/FTP use the wget fetcher
>      0554:        if scheme in ("http", "https", "ftp"):
>      0555:            d.appendVarFlag('do_fetch', 'depends', '
> wget-native:do_populate_sysroot')
>      0556:
> 
> File: '/.../bitbake/lib/bb/fetch2/__init__.py', lineno: 360,
> function: decodeurl
>      0356:    """
>      0357:
>      0358:    m =
> re.compile('(?P<type>[^:]*)
> ://((?P<user>[^/]+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url
> )
>      0359:    if not m:
>  *** 0360:        raise MalformedUrl(url)
>      0361:
>      0362:    type = m.group('type')
>      0363:    location = m.group('location')
>      0364:    if not location:
> 
> Exception: bb.fetch2.MalformedUrl: The URL: '${SRC_URI_FOO}' is
> invalid and cannot be interpreted
> 
> ERROR: Failed to parse recipe: /.../footest_0.1.bb

I'm just guessing but this sounds like an anonymous python ordering
issue. The error above looks to be coming from the anonymous python in
base.bbclass and its depending on the anonymous python in your recipe
having run first. There are no order guarantees about anonymous python.

[As an aside, bitbake simply collects up a list of anonymous python
functions and then calls them all at the end of parsing. Its possible
python3 is being a bit more interesting about the ordering of that
list, or it could be something else]

This is something we likely need to look into from a determinism
perspective but its also a hard problem...

Cheers,

Richard