Modifying SRC_URI from anonymous python

Modifying SRC_URI from anonymous python

[Andre McCurdy]

I have a supplier who provides recipes which set SRC_URI to their
private git servers. To make those same recipes usable by others (ie
me), some anonymous python is used to transform the default SRC_URI
(elements which contain private git URLs are replaced, patches and
other files are left as-is).

This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
python which modifies SRC_URI races with the anonymous python in
base.bbclass which parses SRC_URI to determine additional
do_fetch/do_unpack dependencies and whether or not to call
fetch2.get_srcrev(). Specifically I get failures because
fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
AUTOREV from a repo to which I don't have access.

The proposed solution from the supplier is this patch to base.bbclass:

@@ -598,7 +598,7 @@ python () {
             d.appendVarFlag('do_unpack', 'depends', '
file-native:do_populate_sysroot')

     if needsrcrev:
-        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
+        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}", parsing=True)

     set_packagetriplet(d)

After reading the setVar source I'm not very clear how or why this
works, but it looks dubious. What is parsing=True intended to do?

Is it documented somewhere that modifying SRC_URI from anonymous
python isn't allowed? I've now seen two suppliers both independently
run into the same problem when updating to OE 2.1.

Re: Modifying SRC_URI from anonymous python

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

On Thu, Nov 17, 2016 at 2:28 PM, Andre McCurdy <armccurdy@gmail.com> wrote:

> I have a supplier who provides recipes which set SRC_URI to their
> private git servers. To make those same recipes usable by others (ie
> me), some anonymous python is used to transform the default SRC_URI
> (elements which contain private git URLs are replaced, patches and
> other files are left as-is).
>
> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> python which modifies SRC_URI races with the anonymous python in
> base.bbclass which parses SRC_URI to determine additional
> do_fetch/do_unpack dependencies and whether or not to call
> fetch2.get_srcrev(). Specifically I get failures because
> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> AUTOREV from a repo to which I don't have access.
>

You might want to try using a RecipePreFinalise event handler instead, they
run before anonymous python functions do.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1576 bytes --]

Re: Modifying SRC_URI from anonymous python

[Richard Purdie]

On Thu, 2016-11-17 at 13:28 -0800, Andre McCurdy wrote:
> I have a supplier who provides recipes which set SRC_URI to their
> private git servers. To make those same recipes usable by others (ie
> me), some anonymous python is used to transform the default SRC_URI
> (elements which contain private git URLs are replaced, patches and
> other files are left as-is).
> 
> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> python which modifies SRC_URI races with the anonymous python in
> base.bbclass which parses SRC_URI to determine additional
> do_fetch/do_unpack dependencies and whether or not to call
> fetch2.get_srcrev().

I suspect we made some changes around that time to ensure determinism
in the order certain things happened.

>  Specifically I get failures because
> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> AUTOREV from a repo to which I don't have access.
> 
> The proposed solution from the supplier is this patch to
> base.bbclass:
> 
> @@ -598,7 +598,7 @@ python () {
>              d.appendVarFlag('do_unpack', 'depends', '
> file-native:do_populate_sysroot')
> 
>      if needsrcrev:
> -        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> +        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}",
> parsing=True)
> 
>      set_packagetriplet(d)


parsing=True is a bitbake internal thing, its not documented and
shouldn't be used outside the bitbake codebase. In hindsight, naming
could have been better.

> After reading the setVar source I'm not very clear how or why this
> works, but it looks dubious. What is parsing=True intended to do?

Its internal bitbake data store stuff, don't use or rely on it. It
might happen to work here but its just luck.

> Is it documented somewhere that modifying SRC_URI from anonymous
> python isn't allowed? I've now seen two suppliers both independently
> run into the same problem when updating to OE 2.1.

We have an open bug related to anonymous python ordering. Its hard to
fix easily as something can indicate it wants to run last, but then
everything wants to run last. As Chris mentions, you can use an earlier
event handler as one way to work around it.

I'd also note that anonymous python runs in parsed order. Since
base.bbclass is "up front", its hard to run things before it though :/.

Cheers,

Richard

Re: Modifying SRC_URI from anonymous python

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]

On Thu, Nov 17, 2016 at 4:06 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Thu, 2016-11-17 at 13:28 -0800, Andre McCurdy wrote:
> > I have a supplier who provides recipes which set SRC_URI to their
> > private git servers. To make those same recipes usable by others (ie
> > me), some anonymous python is used to transform the default SRC_URI
> > (elements which contain private git URLs are replaced, patches and
> > other files are left as-is).
> >
> > This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> > python which modifies SRC_URI races with the anonymous python in
> > base.bbclass which parses SRC_URI to determine additional
> > do_fetch/do_unpack dependencies and whether or not to call
> > fetch2.get_srcrev().
>
> I suspect we made some changes around that time to ensure determinism
> in the order certain things happened.
>
> >  Specifically I get failures because
> > fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> > AUTOREV from a repo to which I don't have access.
> >
> > The proposed solution from the supplier is this patch to
> > base.bbclass:
> >
> > @@ -598,7 +598,7 @@ python () {
> >              d.appendVarFlag('do_unpack', 'depends', '
> > file-native:do_populate_sysroot')
> >
> >      if needsrcrev:
> > -        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> > +        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}",
> > parsing=True)
> >
> >      set_packagetriplet(d)
>
>
> parsing=True is a bitbake internal thing, its not documented and
> shouldn't be used outside the bitbake codebase. In hindsight, naming
> could have been better.
>
> > After reading the setVar source I'm not very clear how or why this
> > works, but it looks dubious. What is parsing=True intended to do?
>
> Its internal bitbake data store stuff, don't use or rely on it. It
> might happen to work here but its just luck.
>
> > Is it documented somewhere that modifying SRC_URI from anonymous
> > python isn't allowed? I've now seen two suppliers both independently
> > run into the same problem when updating to OE 2.1.
>
> We have an open bug related to anonymous python ordering. Its hard to
> fix easily as something can indicate it wants to run last, but then
> everything wants to run last. As Chris mentions, you can use an earlier
> event handler as one way to work around it.


I think our best bet there is to move toward more explicit event handler
naming and dependency handling amongst event handlers.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3531 bytes --]

Re: Modifying SRC_URI from anonymous python

[Andre McCurdy]

On Thu, Nov 17, 2016 at 2:00 PM, Christopher Larson <clarson@kergoth.com> wrote:
>
> On Thu, Nov 17, 2016 at 2:28 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
>>
>> I have a supplier who provides recipes which set SRC_URI to their
>> private git servers. To make those same recipes usable by others (ie
>> me), some anonymous python is used to transform the default SRC_URI
>> (elements which contain private git URLs are replaced, patches and
>> other files are left as-is).
>>
>> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
>> python which modifies SRC_URI races with the anonymous python in
>> base.bbclass which parses SRC_URI to determine additional
>> do_fetch/do_unpack dependencies and whether or not to call
>> fetch2.get_srcrev(). Specifically I get failures because
>> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
>> AUTOREV from a repo to which I don't have access.
>
> You might want to try using a RecipePreFinalise event handler instead, they
> run before anonymous python functions do.

Thanks, that works.

(For the benefit of anyone else who may try copying and pasting the
example from the bitbake user manual, the correct eventmask is
"bb.event.RecipePreFinalise" rather than
"bb.build.RecipePreFinalise").

> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics

Re: Modifying SRC_URI from anonymous python

[Patrick Ohly]

On Thu, 2016-11-17 at 13:28 -0800, Andre McCurdy wrote:
> I have a supplier who provides recipes which set SRC_URI to their
> private git servers. To make those same recipes usable by others (ie
> me), some anonymous python is used to transform the default SRC_URI
> (elements which contain private git URLs are replaced, patches and
> other files are left as-is).
> 
> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> python which modifies SRC_URI races with the anonymous python in
> base.bbclass which parses SRC_URI to determine additional
> do_fetch/do_unpack dependencies and whether or not to call
> fetch2.get_srcrev().

I have a patch which I intend to submit to bitbake which adds priorities
for anonymous python methods:
https://github.com/pohly/poky/commit/9d959e50e5f27d149654f5db0aff2fe51365ad68

With that in place, the anonymous python method which transforms the
SRC_URI could request to run sooner than the base.bbclass anonymous
python.

However, I am not sure whether that should be backported to 2.1 and 2.2.

>  Specifically I get failures because
> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> AUTOREV from a repo to which I don't have access.
> 
> The proposed solution from the supplier is this patch to base.bbclass:
> 
> @@ -598,7 +598,7 @@ python () {
>              d.appendVarFlag('do_unpack', 'depends', '
> file-native:do_populate_sysroot')
> 
>      if needsrcrev:
> -        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> +        d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}", parsing=True)
> 
>      set_packagetriplet(d)
> 
> After reading the setVar source I'm not very clear how or why this
> works, but it looks dubious. What is parsing=True intended to do?
> 
> Is it documented somewhere that modifying SRC_URI from anonymous
> python isn't allowed? I've now seen two suppliers both independently
> run into the same problem when updating to OE 2.1.

Sorry, I don't know about that.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.