* Re: Modifying SRC_URI from anonymous python
2016-11-17 21:28 Modifying SRC_URI from anonymous python Andre McCurdy
@ 2016-11-17 22:00 ` Christopher Larson
2016-11-18 3:31 ` Andre McCurdy
2016-11-17 23:06 ` Richard Purdie
2016-11-18 7:44 ` Patrick Ohly
2 siblings, 1 reply; 6+ messages in thread
From: Christopher Larson @ 2016-11-17 22:00 UTC (permalink / raw)
To: Andre McCurdy; +Cc: OE Core mailing list
[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]
On Thu, Nov 17, 2016 at 2:28 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
> I have a supplier who provides recipes which set SRC_URI to their
> private git servers. To make those same recipes usable by others (ie
> me), some anonymous python is used to transform the default SRC_URI
> (elements which contain private git URLs are replaced, patches and
> other files are left as-is).
>
> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> python which modifies SRC_URI races with the anonymous python in
> base.bbclass which parses SRC_URI to determine additional
> do_fetch/do_unpack dependencies and whether or not to call
> fetch2.get_srcrev(). Specifically I get failures because
> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> AUTOREV from a repo to which I don't have access.
>
You might want to try using a RecipePreFinalise event handler instead, they
run before anonymous python functions do.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 1576 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Modifying SRC_URI from anonymous python
2016-11-17 22:00 ` Christopher Larson
@ 2016-11-18 3:31 ` Andre McCurdy
0 siblings, 0 replies; 6+ messages in thread
From: Andre McCurdy @ 2016-11-18 3:31 UTC (permalink / raw)
To: Christopher Larson; +Cc: OE Core mailing list
On Thu, Nov 17, 2016 at 2:00 PM, Christopher Larson <clarson@kergoth.com> wrote:
>
> On Thu, Nov 17, 2016 at 2:28 PM, Andre McCurdy <armccurdy@gmail.com> wrote:
>>
>> I have a supplier who provides recipes which set SRC_URI to their
>> private git servers. To make those same recipes usable by others (ie
>> me), some anonymous python is used to transform the default SRC_URI
>> (elements which contain private git URLs are replaced, patches and
>> other files are left as-is).
>>
>> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
>> python which modifies SRC_URI races with the anonymous python in
>> base.bbclass which parses SRC_URI to determine additional
>> do_fetch/do_unpack dependencies and whether or not to call
>> fetch2.get_srcrev(). Specifically I get failures because
>> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
>> AUTOREV from a repo to which I don't have access.
>
> You might want to try using a RecipePreFinalise event handler instead, they
> run before anonymous python functions do.
Thanks, that works.
(For the benefit of anyone else who may try copying and pasting the
example from the bitbake user manual, the correct eventmask is
"bb.event.RecipePreFinalise" rather than
"bb.build.RecipePreFinalise").
> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Modifying SRC_URI from anonymous python
2016-11-17 21:28 Modifying SRC_URI from anonymous python Andre McCurdy
2016-11-17 22:00 ` Christopher Larson
@ 2016-11-17 23:06 ` Richard Purdie
2016-11-18 0:59 ` Christopher Larson
2016-11-18 7:44 ` Patrick Ohly
2 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2016-11-17 23:06 UTC (permalink / raw)
To: Andre McCurdy, OE Core mailing list
On Thu, 2016-11-17 at 13:28 -0800, Andre McCurdy wrote:
> I have a supplier who provides recipes which set SRC_URI to their
> private git servers. To make those same recipes usable by others (ie
> me), some anonymous python is used to transform the default SRC_URI
> (elements which contain private git URLs are replaced, patches and
> other files are left as-is).
>
> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> python which modifies SRC_URI races with the anonymous python in
> base.bbclass which parses SRC_URI to determine additional
> do_fetch/do_unpack dependencies and whether or not to call
> fetch2.get_srcrev().
I suspect we made some changes around that time to ensure determinism
in the order certain things happened.
> Specifically I get failures because
> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> AUTOREV from a repo to which I don't have access.
>
> The proposed solution from the supplier is this patch to
> base.bbclass:
>
> @@ -598,7 +598,7 @@ python () {
> d.appendVarFlag('do_unpack', 'depends', '
> file-native:do_populate_sysroot')
>
> if needsrcrev:
> - d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> + d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}",
> parsing=True)
>
> set_packagetriplet(d)
parsing=True is a bitbake internal thing, its not documented and
shouldn't be used outside the bitbake codebase. In hindsight, naming
could have been better.
> After reading the setVar source I'm not very clear how or why this
> works, but it looks dubious. What is parsing=True intended to do?
Its internal bitbake data store stuff, don't use or rely on it. It
might happen to work here but its just luck.
> Is it documented somewhere that modifying SRC_URI from anonymous
> python isn't allowed? I've now seen two suppliers both independently
> run into the same problem when updating to OE 2.1.
We have an open bug related to anonymous python ordering. Its hard to
fix easily as something can indicate it wants to run last, but then
everything wants to run last. As Chris mentions, you can use an earlier
event handler as one way to work around it.
I'd also note that anonymous python runs in parsed order. Since
base.bbclass is "up front", its hard to run things before it though :/.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Modifying SRC_URI from anonymous python
2016-11-17 23:06 ` Richard Purdie
@ 2016-11-18 0:59 ` Christopher Larson
0 siblings, 0 replies; 6+ messages in thread
From: Christopher Larson @ 2016-11-18 0:59 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE Core mailing list
[-- Attachment #1: Type: text/plain, Size: 2716 bytes --]
On Thu, Nov 17, 2016 at 4:06 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2016-11-17 at 13:28 -0800, Andre McCurdy wrote:
> > I have a supplier who provides recipes which set SRC_URI to their
> > private git servers. To make those same recipes usable by others (ie
> > me), some anonymous python is used to transform the default SRC_URI
> > (elements which contain private git URLs are replaced, patches and
> > other files are left as-is).
> >
> > This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> > python which modifies SRC_URI races with the anonymous python in
> > base.bbclass which parses SRC_URI to determine additional
> > do_fetch/do_unpack dependencies and whether or not to call
> > fetch2.get_srcrev().
>
> I suspect we made some changes around that time to ensure determinism
> in the order certain things happened.
>
> > Specifically I get failures because
> > fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> > AUTOREV from a repo to which I don't have access.
> >
> > The proposed solution from the supplier is this patch to
> > base.bbclass:
> >
> > @@ -598,7 +598,7 @@ python () {
> > d.appendVarFlag('do_unpack', 'depends', '
> > file-native:do_populate_sysroot')
> >
> > if needsrcrev:
> > - d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> > + d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}",
> > parsing=True)
> >
> > set_packagetriplet(d)
>
>
> parsing=True is a bitbake internal thing, its not documented and
> shouldn't be used outside the bitbake codebase. In hindsight, naming
> could have been better.
>
> > After reading the setVar source I'm not very clear how or why this
> > works, but it looks dubious. What is parsing=True intended to do?
>
> Its internal bitbake data store stuff, don't use or rely on it. It
> might happen to work here but its just luck.
>
> > Is it documented somewhere that modifying SRC_URI from anonymous
> > python isn't allowed? I've now seen two suppliers both independently
> > run into the same problem when updating to OE 2.1.
>
> We have an open bug related to anonymous python ordering. Its hard to
> fix easily as something can indicate it wants to run last, but then
> everything wants to run last. As Chris mentions, you can use an earlier
> event handler as one way to work around it.
I think our best bet there is to move toward more explicit event handler
naming and dependency handling amongst event handlers.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 3531 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Modifying SRC_URI from anonymous python
2016-11-17 21:28 Modifying SRC_URI from anonymous python Andre McCurdy
2016-11-17 22:00 ` Christopher Larson
2016-11-17 23:06 ` Richard Purdie
@ 2016-11-18 7:44 ` Patrick Ohly
2 siblings, 0 replies; 6+ messages in thread
From: Patrick Ohly @ 2016-11-18 7:44 UTC (permalink / raw)
To: Andre McCurdy; +Cc: OE Core mailing list
On Thu, 2016-11-17 at 13:28 -0800, Andre McCurdy wrote:
> I have a supplier who provides recipes which set SRC_URI to their
> private git servers. To make those same recipes usable by others (ie
> me), some anonymous python is used to transform the default SRC_URI
> (elements which contain private git URLs are replaced, patches and
> other files are left as-is).
>
> This apparently worked in OE 2.0 but from 2.1 onwards the anonymous
> python which modifies SRC_URI races with the anonymous python in
> base.bbclass which parses SRC_URI to determine additional
> do_fetch/do_unpack dependencies and whether or not to call
> fetch2.get_srcrev().
I have a patch which I intend to submit to bitbake which adds priorities
for anonymous python methods:
https://github.com/pohly/poky/commit/9d959e50e5f27d149654f5db0aff2fe51365ad68
With that in place, the anonymous python method which transforms the
SRC_URI could request to run sooner than the base.bbclass anonymous
python.
However, I am not sure whether that should be backported to 2.1 and 2.2.
> Specifically I get failures because
> fetch2.get_srcrev() sees the original SRC_URI and tries to resolve
> AUTOREV from a repo to which I don't have access.
>
> The proposed solution from the supplier is this patch to base.bbclass:
>
> @@ -598,7 +598,7 @@ python () {
> d.appendVarFlag('do_unpack', 'depends', '
> file-native:do_populate_sysroot')
>
> if needsrcrev:
> - d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}")
> + d.setVar("SRCPV", "${@bb.fetch2.get_srcrev(d)}", parsing=True)
>
> set_packagetriplet(d)
>
> After reading the setVar source I'm not very clear how or why this
> works, but it looks dubious. What is parsing=True intended to do?
>
> Is it documented somewhere that modifying SRC_URI from anonymous
> python isn't allowed? I've now seen two suppliers both independently
> run into the same problem when updating to OE 2.1.
Sorry, I don't know about that.
--
Best Regards, Patrick Ohly
The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.
^ permalink raw reply [flat|nested] 6+ messages in thread