python function used in variable expansion called multiple times per recipe

python function used in variable expansion called multiple times per recipe

[Daniel Lazzari]

Hi everyone,

I've been wrestling with this problem for over a day now so I figured it's time to call in the cavalry.
I have a class in our layer that needs to fill in a variable (EXTERNPV) with information about svn externals in a project. This process can be very time consuming and resource intensive so I would prefer that it only happen once per run. Unfortunately, it seems to run 5 times for each recipe that inherits the class. You can easily duplicate the behavior  I see with the following:

def testFunc(bb, d):
	print( bb.data.getVar("PN", d, True) )
	return "e1"

EXTERNPV = "${@testFunc(bb, d)}"

Am I doing something wrong? Is there some assignment operator I can use to ensure this only has to be done once? Or maybe a variable flag?

As always, I very much appreciate the help.

Dan Lazzari Jr.
Firmware Engineer
dlazzari@leapfrog.com

Re: python function used in variable expansion called multiple times per recipe

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

${@} contents are evaluated when the variable is expanded. Variables are expanded when they're used, much in the way they are in makefiles. Use := to force immediate expansion, or don't use ${@} (alternatives, anonymous python function or event handler).

-- 
Christopher Larson


On Wednesday, April 18, 2012 at 2:21 PM, Daniel Lazzari wrote:

> Hi everyone,
> 
> I've been wrestling with this problem for over a day now so I figured it's time to call in the cavalry.
> I have a class in our layer that needs to fill in a variable (EXTERNPV) with information about svn externals in a project. This process can be very time consuming and resource intensive so I would prefer that it only happen once per run. Unfortunately, it seems to run 5 times for each recipe that inherits the class. You can easily duplicate the behavior I see with the following:
> 
> def testFunc(bb, d):
> print( bb.data.getVar("PN", d, True) )
> return "e1"
> 
> EXTERNPV = "${@testFunc(bb, d)}"
> 
> Am I doing something wrong? Is there some assignment operator I can use to ensure this only has to be done once? Or maybe a variable flag?
> 
> As always, I very much appreciate the help.
> 
> Dan Lazzari Jr.
> Firmware Engineer
> dlazzari@leapfrog.com (mailto:dlazzari@leapfrog.com)
> 
> 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org (mailto:bitbake-devel@lists.openembedded.org)
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
> 
> 



[-- Attachment #2: Type: text/html, Size: 2445 bytes --]

Re: python function used in variable expansion called multiple times per recipe

[Daniel Lazzari]

[-- Attachment #1: Type: text/plain, Size: 2268 bytes --]

I think immediate expansion is what I’ll need to use, thanks for the tip. That at least brings the invocation down to once per run. Unfortunately, the variable doesn’t seem to get filled in when I change to immediate expansion.

def testFunc(bb, d):
                print( bb.data.getVar("PN", d, True) )
                return "e1"

EXTERNPV := "${@testFunc(bb, d)}"

Now only prints the package name once, but using –e, I don’t see the variable EXTERNPV at all.

From: Christopher Larson [mailto:kergoth@gmail.com]
Sent: Wednesday, April 18, 2012 4:07 PM
To: Daniel Lazzari
Cc: 'bitbake-devel@lists.openembedded.org' (bitbake-devel@lists.openembedded.org)
Subject: Re: [bitbake-devel] python function used in variable expansion called multiple times per recipe

${@} contents are evaluated when the variable is expanded. Variables are expanded when they're used, much in the way they are in makefiles. Use := to force immediate expansion, or don't use ${@} (alternatives, anonymous python function or event handler).

--
Christopher Larson


On Wednesday, April 18, 2012 at 2:21 PM, Daniel Lazzari wrote:
Hi everyone,

I've been wrestling with this problem for over a day now so I figured it's time to call in the cavalry.
I have a class in our layer that needs to fill in a variable (EXTERNPV) with information about svn externals in a project. This process can be very time consuming and resource intensive so I would prefer that it only happen once per run. Unfortunately, it seems to run 5 times for each recipe that inherits the class. You can easily duplicate the behavior I see with the following:

def testFunc(bb, d):
print( bb.data.getVar("PN", d, True) )
return "e1"

EXTERNPV = "${@testFunc(bb, d)}"

Am I doing something wrong? Is there some assignment operator I can use to ensure this only has to be done once? Or maybe a variable flag?

As always, I very much appreciate the help.

Dan Lazzari Jr.
Firmware Engineer
dlazzari@leapfrog.com<mailto:dlazzari@leapfrog.com>


_______________________________________________
bitbake-devel mailing list
bitbake-devel@lists.openembedded.org<mailto:bitbake-devel@lists.openembedded.org>
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel


[-- Attachment #2: Type: text/html, Size: 8443 bytes --]

Re: python function used in variable expansion called multiple times per recipe

[Chris Larson]

On Wed, Apr 18, 2012 at 4:39 PM, Daniel Lazzari <dlazzari@leapfrog.com> wrote:
> I think immediate expansion is what I’ll need to use, thanks for the tip.
> That at least brings the invocation down to once per run. Unfortunately, the
> variable doesn’t seem to get filled in when I change to immediate expansion.
>
> def testFunc(bb, d):
>                 print( bb.data.getVar("PN", d, True) )
>                 return "e1"
>
> EXTERNPV := "${@testFunc(bb, d)}"
>
> Now only prints the package name once, but using –e, I don’t see the
> variable EXTERNPV at all.

Don't know what to tell you. It does work, so it's something wrong on
your end. Either it's returning an empty string, which will result in
-e only showing the commented out definition, or it's raising an
exception, or something else in your setup. Aside: you don't need to
pass 'bb' in. That namespace is always accessible, even from def'd
python functions.
-- 
Christopher Larson

Re: python function used in variable expansion called multiple times per recipe

[Daniel Lazzari]

>-----Original Message-----
>From: Chris Larson [mailto:kergoth@gmail.com]
>Sent: Wednesday, April 18, 2012 4:42 PM
>To: Daniel Lazzari
>Cc: 'bitbake-devel@lists.openembedded.org' (bitbake-
>devel@lists.openembedded.org)
>Subject: Re: [bitbake-devel] python function used in variable expansion called
>multiple times per recipe
>
>On Wed, Apr 18, 2012 at 4:39 PM, Daniel Lazzari <dlazzari@leapfrog.com>
>wrote:
>> I think immediate expansion is what I’ll need to use, thanks for the tip.
>> That at least brings the invocation down to once per run. Unfortunately, the
>> variable doesn’t seem to get filled in when I change to immediate
>expansion.
>>
>> def testFunc(bb, d):
>>                 print( bb.data.getVar("PN", d, True) )
>>                 return "e1"
>>
>> EXTERNPV := "${@testFunc(bb, d)}"
>>
>> Now only prints the package name once, but using –e, I don’t see the
>> variable EXTERNPV at all.
>
>Don't know what to tell you. It does work, so it's something wrong on
>your end. Either it's returning an empty string, which will result in
>-e only showing the commented out definition, or it's raising an
>exception, or something else in your setup. Aside: you don't need to
>pass 'bb' in. That namespace is always accessible, even from def'd
>python functions.
>--
>Christopher Larson

Ah, I was doing -e on the wrong recipe... time to get some more caffeine. The test case is now working. My code is not, but I can debug that pretty easily. Thanks for all of your help!

Re: python function used in variable expansion called multiple times per recipe

[Daniel Lazzari]

>> I think immediate expansion is what I’ll need to use, thanks for the tip.
>> That at least brings the invocation down to once per run. Unfortunately, the
>> variable doesn’t seem to get filled in when I change to immediate
>expansion.
>>
>> def testFunc(bb, d):
>>                 print( bb.data.getVar("PN", d, True) )
>>                 return "e1"
>>
>> EXTERNPV := "${@testFunc(bb, d)}"
>>
>> Now only prints the package name once, but using –e, I don’t see the
>> variable EXTERNPV at all.
>
>Don't know what to tell you. It does work, so it's something wrong on
>your end. Either it's returning an empty string, which will result in
>-e only showing the commented out definition, or it's raising an
>exception, or something else in your setup. Aside: you don't need to
>pass 'bb' in. That namespace is always accessible, even from def'd
>python functions.
>--
>Christopher Larson

Just thought I'd provide an update real quick on this. I discovered, to my dismay, that even doing an immediate expansion still ran the code before every task for that recipe. I switched to using an event handler, but the RecipeParsed event occurs before every task for that recipe. Eventually I discovered that the variable BB_WORKERCONTEXT is set for RecipeParsed events when they are occurring before a task, but not for the initial parse. Also, I discovered that setting a variable in the event handler would not necessarily keep it set through building the recipe. Thus I devised the below code to run the time intensive code during initial parse and cache the value in a file so it can be picked up again in later events. I'm leaving this here in case it helps someone else.

addhandler svn_extern_eventhandler
python svn_extern_eventhandler() {
	from bb.event import getName
	from bb import data
	
	if getName(e) == "RecipeParsed":
		extern_cache_dir = os.path.join( data.getVar('TMPDIR', e.data, True),
		                             'stamps',
		                             data.getVar('MULTIMACH_TARGET_SYS', e.data, True) )
		extern_cache = os.path.join( extern_cache_dir,
		                             data.getVar('PN', e.data, True) + '.extern_cache' )
		#If BB_WORKERCONTEXT is "1", this is NOT the initial recipe parse
		if data.getVar('BB_WORKERCONTEXT', e.data, True) == "1":
			#Pull cached copy
			try:
				cache_file = open(extern_cache, 'r')
				externpv = cache_file.read()
				cache_file.close()
			except IOError:
				#Failed to get cached copy, do it the hard way
				externpv = getExternRevs(e.data)
		else:
			externpv = getExternRevs(e.data)
			try:
				os.makedirs(extern_cache_dir)
			except OSError:
				pass
			#Generate the cache for later use
			cache_file = open(extern_cache, 'w')
			cache_file.write(externpv)
			cache_file.close()

		data.setVar('EXTERNPV', externpv, e.data)
}

Re: python function used in variable expansion called multiple times per recipe

[Richard Purdie]

On Thu, 2012-04-19 at 21:24 +0000, Daniel Lazzari wrote:
> >> I think immediate expansion is what I’ll need to use, thanks for the tip.
> >> That at least brings the invocation down to once per run. Unfortunately, the
> >> variable doesn’t seem to get filled in when I change to immediate
> >expansion.
> >>
> >> def testFunc(bb, d):
> >>                 print( bb.data.getVar("PN", d, True) )
> >>                 return "e1"
> >>
> >> EXTERNPV := "${@testFunc(bb, d)}"
> >>
> >> Now only prints the package name once, but using –e, I don’t see the
> >> variable EXTERNPV at all.
> >
> >Don't know what to tell you. It does work, so it's something wrong on
> >your end. Either it's returning an empty string, which will result in
> >-e only showing the commented out definition, or it's raising an
> >exception, or something else in your setup. Aside: you don't need to
> >pass 'bb' in. That namespace is always accessible, even from def'd
> >python functions.
> >--
> >Christopher Larson
> 
> Just thought I'd provide an update real quick on this. I discovered,
> to my dismay, that even doing an immediate expansion still ran the
> code before every task for that recipe. I switched to using an event
> handler, but the RecipeParsed event occurs before every task for that
> recipe. Eventually I discovered that the variable BB_WORKERCONTEXT is
> set for RecipeParsed events when they are occurring before a task, but
> not for the initial parse. Also, I discovered that setting a variable
> in the event handler would not necessarily keep it set through
> building the recipe. Thus I devised the below code to run the time
> intensive code during initial parse and cache the value in a file so
> it can be picked up again in later events. I'm leaving this here in
> case it helps someone else.

Just as a note, we have the persist_data code in bitbake which was
written for this kind of reason in relation to SRCREV and AUTOINC which
uses a sqlite backend as the storage. There isn't anything wrong with
what you're doing, I just thought I'd mention it.

Cheers,

Richard

Re: python function used in variable expansion called multiple times per recipe

[Daniel Lazzari]

>On Thu, 2012-04-19 at 21:24 +0000, Daniel Lazzari wrote:
>> >> I think immediate expansion is what I’ll need to use, thanks for the tip.
>> >> That at least brings the invocation down to once per run. Unfortunately,
>the
>> >> variable doesn’t seem to get filled in when I change to immediate
>> >expansion.
>> >>
>> >> def testFunc(bb, d):
>> >>                 print( bb.data.getVar("PN", d, True) )
>> >>                 return "e1"
>> >>
>> >> EXTERNPV := "${@testFunc(bb, d)}"
>> >>
>> >> Now only prints the package name once, but using –e, I don’t see the
>> >> variable EXTERNPV at all.
>> >
>> >Don't know what to tell you. It does work, so it's something wrong on
>> >your end. Either it's returning an empty string, which will result in
>> >-e only showing the commented out definition, or it's raising an
>> >exception, or something else in your setup. Aside: you don't need to
>> >pass 'bb' in. That namespace is always accessible, even from def'd
>> >python functions.
>> >--
>> >Christopher Larson
>>
>> Just thought I'd provide an update real quick on this. I discovered,
>> to my dismay, that even doing an immediate expansion still ran the
>> code before every task for that recipe. I switched to using an event
>> handler, but the RecipeParsed event occurs before every task for that
>> recipe. Eventually I discovered that the variable BB_WORKERCONTEXT is
>> set for RecipeParsed events when they are occurring before a task, but
>> not for the initial parse. Also, I discovered that setting a variable
>> in the event handler would not necessarily keep it set through
>> building the recipe. Thus I devised the below code to run the time
>> intensive code during initial parse and cache the value in a file so
>> it can be picked up again in later events. I'm leaving this here in
>> case it helps someone else.
>
>Just as a note, we have the persist_data code in bitbake which was
>written for this kind of reason in relation to SRCREV and AUTOINC which
>uses a sqlite backend as the storage. There isn't anything wrong with
>what you're doing, I just thought I'd mention it.
>
>Cheers,
>
>Richard

That sounds like a better alternative, but I have no idea how to implement it. Is there documentation for it? I'm not even sure how to get at the persist_data from the event handler. Is there a way to access it using d or something?

Re: python function used in variable expansion called multiple times per recipe

[McClintock Matthew-B29882]

On Thu, Apr 19, 2012 at 5:28 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2012-04-19 at 21:24 +0000, Daniel Lazzari wrote:
>> >> I think immediate expansion is what I’ll need to use, thanks for the tip.
>> >> That at least brings the invocation down to once per run. Unfortunately, the
>> >> variable doesn’t seem to get filled in when I change to immediate
>> >expansion.
>> >>
>> >> def testFunc(bb, d):
>> >>                 print( bb.data.getVar("PN", d, True) )
>> >>                 return "e1"
>> >>
>> >> EXTERNPV := "${@testFunc(bb, d)}"
>> >>
>> >> Now only prints the package name once, but using –e, I don’t see the
>> >> variable EXTERNPV at all.
>> >
>> >Don't know what to tell you. It does work, so it's something wrong on
>> >your end. Either it's returning an empty string, which will result in
>> >-e only showing the commented out definition, or it's raising an
>> >exception, or something else in your setup. Aside: you don't need to
>> >pass 'bb' in. That namespace is always accessible, even from def'd
>> >python functions.
>> >--
>> >Christopher Larson
>>
>> Just thought I'd provide an update real quick on this. I discovered,
>> to my dismay, that even doing an immediate expansion still ran the
>> code before every task for that recipe. I switched to using an event
>> handler, but the RecipeParsed event occurs before every task for that
>> recipe. Eventually I discovered that the variable BB_WORKERCONTEXT is
>> set for RecipeParsed events when they are occurring before a task, but
>> not for the initial parse. Also, I discovered that setting a variable
>> in the event handler would not necessarily keep it set through
>> building the recipe. Thus I devised the below code to run the time
>> intensive code during initial parse and cache the value in a file so
>> it can be picked up again in later events. I'm leaving this here in
>> case it helps someone else.
>
> Just as a note, we have the persist_data code in bitbake which was
> written for this kind of reason in relation to SRCREV and AUTOINC which
> uses a sqlite backend as the storage. There isn't anything wrong with
> what you're doing, I just thought I'd mention it.

Richard,

I'm curious if something like an eventhandler is more suitable for me
trying to modify the PV/PR? Seems like eventhandler + BB_WORKERCONTEXT
could be what I'm looking for?

-M

Re: python function used in variable expansion called multiple times per recipe

[Chris Larson]

On Thu, Apr 19, 2012 at 4:13 PM, Daniel Lazzari <dlazzari@leapfrog.com> wrote:
>
>
>>On Thu, 2012-04-19 at 21:24 +0000, Daniel Lazzari wrote:
>>> >> I think immediate expansion is what I’ll need to use, thanks for the tip.
>>> >> That at least brings the invocation down to once per run. Unfortunately,
>>the
>>> >> variable doesn’t seem to get filled in when I change to immediate
>>> >expansion.
>>> >>
>>> >> def testFunc(bb, d):
>>> >>                 print( bb.data.getVar("PN", d, True) )
>>> >>                 return "e1"
>>> >>
>>> >> EXTERNPV := "${@testFunc(bb, d)}"
>>> >>
>>> >> Now only prints the package name once, but using –e, I don’t see the
>>> >> variable EXTERNPV at all.
>>> >
>>> >Don't know what to tell you. It does work, so it's something wrong on
>>> >your end. Either it's returning an empty string, which will result in
>>> >-e only showing the commented out definition, or it's raising an
>>> >exception, or something else in your setup. Aside: you don't need to
>>> >pass 'bb' in. That namespace is always accessible, even from def'd
>>> >python functions.
>>> >--
>>> >Christopher Larson
>>>
>>> Just thought I'd provide an update real quick on this. I discovered,
>>> to my dismay, that even doing an immediate expansion still ran the
>>> code before every task for that recipe. I switched to using an event
>>> handler, but the RecipeParsed event occurs before every task for that
>>> recipe. Eventually I discovered that the variable BB_WORKERCONTEXT is
>>> set for RecipeParsed events when they are occurring before a task, but
>>> not for the initial parse. Also, I discovered that setting a variable
>>> in the event handler would not necessarily keep it set through
>>> building the recipe. Thus I devised the below code to run the time
>>> intensive code during initial parse and cache the value in a file so
>>> it can be picked up again in later events. I'm leaving this here in
>>> case it helps someone else.
>>
>>Just as a note, we have the persist_data code in bitbake which was
>>written for this kind of reason in relation to SRCREV and AUTOINC which
>>uses a sqlite backend as the storage. There isn't anything wrong with
>>what you're doing, I just thought I'd mention it.
>>
>>Cheers,
>>
>>Richard
>
> That sounds like a better alternative, but I have no idea how to implement it. Is there documentation for it? I'm not even sure how to get at the persist_data from the event handler. Is there a way to access it using d or something?

The bb.persist_data python module just provides a convenience
dictionary wrapper around an sqlite3 database.

persist = bb.persist_data.persist('some_identifier', d)
persist['foo'] = 'bar'
-- 
Christopher Larson

Re: python function used in variable expansion called multiple times per recipe

[Daniel Lazzari]

>
>On Thu, Apr 19, 2012 at 4:13 PM, Daniel Lazzari <dlazzari@leapfrog.com>
>wrote:
>>
>>
>>>On Thu, 2012-04-19 at 21:24 +0000, Daniel Lazzari wrote:
>>>> >> I think immediate expansion is what I’ll need to use, thanks for the tip.
>>>> >> That at least brings the invocation down to once per run.
>Unfortunately,
>>>the
>>>> >> variable doesn’t seem to get filled in when I change to immediate
>>>> >expansion.
>>>> >>
>>>> >> def testFunc(bb, d):
>>>> >>                 print( bb.data.getVar("PN", d, True) )
>>>> >>                 return "e1"
>>>> >>
>>>> >> EXTERNPV := "${@testFunc(bb, d)}"
>>>> >>
>>>> >> Now only prints the package name once, but using –e, I don’t see the
>>>> >> variable EXTERNPV at all.
>>>> >
>>>> >Don't know what to tell you. It does work, so it's something wrong on
>>>> >your end. Either it's returning an empty string, which will result in
>>>> >-e only showing the commented out definition, or it's raising an
>>>> >exception, or something else in your setup. Aside: you don't need to
>>>> >pass 'bb' in. That namespace is always accessible, even from def'd
>>>> >python functions.
>>>> >--
>>>> >Christopher Larson
>>>>
>>>> Just thought I'd provide an update real quick on this. I discovered,
>>>> to my dismay, that even doing an immediate expansion still ran the
>>>> code before every task for that recipe. I switched to using an event
>>>> handler, but the RecipeParsed event occurs before every task for that
>>>> recipe. Eventually I discovered that the variable BB_WORKERCONTEXT is
>>>> set for RecipeParsed events when they are occurring before a task, but
>>>> not for the initial parse. Also, I discovered that setting a variable
>>>> in the event handler would not necessarily keep it set through
>>>> building the recipe. Thus I devised the below code to run the time
>>>> intensive code during initial parse and cache the value in a file so
>>>> it can be picked up again in later events. I'm leaving this here in
>>>> case it helps someone else.
>>>
>>>Just as a note, we have the persist_data code in bitbake which was
>>>written for this kind of reason in relation to SRCREV and AUTOINC which
>>>uses a sqlite backend as the storage. There isn't anything wrong with
>>>what you're doing, I just thought I'd mention it.
>>>
>>>Cheers,
>>>
>>>Richard
>>
>> That sounds like a better alternative, but I have no idea how to implement
>it. Is there documentation for it? I'm not even sure how to get at the
>persist_data from the event handler. Is there a way to access it using d or
>something?
>
>The bb.persist_data python module just provides a convenience
>dictionary wrapper around an sqlite3 database.
>
>persist = bb.persist_data.persist('some_identifier', d)
>persist['foo'] = 'bar'
>--
>Christopher Larson

That worked wonderfully! Thanks to both of you for all of your help.