getVar. setVar, "d", etc.

getVar. setVar, "d", etc.

[Rich Pixley]

I'm trying to figure out how to set bitbake variables from within python
escapes but I'm not having much luck.

Could someone please explain the use of the "d" variable in the getVar
and setVar calls?  Or point me towards enlightening pieces of code or
documentation?

--rich

Re: getVar. setVar, "d", etc.

[Richard Purdie]

On Mon, 2008-02-04 at 14:05 -0800, Rich Pixley wrote:
> I'm trying to figure out how to set bitbake variables from within python
> escapes but I'm not having much luck.
> 
> Could someone please explain the use of the "d" variable in the getVar
> and setVar calls?  Or point me towards enlightening pieces of code or
> documentation?

d is the data object you want to set the variables within or get the
variable from.

Regards,

Richard

Re: getVar. setVar, "d", etc.

[Rich Pixley]

Richard Purdie wrote:
> On Mon, 2008-02-04 at 14:05 -0800, Rich Pixley wrote:
>   
>> I'm trying to figure out how to set bitbake variables from within python
>> escapes but I'm not having much luck.
>>
>> Could someone please explain the use of the "d" variable in the getVar
>> and setVar calls?  Or point me towards enlightening pieces of code or
>> documentation?
>>     
>
> d is the data object you want to set the variables within or get the
> variable from.
Ok, so where does it come from?

Eg, in db3_3.2.9.bb, in do_package(), there's a getVar and a setVar but 
it's not at all clear there "d" is assigned.  Is it just a conventional 
global?  What other conventional globals might one want to pass to 
getVar or setVar?

--rich

Re: getVar. setVar, "d", etc.

[Rich Pixley]

Let me try this again.

I have a class.  My class is listed in the INHERIT in my local.conf.

My class intends to set a variable, SUBMISSION_busybox, which will be 
used by a bb file as "PR = ${SUBMISSION_busybox}".

I'm setting the variable via setVar, and pretty printing "d" shows me 
that it's in p.dict.

However, attempting to bitbake yields:

NOTE: <type 'exceptions.SyntaxError'>:EOL while scanning single-quoted 
string (<string>, line 1) while evaluating:
${@base_set_filespath([ 
"/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2-${SUBMISSION_busybox}", 
"/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2", 
"/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox", 
"/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/files", 
"/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox" ], d)}
ERROR: Error in executing:
ERROR: Exception:<type 'exceptions.SyntaxError'> Message:EOL while 
scanning single-quoted string (<string>, line 1)
ERROR: Printing the environment of the function

...

  File "<string>", line 1
    base_set_filespath([ 
"/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2-${SUBMISSION_busybox

(The caret is under the last "o").

And I'm totally confused.  It would appear to me as though 
SUBMISSION_busybox isn't being found and thus isn't being expanded.  I'm 
assuming that my understanding of the various "d"'s is my conceptual 
weakness here.

Could someone please point out my obvious flaw?  Or point me toward 
relevant doc or code which might help clarify this for me?

--rich

Re: getVar. setVar, "d", etc.

[Richard Purdie]

On Mon, 2008-02-04 at 18:37 -0800, Rich Pixley wrote:
> Let me try this again.
> 
> I have a class.  My class is listed in the INHERIT in my local.conf.
> 
> My class intends to set a variable, SUBMISSION_busybox, which will be 
> used by a bb file as "PR = ${SUBMISSION_busybox}".
> 
> I'm setting the variable via setVar, and pretty printing "d" shows me 
> that it's in p.dict.
> 
> However, attempting to bitbake yields:
> 
> NOTE: <type 'exceptions.SyntaxError'>:EOL while scanning single-quoted 
> string (<string>, line 1) while evaluating:
> ${@base_set_filespath([ 
> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2-${SUBMISSION_busybox}", 
> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2", 
> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox", 
> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/files", 
> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox" ], d)}
> ERROR: Error in executing:
> ERROR: Exception:<type 'exceptions.SyntaxError'> Message:EOL while 
> scanning single-quoted string (<string>, line 1)
> ERROR: Printing the environment of the function

The problem is the point at which some variables are accessed. How,
where and when your class sets this variable SUBMISSION_busybox is very
important. 

In this case bitbake is trying to evaluate FILESPATH set in
base.bbclass. To do this it references the ${PF} variable which in turn
contains ${PR} which in turn contains ${SUBMISSION_busybox}. At this
point ${SUBMISSION_busybox} isn't set so it errors.

> And I'm totally confused.  It would appear to me as though 
> SUBMISSION_busybox isn't being found and thus isn't being expanded.  I'm 
> assuming that my understanding of the various "d"'s is my conceptual 
> weakness here.

At the point in time where bitbake is trying to evaluate FILESPATH which
indirectly includes SUBMISSION_busybox, SUBMISSION_busybox isn't in d.

Regards,

Richard

Re: getVar. setVar, "d", etc.

[Rich Pixley]

Richard Purdie wrote:
> On Mon, 2008-02-04 at 18:37 -0800, Rich Pixley wrote:
>   
>> Let me try this again.
>>
>> I have a class.  My class is listed in the INHERIT in my local.conf.
>>
>> My class intends to set a variable, SUBMISSION_busybox, which will be
>> used by a bb file as "PR = ${SUBMISSION_busybox}".
>>
>> I'm setting the variable via setVar, and pretty printing "d" shows me
>> that it's in p.dict.
>>
>> However, attempting to bitbake yields:
>>
>> NOTE: <type 'exceptions.SyntaxError'>:EOL while scanning single-quoted
>> string (<string>, line 1) while evaluating:
>> ${@base_set_filespath([
>> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2-${SUBMISSION_busybox}",
>> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox-1.7.2",
>> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/busybox",
>> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox/files",
>> "/home/rich/projects/nova/oe-bom/nova/oe/packages/busybox" ], d)}
>> ERROR: Error in executing:
>> ERROR: Exception:<type 'exceptions.SyntaxError'> Message:EOL while
>> scanning single-quoted string (<string>, line 1)
>> ERROR: Printing the environment of the function
>>     
> The problem is the point at which some variables are accessed. How,
> where and when your class sets this variable SUBMISSION_busybox is very
> important.
>
> In this case bitbake is trying to evaluate FILESPATH set in
> base.bbclass. To do this it references the ${PF} variable which in turn
> contains ${PR} which in turn contains ${SUBMISSION_busybox}. At this
> point ${SUBMISSION_busybox} isn't set so it errors
>> And I'm totally confused.  It would appear to me as though
>> SUBMISSION_busybox isn't being found and thus isn't being expanded.  I'm
>> assuming that my understanding of the various "d"'s is my conceptual
>> weakness here.
>>     
> At the point in time where bitbake is trying to evaluate FILESPATH which
> indirectly includes SUBMISSION_busybox, SUBMISSION_busybox isn't in d.
Thank you for your analysis.  That's about the conclusion that I'd come 
to as well.

However, with judicious use of print commands I can see that 
SUBMISSION_busybox is indeed set in d at that time, even in d which has 
"${PN}" set to "busybox" but apparently either that's not the right d, 
or it's not set soon enough, or something else it going on that I don't 
understand.

Using:

def set_submissions(d):
    import bb
    import pprint   
    bb.data.setVar('SUBMISSION_busybox', '%s' % 1, d)
    print "d.dict is"
    pp = pprint.PrettyPrinter(indent=4);
    pp.pprint(d.dict)
    print "\n"
    return "garbage"

garbage := "${@set_submissions(d)}"

Before the FILESPATH assignment in base.bbclass, I can see that 
SUBMISSION_busybox is being set, and it's being set in the current d, 
but it's still not being expanded.

Any hints?

--rich

Re: getVar. setVar, "d", etc.

[Rich Pixley]

When does d change?

I've got the following bb file which produces results which puzzle me.  
What I'm seeing is that the two values for d are different instances and 
that the second function doesn't see the variable which was set in the 
first because they're using different dictionaries.

--rich

def setting(d):

    import bb

    print "setting sets in ", d, "\n"

    bb.data.setVar('FOO', 'bar', d)

    return "setting"

X := ${@setting(d)}

def getting(d):

    import bb

    got = bb.data.getVar('FOO', d)

    print "getting gets %s from " % (got), d, "\n"

    return "getting"

Y := ${@getting(d)}

Re: getVar. setVar, "d", etc.

[Richard Purdie]

On Tue, 2008-02-05 at 18:29 -0800, Rich Pixley wrote:
> When does d change?
> 
> I've got the following bb file which produces results which puzzle me.  
> What I'm seeing is that the two values for d are different instances and 
> that the second function doesn't see the variable which was set in the 
> first because they're using different dictionaries.
> 
> --rich
> 
> def setting(d):
> 
>     import bb
> 
>     print "setting sets in ", d, "\n"
> 
>     bb.data.setVar('FOO', 'bar', d)
> 
>     return "setting"
> 
> X := ${@setting(d)}
> 
> def getting(d):
> 
>     import bb
> 
>     got = bb.data.getVar('FOO', d)
> 
>     print "getting gets %s from " % (got), d, "\n"
> 
>     return "getting"
> 
> Y := ${@getting(d)}

The problem is that you're doing this in ":=" context. The immediate
expansion operator means we have to expand the data overrides before
parsing the variables so overrides work as expected. This makes a copy
of the data object so the result of the setVar is lost inside the
setting function.

Using "X = ${@setting(d)}" instead should make the above example work if
I remember how this works correctly.

Once again I should remind people that := is usually a bad idea (see
recent discussions).

Regards,

Richard

Re: getVar. setVar, "d", etc.

[K. Richard Pixley]

Richard Purdie wrote:
> Once again I should remind people that := is usually a bad idea (see
> recent discussions).
That's counterintuitive and backwards from other build systems.  I've 
often pulled significant savings out of makefile based systems by 
switching from = to :=.  The problem being that "x = ${@ 
a_long_running_function() }" is a killer, especially when 
a_long_running_function is deterministic.  In that case, := requires 
exactly one call during assignment while = requires a fresh call for 
each and every reference.  The only down side of := assignments is that 
one needs to be savvy about sequencing assignments.  Luckily, with 
conditionals, any sequence of assignments can be expressed.

I used := in this situation specifically because of the sequence 
ordering.   The assignments in my example are dummies whose sole purpose 
is to force the side effects to happen at a predictable and early point 
in the run - specifically, before the assignment to SRC_URI.  If I don't 
force the side effects to happen at := time, then there's no reason to 
believe that they will have occurred prior to the expansion of ${PR} in 
${SRC_URI}.  (And, in fact, changing this doesn't affect my results).

I have discovered that converting the constructions into "python 
setting() {" seem to give them consistent contexts.  I don't understand 
why ${@ ... } lacks that context, but at least I have a way to proceed.  
Event handlers also seem to have consistent context so if I can reverse 
engineer what events actually exist, or force one to occur at the right 
time, then I should be able to get my assignments done early enough.

Neither these differences nor event handlers are very well described in 
the manual, btw.

Thanks for your help.  I think I'm over my hump now.  "python setting()" 
and event handlers should get me where I want to go now.

--rich

Re: getVar. setVar, "d", etc.

[Richard Purdie]

On Wed, 2008-02-06 at 10:54 -0800, K. Richard Pixley wrote:
> Richard Purdie wrote:
> > Once again I should remind people that := is usually a bad idea (see
> > recent discussions).
> That's counterintuitive and backwards from other build systems.  

Counterintuitive, yes, backwards, no. It does this differently since
bitbake works differently in general to other build systems. The
overrides syntax OE/bitbake uses is extremely powerful but has some
costs and this is one of them.

> I've often pulled significant savings out of makefile based systems by 
> switching from = to :=.  

This isn't a makefile based system.

> The problem being that "x = ${@ 
> a_long_running_function() }" is a killer, especially when 
> a_long_running_function is deterministic.  In that case, := requires 
> exactly one call during assignment while = requires a fresh call for 
> each and every reference. 

You need to think carefully about when bitbake will execute these
expressions. If a variable is never used, the code in it will never be
run. Most variables are not expanded and hence any functions are not run
during initial parsing. They may run when the recipe is executed but any
time taken at this point isn't usually a problem.

> I used := in this situation specifically because of the sequence 
> ordering.   The assignments in my example are dummies whose sole purpose 
> is to force the side effects to happen at a predictable and early point 
> in the run - specifically, before the assignment to SRC_URI.  If I don't 
> force the side effects to happen at := time, then there's no reason to 
> believe that they will have occurred prior to the expansion of ${PR} in 
> ${SRC_URI}.  (And, in fact, changing this doesn't affect my results).
> 
> I have discovered that converting the constructions into "python 
> setting() {" seem to give them consistent contexts.  I don't understand 
> why ${@ ... } lacks that context, but at least I have a way to proceed. 

The "anonymous" functions are run after the file is parsed but before
the data is used.

${@...} runs when the variable is expanded.

> Event handlers also seem to have consistent context so if I can reverse 
> engineer what events actually exist, or force one to occur at the right 
> time, then I should be able to get my assignments done early enough.
> 
> Neither these differences nor event handlers are very well described in 
> the manual, btw.

Patches welcome!

Regards,

Richard