writeup on the mechanics of FILESEXTRAPATHS and bbappend files

writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Robert P. J. Day]

  a while back, i dug into how FILESEXTRAPATHS worked with respect to
.bbappend files and i wrote it up, just updated it a bit here:

http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files

do i have it about right? it seemed that the underlying details
weren't really explained well in any of the current yocto docs, and i
wanted something i could point students at.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Paul Eggleton]

On Wednesday 17 April 2013 10:24:42 Robert P. J. Day wrote:
>   a while back, i dug into how FILESEXTRAPATHS worked with respect to
> .bbappend files and i wrote it up, just updated it a bit here:
> 
> http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files
> 
> do i have it about right? it seemed that the underlying details
> weren't really explained well in any of the current yocto docs, and i
> wanted something i could point students at.

It looks OK to me although if you're going to point students at it I might 
suggest moving the "don't look here!" stuff elsewhere ;)

Also as I believe I mentioned before, FILESEXTRAPATHS adds not just one 
directory but an "entire list of directories" is it allows bbappend file path 
behaviour to match that of the recipe itself.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Robert P. J. Day]

On Wed, 17 Apr 2013, Paul Eggleton wrote:

> On Wednesday 17 April 2013 10:24:42 Robert P. J. Day wrote:
> >   a while back, i dug into how FILESEXTRAPATHS worked with respect to
> > .bbappend files and i wrote it up, just updated it a bit here:
> >
> > http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files
> >
> > do i have it about right? it seemed that the underlying details
> > weren't really explained well in any of the current yocto docs, and i
> > wanted something i could point students at.
>
> It looks OK to me although if you're going to point students at it I
> might suggest moving the "don't look here!" stuff elsewhere ;)

  yes, that's going to be updated with more examples ripped from the
headlines. :-)

> Also as I believe I mentioned before, FILESEXTRAPATHS adds not just
> one directory but an "entire list of directories" is it allows
> bbappend file path behaviour to match that of the recipe itself.

  quite so. the reason i did that writeup is that, as i mentioned, i
don't think the behaviour of FILESEXTRAPATHS is well explained. for
example, the entry in the reference manual variable glossary here:

http://www.yoctoproject.org/docs/1.2/poky-ref-manual/poky-ref-manual.html#var-FILESEXTRAPATHS

doesn't even *hint* at the underlying behaviour, which i think is
important to understand.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Rifenbark, Scott M]

Just a note here regarding which Yocto Project manuals to look at for the most current information.  Use the 1.4 manuals.  So for the FILESEXTRAPATHS variable description you should look at http://www.yoctoproject.org/docs/1.4/ref-manual/ref-manual.html#var-FILESEXTRAPATHS entry.  The description in this more recent manual may be lacking as well.  I will try and work on that going forward.

Also, the "poky-ref-manual" part of the name has been changed to a symlink for releases 1.3 forward.  This manual path for 1.3 on should direct the user to the manual whose location includes "ref-manual".

Scott

>-----Original Message-----
>From: yocto-bounces@yoctoproject.org [mailto:yocto-
>bounces@yoctoproject.org] On Behalf Of Robert P. J. Day
>Sent: Wednesday, April 17, 2013 8:12 AM
>To: Paul Eggleton
>Cc: yocto@yoctoproject.org
>Subject: Re: [yocto] writeup on the mechanics of FILESEXTRAPATHS and
>bbappend files
>
>On Wed, 17 Apr 2013, Paul Eggleton wrote:
>
>> On Wednesday 17 April 2013 10:24:42 Robert P. J. Day wrote:
>> >   a while back, i dug into how FILESEXTRAPATHS worked with respect
>to
>> > .bbappend files and i wrote it up, just updated it a bit here:
>> >
>> >
>http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_fi
>les
>> >
>> > do i have it about right? it seemed that the underlying details
>> > weren't really explained well in any of the current yocto docs, and
>i
>> > wanted something i could point students at.
>>
>> It looks OK to me although if you're going to point students at it I
>> might suggest moving the "don't look here!" stuff elsewhere ;)
>
>  yes, that's going to be updated with more examples ripped from the
>headlines. :-)
>
>> Also as I believe I mentioned before, FILESEXTRAPATHS adds not just
>> one directory but an "entire list of directories" is it allows
>> bbappend file path behaviour to match that of the recipe itself.
>
>  quite so. the reason i did that writeup is that, as i mentioned, i
>don't think the behaviour of FILESEXTRAPATHS is well explained. for
>example, the entry in the reference manual variable glossary here:
>
>http://www.yoctoproject.org/docs/1.2/poky-ref-manual/poky-ref-
>manual.html#var-FILESEXTRAPATHS
>
>doesn't even *hint* at the underlying behaviour, which i think is
>important to understand.
>
>rday
>
>--
>
>========================================================================
>Robert P. J. Day                                 Ottawa, Ontario, CANADA
>                        http://crashcourse.ca
>
>Twitter:                                       http://twitter.com/rpjday
>LinkedIn:                               http://ca.linkedin.com/in/rpjday
>========================================================================
>_______________________________________________
>yocto mailing list
>yocto@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/yocto

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Robert P. J. Day]

On Wed, 17 Apr 2013, Rifenbark, Scott M wrote:

> Just a note here regarding which Yocto Project manuals to look at
> for the most current information.  Use the 1.4 manuals.  So for the
> FILESEXTRAPATHS variable description you should look at
> http://www.yoctoproject.org/docs/1.4/ref-manual/ref-manual.html#var-FILESEXTRAPATHS
> entry.  The description in this more recent manual may be lacking as
> well.  I will try and work on that going forward.

  i use the "in progress" version of all of the manuals, so i still
think that glossary entry is a bit skimpy on details. on the other
hand, glossary entries are *normally* fairly brief, so a full
explanation of something like that probably belongs elsewhere, anyway.

rday

p.s.  i think one of the manuals should mention chris larson's
"bitbake-env" utility, which i use for debugging and explanations all
the time.

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Paul Eggleton]

On Wednesday 17 April 2013 11:39:09 Robert P. J. Day wrote:
> p.s.  i think one of the manuals should mention chris larson's
> "bitbake-env" utility, which i use for debugging and explanations all
> the time.

I'd rather wait until we actually integrate that into BitBake (i.e. provide it 
with the BitBake source, which I hope we can look at within the next 
development cycle).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[William M.A. Traynor]

On Wed, Apr 17, 2013 at 11:42 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Wednesday 17 April 2013 11:39:09 Robert P. J. Day wrote:
>> p.s.  i think one of the manuals should mention chris larson's
>> "bitbake-env" utility, which i use for debugging and explanations all
>> the time.
>
> I'd rather wait until we actually integrate that into BitBake (i.e. provide it
> with the BitBake source, which I hope we can look at within the next
> development cycle).

Is it as simple as including it in the contrib directory?

>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Gary Thomas]

On 2013-04-17 09:03, Paul Eggleton wrote:
> On Wednesday 17 April 2013 10:24:42 Robert P. J. Day wrote:
>>    a while back, i dug into how FILESEXTRAPATHS worked with respect to
>> .bbappend files and i wrote it up, just updated it a bit here:
>>
>> http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files
>>
>> do i have it about right? it seemed that the underlying details
>> weren't really explained well in any of the current yocto docs, and i
>> wanted something i could point students at.
>
> It looks OK to me although if you're going to point students at it I might
> suggest moving the "don't look here!" stuff elsewhere ;)
>
> Also as I believe I mentioned before, FILESEXTRAPATHS adds not just one
> directory but an "entire list of directories" is it allows bbappend file path
> behaviour to match that of the recipe itself.

I also think it would be good to point out the use of := in the FILESEXTRAPATHS
assignment and why it's important.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Robert P. J. Day]

On Wed, 17 Apr 2013, Gary Thomas wrote:

> On 2013-04-17 09:03, Paul Eggleton wrote:
> > On Wednesday 17 April 2013 10:24:42 Robert P. J. Day wrote:
> > >    a while back, i dug into how FILESEXTRAPATHS worked with respect to
> > > .bbappend files and i wrote it up, just updated it a bit here:
> > >
> > > http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files
> > >
> > > do i have it about right? it seemed that the underlying details
> > > weren't really explained well in any of the current yocto docs, and i
> > > wanted something i could point students at.
> >
> > It looks OK to me although if you're going to point students at it I might
> > suggest moving the "don't look here!" stuff elsewhere ;)
> >
> > Also as I believe I mentioned before, FILESEXTRAPATHS adds not just one
> > directory but an "entire list of directories" is it allows bbappend file
> > path
> > behaviour to match that of the recipe itself.
>
> I also think it would be good to point out the use of := in the
> FILESEXTRAPATHS assignment and why it's important.

  done:

http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files#Adding_the_meta-yocto_layer_and_building_for_a_BeagleBoard

i'm already unhappy with the way that whole page is written so i'll
redesign it this weekend, and maybe add a bit more. i just remember it
taking me a while to truly understand what was happening there so i
figured i'd better write it down before i forgot.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Gary Thomas]

On 2013-04-17 09:34, Robert P. J. Day wrote:
> On Wed, 17 Apr 2013, Gary Thomas wrote:
>
>> On 2013-04-17 09:03, Paul Eggleton wrote:
>>> On Wednesday 17 April 2013 10:24:42 Robert P. J. Day wrote:
>>>>     a while back, i dug into how FILESEXTRAPATHS worked with respect to
>>>> .bbappend files and i wrote it up, just updated it a bit here:
>>>>
>>>> http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files
>>>>
>>>> do i have it about right? it seemed that the underlying details
>>>> weren't really explained well in any of the current yocto docs, and i
>>>> wanted something i could point students at.
>>>
>>> It looks OK to me although if you're going to point students at it I might
>>> suggest moving the "don't look here!" stuff elsewhere ;)
>>>
>>> Also as I believe I mentioned before, FILESEXTRAPATHS adds not just one
>>> directory but an "entire list of directories" is it allows bbappend file
>>> path
>>> behaviour to match that of the recipe itself.
>>
>> I also think it would be good to point out the use of := in the
>> FILESEXTRAPATHS assignment and why it's important.
>
>    done:
>
> http://www.crashcourse.ca/wiki/index.php/FILESEXTRAPATHS_and_bbappend_files#Adding_the_meta-yocto_layer_and_building_for_a_BeagleBoard

Thanks.  You might add a comment about what happens if you do it wrong (I should know!)
that it will search in the directory that has the .bb file and not the .bbappend.

>
> i'm already unhappy with the way that whole page is written so i'll
> redesign it this weekend, and maybe add a bit more. i just remember it
> taking me a while to truly understand what was happening there so i
> figured i'd better write it down before i forgot.
>
> rday
>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Trevor Woerner]

On Wed, Apr 17, 2013 at 11:12 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> I also think it would be good to point out the use of := in the
> FILESEXTRAPATHS
> assignment and why it's important.

Knowing that the := operator will put your directories first is only
useful if you also know that the underlying mechanism will take the
first such files from the given directories it encounters :-) What's
to say bitbake doesn't look at all the directories/files and take the
last ones?

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Robert P. J. Day]

On Wed, 17 Apr 2013, Trevor Woerner wrote:

> On Wed, Apr 17, 2013 at 11:12 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> > I also think it would be good to point out the use of := in the
> > FILESEXTRAPATHS assignment and why it's important.
>
> Knowing that the := operator will put your directories first

  whoa, i'm not sure i'd phrase it *that* way but i know what you're
getting at.

> is only useful if you also know that the underlying mechanism will
> take the first such files from the given directories it encounters
> :-) What's to say bitbake doesn't look at all the directories/files
> and take the last ones?

  ok, so i should add that it's left-to-right parsing? fair enough.
although i suspect most people would just *assume* that but it's
always better to be explicit.

  at the moment, i'm more interested in egregious errors in my
explanation. again, i don't think FILESEXTRAPATHS is explained in
sufficient detail to prevent newbies from getting confused and into
real trouble.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Gary Thomas]

On 2013-04-17 16:26, Trevor Woerner wrote:
> On Wed, Apr 17, 2013 at 11:12 AM, Gary Thomas <gary@mlbassoc.com> wrote:
>> I also think it would be good to point out the use of := in the
>> FILESEXTRAPATHS
>> assignment and why it's important.
>
> Knowing that the := operator will put your directories first is only
> useful if you also know that the underlying mechanism will take the
> first such files from the given directories it encounters :-) What's
> to say bitbake doesn't look at all the directories/files and take the
> last ones?

This has more to do with how the path is constructed (using _prepend).

My point, which was totally unclear to me when using this line, was
that with ':=' ${THISDIR} evaluates to the directory that contains
the .bbappend file (what you want/need) whereas with '=' it evaluates
to the directory containing the main .bb file.  Incredibly confusing
if you miss it...

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Robert P. J. Day]

On Thu, 18 Apr 2013, Gary Thomas wrote:

> On 2013-04-17 16:26, Trevor Woerner wrote:
> > On Wed, Apr 17, 2013 at 11:12 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> > > I also think it would be good to point out the use of := in the
> > > FILESEXTRAPATHS
> > > assignment and why it's important.
> >
> > Knowing that the := operator will put your directories first is only
> > useful if you also know that the underlying mechanism will take the
> > first such files from the given directories it encounters :-) What's
> > to say bitbake doesn't look at all the directories/files and take the
> > last ones?
>
> This has more to do with how the path is constructed (using _prepend).
>
> My point, which was totally unclear to me when using this line, was
> that with ':=' ${THISDIR} evaluates to the directory that contains
> the .bbappend file (what you want/need) whereas with '=' it evaluates
> to the directory containing the main .bb file.  Incredibly confusing
> if you miss it...

  ah, quite so, must make a note of that. i wonder how many other
places that point should be made forcefully.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: writeup on the mechanics of FILESEXTRAPATHS and bbappend files

[Trevor Woerner]

Sorry if I'm asking a really stupid question, but if so, it means I'm
way off the mark :-)

On Thu, Apr 18, 2013 at 6:23 AM, Gary Thomas <gary@mlbassoc.com> wrote:
> My point, which was totally unclear to me when using this line, was
> that with ':=' ${THISDIR} evaluates to the directory that contains
> the .bbappend file (what you want/need) whereas with '=' it evaluates
> to the directory containing the main .bb file.  Incredibly confusing
> if you miss it...

So it's (sort of like) a matter of timing?

By evaluating THISDIR now (though the use of ':='), we get the
directory we are in when this directive is encountered. Whereas with
'=' it will be evaluated sometime in the future when we are no longer
in THISDIR but rather in SOMEOTHERDIR?