Examples of srctree and gitver

Examples of srctree and gitver

[Ambrose, Martin]

Hello.

Are there any recipes/examples which use these classes?

I grep'ed oe mainline/poky/arago/kergoth but no luck.

Regards,
Martin

Re: Examples of srctree and gitver

[Chris Larson]

On Sun, May 30, 2010 at 8:07 AM, Ambrose, Martin <martin@ti.com> wrote:

> Hello.
>
> Are there any recipes/examples which use these classes?
>
> I grep'ed oe mainline/poky/arago/kergoth but no luck.


I expect they'd be most useful in development, so no, I don't know of any
examples in an OE tree today.  It's pretty straightforward.  Add inherit
srctree, point S at your local source tree (or put the recipe inside the
sourcetree, it defaults to the dir the recipe is in), and it'll build from
that local tree rather than doing the fetch+unpack+patch process.  The only
real potential complexity there is the 'clean' task -- how you want to clean
your source tree may vary.  Iirc, the default will run a 'make clean', but
there's a function provided to use git to clean it.  gitver just sets GITVER
with 'git describe' information, to give you a version based on the current
contents of the local git tree, and it attempts to arrange things so it will
automatically be updated if you make any local commits to the tree.  Iirc
there's a hook provided to let you mangle the tags in the git repository
into the GITVER value -- an example of this would be stripping off the
leading 'v' in the tags in the linux kernel repository.

I'll see if I can find some example recipes for you, I know I had an example
repo set up when I was writing the classes.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Examples of srctree and gitver

[Ambrose, Martin]

On Sun, May 30, 2010 at 11:16:16, Chris Larson wrote:

> On Sun, May 30, 2010 at 8:07 AM, Ambrose, Martin <martin@ti.com> wrote:
...

> > Are there any recipes/examples which use these classes?
> 
> I expect they'd be most useful in development, so no, I don't know of any
> examples in an OE tree today.

Makes sense.

> It's pretty straightforward.  Add inherit
> srctree, point S at your local source tree (or put the recipe inside the
> sourcetree, it defaults to the dir the recipe is in), and it'll build from
> that local tree rather than doing the fetch+unpack+patch process.

Ack. I now think of the class as providing a 'skip' to the fetch+unpack+patch
steps but everything afterwards is then vanilla OE.

Ideally I could have a single set of recipes which, through some command line switches,
would either build in-tree using local sources or out-of-tree using a tarball.
But maybe this really requires separate recipes with commonality factored into includes.

Thanks for the explanation/help.

Regards, 
Martin

Re: Examples of srctree and gitver

[Chris Larson]

On Sun, May 30, 2010 at 12:26 PM, Ambrose, Martin <martin@ti.com> wrote:

> On Sun, May 30, 2010 at 11:16:16, Chris Larson wrote:
>
> > On Sun, May 30, 2010 at 8:07 AM, Ambrose, Martin <martin@ti.com> wrote:
> ...
>
> > > Are there any recipes/examples which use these classes?
> >
> > I expect they'd be most useful in development, so no, I don't know of any
> > examples in an OE tree today.
>
> Makes sense.
>
> > It's pretty straightforward.  Add inherit
> > srctree, point S at your local source tree (or put the recipe inside the
> > sourcetree, it defaults to the dir the recipe is in), and it'll build
> from
> > that local tree rather than doing the fetch+unpack+patch process.
>
> Ack. I now think of the class as providing a 'skip' to the
> fetch+unpack+patch
> steps but everything afterwards is then vanilla OE.
>

Yeah, that's the idea, and is the only reason the class needs to exist --
otherwise you could just point S to a local directory in the recipe and not
need it.  Aside: the way it does so is quite the hack, we need to figure out
a better way to manipulate the task graph like this.

Ideally I could have a single set of recipes which, through some command
> line switches,
> would either build in-tree using local sources or out-of-tree using a
> tarball.
> But maybe this really requires separate recipes with commonality factored
> into includes.


It would be possible to add a variable to toggle the behavior of the class,
but it isn't there today.  Note, you *should* be able to use an amend.inc
(w/ INHERIT += amend) to switch an existing recipe in an existing overlay
without modifying the recipe itself, which I could being quite useful to
temporarily switch it over and then back again, but I haven't tested that
myself.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Examples of srctree and gitver

[Ambrose, Martin]

On Sun, May 30, 2010 at 14:49:32, Chris Larson wrote:

> On Sun, May 30, 2010 at 12:26 PM, Ambrose, Martin <martin@ti.com> wrote:
> > On Sun, May 30, 2010 at 11:16:16, Chris Larson wrote:
> > > On Sun, May 30, 2010 at 8:07 AM, Ambrose, Martin <martin@ti.com> wrote:

...

> > Ack. I now think of the class as providing a 'skip' to the
> > fetch+unpack+patch steps but everything afterwards is then vanilla OE.
> 
> Yeah, that's the idea, and is the only reason the class needs to exist --
> otherwise you could just point S to a local directory in the recipe and not
> need it.  Aside: the way it does so is quite the hack, we need to figure out
> a better way to manipulate the task graph like this.

Maybe that explains the problem I'm seeing with the dependency graph when using srctree.
Specifically if my recipe has
	inherit qt4e

Then it doesn't try to run the do_configure before the qt native tools are present.
This of course being expected/desired. However if I have
	inherit srctree qt4e

Then it goes straight to the configure phase without ensuring the dependencies are present.
	bitbake blah
	...
	/home/user/oe/arago-tmp/staging/i686-linux/usr/bin/qmake2: No such file or directory

Is this known behavior? Maybe even a workaround exists?

Regards, 
Martin

Re: Examples of srctree and gitver

[Michael Smith]

Ambrose, Martin wrote:
> On Sun, May 30, 2010 at 14:49:32, Chris Larson wrote:
>> Aside: the way it does so is quite the hack, we need to figure out
>> a better way to manipulate the task graph like this.
 >
> Maybe that explains the problem I'm seeing with the dependency graph when using srctree.

> Then it goes straight to the configure phase without ensuring the dependencies are present.

Yes, I see similar things. I'm still on an ancient tree with 
do_populate_staging(). If memory serves, the prerequisite packages are 
unpacked, compiled and installed, but do_stage isn't run before the 
srctree recipe's do_configure.

As long as you're not doing parallel builds it's easy enough to work 
around: just put the libraries you need in your IMAGE_INSTALL before the 
srctree packages.

Mike

Re: Examples of srctree and gitver

[Chris Larson]

On Tue, Jun 1, 2010 at 6:36 AM, Michael Smith <msmith@cbnco.com> wrote:

> Ambrose, Martin wrote:
>
>> On Sun, May 30, 2010 at 14:49:32, Chris Larson wrote:
>>
>>> Aside: the way it does so is quite the hack, we need to figure out
>>> a better way to manipulate the task graph like this.
>>>
>> >
>
>> Maybe that explains the problem I'm seeing with the dependency graph when
>> using srctree.
>>
>
>  Then it goes straight to the configure phase without ensuring the
>> dependencies are present.
>>
>
> Yes, I see similar things. I'm still on an ancient tree with
> do_populate_staging(). If memory serves, the prerequisite packages are
> unpacked, compiled and installed, but do_stage isn't run before the srctree
> recipe's do_configure.
>
> As long as you're not doing parallel builds it's easy enough to work
> around: just put the libraries you need in your IMAGE_INSTALL before the
> srctree packages.


I'll look into this today.  The class may need adjusting for the new staging
stuff, and some of what it does can be undone entirely in certain cases.
 It:
1) empties SRC_URI, sets S to FILE_DIRNAME
2) inherits 'clean' to set up a different default clean method and pull in
helpers
3) removes do_patch and its deps from the task graph entirely
4) merges operations above patch, up to install & stage

4) is the particularly ugly bit, and is only necessary due to the recipe not
using a separate build vs source dir.  The issue is variants, bbclassextend,
etc, from what I can recall.  You can't risk an interleaving of the
configure/compile/install tasks between the recipes that share the same
source tree.  A per task lockfile is insufficient, as it's released between
the tasks.  I'll see about adding a flag to control this behavior, and also
automatically disable it when a separate build dir is used (B variable), and
will see if it can be done in a better way, unless someone else gets to it
before me :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Examples of srctree and gitver

[Ambrose, Martin]

On Tue, Jun 01, 2010 at 08:36:40, Michael Smith wrote:

> Ambrose, Martin wrote:
> > On Sun, May 30, 2010 at 14:49:32, Chris Larson wrote:
> >> Aside: the way it does so is quite the hack, we need to figure out
> >> a better way to manipulate the task graph like this.
>  >
> > Maybe that explains the problem I'm seeing with the dependency graph when using srctree.
> 
> > Then it goes straight to the configure phase without ensuring the dependencies are present.
> 
> Yes, I see similar things. I'm still on an ancient tree with 
> do_populate_staging(). If memory serves, the prerequisite packages are 
> unpacked, compiled and installed, but do_stage isn't run before the 
> srctree recipe's do_configure.
> 
> As long as you're not doing parallel builds it's easy enough to work 
> around: just put the libraries you need in your IMAGE_INSTALL before the 
> srctree packages.

Thanks for the pointer. Working around for now by building the dependencies
by hand so I could get to the next problem. This being that my do_install function
is not being populated from what's in my recipe.

My recipe has

do_install() {                                                                                                                                                                                                
    oenote "RUNNING install: ${D}"                                                                                                                                                                            
    install -d ${D}/home/root/appdir                                                                                                                                                                            
    install -m 0755 ${S}/myapp ${D}/home/root/myapp
}

But the resultant run.do_install script generates an empty function

base_do_install() {                                                                                                                                                                                           
        :                                                                                                                                                                                                     
                                                                                                                                                                                                              
}   
...
do_install() {                                                                                                                                                                                                
        base_do_install                                                                                                                                                                                       
                                                                                                                                                                                                              
}     
...
cd /home/user/mysrc                                                                                                                                                                                     
do_install  


Looks like the .bbclass which generates the script is not happy about something
but not sure where to look to find out what I'm missing.

Regards, 
Martin

Re: Examples of srctree and gitver

[Ambrose, Martin]

On Wed, Jun 02, 2010 at 16:47:17, Ambrose, Martin wrote:

> On Tue, Jun 01, 2010 at 08:36:40, Michael Smith wrote:
> 
> > Ambrose, Martin wrote:
> > > On Sun, May 30, 2010 at 14:49:32, Chris Larson wrote:
> > >> Aside: the way it does so is quite the hack, we need to figure out
> > >> a better way to manipulate the task graph like this.
> >  >
> > > Maybe that explains the problem I'm seeing with the dependency graph when using srctree.
> > 
> > > Then it goes straight to the configure phase without ensuring the dependencies are present.
> > 
> > Yes, I see similar things. I'm still on an ancient tree with 
> > do_populate_staging(). If memory serves, the prerequisite packages are 
> > unpacked, compiled and installed, but do_stage isn't run before the 
> > srctree recipe's do_configure.
> > 
> > As long as you're not doing parallel builds it's easy enough to work 
> > around: just put the libraries you need in your IMAGE_INSTALL before the 
> > srctree packages.
> 
> Thanks for the pointer. Working around for now by building the dependencies
> by hand so I could get to the next problem. This being that my do_install function
> is not being populated from what's in my recipe.
> 
> My recipe has
> 
> do_install() {                                                                                                                                                                                                
>     oenote "RUNNING install: ${D}"                                                                                                                                                                            
>     install -d ${D}/home/root/appdir                                                                                                                                                                            
>     install -m 0755 ${S}/myapp ${D}/home/root/myapp
> }

Actually my closing bracket here "}" wasn't in column zero. 
My emacs python mode doesn't do proper indentation I guess.
Anyway this seems to be a python/parsing error, with a silent failure,
and nothing to do with srctree AFAICT.

Regards,
Martin

> 
> But the resultant run.do_install script generates an empty function
> 
> base_do_install() {                                                                                                                                                                                           
>         :                                                                                                                                                                                                     
>                                                                                                                                                                                                               
> }   
> ...
> do_install() {                                                                                                                                                                                                
>         base_do_install                                                                                                                                                                                       
>                                                                                                                                                                                                               
> }     
> ...
> cd /home/user/mysrc                                                                                                                                                                                     
> do_install  
> 
> 
> Looks like the .bbclass which generates the script is not happy about something
> but not sure where to look to find out what I'm missing.
> 
> Regards, 
> Martin
> 



Regards, 
Martin

Re: Examples of srctree and gitver

[Michael Smith]

On Sun, 30 May 2010, Ambrose, Martin wrote:

> Are there any recipes/examples which use these classes?

Hi Martin,

I think you'll find srctree used mostly in out-of-tree stuff. Here's a 
sample.

inherit srctree gitver

# Strip off the leading 'v' from the Linux kernel git tags
GITVER_NO_V = "${@'${GITVER}'[1:]}"
PV = "${GITVER_NO_V}"

# Clean up between builds for different architectures
do_configure() {
    oe_runmake clean
}

do_compile() {
    oe_runmake
}

do_install() {
    # ...
}



Mike

Re: Examples of srctree and gitver

[Chris Larson]

On Sun, May 30, 2010 at 9:27 AM, Michael Smith <msmith@cbnco.com> wrote:

> On Sun, 30 May 2010, Ambrose, Martin wrote:
>
> > Are there any recipes/examples which use these classes?
>
> Hi Martin,
>
> I think you'll find srctree used mostly in out-of-tree stuff. Here's a
> sample.
>
> inherit srctree gitver
>
> # Strip off the leading 'v' from the Linux kernel git tags
> GITVER_NO_V = "${@'${GITVER}'[1:]}"
> PV = "${GITVER_NO_V}"
>
> # Clean up between builds for different architectures
> do_configure() {
>    oe_runmake clean
> }
>
> do_compile() {
>    oe_runmake
> }
>
> do_install() {
>    # ...
> }
>

Thanks, glad someone had an example for him handy :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: Examples of srctree and gitver

[Ambrose, Martin]

On Sun, May 30, 2010 at 11:27:45, Michael Smith wrote:

> On Sun, 30 May 2010, Ambrose, Martin wrote:

> > Are there any recipes/examples which use these classes?
> 

> # Strip off the leading 'v' from the Linux kernel git tags
> GITVER_NO_V = "${@'${GITVER}'[1:]}"
> PV = "${GITVER_NO_V}"
> 

Thanks for the pointer.

Looking at the class I see 
    ver = popen(["git", "describe", "--tags"], cwd=path)                                                                                                                                                                                     
    if not ver:                                                                                                                                                                                                                              
        ver = popen(["git", "rev-parse", "--short", "HEAD"], cwd=path)

Normally I work on a named branch so the PV here becomes the capitalized
branch name. My preference would be the shortened commit id which is the
fallback option above. I guess no way to get this functionality unless I
either don't work a branch or locally modify gitver.bbclass.

Regards, 
Martin

Re: Examples of srctree and gitver

[Michael Smith]

> From: Ambrose, Martin <martin@ti.com>
> Looking at the class I see
>     ver = popen(["git", "describe", "--tags"], cwd=path)

> Normally I work on a named branch so the PV here becomes the capitalized
> branch name. My preference would be the shortened commit id which is the
> fallback option above. I guess no way to get this functionality unless I
> either don't work a branch or locally modify gitver.bbclass.

I think if you tag a commit somewhere on your branch, you'll get 
<tag>-<count>-<shortened commit ID> as your version for all future commits.

Mike