bitbake and OE opperation

bitbake and OE opperation

[Mark Gross]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

I'm trying to wrap my head around bit bake and the open embedded recipe
files design and structure.

Also, I'm a relative newbie WRT Python.

I'm trying to understand the way bit bake builds the dependency trees
and works at a low level.   Trying to reverse engineer it from the
python code is difficult.  Does anyone recommend any browser tools for
navigating a large python code base like bit bake?

I've read the documentation pages I could find but I haven't found any
"theory of operation" types of documents.  Are there any such things?

The questions I have are:
How does a simple command line "bitbake slogos-image" translate into
building all the components?  

How does bitbake push in the ./configure command line options?

Are there tools for doing analysis on the open embedded bb library?  

Thanks,

--mgross

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: bitbake and OE opperation

[Richard Purdie]

Hi,

On Mon, 2007-03-19 at 20:31 -0700, Mark Gross wrote:
> I'm trying to understand the way bit bake builds the dependency trees
> and works at a low level.   Trying to reverse engineer it from the
> python code is difficult.  Does anyone recommend any browser tools for
> navigating a large python code base like bit bake?

I don't have any tools to recommend to help with this but you might find
the recent trunk versions easier to follow as the parts are more split
into modules.

> I've read the documentation pages I could find but I haven't found any
> "theory of operation" types of documents.  Are there any such things?

Did you read the bitbake manual? If so, thats around as low level as the
documentation goes since most OE users don't usually need to delve into
what bitbake does behind the scenes and the manual is a mostly complete
document of its API.

If you are interested, there are emails in the archive talking about
specific changes that have been made to the code such as multithreading.
Also, if there is anything you want clarified, I'm happy to explain
things if I can.

> The questions I have are:
> How does a simple command line "bitbake slogos-image" translate into
> building all the components?  

Bitbake only looks at a small subset of the metadata and is conceptually
quite simple. Speaking in simply terms, bitbake looks at what each file
PROVIDES and also what each file DEPENDS on.

Once it has a target, it can look at what dependencies it needs (from
DEPENDS and maybe RDEPENDS). It can then find providers for those
dependencies which can be influenced by things like PREFERRED_PROVIDER
and PREFERRED_VERSION. This cycle then repeats until it has a build
chain. It converts that chain into a list of tasks and then executes
each task in turn.

In bitbake trunk, taskdata.py generates the metadata model in memory and
then runqueue.py "compiles" that metedata into a single queue and builds
it. If a task fails, it can recompile the metadata and try a different
build path should the user wish.

Note that bitbake 1.6 or less handles this totally differently and the
code was rewritten in 1.7 onwards.

> How does bitbake push in the ./configure command line options?

Bitbake knows nothing about this, it just knows about a list of tasks.
autotools.bbclass defines a do_configure task which knows how to pass
options set in an EXTRA_OECONF variable to a configure commandline. The
classes are a powerful abstraction feature but are part of OE, not
bitbake.

> Are there tools for doing analysis on the open embedded bb library?  

What kind of analysis?

Regards,

Richard

Re: bitbake and OE opperation

[Lee Studley]

Hi,
I personally like www.activestate.com/Products/Komodo/   komodo
( linux and Windows ) for searching large projects. About $30 and is well 
worth it.

It has good grepping and module/object search tools. It can do single 
stepping and breakpoints in Python ( and other langs ) within the gui and 
some pretty good variable watch windows.
-Lee


At 02:34 PM 3/22/2007 +0000, you wrote:
>Hi,
>
>On Mon, 2007-03-19 at 20:31 -0700, Mark Gross wrote:
> > I'm trying to understand the way bit bake builds the dependency trees
> > and works at a low level.   Trying to reverse engineer it from the
> > python code is difficult.  Does anyone recommend any browser tools for
> > navigating a large python code base like bit bake?
>
>I don't have any tools to recommend to help with this but you might find
>the recent trunk versions easier to follow as the parts are more split
>into modules.
>
> > I've read the documentation pages I could find but I haven't found any
> > "theory of operation" types of documents.  Are there any such things?
>
>Did you read the bitbake manual? If so, thats around as low level as the
>documentation goes since most OE users don't usually need to delve into
>what bitbake does behind the scenes and the manual is a mostly complete
>document of its API.
>
>If you are interested, there are emails in the archive talking about
>specific changes that have been made to the code such as multithreading.
>Also, if there is anything you want clarified, I'm happy to explain
>things if I can.
>
> > The questions I have are:
> > How does a simple command line "bitbake slogos-image" translate into
> > building all the components?
>
>Bitbake only looks at a small subset of the metadata and is conceptually
>quite simple. Speaking in simply terms, bitbake looks at what each file
>PROVIDES and also what each file DEPENDS on.
>
>Once it has a target, it can look at what dependencies it needs (from
>DEPENDS and maybe RDEPENDS). It can then find providers for those
>dependencies which can be influenced by things like PREFERRED_PROVIDER
>and PREFERRED_VERSION. This cycle then repeats until it has a build
>chain. It converts that chain into a list of tasks and then executes
>each task in turn.
>
>In bitbake trunk, taskdata.py generates the metadata model in memory and
>then runqueue.py "compiles" that metedata into a single queue and builds
>it. If a task fails, it can recompile the metadata and try a different
>build path should the user wish.
>
>Note that bitbake 1.6 or less handles this totally differently and the
>code was rewritten in 1.7 onwards.
>
> > How does bitbake push in the ./configure command line options?
>
>Bitbake knows nothing about this, it just knows about a list of tasks.
>autotools.bbclass defines a do_configure task which knows how to pass
>options set in an EXTRA_OECONF variable to a configure commandline. The
>classes are a powerful abstraction feature but are part of OE, not
>bitbake.
>
> > Are there tools for doing analysis on the open embedded bb library?
>
>What kind of analysis?
>
>Regards,
>
>Richard
>
>
>
>
>_______________________________________________
>Openembedded-devel mailing list
>Openembedded-devel@lists.openembedded.org
>http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: bitbake and OE opperation

[Lee Studley]

Wow the price has jumped, but use the Student license: $49 version


At 11:37 AM 3/22/2007 -0700, you wrote:
>Hi,
>I personally like www.activestate.com/Products/Komodo/   komodo
>( linux and Windows ) for searching large projects. About $30 and is well
>worth it.
>
>It has good grepping and module/object search tools. It can do single
>stepping and breakpoints in Python ( and other langs ) within the gui and
>some pretty good variable watch windows.
>-Lee
>
>
>At 02:34 PM 3/22/2007 +0000, you wrote:
> >Hi,
> >
> >On Mon, 2007-03-19 at 20:31 -0700, Mark Gross wrote:
> > > I'm trying to understand the way bit bake builds the dependency trees
> > > and works at a low level.   Trying to reverse engineer it from the
> > > python code is difficult.  Does anyone recommend any browser tools for
> > > navigating a large python code base like bit bake?
> >
> >I don't have any tools to recommend to help with this but you might find
> >the recent trunk versions easier to follow as the parts are more split
> >into modules.
> >
> > > I've read the documentation pages I could find but I haven't found any
> > > "theory of operation" types of documents.  Are there any such things?
> >
> >Did you read the bitbake manual? If so, thats around as low level as the
> >documentation goes since most OE users don't usually need to delve into
> >what bitbake does behind the scenes and the manual is a mostly complete
> >document of its API.
> >
> >If you are interested, there are emails in the archive talking about
> >specific changes that have been made to the code such as multithreading.
> >Also, if there is anything you want clarified, I'm happy to explain
> >things if I can.
> >
> > > The questions I have are:
> > > How does a simple command line "bitbake slogos-image" translate into
> > > building all the components?
> >
> >Bitbake only looks at a small subset of the metadata and is conceptually
> >quite simple. Speaking in simply terms, bitbake looks at what each file
> >PROVIDES and also what each file DEPENDS on.
> >
> >Once it has a target, it can look at what dependencies it needs (from
> >DEPENDS and maybe RDEPENDS). It can then find providers for those
> >dependencies which can be influenced by things like PREFERRED_PROVIDER
> >and PREFERRED_VERSION. This cycle then repeats until it has a build
> >chain. It converts that chain into a list of tasks and then executes
> >each task in turn.
> >
> >In bitbake trunk, taskdata.py generates the metadata model in memory and
> >then runqueue.py "compiles" that metedata into a single queue and builds
> >it. If a task fails, it can recompile the metadata and try a different
> >build path should the user wish.
> >
> >Note that bitbake 1.6 or less handles this totally differently and the
> >code was rewritten in 1.7 onwards.
> >
> > > How does bitbake push in the ./configure command line options?
> >
> >Bitbake knows nothing about this, it just knows about a list of tasks.
> >autotools.bbclass defines a do_configure task which knows how to pass
> >options set in an EXTRA_OECONF variable to a configure commandline. The
> >classes are a powerful abstraction feature but are part of OE, not
> >bitbake.
> >
> > > Are there tools for doing analysis on the open embedded bb library?
> >
> >What kind of analysis?
> >
> >Regards,
> >
> >Richard
> >
> >
> >
> >
> >_______________________________________________
> >Openembedded-devel mailing list
> >Openembedded-devel@lists.openembedded.org
> >http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>_______________________________________________
>Openembedded-devel mailing list
>Openembedded-devel@lists.openembedded.org
>http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

Re: bitbake and OE opperation

[Mark Gross]

[-- Attachment #1: Type: text/plain, Size: 5182 bytes --]

On Thu, Mar 22, 2007 at 02:34:45PM +0000, Richard Purdie wrote:
> Hi,
> 
> On Mon, 2007-03-19 at 20:31 -0700, Mark Gross wrote:
> > I'm trying to understand the way bit bake builds the dependency trees
> > and works at a low level.   Trying to reverse engineer it from the
> > python code is difficult.  Does anyone recommend any browser tools for
> > navigating a large python code base like bit bake?
> 
> I don't have any tools to recommend to help with this but you might find
> the recent trunk versions easier to follow as the parts are more split
> into modules.
> 
> > I've read the documentation pages I could find but I haven't found any
> > "theory of operation" types of documents.  Are there any such things?
> 
> Did you read the bitbake manual? If so, thats around as low level as the
> documentation goes since most OE users don't usually need to delve into
> what bitbake does behind the scenes and the manual is a mostly complete
> document of its API.

Yup, I read it.  I still have questions (I've figured out a number of
things since I posted this email.) Like, what triggers a given bbclass
file to be parsed, and what determines when if the functions withing the
class definition are executed.  I think I "get" the tasks and bb files
but the classes are still a bit mysterious.

> If you are interested, there are emails in the archive talking about
> specific changes that have been made to the code such as multithreading.
> Also, if there is anything you want clarified, I'm happy to explain
> things if I can.

Mostly I was looking for the starting point in the dynamical program
space that the represent the dependency tree within the OE recipe
trees.  I think I get this.

BB parses all the bb files which build up a network of dependencies that
is then traversed based on a seed point defined by the command line.

Some tricks I found to identify the bb files involved in my slugos-image
build was to run nohup bitbake -n -v slugos-image, and nohup bitbake -e
slugos-image.  I also got a fair amount of insight by running bitbake
in interactive mode and using the which command.

So now I know how to find all the bb files that are involved with
building my image.  

> 
> > The questions I have are:
> > How does a simple command line "bitbake slogos-image" translate into
> > building all the components?  
> 
> Bitbake only looks at a small subset of the metadata and is conceptually
> quite simple. Speaking in simply terms, bitbake looks at what each file
> PROVIDES and also what each file DEPENDS on.
> 
> Once it has a target, it can look at what dependencies it needs (from
> DEPENDS and maybe RDEPENDS). It can then find providers for those
> dependencies which can be influenced by things like PREFERRED_PROVIDER
> and PREFERRED_VERSION. This cycle then repeats until it has a build
> chain. It converts that chain into a list of tasks and then executes
> each task in turn.
> 
> In bitbake trunk, taskdata.py generates the metadata model in memory and
> then runqueue.py "compiles" that metedata into a single queue and builds
> it. If a task fails, it can recompile the metadata and try a different
> build path should the user wish.
> 
> Note that bitbake 1.6 or less handles this totally differently and the
> code was rewritten in 1.7 onwards.

thanks!

> 
> > How does bitbake push in the ./configure command line options?
> 
> Bitbake knows nothing about this, it just knows about a list of tasks.
> autotools.bbclass defines a do_configure task which knows how to pass
> options set in an EXTRA_OECONF variable to a configure commandline. The
> classes are a powerful abstraction feature but are part of OE, not
> bitbake.

How is autotools.bbclass identified to be parsed?
What triggers do_configure to be executed and when?
What do I need to hack if I want to change the way or when they are
called?
There has to be some initial entry points for the classes.

Are there a set of pre-defined class functions that will be called or
does some bb file reference the class member functions?  (I haven't found
them referenced in the bb files I looked at, and it makes me think there
is some meta knowledge in the bitbake code that expects to call things
like do_configure, or do_patch, or do_build, but I'm having a hard time
finding them.

basically I'm trying to understand the glue that holds everything
together.  I'm staring to feel dangerous at hacking the OK bb tasks but
I'm still lost on the classes.

> 
> > Are there tools for doing analysis on the open embedded bb library?  
> 
> What kind of analysis?

Dependency trees, configuration tweaking, the export of a minimal tree
of bbfiles for a specific build, a build browsing tool for exploring the
OE dependency cloud, for providers-bb files, classes and config files.

Its taken me a long time figure out all the OE files that effect the
build of my image.  It be nice to make things easier for folks that come
after me to do the same type of thing....

> 
> Regards,
>

Thanks a lot!  I think in a few more weekends I may start to "get it".

--mgross


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: bitbake and OE opperation

[Richard Purdie]

On Thu, 2007-03-22 at 21:31 -0700, Mark Gross wrote:
> On Thu, Mar 22, 2007 at 02:34:45PM +0000, Richard Purdie wrote:
> Yup, I read it.  I still have questions (I've figured out a number of
> things since I posted this email.) Like, what triggers a given bbclass
> file to be parsed, and what determines when if the functions withing the
> class definition are executed.  I think I "get" the tasks and bb files
> but the classes are still a bit mysterious.

To trigger the inclusion of autotools.bbclass, you'd see "inherit
autotools". Each .bb file has an assumed "inherit base". Also, conf
files can add a class file for inclusion with IHHERIT = "xyz".

> Mostly I was looking for the starting point in the dynamical program
> space that the represent the dependency tree within the OE recipe
> trees.  I think I get this.
> 
> BB parses all the bb files which build up a network of dependencies that
> is then traversed based on a seed point defined by the command line.

Or seed points as it can have more than one, but yes.

> > > How does bitbake push in the ./configure command line options?
> > 
> > Bitbake knows nothing about this, it just knows about a list of tasks.
> > autotools.bbclass defines a do_configure task which knows how to pass
> > options set in an EXTRA_OECONF variable to a configure commandline. The
> > classes are a powerful abstraction feature but are part of OE, not
> > bitbake.
> 
> How is autotools.bbclass identified to be parsed?

See above.

> What triggers do_configure to be executed and when?

Read base.bbclass. It sets up a number of tasks and a default task,
"build". You'd add configure after patch, before compile. compile would
be before build. "bitbake somefile" would cause the default task (and
its dependent tasks) to be executed. You could also do "bitbake somefile
-c configure" to specifically execute the configure task.

> What do I need to hack if I want to change the way or when they are
> called?
> There has to be some initial entry points for the classes.

An assumed inherit base, then inheriting other classes like autotools
which can change things. The order of the inherits matters.

> Are there a set of pre-defined class functions that will be called or
> does some bb file reference the class member functions?  (I haven't found
> them referenced in the bb files I looked at, and it makes me think there
> is some meta knowledge in the bitbake code that expects to call things
> like do_configure, or do_patch, or do_build, but I'm having a hard time
> finding them.
> 
> basically I'm trying to understand the glue that holds everything
> together.  I'm staring to feel dangerous at hacking the OK bb tasks but
> I'm still lost on the classes.

Hopefully I've covered this above.

> Dependency trees, configuration tweaking, the export of a minimal tree
> of bbfiles for a specific build, a build browsing tool for exploring the
> OE dependency cloud, for providers-bb files, classes and config files.

You might fine bitbake -g interesting, particularly in trunk. It will
dump a number of dependency graph files.

bitbake -DDD (enabling a ton of debugging) will also show you which
files it touches and why.

> Its taken me a long time figure out all the OE files that effect the
> build of my image.  It be nice to make things easier for folks that come
> after me to do the same type of thing....

Help with the documentation is more than welcome!

Cheers,

Richard

Re: bitbake and OE opperation

[Mark Gross]

[-- Attachment #1: Type: text/plain, Size: 4133 bytes --]

On Fri, Mar 23, 2007 at 09:20:15AM +0000, Richard Purdie wrote:
> On Thu, 2007-03-22 at 21:31 -0700, Mark Gross wrote:
> > On Thu, Mar 22, 2007 at 02:34:45PM +0000, Richard Purdie wrote:
> > Yup, I read it.  I still have questions (I've figured out a number of
> > things since I posted this email.) Like, what triggers a given bbclass
> > file to be parsed, and what determines when if the functions withing the
> > class definition are executed.  I think I "get" the tasks and bb files
> > but the classes are still a bit mysterious.
> 
> To trigger the inclusion of autotools.bbclass, you'd see "inherit
> autotools". Each .bb file has an assumed "inherit base". Also, conf
> files can add a class file for inclusion with IHHERIT = "xyz".
> 

Ah, I'll take a closer look at base.


> > Mostly I was looking for the starting point in the dynamical program
> > space that the represent the dependency tree within the OE recipe
> > trees.  I think I get this.
> > 
> > BB parses all the bb files which build up a network of dependencies that
> > is then traversed based on a seed point defined by the command line.
> 
> Or seed points as it can have more than one, but yes.
> 
> > > > How does bitbake push in the ./configure command line options?
> > > 
> > > Bitbake knows nothing about this, it just knows about a list of tasks.
> > > autotools.bbclass defines a do_configure task which knows how to pass
> > > options set in an EXTRA_OECONF variable to a configure commandline. The
> > > classes are a powerful abstraction feature but are part of OE, not
> > > bitbake.
> > 
> > How is autotools.bbclass identified to be parsed?
> 
> See above.
> 
> > What triggers do_configure to be executed and when?
> 
> Read base.bbclass. It sets up a number of tasks and a default task,
> "build". You'd add configure after patch, before compile. compile would
> be before build. "bitbake somefile" would cause the default task (and
> its dependent tasks) to be executed. You could also do "bitbake somefile
> -c configure" to specifically execute the configure task.

Ok.

> 
> > What do I need to hack if I want to change the way or when they are
> > called?
> > There has to be some initial entry points for the classes.
> 
> An assumed inherit base, then inheriting other classes like autotools
> which can change things. The order of the inherits matters.
> 
> > Are there a set of pre-defined class functions that will be called or
> > does some bb file reference the class member functions?  (I haven't found
> > them referenced in the bb files I looked at, and it makes me think there
> > is some meta knowledge in the bitbake code that expects to call things
> > like do_configure, or do_patch, or do_build, but I'm having a hard time
> > finding them.
> > 
> > basically I'm trying to understand the glue that holds everything
> > together.  I'm staring to feel dangerous at hacking the OK bb tasks but
> > I'm still lost on the classes.
> 
> Hopefully I've covered this above.
> 
> > Dependency trees, configuration tweaking, the export of a minimal tree
> > of bbfiles for a specific build, a build browsing tool for exploring the
> > OE dependency cloud, for providers-bb files, classes and config files.
> 
> You might fine bitbake -g interesting, particularly in trunk. It will
> dump a number of dependency graph files.

I forgot that one.  I found that the class dependencies aren't
represented in the graph.  It seems to be just the providers(?) / bb files.

> 
> bitbake -DDD (enabling a ton of debugging) will also show you which
> files it touches and why.
>


I'll give this a spin!


> > Its taken me a long time figure out all the OE files that effect the
> > build of my image.  It be nice to make things easier for folks that come
> > after me to do the same type of thing....
> 
> Help with the documentation is more than welcome!

That is my intention, at least to start with.

Something of a draft will be coming in a few weekends, as this all sinks
in a bit more.

Thanks so much for your help!

--mgross


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: bitbake and OE opperation

[Tim Bird]

Richard Purdie wrote:
> 
>> What triggers do_configure to be executed and when?
> 
> Read base.bbclass. It sets up a number of tasks and a default task,
> "build". You'd add configure after patch, before compile. compile would
> be before build. "bitbake somefile" would cause the default task (and
> its dependent tasks) to be executed. You could also do "bitbake somefile
> -c configure" to specifically execute the configure task.

I'm sorry, but I don't understand most of that paragraph.

I've read the manuals - maybe I missed it, but I have the same question.

In <oe>/classes/base.bbclass I see:

BB_DEFAULT_TASK = "build"

at the top of the file.

There's no "def build()" or "def do_build()"
There are the following lines - I don't know if they're related or not:

addtask build after do_populate_staging
do_build = ""
do_build[func] = "1"

In <bitbake>/classes/base.bbclass I see:

addtask build
do_build[dirs] = "${TOPDIR}"
do_build[nostamp] = "1"
python base_do_build () {
        bb.note("The included, default BB base.bbclass does not define a useful default task.")
        bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
}

Maybe I'm thinking procedurally, and this is all declarative, but where
the heck are the actual statements executed during a "build"?

Back to Mark's point, how are the stages "fetch", "compile", "install"
known to the system, and sequenced.  Are these hardcoded in bitbake?
I'm guessing not, from the answer above and the structure of the
"addtask <foo> before|after <bar>" statements.  But I can't find
anywhere in the bbclass files where this is expressed.

Please help me understand this.

Thanks,
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: bitbake and OE opperation

[Richard Purdie]

On Wed, 2007-03-28 at 10:54 -0700, Tim Bird wrote:
> Richard Purdie wrote:
> >> What triggers do_configure to be executed and when?
> > 
> > Read base.bbclass. It sets up a number of tasks and a default task,
> > "build". You'd add configure after patch, before compile. compile would
> > be before build. "bitbake somefile" would cause the default task (and
> > its dependent tasks) to be executed. You could also do "bitbake somefile
> > -c configure" to specifically execute the configure task.
> 
> I'm sorry, but I don't understand most of that paragraph.
> 
> I've read the manuals - maybe I missed it, but I have the same question.
> 
> In <oe>/classes/base.bbclass I see:
> 
> BB_DEFAULT_TASK = "build"
> 
> at the top of the file.
> 
> There's no "def build()" or "def do_build()"
> There are the following lines - I don't know if they're related or not:
> 
> addtask build after do_populate_staging
> do_build = ""
> do_build[func] = "1"

These are very much related, it defines that there is a build task (with
an associated function which defaults to empty). It says this build task
depends on a "populate_staging" task.

> In <bitbake>/classes/base.bbclass I see:
> 
> addtask build
> do_build[dirs] = "${TOPDIR}"
> do_build[nostamp] = "1"
> python base_do_build () {
>         bb.note("The included, default BB base.bbclass does not define a useful default task.")
>         bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
> }

Please ignore <bitbake>/classes/base.bbclass, they're examples in
bitbake and the files in OE take precedence.

> Maybe I'm thinking procedurally, and this is all declarative, but where
> the heck are the actual statements executed during a "build"?

Nothing is executed by "build" itself, the key is in the dependency on
"populate_staging".

> Back to Mark's point, how are the stages "fetch", "compile", "install"
> known to the system, and sequenced.  Are these hardcoded in bitbake?
> I'm guessing not, from the answer above and the structure of the
> "addtask <foo> before|after <bar>" statements.  But I can't find
> anywhere in the bbclass files where this is expressed.
> 
> Please help me understand this.

They're not hardcoded, the sequence is made through the definition of
tasks, each of which depend on each other. Very roughly:

build depends on populate_staging
populate_staging depends on package_write
package_write depends on package
package depends on install
install depends on compile
compile depends on configure
configure depends on patch
patch depends on unpack
unpack depends on fetch

This is all in base.bbclass, albeit rather difficult to isolate on its
own. Tasks like fetch, configure, compile etc. all have functions
associated with them and build is effectively a combination of all of
these.

Incidentally, don't be put off by the "do_" prefix, this is just to
highlight things as tasks when their usage might otherwise be
ambiguous. 

Does that make it clearer?

Regards,

Richard

Re: bitbake and OE opperation

[Tim Bird]

Not to belabor this, but I'm going to try to repeat
this stuff back, to make sure I understand.

Richard Purdie wrote:
>> There are the following lines - I don't know if they're related or not:
>>
>> addtask build after do_populate_staging
>> do_build = ""
>> do_build[func] = "1"
> 
> These are very much related, it defines that there is a build task (with
> an associated function which defaults to empty). It says this build task
> depends on a "populate_staging" task.

So this is similar to:

build: do_populate_staging

in a makefile.

>> Maybe I'm thinking procedurally, and this is all declarative, but where
>> the heck are the actual statements executed during a "build"?
> 
> Nothing is executed by "build" itself, the key is in the dependency on
> "populate_staging".
> 
In this case, since build has no actual
commands associated with it, it is like a PHONY target in
a makefile?

> They're not hardcoded, the sequence is made through the definition of
> tasks, each of which depend on each other. Very roughly:
> 
> build depends on populate_staging
> populate_staging depends on package_write
> package_write depends on package
> package depends on install
> install depends on compile
> compile depends on configure
> configure depends on patch
> patch depends on unpack
> unpack depends on fetch
>

From base.bbclass, if I grep addtask, I see:
addtask showdata
addtask listtasks
addtask clean
addtask rebuild
addtask mrproper
addtask fetch
addtask fetchall after do_fetch
addtask unpack after do_fetch
addtask configure after do_unpack do_patch
addtask compile after do_configure
addtask populate_staging after do_package_write
addtask install after do_compile
addtask build after do_populate_staging

So there appear to be "island" tasks (e.g.showdata, clean),
that are not part of the main sequence for a build.  I presume these
are accessible via the command line in bitbake?

I don't see an addtask for "patch" in there.  Is this added
to the dependency tree somewhere else?  Is this something
that is package-specific?

Also, there appear to be multiple parallel
dependencies (fetchall and unpack are both after fetch).
How are these ordered?  (What is "fetchall" anyway?)

Tasks package and package_write are also referenced but not "addtask"ed
to the graph.

> Incidentally, don't be put off by the "do_" prefix, this is just to
> highlight things as tasks when their usage might otherwise be
> ambiguous.

Is "do_" ignored in task declarations, references, or both?

> Does that make it clearer?
Yes, thanks.

I assume that .bb files can arbitrarily define some new
task and graft it into the dependency tree.

In your experience, are there many other custom (package-specific)
tasks like this?

Here is my idea of what each task does:
Can you please check this?

build - meta-task to cover the whole build process
populate_staging - (same as do_stage?) - put package contents in staging area?
package_write - ??
package - collect files files from install locations into some kind of collection (dependent on package format)
install - place files (e.g. built binaries) at install locations
compile - make
configure - run configure
patch - apply patches to source
unpack - unpack tarballs
fetch - grab sources from Internet - turn into tarballs

Are there any other major tasks?

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Electronics
=============================

Re: bitbake and OE opperation

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim Bird schreef:

> build - meta-task to cover the whole build process
> populate_staging - (same as do_stage?) - put package contents in staging area?

do_stage was a wrapper around populate_staging and something arcane, and nowadays just
calls populate_staging directly

> package_write - ??

Write out the actual packages

> package - collect files files from install locations into some kind of collection (dependent on package format)

collect files files from install locations (${WORKDIR}/image/) into package directories (
${WORKDIR}/install/<package-name>) (defined by PACKAGES and FILES), independant of package
format.

> install - place files (e.g. built binaries) at install locations
> compile - make
> configure - run configure
> patch - apply patches to source
> unpack - unpack tarballs
> fetch - grab sources from Internet - turn into tarballs
> 
> Are there any other major tasks?

do_deploy and do_rootfs are pretty widely used as well.

regards,

KOen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFGCs3YMkyGM64RGpERArHhAJ9RciWFDVNgpidR3QMk7m1Mho8c1gCfftes
RVyXY/27NNgKh57WDcDXShY=
=af3p
-----END PGP SIGNATURE-----

Re: bitbake and OE opperation

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 266 bytes --]

Tim Bird wrote:
> So there appear to be "island" tasks (e.g.showdata, clean),
> that are not part of the main sequence for a build.  I presume these
> are accessible via the command line in bitbake?

bitbake -c clean <thing to clean>

For example.

Philip

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3303 bytes --]

Re: bitbake and OE opperation

[Richard Purdie]

On Wed, 2007-03-28 at 13:10 -0700, Tim Bird wrote:
> Richard Purdie wrote:
> >> addtask build after do_populate_staging
> >> do_build = ""
> >> do_build[func] = "1"
> > 
> > These are very much related, it defines that there is a build task (with
> > an associated function which defaults to empty). It says this build task
> > depends on a "populate_staging" task.
> 
> So this is similar to:
> 
> build: do_populate_staging
> 
> in a makefile.

Yes.

> >> Maybe I'm thinking procedurally, and this is all declarative, but where
> >> the heck are the actual statements executed during a "build"?
> > 
> > Nothing is executed by "build" itself, the key is in the dependency on
> > "populate_staging".
> > 
> In this case, since build has no actual
> commands associated with it, it is like a PHONY target in
> a makefile?

Yes.

> From base.bbclass, if I grep addtask, I see:
> addtask showdata
> addtask listtasks
> addtask clean
> addtask rebuild
> addtask mrproper
> addtask fetch
> addtask fetchall after do_fetch
> addtask unpack after do_fetch
> addtask configure after do_unpack do_patch
> addtask compile after do_configure
> addtask populate_staging after do_package_write
> addtask install after do_compile
> addtask build after do_populate_staging
> 
> So there appear to be "island" tasks (e.g.showdata, clean),
> that are not part of the main sequence for a build.  I presume these
> are accessible via the command line in bitbake?

Yes, e.g. "bitbake something -c listtasks".

> I don't see an addtask for "patch" in there.  Is this added
> to the dependency tree somewhere else?  Is this something
> that is package-specific?

base.bbclass has "inherit patch" and you'll find addtask patch in
patch.bbclass. The patch functionality became big enough to warrant its
own file. Its unconditionally included in the build.

> Also, there appear to be multiple parallel
> dependencies (fetchall and unpack are both after fetch).
> How are these ordered?  (What is "fetchall" anyway?)

fetchall is an independent task which fetches all sources for all
dependent packages as well as the sources for the package itself (e.g.
"bitbake busybox -c fetchall" would fetch busybox's sources but also
glibc amongst other things). It was one possible proposal for fetching
sources for subsequent offline use. Dependency wise, bitbake will make
sure fetch has run before fetchall runs.

> Tasks package and package_write are also referenced but not "addtask"ed
> to the graph.

This is because you have to specifically enable packaging by inheriting
package_ipk or package_deb or some combination of packaging classes.
This is normally done as part of the distribution configuration and the
addtasks appear in package.bbclass.

> > Incidentally, don't be put off by the "do_" prefix, this is just to
> > highlight things as tasks when their usage might otherwise be
> > ambiguous.
> 
> Is "do_" ignored in task declarations, references, or both?

Its present in task declarations and some references when its unclear
what you'd be referring to is the best summary I can give. Anything
passed to a -c option on the bitbake commandline is obviously a task
name for example, with a function definition, its not obvious so its
prefixed. I suspect there are some exceptions to the above syntax
summary but they existed before I worked with things and I don't know
the historical reasons.

> I assume that .bb files can arbitrarily define some new
> task and graft it into the dependency tree.

Correct, and some packages do this.

> In your experience, are there many other custom (package-specific)
> tasks like this?

glibc has the infamous "do_munge" which adds in the glibc ports tree.
There are also class related tasks like do_rootfs for generating images.
do_deploy is a convention for placing output of things like kernel
builds or bootloaders into tmp/deploy/whereever.

> Here is my idea of what each task does:
> Can you please check this?
> 
> build - meta-task to cover the whole build process
> populate_staging - (same as do_stage?) - put package contents in staging area?

Yes, there used to be two confusing staging functions, do_stage and
do_populate_staging. do_stage was removed.

> install - place files (e.g. built binaries) at install locations

Yes, as a prelude for splitting them for packaging in "package" below.

> package - collect files files from install locations into some kind of collection (dependent on package format)

Splits the into separate directories ready for packaging.

> package_write - ??

Actually writes the files into the specified package formats.

> compile - make
> configure - run configure
> patch - apply patches to source
> unpack - unpack tarballs
> fetch - grab sources from Internet - turn into tarballs

deploy - put some kind of output into tmp/deploy (kernel, bootloader)
rootfs - create an image
devshell - drop to a shell within the work directory (interactive, see 
		recent email discussion, need to inherit devshell)
listtasks - lists the available tasks for a target
clean - Delete the workdir and stamps (does not depopulate staging atm)
rebuild - "clean", then "build"
fetchall - fetch all sources + dependent sources


Cheers,

Richard