RFC: Layer tooling brainstorming

RFC: Layer tooling brainstorming

[Richard Purdie]

Layer Tooling for OpenEmbedded
==============================

As we move forward with encouraging the use of metadata layers, a number
of issues are appearing which imply the need for either infrastructure
improvements to the layer construction, or better tools to handle the
layers. The OS TSC discussed layers and layer tooling at the face to
face TSC meeting in San Francisco. Some example problems that were
identified that currently need addressing in some form are:

* There is no dependency information between layers (layer X depends on 
  layer Y and Z)
* There is no version information present (layer X needs version A of 
  layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 
  1.1 roadmap)
* Ability to fetch layers from other repositories
  - Multi SCM?
  - Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky,
  RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only 
  beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer 
  components present
* bitbake -e on steroids to highlight differences due to layer changes? 
  (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack 
  than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseduo is 
  established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least 
  enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs 
  and ability to control tool evolution, likely need own tool.

This hopefully summarises the ideas we discussed in person, we therefore
need to start looking at how to move forward from here with this but I
wanted to document this for starters!

Cheers,

Richard

Re: RFC: Layer tooling brainstorming

[Martin Jansa]

On Mon, Apr 18, 2011 at 07:17:01AM +0100, Richard Purdie wrote:
> Layer Tooling for OpenEmbedded
> ==============================
> 
> As we move forward with encouraging the use of metadata layers, a number
> of issues are appearing which imply the need for either infrastructure
> improvements to the layer construction, or better tools to handle the
> layers. The OS TSC discussed layers and layer tooling at the face to
> face TSC meeting in San Francisco. Some example problems that were
> identified that currently need addressing in some form are:
> 
> * There is no dependency information between layers (layer X depends on 
>   layer Y and Z)
> * There is no version information present (layer X needs version A of 
>   layer Y)
> * There is no layer "ABI" version number present in layers
> * Layer specific sanity checks? (other types of sanity checks on Yocto 
>   1.1 roadmap)
> * Ability to fetch layers from other repositories
>   - Multi SCM?
>   - Use bitbake fetcher?
> * Ability to generate repositories representing composited layers (Poky,
>   RP has hacky scripts)
> * Tool to seperate patch into one for different composited parts
> * Ability to include partial components of repositories (e.g. only 
>   beagleboard from meta-ti)
> * Tool to merge (flatten) several layers into one
> * Able to visualise dependencies between layers
> * When bitbake prints banner of branch/commit, need to cover all layer 
>   components present
> * bitbake -e on steroids to highlight differences due to layer changes? 
>   (long term goal)

maybe I've missed something, but
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in
  upper layer was upgraded to foo_1.1.bb

This can cause big problems when distribution explicitly enables
something in .bbappend with EXTRA_OECONF, and someone updates all
layers, builds foo_1.1.bb and installs it on targets before noticing
that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding
foo_1.1.bbappend to distro layer (to fix foo on targets), so I would
prefer parsing error in this case (won't help if upper layer keeps both
versions foo_1.0.bb and foo_1.1.bb ;/).

This should be combined with your 2nd point and person who changes
versioned dependency on upper layer should check that all needed
.bbappends still apply.

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: RFC: Layer tooling brainstorming

[Richard Purdie]

On Mon, 2011-04-18 at 08:50 +0200, Martin Jansa wrote:
> On Mon, Apr 18, 2011 at 07:17:01AM +0100, Richard Purdie wrote:
> > Layer Tooling for OpenEmbedded
> > ==============================
> > 
> > As we move forward with encouraging the use of metadata layers, a number
> > of issues are appearing which imply the need for either infrastructure
> > improvements to the layer construction, or better tools to handle the
> > layers. The OS TSC discussed layers and layer tooling at the face to
> > face TSC meeting in San Francisco. Some example problems that were
> > identified that currently need addressing in some form are:
> > 
> > * There is no dependency information between layers (layer X depends on 
> >   layer Y and Z)
> > * There is no version information present (layer X needs version A of 
> >   layer Y)
> > * There is no layer "ABI" version number present in layers
> > * Layer specific sanity checks? (other types of sanity checks on Yocto 
> >   1.1 roadmap)
> > * Ability to fetch layers from other repositories
> >   - Multi SCM?
> >   - Use bitbake fetcher?
> > * Ability to generate repositories representing composited layers (Poky,
> >   RP has hacky scripts)
> > * Tool to seperate patch into one for different composited parts
> > * Ability to include partial components of repositories (e.g. only 
> >   beagleboard from meta-ti)
> > * Tool to merge (flatten) several layers into one
> > * Able to visualise dependencies between layers
> > * When bitbake prints banner of branch/commit, need to cover all layer 
> >   components present
> > * bitbake -e on steroids to highlight differences due to layer changes? 
> >   (long term goal)
> 
> maybe I've missed something, but
> * Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in
>   upper layer was upgraded to foo_1.1.bb
>
> This can cause big problems when distribution explicitly enables
> something in .bbappend with EXTRA_OECONF, and someone updates all
> layers, builds foo_1.1.bb and installs it on targets before noticing
> that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding
> foo_1.1.bbappend to distro layer (to fix foo on targets), so I would
> prefer parsing error in this case (won't help if upper layer keeps both
> versions foo_1.0.bb and foo_1.1.bb ;/).
> 
> This should be combined with your 2nd point and person who changes
> versioned dependency on upper layer should check that all needed
> .bbappends still apply.

Agreed but I thought we already had a tool to handle at least some of
this though? Certainly integration of that with the above is essential
though.

Cheers,

Richard

Re: RFC: Layer tooling brainstorming

[Andreas Mueller]

On Monday 18 April 2011 08:17:01 Richard Purdie wrote:
> Layer Tooling for OpenEmbedded
> ==============================
> 
> As we move forward with encouraging the use of metadata layers, a number
> of issues are appearing which imply the need for either infrastructure
> improvements to the layer construction, or better tools to handle the
> layers. The OS TSC discussed layers and layer tooling at the face to
> face TSC meeting in San Francisco. Some example problems that were
> identified that currently need addressing in some form are:
> 
> * There is no dependency information between layers (layer X depends on
>   layer Y and Z)
> * There is no version information present (layer X needs version A of
>   layer Y)
> * There is no layer "ABI" version number present in layers
> * Layer specific sanity checks? (other types of sanity checks on Yocto
>   1.1 roadmap)
> * Ability to fetch layers from other repositories
>   - Multi SCM?
>   - Use bitbake fetcher?
> * Ability to generate repositories representing composited layers (Poky,
>   RP has hacky scripts)
> * Tool to seperate patch into one for different composited parts
> * Ability to include partial components of repositories (e.g. only
>   beagleboard from meta-ti)
> * Tool to merge (flatten) several layers into one
> * Able to visualise dependencies between layers
> * When bitbake prints banner of branch/commit, need to cover all layer
>   components present
> * bitbake -e on steroids to highlight differences due to layer changes?
>   (long term goal)
> 
> Implementation thoughts:
> 
> * This layer tooling probably belongs at a higher level on the stack
>   than bitbake itself
> * Maybe need to split into "bootstrap" steps (e.g where pseduo is
>   established, layers downloaded etc)
> * Need some kind of config file defining layer's properties or at least
>   enhance layer.conf significantly
> * git submodules and repo should be looked at but due to specific needs
>   and ability to control tool evolution, likely need own tool.
> 
> This hopefully summarises the ideas we discussed in person, we therefore
> need to start looking at how to move forward from here with this but I
> wanted to document this for starters!
> 
> Cheers,
> 
> Richard
Afraid being killed I didn't dare to ask - but it's getting time to do so: What 
exactly are layers in OE and what are their advantages?

Andreas

Re: RFC: Layer tooling brainstorming

[Koen Kooi]

Op 18 apr 2011, om 09:09 heeft Richard Purdie het volgende geschreven:

> On Mon, 2011-04-18 at 08:50 +0200, Martin Jansa wrote:
>> On Mon, Apr 18, 2011 at 07:17:01AM +0100, Richard Purdie wrote:
>> 
>> maybe I've missed something, but
>> * Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in
>>  upper layer was upgraded to foo_1.1.bb
>> 
>> This can cause big problems when distribution explicitly enables
>> something in .bbappend with EXTRA_OECONF, and someone updates all
>> layers, builds foo_1.1.bb and installs it on targets before noticing
>> that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding
>> foo_1.1.bbappend to distro layer (to fix foo on targets), so I would
>> prefer parsing error in this case (won't help if upper layer keeps both
>> versions foo_1.0.bb and foo_1.1.bb ;/).
>> 
>> This should be combined with your 2nd point and person who changes
>> versioned dependency on upper layer should check that all needed
>> .bbappends still apply.
> 
> Agreed but I thought we already had a tool to handle at least some of
> this though? Certainly integration of that with the above is essential
> though.

This is what I get for my setup:

koen@dominion:/OE/tentacle/build$ bitbake-layers 
<string>:1: DeprecationWarning: Call to deprecated function bb.parse.parse_py.BBHandler.vars_from_file: Please use bb.parse.vars_from_file instead.
<string>:6: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
WARNING: Bitbake has not been run using the bitbake wrapper (scripts/bitbake); this is likely because your PATH has been altered from that normally set up by the poky-init-build-env script. Not using the wrapper may result in failures during package installation, so it is highly recommended that you set your PATH back so that the wrapper script is being executed.
WARNING: Pseudo is not functioning correctly, which will cause failures during package installation. Please check your configuration.
Parsing recipes..<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
done.
INFO: State of append files:
INFO: base-files_3.0.14.bb:
INFO:   /OE/tentacle/sources/meta-angstrom/recipes-core/base-files/base-files_3.0.14.bbappend
INFO: libusb-compat_0.1.3.bb:
INFO:   /OE/tentacle/sources/meta-openembedded/meta-oe/recipes-support/libusb/libusb-compat_0.1.3.bbappend
INFO: sysvinit_2.88dsf.bb:
INFO:   /OE/tentacle/sources/meta-angstrom/recipes-core/sysvinit/sysvinit_2.88dsf.bbappend
koen@dominion:/OE/tentacle/build$ 

regards,

Koen

Re: RFC: Layer tooling brainstorming

[Richard Purdie]

On Mon, 2011-04-18 at 09:33 +0200, Andreas Mueller wrote:
> On Monday 18 April 2011 08:17:01 Richard Purdie wrote:
> > Layer Tooling for OpenEmbedded
> > ==============================
> > 
> > As we move forward with encouraging the use of metadata layers, a number
> > of issues are appearing which imply the need for either infrastructure
> > improvements to the layer construction, or better tools to handle the
> > layers. The OS TSC discussed layers and layer tooling at the face to
> > face TSC meeting in San Francisco. Some example problems that were
> > identified that currently need addressing in some form are:
> > 
> > * There is no dependency information between layers (layer X depends on
> >   layer Y and Z)
> > * There is no version information present (layer X needs version A of
> >   layer Y)
> > * There is no layer "ABI" version number present in layers
> > * Layer specific sanity checks? (other types of sanity checks on Yocto
> >   1.1 roadmap)
> > * Ability to fetch layers from other repositories
> >   - Multi SCM?
> >   - Use bitbake fetcher?
> > * Ability to generate repositories representing composited layers (Poky,
> >   RP has hacky scripts)
> > * Tool to seperate patch into one for different composited parts
> > * Ability to include partial components of repositories (e.g. only
> >   beagleboard from meta-ti)
> > * Tool to merge (flatten) several layers into one
> > * Able to visualise dependencies between layers
> > * When bitbake prints banner of branch/commit, need to cover all layer
> >   components present
> > * bitbake -e on steroids to highlight differences due to layer changes?
> >   (long term goal)
> > 
> > Implementation thoughts:
> > 
> > * This layer tooling probably belongs at a higher level on the stack
> >   than bitbake itself
> > * Maybe need to split into "bootstrap" steps (e.g where pseduo is
> >   established, layers downloaded etc)
> > * Need some kind of config file defining layer's properties or at least
> >   enhance layer.conf significantly
> > * git submodules and repo should be looked at but due to specific needs
> >   and ability to control tool evolution, likely need own tool.
> > 
> > This hopefully summarises the ideas we discussed in person, we therefore
> > need to start looking at how to move forward from here with this but I
> > wanted to document this for starters!
> > 
> > Cheers,
> > 
> > Richard
> Afraid being killed I didn't dare to ask - but it's getting time to do so: What 
> exactly are layers in OE and what are their advantages?

Layers are the things that have also been called overlays or
collections. If neither of those things mean anything either, they're a
method of separating metadata and customisations into containers.

The concept itself has been around for a long time but we've recently
made enhancements to make it much more useful/usable.

http://www.yoctoproject.org/docs/bsp-guide/bsp-guide.html shows an
example layer.

Cheers,

Richard

Re: RFC: Layer tooling brainstorming

[Martin Jansa]

On Mon, Apr 18, 2011 at 10:01:59AM +0200, Koen Kooi wrote:
> 
> Op 18 apr 2011, om 09:09 heeft Richard Purdie het volgende geschreven:
> 
> > On Mon, 2011-04-18 at 08:50 +0200, Martin Jansa wrote:
> >> On Mon, Apr 18, 2011 at 07:17:01AM +0100, Richard Purdie wrote:
> >> 
> >> maybe I've missed something, but
> >> * Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in
> >>  upper layer was upgraded to foo_1.1.bb
> >> 
> >> This can cause big problems when distribution explicitly enables
> >> something in .bbappend with EXTRA_OECONF, and someone updates all
> >> layers, builds foo_1.1.bb and installs it on targets before noticing
> >> that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding
> >> foo_1.1.bbappend to distro layer (to fix foo on targets), so I would
> >> prefer parsing error in this case (won't help if upper layer keeps both
> >> versions foo_1.0.bb and foo_1.1.bb ;/).
> >> 
> >> This should be combined with your 2nd point and person who changes
> >> versioned dependency on upper layer should check that all needed
> >> .bbappends still apply.
> > 
> > Agreed but I thought we already had a tool to handle at least some of
> > this though? Certainly integration of that with the above is essential
> > though.
> 
> This is what I get for my setup:
> 
> koen@dominion:/OE/tentacle/build$ bitbake-layers 
> <string>:1: DeprecationWarning: Call to deprecated function bb.parse.parse_py.BBHandler.vars_from_file: Please use bb.parse.vars_from_file instead.
> <string>:6: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
> WARNING: Bitbake has not been run using the bitbake wrapper (scripts/bitbake); this is likely because your PATH has been altered from that normally set up by the poky-init-build-env script. Not using the wrapper may result in failures during package installation, so it is highly recommended that you set your PATH back so that the wrapper script is being executed.
> WARNING: Pseudo is not functioning correctly, which will cause failures during package installation. Please check your configuration.
> Parsing recipes..<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
> <string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
> <string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
> done.
> INFO: State of append files:
> INFO: base-files_3.0.14.bb:
> INFO:   /OE/tentacle/sources/meta-angstrom/recipes-core/base-files/base-files_3.0.14.bbappend
> INFO: libusb-compat_0.1.3.bb:
> INFO:   /OE/tentacle/sources/meta-openembedded/meta-oe/recipes-support/libusb/libusb-compat_0.1.3.bbappend
> INFO: sysvinit_2.88dsf.bb:
> INFO:   /OE/tentacle/sources/meta-angstrom/recipes-core/sysvinit/sysvinit_2.88dsf.bbappend
> koen@dominion:/OE/tentacle/build$ 

This is great for distro maintainer who maintains also those required
revisions for upper layers, if distro building scripts (oebb.sh or our
SHR Makefile) allows such revision locking.

But I don't expect regular distro builder to do such test after every
git pull in some layer (that's why I think parsing error would be
better).

My workaround for revision locking is using branches oe-core-contrib/shr
meta-oe-contrib/shr from SHR Makefile, where I can stash some patches
not yet applied from pull request or even not yet sent in any pull
request and also control which upstream revision will be used (rebased
on).

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: RFC: Layer tooling brainstorming

[Paul Eggleton]

Some of the ideas I've been thinking of for layer tools, other than what's 
been mentioned already:

* Show easily which recipes are being used and which are overridden. We have 
bitbake-layers, but this clearly needs some extension. I think long term we 
would want to be able to analyse overrides accross a set of layers to figure 
out where common stuff in non-core layers needs pushing up to oe-core.

* Allow layer maintainers to easily pull a version of a recipe into their own 
layer if it's about to be removed from oe-core - with some complicated recipes 
that could be painful/annoying to pick out all the necessary files by hand.

* Recipe maintenance tools that take advantage of bitbake's parsing logic to 
aid blanket updates (e.g. variable renames). This should help mitigate the 
increased difficulty of having recipes spread out over multiple repositories 
when doing these kinds of updates.

* Some kind of central registration of layers. I'm guessing a wiki page would 
do for starters but something with a database behind it would enable us to 
show "live" metadata such as when the layer was last updated, info read from 
layer.conf etc.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre (UK)

Re: RFC: Layer tooling brainstorming

[Chris Larson]

On Mon, Apr 18, 2011 at 6:59 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> * Show easily which recipes are being used and which are overridden. We have
> bitbake-layers, but this clearly needs some extension. I think long term we
> would want to be able to analyse overrides accross a set of layers to figure
> out where common stuff in non-core layers needs pushing up to oe-core.

I think we'll definitely need multiple views into the information.

> * Allow layer maintainers to easily pull a version of a recipe into their own
> layer if it's about to be removed from oe-core - with some complicated recipes
> that could be painful/annoying to pick out all the necessary files by hand.

The newcollection class in OE does this.  It's not perfect, but it
gets you 90% of the way there.  I was thinking it may be useful as a
separate script, indeed.

> * Recipe maintenance tools that take advantage of bitbake's parsing logic to
> aid blanket updates (e.g. variable renames). This should help mitigate the
> increased difficulty of having recipes spread out over multiple repositories
> when doing these kinds of updates.

This is non-trivial, but doable given zecke's ast implementation.  I
expect we need an API function in the parse package to parse a
string/file and hand back the AST nodes, which we could then
manipulate, and write code to generate the file back out of the ast.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: RFC: Layer tooling brainstorming

[Jeremy Puhlman]

So we(montavista) basically have run through this process before, though
using collections implementation. Difference in overall implementation
aside, the issues are largely the same.

> * This layer tooling probably belongs at a higher level on the stack 
>   than bitbake itself

This is most definately true, and has worked well for us. We have a set
of content tools that describes the distribution of each of the
collections and prebuilt componets that get shipped to the end user.

> * Maybe need to split into "bootstrap" steps (e.g where pseduo is 
>   established, layers downloaded etc)

Internally we extended collections to act upon remote collections of
with any kind fetching bitbake supports. The collections implementation
has a "built-in" bootstrap step. The actual nice thing about the layers
implimentation, is you could implement the remote fetching with out
having the re-execution of bitbake. That being said there is a clear
need for a method of adding bootstrap steps to address things like
pseudo. I think trying to put something extensible here would probably
be good.

Re: RFC: Layer tooling brainstorming

[Jeremy Puhlman]

> * Maybe need to split into "bootstrap" steps (e.g where pseduo is
>   established, layers downloaded etc)

So for the most part, the last product we shipped was largely arranged
in this manner. We were using collections, but in many cases layers is
just a reimplementation at a lower level(which is definitely goodness).
During ELC I had a number of discussions primarily around remote layer
fetch and most of the responses were meh, or that they wanted to make
the tool external to bitbake. So after the conference was over I went
ahead and worked out a version of the implementation  that could be done
either in a include or in the bitbake code itself. The version in the
git repo below is obviously just the bitbake implementation.

I have a rough implementation largely based off of collection.inc + some
enhancements we made internally for remote collection fetching. Some of
it is messy(setting up the early fetching vars with out using another
config), but for the ones I list in the patch header it appears to work
well.

I do not believe that this is complete, but it is probably a reasonable
starting point.

Pull URL: git://gitorious.org/jpwork/bb.git
Branch: remote-layering
Browse: https://gitorious.org/jpwork/bb/trees/remote-layering


---
Add support for remote layering.

The bulk of this patch is based on Chris Larson's collection.inc,
but refactored for use inside bitbake, and for layers.

Adds two new configuration options:

LAYER_UNPACKDIR - directory to unpack downloaded layers for use in
configuration.
LAYER_REMOTE_STASH - location to store downloaded layers for use in
configuration.

Layers can be specified in the current manner:
BBLAYERS = " \
   /path/to/layer/one \
   /path/to/layer/two \
"

Or you can specify uris:

BBLAYERS = " \
	http://url.to.my.tarball/tarball.tar.gz \
	git://gitserver.com/git/repo;proto=http \
        /path/to/layer/three \
        file:///path/to/layer/four \
"

Currently there is a single layer option, that can be added to a uri,
layerBase=<subpath>.
This option would be used to specify if a layer starts somewhere other
then the base of the
tarball/scm repository. For example if you wanted to add oe-core you
would do:

BBLAYERS  =
"git://git.openembedded.org/openembedded-core;protocol=git;tag=master;layerBase=meta"

Currently for fetch the following uris should work  :
'git','http','https','ftp','file'
Currently for fetch2 the following uris should work :
'git','http','https','ftp','file','svn'

There are certainly some ugly bits in here, and there are certainly lots
of places to improve,
but this gets a good chunk of the base support in.

Signed-off-by: Jeremy Puhlman <jpuhlman@mvista.com>
Diff rendering mode:inlineside by side