RFC: Improve our default conf file setup

RFC: Improve our default conf file setup

[Richard Purdie]

Hi,

I've been thinking for a while that our "look for a conf/bitbake.conf"
approach to finding our configuration is rather dated and inflexible.
Talking to others I think they feel the same way and its time to take a
step back and think about what we actually need. I've tried to do this
and I have a proposal based on what I found.

The fundamental problem currently is that the build directory is not in
control of what bitbake does. We require BBPATH to be set to some magic
value, find bitbake.conf, this transfers control back to a single file
(local.conf) which then further influences things further. The build
directory has to be combined with the right BBPATH setting.

Furthermore, we have the powerful overlay extension mechanism but when
adding an overlay, you have to change BBPATH, BBFILES and a load of
other things to correctly integrate any overlay into a build.

So taking a step back, what's needed is for the build directory to have
the basic configuration contained within it and no need for some magic
BBPATH. Overlays should ship with configuration information attached to
them.

I therefore propose that before conf/bitbake.conf, bitbake looks for a
conf/bblayers.conf. This file sets a variable BBLAYERS.

BBLAYERS is simply a list of overlay directories to include for the
given build directory.

For *each* overlay listed, bitbake will then include conf/layer.conf.
This is the main change in behaviour for bitbake since normally only one
file of a given name would ever be included but I think this makes sense
to give us some new functionality.

These layer.conf files are free to do whatever they need such as adding
paths to BBPATH, BBFILES and so on (I see new variables being added
which is to be encouraged e.g. extra site files). Experimenting, its
obvious the one thing you need in a layer.conf file is the layer
directory name so I've let bitbake set this in the LAYERDIR variable.

LAYERDIR is a tricky thing since you always want to do immediate
expansion on it since it will change later. This is going to catch
people out but so be it, it works really well.

The nice thing about this approach is that its in keeping with the way
bitbake works, but its powerful and easily extensible and uses the
existing configuration syntax, parser and so on.

I wrote a patch for Poky illustrating how this thing could be used which
is included below.

Its also totally backwards compatible. If bblayers.conf doesn't exist,
it uses the old behaviour and you can mix and match them if you're
careful too.

Does anyone have any feedback on this approach?

Cheers,

Richard


diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 5f9becc..002dfa7 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -87,10 +87,7 @@ class BBCooker:
 
         bb.data.inheritFromOS(self.configuration.data)
 
-        for f in self.configuration.file:
-            self.parseConfigurationFile( f )
-
-        self.parseConfigurationFile( os.path.join( "conf", "bitbake.conf" ) )
+        self.parseConfigurationFiles(self.configuration.file)
 
         if not self.configuration.cmd:
             self.configuration.cmd = bb.data.getVar("BB_DEFAULT_TASK", self.configuration.data, True) or "build"
@@ -527,9 +524,27 @@ class BBCooker:
         else:
             shell.start( self )
 
-    def parseConfigurationFile( self, afile ):
+    def parseConfigurationFiles(self, files):
         try:
-            self.configuration.data = bb.parse.handle( afile, self.configuration.data )
+            data = self.configuration.data
+            for f in files:
+                data = bb.parse.handle(f, data)
+
+            layerconf = os.path.join(os.getcwd(), "conf", "bblayers.conf")
+            if os.path.exists(layerconf):
+                bb.msg.debug(2, bb.msg.domain.Parsing, "Found bblayers.conf (%s)" % layerconf)
+                data = bb.parse.handle(layerconf, data)
+
+                layers = (bb.data.getVar('BBLAYERS', data, True) or "").split()
+
+                for layer in layers:
+                    bb.msg.debug(2, bb.msg.domain.Parsing, "Adding layer %s" % layer)
+                    bb.data.setVar('LAYERDIR', layer, data)
+                    data = bb.parse.handle(os.path.join(layer, "conf", "layer.conf"), data)
+
+            data = bb.parse.handle(os.path.join("conf", "bitbake.conf"), data )
+
+            self.configuration.data = data
 
             # Handle any INHERITs and inherit the base class
             inherits  = ["base"] + (bb.data.getVar('INHERIT', self.configuration.data, True ) or "").split()
@@ -546,9 +561,9 @@ class BBCooker:
             bb.event.fire(bb.event.ConfigParsed(), self.configuration.data)
 
         except IOError, e:
-            bb.msg.fatal(bb.msg.domain.Parsing, "Error when parsing %s: %s" % (afile, str(e)))
+            bb.msg.fatal(bb.msg.domain.Parsing, "Error when parsing %s: %s" % (files, str(e)))
         except bb.parse.ParseError, details:
-            bb.msg.fatal(bb.msg.domain.Parsing, "Unable to parse %s (%s)" % (afile, details) )
+            bb.msg.fatal(bb.msg.domain.Parsing, "Unable to parse %s (%s)" % (files, details) )
 
     def handleCollections( self, collections ):
         """Handle collections"""
diff --git a/build/conf/bblayers.conf.sample b/build/conf/bblayers.conf.sample
new file mode 100644
index 0000000..0b56d10
--- /dev/null
+++ b/build/conf/bblayers.conf.sample
@@ -0,0 +1,2 @@
+BBFILES ?= ""
+BBLAYERS = "${OEROOT}/meta ${OEROOT}/meta-moblin"
diff --git a/meta-moblin/conf/layer.conf b/meta-moblin/conf/layer.conf
new file mode 100644
index 0000000..e6d3b49
--- /dev/null
+++ b/meta-moblin/conf/layer.conf
@@ -0,0 +1,5 @@
+# We have a conf and classes directory, add to BBPATH
+BBPATH := "${BBPATH}${LAYERDIR}"
+
+# We have a packages directory, add to BBFILES
+BBFILES := "${BBFILES} ${LAYERDIR}/packages/*/*.bb"
diff --git a/meta/conf/layer.conf b/meta/conf/layer.conf
new file mode 100644
index 0000000..ba9c87b
--- /dev/null
+++ b/meta/conf/layer.conf
@@ -0,0 +1,8 @@
+# We have a conf and classes directory, add to BBPATH
+BBPATH := "${BBPATH}${LAYERDIR}"
+
+# We have a packages directory, add to BBFILES
+BBFILES := "${BBFILES} ${LAYERDIR}/packages/*/*.bb"
+
+
+

Re: RFC: Improve our default conf file setup

[Frans Meulenbroeks]

2010/2/16 Richard Purdie <rpurdie@rpsys.net>:
> Hi,
>
> I've been thinking for a while that our "look for a conf/bitbake.conf"
> approach to finding our configuration is rather dated and inflexible.
> Talking to others I think they feel the same way and its time to take a
> step back and think about what we actually need. I've tried to do this
> and I have a proposal based on what I found.
>
> The fundamental problem currently is that the build directory is not in
> control of what bitbake does. We require BBPATH to be set to some magic
> value, find bitbake.conf, this transfers control back to a single file
> (local.conf) which then further influences things further. The build
> directory has to be combined with the right BBPATH setting.
>
> Furthermore, we have the powerful overlay extension mechanism but when
> adding an overlay, you have to change BBPATH, BBFILES and a load of
> other things to correctly integrate any overlay into a build.
>
> So taking a step back, what's needed is for the build directory to have
> the basic configuration contained within it and no need for some magic
> BBPATH. Overlays should ship with configuration information attached to
> them.
>
> I therefore propose that before conf/bitbake.conf, bitbake looks for a
> conf/bblayers.conf. This file sets a variable BBLAYERS.
>
> BBLAYERS is simply a list of overlay directories to include for the
> given build directory.
>
> For *each* overlay listed, bitbake will then include conf/layer.conf.
> This is the main change in behaviour for bitbake since normally only one
> file of a given name would ever be included but I think this makes sense
> to give us some new functionality.
>
> These layer.conf files are free to do whatever they need such as adding
> paths to BBPATH, BBFILES and so on (I see new variables being added
> which is to be encouraged e.g. extra site files). Experimenting, its
> obvious the one thing you need in a layer.conf file is the layer
> directory name so I've let bitbake set this in the LAYERDIR variable.
>
> LAYERDIR is a tricky thing since you always want to do immediate
> expansion on it since it will change later. This is going to catch
> people out but so be it, it works really well.
>
> The nice thing about this approach is that its in keeping with the way
> bitbake works, but its powerful and easily extensible and uses the
> existing configuration syntax, parser and so on.
>
> I wrote a patch for Poky illustrating how this thing could be used which
> is included below.
>http://picasaweb.google.com/fransmeulenbroeks/Fosdem2010?authkey=Gv1sRgCPfSh7qcjv2siQE#5435938102383709010
> Its also totally backwards compatible. If bblayers.conf doesn't exist,
> it uses the old behaviour and you can mix and match them if you're
> careful too.
>
> Does anyone have any feedback on this approach?

Sounds quite nice.
Didn't study the class code, but it would be nice if within layer.conf
I could use a relative path, which then is turned into an absolute
path when the layer.conf file is read
That means layer.conf can become very standard wrt BBPATH etc. and you
can even move layers around.

Frans

Re: RFC: Improve our default conf file setup

[Frans Meulenbroeks]

2010/2/16 Frans Meulenbroeks <fransmeulenbroeks@gmail.com>:
> 2010/2/16 Richard Purdie <rpurdie@rpsys.net>:
>> Hi,
>>
>> I've been thinking for a while that our "look for a conf/bitbake.conf"
>> approach to finding our configuration is rather dated and inflexible.
>> Talking to others I think they feel the same way and its time to take a
>> step back and think about what we actually need. I've tried to do this
>> and I have a proposal based on what I found.
>>
>> The fundamental problem currently is that the build directory is not in
>> control of what bitbake does. We require BBPATH to be set to some magic
>> value, find bitbake.conf, this transfers control back to a single file
>> (local.conf) which then further influences things further. The build
>> directory has to be combined with the right BBPATH setting.
>>
>> Furthermore, we have the powerful overlay extension mechanism but when
>> adding an overlay, you have to change BBPATH, BBFILES and a load of
>> other things to correctly integrate any overlay into a build.
>>
>> So taking a step back, what's needed is for the build directory to have
>> the basic configuration contained within it and no need for some magic
>> BBPATH. Overlays should ship with configuration information attached to
>> them.
>>
>> I therefore propose that before conf/bitbake.conf, bitbake looks for a
>> conf/bblayers.conf. This file sets a variable BBLAYERS.
>>
>> BBLAYERS is simply a list of overlay directories to include for the
>> given build directory.
>>
>> For *each* overlay listed, bitbake will then include conf/layer.conf.
>> This is the main change in behaviour for bitbake since normally only one
>> file of a given name would ever be included but I think this makes sense
>> to give us some new functionality.
>>
>> These layer.conf files are free to do whatever they need such as adding
>> paths to BBPATH, BBFILES and so on (I see new variables being added
>> which is to be encouraged e.g. extra site files). Experimenting, its
>> obvious the one thing you need in a layer.conf file is the layer
>> directory name so I've let bitbake set this in the LAYERDIR variable.
>>
>> LAYERDIR is a tricky thing since you always want to do immediate
>> expansion on it since it will change later. This is going to catch
>> people out but so be it, it works really well.
>>
>> The nice thing about this approach is that its in keeping with the way
>> bitbake works, but its powerful and easily extensible and uses the
>> existing configuration syntax, parser and so on.
>>
>> I wrote a patch for Poky illustrating how this thing could be used which
>> is included below.
>>http://picasaweb.google.com/fransmeulenbroeks/Fosdem2010?authkey=Gv1sRgCPfSh7qcjv2siQE#5435938102383709010
>> Its also totally backwards compatible. If bblayers.conf doesn't exist,
>> it uses the old behaviour and you can mix and match them if you're
>> careful too.
>>
>> Does anyone have any feedback on this approach?
>
> Sounds quite nice.
> Didn't study the class code, but it would be nice if within layer.conf
> I could use a relative path, which then is turned into an absolute
> path when the layer.conf file is read
> That means layer.conf can become very standard wrt BBPATH etc. and you
> can even move layers around.
>
> Frans
>
Oops apolgies for accidently pasting in the link to my fosdem pics.
FM

Re: RFC: Improve our default conf file setup

[Richard Purdie]

On Tue, 2010-02-16 at 11:58 +0100, Frans Meulenbroeks wrote:
> Sounds quite nice.
> Didn't study the class code, but it would be nice if within layer.conf
> I could use a relative path, which then is turned into an absolute
> path when the layer.conf file is read

This is what the LAYERDIR variable gives us. 

Having relative paths would lose all context outside the layer.conf
file. We could hardcode a list of variables that needed to be processed
and so on but LAYERDIR removes all that complexity whilst still letting
you move things around.

> That means layer.conf can become very standard wrt BBPATH etc. and you
> can even move layers around.

You should be able to do that with my proposal, the only file that would
need changing is bblayers.conf, not the layers themselves.

Cheers,

Richard

Re: RFC: Improve our default conf file setup

[Chris Larson]

On Tue, Feb 16, 2010 at 4:49 AM, Richard Purdie <rpurdie@rpsys.net> wrote:

> On Tue, 2010-02-16 at 11:58 +0100, Frans Meulenbroeks wrote:
> > Sounds quite nice.
> > Didn't study the class code, but it would be nice if within layer.conf
> > I could use a relative path, which then is turned into an absolute
> > path when the layer.conf file is read
>
> This is what the LAYERDIR variable gives us.
>
> Having relative paths would lose all context outside the layer.conf
> file. We could hardcode a list of variables that needed to be processed
> and so on but LAYERDIR removes all that complexity whilst still letting
> you move things around.
>
> > That means layer.conf can become very standard wrt BBPATH etc. and you
> > can even move layers around.
>
> You should be able to do that with my proposal, the only file that would
> need changing is bblayers.conf, not the layers themselves.
>

Can you explain what the use case is for needing this much control?  I find
it unlikely that it would be problematic to simply add each layer as a full
path to a list of layers in a single file, and leave it at that (i.e.
bblayers.conf).  If site files aren't in the layer, they won't be used.  If
recipes aren't around, they won't be used.  It seems a bit silly if every
layer.conf ends up being a copy of every other layer.conf.  And this sort of
blind copying tends to lead to problems and cruft (just look at distros that
copy angstrom).
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: RFC: Improve our default conf file setup

[Richard Purdie]

On Tue, 2010-02-16 at 07:16 -0700, Chris Larson wrote:
> On Tue, Feb 16, 2010 at 4:49 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
> 
> > On Tue, 2010-02-16 at 11:58 +0100, Frans Meulenbroeks wrote:
> > > Sounds quite nice.
> > > Didn't study the class code, but it would be nice if within layer.conf
> > > I could use a relative path, which then is turned into an absolute
> > > path when the layer.conf file is read
> >
> > This is what the LAYERDIR variable gives us.
> >
> > Having relative paths would lose all context outside the layer.conf
> > file. We could hardcode a list of variables that needed to be processed
> > and so on but LAYERDIR removes all that complexity whilst still letting
> > you move things around.
> >
> > > That means layer.conf can become very standard wrt BBPATH etc. and you
> > > can even move layers around.
> >
> > You should be able to do that with my proposal, the only file that would
> > need changing is bblayers.conf, not the layers themselves.
> 
> Can you explain what the use case is for needing this much control?  I find
> it unlikely that it would be problematic to simply add each layer as a full
> path to a list of layers in a single file, and leave it at that (i.e.
> bblayers.conf).  If site files aren't in the layer, they won't be used.  If
> recipes aren't around, they won't be used.  It seems a bit silly if every
> layer.conf ends up being a copy of every other layer.conf.  And this sort of
> blind copying tends to lead to problems and cruft (just look at distros that
> copy angstrom).

Several reasons:

a) I like the idea of a given overlay shipping with a description in
some form of what it contains. Its intuitive.
b) I don't want to hardcode behaviour in bitbake (see c/d)
c) Are recipes in a recipes or a packages directory?
d) How deep are the paths to the .bb files?
e) What priority are assigned to the layers?
f) Should a given layer append or prepend to BBPATH? 

It also means that if we add some kind of new functionality we have to
update bitbake to add the appropriate variables, or add horrible
anonymous methods to poke around BBLAYERS.

When the alternative is a single .conf file describing a layer, I know
which I prefer even if most might be two lines of boilerplate.

I was also thinking of your recursive bitbake 'abomination' ;-). With
this new functionality, its possible some code in one of the layers
could check ahead and checkout other layers if it noticed they were
missing. We'd need some additional code for that but its an idea in the
back of my mind.

Cheers,

Richard

Re: RFC: Improve our default conf file setup

[Frans Meulenbroeks]

2010/2/16 Richard Purdie <rpurdie@rpsys.net>:
> On Tue, 2010-02-16 at 07:16 -0700, Chris Larson wrote:
>> On Tue, Feb 16, 2010 at 4:49 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
>>
>> > On Tue, 2010-02-16 at 11:58 +0100, Frans Meulenbroeks wrote:
>> > > Sounds quite nice.
>> > > Didn't study the class code, but it would be nice if within layer.conf
>> > > I could use a relative path, which then is turned into an absolute
>> > > path when the layer.conf file is read
>> >
>> > This is what the LAYERDIR variable gives us.
>> >
>> > Having relative paths would lose all context outside the layer.conf
>> > file. We could hardcode a list of variables that needed to be processed
>> > and so on but LAYERDIR removes all that complexity whilst still letting
>> > you move things around.
>> >
>> > > That means layer.conf can become very standard wrt BBPATH etc. and you
>> > > can even move layers around.
>> >
>> > You should be able to do that with my proposal, the only file that would
>> > need changing is bblayers.conf, not the layers themselves.
>>
>> Can you explain what the use case is for needing this much control?  I find
>> it unlikely that it would be problematic to simply add each layer as a full
>> path to a list of layers in a single file, and leave it at that (i.e.
>> bblayers.conf).  If site files aren't in the layer, they won't be used.  If
>> recipes aren't around, they won't be used.  It seems a bit silly if every
>> layer.conf ends up being a copy of every other layer.conf.  And this sort of
>> blind copying tends to lead to problems and cruft (just look at distros that
>> copy angstrom).
>
> Several reasons:
>
> a) I like the idea of a given overlay shipping with a description in
> some form of what it contains. Its intuitive.
> b) I don't want to hardcode behaviour in bitbake (see c/d)
> c) Are recipes in a recipes or a packages directory?
> d) How deep are the paths to the .bb files?
> e) What priority are assigned to the layers?
> f) Should a given layer append or prepend to BBPATH?
>
> It also means that if we add some kind of new functionality we have to
> update bitbake to add the appropriate variables, or add horrible
> anonymous methods to poke around BBLAYERS.
>
> When the alternative is a single .conf file describing a layer, I know
> which I prefer even if most might be two lines of boilerplate.
>
> I was also thinking of your recursive bitbake 'abomination' ;-). With
> this new functionality, its possible some code in one of the layers
> could check ahead and checkout other layers if it noticed they were
> missing. We'd need some additional code for that but its an idea in the
> back of my mind.
>
> Cheers,
>
> Richard

We could also use the layering mechanism to give distro's their own
layer. That way it is clear what files belong to the distro and what
files are generic.

Frans.

Re: RFC: Improve our default conf file setup

[Marcin Juszkiewicz]

Dnia wtorek, 16 lutego 2010 o 16:36:44 Richard Purdie napisał(a):


> c) Are recipes in a recipes or a packages directory?
> d) How deep are the paths to the .bb files?

BBFILES += "layer-top-dir" and BitBake will traverse subdirs and collect all 
recipes from there. Never mind are they in recipes/ packages/ stuff/ or other 
dir. It is working for quite long time now.

> e) What priority are assigned to the layers?

Priorities looks easy when you look at numbers. But problem arrives when you 
want layer3 to have higher priority then layer2 and they are not connected.

> f) Should a given layer append or prepend to BBPATH? 

Those two are important. I remember situation with BugLabs r1.4 overlays when 
I had to recreate BBPATH to get proper order of overlays.

Regards, 
-- 
JID:      hrw@jabber.org
Website:  http://marcin.juszkiewicz.com.pl/
LinkedIn: http://www.linkedin.com/in/marcinjuszkiewicz

Re: RFC: Improve our default conf file setup

[Chris Larson]

On Thu, Feb 18, 2010 at 2:09 AM, Marcin Juszkiewicz <
marcin@juszkiewicz.com.pl> wrote:

> Dnia wtorek, 16 lutego 2010 o 16:36:44 Richard Purdie napisał(a):
>
>
> > c) Are recipes in a recipes or a packages directory?
> > d) How deep are the paths to the .bb files?
>
> BBFILES += "layer-top-dir" and BitBake will traverse subdirs and collect
> all
> recipes from there. Never mind are they in recipes/ packages/ stuff/ or
> other
> dir. It is working for quite long time now.
>

Yeah, exactly.  And of course, you can utilize BBMASK to exclude obsolete,
etc.


> e) What priority are assigned to the layers?

 Priorities looks easy when you look at numbers. But problem arrives when
> you
> want layer3 to have higher priority then layer2 and they are not connected.
>
> > f) Should a given layer append or prepend to BBPATH?
>
> Those two are important. I remember situation with BugLabs r1.4 overlays
> when
> I had to recreate BBPATH to get proper order of overlays.
>

With COLLECTIONS (and BBLAYERS could work similarly), all the overlay
priorities and bbpath ordering are automatically determined based on the
order of entries in the variable.  Documented as highest-to-lowest.

Note that I'm not necessarily arguing against the use of a per-layer
configuration file, as this seems like it would be quite useful, but I
question the need for the boilerplate.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: RFC: Improve our default conf file setup

[Richard Purdie]

On Thu, 2010-02-18 at 09:13 -0700, Chris Larson wrote:
> With COLLECTIONS (and BBLAYERS could work similarly), all the overlay
> priorities and bbpath ordering are automatically determined based on the
> order of entries in the variable.  Documented as highest-to-lowest.
> 
> Note that I'm not necessarily arguing against the use of a per-layer
> configuration file, as this seems like it would be quite useful, but I
> question the need for the boilerplate.

Its certainly possible to put defaults in which would handle the
"standard" case. My main dislike is having to hardcode behaviour and
variable handing into bitbake in that case. 

The idea that adding an empty layer.conf file breaks a previously
working overlay seems counter intuitive too...

Cheers,

Richard

Re: RFC: Improve our default conf file setup

[Chris Larson]

On Thu, Feb 18, 2010 at 9:58 AM, Richard Purdie <rpurdie@rpsys.net> wrote:

> On Thu, 2010-02-18 at 09:13 -0700, Chris Larson wrote:
> > With COLLECTIONS (and BBLAYERS could work similarly), all the overlay
> > priorities and bbpath ordering are automatically determined based on the
> > order of entries in the variable.  Documented as highest-to-lowest.
> >
> > Note that I'm not necessarily arguing against the use of a per-layer
> > configuration file, as this seems like it would be quite useful, but I
> > question the need for the boilerplate.
>
> Its certainly possible to put defaults in which would handle the
> "standard" case. My main dislike is having to hardcode behaviour and
> variable handing into bitbake in that case.
>

Not hardcoding *any* behavior into bitbake is how we got into the mess that
we're in, and is likely the source of a large amount of confusion on the
part of our userbase, and is likely the cause of some of the performance
issues as well.  Flexibility is good, but there's a point at which the
benefits no longer outweigh the drawbacks.


> The idea that adding an empty layer.conf file breaks a previously
> working overlay seems counter intuitive too...
>

The layer itself deciding where it goes in the BBPATH is not obvious or nice
or clear to me the way it seems to be to you.  How can a given layer claim
to know where it should go in the bbpath, what its priority should be?  I
think it makes a great deal more sense to let the order/information in
bblayers.conf define that, as the user has direct control over that.  Can
you think of any cases where a given layer could possibly know better than
the user, or where defining the priority via BBLAYERS is insufficient?
 Regarding BBFILES, there's nothing to hardcode.  Add the layer to BBFILES,
let bitbake find the recipes, and you have BBMASK to remove the ones you
don't want.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: RFC: Improve our default conf file setup

[Richard Purdie]

On Fri, 2010-02-19 at 18:39 -0700, Chris Larson wrote:
> On Thu, Feb 18, 2010 at 9:58 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
> > On Thu, 2010-02-18 at 09:13 -0700, Chris Larson wrote:
> > > With COLLECTIONS (and BBLAYERS could work similarly), all the overlay
> > > priorities and bbpath ordering are automatically determined based on the
> > > order of entries in the variable.  Documented as highest-to-lowest.
> > >
> > > Note that I'm not necessarily arguing against the use of a per-layer
> > > configuration file, as this seems like it would be quite useful, but I
> > > question the need for the boilerplate.
> >
> > Its certainly possible to put defaults in which would handle the
> > "standard" case. My main dislike is having to hardcode behaviour and
> > variable handing into bitbake in that case.
> 
> Not hardcoding *any* behavior into bitbake is how we got into the mess that
> we're in,

In many ways, its helped make bitbake what it is IMO. Bitbake and OE
have a fairly clearly modularised structure where you can replace
individual elements and this is a major attraction of it. Yes it causes
some pain at times but it also makes many things possible too.

>  and is likely the source of a large amount of confusion on the
> part of our userbase, and is likely the cause of some of the performance
> issues as well.  Flexibility is good, but there's a point at which the
> benefits no longer outweigh the drawbacks.

The performance issues seem to stem from two things, the anonymous
methods and the way our data store works as far as I can tell. We could
do with finding a better way to handle the methods I agree but they also
have massive benefits too. 

As for confusion of the userbase, this i still more a documentation
problem. Those who really understand it don't document it. The
documentation I have written is for Poky and few people bother to read
it, even if most of it is generic :(. Starting to hardcode behaviour is
unlikely to lead to much improvement for the users IMO.

> > The idea that adding an empty layer.conf file breaks a previously
> > working overlay seems counter intuitive too...
> >
> 
> The layer itself deciding where it goes in the BBPATH is not obvious or nice
> or clear to me the way it seems to be to you.  How can a given layer claim
> to know where it should go in the bbpath, what its priority should be?

Think about how people are going to use layers. Its never likely that a
random collection of layers are going to work together without some
(possibly minor) effort. Layers are going to end up advertised as being
enhancements for OE, Poky or some specific choice.

The "policy" on where in BBPATH (prepend or append) and actual priority
numbers are going to be a policy decision for those projects, just like
the bitbake.conf.

I'm quite happy if a standard emerges. My point is that policy should
not be in bitbake if we can help it.

>   I
> think it makes a great deal more sense to let the order/information in
> bblayers.conf define that, as the user has direct control over that.  Can
> you think of any cases where a given layer could possibly know better than
> the user, or where defining the priority via BBLAYERS is insufficient?
>  Regarding BBFILES, there's nothing to hardcode.  Add the layer to BBFILES,
> let bitbake find the recipes, and you have BBMASK to remove the ones you
> don't want.

I disagree but its pointless to take this further and spoil what
otherwise is a useful patch, we'll just add the default behaviour you
describe.

Cheers,

Richard

Re: RFC: Improve our default conf file setup

[Chris Larson]

On Sat, Feb 20, 2010 at 3:58 PM, Richard Purdie <rpurdie@rpsys.net> wrote:

> On Fri, 2010-02-19 at 18:39 -0700, Chris Larson wrote:
> > On Thu, Feb 18, 2010 at 9:58 AM, Richard Purdie <rpurdie@rpsys.net>
> wrote:
> > > On Thu, 2010-02-18 at 09:13 -0700, Chris Larson wrote:
> > > > With COLLECTIONS (and BBLAYERS could work similarly), all the overlay
> > > > priorities and bbpath ordering are automatically determined based on
> the
> > > > order of entries in the variable.  Documented as highest-to-lowest.
> > > >
> > > > Note that I'm not necessarily arguing against the use of a per-layer
> > > > configuration file, as this seems like it would be quite useful, but
> I
> > > > question the need for the boilerplate.
> > >
> > > Its certainly possible to put defaults in which would handle the
> > > "standard" case. My main dislike is having to hardcode behaviour and
> > > variable handing into bitbake in that case.
> >
> > Not hardcoding *any* behavior into bitbake is how we got into the mess
> that
> > we're in,
>
> In many ways, its helped make bitbake what it is IMO. Bitbake and OE
> have a fairly clearly modularised structure where you can replace
> individual elements and this is a major attraction of it. Yes it causes
> some pain at times but it also makes many things possible too.
>
> >  and is likely the source of a large amount of confusion on the
> > part of our userbase, and is likely the cause of some of the performance
> > issues as well.  Flexibility is good, but there's a point at which the
> > benefits no longer outweigh the drawbacks.
>
> The performance issues seem to stem from two things, the anonymous
> methods and the way our data store works as far as I can tell. We could
> do with finding a better way to handle the methods I agree but they also
> have massive benefits too.
>
> As for confusion of the userbase, this i still more a documentation
> problem. Those who really understand it don't document it. The
> documentation I have written is for Poky and few people bother to read
> it, even if most of it is generic :(. Starting to hardcode behaviour is
> unlikely to lead to much improvement for the users IMO.
>
> > > The idea that adding an empty layer.conf file breaks a previously
> > > working overlay seems counter intuitive too...
> > >
> >
> > The layer itself deciding where it goes in the BBPATH is not obvious or
> nice
> > or clear to me the way it seems to be to you.  How can a given layer
> claim
> > to know where it should go in the bbpath, what its priority should be?
>
> Think about how people are going to use layers. Its never likely that a
> random collection of layers are going to work together without some
> (possibly minor) effort. Layers are going to end up advertised as being
> enhancements for OE, Poky or some specific choice.
>
> The "policy" on where in BBPATH (prepend or append) and actual priority
> numbers are going to be a policy decision for those projects, just like
> the bitbake.conf.
>
> I'm quite happy if a standard emerges. My point is that policy should
> not be in bitbake if we can help it.
>
> >   I
> > think it makes a great deal more sense to let the order/information in
> > bblayers.conf define that, as the user has direct control over that.  Can
> > you think of any cases where a given layer could possibly know better
> than
> > the user, or where defining the priority via BBLAYERS is insufficient?
> >  Regarding BBFILES, there's nothing to hardcode.  Add the layer to
> BBFILES,
> > let bitbake find the recipes, and you have BBMASK to remove the ones you
> > don't want.
>
> I disagree but its pointless to take this further and spoil what
> otherwise is a useful patch, we'll just add the default behaviour you
> describe.



I see a few problems here.  First, there's more to a notion of priority than
just 'prepend' and 'append'.  It's unlikely that a given layer will either
way to override everything else, or be overridden by everything else.
 Second, the final resulting BBPATH will end up depending upon ordering
anyway, because the prepend/appends will be applied based on the order of
the parsing of the bblayer.conf files.  Third, you seem to be retaining the
separation between configuration file / class priority (BBPATH) and
collection priority (BBFILE_COLLECTIONS, priority number).  Users see a
layer/overlay/collection as a single entity, not two components whose
interactions with everyone else may vary.

If you're going to use a priority numbering scheme or something similar,
defined by the layer, then everything should obey it, in my opinion, and
BBPATH_prepend/append is not sufficient to reflect that priority scheme.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: RFC: Improve our default conf file setup

[Richard Purdie]

On Sat, 2010-02-20 at 18:19 -0700, Chris Larson wrote:
> > I disagree but its pointless to take this further and spoil what
> > otherwise is a useful patch, we'll just add the default behaviour you
> > describe.
> 
> I see a few problems here.  First, there's more to a notion of priority than
> just 'prepend' and 'append'.  It's unlikely that a given layer will either
> way to override everything else, or be overridden by everything else.

I assume s/way/want/ above?

I agree there is more to it than that but you're arguing for a simple
default case which is just "prepend" or "append" effectively. I expect
most layers do want to override the preceding layer.

>  Second, the final resulting BBPATH will end up depending upon ordering
> anyway, because the prepend/appends will be applied based on the order of
> the parsing of the bblayer.conf files.

Most likely BBPATH and optionally BBFILE_COLLECTIONS will be appended or
prepended to which seems reasonable and BBFILES it doesn't matter.

>   Third, you seem to be retaining the
> separation between configuration file / class priority (BBPATH) and
> collection priority (BBFILE_COLLECTIONS, priority number).  Users see a
> layer/overlay/collection as a single entity, not two components whose
> interactions with everyone else may vary.

Well, the default case where there is no layer.conf file just needs to
become more complex and append to BBFILE_COLLECTIONS too then. You're
the one arguing for it :).

The default case should make them function as one I agree but where
there is a layer.conf I'd argue its up to that file to setup the righr
variables.

> If you're going to use a priority numbering scheme or something similar,
> defined by the layer, then everything should obey it, in my opinion, and
> BBPATH_prepend/append is not sufficient to reflect that priority scheme.

I propose no numbering scheme, just a convention for priorities e.g.:

Core metadata: 100
Supplementary metadata: 500
User metadata: 1000

So a user overlay would override supplementary metadata and so on
(assuming I have my numbers going the right way, I never can remember).


Take a step back again - we want a system where each overlay contains
data about what it contains and sets up bitbake's variables and any OE
specific variables to match the setup. The layer.conf in this case does
that well.

What it does badly is let the conf files know about each other so they
can fight over who does what, or let that be controlled from the
bblayers.conf file other than through ordering the BBLAYERS variable.
I'm open to ideas on how to work this better but think what I'm
proposing has ways of meeting all real world use cases?

Better ideas more than welcome!

Cheers,

Richard

Re: RFC: Improve our default conf file setup

[Chris Larson]

On Tue, Feb 23, 2010 at 10:55 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
> On Sat, 2010-02-20 at 18:19 -0700, Chris Larson wrote:
>> > I disagree but its pointless to take this further and spoil what
>> > otherwise is a useful patch, we'll just add the default behaviour you
>> > describe.
>>
>> I see a few problems here.  First, there's more to a notion of priority than
>> just 'prepend' and 'append'.  It's unlikely that a given layer will either
>> way to override everything else, or be overridden by everything else.
>
> I assume s/way/want/ above?
>
> I agree there is more to it than that but you're arguing for a simple
> default case which is just "prepend" or "append" effectively. I expect
> most layers do want to override the preceding layer.
>
>>  Second, the final resulting BBPATH will end up depending upon ordering
>> anyway, because the prepend/appends will be applied based on the order of
>> the parsing of the bblayer.conf files.
>
> Most likely BBPATH and optionally BBFILE_COLLECTIONS will be appended or
> prepended to which seems reasonable and BBFILES it doesn't matter.
>
>>   Third, you seem to be retaining the
>> separation between configuration file / class priority (BBPATH) and
>> collection priority (BBFILE_COLLECTIONS, priority number).  Users see a
>> layer/overlay/collection as a single entity, not two components whose
>> interactions with everyone else may vary.
>
> Well, the default case where there is no layer.conf file just needs to
> become more complex and append to BBFILE_COLLECTIONS too then. You're
> the one arguing for it :).
>
> The default case should make them function as one I agree but where
> there is a layer.conf I'd argue its up to that file to setup the righr
> variables.
>
>> If you're going to use a priority numbering scheme or something similar,
>> defined by the layer, then everything should obey it, in my opinion, and
>> BBPATH_prepend/append is not sufficient to reflect that priority scheme.
>
> I propose no numbering scheme, just a convention for priorities e.g.:
>
> Core metadata: 100
> Supplementary metadata: 500
> User metadata: 1000
>
> So a user overlay would override supplementary metadata and so on
> (assuming I have my numbers going the right way, I never can remember).
>
>
> Take a step back again - we want a system where each overlay contains
> data about what it contains and sets up bitbake's variables and any OE
> specific variables to match the setup. The layer.conf in this case does
> that well.
>
> What it does badly is let the conf files know about each other so they
> can fight over who does what, or let that be controlled from the
> bblayers.conf file other than through ordering the BBLAYERS variable.
> I'm open to ideas on how to work this better but think what I'm
> proposing has ways of meeting all real world use cases?

The point I'm attempting to make is, an individual layer doesn't know
enough to make the call.  "Override the preceding" is meaningless if
you don't know what the preceding *is*.  You haven't given the
individual layer that information.  You haven't moved the
responsibility from the user (BBLAYERS) to the layer (layer.conf) as
you seemed to be implying (by noting that a given layer will be
expected to be part of a set of layers, or operate against a specific
set of layers, you imply that the responsibility doesn't belong in the
hands of the user).  You're moving part of the responsibility, not all
of it, and what responsibility you do move over is useless without the
rest of the information.  You're giving them control without
knowledge, which, as far as I can tell, won't meet any additional use
cases that a simpler mechanism wouldn't, yet does increase the amount
of setup necessary, and, therefore, more potential points of failure.
In my opinion, you're better off leaving it entirely in the hands of
the user, or entirely in the hands of the layers, not a hodgepodge of
the two.

Now, to be clear, I'm not trying to rain on your parade here.  The
patch is an improvement, absolutely, and is in the right direction for
bitbake, imo, as users would find a more project-oriented solution to
be less confusing, but I think the design needs more thought and
discussion before it goes in.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: RFC: Improve our default conf file setup

[Marco Cavallini]

Richard Purdie ha scritto, Il 16/02/2010 11:34:
> Hi,
> 
> I've been thinking for a while that our "look for a conf/bitbake.conf"
> approach to finding our configuration is rather dated and inflexible.
> Talking to others I think they feel the same way and its time to take a
> step back and think about what we actually need. I've tried to do this
> and I have a proposal based on what I found.
> 
> The fundamental problem currently is that the build directory is not in
> control of what bitbake does. We require BBPATH to be set to some magic
> value, find bitbake.conf, this transfers control back to a single file
> (local.conf) which then further influences things further. The build
> directory has to be combined with the right BBPATH setting.
> 
> Furthermore, we have the powerful overlay extension mechanism but when
> adding an overlay, you have to change BBPATH, BBFILES and a load of
> other things to correctly integrate any overlay into a build.
> 
> So taking a step back, what's needed is for the build directory to have
> the basic configuration contained within it and no need for some magic
> BBPATH. Overlays should ship with configuration information attached to
> them.
> 
> I therefore propose that before conf/bitbake.conf, bitbake looks for a
> conf/bblayers.conf. This file sets a variable BBLAYERS.
> 
> BBLAYERS is simply a list of overlay directories to include for the
> given build directory.
> 
> For *each* overlay listed, bitbake will then include conf/layer.conf.
> This is the main change in behaviour for bitbake since normally only one
> file of a given name would ever be included but I think this makes sense
> to give us some new functionality.
> 
> These layer.conf files are free to do whatever they need such as adding
> paths to BBPATH, BBFILES and so on (I see new variables being added
> which is to be encouraged e.g. extra site files). Experimenting, its
> obvious the one thing you need in a layer.conf file is the layer
> directory name so I've let bitbake set this in the LAYERDIR variable.
> 
> LAYERDIR is a tricky thing since you always want to do immediate
> expansion on it since it will change later. This is going to catch
> people out but so be it, it works really well.
> 
> The nice thing about this approach is that its in keeping with the way
> bitbake works, but its powerful and easily extensible and uses the
> existing configuration syntax, parser and so on.
> 
> I wrote a patch for Poky illustrating how this thing could be used which
> is included below.
> 
> Its also totally backwards compatible. If bblayers.conf doesn't exist,
> it uses the old behaviour and you can mix and match them if you're
> careful too.
> 
> Does anyone have any feedback on this approach?
> 
> Cheers,
> 

Richard,
I agree with this proposal, also beacuse I have been doing an intensive
use of overlays.
But I'd prefer a single configuration file containing everything rather
than a myriad of small ones.
BTW the totally backwards compatibility makes me comfortable with your
proposal.

Cheers,

--
Marco Cavallini | KOAN sas | Bergamo - Italia
 embedded and real-time software engineering
   Atmel third party certified consultant
Phone:+39-035-255.235 - Fax:+39-178-22.39.748
      http://www.KoanSoftware.com
         http://www.KaeilOS.com

Re: RFC: Improve our default conf file setup

[Richard Purdie]

On Tue, 2010-02-16 at 12:15 +0100, Marco Cavallini wrote:
> I agree with this proposal, also beacuse I have been doing an intensive
> use of overlays.
> But I'd prefer a single configuration file containing everything rather
> than a myriad of small ones.

The trouble with this is that it doesn't let us escape from the current
local.conf situation.

The idea in the proposal is that each file has a specific purpose and
associates the data with the appropriate place. E.g. an overlay is in
the best position to know whether it has extra .bb files, .conf files,
site files and so on. A build directory is in the best position to know
which overlays should be included for a given build and so on.

Cheers,

Richard