Layer priorities influencing default version selection

Layer priorities influencing default version selection

[Paul Eggleton]

Hi all,

Andrea pointed out a situation where he has seen the layer priority overriding 
version selection, and I've been able to confirm it.

Basically, if you have a recipe with a lower version in a layer with a higher 
priority it selects the lower version. What's more after digging further I 
found there were some rather anomalous interactions with the version logic and 
BBCLASSEXTEND. Here's an example using Poky:

1. Firstly, copy meta/recipes-support/curl to meta-yocto/recipes-support, then 
rename the version in meta-yocto so that its version is 7.20.0. At this point 
both meta/ and meta-yocto/ have the same layer priority of 5.

2.  "bitbake -s | grep ^curl" reports:
curl                                               :7.21.7-r0                          
curl-native                                        :7.21.7-r0                          
curl-nativesdk                                     :7.21.7-r0                          

3. Now increase the layer priority in meta-yocto/conf/layer.conf to 6, and run  
"bitbake -s | grep ^curl" again:
curl                                               :7.20.0-r0                          
curl-native                                        :7.21.7-r0                          
curl-nativesdk                                     :7.21.7-r0                          

So the latest version here is a lie, this is not the latest version available. 
Furthermore it seems not to have affected the BBCLASSEXTENDs.

4. Now add PREFERRED_VERSION_curl = "7.21.7" to conf/layer.conf and run 
"bitbake -s | grep ^curl" again:
curl                                 :7.20.0-r0                :7.21.7-r0
curl-native                    :7.21.7-r0                          
curl-nativesdk               :7.21.7-r0                          

So it can clearly see the other recipe, it just doesn't acknowledge it until 
you force the matter.

This is all rather undesirable behaviour IMHO - even if the BBCLASSEXTEND and 
reported "latest version available" issues were corrected, I think the policy 
of "latest version wins unless DEFAULT_PREFERENCE or PREFERRED_VERSION says 
otherwise" should not be affected by layer priority.

Thoughts?

Cheers,
Paul

(PS: this is not unique to Poky, Andrea and I first reproduced this with 
Angstrom using bitbake master.)

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] Layer priorities influencing default version selection

[Richard Purdie]

On Tue, 2011-08-02 at 12:26 +0100, Paul Eggleton wrote:
> Hi all,
> 
> Andrea pointed out a situation where he has seen the layer priority overriding 
> version selection, and I've been able to confirm it.
> 
> Basically, if you have a recipe with a lower version in a layer with a higher 
> priority it selects the lower version. What's more after digging further I 
> found there were some rather anomalous interactions with the version logic and 
> BBCLASSEXTEND. Here's an example using Poky:
> 
> 1. Firstly, copy meta/recipes-support/curl to meta-yocto/recipes-support, then 
> rename the version in meta-yocto so that its version is 7.20.0. At this point 
> both meta/ and meta-yocto/ have the same layer priority of 5.
> 
> 2.  "bitbake -s | grep ^curl" reports:
> curl                                               :7.21.7-r0                          
> curl-native                                        :7.21.7-r0                          
> curl-nativesdk                                     :7.21.7-r0                          
> 
> 3. Now increase the layer priority in meta-yocto/conf/layer.conf to 6, and run  
> "bitbake -s | grep ^curl" again:
> curl                                               :7.20.0-r0                          
> curl-native                                        :7.21.7-r0                          
> curl-nativesdk                                     :7.21.7-r0                          

This is clearly broken and needs to be consistent at the very least.

> So the latest version here is a lie, this is not the latest version available. 
> Furthermore it seems not to have affected the BBCLASSEXTENDs.
> 
> 4. Now add PREFERRED_VERSION_curl = "7.21.7" to conf/layer.conf and run 
> "bitbake -s | grep ^curl" again:
> curl                                 :7.20.0-r0                :7.21.7-r0
> curl-native                    :7.21.7-r0                          
> curl-nativesdk               :7.21.7-r0                          
> 
> So it can clearly see the other recipe, it just doesn't acknowledge it until 
> you force the matter.
> 
> This is all rather undesirable behaviour IMHO - even if the BBCLASSEXTEND and 
> reported "latest version available" issues were corrected, I think the policy 
> of "latest version wins unless DEFAULT_PREFERENCE or PREFERRED_VERSION says 
> otherwise" should not be affected by layer priority.
> 
> Thoughts?

We could do with clearly documenting this in the bitbake manual. I
suspect users would expect the highest version to win and we probably
should change the behaviour but I'm open to other opinions.

Cheers,

Richard

Re: [OE-core] Layer priorities influencing default version selection

[Phil Blundell]

On Tue, 2011-08-02 at 14:45 +0100, Richard Purdie wrote:
> We could do with clearly documenting this in the bitbake manual. I
> suspect users would expect the highest version to win and we probably
> should change the behaviour but I'm open to other opinions.

The original intent with BBFILE_COLLECTIONS was that the recipe from the
highest-priority collection would win even if it was an older version.
(The rationale for this was that the one in the higher priority
collection was the most likely to have local customisations, and you
wouldn't want those to suddenly be negated if a newer version popped up
in some upstream collection.)

I suspect the same logic probably does apply to layers to some extent
for the same reason.  If there is a situation where you are just trying
to aggregate the recipes together from a bunch of layers and you just
want the highest version from that set then my first reaction would be
that the layers should probably all have the same priority.  

Actually, to some extent I consider it a bit of a misfeature that the
layer priority is specified by the layer rather than by the user in
bblayers.conf, since this makes it harder to vary the stack-up order
without local hackery to the layer files.  For example, meta-oe
currently sets itself to a higher priority than oe-core, but I've found
that it generally seems to suit me better if meta-oe is actually the
lower-priority layer.

p.

Re: [OE-core] Layer priorities influencing default version selection

[Chris Larson]

On Tue, Aug 2, 2011 at 6:52 AM, Phil Blundell <philb@gnu.org> wrote:
> Actually, to some extent I consider it a bit of a misfeature that the
> layer priority is specified by the layer rather than by the user in
> bblayers.conf, since this makes it harder to vary the stack-up order
> without local hackery to the layer files.  For example, meta-oe
> currently sets itself to a higher priority than oe-core, but I've found
> that it generally seems to suit me better if meta-oe is actually the
> lower-priority layer.

Agreed. It also means that the priority knowledge is split between
layer.conf and bblayers.conf today, as config/class priority is
determined by order of entries in BBLAYERS, whereas recipe priority is
determined by layer.conf.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [OE-core] Layer priorities influencing default version selection

[Paul Eggleton]

On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
> as config/class priority is
> determined by order of entries in BBLAYERS, whereas recipe priority is
> determined by layer.conf.

Is that a good thing though? At the moment I'm not convinced that it is.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] Layer priorities influencing default version selection

[Chris Larson]

On Tue, Aug 2, 2011 at 7:21 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
> On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
>> as config/class priority is
>> determined by order of entries in BBLAYERS, whereas recipe priority is
>> determined by layer.conf.
>
> Is that a good thing though? At the moment I'm not convinced that it is.

I probably could have worded my reply better. I was pointing out the
fact that the order is determined in two places right now as being one
reason why I agreed with his statement that ordering determined by the
layer isn't ideal.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: [OE-core] Layer priorities influencing default version selection

[Paul Eggleton]

On Tuesday 02 August 2011 15:27:58 Chris Larson wrote:
> On Tue, Aug 2, 2011 at 7:21 AM, Paul Eggleton
> 
> <paul.eggleton@linux.intel.com> wrote:
> > On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
> >> as config/class priority is
> >> determined by order of entries in BBLAYERS, whereas recipe priority is
> >> determined by layer.conf.
> > 
> > Is that a good thing though? At the moment I'm not convinced that it is.
> 
> I probably could have worded my reply better. I was pointing out the
> fact that the order is determined in two places right now as being one
> reason why I agreed with his statement that ordering determined by the
> layer isn't ideal.

Well, all the same I'd like to have that question answered, maybe that's a 
different thread.

Back to Phil's original point though, I tend to think issues with meta-oe 
ought to be solved by fixing meta-oe - if some things in meta-oe are causing so 
much trouble that we have to down-grade the priority of the layer in order to 
squelch them, that's bad.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] Layer priorities influencing default version selection

[Mark Hatle]

At Wind River, in our non bitbake based build system, the way layers work is
based on ordering.

The last layer listed has the highest priority.  So whatever is specified in
that layer (lower or higher versions) wins.  All of the priority information is
selected based on the equivalent of the order of entries in the BBLAYERS.
However, unlike bitbake -- in the WR build system we can only have one version
of a package available at any time.

(There is a second level of ordering for configuration files.  For instance if
you are looking for foo.conf, it will search the highest priority layer first
and work it's way down.  If foo.conf includes itself, foo.conf, it will
automatically search starting with the next lower priority layer.  If you
include a differently named configuration file it starts back at the highest
priority layer.  This is what allows a layer to provide snippets of
configuration files. -- I'm not sure this is relevant to this conversation
though... but I thought I'd point out a characteristic of the ordering.)

--Mark

On 8/2/11 9:27 AM, Chris Larson wrote:
> On Tue, Aug 2, 2011 at 7:21 AM, Paul Eggleton
> <paul.eggleton@linux.intel.com> wrote:
>> On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
>>> as config/class priority is
>>> determined by order of entries in BBLAYERS, whereas recipe priority is
>>> determined by layer.conf.
>>
>> Is that a good thing though? At the moment I'm not convinced that it is.
> 
> I probably could have worded my reply better. I was pointing out the
> fact that the order is determined in two places right now as being one
> reason why I agreed with his statement that ordering determined by the
> layer isn't ideal.

Re: [OE-core] Layer priorities influencing default version selection

[Khem Raj]

On Aug 2, 2011, at 7:14 AM, Chris Larson <clarson@kergoth.com> wrote:

> On Tue, Aug 2, 2011 at 6:52 AM, Phil Blundell <philb@gnu.org> wrote:
>> Actually, to some extent I consider it a bit of a misfeature that the
>> layer priority is specified by the layer rather than by the user in
>> bblayers.conf, since this makes it harder to vary the stack-up order
>> without local hackery to the layer files.  For example, meta-oe
>> currently sets itself to a higher priority than oe-core, but I've found
>> that it generally seems to suit me better if meta-oe is actually the
>> lower-priority layer.
> 
> Agreed. It also means that the priority knowledge is split between
> layer.conf and bblayers.conf today, as config/class priority is
> determined by order of entries in BBLAYERS, whereas recipe priority is
> determined by layer.conf.


In my opinion the layer priority for all kind of meta data should be consistent and selected using bblayers.conf 


> -- 
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics
> 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel

Re: [OE-core] Layer priorities influencing default version selection

[Paul Eggleton]

On Tuesday 02 August 2011 16:35:12 Khem Raj wrote:
> On Aug 2, 2011, at 7:14 AM, Chris Larson <clarson@kergoth.com> wrote:
> > On Tue, Aug 2, 2011 at 6:52 AM, Phil Blundell <philb@gnu.org> wrote:
> >> Actually, to some extent I consider it a bit of a misfeature that the
> >> layer priority is specified by the layer rather than by the user in
> >> bblayers.conf, since this makes it harder to vary the stack-up order
> >> without local hackery to the layer files.  For example, meta-oe
> >> currently sets itself to a higher priority than oe-core, but I've found
> >> that it generally seems to suit me better if meta-oe is actually the
> >> lower-priority layer.
> > 
> > Agreed. It also means that the priority knowledge is split between
> > layer.conf and bblayers.conf today, as config/class priority is
> > determined by order of entries in BBLAYERS, whereas recipe priority is
> > determined by layer.conf.
> 
> In my opinion the layer priority for all kind of meta data should be
> consistent and selected using bblayers.conf

So I've thought about this more, and I don't think it is a good idea, for two 
reasons:

1) It increases the complexity by a huge amount. New users will have to 
understand how to set the priorities, and if someone reports a problem, we now 
not only need to know what layers they are using, we also need to know what 
mangling of the priorities they might have done. We'd just be introducing more 
potentially untested configurations (as if we don't have enough already.)

2) It just papers over real problems in meta-oe or any other layer you might 
choose to use on top. As I said, if meta-oe has stuff in it that is not 
working, it should be fixed or removed. Let's fix problems instead of finding 
ways of ignoring them.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] Layer priorities influencing default version selection

[Khem Raj]

On Thu, Aug 25, 2011 at 3:50 AM, Paul Eggleton
<paul.eggleton@linux.intel.com> wrote:
>>
>> In my opinion the layer priority for all kind of meta data should be
>> consistent and selected using bblayers.conf
>
> So I've thought about this more, and I don't think it is a good idea, for two
> reasons:
>
> 1) It increases the complexity by a huge amount. New users will have to
> understand how to set the priorities, and if someone reports a problem, we now
> not only need to know what layers they are using, we also need to know what
> mangling of the priorities they might have done. We'd just be introducing more
> potentially untested configurations (as if we don't have enough already.)

with layers we dont control policies of all the layers that may be used on top
so fixing meta-oe does not solve problem completely and we can not ask
for exclusive
recipes. People would want to override the recipes from other layers.

bitbake could report the layer the recipe comes from which can make it evident
or may be special command to inform the layer priorities. This will guide the
users to diagnose the problems quickly and help developers to identify
the issues
faster. There could be a complete bill of recipes printed for a given
target as well
so if someone wants to check all the recipes that were built.

>
> 2) It just papers over real problems in meta-oe or any other layer you might
> choose to use on top. As I said, if meta-oe has stuff in it that is not
> working, it should be fixed or removed. Let's fix problems instead of finding
> ways of ignoring them.
>

In any case I agree that problems should be fixed. However this does
not scale to all
layers and we can not police all the layers and we should not. We
should try to make it possible
for people to glue layers together without  issues.

Re: [OE-core] Layer priorities influencing default version selection

[Paul Eggleton]

On Thursday 25 August 2011 16:56:28 you wrote:
> with layers we dont control policies of all the layers that may be used on
> top so fixing meta-oe does not solve problem completely and we can not ask
> for exclusive  recipes. People would want to override the recipes from other
> layers.

Generally, the less duplication we have across layers the better; if a 
maintainer is duplicating a recipe they should have a very good reason to do 
so. We can avoid some of this duplication by making it easier to figure out 
where the right place for recipes is and doing everything we can encourage 
people to get stuff merged there. I'm not convinced we are doing very well on 
either count yet, but I expect this will improve over time.

> bitbake could report the layer the recipe comes from which can make it
> evident or may be special command to inform the layer priorities. This will
> guide the users to diagnose the problems quickly and help developers to
> identify the issues faster. There could be a complete bill of recipes
> printed for a given target as well so if someone wants to check all
> the recipes that were built.

I had not considered BitBake itself reporting these, that might be useful 
especially in the case of errors (although the full path to the recipe is 
already reported during the build, so you should be able to tell in most cases 
from that if you need to). Outputting a specific report of parsed recipes and 
bbappends as part of the build also might be useful.

FYI, bitbake-layers exists as a separate utility to answer a lot of these 
kinds of questions, and recently has become a lot more powerful - I would 
strongly recommend everyone check it out and if it's not able to report what 
you want then please tell me and I'll try to make it do so. I already have a 
few ideas for future improvements there which I will hopefully get time to 
look into in the Yocto 1.2 cycle.

> In any case I agree that problems should be fixed. However this does
> not scale to all layers and we can not police all the layers and we should
> not. We should try to make it possible for people to glue layers together
> without  issues.

How does merely being able to alter the priorities help here though? I could 
begin to understand if you were asking for some way to blacklist individual 
recipes in individual layers, but moving the priority of an entire layer is 
likely to have much more of an effect than just obscuring a few errant recipes 
that you don't want.

We can't "police" all the layers, no. But if those who maintain that layer 
can't respond to and resolve problems then they shouldn't be maintainers or 
those layers can't be considered well-maintained. When you add a layer to your 
build you are placing some trust in the maintainer of that layer; it is up to 
the maintainer to make sure they don't abuse that trust.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [OE-core] Layer priorities influencing default version selection

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3604 bytes --]

On Thu, Aug 25, 2011 at 05:58:23PM +0100, Paul Eggleton wrote:
> On Thursday 25 August 2011 16:56:28 you wrote:
> > with layers we dont control policies of all the layers that may be used on
> > top so fixing meta-oe does not solve problem completely and we can not ask
> > for exclusive  recipes. People would want to override the recipes from other
> > layers.
> 
> Generally, the less duplication we have across layers the better; if a 
> maintainer is duplicating a recipe they should have a very good reason to do 
> so. We can avoid some of this duplication by making it easier to figure out 
> where the right place for recipes is and doing everything we can encourage 
> people to get stuff merged there. I'm not convinced we are doing very well on 
> either count yet, but I expect this will improve over time.
> 
> > bitbake could report the layer the recipe comes from which can make it
> > evident or may be special command to inform the layer priorities. This will
> > guide the users to diagnose the problems quickly and help developers to
> > identify the issues faster. There could be a complete bill of recipes
> > printed for a given target as well so if someone wants to check all
> > the recipes that were built.
> 
> I had not considered BitBake itself reporting these, that might be useful 
> especially in the case of errors (although the full path to the recipe is 
> already reported during the build, so you should be able to tell in most cases 
> from that if you need to). Outputting a specific report of parsed recipes and 
> bbappends as part of the build also might be useful.
> 
> FYI, bitbake-layers exists as a separate utility to answer a lot of these 
> kinds of questions, and recently has become a lot more powerful - I would 
> strongly recommend everyone check it out and if it's not able to report what 
> you want then please tell me and I'll try to make it do so. I already have a 
> few ideas for future improvements there which I will hopefully get time to 
> look into in the Yocto 1.2 cycle.
> 
> > In any case I agree that problems should be fixed. However this does
> > not scale to all layers and we can not police all the layers and we should
> > not. We should try to make it possible for people to glue layers together
> > without  issues.
> 
> How does merely being able to alter the priorities help here though? I could 
> begin to understand if you were asking for some way to blacklist individual 
> recipes in individual layers, but moving the priority of an entire layer is 
> likely to have much more of an effect than just obscuring a few errant recipes 
> that you don't want.

Well in this I agree with khem that it would be more consistent to use
priority from bblayers.conf. You're right that changing priority of an
entire layer is having too big impact, but whoever is writing his
bblayers.conf has better information about what he needs/wants from
particular layer then layer maintainer who has no clue in which layer
stack his layer will end up.

> We can't "police" all the layers, no. But if those who maintain that layer 
> can't respond to and resolve problems then they shouldn't be maintainers or 
> those layers can't be considered well-maintained. When you add a layer to your 
> build you are placing some trust in the maintainer of that layer; it is up to 
> the maintainer to make sure they don't abuse that trust.

And with priority in bblayers.conf I can say how much I trust him.

my 2c

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]