RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

[Rod Whitby]

Whenever I build virtual/kernel, and for some reason (perhaps a git
fetcher problem, or a patch doesn't apply cleanly) it doesn't build,
bitbake decides to then go and build every other kernel under the sun
trying to complete virtual/kernel.

This is undesirable to the extreme, but I realise that there might have
in the history of OE been a good reason for that behaviour.

My proposal is that bitbake is changed so that if PREFERRED_PROVIDER is
defined, then bitbake does *not* try other providers.  Only if
PREFERRED_PROVIDER is *not* defined should bitbake ever try more than
one package.

Objections?

Here is the IRC log of the discussion with RP about this issue:

<rwhitby> RP: a question for you
<rwhitby> when using virtual/kernel, and your PREFERRED kernel fails to
build, why does bitbake then go and build every other kernel under the sun?
<rwhitby> (and is there a way to stop that happening)
<RP> rwhitby: That is the traditional bitbake behaviour
<RP> rwhitby: I don't actually agree with it but changing it would
breakbacwards compatibilty
<RP> I guess it was once a feature but its turned out no to be so useful
<RP> We should make that behaviour configurable. It only happens with
the -k option anyway
<rwhitby> RP: what was the use case in which it was useful, when
PREFERRED_PROVIDER was defined?
<RP> rwhitby: The idea was it would allow builds to complete rather than
fail
<rwhitby> I can understand it if PREFERRED_PROVIDER is not defined, but
cannot think of a good reason why bitbake should go against explicit
directions for a PREFERRED_PROVIDER
<RP> rwhitby: It comes does to the variable name - "PREFERRED"
<RP> Would it be useful to change the design, I say yes
<RP> but there is history there and as the maintainer I need to be
careful with this
<rwhitby> understood
<rwhitby> can we make it configurable and configure to not do it by
default ?
<RP> We can make it configurable
<RP> I would want to see a discussion about defaults
<RP> Feel free to start that on the bitbake+OE list
<RP> I'm actually in favour of changing it
<RP> but it must be discussed
<rwhitby> nod

-- Rod

Re: RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

[Phil Blundell]

On Sat, 2008-08-30 at 17:40 +0930, Rod Whitby wrote:
> Whenever I build virtual/kernel, and for some reason (perhaps a git
> fetcher problem, or a patch doesn't apply cleanly) it doesn't build,
> bitbake decides to then go and build every other kernel under the sun
> trying to complete virtual/kernel.

It shouldn't be doing that in the first place.  If PREFERRED_PROVIDER is
set and that package is considered statically buildable, bitbake will
try to build it.  If PREFERRED_PROVIDER names a package which is
statically unbuildable, bitbake will proceed as if that variable wasn't
set at all and select one according to its other rules.  In either case,
once it has made its determination, it will try to build that one
package and no other.  If the selected package turns out to be
dynamically unbuildable (which would include the reasons you mentioned
above) then it will stop.

So, it sounds like the behaviour you are experiencing is a bug rather
than a feature.

p.

Re: RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

[Rod Whitby]

Phil Blundell wrote:
> On Sat, 2008-08-30 at 17:40 +0930, Rod Whitby wrote:
>> Whenever I build virtual/kernel, and for some reason (perhaps a git
>> fetcher problem, or a patch doesn't apply cleanly) it doesn't build,
>> bitbake decides to then go and build every other kernel under the sun
>> trying to complete virtual/kernel.
> 
> It shouldn't be doing that in the first place.  If PREFERRED_PROVIDER is
> set and that package is considered statically buildable, bitbake will
> try to build it.  If PREFERRED_PROVIDER names a package which is
> statically unbuildable, bitbake will proceed as if that variable wasn't
> set at all and select one according to its other rules.  In either case,
> once it has made its determination, it will try to build that one
> package and no other.  If the selected package turns out to be
> dynamically unbuildable (which would include the reasons you mentioned
> above) then it will stop.
> 
> So, it sounds like the behaviour you are experiencing is a bug rather
> than a feature.

Ah, I forgot to say that the circumstances in which I see this are
probably coincident with when I invoke bitbake with -k.

-- Rod

Re: RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

[Phil Blundell]

On Sat, 2008-08-30 at 19:24 +0930, Rod Whitby wrote:
> Ah, I forgot to say that the circumstances in which I see this are
> probably coincident with when I invoke bitbake with -k.

Oh right.  Yes, in that case, what you are seeing is expected.  As the
name suggests, PREFERRED_PROVIDER only affects the relative
attractiveness of the different possible providers for a given virtual:
it will not cause bitbake to build a package that wouldn't otherwise
have been considered, nor will it prevent other providers from being
considered if the preferred one is unavailable.  All it does is change
the ordering of bitbake's internal list of build candidates.

So, I think the problem you're seeing is actually unrelated to
PREFERRED_PROVIDER.  It sounds like what you really want is a variant of
"-k" that will cause bitbake to make only one attempt at building a
given virtual (as opposed to trying all the providers in descending
preference order, which is what happens now) but still allow it to
continue with other packages even if that one attempt fails.  That would
be easy enough to add, I imagine, and wouldn't have any compatibility
implications.  Patches welcome! :-)

p.

Re: RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

[K. Richard Pixley]

I have a similar issue to address.  I haven't gone looking, but what I'm 
hoping for is a hard fail exit from bitbake if the PROVIDER isn't 
available.  I'm hoping for either a global switch that turns 
PREFERRED_PROVIDER from advisory to mandatory.  Or, even better, a 
MANDATORY_PROVIDER which is hard fail while other PREFERRED_PROVIDERs 
remain optional.

It's on my of features to add, but our OE tree is over a year and a half 
old now, and an OE merge doesn't seem to be much of a priority here, so 
it could be some time before I get to it.

The rationale here is the same difference I had with Cygnus configure vs 
Autoconf.  Autoconf's job is to infer itself into new environments as 
best it can.  This is great for systems that are compiled once each on 
millions of personal and academic systems of which no two are exactly alike.

However, for commercial grade software, we really do know exactly what 
our environment is and any time the compiling environment doesn't fit 
our expectations, we really would prefer a hard fail immediately so that 
we can track down that difference.

In our case, we want to use virtual packages as a way of selecting 
one-of-N options for our different products.  But there's absolutely no 
reason to even consider using an option that hasn't been specifically 
selected for a particular product.

--rich

Rod Whitby wrote:
> Whenever I build virtual/kernel, and for some reason (perhaps a git
> fetcher problem, or a patch doesn't apply cleanly) it doesn't build,
> bitbake decides to then go and build every other kernel under the sun
> trying to complete virtual/kernel.
>
> This is undesirable to the extreme, but I realise that there might have
> in the history of OE been a good reason for that behaviour.
>
> My proposal is that bitbake is changed so that if PREFERRED_PROVIDER is
> defined, then bitbake does *not* try other providers.  Only if
> PREFERRED_PROVIDER is *not* defined should bitbake ever try more than
> one package.
>
> Objections?
>
> Here is the IRC log of the discussion with RP about this issue:
>
> <rwhitby> RP: a question for you
> <rwhitby> when using virtual/kernel, and your PREFERRED kernel fails to
> build, why does bitbake then go and build every other kernel under the sun?
> <rwhitby> (and is there a way to stop that happening)
> <RP> rwhitby: That is the traditional bitbake behaviour
> <RP> rwhitby: I don't actually agree with it but changing it would
> breakbacwards compatibilty
> <RP> I guess it was once a feature but its turned out no to be so useful
> <RP> We should make that behaviour configurable. It only happens with
> the -k option anyway
> <rwhitby> RP: what was the use case in which it was useful, when
> PREFERRED_PROVIDER was defined?
> <RP> rwhitby: The idea was it would allow builds to complete rather than
> fail
> <rwhitby> I can understand it if PREFERRED_PROVIDER is not defined, but
> cannot think of a good reason why bitbake should go against explicit
> directions for a PREFERRED_PROVIDER
> <RP> rwhitby: It comes does to the variable name - "PREFERRED"
> <RP> Would it be useful to change the design, I say yes
> <RP> but there is history there and as the maintainer I need to be
> careful with this
> <rwhitby> understood
> <rwhitby> can we make it configurable and configure to not do it by
> default ?
> <RP> We can make it configurable
> <RP> I would want to see a discussion about defaults
> <RP> Feel free to start that on the bitbake+OE list
> <RP> I'm actually in favour of changing it
> <RP> but it must be discussed
> <rwhitby> nod
>
> -- Rod
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>   

Re: RFC: When PREFERRED_PROVIDER does not build, don't try other alternatives by default

[Koen Kooi]

Rod Whitby wrote:
> Whenever I build virtual/kernel, and for some reason (perhaps a git
> fetcher problem, or a patch doesn't apply cleanly) it doesn't build,
> bitbake decides to then go and build every other kernel under the sun
> trying to complete virtual/kernel.
>
> This is undesirable to the extreme,

Related to that: if for some reason you add a linux-myleetmachine.bb 
recipe instead of extending existing kernel recipes, please set 
COMPATIBLE_MACHINE = "myleetmachine" inside that recipe.
The same goes for other machine specific recipes (e.g. omapfbplay_git.bb 
should have COMPATIBLE_MACHINE set to all omap3 boards).

regards,

Koen