USE flags mumbling

USE flags mumbling

[Graeme Gregory]

We already have BBCLASSEXTENDS which modifies ${PN} of a package and
can use overrides to change behaviors of recipes.

Maybe USE flags could be implemented in a similar fashing.

DISTRO_USE = "nossl nox11"

EXTRA_OECONF_append_use-nossl = "--disable-ssl"

${PN} of the recipe becomes XXXX-nossl

Thoughts?

Graeme

Re: USE flags mumbling

[Chris Larson]

On Thu, Jul 1, 2010 at 9:24 AM, Graeme Gregory <dp@xora.org.uk> wrote:

> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
> can use overrides to change behaviors of recipes.
>
> Maybe USE flags could be implemented in a similar fashing.
>
> DISTRO_USE = "nossl nox11"
>
> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
>
> ${PN} of the recipe becomes XXXX-nossl


Any particular reason to not leverage the existing DISTRO_FEATURES,
MACHINE_FEATURES, COMBINED_FEATURES for this?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: USE flags mumbling

[Graeme Gregory]

On Thu, 1 Jul 2010 09:50:11 -0700
Chris Larson <clarson@kergoth.com> wrote:

> On Thu, Jul 1, 2010 at 9:24 AM, Graeme Gregory <dp@xora.org.uk> wrote:
> 
> > We already have BBCLASSEXTENDS which modifies ${PN} of a package and
> > can use overrides to change behaviors of recipes.
> >
> > Maybe USE flags could be implemented in a similar fashing.
> >
> > DISTRO_USE = "nossl nox11"
> >
> > EXTRA_OECONF_append_use-nossl = "--disable-ssl"
> >
> > ${PN} of the recipe becomes XXXX-nossl
> 
> 
> Any particular reason to not leverage the existing DISTRO_FEATURES,
> MACHINE_FEATURES, COMBINED_FEATURES for this?

No reason, doesn't really matter how the choices are generated.

G

Re: USE flags mumbling

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01-07-10 18:24, Graeme Gregory wrote:
> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
> can use overrides to change behaviors of recipes.
> 
> Maybe USE flags could be implemented in a similar fashing.
> 
> DISTRO_USE = "nossl nox11"
> 
> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
> 
> ${PN} of the recipe becomes XXXX-nossl
> 
> Thoughts?

Just that USE flags shouldn't be used if seperate recipes can solve it
as well.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMLQBEMkyGM64RGpERAikeAJ9L/lRCs6ZFrvBJfEszjZDEmlNIxQCeJURs
ag7eTOeSBZjR6IroPtaPKbE=
=USZM
-----END PGP SIGNATURE-----

Re: USE flags mumbling

[Chris Larson]

On Thu, Jul 1, 2010 at 1:53 PM, Koen Kooi <k.kooi@student.utwente.nl> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 01-07-10 18:24, Graeme Gregory wrote:
> > We already have BBCLASSEXTENDS which modifies ${PN} of a package and
> > can use overrides to change behaviors of recipes.
> >
> > Maybe USE flags could be implemented in a similar fashing.
> >
> > DISTRO_USE = "nossl nox11"
> >
> > EXTRA_OECONF_append_use-nossl = "--disable-ssl"
> >
> > ${PN} of the recipe becomes XXXX-nossl
> >
> > Thoughts?
>
> Just that USE flags shouldn't be used if seperate recipes can solve it
> as well.


In general I disagree with this, though I understand where you're coming
from (Tom Rini made it clear in a discussion earlier).  I don't think it'd
be terribly difficult to be able to automatically encode the use flags (at
least for non-default) in the PKG, while having it RPROVIDES ${PN}, so that
it would be obvious looking at the feeds.  Alternatively, we can in the
future utilize the code I'm working on for checksumming the metadata or
portions of the metadata to ensure that variants are encoded in the package
naming.  Opinions?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: USE flags mumbling

[Tom Rini]

Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 01-07-10 18:24, Graeme Gregory wrote:
>> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
>> can use overrides to change behaviors of recipes.
>>
>> Maybe USE flags could be implemented in a similar fashing.
>>
>> DISTRO_USE = "nossl nox11"
>>
>> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
>>
>> ${PN} of the recipe becomes XXXX-nossl
>>
>> Thoughts?
> 
> Just that USE flags shouldn't be used if seperate recipes can solve it
> as well.

If I may, I'd like to articulate what I believe to be the technical 
argument behind this statement.

One of the issues with some form of USE flags, and I believe this is one 
of the big ones for Angstrom as well as any other public feed publishing 
distribution is that having a single recipe that does different things 
based on variables makes maintaining their feed (and allowing users to 
publish their own compatible feeds) a nightmare.

This is because there isn't any form of use flag checking (a simple case 
of 1:1 or the more complex case of flagA needs flagB and flagC doesn't 
care about flagD -> flagZ).

-- 
Tom Rini
Mentor Graphics Corporation

Re: USE flags mumbling

[Chris Larson]

On Thu, Jul 1, 2010 at 3:16 PM, Tom Rini <tom_rini@mentor.com> wrote:

> Koen Kooi wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 01-07-10 18:24, Graeme Gregory wrote:
>>
>>> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
>>> can use overrides to change behaviors of recipes.
>>>
>>> Maybe USE flags could be implemented in a similar fashing.
>>>
>>> DISTRO_USE = "nossl nox11"
>>>
>>> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
>>>
>>> ${PN} of the recipe becomes XXXX-nossl
>>>
>>> Thoughts?
>>>
>>
>> Just that USE flags shouldn't be used if seperate recipes can solve it
>> as well.
>>
>
> If I may, I'd like to articulate what I believe to be the technical
> argument behind this statement.
>
> One of the issues with some form of USE flags, and I believe this is one of
> the big ones for Angstrom as well as any other public feed publishing
> distribution is that having a single recipe that does different things based
> on variables makes maintaining their feed (and allowing users to publish
> their own compatible feeds) a nightmare.
>
> This is because there isn't any form of use flag checking (a simple case of
> 1:1 or the more complex case of flagA needs flagB and flagC doesn't care
> about flagD -> flagZ).


Hmm, good point, dealing with dependency between recipes with specific flags
set or unset gets ugly fast.  *ponders*
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: USE flags mumbling

[Tom Rini]

Graeme Gregory wrote:
> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
> can use overrides to change behaviors of recipes.
> 
> Maybe USE flags could be implemented in a similar fashing.
> 
> DISTRO_USE = "nossl nox11"
> 
> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
> 
> ${PN} of the recipe becomes XXXX-nossl
> 
> Thoughts?

First we'd have to have a discussion on what the default should be and 
get agreement everywhere (ssl? x11? bluetooth (bluez3? bluez4?)? alsa?) 
on the whole raft of things that it would be nice to globally turn off 
or on.  Then we have to know which ones a given recipe actually made use 
of as opkg-nox11 is quite silly but conversely it'd be quite nice to 
have in autotools.bbclass the magic to always pass --disable-x11 (since 
it sure feels like everyone uses the same enable/disable switch finally).

Second, we also need a raft of, and perhaps a much easier way to, add 
binary package output virtuals.

I wonder, and I have to admit to having no real gentoo background here, 
how do they solve this problem?  Did they invent their own package 
format and add another field that consists of use flags?  That'd make 
some stuff a lot easier, but making deb/rpm get that mapping somehow 
seems hard at first.

-- 
Tom Rini
Mentor Graphics Corporation

Re: USE flags mumbling

[Phil Blundell]

On Thu, 2010-07-01 at 15:16 -0700, Tom Rini wrote:
> One of the issues with some form of USE flags, and I believe this is one 
> of the big ones for Angstrom as well as any other public feed publishing 
> distribution is that having a single recipe that does different things 
> based on variables makes maintaining their feed (and allowing users to 
> publish their own compatible feeds) a nightmare.

It's only a nightmare if the flags in question are user-frobbable.  If
they are all nailed down in the DISTRO configuration (which
DISTRO_FEATURES certainly ought to be), and those folks who want to
build compatible binaries just leave them alone, then there oughtn't to
be any real problem.  To that extent it doesn't really seem any
different to the choice of compiler or tuning flags or glibc version or
any of the other ways in which you can already produce incompatible
binaries by flipping the wrong switches.

p.

Re: USE flags mumbling

[Michael 'Mickey' Lauer]

Am Donnerstag, den 01.07.2010, 23:29 +0100 schrieb Phil Blundell:
> On Thu, 2010-07-01 at 15:16 -0700, Tom Rini wrote:
> > One of the issues with some form of USE flags, and I believe this is one 
> > of the big ones for Angstrom as well as any other public feed publishing 
> > distribution is that having a single recipe that does different things 
> > based on variables makes maintaining their feed (and allowing users to 
> > publish their own compatible feeds) a nightmare.
> 
> It's only a nightmare if the flags in question are user-frobbable.  If
> they are all nailed down in the DISTRO configuration (which
> DISTRO_FEATURES certainly ought to be), and those folks who want to
> build compatible binaries just leave them alone, then there oughtn't to
> be any real problem.  To that extent it doesn't really seem any
> different to the choice of compiler or tuning flags or glibc version or
> any of the other ways in which you can already produce incompatible
> binaries by flipping the wrong switches.

Agreed. I think we should be more open to this concept. At least for
packages with optional (but "infecting") X11 support -- such as EFL -- I
plan to use such a mechanism soon.

-- 
:M:

Re: USE flags mumbling

[Tom Rini]

Michael 'Mickey' Lauer wrote:
> Am Donnerstag, den 01.07.2010, 23:29 +0100 schrieb Phil Blundell:
>> On Thu, 2010-07-01 at 15:16 -0700, Tom Rini wrote:
>>> One of the issues with some form of USE flags, and I believe this is one 
>>> of the big ones for Angstrom as well as any other public feed publishing 
>>> distribution is that having a single recipe that does different things 
>>> based on variables makes maintaining their feed (and allowing users to 
>>> publish their own compatible feeds) a nightmare.
>> It's only a nightmare if the flags in question are user-frobbable.  If
>> they are all nailed down in the DISTRO configuration (which
>> DISTRO_FEATURES certainly ought to be), and those folks who want to
>> build compatible binaries just leave them alone, then there oughtn't to
>> be any real problem.  To that extent it doesn't really seem any
>> different to the choice of compiler or tuning flags or glibc version or
>> any of the other ways in which you can already produce incompatible
>> binaries by flipping the wrong switches.
> 
> Agreed. I think we should be more open to this concept. At least for
> packages with optional (but "infecting") X11 support -- such as EFL -- I
> plan to use such a mechanism soon.

Putting on my devil's advocate hat again, unless something is really and 
truly locked down, it's going to get modified.  While most end users 
won't want to frob gcc versions, frobbing bluetooth or alsa or x11 or 
... is more common.  For example, Gentoo.  And this is a problem if it's 
not easily detectable that you're going to have a clash.  All the DANGER 
DANGER DANGER comments in the world won't stop users from putting up the 
incompatible feed (with their own warning that someone else will ignore).

-- 
Tom Rini
Mentor Graphics Corporation

Re: USE flags mumbling

[Roman I Khimov]

[-- Attachment #1: Type: Text/Plain, Size: 2866 bytes --]

В сообщении от Пятница 02 июля 2010 02:39:54 автор Tom Rini написал:
> Michael 'Mickey' Lauer wrote:
> > Am Donnerstag, den 01.07.2010, 23:29 +0100 schrieb Phil Blundell:
> >> On Thu, 2010-07-01 at 15:16 -0700, Tom Rini wrote:
> >>> One of the issues with some form of USE flags, and I believe this is
> >>> one of the big ones for Angstrom as well as any other public feed
> >>> publishing distribution is that having a single recipe that does
> >>> different things based on variables makes maintaining their feed (and
> >>> allowing users to publish their own compatible feeds) a nightmare.
> >>
> >> It's only a nightmare if the flags in question are user-frobbable.  If
> >> they are all nailed down in the DISTRO configuration (which
> >> DISTRO_FEATURES certainly ought to be), and those folks who want to
> >> build compatible binaries just leave them alone, then there oughtn't to
> >> be any real problem.  To that extent it doesn't really seem any
> >> different to the choice of compiler or tuning flags or glibc version or
> >> any of the other ways in which you can already produce incompatible
> >> binaries by flipping the wrong switches.
> >
> > Agreed. I think we should be more open to this concept. At least for
> > packages with optional (but "infecting") X11 support -- such as EFL -- I
> > plan to use such a mechanism soon.
> 
> Putting on my devil's advocate hat again, unless something is really and
> truly locked down, it's going to get modified.  While most end users
> won't want to frob gcc versions, frobbing bluetooth or alsa or x11 or
> ... is more common.  For example, Gentoo. 

IMO this is not that much of a problem for a thing like OE. I view OE as a 
powerful tool that is used by developers to make products for end-users. As a 
developer I love an idea of USE-flags as that's what you generally do on 
embedded device - tune it for specific use-cases, leaving only things you 
really need.

And as for an end users, there are already tons of ways to make incompatible 
package feeds for them: toolchain components versions, gcc flags, etc. Or take 
a look at shadow recipe, for example, it has dynamic PAM dependency. And know 
what? I'd really love to see the same dynamic PAM dep for busybox, for 
example, as we're building it with PAM support and with current state of 
things there is no sane way I could bring such a change in oe.dev.

Real end users mostly follow one feed, I think. If they don't - they can be in 
trouble already. So there is not that much USE flags change here, IMO. And 
with a mechanism to encode used USE flags in package name with proper 
RPROVIDES and such I don't see any real problem at all.

-- 
 http://roman.khimov.ru
mailto: roman@khimov.ru
gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: USE flags mumbling

[Michael Lippautz]

2010/7/2 Tom Rini <tom_rini@mentor.com>:
> ...
> I wonder, and I have to admit to having no real gentoo background here, how
> do they solve this problem?  Did they invent their own package format and
> add another field that consists of use flags?  That'd make some stuff a lot
> easier, but making deb/rpm get that mapping somehow seems hard at first.

On Gentoo you don't pull any binary packages but you load the ebuild (<=>
recipe) from a feed. Thus you can build what you need AT your machine, which
I don't think is possible here. Dependency management is done through this
ebuilds. Package A has useflags B? and C? and pulls other packages in
depending on if these flags are enabled.

Regards,
Michael

Re: USE flags mumbling

[Martin Jansa]

On Thu, Jul 01, 2010 at 03:22:54PM -0700, Tom Rini wrote:
> I wonder, and I have to admit to having no real gentoo background
> here, how do they solve this problem?  Did they invent their own
> package format and add another field that consists of use flags?
> That'd make some stuff a lot easier, but making deb/rpm get that
> mapping somehow seems hard at first.

They store all information about built packages in
/var/db/pkg/category/name/*

So they know which USE flags were ON and OFF when you built it, there is
also environment.bz2 where you can find which gcc version was used to
build it.

So it's easy to rebuild all installed packages if current systems
settings gives different USE flag sets then before (--newuse param) or
with small script I used to rebuild all remaining packages which weren't
rebuild after system gcc upgrade.

Lately (imho with new EAPI) they can also narrow dependencies not only
with requested version, but also with some USE flag ON or OFF.

And they have it much easier as they resolve it on target device, to
provide --newuse functionality in OE we would probably do PR bump of
all "changed" recipes.

So in the end it will be far from ideal if we try just to encode "our" USE
flags to ${PN}. 

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: USE flags mumbling

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02-07-10 00:16, Tom Rini wrote:
> Koen Kooi wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 01-07-10 18:24, Graeme Gregory wrote:
>>> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
>>> can use overrides to change behaviors of recipes.
>>>
>>> Maybe USE flags could be implemented in a similar fashing.
>>>
>>> DISTRO_USE = "nossl nox11"
>>>
>>> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
>>>
>>> ${PN} of the recipe becomes XXXX-nossl
>>>
>>> Thoughts?
>>
>> Just that USE flags shouldn't be used if seperate recipes can solve it
>> as well.
> 
> If I may, I'd like to articulate what I believe to be the technical
> argument behind this statement.
> 
> One of the issues with some form of USE flags, and I believe this is one
> of the big ones for Angstrom as well as any other public feed publishing
> distribution is that having a single recipe that does different things
> based on variables makes maintaining their feed (and allowing users to
> publish their own compatible feeds) a nightmare.

That's not what I'm getting at. If 2 *packages* can safely co-exist in
the feeds *and* and image can choose which it wants to install USE flags
artificially limit the choice.
Example: opkg, nopkg-nopgp

This extends to things like "shadow or tinylogin?" as well.

regards,

Koen

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMLYzLMkyGM64RGpERAqz5AJ0WYMyg9VkgKfHEz22ZWkXcX8ik0ACfdJir
AzXGjeywXg5XnbtRIqXIEhI=
=8aWE
-----END PGP SIGNATURE-----

Re: USE flags mumbling

[Richard Purdie]

On Fri, 2010-07-02 at 08:52 +0200, Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 02-07-10 00:16, Tom Rini wrote:
> > Koen Kooi wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> On 01-07-10 18:24, Graeme Gregory wrote:
> >>> We already have BBCLASSEXTENDS which modifies ${PN} of a package and
> >>> can use overrides to change behaviors of recipes.
> >>>
> >>> Maybe USE flags could be implemented in a similar fashing.
> >>>
> >>> DISTRO_USE = "nossl nox11"
> >>>
> >>> EXTRA_OECONF_append_use-nossl = "--disable-ssl"
> >>>
> >>> ${PN} of the recipe becomes XXXX-nossl
> >>>
> >>> Thoughts?
> >>
> >> Just that USE flags shouldn't be used if seperate recipes can solve it
> >> as well.
> > 
> > If I may, I'd like to articulate what I believe to be the technical
> > argument behind this statement.
> > 
> > One of the issues with some form of USE flags, and I believe this is one
> > of the big ones for Angstrom as well as any other public feed publishing
> > distribution is that having a single recipe that does different things
> > based on variables makes maintaining their feed (and allowing users to
> > publish their own compatible feeds) a nightmare.
> 
> That's not what I'm getting at. If 2 *packages* can safely co-exist in
> the feeds *and* and image can choose which it wants to install USE flags
> artificially limit the choice.
> Example: opkg, nopkg-nopgp
> 
> This extends to things like "shadow or tinylogin?" as well.

Graeme suggested something like BBCLASSEXTEND for USE flags. I've also
wondered about it.

Any "USE" case contains at least three pieces of information:

a) Which configure flag we're playing with
b) What DEPENDS needs to change
c) Some kind of associated name for the flag

Creating an example, lets have a case where we have "ssl, x11, dbus,
bluez and wifi" as features. This example isn't so crazy as once we open
the box, someone will want every combination of every possible option.

With 5 options there are 32 different possible variants.

Now I realise not all combinations of "USE" cases work together or
actually make sense. Do we list the combinations that are allowed? or
disallowed?

By the time you delve into adding some complex configuration system to
merge together all this data, I'd suggest it makes a lot more sense to
have setups like opkg where we have a simple .inc file which does:

require foo_${PV}.bb

DEPENDS += "xyz"
EXTRA_OECONF = "--with-xyz"

and the addition of a -= operator in bitbake could be very useful.

BBCLASSEXTEND grew out of the shear number of recipes that followed a
form of just inheriting native. I'd suggest that if we need an extension
to bitbake in the future we can add one but lets make it work without
that, at least at first and see what makes sense.

I'd suggest that adding 32 different variants of a package just dilutes
our testing further and will become an unmaintainable nightmare so any
system which limits the number of variants to those we actually need is
desirable.

This whole discussion also doesn't get to how to select which
combination you actually want to use in an image. If you pull in
something with ssl enabled, you probably want to change to the ssl
versions of all the other packages with that option for example. This
implies that the package manager needs to be taught something about
this.

So that covers generating all the multiple outputs of USE cases. What
about users wishing to modify a given recipes flags without changing its
name? This is its own can of worms.

I'd point out you can already do fun things like:

EXTRA_OECONF_append_pn-opkg = " --without-ssl"

from your distro.conf or local.conf.

I don't really have a conclusion. In some cases providing multiple
variants makes sense. In others its just impractical and being able to
change the flags of packages does make sense for some people's usecases.
Regardless of what we do, it needs a lot of thought and planning going
into it and I don't think anyone has followed through the implications
to the level of detail required :/. It is a case where forward planning
would help a lot.

Cheers,

Richard

Re: USE flags mumbling

[Tom Rini]

Martin Jansa wrote:
> On Thu, Jul 01, 2010 at 03:22:54PM -0700, Tom Rini wrote:
>> I wonder, and I have to admit to having no real gentoo background
>> here, how do they solve this problem?  Did they invent their own
>> package format and add another field that consists of use flags?
>> That'd make some stuff a lot easier, but making deb/rpm get that
>> mapping somehow seems hard at first.
> 
> They store all information about built packages in
> /var/db/pkg/category/name/*

OK.  I thought they had finally added binary packages as an option a 
while back but I see that's really just a few really big things now.

-- 
Tom Rini
Mentor Graphics Corporation