RFD: Recipe variants, multilib and package handling

RFD: Recipe variants, multilib and package handling

[Richard Purdie]

Why are variants of recipes important at all? I think we've all seen the
benefit of BBCLASSEXTEND massively simplifying native recipes and the
nativesdk class gives some glimpse of the generic power. We're in the
process of adding multilib handling to OE-Core and this pushes that
mechanism. It turns out there are some complexities here to do with the
naming conventions and how we namespace various targets which need
discussion.

Possible Namespaces
===================

For the purposes of this email lets assume we have the variants
"<default target> <native> <nativesdk> <multilibA> <multilibB>". The
exact format of the namespace is flexible but there are fours main
viable approaches to package namespace of variants:

a) Something gets appended to the end of the name:

<def target> : pkgconfig pkgconfig-dev pkgconfig-locale-foo
<native>     : pkgconfig-native
<nativesdk>  : pkgconfig-nativesdk pkgconfig-dev-nativesdk pkgconfig-locale-foo-nativesdk
<multilibA>  : pkgconfig-mlA pkgconfig-dev-mlA pkgconfig-locale-foo-mlA
<multilibB>  : pkgconfig-mlB pkgconfig-dev-mlB pkgconfig-locale-foo-mlB

b) Something gets appended to the end of PN:

<def target> : pkgconfig pkgconfig-dev pkgconfig-locale-foo
<native>     : pkgconfig-native
<nativesdk>  : pkgconfig-nativesdk pkgconfig-nativesdk-dev pkgconfig-nativesdk-locale-foo
<multilibA>  : pkgconfig-mlA pkgconfig-mlA-dev pkgconfig-mlA-locale-foo
<multilibB>  : pkgconfig-mlB pkgconfig-mlB-dev pkgconfig-mlB-locale-foo

c) Something gets prepended to the start of the name:

<def target> : pkgconfig pkgconfig-dev pkgconfig-locale-foo
<native>     : native-pkgconfig
<nativesdk>  : nativesdk-pkgconfig nativesdk-pkgconfig-dev nativesdk-pkgconfig-locale-foo
<multilibA>  : mlA-pkgconfig mlA-pkgconfig-dev mlA-pkgconfig-locale-foo
<multilibB>  : mlB-pkgconfig mlB-pkgconfig-dev mlB-pkgconfig-locale-foo

Variant d) where something gets prepended to PN works out the same as c)
except where we have package names like lib${PN}.

Currently OE operates on b) or some variant thereof. BBCLASSEXTEND
automatically adds something to the end of PN.

The two conflicting problems
============================

So what's the problem? It all comes down to how to process a recipe and
turn it into one of these variants. There are two classes of problem:

i) Translating DEPENDS/RDEPENDS style variables

Given a general dependency like "pkgconfig-locale-foo", how do you turn
it into the variant? For the cases like b) and d) above where some
transformation of ${PN} is made, you don't know where to place the
variant in the general string. In this example, its obvious to the human
eye that it should be after "pkgconfig" but a computer can't do that
generically.

That turns out to be hard so then you consider appending to the end of
the name. You can search for any reference to the variant in the name,
remove it and place it at the end.

ii) Dealing with assumptions about ${PN}

The whole of the metadata does things like:

PACKAGES = "${PN} ${PN}-dev"
FILES_${PN}-dev = "x"

The use of PN like this is why there is the attraction of simply
appending to PN. The trouble is that i) above means you can't do this
and reliably translate dependencies.

Options
=======

a) We could introduce some kind of suffix variable ${PKGSFX} which would
contain the variant, maybe in conjuction with BPN instead of PN:

PACKAGES = "${BPN}${PKGSFX} ${BPN}-dev${PKGSFX}"
FILES_${BPN}-dev${PKGSFX} = "x"

This has the downside of having to touch the whole metadata and it looks
ugly.

b) We could try and remap all the variables at BBCLASSEXTEND time. This
would mean remapping PACKAGES and renaming all the package specific
variables such as FILES, SUMMARY, DESCRIPTION, RDEPENDS, RRECOMMENDS,
RSUGGESTS, RPROVIDES, RREPLACES, ALLOW_EMPTY, SECTION and likely others.

I knew were were going to have to read through some of those variables
and remap the dependencies but having to totally move them into new
namespaces is a scary thought (but still possible).

c) Switch to prepending to PN. This would only break cases where we have
packages which don't start with ${PN} of which there are comparatively
few. We could make the simplifying assumption that all PACKAGES start
with ${PN} meaning that all manipulations would be simple prepends. If
we do this, problems i) and ii) above are no longer an issue.

Discussion
==========

I don't think option a) above is viable and the current plan implies
we'd do b) but its extremely ugly. I'm therefore tempted to look more
seriously at c). The bigger issues would appear to be:

* It breaks with convention/tradition for OE (xxx-native vs native:xxx)
* It would add the constraint of packaging starting with ${PN}
* It would require changes to the likes of debian.bbclass to account 
  for package prefixes when performing auto renaming
* It would break a small set of the metadata where packages don't start 
  with ${PN} (although the class could simply refuse to extend these 
  automatically).

Things to consider:

* Would we just do this for multilibs or would we transition native 
  recipes to the new style of naming? We don't have PACKAGES problems 
  for native recipes.
* Likewise, would nativesdk transition? Is has more PACKAGES problems 
  so likely yes, it would make sense to transition.
* Would we stick with "-" as a delimiter or switch to something like 
  ":"?

Thoughts/suggestions/better ideas welcome...

Cheers,

Richard

Re: RFD: Recipe variants, multilib and package handling

[Esben Haabendal]

On Mon, 2011-06-13 at 12:52 +0100, Richard Purdie wrote:
> Options
> =======
> 
> a) We could introduce some kind of suffix variable ${PKGSFX} which would
> contain the variant, maybe in conjuction with BPN instead of PN:
> 
> PACKAGES = "${BPN}${PKGSFX} ${BPN}-dev${PKGSFX}"
> FILES_${BPN}-dev${PKGSFX} = "x"
> 
> This has the downside of having to touch the whole metadata and it looks
> ugly.

In OE-lite, we have tried out this option a.  It works out pretty well,
but is ugly, and does to some degree make it more complex to manage
simple recipe meta-data.

For this reason, next OE-lite version will transition to something more
like the option b you suggest.

> b) We could try and remap all the variables at BBCLASSEXTEND time. This
> would mean remapping PACKAGES and renaming all the package specific
> variables such as FILES, SUMMARY, DESCRIPTION, RDEPENDS, RRECOMMENDS,
> RSUGGESTS, RPROVIDES, RREPLACES, ALLOW_EMPTY, SECTION and likely others.

In next OE-lite version, I have introduce something called "recipe
type", which is nothing else than the name from "BBCLASSEXEND", or
"machine" for default recipe type.

When resolving dependencies, I have introduced a recipe type context.
One for DEPENDS for each recipe type, and one for RDEPENDS for the
recipe types that is relevant for run-time dependency handling.

In recipe meta-data, you can fx. say

DEPENDS = "libncurses native:libtool"

Based on the recipe type context, the "libncurses" is prefixed with the
recipe type of the context, fx. "machine:libncurses" for machine recipe
type.

To further simplify recipe meta-data, I have introduced some generic
prefix aliases.  I have "host" and "target", and "host-cross" and
"target-cross", which is then mapped to the proper recipe type based on
the context.

This seems to work out very nicely.  I seems effective in reducing the
need for recipe type specfic overrides of DEPENDS variables, and makes
recipe meta-data much easier to read IMHO.

Of-course, OE-lite does only have package based dependencies, and I
haven't given any thought to how this would work out for OE's recipe
based build dependencies.

But perhaps it would also be a good choice for OE.

> I knew were were going to have to read through some of those variables
> and remap the dependencies but having to totally move them into new
> namespaces is a scary thought (but still possible).

Hehe.  In danish, we have a saying: "Hvo intet vover, intet vinder",
which I belive is something like "Nothing ventured, nothing gained" in
english.

> c) Switch to prepending to PN. This would only break cases where we have
> packages which don't start with ${PN} of which there are comparatively
> few. We could make the simplifying assumption that all PACKAGES start
> with ${PN} meaning that all manipulations would be simple prepends. If
> we do this, problems i) and ii) above are no longer an issue.
> 
> Discussion
> ==========
> 
> I don't think option a) above is viable and the current plan implies
> we'd do b) but its extremely ugly. I'm therefore tempted to look more
> seriously at c). The bigger issues would appear to be:
> 
> * It breaks with convention/tradition for OE (xxx-native vs native:xxx)
> * It would add the constraint of packaging starting with ${PN}
> * It would require changes to the likes of debian.bbclass to account 
>   for package prefixes when performing auto renaming
> * It would break a small set of the metadata where packages don't start 
>   with ${PN} (although the class could simply refuse to extend these 
>   automatically).

Looking into the future, wouldn't it make most sense to use the solution
broadly in OE, to avoid making it even harder to get started with OE?

> Things to consider:
> 
> * Would we just do this for multilibs or would we transition native 
>   recipes to the new style of naming? We don't have PACKAGES problems 
>   for native recipes.
> * Likewise, would nativesdk transition? Is has more PACKAGES problems 
>   so likely yes, it would make sense to transition.
> * Would we stick with "-" as a delimiter or switch to something like 
>   ":"?

I decided to switch from "-" to ":", to allow robust translation of
non-prefixed dependency names.

> Thoughts/suggestions/better ideas welcome...

Thanks,

Esben

Re: RFD: Recipe variants, multilib and package handling

[Tom Rini]

On 06/13/2011 04:52 AM, Richard Purdie wrote:

[snip]
> Discussion
> ==========
> 
> I don't think option a) above is viable and the current plan implies
> we'd do b) but its extremely ugly. I'm therefore tempted to look more
> seriously at c). The bigger issues would appear to be:
> 
> * It breaks with convention/tradition for OE (xxx-native vs native:xxx)

True, but how long do we stick with things that are limiting us when we
need a change to fix a real problem?  And this is a little easier to
deal with, now that we do really have a notion of doing releases so we
can more easily explain to folks when the change is.

> * It would add the constraint of packaging starting with ${PN}

I know we have some, but do they also really have a good reason for not
being ${PN}-foo now, possibly other than we borrowed the notion there
from someone else?  For example, I know Ubuntu is still 'ssh' for
'openssh', but we don't do that one.

> * It would require changes to the likes of debian.bbclass to account 
>   for package prefixes when performing auto renaming

Maybe we need to rename debian.bbclass while we're at it and yes, taking
into account multilib is, to me, just a 'yeah, gonna have to' as part of
the problem.

> * It would break a small set of the metadata where packages don't start 
>   with ${PN} (although the class could simply refuse to extend these 
>   automatically).

I think refusing is a good starting point to encourage someone that
needs it to update the recipe, or it can be a janitor project in the end
if the set is small enough...

> Things to consider:
> 
> * Would we just do this for multilibs or would we transition native 
>   recipes to the new style of naming? We don't have PACKAGES problems 
>   for native recipes.

I see a positive here being one less thing to change, but the downside
being one more set of logic sitting around.  Perhaps as a second pass
migrating native over...

> * Likewise, would nativesdk transition? Is has more PACKAGES problems 
>   so likely yes, it would make sense to transition.

I think it'll have to, at least before it's all said and done, otherwise
you will run into someone extending for both and puzzling over the
different names they get.

> * Would we stick with "-" as a delimiter or switch to something like 
>   ":"?

Internally, that might make things easier but in terms of writing out
the packages, that could be a problem...

> Thoughts/suggestions/better ideas welcome...

I wish it was easier to abstract things away so we could get namespace B
and keep that information around to solve problem 'i'.  Then problem
'ii' would just be about changing how we define PN.

-- 
Tom Rini
Mentor Graphics Corporation