Proposed Multilib Implementation Brainstorming

Proposed Multilib Implementation Brainstorming

[Richard Purdie]

One of the items on our post 1.0 schedule is multilib and we need a plan
of implementation. I've been thinking about this for a while and at
least have some ideas how some of the issues can be handled.

In case anyone isn't familiar with the idea of multilibs, the most
common example are x86 systems which have both 32 bit and 64 bit
libraries installed at the same time in /lib/ and /lib64/ directories or
similar and hence both types of binaries can be used.

The number of libs isn't limited to two choices as for example the mips
platform has three choices which can be used in parallel. The x32 ABI
would mean three possible x86 options too. 

The first step is being able to build given recipes for each of a set of
given multilibs. This is very similar in principle to the way
BBCLASSEXTEND is used, the difference it that whilst the native or
nativesdk classes are used once, for multilib we need a kind of multiple
inheritance of a class assuming we don't want one class per multilib.

To do this I'm thinking of a new set of include files of the form
conf/machine/include/multilib-xxxx.inc, similar in nature to the
tune-xxx.inc files. A given setup would include each of the multilibs it
requred. Each multilib include file would look something like:

conf/machine/include/multilib-x86-lib32.inc:

MULTILIBS += "lib32"

BASE_PACKAGE_ARCH_virtclass-multilib-lib32 = "core2-lib32"
TARGET_CC_ARCH_virtclass-multilib-lib32 = "-m32 -march=core2"
PACKAGE_EXTRA_ARCHS += "core2-lib32"

libdir_virtclass-multilib-lib32 = "${exec_prefix}/lib32"
base_libdir_virtclass-multilib-lib32 = "${base_prefix}/lib32"

i.e. use overrides to change the target cc architecture specifying the
required flags to the compiler and also alter the library paths as
required.

I'm envisaging passing an optional new parameter to classes in
BBCLASSEXTEND so for example the definition of a multilib could look
something like:

BBCLASSEXTEND_append = " multilib:x86-lib32"

and all the multilib class would need to do is to manipulation of
variables including OVERRIDES in a similar way to native.bbclass with
the complication of handling the parameter.

The toolchain bootstrap process would become a little complicated by
this. We'll need to be able to iterate over the list of multilibs and
configure the compiler with a configuration appropriate to the multilibs
requested. The compiler should then take care of generating a suitable
libgcc for each multilib. Where the compiler currently depends on
virtual/libc-initial, we'll need with a function called to generate the
dependecies so for example: "virtual/libc virtual/libc-initial-lib32"

The toolchain bootstrap process would hence become something like:

gcc-cross-initial
libc-initial
libc-initial-lib32
gcc-cross-intermediate
libc
libc-lib32
gcc-cross
libgcc (package)
libgcc-lib32 (package)
gcc-runtime
gcc-runtime-lib32

So for any multilib, we're stuck with always building the libc due to
the way gcc handles multilib. For subsequent tasks, only the libs
requested in a give image in a given ABI would be built.

Note that as far as the build system is concerned there become distinct
targets for each ABI so "bitbake mysql" would build the default library
type, "bitbake mysql-lib32" would build the 32 bit version. Including
both "mysql" and "mysql-lib32" in an image is where things get
interesting, the build system will ensure they're both built, then it
becomes the problem of the image creation and package
creation/management processes.

The final tricky part of the puzzle which is under the guidance of the
package manager. rpm has extensive pre-existing mutlilib support. opkg
has none.

It should be possible to convert the build system package name
libmysql-libx32 into just "libmysql" for the benefit of rpm and use
rpm's package flavours to handle the multilib issues.

For opkg there are multiple levels we can take the integration to. It
should be possible for example to install the "weaker" multilibs first,
then just install the stronger ones over the top, assuming any file
overwriting is ok and can be ignored. Teaching opkg about proper
multilib support would be a bigger project but would be much less of a
hack/workaround.

So to summarise the steps needed are:

a) bitbake enhancement for BBCLASSEXTEND parameter support
b) Add simple multilib build configuration
c) Update toolchain bootstrap with multilibs support
d) Add RPM multilib packaging
e) Look at opkg multilib integration


Does this make sense to everyone, are there any questions/ objections/
concerns/ things I've missed?

Cheers,

Richard

Re: Proposed Multilib Implementation Brainstorming

[Jeremy Puhlman]

> 
> To do this I'm thinking of a new set of include files of the form
> conf/machine/include/multilib-xxxx.inc, similar in nature to the
> tune-xxx.inc files. A given setup would include each of the multilibs it
> requred. Each multilib include file would look something like:
> 
> conf/machine/include/multilib-x86-lib32.inc:
> 
> MULTILIBS += "lib32"
> 
> BASE_PACKAGE_ARCH_virtclass-multilib-lib32 = "core2-lib32"
> TARGET_CC_ARCH_virtclass-multilib-lib32 = "-m32 -march=core2"
> PACKAGE_EXTRA_ARCHS += "core2-lib32"
> 
> libdir_virtclass-multilib-lib32 = "${exec_prefix}/lib32"
> base_libdir_virtclass-multilib-lib32 = "${base_prefix}/lib32"
> 
> i.e. use overrides to change the target cc architecture specifying the
> required flags to the compiler and also alter the library paths as
> required.
> 
> I'm envisaging passing an optional new parameter to classes in
> BBCLASSEXTEND so for example the definition of a multilib could look
> something like:
> 
> BBCLASSEXTEND_append = " multilib:x86-lib32"
> 
> and all the multilib class would need to do is to manipulation of
> variables including OVERRIDES in a similar way to native.bbclass with
> the complication of handling the parameter.

It is possible to do it with out changes to bitbake, but adding
parametrization to BBCLASSEXTEND would be useful in a number of ways,
and much more elegant then doing it with out it, and cleaner for that
matter. Unrelated to multilib, would the full path to the sub parameters
be configurable via the extended class? i.e. "conf/machine/include/" be
a configurable value from the multilib class?

> 
> For opkg there are multiple levels we can take the integration to. It
> should be possible for example to install the "weaker" multilibs first,
> then just install the stronger ones over the top, assuming any file
> overwriting is ok and can be ignored. 

If there isn't any kind of compartmentalization of binaries(i.e. the
alternate abi having a set of packages that only contian runtime libs),
wouldn't this render the package management on opkg based systems post
install largely useless other then just adding new components? Most
applications are getting better about it(openssl comes to mind), but
some may still place arch specific headers in common locations. Many
like openssl started switching over to arch wrapper headers, but there
may be some stuff still lurking out there. Could probably be handled on
a case by case basis.

> So to summarise the steps needed are:
> 
> a) bitbake enhancement for BBCLASSEXTEND parameter support
> b) Add simple multilib build configuration
> c) Update toolchain bootstrap with multilibs support
> d) Add RPM multilib packaging
> e) Look at opkg multilib integration
> 
> 
> Does this make sense to everyone, are there any questions/ objections/
> concerns/ things I've missed?
> 

I have been kicking around a still broken POC implementation to this for
a bit, but this is basically the line of thought(minus the
parametrization of BBCLASSEXTEND) for a while on the subject. This looks
mostly right to me.

Jeremy Puhlman
MontaVista Software, LLC.

Re: Proposed Multilib Implementation Brainstorming

[Esben Haabendal]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> One of the items on our post 1.0 schedule is multilib and we need a plan
> of implementation. I've been thinking about this for a while and at
> least have some ideas how some of the issues can be handled.
> ....
> Does this make sense to everyone, are there any questions/ objections/
> concerns/ things I've missed?

Which actual OE use-cases justify this kind of addition to OE?

I know it is on the Yocto post 1.0 schedule, but is it actually a good
thing for OE?  Maintaining OE recipes is clearly not going to get any
easier with multilibs support.

/Esben

Re: Proposed Multilib Implementation Brainstorming

[Koen Kooi]

Op 5 apr 2011, om 13:02 heeft Richard Purdie het volgende geschreven:

> One of the items on our post 1.0 schedule is multilib and we need a plan
> of implementation. I've been thinking about this for a while and at
> least have some ideas how some of the issues can be handled.
> 
> In case anyone isn't familiar with the idea of multilibs, the most
> common example are x86 systems which have both 32 bit and 64 bit
> libraries installed at the same time in /lib/ and /lib64/ directories or
> similar and hence both types of binaries can be used.
> 
> The number of libs isn't limited to two choices as for example the mips
> platform has three choices which can be used in parallel. The x32 ABI
> would mean three possible x86 options too. 

Sweet, this opens the door for arm multilibs as well. With this can finally build an armv7a without NEON and with NEON in "one" go and use hwcaps to load the right lib.

regards,

Koen

Re: Proposed Multilib Implementation Brainstorming

[Frans Meulenbroeks]

2011/4/5 Richard Purdie <richard.purdie@linuxfoundation.org>:

[...]
>
> Does this make sense to everyone, are there any questions/ objections/
> concerns/ things I've missed?

I think most embedded systems would only use one lib. To take your
lib/lib64 example:
If I am developing for an embedded system I know whether it will run
as 32 or 64 bit, so there is no need to have both.

multilib has its merits when it comes to supporting multiple hardware systems.
However as in the embedded world one is typically targeting a specific
hardware configuration.
(actually I don't recall having seen requests for multilib on the ML
before, although I could have missed it).

Also I'm somewhat worried by the actual complexity this adds (to the
build process and the recipes, and timewise probably also to the
bootstrap process as additional packages have to be built).

Not sure if that is a desirable route forward, but if we (we as in OE
members + developers) feel that OE should go that way, I would
sugggest to have a way to opt-in or opt-out

Frans

Re: [poky] Proposed Multilib Implementation Brainstorming

[Richard Purdie]

On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
> > One of the items on our post 1.0 schedule is multilib and we need a plan
> > of implementation. I've been thinking about this for a while and at
> > least have some ideas how some of the issues can be handled.
> > ....
> > Does this make sense to everyone, are there any questions/ objections/
> > concerns/ things I've missed?
> 
> Which actual OE use-cases justify this kind of addition to OE?

Several people wanting to use OECore have a requirement of multilib
support. The typical embedded use case is where you have one main
application which you might want to run in some kind of large memory
mode or with some special optimisation (think a database engine) whilst
the rest of the system is "standard". This requires the ability to mix
different libraries.

> I know it is on the Yocto post 1.0 schedule, but is it actually a good
> thing for OE?  Maintaining OE recipes is clearly not going to get any
> easier with multilibs support.

As detailed in the proposal you will see that the complexity added is
minimal. It requires a simple enhancement to BBCLASSEXTEND which is
likely desirable for other reasons too and that is the only real bitbake
change required. For the metadata, individual recipes remain unaffected
and also the core conf files are unchanged too. The toolchain dependency
changes will be the only change affecting users at the recipe level and
most of the class/machine configuration will be opt in by anyone using
multilib. The only other invasive change is the package manager
integration. For rpm, it has good support for multilib already and we're
just enabling that. For opkg, we still need to determine the best
approach but the simplistic approach I mentioned will probably suffice
and anyone wanting true support at the package manager level can use
rpm.

For day to day recipe maintenance I don't see much direct impact.

Cheers,

Richard

Re: [poky] Proposed Multilib Implementation Brainstorming

[Esben Haabendal]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> 
>> > One of the items on our post 1.0 schedule is multilib and we need a plan
>> > of implementation. I've been thinking about this for a while and at
>> > least have some ideas how some of the issues can be handled.
>> > ....
>> > Does this make sense to everyone, are there any questions/ objections/
>> > concerns/ things I've missed?
>> 
>> Which actual OE use-cases justify this kind of addition to OE?
>
> Several people wanting to use OECore have a requirement of multilib
> support.

Just out of curiosity, could you come with some examples of such
people or projects?

> The typical embedded use case is where you have one main
> application which you might want to run in some kind of large memory
> mode or with some special optimisation (think a database engine) whilst
> the rest of the system is "standard". This requires the ability to mix
> different libraries.
>
>> I know it is on the Yocto post 1.0 schedule, but is it actually a good
>> thing for OE?  Maintaining OE recipes is clearly not going to get any
>> easier with multilibs support.
>
> As detailed in the proposal you will see that the complexity added is
> minimal. It requires a simple enhancement to BBCLASSEXTEND which is
> likely desirable for other reasons too and that is the only real bitbake
> change required. For the metadata, individual recipes remain unaffected
> and also the core conf files are unchanged too.

I thought that all recipes (fx. library recipes) involved in a multilib
build must have the appropriate multilib parts put in BBCLASSEXTEND, and
after that, anybody touch that recipe is suddenly faced with having to
take care not to break multilib builds (which most developers probably
will not really care for).

> The toolchain dependency changes will be the only change affecting
> users at the recipe level and most of the class/machine configuration
> will be opt in by anyone using multilib. The only other invasive
> change is the package manager integration. For rpm, it has good
> support for multilib already and we're just enabling that. For opkg,
> we still need to determine the best approach but the simplistic
> approach I mentioned will probably suffice and anyone wanting true
> support at the package manager level can use rpm.
>
> For day to day recipe maintenance I don't see much direct impact.

I guess time will show...

/Esben

Re: [poky] Proposed Multilib Implementation Brainstorming

[Richard Purdie]

On Wed, 2011-04-06 at 14:16 +0200, Esben Haabendal wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
> > On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
> >> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> >> 
> >> > One of the items on our post 1.0 schedule is multilib and we need a plan
> >> > of implementation. I've been thinking about this for a while and at
> >> > least have some ideas how some of the issues can be handled.
> >> > ....
> >> > Does this make sense to everyone, are there any questions/ objections/
> >> > concerns/ things I've missed?
> >> 
> >> Which actual OE use-cases justify this kind of addition to OE?
> >
> > Several people wanting to use OECore have a requirement of multilib
> > support.
> 
> Just out of curiosity, could you come with some examples of such
> people or projects?

Well, you've seen Koen's reaction to the prospect of multilib support,
there is one example. I've given x32 as another at the start of this
thread. Its an issue I keep hearing about from companies too.

Fundamentally, multilib is one of the biggest remaining weaknesses of
the OE architecture. One of Yoctos objectives is to reduce the number of
build systems out there and to do this, we need to make one good fully
featured one. Until OE has multilib support there is a functionality
gap. I don't see any technical reason OE can't support mulitlibs and I
don't believe the complexity is high either, we just need to tweak some
things. 

I actually believe the core in OE is well suited to multilib support and
using it will show off some of the power in the OE architecture as I
think we can do it better than anyone else before!

> >> I know it is on the Yocto post 1.0 schedule, but is it actually a good
> >> thing for OE?  Maintaining OE recipes is clearly not going to get any
> >> easier with multilibs support.
> >
> > As detailed in the proposal you will see that the complexity added is
> > minimal. It requires a simple enhancement to BBCLASSEXTEND which is
> > likely desirable for other reasons too and that is the only real bitbake
> > change required. For the metadata, individual recipes remain unaffected
> > and also the core conf files are unchanged too.
> 
> I thought that all recipes (fx. library recipes) involved in a multilib
> build must have the appropriate multilib parts put in BBCLASSEXTEND, and
> after that, anybody touch that recipe is suddenly faced with having to
> take care not to break multilib builds (which most developers probably
> will not really care for).

The multilib conf files are responsible for changing BBCLASSEXTEND, I
don't want to see the need for many, if any recipe specific changes.

The recipe specific problems are likely to be limited to the likes of
hardcoded library paths and those already break things like
DISTRO="minimal" in OE.dev today.

Cheers,

Richard

Re: [poky] Proposed Multilib Implementation Brainstorming

[Esben Haabendal]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> On Wed, 2011-04-06 at 14:16 +0200, Esben Haabendal wrote:
>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> 
>> > On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
>> >> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> >> 
>> >> > One of the items on our post 1.0 schedule is multilib and we need a plan
>> >> > of implementation. I've been thinking about this for a while and at
>> >> > least have some ideas how some of the issues can be handled.
>> >> > ....
>> >> > Does this make sense to everyone, are there any questions/ objections/
>> >> > concerns/ things I've missed?
>> >> 
>> >> Which actual OE use-cases justify this kind of addition to OE?
>> >
>> > Several people wanting to use OECore have a requirement of multilib
>> > support.
>> 
>> Just out of curiosity, could you come with some examples of such
>> people or projects?
>
> Well, you've seen Koen's reaction to the prospect of multilib support,
> there is one example. I've given x32 as another at the start of this
> thread. Its an issue I keep hearing about from companies too.
>
> Fundamentally, multilib is one of the biggest remaining weaknesses of
> the OE architecture.
>
> One of Yoctos objectives is to reduce the number of
> build systems out there and to do this, we need to make one good fully
> featured one. Until OE has multilib support there is a functionality
> gap. I don't see any technical reason OE can't support mulitlibs and I
> don't believe the complexity is high either, we just need to tweak some
> things. 
>
> I actually believe the core in OE is well suited to multilib support and
> using it will show off some of the power in the OE architecture as I
> think we can do it better than anyone else before!

The one thing that in my experience pushes most people in other
directiosn than OE is the complexity.  Unless OE made simpler, I think
it will be quite hard to reach the goal of significantly reducing the
number of build systems.

I am uncertain to what effect multilib in OE will have (positive and
negative), but it is clearly not making OE simpler.

I just kind of makes things a bit ehmm... hopeless that the requests I
(and others with similar customers) hear from companies are considered
out of line for OE evolution, whereas the requests you and others hear
can be used as valid arguments for which directions to take.

It is not that I think it will add considerable burden, but is just yet
another brick to the wall to pass for developers trying to understand
OE.

Anyways, you asked and I raised my concern.

/Esben

Re: [poky] Proposed Multilib Implementation Brainstorming

[Richard Purdie]

On Wed, 2011-04-06 at 15:21 +0200, Esben Haabendal wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
> > On Wed, 2011-04-06 at 14:16 +0200, Esben Haabendal wrote:
> > I actually believe the core in OE is well suited to multilib support and
> > using it will show off some of the power in the OE architecture as I
> > think we can do it better than anyone else before!
> 
> The one thing that in my experience pushes most people in other
> directiosn than OE is the complexity.  Unless OE made simpler, I think
> it will be quite hard to reach the goal of significantly reducing the
> number of build systems.
>
> I am uncertain to what effect multilib in OE will have (positive and
> negative), but it is clearly not making OE simpler.
>
> I just kind of makes things a bit ehmm... hopeless that the requests I
> (and others with similar customers) hear from companies are considered
> out of line for OE evolution, whereas the requests you and others hear
> can be used as valid arguments for which directions to take.
> 
> It is not that I think it will add considerable burden, but is just yet
> another brick to the wall to pass for developers trying to understand
> OE.

Nobody is saying requests for simplicity or ease of use are out of line,
far from it. Its something that is being worked on by Yocto quite
heavily. What does differ is the approach though, yours is to rip out
anything you personally don't need, Yocto is taking a more reasoned
approach to this and asking questions like:

"What can we do to make things easier for new users?"
"Can we make error messages more user friendly?"
"What code do we have that is needlessly complex and can be simplified?"
"If we are going to add a new feature, how was we reuse existing
established concepts?"
"If something is complex and hard to understand but required, how can we
document or better explain the concepts to people?"

I'd go as far as to say that things are getting more understandable and
easier too. Yocto ran an alpha test programme where we took a selection
of people, some with experience and some with no experience of
embeddded, pointed them at the quick start guide and asked them to build
an image, boot it and experiment. The results of this programme with
Yocto 1.0 are orders of magnitude better than 0.9. This at least shows
the new user experience is better than it was. Its not the only area
that needs improvement and there is a long way to go but its a very
positive start IMO.

Cheers,

Richard

Re: Proposed Multilib Implementation Brainstorming

[Richard Purdie]

On Wed, 2011-04-06 at 10:47 +0200, Frans Meulenbroeks wrote:
> I think most embedded systems would only use one lib. To take your
> lib/lib64 example:
> If I am developing for an embedded system I know whether it will run
> as 32 or 64 bit, so there is no need to have both.

I agree that this is the most common usecase and that remains unchanged.

> multilib has its merits when it comes to supporting multiple hardware systems.
> However as in the embedded world one is typically targeting a specific
> hardware configuration.
> (actually I don't recall having seen requests for multilib on the ML
> before, although I could have missed it).

These have been requests I've received verbally in general but you'll
see from the replies on the mailing list, Montavista is interested, Koen
is as are a number of others.

> Also I'm somewhat worried by the actual complexity this adds (to the
> build process and the recipes, and timewise probably also to the
> bootstrap process as additional packages have to be built).
> 
> Not sure if that is a desirable route forward, but if we (we as in OE
> members + developers) feel that OE should go that way, I would
> sugggest to have a way to opt-in or opt-out

Multilib will be opt-in. Things will operate just as they do today
unless you specify you want a multilib configuration.

Cheers,

Richard

Re: [poky] Proposed Multilib Implementation Brainstorming

[Esben Haabendal]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> On Wed, 2011-04-06 at 15:21 +0200, Esben Haabendal wrote:
>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> 
>> > On Wed, 2011-04-06 at 14:16 +0200, Esben Haabendal wrote:
>> > I actually believe the core in OE is well suited to multilib support and
>> > using it will show off some of the power in the OE architecture as I
>> > think we can do it better than anyone else before!
>> 
>> The one thing that in my experience pushes most people in other
>> directiosn than OE is the complexity.  Unless OE made simpler, I think
>> it will be quite hard to reach the goal of significantly reducing the
>> number of build systems.
>>
>> I am uncertain to what effect multilib in OE will have (positive and
>> negative), but it is clearly not making OE simpler.
>>
>> I just kind of makes things a bit ehmm... hopeless that the requests I
>> (and others with similar customers) hear from companies are considered
>> out of line for OE evolution, whereas the requests you and others hear
>> can be used as valid arguments for which directions to take.
>> 
>> It is not that I think it will add considerable burden, but is just yet
>> another brick to the wall to pass for developers trying to understand
>> OE.
>
> Nobody is saying requests for simplicity or ease of use are out of line,
> far from it. Its something that is being worked on by Yocto quite
> heavily. What does differ is the approach though, yours is to rip out
> anything you personally don't need,

That's not fair.  While I do "rip out" (to use your choice of colorful
term) stuff in OE-lite, I do and did not proppose to rip out any
features from OE.  I did comment that sstate as a pice of code might not
be give as much value if some of the concepts from OE-lite were
implemented.

> Yocto is taking a more reasoned approach to this and asking questions
> like:

Sorry, but I must take offence on that remark.  I do not feel that your
approach is anymore reasoned than mine.  It is different, and you
clearly prefer your approach, I don't question that.

> "What can we do to make things easier for new users?"
> "Can we make error messages more user friendly?"
> "What code do we have that is needlessly complex and can be simplified?"
> "If we are going to add a new feature, how was we reuse existing
> established concepts?"
> "If something is complex and hard to understand but required, how can we
> document or better explain the concepts to people?"
>
> I'd go as far as to say that things are getting more understandable and
> easier too. Yocto ran an alpha test programme where we took a selection
> of people, some with experience and some with no experience of
> embeddded, pointed them at the quick start guide and asked them to build
> an image, boot it and experiment. The results of this programme with
> Yocto 1.0 are orders of magnitude better than 0.9. This at least shows
> the new user experience is better than it was. Its not the only area
> that needs improvement and there is a long way to go but its a very
> positive start IMO.

Yes, that is measuring Yocto "user" usability.

I am at least as much concerned with long-time developer usability.
Given the amount of long-time developers normally available in OE, I
find it disappointing that at any time, almost noone (at most a very
small handful) dares to comment on RFC's touch core parts, such as
BitBake and gcc/glibc recipes.  I am firmly convinced that "we" (whoever
that might be) can do better than that.  I also believe that this is one
of the biggest reasons that there are so many embedded Linux
buildsystems.  Most people agrees that OE is among (if not the) most
advanced embedded Linux build system, but many of them end up using some
of the simpler tools.  With careful design, and constant refactoring, I
believe that you can lead in both features and simplicity, but you have
to accept that code and design concepts will be discarded when something
better turns up.

/Esben

Re: [poky] Proposed Multilib Implementation Brainstorming

[Hatle, Mark]

On Apr 6, 2011, at 5:06 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>> 
>>> One of the items on our post 1.0 schedule is multilib and we need a plan
>>> of implementation. I've been thinking about this for a while and at
>>> least have some ideas how some of the issues can be handled.
>>> ....
>>> Does this make sense to everyone, are there any questions/ objections/
>>> concerns/ things I've missed?
>> 
>> Which actual OE use-cases justify this kind of addition to OE?
> 
> Several people wanting to use OECore have a requirement of multilib
> support. The typical embedded use case is where you have one main
> application which you might want to run in some kind of large memory
> mode or with some special optimisation (think a database engine) whilst
> the rest of the system is "standard". This requires the ability to mix
> different libraries.

This is a very popular use-case in carrier grade systems.  Back haul voice, data, etc applications that require large routing databases or billing databases.  The other use I've seen for multilibs is some very specialized DVR systems where large amount of video need to stay in memory for quick retrieval.

>> I know it is on the Yocto post 1.0 schedule, but is it actually a good
>> thing for OE?  Maintaining OE recipes is clearly not going to get any
>> easier with multilibs support.
> 
> As detailed in the proposal you will see that the complexity added is
> minimal. It requires a simple enhancement to BBCLASSEXTEND which is
> likely desirable for other reasons too and that is the only real bitbake
> change required. For the metadata, individual recipes remain unaffected
> and also the core conf files are unchanged too. The toolchain dependency
> changes will be the only change affecting users at the recipe level and
> most of the class/machine configuration will be opt in by anyone using
> multilib. The only other invasive change is the package manager
> integration. For rpm, it has good support for multilib already and we're
> just enabling that. For opkg, we still need to determine the best
> approach but the simplistic approach I mentioned will probably suffice
> and anyone wanting true support at the package manager level can use
> rpm.
> 
> For day to day recipe maintenance I don't see much direct impact.
> 

What I've seen in the past is the ability to build multilibs has improved the quality of the software integration by finding poorly constructed headers, hard coded library paths, etc... All things we should fix.

> Cheers,
> 
> Richard
> 
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [poky] Proposed Multilib Implementation Brainstorming

[Hatle, Mark]

On Apr 6, 2011, at 7:29 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2011-04-06 at 10:47 +0200, Frans Meulenbroeks wrote:
>> I think most embedded systems would only use one lib. To take your
>> lib/lib64 example:
>> If I am developing for an embedded system I know whether it will run
>> as 32 or 64 bit, so there is no need to have both.
> 
> I agree that this is the most common usecase and that remains unchanged.
> 

I'd like to stress that existing use-case behavior (non multilib) is imperative to keep the same as we have today.  Multilib is a specialized use case that covers large enough space that it needs to be supported.

>> multilib has its merits when it comes to supporting multiple hardware systems.
>> However as in the embedded world one is typically targeting a specific
>> hardware configuration.
>> (actually I don't recall having seen requests for multilib on the ML
>> before, although I could have missed it).
> 
> These have been requests I've received verbally in general but you'll
> see from the replies on the mailing list, Montavista is interested, Koen
> is as are a number of others.
> 

I'll add Wind River has customers that directly need this functionality.  Many of their products are similar to specialized servers, but really are embedded.

>> Also I'm somewhat worried by the actual complexity this adds (to the
>> build process and the recipes, and timewise probably also to the
>> bootstrap process as additional packages have to be built).
>> 
>> Not sure if that is a desirable route forward, but if we (we as in OE
>> members + developers) feel that OE should go that way, I would
>> sugggest to have a way to opt-in or opt-out
> 
> Multilib will be opt-in. Things will operate just as they do today
> unless you specify you want a multilib configuration.
> 
> Cheers,
> 
> Richard
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: Proposed Multilib Implementation Brainstorming

[Tom Rini]

On 04/06/2011 01:47 AM, Frans Meulenbroeks wrote:
> 2011/4/5 Richard Purdie <richard.purdie@linuxfoundation.org>:
> 
> [...]
>>
>> Does this make sense to everyone, are there any questions/ objections/
>> concerns/ things I've missed?
> 
> I think most embedded systems would only use one lib. To take your
> lib/lib64 example:
> If I am developing for an embedded system I know whether it will run
> as 32 or 64 bit, so there is no need to have both.

What about the case of new 64bit hardware and legacy software that's
still 32bit?  This is a problem that embedded systems have and OE needs
to support

> multilib has its merits when it comes to supporting multiple hardware systems.
> However as in the embedded world one is typically targeting a specific
> hardware configuration.
> (actually I don't recall having seen requests for multilib on the ML
> before, although I could have missed it).

You missed it :)  Roman Khimov posted a very basic thing years ago for
dealing with the x86/x86_64 problem.

> Also I'm somewhat worried by the actual complexity this adds (to the
> build process and the recipes, and timewise probably also to the
> bootstrap process as additional packages have to be built).

I agree, we do need to be careful to make sure the non-multilib case
isn't made more difficult by all of this.

-- 
Tom Rini
Mentor Graphics Corporation

Re: Proposed Multilib Implementation Brainstorming

[Hatle, Mark]

A few comments to the original proposal below...



On Apr 5, 2011, at 2:24 PM, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

> One of the items on our post 1.0 schedule is multilib and we need a plan
> of implementation. I've been thinking about this for a while and at
> least have some ideas how some of the issues can be handled.
> 
> In case anyone isn't familiar with the idea of multilibs, the most
> common example are x86 systems which have both 32 bit and 64 bit
> libraries installed at the same time in /lib/ and /lib64/ directories or
> similar and hence both types of binaries can be used.
> 
> The number of libs isn't limited to two choices as for example the mips
> platform has three choices which can be used in parallel. The x32 ABI
> would mean three possible x86 options too. 
> 

There are really two facets when people talk about multilibs.  The first is what is discussed above, incompatible ABI that are compatible in regards of running on the same hardware and can be collocated within a single filesystem.   The other type of multilibs, common on ARM, is same ABI with different optimizations, machine specific code.

Both types of multilibs need to be supported over time.


> The first step is being able to build given recipes for each of a set of
> given multilibs. This is very similar in principle to the way
> BBCLASSEXTEND is used, the difference it that whilst the native or
> nativesdk classes are used once, for multilib we need a kind of multiple
> inheritance of a class assuming we don't want one class per multilib.
> 
> To do this I'm thinking of a new set of include files of the form
> conf/machine/include/multilib-xxxx.inc, similar in nature to the
> tune-xxx.inc files. A given setup would include each of the multilibs it
> requred. Each multilib include file would look something like:
> 
> conf/machine/include/multilib-x86-lib32.inc:
> 
> MULTILIBS += "lib32"
> 
> BASE_PACKAGE_ARCH_virtclass-multilib-lib32 = "core2-lib32"
> TARGET_CC_ARCH_virtclass-multilib-lib32 = "-m32 -march=core2"
> PACKAGE_EXTRA_ARCHS += "core2-lib32"
> 
> libdir_virtclass-multilib-lib32 = "${exec_prefix}/lib32"
> base_libdir_virtclass-multilib-lib32 = "${base_prefix}/lib32"
> 
> i.e. use overrides to change the target cc architecture specifying the
> required flags to the compiler and also alter the library paths as
> required.
> 
> I'm envisaging passing an optional new parameter to classes in
> BBCLASSEXTEND so for example the definition of a multilib could look
> something like:
> 
> BBCLASSEXTEND_append = " multilib:x86-lib32"
> 
> and all the multilib class would need to do is to manipulation of
> variables including OVERRIDES in a similar way to native.bbclass with
> the complication of handling the parameter.
> 

One of the complications though is deciding what packages to build for which multilibs.  We will need a way to say the default system libraries and components should be say 32-bit.  But I need this one component to be 64-bit....

> The toolchain bootstrap process would become a little complicated by
> this. We'll need to be able to iterate over the list of multilibs and
> configure the compiler with a configuration appropriate to the multilibs
> requested. The compiler should then take care of generating a suitable
> libgcc for each multilib. Where the compiler currently depends on
> virtual/libc-initial, we'll need with a function called to generate the
> dependecies so for example: "virtual/libc virtual/libc-initial-lib32"
> 

I think the toolchain bootstrap is a place where the implementation of the toolchain can make this easier or MUCH harder.  In the environments I am used to using (code sorcery based toolchain) its a single binary that enables the user to use many different multilibs.  However, this is likely not the best/easiest approach for source based toolchain builds.  It's much easier for use to just iteratively build toolchains for each multlib.  Using unique filenames can make this fairly easy to implement with minimal changes to the recipes.

Using something more complex like the CS toolchains, you can do simply by creating specific wrappers that call the base common toolchain with the right parameters to switch the multlib settings..


> The toolchain bootstrap process would hence become something like:
> 
> gcc-cross-initial
> libc-initial
> libc-initial-lib32
> gcc-cross-intermediate
> libc
> libc-lib32
> gcc-cross
> libgcc (package)
> libgcc-lib32 (package)
> gcc-runtime
> gcc-runtime-lib32
> 
> So for any multilib, we're stuck with always building the libc due to
> the way gcc handles multilib. For subsequent tasks, only the libs
> requested in a give image in a given ABI would be built.
> 

Yes, when building multilibs there is a key set that has to exist before you can build anything else.  This normally includes libgcc, libstdcxx, and libc.  Otherwise general depenedencies can handle anything we need...  But as mentioned before we need to understand which items to build and use for filesystem construction.

> Note that as far as the build system is concerned there become distinct
> targets for each ABI so "bitbake mysql" would build the default library
> type, "bitbake mysql-lib32" would build the 32 bit version. Including
> both "mysql" and "mysql-lib32" in an image is where things get
> interesting, the build system will ensure they're both built, then it
> becomes the problem of the image creation and package
> creation/management processes.
> 

So when constructing a task or dependency list set we can do mysql, if we want the default or mysql-lib64 if we want the specific 64 bit version?  I think this is a place where we'll want to evaluate how people interact with the system and configuration files to find the best way to implement it.

> The final tricky part of the puzzle which is under the guidance of the
> package manager. rpm has extensive pre-existing mutlilib support. opkg
> has none.
> 
> It should be possible to convert the build system package name
> libmysql-libx32 into just "libmysql" for the benefit of rpm and use
> rpm's package flavours to handle the multilib issues.
> 
> For opkg there are multiple levels we can take the integration to. It
> should be possible for example to install the "weaker" multilibs first,
> then just install the stronger ones over the top, assuming any file
> overwriting is ok and can be ignored. Teaching opkg about proper
> multilib support would be a bigger project but would be much less of a
> hack/workaround.
> 
> So to summarise the steps needed are:
> 
> a) bitbake enhancement for BBCLASSEXTEND parameter support
> b) Add simple multilib build configuration
> c) Update toolchain bootstrap with multilibs support
> d) Add RPM multilib packaging
> e) Look at opkg multilib integration
> 
> 
> Does this make sense to everyone, are there any questions/ objections/
> concerns/ things I've missed?
> 
> Cheers,
> 
> Richard
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: Proposed Multilib Implementation Brainstorming

[Tom Rini]

On 04/06/2011 11:26 AM, Hatle, Mark wrote:
> A few comments to the original proposal below...
> 
> 
> 
> On Apr 5, 2011, at 2:24 PM, "Richard Purdie"
> <richard.purdie@linuxfoundation.org> wrote:
[snip]
>> and all the multilib class would need to do is to manipulation of 
>> variables including OVERRIDES in a similar way to native.bbclass
>> with the complication of handling the parameter.
>> 
> 
> One of the complications though is deciding what packages to build
> for which multilibs.  We will need a way to say the default system
> libraries and components should be say 32-bit.  But I need this one
> component to be 64-bit....

To be clear (and I think you agree), it also needs to be vice-versa
(64bit except...) and rather arbitrary on what recipes get which treatment.

>> The toolchain bootstrap process would become a little complicated
>> by this. We'll need to be able to iterate over the list of
>> multilibs and configure the compiler with a configuration
>> appropriate to the multilibs requested. The compiler should then
>> take care of generating a suitable libgcc for each multilib. Where
>> the compiler currently depends on virtual/libc-initial, we'll need
>> with a function called to generate the dependecies so for example:
>> "virtual/libc virtual/libc-initial-lib32"
>> 
> 
> I think the toolchain bootstrap is a place where the implementation
> of the toolchain can make this easier or MUCH harder.  In the
> environments I am used to using (code sorcery based toolchain) its a
> single binary that enables the user to use many different multilibs.
> However, this is likely not the best/easiest approach for source
> based toolchain builds.  It's much easier for use to just iteratively
> build toolchains for each multlib.  Using unique filenames can make
> this fairly easy to implement with minimal changes to the recipes.
> 
> Using something more complex like the CS toolchains, you can do
> simply by creating specific wrappers that call the base common
> toolchain with the right parameters to switch the multlib settings..

Putting my community hat on, why not replicate what the CS folks do?
Even with sstate and the yocto 1.1 goal of quicker builds, no one likes
how long the toolchain bottleneck is.

-- 
Tom Rini
Mentor Graphics Corporation

Re: Proposed Multilib Implementation Brainstorming

[Richard Purdie]

On Tue, 2011-04-05 at 16:28 -0700, Jeremy Puhlman wrote:
> > 
> > To do this I'm thinking of a new set of include files of the form
> > conf/machine/include/multilib-xxxx.inc, similar in nature to the
> > tune-xxx.inc files. A given setup would include each of the multilibs it
> > requred. Each multilib include file would look something like:
> > 
> > conf/machine/include/multilib-x86-lib32.inc:
> > 
> > MULTILIBS += "lib32"
> > 
> > BASE_PACKAGE_ARCH_virtclass-multilib-lib32 = "core2-lib32"
> > TARGET_CC_ARCH_virtclass-multilib-lib32 = "-m32 -march=core2"
> > PACKAGE_EXTRA_ARCHS += "core2-lib32"
> > 
> > libdir_virtclass-multilib-lib32 = "${exec_prefix}/lib32"
> > base_libdir_virtclass-multilib-lib32 = "${base_prefix}/lib32"
> > 
> > i.e. use overrides to change the target cc architecture specifying the
> > required flags to the compiler and also alter the library paths as
> > required.
> > 
> > I'm envisaging passing an optional new parameter to classes in
> > BBCLASSEXTEND so for example the definition of a multilib could look
> > something like:
> > 
> > BBCLASSEXTEND_append = " multilib:x86-lib32"
> > 
> > and all the multilib class would need to do is to manipulation of
> > variables including OVERRIDES in a similar way to native.bbclass with
> > the complication of handling the parameter.
> 
> It is possible to do it with out changes to bitbake, but adding
> parametrization to BBCLASSEXTEND would be useful in a number of ways,
> and much more elegant then doing it with out it, and cleaner for that
> matter. Unrelated to multilib, would the full path to the sub parameters
> be configurable via the extended class? i.e. "conf/machine/include/" be
> a configurable value from the multilib class?

I've been thinking this would be the other way around. The act of adding
some conf/machine/include/* files would then trigger the inheritance of
the multilib class with the parameter specified at the point of the
inclusion.

> > For opkg there are multiple levels we can take the integration to. It
> > should be possible for example to install the "weaker" multilibs first,
> > then just install the stronger ones over the top, assuming any file
> > overwriting is ok and can be ignored. 
> 
> If there isn't any kind of compartmentalization of binaries(i.e. the
> alternate abi having a set of packages that only contian runtime libs),
> wouldn't this render the package management on opkg based systems post
> install largely useless other then just adding new components?

Its use case specific and I agree the above is hacky. It would be better
to have that option with known caveats that none at all though.

>  Most
> applications are getting better about it(openssl comes to mind), but
> some may still place arch specific headers in common locations. Many
> like openssl started switching over to arch wrapper headers, but there
> may be some stuff still lurking out there. Could probably be handled on
> a case by case basis.

Yes, we've going to have to watch for this kind of issue and get it
fixed as and where we find it.

> I have been kicking around a still broken POC implementation to this for
> a bit, but this is basically the line of thought(minus the
> parametrization of BBCLASSEXTEND) for a while on the subject. This looks
> mostly right to me.

Great, thanks for the feedback!

Cheers,

Richard

Re: [poky] Proposed Multilib Implementation Brainstorming

[Esben Haabendal]

"Hatle, Mark" <mark.hatle@windriver.com> writes:

> On Apr 6, 2011, at 5:06 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:
>
>> On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
>>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>>> 
>>>> One of the items on our post 1.0 schedule is multilib and we need a plan
>>>> of implementation. I've been thinking about this for a while and at
>>>> least have some ideas how some of the issues can be handled.
>>>> ....
>>>> Does this make sense to everyone, are there any questions/ objections/
>>>> concerns/ things I've missed?
>>> 
>>> Which actual OE use-cases justify this kind of addition to OE?
>> 
>> Several people wanting to use OECore have a requirement of multilib
>> support. The typical embedded use case is where you have one main
>> application which you might want to run in some kind of large memory
>> mode or with some special optimisation (think a database engine) whilst
>> the rest of the system is "standard". This requires the ability to mix
>> different libraries.
>
> This is a very popular use-case in carrier grade systems.  Back haul
> voice, data, etc applications that require large routing databases or
> billing databases.  The other use I've seen for multilibs is some very
> specialized DVR systems where large amount of video need to stay in
> memory for quick retrieval.

None of which really resembles the systems the typical OE developer is
working on.  My point being that what might be important goals for YP
might not be equally important for community OE as such.  Not that it
necesarely is a real conflict, but I believe it is worth keeping in
mind.

>>> I know it is on the Yocto post 1.0 schedule, but is it actually a good
>>> thing for OE?  Maintaining OE recipes is clearly not going to get any
>>> easier with multilibs support.
>> 
>> As detailed in the proposal you will see that the complexity added is
>> minimal. It requires a simple enhancement to BBCLASSEXTEND which is
>> likely desirable for other reasons too and that is the only real bitbake
>> change required. For the metadata, individual recipes remain unaffected
>> and also the core conf files are unchanged too. The toolchain dependency
>> changes will be the only change affecting users at the recipe level and
>> most of the class/machine configuration will be opt in by anyone using
>> multilib. The only other invasive change is the package manager
>> integration. For rpm, it has good support for multilib already and we're
>> just enabling that. For opkg, we still need to determine the best
>> approach but the simplistic approach I mentioned will probably suffice
>> and anyone wanting true support at the package manager level can use
>> rpm.
>> 
>> For day to day recipe maintenance I don't see much direct impact.
>
> What I've seen in the past is the ability to build multilibs has
> improved the quality of the software integration by finding poorly
> constructed headers, hard coded library paths, etc... All things we
> should fix.

Cannot really be against improving software quality :-)

/Esben

Re: [poky] Proposed Multilib Implementation Brainstorming

[Esben Haabendal]

"Hatle, Mark" <mark.hatle@windriver.com> writes:

> On Apr 6, 2011, at 5:06 AM, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:
>
>> On Wed, 2011-04-06 at 09:08 +0200, Esben Haabendal wrote:
>>> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>>> 
>>>> One of the items on our post 1.0 schedule is multilib and we need a plan
>>>> of implementation. I've been thinking about this for a while and at
>>>> least have some ideas how some of the issues can be handled.
>>>> ....
>>>> Does this make sense to everyone, are there any questions/ objections/
>>>> concerns/ things I've missed?
>>> 
>>> Which actual OE use-cases justify this kind of addition to OE?
>> 
>> Several people wanting to use OECore have a requirement of multilib
>> support. The typical embedded use case is where you have one main
>> application which you might want to run in some kind of large memory
>> mode or with some special optimisation (think a database engine) whilst
>> the rest of the system is "standard". This requires the ability to mix
>> different libraries.
>
> This is a very popular use-case in carrier grade systems.  Back haul
> voice, data, etc applications that require large routing databases or
> billing databases.  The other use I've seen for multilibs is some very
> specialized DVR systems where large amount of video need to stay in
> memory for quick retrieval.

None of which really resembles the systems the typical OE developer is
working on.  My point being that what might be important goals for YP
might not be equally important for community OE as such.  Not that it
necesarely is a real conflict, but I believe it is worth keeping in
mind.

>>> I know it is on the Yocto post 1.0 schedule, but is it actually a good
>>> thing for OE?  Maintaining OE recipes is clearly not going to get any
>>> easier with multilibs support.
>> 
>> As detailed in the proposal you will see that the complexity added is
>> minimal. It requires a simple enhancement to BBCLASSEXTEND which is
>> likely desirable for other reasons too and that is the only real bitbake
>> change required. For the metadata, individual recipes remain unaffected
>> and also the core conf files are unchanged too. The toolchain dependency
>> changes will be the only change affecting users at the recipe level and
>> most of the class/machine configuration will be opt in by anyone using
>> multilib. The only other invasive change is the package manager
>> integration. For rpm, it has good support for multilib already and we're
>> just enabling that. For opkg, we still need to determine the best
>> approach but the simplistic approach I mentioned will probably suffice
>> and anyone wanting true support at the package manager level can use
>> rpm.
>> 
>> For day to day recipe maintenance I don't see much direct impact.
>
> What I've seen in the past is the ability to build multilibs has
> improved the quality of the software integration by finding poorly
> constructed headers, hard coded library paths, etc... All things we
> should fix.

Cannot really be against improving software quality :-)

/Esben

Re: Proposed Multilib Implementation Brainstorming

[Koen Kooi]

Op 6 apr 2011, om 20:25 heeft Tom Rini het volgende geschreven:

> On 04/06/2011 01:47 AM, Frans Meulenbroeks wrote:
>> 2011/4/5 Richard Purdie <richard.purdie@linuxfoundation.org>:
>> 
>> [...]
>>> 
>>> Does this make sense to everyone, are there any questions/ objections/
>>> concerns/ things I've missed?
>> 
>> I think most embedded systems would only use one lib. To take your
>> lib/lib64 example:
>> If I am developing for an embedded system I know whether it will run
>> as 32 or 64 bit, so there is no need to have both.
> 
> What about the case of new 64bit hardware and legacy software that's
> still 32bit?  This is a problem that embedded systems have and OE needs
> to support
> 
>> multilib has its merits when it comes to supporting multiple hardware systems.
>> However as in the embedded world one is typically targeting a specific
>> hardware configuration.
>> (actually I don't recall having seen requests for multilib on the ML
>> before, although I could have missed it).
> 
> You missed it :)  Roman Khimov posted a very basic thing years ago for
> dealing with the x86/x86_64 problem.

And Graeme fixed another batch of multilib problems way back at OEDEM in Berlin (2006?). So it has been on the table for a while :)

regards,

Koen

Re: [poky] Proposed Multilib Implementation Brainstorming

[Colin Walters]

On Tue, Apr 5, 2011 at 7:02 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> One of the items on our post 1.0 schedule is multilib and we need a plan
> of implementation. I've been thinking about this for a while and at
> least have some ideas how some of the issues can be handled.
>
> In case anyone isn't familiar with the idea of multilibs, the most
> common example are x86 systems which have both 32 bit and 64 bit
> libraries installed at the same time in /lib/ and /lib64/ directories or
> similar and hence both types of binaries can be used.

I call this "rpm style" multilib.

> The number of libs isn't limited to two choices as for example the mips
> platform has three choices which can be used in parallel. The x32 ABI
> would mean three possible x86 options too.

I don't think there's any precedent in RPM for anything beyond 2 (is there?)

In this discussion we should be sure we're clear on the distinction
between "multilib" and "multiarch".  Multilib's goal is simply
parallel-installable libraries mainly; for the use case stated above,
where you have a 64 bit system and want to run a 32 bit Unix
application, or vice versa.

Multiarch is much more ambitious - it includes cross compilation for
example (something that people keep thinking multilib was designed
for, but it never was).

So are you proposing multilib or multiarch for OE?  It looks like
multilib, but I want to be sure we're clear on the goal.

In your proposal, it looks like you're suggesting the libdir names are
up to the image creator?  If we're doing something new *now* without
any backwards compatibility concerns, we might consider using the
Debian/Ubuntu scheme for libdirs.  This may come at the expense of
having to patch RPM, but I'm not sure how hardcoded the lib/lib64
paths are in the ecosystem; it may be pretty easy.

Re: [poky] Proposed Multilib Implementation Brainstorming

[Hatle, Mark]

On Apr 7, 2011, at 8:36 AM, "Colin Walters" <walters@verbum.org> wrote:

> On Tue, Apr 5, 2011 at 7:02 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> One of the items on our post 1.0 schedule is multilib and we need a plan
>> of implementation. I've been thinking about this for a while and at
>> least have some ideas how some of the issues can be handled.
>> 
>> In case anyone isn't familiar with the idea of multilibs, the most
>> common example are x86 systems which have both 32 bit and 64 bit
>> libraries installed at the same time in /lib/ and /lib64/ directories or
>> similar and hence both types of binaries can be used.
> 
> I call this "rpm style" multilib.
> 
>> The number of libs isn't limited to two choices as for example the mips
>> platform has three choices which can be used in parallel. The x32 ABI
>> would mean three possible x86 options too.
> 
> I don't think there's any precedent in RPM for anything beyond 2 (is there?)
> 

RPM uses file 'colors' to handle the multilib conflict and installation policy.  The colors are defined as a bit field where so far only 3 bit are defined.  Elf 32, elf 64, and MIPS n32.

So we have a way to do more then three, currently MIPS64 n32 is the only architecture this is defined on.

> In this discussion we should be sure we're clear on the distinction
> between "multilib" and "multiarch".  Multilib's goal is simply
> parallel-installable libraries mainly; for the use case stated above,
> where you have a 64 bit system and want to run a 32 bit Unix
> application, or vice versa.
> 

I think definition may be an issue here.  When I speak of multilibs I am talking primarily about multiliple non-conflicting, ABI incompatible libraries being installed at the same time.  This is the ELF 32 and ELF 64 case where the items can't be linked together, but can be installed and jrun on the same machine.

The alternative is the case on ARM where we have one single ABI but multiple ways to implement the code.. Thumb, neon, vfp, etc being options.  We want to do this as well, but while related, I see it as a diffent use case.

> Multiarch is much more ambitious - it includes cross compilation for
> example (something that people keep thinking multilib was designed
> for, but it never was).
> 
> So are you proposing multilib or multiarch for OE?  It looks like
> multilib, but I want to be sure we're clear on the goal.
> 

Multilib, install of binaries that will execute natively on the machine.

> In your proposal, it looks like you're suggesting the libdir names are
> up to the image creator?  If we're doing something new *now* without
> any backwards compatibility concerns, we might consider using the
> Debian/Ubuntu scheme for libdirs.  This may come at the expense of
> having to patch RPM, but I'm not sure how hardcoded the lib/lib64
> paths are in the ecosystem; it may be pretty easy.

Lib dir names are already defined within the system configuration, but there hasn't been a lot of work to prove it's being used and is working properly.  (everyone uses /lib today...)

Rpm really doesn't care or pay attention to the path, it's the elf file type that matters...

> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [poky] Proposed Multilib Implementation Brainstorming

[Colin Walters]

On Thu, Apr 7, 2011 at 12:10 PM, Hatle, Mark <mark.hatle@windriver.com> wrote:
>
> I think definition may be an issue here.  When I speak of multilibs I am talking primarily about multiliple non-conflicting, ABI incompatible libraries being installed at the same time.  This is the ELF 32 and ELF 64 case where the items can't be linked together, but can be installed and jrun on the same machine.

Right, that's "multilib".

> The alternative is the case on ARM where we have one single ABI but multiple ways to implement the code.. Thumb, neon, vfp, etc being options.  We want to do this as well, but while related, I see it as a diffent use case.

Isn't this already covered by glibc's support for
/lib/$processor_feature?  On my Fedora 14 x86_64 system I have
/lib/i686/nosegneg for example, and a simple "strace true" shows us
trying to find a variant of libc that is built for TLS (and not
finding it):

$ strace true
execve("/bin/true", ["true"], [/* 54 vars */]) = 0
brk(0)                                  = 0xe59000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0x7f54472b0000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/src/build/jhbuild/lib64/tls/x86_64/libc.so.6", O_RDONLY) = -1
ENOENT (No such file or directory)

Re: [poky] Proposed Multilib Implementation Brainstorming

[Hatle, Mark]

On Apr 7, 2011, at 9:53 AM, "Colin Walters" <walters@verbum.org> wrote:

> On Thu, Apr 7, 2011 at 12:10 PM, Hatle, Mark <mark.hatle@windriver.com> wrote:
>> 
>> I think definition may be an issue here.  When I speak of multilibs I am talking primarily about multiliple non-conflicting, ABI incompatible libraries being installed at the same time.  This is the ELF 32 and ELF 64 case where the items can't be linked together, but can be installed and jrun on the same machine.
> 
> Right, that's "multilib".
> 
>> The alternative is the case on ARM where we have one single ABI but multiple ways to implement the code.. Thumb, neon, vfp, etc being options.  We want to do this as well, but while related, I see it as a diffent use case.
> 
> Isn't this already covered by glibc's support for
> /lib/$processor_feature?  On my Fedora 14 x86_64 system I have
> /lib/i686/nosegneg for example, and a simple "strace true" shows us
> trying to find a variant of libc that is built for TLS (and not
> finding it):
> 

The needs for the arm case include both executables and libraries.  The processor feature can be used, but that only covers the libraries themselves.  We want a mechanism to say that the whole system is built thumb2, except for a particular compilation modules, library and/or application.  (generally is done on a full recipe basis, and not on a single compilation module..)

The other issue on ARM embedded systems is that the processor feature approach typically makes the system larger, by providing both the lesser optimized and more completely optimized libraries....  I've had resistence to this approach from customer over that reason.  (mind you I'm in favor of making that easier to use in OE as well.)

> $ strace true
> execve("/bin/true", ["true"], [/* 54 vars */]) = 0
> brk(0)                                  = 0xe59000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7f54472b0000
> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
> open("/src/build/jhbuild/lib64/tls/x86_64/libc.so.6", O_RDONLY) = -1
> ENOENT (No such file or directory)

Re: [poky] Proposed Multilib Implementation Brainstorming

[Koen Kooi]

Op 7 apr 2011, om 18:53 heeft Colin Walters het volgende geschreven:

> On Thu, Apr 7, 2011 at 12:10 PM, Hatle, Mark <mark.hatle@windriver.com> wrote:
>> 
>> I think definition may be an issue here.  When I speak of multilibs I am talking primarily about multiliple non-conflicting, ABI incompatible libraries being installed at the same time.  This is the ELF 32 and ELF 64 case where the items can't be linked together, but can be installed and jrun on the same machine.
> 
> Right, that's "multilib".
> 
>> The alternative is the case on ARM where we have one single ABI but multiple ways to implement the code.. Thumb, neon, vfp, etc being options.  We want to do this as well, but while related, I see it as a diffent use case.
> 
> Isn't this already covered by glibc's support for
> /lib/$processor_feature?  On my Fedora 14 x86_64 system I have
> /lib/i686/nosegneg for example, and a simple "strace true" shows us
> trying to find a variant of libc that is built for TLS (and not
> finding it):
> 
> $ strace true
> execve("/bin/true", ["true"], [/* 54 vars */]) = 0
> brk(0)                                  = 0xe59000
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
> 0) = 0x7f54472b0000
> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
> open("/src/build/jhbuild/lib64/tls/x86_64/libc.so.6", O_RDONLY) = -1
> ENOENT (No such file or directory)

That's what I call 'hwcap'. OE currently lacks a decent way to build and install the permuations people want. For armv7a this boils down to NEON/VENUM or not, TI, Freescale and Qualcomm have it, Nvidia and Marvell don't.
This type of setup on ARM is only usefull for people who think that one rootfs to rule all boards is the best thing since sliced bread. It does make it really easy to hand people a rootfs and have it "just work" and not get caught in "why didn't you license NEON?" debates :)

regards,

Koen

Re: [poky] Proposed Multilib Implementation Brainstorming

[Hatle, Mark]

Sorry to reply to myself but I forgot one thing..



On Apr 7, 2011, at 10:05 AM, "Hatle, Mark" <mark.hatle@windriver.com> wrote:

> 
> 
> 
> 
> On Apr 7, 2011, at 9:53 AM, "Colin Walters" <walters@verbum.org> wrote:
> 
>> On Thu, Apr 7, 2011 at 12:10 PM, Hatle, Mark <mark.hatle@windriver.com> wrote:
>>> 
>>> I think definition may be an issue here.  When I speak of multilibs I am talking primarily about multiliple non-conflicting, ABI incompatible libraries being installed at the same time.  This is the ELF 32 and ELF 64 case where the items can't be linked together, but can be installed and jrun on the same machine.
>> 
>> Right, that's "multilib".
>> 
>>> The alternative is the case on ARM where we have one single ABI but multiple ways to implement the code.. Thumb, neon, vfp, etc being options.  We want to do this as well, but while related, I see it as a diffent use case.
>> 
>> Isn't this already covered by glibc's support for
>> /lib/$processor_feature?  On my Fedora 14 x86_64 system I have
>> /lib/i686/nosegneg for example, and a simple "strace true" shows us
>> trying to find a variant of libc that is built for TLS (and not
>> finding it):
>> 
> 
> The needs for the arm case include both executables and libraries.  The processor feature can be used, but that only covers the libraries themselves.  We want a mechanism to say that the whole system is built thumb2, except for a particular compilation modules, library and/or application.  (generally is done on a full recipe basis, and not on a single compilation module..)
> 

From an RPM perspective the real difference is that one type uses multiple recipe builds and let's install time policy decide what to install.. (this is the multilib mentioned above)

The ARM case I mentioned is really optimization based.. There are a few key components like gcc that need to be configured for the multiple types of systems, but the decision on what to install (a.k.a. What special optimizations are required) are made at recipe compile time.

This is why they're related cases, we need a common way to specify these multiple library type optimization and configuration flags... But the eventual result install-time vs compile-time is what makes them different.

> The other issue on ARM embedded systems is that the processor feature approach typically makes the system larger, by providing both the lesser optimized and more completely optimized libraries....  I've had resistence to this approach from customer over that reason.  (mind you I'm in favor of making that easier to use in OE as well.)
> 
>> $ strace true
>> execve("/bin/true", ["true"], [/* 54 vars */]) = 0
>> brk(0)                                  = 0xe59000
>> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
>> 0) = 0x7f54472b0000
>> access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
>> open("/src/build/jhbuild/lib64/tls/x86_64/libc.so.6", O_RDONLY) = -1
>> ENOENT (No such file or directory)
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky