RFC: Secondary Toolchain

RFC: Secondary Toolchain

[Mark Hatle]

We have an issue where we'd like to have an alternative toolchain (assembler, 
linker, compiler) available for our customers to selectively use.  However, 
before we just go off and implement something, I'd like at least some basic 
consensus on what the best practice is for doing this work.  Below is my attempt 
at an early proposal.

Background
----------------

Many companies have commercial / highly optimized toolchains for specific 
purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For 
(potentially) better performance with some applications a mechanism to both 
provide the hooks for the alternative toolchain as well as a way to active it 
per-package is desired.

This work assumes that the third party toolchain is generally compatible with 
the idea of sysroots, linking to the libc provided by OE, and generally 
compatible with GNU conventions.

However, as part of the third party toolchain, it may not be GNU compatible. 
This means many Open Source applications simply may not work with this 
toolchain.  That means that we need to have a way for a toolchain to blacklist 
(or whitelist) specific recipes.  This way only supported components can be 
built by the user, avoiding numerous complaints.

Currently OE has a method to generate an SDK based on the GNU toolchain.  We 
would like to be able to also export the external toolchain along with the SDK, 
effectively providing both the GNU toolchain and the third party toolchain using 
the common sysroot.

We need a way to active the third party toolchain on a per-package basis.  This 
activation will need to use the existing sysroot, but be able to pass different 
C, C++, LD, AS and other flags as specified by the third party toolchain.

Finally third party toolchains should be implemented as layers that can easily 
plug into OE.

Implementation
---------------------

Add an OVERRIDE to specify the alternative toolchain.  Can this be done within 
the layer by doing something like:

OVERRIDE_append = ":toolchain-${TOOLCHAIN}"

TOOLCHAIN = "gnu" (or "icc")

To activate the toolchain you would use things like:

TOOLCHAIN_pn-bash = 'icc'

To define the correct behavior for the toolchain, the following would need to be 
defined (with _toolchain-<toolchain>):

TARGET_CPPFLAGS
TARGET_CFLAGS
TARGET_CXXFLAGS
TARGET_LDFLAGS
CC
CXX
F77
CPP
LD
CCLD
AR
AS
RANLIB
STRIP
OBJCOPY
OBJDUMP
NM
FULL_OPTIMIZATIONS
DEBUG_OPTIMIZATION

Is anyone aware of any other items that may require additional items?  Will the 
above actually work?  Using the override of the TOOLCHAIN_… will that actually 
change the override values or do we get stuck?

Comments/suggestions appreciated!
--Mark

Re: RFC: Secondary Toolchain

[Trevor Woerner]

I'm curious to know if anyone (I certainly wouldn't be able to!) can
take a guess whether this would "play nicely" with external
toolchains?

In other words, if some recipe is already PROVIDES'ing
virtual/${TARGET_PREFIX}gcc etc would the correct toolchain be used
for the special packages needing the secondary toolchain?

Re: RFC: Secondary Toolchain

[Mark Hatle]

On 10/4/12 1:15 PM, Trevor Woerner wrote:
> I'm curious to know if anyone (I certainly wouldn't be able to!) can
> take a guess whether this would "play nicely" with external
> toolchains?
>
> In other words, if some recipe is already PROVIDES'ing
> virtual/${TARGET_PREFIX}gcc etc would the correct toolchain be used
> for the special packages needing the secondary toolchain?

My expectations is that any existing dependency set will be managed and 
maintained by the existing primary toolchain, unless a given recipe has a 
specific dependency on a secondary toolchain item.

I.e. the example above will -always- be the primary toolchain from a dependency 
resolution standpoint..

If the recipe adds "virtual/icc", then something, such as icc, needs to exist to 
provide that.

Does this seem like a reasonable behavior and expectation?  (The thing to 
remember is this secondary toolchain is just that.. an alternative to the 
primary for specific users and NOT general purpose....)

--Mark

> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: RFC: Secondary Toolchain

[McClintock Matthew-B29882]

On Thu, Oct 4, 2012 at 1:02 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> We have an issue where we'd like to have an alternative toolchain
> (assembler, linker, compiler) available for our customers to selectively
> use.  However, before we just go off and implement something, I'd like at
> least some basic consensus on what the best practice is for doing this work.
> Below is my attempt at an early proposal.
>
> Background
> ----------------
>
> Many companies have commercial / highly optimized toolchains for specific
> purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For
> (potentially) better performance with some applications a mechanism to both
> provide the hooks for the alternative toolchain as well as a way to active
> it per-package is desired.
>
> This work assumes that the third party toolchain is generally compatible
> with the idea of sysroots, linking to the libc provided by OE, and generally
> compatible with GNU conventions.
>
> However, as part of the third party toolchain, it may not be GNU compatible.
> This means many Open Source applications simply may not work with this
> toolchain.  That means that we need to have a way for a toolchain to
> blacklist (or whitelist) specific recipes.  This way only supported
> components can be built by the user, avoiding numerous complaints.
>
> Currently OE has a method to generate an SDK based on the GNU toolchain.  We
> would like to be able to also export the external toolchain along with the
> SDK, effectively providing both the GNU toolchain and the third party
> toolchain using the common sysroot.
>
> We need a way to active the third party toolchain on a per-package basis.
> This activation will need to use the existing sysroot, but be able to pass
> different C, C++, LD, AS and other flags as specified by the third party
> toolchain.
>
> Finally third party toolchains should be implemented as layers that can
> easily plug into OE.
>
> Implementation
> ---------------------
>
> Add an OVERRIDE to specify the alternative toolchain.  Can this be done
> within the layer by doing something like:
>
> OVERRIDE_append = ":toolchain-${TOOLCHAIN}"
>
> TOOLCHAIN = "gnu" (or "icc")
>
> To activate the toolchain you would use things like:
>
> TOOLCHAIN_pn-bash = 'icc'
>
> To define the correct behavior for the toolchain, the following would need
> to be defined (with _toolchain-<toolchain>):
>
> TARGET_CPPFLAGS
> TARGET_CFLAGS
> TARGET_CXXFLAGS
> TARGET_LDFLAGS
> CC
> CXX
> F77
> CPP
> LD
> CCLD
> AR
> AS
> RANLIB
> STRIP
> OBJCOPY
> OBJDUMP
> NM
> FULL_OPTIMIZATIONS
> DEBUG_OPTIMIZATION
>
> Is anyone aware of any other items that may require additional items?  Will
> the above actually work?  Using the override of the TOOLCHAIN_… will that
> actually change the override values or do we get stuck?

This seems orthogonal to actually implementing the recipe which would
procide 'icc'?

-M

>
> Comments/suggestions appreciated!
> --Mark
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: RFC: Secondary Toolchain

[Mark Hatle]

On 10/4/12 2:03 PM, McClintock Matthew-B29882 wrote:
> On Thu, Oct 4, 2012 at 1:02 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
>> We have an issue where we'd like to have an alternative toolchain
>> (assembler, linker, compiler) available for our customers to selectively
>> use.  However, before we just go off and implement something, I'd like at
>> least some basic consensus on what the best practice is for doing this work.
>> Below is my attempt at an early proposal.
>>
>> Background
>> ----------------
>>
>> Many companies have commercial / highly optimized toolchains for specific
>> purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For
>> (potentially) better performance with some applications a mechanism to both
>> provide the hooks for the alternative toolchain as well as a way to active
>> it per-package is desired.
>>
>> This work assumes that the third party toolchain is generally compatible
>> with the idea of sysroots, linking to the libc provided by OE, and generally
>> compatible with GNU conventions.
>>
>> However, as part of the third party toolchain, it may not be GNU compatible.
>> This means many Open Source applications simply may not work with this
>> toolchain.  That means that we need to have a way for a toolchain to
>> blacklist (or whitelist) specific recipes.  This way only supported
>> components can be built by the user, avoiding numerous complaints.
>>
>> Currently OE has a method to generate an SDK based on the GNU toolchain.  We
>> would like to be able to also export the external toolchain along with the
>> SDK, effectively providing both the GNU toolchain and the third party
>> toolchain using the common sysroot.
>>
>> We need a way to active the third party toolchain on a per-package basis.
>> This activation will need to use the existing sysroot, but be able to pass
>> different C, C++, LD, AS and other flags as specified by the third party
>> toolchain.
>>
>> Finally third party toolchains should be implemented as layers that can
>> easily plug into OE.
>>
>> Implementation
>> ---------------------
>>
>> Add an OVERRIDE to specify the alternative toolchain.  Can this be done
>> within the layer by doing something like:
>>
>> OVERRIDE_append = ":toolchain-${TOOLCHAIN}"
>>
>> TOOLCHAIN = "gnu" (or "icc")
>>
>> To activate the toolchain you would use things like:
>>
>> TOOLCHAIN_pn-bash = 'icc'
>>
>> To define the correct behavior for the toolchain, the following would need
>> to be defined (with _toolchain-<toolchain>):
>>
>> TARGET_CPPFLAGS
>> TARGET_CFLAGS
>> TARGET_CXXFLAGS
>> TARGET_LDFLAGS
>> CC
>> CXX
>> F77
>> CPP
>> LD
>> CCLD
>> AR
>> AS
>> RANLIB
>> STRIP
>> OBJCOPY
>> OBJDUMP
>> NM
>> FULL_OPTIMIZATIONS
>> DEBUG_OPTIMIZATION
>>
>> Is anyone aware of any other items that may require additional items?  Will
>> the above actually work?  Using the override of the TOOLCHAIN_… will that
>> actually change the override values or do we get stuck?
>
> This seems orthogonal to actually implementing the recipe which would
> procide 'icc'?

That is correct.  I'm trying to establish a best practice for the layer 
configuration, as well as general distribution/recipe configuration.

What I really don't want to see is 5 companies implementing similar 
functionality and doing it in a completely incompatible way.  If the variables 
and override mechanism above is reasonable, then it gives people a roadmap to 
get started.

--Mark

> -M
>
>>
>> Comments/suggestions appreciated!
>> --Mark
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: RFC: Secondary Toolchain

[Khem Raj]

On Thu, Oct 4, 2012 at 11:02 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
> We have an issue where we'd like to have an alternative toolchain
> (assembler, linker, compiler) available for our customers to selectively
> use.  However, before we just go off and implement something, I'd like at
> least some basic consensus on what the best practice is for doing this work.
> Below is my attempt at an early proposal.
>
> Background
> ----------------
>
> Many companies have commercial / highly optimized toolchains for specific
> purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For
> (potentially) better performance with some applications a mechanism to both
> provide the hooks for the alternative toolchain as well as a way to active
> it per-package is desired.
>
> This work assumes that the third party toolchain is generally compatible
> with the idea of sysroots, linking to the libc provided by OE, and generally
> compatible with GNU conventions.
>
> However, as part of the third party toolchain, it may not be GNU compatible.
> This means many Open Source applications simply may not work with this
> toolchain.  That means that we need to have a way for a toolchain to
> blacklist (or whitelist) specific recipes.  This way only supported
> components can be built by the user, avoiding numerous complaints.
>
> Currently OE has a method to generate an SDK based on the GNU toolchain.  We
> would like to be able to also export the external toolchain along with the
> SDK, effectively providing both the GNU toolchain and the third party
> toolchain using the common sysroot.
>
> We need a way to active the third party toolchain on a per-package basis.

Per package bases is a bit iffy from my POV if you consider the compiler runtime
and C library runtime and so forth. Is it intended to just compile and
be able to
cope with the default runtimes on images or do you also intend to figure out
ways for multiple runtimes to live together.

> This activation will need to use the existing sysroot, but be able to pass
> different C, C++, LD, AS and other flags as specified by the third party
> toolchain.

that sort of answers the above. So we are assuming that toolchain that
can interoperate
will be using GNU runtime

There are cases where you might want something like ELLCC which avoids
complete GPL runtime e.g.
I think it could be divided into runtime  + tools where runtime lives
in a different layer but
essentially offering to let people use builtin C library and runtime
may not be a bad choice


>
> Finally third party toolchains should be implemented as layers that can
> easily plug into OE.

agreed. I think we need to document the interfaces in OE-Core for this
too so that
people can rely on the interface. Otherwise it will be constant pain
for keeping those
layers working together.

>
> Implementation
> ---------------------
>
> Add an OVERRIDE to specify the alternative toolchain.  Can this be done
> within the layer by doing something like:
>
> OVERRIDE_append = ":toolchain-${TOOLCHAIN}"
>
> TOOLCHAIN = "gnu" (or "icc")
>
> To activate the toolchain you would use things like:
>
> TOOLCHAIN_pn-bash = 'icc'
>
> To define the correct behavior for the toolchain, the following would need
> to be defined (with _toolchain-<toolchain>):
>
> TARGET_CPPFLAGS
> TARGET_CFLAGS
> TARGET_CXXFLAGS
> TARGET_LDFLAGS
> CC
> CXX
> F77
> CPP
> LD
> CCLD
> AR
> AS
> RANLIB
> STRIP
> OBJCOPY
> OBJDUMP
> NM
> FULL_OPTIMIZATIONS
> DEBUG_OPTIMIZATION
>
> Is anyone aware of any other items that may require additional items?  Will
> the above actually work?  Using the override of the TOOLCHAIN_… will that
> actually change the override values or do we get stuck?
>
> Comments/suggestions appreciated!
> --Mark
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: RFC: Secondary Toolchain

[Mark Hatle]

On 10/4/12 3:36 PM, Khem Raj wrote:
> On Thu, Oct 4, 2012 at 11:02 AM, Mark Hatle <mark.hatle@windriver.com> wrote:
>> We have an issue where we'd like to have an alternative toolchain
>> (assembler, linker, compiler) available for our customers to selectively
>> use.  However, before we just go off and implement something, I'd like at
>> least some basic consensus on what the best practice is for doing this work.
>> Below is my attempt at an early proposal.
>>
>> Background
>> ----------------
>>
>> Many companies have commercial / highly optimized toolchains for specific
>> purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For
>> (potentially) better performance with some applications a mechanism to both
>> provide the hooks for the alternative toolchain as well as a way to active
>> it per-package is desired.
>>
>> This work assumes that the third party toolchain is generally compatible
>> with the idea of sysroots, linking to the libc provided by OE, and generally
>> compatible with GNU conventions.
>>
>> However, as part of the third party toolchain, it may not be GNU compatible.
>> This means many Open Source applications simply may not work with this
>> toolchain.  That means that we need to have a way for a toolchain to
>> blacklist (or whitelist) specific recipes.  This way only supported
>> components can be built by the user, avoiding numerous complaints.
>>
>> Currently OE has a method to generate an SDK based on the GNU toolchain.  We
>> would like to be able to also export the external toolchain along with the
>> SDK, effectively providing both the GNU toolchain and the third party
>> toolchain using the common sysroot.
>>
>> We need a way to active the third party toolchain on a per-package basis.
>
> Per package bases is a bit iffy from my POV if you consider the compiler runtime
> and C library runtime and so forth. Is it intended to just compile and
> be able to
> cope with the default runtimes on images or do you also intend to figure out
> ways for multiple runtimes to live together.

This is only one runtime.  You have multiple compilers all capable of producing 
software compatible with the same ABI.  The default (oe) compiler is used, 
unless otherwise configured.  The alternative(s) are used for optimization of 
various items.  I mentioned ICC, because it's one that I know today people are 
using and it is capable of replacing gcc in many applications.

The issue is that I want to enable someone to use an alternative compiler, but I 
don't want to do anything beyond enabling it if it doesn't work.

>> This activation will need to use the existing sysroot, but be able to pass
>> different C, C++, LD, AS and other flags as specified by the third party
>> toolchain.
>
> that sort of answers the above. So we are assuming that toolchain that
> can interoperate
> will be using GNU runtime
>
> There are cases where you might want something like ELLCC which avoids
> complete GPL runtime e.g.
> I think it could be divided into runtime  + tools where runtime lives
> in a different layer but
> essentially offering to let people use builtin C library and runtime
> may not be a bad choice

When the runtime changes, I consider that to be a different abi and outside the 
scope of this.  Thats where the machine/multilib/etc configuration items come 
into play.  (I.e. some architectures support an alternative compiler for bare 
metal applications -- in those cases the canonical arch is no longer 
-linux[-gnu]... so they would simply be treated as a different architecture.)

>
>>
>> Finally third party toolchains should be implemented as layers that can
>> easily plug into OE.
>
> agreed. I think we need to document the interfaces in OE-Core for this
> too so that
> people can rely on the interface. Otherwise it will be constant pain
> for keeping those
> layers working together.

Exactly.. I don't want people coming to me saying I want XYZ compiler, but it's 
implemented for my semi-conduction specific Linux and won't work with your OSV 
based Linux.  That doesn't help anyone (unless they are intentionally trying to 
be incompatible!)  So if we can provide a functional best practice and example 
of how to do this, then when people want to provide (and use) a secondary 
toolchain -- it's available to them.

>>
>> Implementation
>> ---------------------
>>
>> Add an OVERRIDE to specify the alternative toolchain.  Can this be done
>> within the layer by doing something like:
>>
>> OVERRIDE_append = ":toolchain-${TOOLCHAIN}"
>>
>> TOOLCHAIN = "gnu" (or "icc")
>>
>> To activate the toolchain you would use things like:
>>
>> TOOLCHAIN_pn-bash = 'icc'
>>
>> To define the correct behavior for the toolchain, the following would need
>> to be defined (with _toolchain-<toolchain>):
>>
>> TARGET_CPPFLAGS
>> TARGET_CFLAGS
>> TARGET_CXXFLAGS
>> TARGET_LDFLAGS
>> CC
>> CXX
>> F77
>> CPP
>> LD
>> CCLD
>> AR
>> AS
>> RANLIB
>> STRIP
>> OBJCOPY
>> OBJDUMP
>> NM
>> FULL_OPTIMIZATIONS
>> DEBUG_OPTIMIZATION
>>
>> Is anyone aware of any other items that may require additional items?  Will
>> the above actually work?  Using the override of the TOOLCHAIN_… will that
>> actually change the override values or do we get stuck?
>>
>> Comments/suggestions appreciated!
>> --Mark
>>
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: RFC: Secondary Toolchain

[Khem Raj]

On Thu, Oct 4, 2012 at 2:00 PM, Mark Hatle <mark.hatle@windriver.com> wrote:
> On 10/4/12 3:36 PM, Khem Raj wrote:
>>
>> On Thu, Oct 4, 2012 at 11:02 AM, Mark Hatle <mark.hatle@windriver.com>
>> wrote:
>>>
>>> We have an issue where we'd like to have an alternative toolchain
>>> (assembler, linker, compiler) available for our customers to selectively
>>> use.  However, before we just go off and implement something, I'd like at
>>> least some basic consensus on what the best practice is for doing this
>>> work.
>>> Below is my attempt at an early proposal.
>>>
>>> Background
>>> ----------------
>>>
>>> Many companies have commercial / highly optimized toolchains for specific
>>> purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For
>>> (potentially) better performance with some applications a mechanism to
>>> both
>>> provide the hooks for the alternative toolchain as well as a way to
>>> active
>>> it per-package is desired.
>>>
>>> This work assumes that the third party toolchain is generally compatible
>>> with the idea of sysroots, linking to the libc provided by OE, and
>>> generally
>>> compatible with GNU conventions.
>>>
>>> However, as part of the third party toolchain, it may not be GNU
>>> compatible.
>>> This means many Open Source applications simply may not work with this
>>> toolchain.  That means that we need to have a way for a toolchain to
>>> blacklist (or whitelist) specific recipes.  This way only supported
>>> components can be built by the user, avoiding numerous complaints.
>>>
>>> Currently OE has a method to generate an SDK based on the GNU toolchain.
>>> We
>>> would like to be able to also export the external toolchain along with
>>> the
>>> SDK, effectively providing both the GNU toolchain and the third party
>>> toolchain using the common sysroot.
>>>
>>> We need a way to active the third party toolchain on a per-package basis.
>>
>>
>> Per package bases is a bit iffy from my POV if you consider the compiler
>> runtime
>> and C library runtime and so forth. Is it intended to just compile and
>> be able to
>> cope with the default runtimes on images or do you also intend to figure
>> out
>> ways for multiple runtimes to live together.
>
>
> This is only one runtime.  You have multiple compilers all capable of
> producing software compatible with the same ABI.  The default (oe) compiler
> is used, unless otherwise configured.  The alternative(s) are used for
> optimization of various items.  I mentioned ICC, because it's one that I
> know today people are using and it is capable of replacing gcc in many
> applications.

Thanks that clarifies the scope. I thought you were going for a
general solution where you
could insert any toolchain.

>
> The issue is that I want to enable someone to use an alternative compiler,
> but I don't want to do anything beyond enabling it if it doesn't work.
>
>
>>> This activation will need to use the existing sysroot, but be able to
>>> pass
>>> different C, C++, LD, AS and other flags as specified by the third party
>>> toolchain.
>>
>>
>> that sort of answers the above. So we are assuming that toolchain that
>> can interoperate
>> will be using GNU runtime
>>
>> There are cases where you might want something like ELLCC which avoids
>> complete GPL runtime e.g.
>> I think it could be divided into runtime  + tools where runtime lives
>> in a different layer but
>> essentially offering to let people use builtin C library and runtime
>> may not be a bad choice
>
>
> When the runtime changes, I consider that to be a different abi and outside
> the scope of this.  Thats where the machine/multilib/etc configuration items
> come into play.  (I.e. some architectures support an alternative compiler
> for bare metal applications -- in those cases the canonical arch is no
> longer -linux[-gnu]... so they would simply be treated as a different
> architecture.)
>
>
>>
>>>
>>> Finally third party toolchains should be implemented as layers that can
>>> easily plug into OE.
>>
>>
>> agreed. I think we need to document the interfaces in OE-Core for this
>> too so that
>> people can rely on the interface. Otherwise it will be constant pain
>> for keeping those
>> layers working together.
>
>
> Exactly.. I don't want people coming to me saying I want XYZ compiler, but
> it's implemented for my semi-conduction specific Linux and won't work with
> your OSV based Linux.  That doesn't help anyone (unless they are
> intentionally trying to be incompatible!)  So if we can provide a functional
> best practice and example of how to do this, then when people want to
> provide (and use) a secondary toolchain -- it's available to them.
>
>
>>>
>>> Implementation
>>> ---------------------
>>>
>>> Add an OVERRIDE to specify the alternative toolchain.  Can this be done
>>> within the layer by doing something like:
>>>
>>> OVERRIDE_append = ":toolchain-${TOOLCHAIN}"
>>>
>>> TOOLCHAIN = "gnu" (or "icc")
>>>
>>> To activate the toolchain you would use things like:
>>>
>>> TOOLCHAIN_pn-bash = 'icc'
>>>
>>> To define the correct behavior for the toolchain, the following would
>>> need
>>> to be defined (with _toolchain-<toolchain>):
>>>
>>> TARGET_CPPFLAGS
>>> TARGET_CFLAGS
>>> TARGET_CXXFLAGS
>>> TARGET_LDFLAGS
>>> CC
>>> CXX
>>> F77
>>> CPP
>>> LD
>>> CCLD
>>> AR
>>> AS
>>> RANLIB
>>> STRIP
>>> OBJCOPY
>>> OBJDUMP
>>> NM
>>> FULL_OPTIMIZATIONS
>>> DEBUG_OPTIMIZATION
>>>
>>> Is anyone aware of any other items that may require additional items?
>>> Will
>>> the above actually work?  Using the override of the TOOLCHAIN_… will that
>>> actually change the override values or do we get stuck?
>>>
>>> Comments/suggestions appreciated!
>>> --Mark
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>
>

Re: RFC: Secondary Toolchain

[Phil Blundell]

On Thu, 2012-10-04 at 16:00 -0500, Mark Hatle wrote:
> This is only one runtime.  You have multiple compilers all capable of producing 
> software compatible with the same ABI.  The default (oe) compiler is used, 
> unless otherwise configured.  The alternative(s) are used for optimization of 
> various items.  I mentioned ICC, because it's one that I know today people are 
> using and it is capable of replacing gcc in many applications.

If that's the case then it doesn't seem like it ought to be necessary to
override most of the variables you mentioned.  Tools like ar, ranlib and
strip are generic to a binary format and should work fine with the
output from any compiler.  Also, if you're linking with GNU libc then
the chances are that you need to use one of the GNU linkers (be that
ld.bfd or gold) since I suspect most third party link editors will not
work very well.  And, if you have assembly source on hand, it's probably
going to be written in GAS dialect so selecting an alternative assembler
as ${AS} is unlikely to gain much.

So, it kind of seems like the only variables you're likely to need to
change are CC, CXX and the corresponding FLAGS.  I guess you might also
want to override F77 in theory though I'm not sure there are many people
using Fortran in an embedded environment.

Your proposed "toolchain-icc" override sounds like a reasonable enough
way to accomplish that, anyway.  The only problem it doesn't seem to
solve is the use-case of "everything that supports icc, I want to build
with icc; everything that doesn't support icc, I want to build with
gcc".  But I'm not sure this really matters much.

p.

Re: RFC: Secondary Toolchain

[Mark Hatle]

On 10/4/12 4:16 PM, Phil Blundell wrote:
> On Thu, 2012-10-04 at 16:00 -0500, Mark Hatle wrote:
>> This is only one runtime.  You have multiple compilers all capable of producing
>> software compatible with the same ABI.  The default (oe) compiler is used,
>> unless otherwise configured.  The alternative(s) are used for optimization of
>> various items.  I mentioned ICC, because it's one that I know today people are
>> using and it is capable of replacing gcc in many applications.
>
> If that's the case then it doesn't seem like it ought to be necessary to
> override most of the variables you mentioned.  Tools like ar, ranlib and
> strip are generic to a binary format and should work fine with the
> output from any compiler.  Also, if you're linking with GNU libc then
> the chances are that you need to use one of the GNU linkers (be that
> ld.bfd or gold) since I suspect most third party link editors will not
> work very well.  And, if you have assembly source on hand, it's probably
> going to be written in GAS dialect so selecting an alternative assembler
> as ${AS} is unlikely to gain much.

I realize not everything will be overridden, and I even expect in many cases 
only the CC and CXX will be defined.

I'm aware of at least one third party compiler that uses a different dialect 
then GAS.  And in that case the compiler and assembly would be overwritten.. 
(mind you anything that uses gas syntax would of course be incompatible and blow 
up, which begs the question -- do you really want to use AS then?)

But I figure if we cover all of the cases then we should be good no matter what.

> So, it kind of seems like the only variables you're likely to need to
> change are CC, CXX and the corresponding FLAGS.  I guess you might also
> want to override F77 in theory though I'm not sure there are many people
> using Fortran in an embedded environment.

I'm aware of one company that -was- using Fortran, I don't know if they still 
are.  (And they were not using the GNU F77....)

> Your proposed "toolchain-icc" override sounds like a reasonable enough
> way to accomplish that, anyway.  The only problem it doesn't seem to
> solve is the use-case of "everything that supports icc, I want to build
> with icc; everything that doesn't support icc, I want to build with
> gcc".  But I'm not sure this really matters much.

There needs to be a way to set the TOOLCHAIN 'type' on a blacklist or whitelist 
basis.  I.e. in a whitelist, you simply set it to 'gnu' (or whatever), which is 
more or less what I'd expect the default behavior to be.  Then for every recipe 
you are aware of that works -- you'd overriver the TOOLCHAIN-pn_package w/ 'alt'.

Alternatively you could do it as a blacklist.. set TOOLCHAIN = 'alt' and then 
blacklist items by setting the items to 'gnu' (or whatever).

Both have advantages.. personally I'd suggest the former rather then later 
approach, it's easier to support and then you can focus porting efforts on the 
packages that you believe you need the most optimizations in.

--Mark

> p.
>
>

RFC: Secondary Toolchain -- Followup

[Mark Hatle]

I have taken the RFC added it to the OE Wiki -- as well as added example 
details.  I've got most of the information implemented now, only the SDK 
specific items are remaining.

I'm looking for people to review this and let me know if the contents are 
reasonable or not -- and if you have any suggestions or comments.

http://www.openembedded.org/wiki/Adding_a_secondary_toolchain

Once the document is complete, and all of the comments are addressed, I expect 
this will become the 'standard' way to implement and control a secondary toolchain.

On 10/4/12 1:02 PM, Mark Hatle wrote:
> We have an issue where we'd like to have an alternative toolchain (assembler,
> linker, compiler) available for our customers to selectively use.  However,
> before we just go off and implement something, I'd like at least some basic
> consensus on what the best practice is for doing this work.  Below is my attempt
> at an early proposal.
>
> Background
> ----------------
>
> Many companies have commercial / highly optimized toolchains for specific
> purpose, such as ICC from Intel, LLVM, ARM specific toolchain, etc..  For
> (potentially) better performance with some applications a mechanism to both
> provide the hooks for the alternative toolchain as well as a way to active it
> per-package is desired.
>
> This work assumes that the third party toolchain is generally compatible with
> the idea of sysroots, linking to the libc provided by OE, and generally
> compatible with GNU conventions.
>
> However, as part of the third party toolchain, it may not be GNU compatible.
> This means many Open Source applications simply may not work with this
> toolchain.  That means that we need to have a way for a toolchain to blacklist
> (or whitelist) specific recipes.  This way only supported components can be
> built by the user, avoiding numerous complaints.
>
> Currently OE has a method to generate an SDK based on the GNU toolchain.  We
> would like to be able to also export the external toolchain along with the SDK,
> effectively providing both the GNU toolchain and the third party toolchain using
> the common sysroot.
>
> We need a way to active the third party toolchain on a per-package basis.  This
> activation will need to use the existing sysroot, but be able to pass different
> C, C++, LD, AS and other flags as specified by the third party toolchain.
>
> Finally third party toolchains should be implemented as layers that can easily
> plug into OE.
>
> Implementation
> ---------------------
>
> Add an OVERRIDE to specify the alternative toolchain.  Can this be done within
> the layer by doing something like:
>
> OVERRIDE_append = ":toolchain-${TOOLCHAIN}"
>
> TOOLCHAIN = "gnu" (or "icc")
>
> To activate the toolchain you would use things like:
>
> TOOLCHAIN_pn-bash = 'icc'
>
> To define the correct behavior for the toolchain, the following would need to be
> defined (with _toolchain-<toolchain>):
>
> TARGET_CPPFLAGS
> TARGET_CFLAGS
> TARGET_CXXFLAGS
> TARGET_LDFLAGS
> CC
> CXX
> F77
> CPP
> LD
> CCLD
> AR
> AS
> RANLIB
> STRIP
> OBJCOPY
> OBJDUMP
> NM
> FULL_OPTIMIZATIONS
> DEBUG_OPTIMIZATION
>
> Is anyone aware of any other items that may require additional items?  Will the
> above actually work?  Using the override of the TOOLCHAIN_… will that actually
> change the override values or do we get stuck?
>
> Comments/suggestions appreciated!
> --Mark
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>