RFC: "Virtual" native and sdk recipes

RFC: "Virtual" native and sdk recipes

[Richard Purdie]

Its nice to try something a bit different occasionally. For a long time
the mechanical repetitive nature of -native and -sdk recipes has
bothered me and people have talked about getting rid of them since
forever. I've had ideas floating around on how to address this for a
while and I now have a proper proposal and better still a proof of
concept.

Some -native recipes are already very simple, others are quite complex.
My hypothetical foo package files contain some common constructs and
look like:

foo/foo.inc:

SRC_URI = "http://somwehere/foo-${PV}.tar.bz2 \
           file://some.patch;patch=1"
S = "${WORKDIR}/foo-${PV}
inherit autotools

foo/foo_1.0.bb:

require foo.inc

foo/foo-native_1.0.bb:
require foo_${PV}.bb
FILESDIR = "${@os.path.dirname(bb.data.getVar('FILE',d,1))}/foo-${PV}"
inherit native

and foo/foo-1.0/some.patch exists.


Proposal Step A:

Having to set:

FILESDIR = "${@os.path.dirname(bb.data.getVar('FILE',d,1))}/foo-${PV}"

and

S = "${WORKDIR}/foo-${PV}

is fairly pointless and it would be nice the the defaults just did the
right thing. 

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=b8f0510a5567f6d1b90934fe513fd40423f11086

is a patch introducing BASEPN which is PN with a range of specified
suffixes removed (-native, -cross-sdk, -cross, -sdk as specified by
SPECIAL_PKGSUFFIX).

FILESPATH and S can then be constructed with BASEPN instead of PN
removing the need to need to hardcode them:

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=ed829661d51b8f23628e97404ed507c2e56d8495

This reduces my hypothetical example to:

foo/foo.inc:

SRC_URI = "http://somwehere/${BASEPN}-${PV}.tar.bz2 \
           file://some.patch;patch=1"
inherit autotools

foo/foo-native_1.0.bb:
require foo_${PV}.bb
inherit native

The above proposal is worthwhile in itself alone.


Proposal Step B:

To add a new variable which is a list of classes to additionally parse
after parsing the 'base' recipe. In the above example, foo-native-1.0.bb
would be removed and we'd instead have:

foo/foo.inc:

SRC_URI = "http://somwehere/${BASEPN}-${PV}.tar.bz2 \
           file://some.patch;patch=1"
inherit autotools
BBCLASSEXTEND = "native"

Internally bitbake would see BBCLASSEXTEND and do something like:

"""
cls = value in BBCLASSEXTEND
data.setVar('PN', pn + '-' + cls, based)
inherit cls
"""

The big change here is that one .bb file can now provide multiple
targets. This causes a few issues but as a hacked up proof of concept:

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=f614d1e2ff10938167ba6c29c49677a03b4a19ef

where we append magic values to the "filename" and inherit the
appropriate class when we notice the special names.


The advantages of doing this are:

* We can convert to the system incrementally
* Simple syntax, the complexity is hidden in bitbake
* Removes needless duplication but allows custom -native and -sdk 
  packages to be used where automation isn't possible/desirable
* Parsing speed should increase since the number of full recipes we 
  parse drops and only the end stage of parsing is done for the 
  BBCLASSEXTEND packages.
* Even just step A simplifies recipes.

The disadvantages:

* More complexity in bitbake (but localised mainly to cache.py)
* The .bb file is no longer one recipe but can run in different 
  flavours which will be confusing to people 
* Would require a new bitbake release and adoption of that release 
  before OE.dev can use step B functionality. Step A functionality 
  could be added without a new bitbake by putting the special function 
  in base.bbclass instead of bitbake.
* Certain recipe name extensions become "special". The values are not 
  hardcoded in bitbake though and its already a common convention so 
  I'm not sure this is a problem.


I've shared my changes in a branch of Poky:

http://git.pokylinux.org/cgit.cgi/poky/log/?h=experimental-virtualnative

Obviously there are some heavy cleanups needed to the last bitbake
changes but it works as a proof of concept so its time to discuss it.

The main problem I currently have before being able to deploy this more
widely is auto adjustment of DEPENDS. In most cases this consists of
adding "-native" or "-sdk" to the values already in DEPENDS so we could
probably add a function to native.bbclass for that. Alternatively or
additionally, I'm considering having native.bbclass append to OVERRIDES
so we can override DEPENDS that way instead.

Comments welcome.

Cheers,

Richard

-- 
Richard Purdie
Intel Open Source Technology Centre

Re: RFC: "Virtual" native and sdk recipes

[Koen Kooi]

On 01-01-09 17:25, Richard Purdie wrote:
> Its nice to try something a bit different occasionally. For a long time
> the mechanical repetitive nature of -native and -sdk recipes has
> bothered me and people have talked about getting rid of them since
> forever. I've had ideas floating around on how to address this for a
> while and I now have a proper proposal and better still a proof of
> concept.
>
> Some -native recipes are already very simple, others are quite complex.
> My hypothetical foo package files contain some common constructs and
> look like:
>
> foo/foo.inc:
>
> SRC_URI = "http://somwehere/foo-${PV}.tar.bz2 \
>             file://some.patch;patch=1"
> S = "${WORKDIR}/foo-${PV}
> inherit autotools
>
> foo/foo_1.0.bb:
>
> require foo.inc
>
> foo/foo-native_1.0.bb:
> require foo_${PV}.bb
> FILESDIR = "${@os.path.dirname(bb.data.getVar('FILE',d,1))}/foo-${PV}"
> inherit native
>
> and foo/foo-1.0/some.patch exists.
>
>
> Proposal Step A:
>
> Having to set:
>
> FILESDIR = "${@os.path.dirname(bb.data.getVar('FILE',d,1))}/foo-${PV}"
>
> and
>
> S = "${WORKDIR}/foo-${PV}
>
> is fairly pointless and it would be nice the the defaults just did the
> right thing.
>
> http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=b8f0510a5567f6d1b90934fe513fd40423f11086
>
> is a patch introducing BASEPN which is PN with a range of specified
> suffixes removed (-native, -cross-sdk, -cross, -sdk as specified by
> SPECIAL_PKGSUFFIX).


How is the above different from the BPN used in OE?

# Base package name
# Automatically derives "foo" from "foo-native", "foo-cross" or 
"foo-initial"
# otherwise it is the same as PN and P
BPN = "${@base_package_name(d)}"
BP = "${BPN}-${PV}"

[..]

S = "${WORKDIR}/${BP}"
B = "${S}"

regards,

Koen




>
> FILESPATH and S can then be constructed with BASEPN instead of PN
> removing the need to need to hardcode them:
>
> http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=ed829661d51b8f23628e97404ed507c2e56d8495
>
> This reduces my hypothetical example to:
>
> foo/foo.inc:
>
> SRC_URI = "http://somwehere/${BASEPN}-${PV}.tar.bz2 \
>             file://some.patch;patch=1"
> inherit autotools
>
> foo/foo-native_1.0.bb:
> require foo_${PV}.bb
> inherit native
>
> The above proposal is worthwhile in itself alone.
>
>
> Proposal Step B:
>
> To add a new variable which is a list of classes to additionally parse
> after parsing the 'base' recipe. In the above example, foo-native-1.0.bb
> would be removed and we'd instead have:
>
> foo/foo.inc:
>
> SRC_URI = "http://somwehere/${BASEPN}-${PV}.tar.bz2 \
>             file://some.patch;patch=1"
> inherit autotools
> BBCLASSEXTEND = "native"
>
> Internally bitbake would see BBCLASSEXTEND and do something like:
>
> """
> cls = value in BBCLASSEXTEND
> data.setVar('PN', pn + '-' + cls, based)
> inherit cls
> """
>
> The big change here is that one .bb file can now provide multiple
> targets. This causes a few issues but as a hacked up proof of concept:
>
> http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=f614d1e2ff10938167ba6c29c49677a03b4a19ef
>
> where we append magic values to the "filename" and inherit the
> appropriate class when we notice the special names.
>
>
> The advantages of doing this are:
>
> * We can convert to the system incrementally
> * Simple syntax, the complexity is hidden in bitbake
> * Removes needless duplication but allows custom -native and -sdk
>    packages to be used where automation isn't possible/desirable
> * Parsing speed should increase since the number of full recipes we
>    parse drops and only the end stage of parsing is done for the
>    BBCLASSEXTEND packages.
> * Even just step A simplifies recipes.
>
> The disadvantages:
>
> * More complexity in bitbake (but localised mainly to cache.py)
> * The .bb file is no longer one recipe but can run in different
>    flavours which will be confusing to people
> * Would require a new bitbake release and adoption of that release
>    before OE.dev can use step B functionality. Step A functionality
>    could be added without a new bitbake by putting the special function
>    in base.bbclass instead of bitbake.
> * Certain recipe name extensions become "special". The values are not
>    hardcoded in bitbake though and its already a common convention so
>    I'm not sure this is a problem.
>
>
> I've shared my changes in a branch of Poky:
>
> http://git.pokylinux.org/cgit.cgi/poky/log/?h=experimental-virtualnative
>
> Obviously there are some heavy cleanups needed to the last bitbake
> changes but it works as a proof of concept so its time to discuss it.
>
> The main problem I currently have before being able to deploy this more
> widely is auto adjustment of DEPENDS. In most cases this consists of
> adding "-native" or "-sdk" to the values already in DEPENDS so we could
> probably add a function to native.bbclass for that. Alternatively or
> additionally, I'm considering having native.bbclass append to OVERRIDES
> so we can override DEPENDS that way instead.
>
> Comments welcome.
>
> Cheers,
>
> Richard
>

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Thu, Jan 01, 2009 at 04:25:14PM +0000, Richard Purdie wrote:

> Its nice to try something a bit different occasionally. For a long time
> the mechanical repetitive nature of -native and -sdk recipes has
> bothered me and people have talked about getting rid of them since
> forever. I've had ideas floating around on how to address this for a
> while and I now have a proper proposal and better still a proof of
> concept.

I'll read the whole thing again later, but in working on the canadian
SDK stuff (*poke*) I've been thinking why don't we just build the "SDK"
packages for building our own stuff?  gcc/etc are relocatible if we
build them right...

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Richard Purdie]

On Thu, 2009-01-01 at 19:11 +0100, Koen Kooi wrote:
> How is the above different from the BPN used in OE?
> 
> # Base package name
> # Automatically derives "foo" from "foo-native", "foo-cross" or 
> "foo-initial"
> # otherwise it is the same as PN and P
> BPN = "${@base_package_name(d)}"
> BP = "${BPN}-${PV}"
> 
> [..]
> 
> S = "${WORKDIR}/${BP}"
> B = "${S}"

Heh, I could have sworn I'd seen that done before, OE is obviously
where. 

The rest still stands though and is the interesting bit :)

Cheers,

Richard

Re: RFC: "Virtual" native and sdk recipes

[Richard Purdie]

On Thu, 2009-01-01 at 11:25 -0700, Tom Rini wrote:
> On Thu, Jan 01, 2009 at 04:25:14PM +0000, Richard Purdie wrote:
> 
> > Its nice to try something a bit different occasionally. For a long time
> > the mechanical repetitive nature of -native and -sdk recipes has
> > bothered me and people have talked about getting rid of them since
> > forever. I've had ideas floating around on how to address this for a
> > while and I now have a proper proposal and better still a proof of
> > concept.
> 
> I'll read the whole thing again later, but in working on the canadian
> SDK stuff (*poke*) I've been thinking why don't we just build the "SDK"
> packages for building our own stuff?  gcc/etc are relocatible if we
> build them right...

Even if gcc is relocatable, some software (quilt springs to mind) has to
be built using the paths it will be installed to. I'll happily support
any effort to make as much as possible support relocation but at the
very least we'll continue having -native packages and its them this is
primarily aimed at. I wouldn't want to miss the opportunity to kill off
many of the -sdk packages in poky at the same time though!

I've been wondering how much the canadian SDK code will benefit from
this and perhaps you can give some input on that?

Cheers,

Richard

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Thu, Jan 01, 2009 at 08:11:27PM +0000, Richard Purdie wrote:
> 
> On Thu, 2009-01-01 at 11:25 -0700, Tom Rini wrote:
> > On Thu, Jan 01, 2009 at 04:25:14PM +0000, Richard Purdie wrote:
> > 
> > > Its nice to try something a bit different occasionally. For a long time
> > > the mechanical repetitive nature of -native and -sdk recipes has
> > > bothered me and people have talked about getting rid of them since
> > > forever. I've had ideas floating around on how to address this for a
> > > while and I now have a proper proposal and better still a proof of
> > > concept.
> > 
> > I'll read the whole thing again later, but in working on the canadian
> > SDK stuff (*poke*) I've been thinking why don't we just build the "SDK"
> > packages for building our own stuff?  gcc/etc are relocatible if we
> > build them right...
> 
> Even if gcc is relocatable, some software (quilt springs to mind) has to
> be built using the paths it will be installed to. I'll happily support
> any effort to make as much as possible support relocation but at the
> very least we'll continue having -native packages and its them this is
> primarily aimed at. I wouldn't want to miss the opportunity to kill off
> many of the -sdk packages in poky at the same time though!

I know for quilt specifically anything you can compile in as a path you
can override in quiltrc, and you can pass-in --quiltrc, so... next? :)
But yes, I too wouldn't mind having a few less -sdk packages in my own
stuff too.

> I've been wondering how much the canadian SDK code will benefit from
> this and perhaps you can give some input on that?

Off the top of my head, it depends on if we want to allow one env to
make both a Linux and a Windows SDK.  The foo-canadian-sdk packages can
fit very easily into any reuse scheme (the gcc ones look like all of the
others now).  I was thinking there's no reason really (esp if we build
gcc correctly for relcation, which we don't now) that any 'target gcc
for sdk' recipe couldn't build for any one Linux or Windows target, but
not both.  For my needs, I want both Linux and Windows SDKs kicked out
from one bitbake invokation.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Thu, Jan 01, 2009 at 04:25:14PM +0000, Richard Purdie wrote:

[snip]
> Proposal Step B:
> 
> To add a new variable which is a list of classes to additionally parse
> after parsing the 'base' recipe. In the above example, foo-native-1.0.bb
> would be removed and we'd instead have:

I think the canadian SDK stuff should fit into this scheme as well.
There's a few very trivial packages (mpfr, gmp) and most of the rest of
the magic ends up being in canadian-sdk.bbclass.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Richard Purdie]

On Thu, 2009-01-01 at 16:25 +0000, Richard Purdie wrote:
> The main problem I currently have before being able to deploy this more
> widely is auto adjustment of DEPENDS. In most cases this consists of
> adding "-native" or "-sdk" to the values already in DEPENDS so we could
> probably add a function to native.bbclass for that. Alternatively or
> additionally, I'm considering having native.bbclass append to OVERRIDES
> so we can override DEPENDS that way instead.

Just to follow up, I added some magic to native.bbclass and sdk.bbclass
which meant things would work for a larger test sample in Poky. This
commit has the details:

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=8bc63a922d237b0c2797e231d3b5894bf88e0530

Cheers,

Richard

Re: RFC: "Virtual" native and sdk recipes

[Richard Purdie]

On Thu, 2009-01-01 at 15:02 -0700, Tom Rini wrote: 
> On Thu, Jan 01, 2009 at 08:11:27PM +0000, Richard Purdie wrote:
> I know for quilt specifically anything you can compile in as a path you
> can override in quiltrc, and you can pass-in --quiltrc, so... next? :)

I'll have to remember that :)

> But yes, I too wouldn't mind having a few less -sdk packages in my own
> stuff too.
> 
> > I've been wondering how much the canadian SDK code will benefit from
> > this and perhaps you can give some input on that?
> 
> Off the top of my head, it depends on if we want to allow one env to
> make both a Linux and a Windows SDK.  The foo-canadian-sdk packages can
> fit very easily into any reuse scheme (the gcc ones look like all of the
> others now).  I was thinking there's no reason really (esp if we build
> gcc correctly for relcation, which we don't now) that any 'target gcc
> for sdk' recipe couldn't build for any one Linux or Windows target, but
> not both.  For my needs, I want both Linux and Windows SDKs kicked out
> from one bitbake invokation.

Ideally I want to be able to generate a Linux 32 bit toolchain, a Linux
64 bit toolchain and a windows one. If I can do that using canadian-sdk,
the current -sdk stuff can be removed.

I'm not sure the one bitbake invocation is going to work though or is
desirable.

I just had a look through the trini/canadian-sdk branch and there are
bits I like and bits I dislike. I'll try and provide some feedback in
due course with a view to getting the less controversial bits merged. I
have some tweaks in poky to do with dynamic library extension handling
for example (from playing with darwin targets) where it would pay us to
find a common solution.

More fundamentally I'm actually very nervous about something here which
is very difficult to put into words but I'll try. To do this, I'd like
to take a step back and think about how this should work. Our "normal"
OE usage has HOST == TARGET where the value is determined by MACHINE.
With the introduction of Canadian where BUILD != HOST != TARGET we
obviously need one more input. 

Correct me if I'm wrong but your approach seems to assume MACHINE is the
TARGET value? If so we may be missing an opportunity.

Lets for a second make the decision that MACHINE is actually the HOST
value. This makes sense in that everything we usually build is to run on
the HOST. From here we only need to add gcc-canadian recipe which builds
a compiler that runs on HOST but targets a different TARGET to normal to
have our BUILD != HOST != TARGET.

This does reverse the logic that the current sdk class we use however
since MACHINE is now the machine we want to run the compiler on, not the
machine we want to compile for. It should be simple enough to add some
MACHINEs and version setups which correspond to a Linux 32 bit system, a
Linux 64 bit system and a windows system though. Assuming the choice of
TARGET for gcc-canadian is controlled by a variable like SDKTARGET, to
run the builds I'd want, I'd run:

MACHINE=i686-generic   SDKTARGET=armv5te-generic bitbake gcc-canadian
MACHINE=x86-64-generic SDKTARGET=armv5te-generic bitbake gcc-canadian
MACHINE=winxp-generic  SDKTARGET=armv5te-generic bitbake gcc-canadian


By contrast, the scheme you are aiming for (as I understand it) would
be:

MACHINE=armv5te-generic SDKARCH=i686-generic   bitbake gcc-canadian
MACHINE=armv5te-generic SDKARCH=x86-64-generic bitbake gcc-canadian
MACHINE=armv5te-generic SDKARCH=winxp-generic  bitbake gcc-canadian

Can you see the point I'm making here? If not I'll try again tomorrow
after some sleep! :)

Also note why I don't see this being possible to combine into one
bitbake command, much the same way you can't build 10 machines with one
bitbake command.

Cheers,

Richard

-- 
Richard Purdie
Intel Open Source Technology Centre

Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Tom Rini]

On Fri, Jan 02, 2009 at 01:11:53AM +0000, Richard Purdie wrote:

[snip]
> Ideally I want to be able to generate a Linux 32 bit toolchain, a Linux
> 64 bit toolchain and a windows one. If I can do that using canadian-sdk,
> the current -sdk stuff can be removed.

It's a theory, and not one I've tested (and I know it won't work _today_
as there's some hard-coded C:\ paths right now.  Those can die if we
build it properly relocatible like CodeSourcery does).

> I'm not sure the one bitbake invocation is going to work though or is
> desirable.

Well, here's my usage.  Right now, we produce an SDK for a specific
target.  It's quite handy that I can do a 'bitbake sdk-release' and
that builds me my Linux SDK in one file, my Windows SDK in another.  I
could live if I had to do 'bitbake linux-sdk-release && bitbake
windows-sdk-release' if I must.  But I'm lazy :)

> I just had a look through the trini/canadian-sdk branch and there are
> bits I like and bits I dislike. I'll try and provide some feedback in
> due course with a view to getting the less controversial bits merged. I

Great, thanks.

> have some tweaks in poky to do with dynamic library extension handling
> for example (from playing with darwin targets) where it would pay us to
> find a common solution.

Sounds good.

> More fundamentally I'm actually very nervous about something here which
> is very difficult to put into words but I'll try. To do this, I'd like
> to take a step back and think about how this should work. Our "normal"
> OE usage has HOST == TARGET where the value is determined by MACHINE.

I think saying the current usage is "BUILD == HOST" would be more
correct.  But I see what you're saying.  We set a machine (which sets
our output target) and assume everything we're building will run on the
machine that's doing the building (and only that machine, which is why
I've switched to doing all my builds inside a minimal Ubuntu chroot.
Hit a few dep issues I need to try and sort out properly sometime w/
ext2 stuff).

> With the introduction of Canadian where BUILD != HOST != TARGET we
> obviously need one more input. 

Yes.  We stop assuming BUILD == HOST, as it may or may not.

> Correct me if I'm wrong but your approach seems to assume MACHINE is the
> TARGET value? If so we may be missing an opportunity.

Yeah.  But it seems to me that's what we always do.  It's currently
always a "build to run on this machine" or "build to run on the target
machine".  But I think I see what you're saying and agree.

> Lets for a second make the decision that MACHINE is actually the HOST
> value. This makes sense in that everything we usually build is to run on
> the HOST. From here we only need to add gcc-canadian recipe which builds
> a compiler that runs on HOST but targets a different TARGET to normal to
> have our BUILD != HOST != TARGET.

Right.  We  get that in the current Canadian stuff, just using different
variable names, I think.  I _think_ if the new SDK_foo variables were
set right, it would in fact kick out a working Linux SDK.

> This does reverse the logic that the current sdk class we use however
> since MACHINE is now the machine we want to run the compiler on, not the
> machine we want to compile for. It should be simple enough to add some
> MACHINEs and version setups which correspond to a Linux 32 bit system, a
> Linux 64 bit system and a windows system though. Assuming the choice of
> TARGET for gcc-canadian is controlled by a variable like SDKTARGET, to
> run the builds I'd want, I'd run:
> 
> MACHINE=i686-generic   SDKTARGET=armv5te-generic bitbake gcc-canadian
> MACHINE=x86-64-generic SDKTARGET=armv5te-generic bitbake gcc-canadian
> MACHINE=winxp-generic  SDKTARGET=armv5te-generic bitbake gcc-canadian

Or 'meta-toolchain-sbox' for an existing SDK type target.

> By contrast, the scheme you are aiming for (as I understand it) would
> be:
> 
> MACHINE=armv5te-generic SDKARCH=i686-generic   bitbake gcc-canadian
> MACHINE=armv5te-generic SDKARCH=x86-64-generic bitbake gcc-canadian
> MACHINE=armv5te-generic SDKARCH=winxp-generic  bitbake gcc-canadian

With a "but I'm lazy and we'll just re-use current SDK recipes for
the first choice, canadian for the second and, what? a third choice?
la-la-la I can't hear you!".  The biggest hiccup I see here is my usage
would be more like (using current machine.conf's):
MACHINE=qemux86 SDKTARGET=db1200 bitbake meta-toolchain-sbox

> Can you see the point I'm making here? If not I'll try again tomorrow
> after some sleep! :)

I think so.  Right now what's bugging me personally the most is the
non-relocatible SDK toolchain, which once fixed could let us kill off
much of the -cross stuff.  Build the SDK version and then use that to
build our regular stuff, always.

> Also note why I don't see this being possible to combine into one
> bitbake command, much the same way you can't build 10 machines with one
> bitbake command.

Trying to get more than one machine into an SDK is something else I've
pondered, but only lightly so far.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Koen Kooi]

On 01-01-09 17:25, Richard Purdie wrote:

> Proposal Step B:
>
> To add a new variable which is a list of classes to additionally parse
> after parsing the 'base' recipe. In the above example, foo-native-1.0.bb
> would be removed and we'd instead have:
>
> foo/foo.inc:
>
> SRC_URI = "http://somwehere/${BASEPN}-${PV}.tar.bz2 \
>             file://some.patch;patch=1"
> inherit autotools
> BBCLASSEXTEND = "native"
>
> Internally bitbake would see BBCLASSEXTEND and do something like:
>
> """
> cls = value in BBCLASSEXTEND
> data.setVar('PN', pn + '-' + cls, based)
> inherit cls
> """
>

[..]

> The disadvantages:
>
> * More complexity in bitbake (but localised mainly to cache.py)
> * The .bb file is no longer one recipe but can run in different
>    flavours which will be confusing to people
> * Would require a new bitbake release and adoption of that release
>    before OE.dev can use step B functionality. Step A functionality
>    could be added without a new bitbake by putting the special function
>    in base.bbclass instead of bitbake.

If you put that code into bitbake trunk *now* and do a new release in a 
week or so people can start testing your work. OE .dev can require that 
bitbake release when the branch with BBCLASSEXTEND lands.
The important part is that bitbake has BBCLASSEXTEND in a release real 
soon :)

regards,

Koen

Re: RFC: "Virtual" native and sdk recipes

[Richard Purdie]

To further update, I merged what was in OE and my Step A into poky. I
also merged some code rearranging into bitbake. This leaves the
following two commits:

This is the patch to add support to bitbake:

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=7e4d5d677f996a9ca621f3ab6aa22cc7c3ef80d8

(3 files changed, 64 insertions, 12 deletions)

and this then changes some recipes in Poky to use BBCLASSEXTEND:

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=7f3769423fa81163be97c744e3ee274a8cf45aa1

Cheers,

Richard

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Esben Haabendal]

On Fri, Jan 2, 2009 at 5:37 AM, Tom Rini <trini@kernel.crashing.org> wrote:
> On Fri, Jan 02, 2009 at 01:11:53AM +0000, Richard Purdie wrote:
>
> [snip]
>> Ideally I want to be able to generate a Linux 32 bit toolchain, a Linux
>> 64 bit toolchain and a windows one. If I can do that using canadian-sdk,
>> the current -sdk stuff can be removed.
>
> It's a theory, and not one I've tested (and I know it won't work _today_
> as there's some hard-coded C:\ paths right now.  Those can die if we
> build it properly relocatible like CodeSourcery does).

Sorry, but where would that be?

In class/canadian-sdk.bbclass:

SDK_PATH_sdk-mingw32 = "/OpenEmbedded/${SDK_NAME}"
SDK_REALPATH = "${SDK_PATH}"
SDK_REALPATH_sdk-mingw32 = "C:/OpenEmbedded/${SDK_NAME}"

Which should only hard-code C:/ into mingw32 toolchains.  I don't
remember seing C:\
in any other places, so that shouldn't get in the way of building
Linux toolchains
with canadian-sdk.

>> Lets for a second make the decision that MACHINE is actually the HOST
>> value. This makes sense in that everything we usually build is to run on
>> the HOST. From here we only need to add gcc-canadian recipe which builds
>> a compiler that runs on HOST but targets a different TARGET to normal to
>> have our BUILD != HOST != TARGET.
>
> Right.  We  get that in the current Canadian stuff, just using different
> variable names, I think.  I _think_ if the new SDK_foo variables were
> set right, it would in fact kick out a working Linux SDK.

I agree on that, although this must come with a YMMV for now ;-)

It would be really great if someone give this a spin, and perhaps a few small
fixes.  I really don't think there should be any big showstoppers ahead.

>> This does reverse the logic that the current sdk class we use however
>> since MACHINE is now the machine we want to run the compiler on, not the
>> machine we want to compile for. It should be simple enough to add some
>> MACHINEs and version setups which correspond to a Linux 32 bit system, a
>> Linux 64 bit system and a windows system though. Assuming the choice of
>> TARGET for gcc-canadian is controlled by a variable like SDKTARGET, to
>> run the builds I'd want, I'd run:
>>
>> MACHINE=i686-generic   SDKTARGET=armv5te-generic bitbake gcc-canadian
>> MACHINE=x86-64-generic SDKTARGET=armv5te-generic bitbake gcc-canadian
>> MACHINE=winxp-generic  SDKTARGET=armv5te-generic bitbake gcc-canadian
>
> Or 'meta-toolchain-sbox' for an existing SDK type target.

I'm sorry, but this seems like a dangerous way of starting confusion of terms.
MACHINE in how I see OE is really the _TARGET_, ie. the small device this all
is targeted at.

And AFAICS, the end result is at best the same as the current canadian-sdk
approach.

>> By contrast, the scheme you are aiming for (as I understand it) would
>> be:
>>
>> MACHINE=armv5te-generic SDKARCH=i686-generic   bitbake gcc-canadian
>> MACHINE=armv5te-generic SDKARCH=x86-64-generic bitbake gcc-canadian
>> MACHINE=armv5te-generic SDKARCH=winxp-generic  bitbake gcc-canadian
>
> With a "but I'm lazy and we'll just re-use current SDK recipes for
> the first choice, canadian for the second and, what? a third choice?
> la-la-la I can't hear you!".  The biggest hiccup I see here is my usage
> would be more like (using current machine.conf's):
> MACHINE=qemux86 SDKTARGET=db1200 bitbake meta-toolchain-sbox

It would be rather nice if local.conf could be setup with the set of SDKARCH's
that is supported by the particular project, so that bitbake canadian-sdk would
actually build sdk's for all required archs.  And a default of
building i686-linux,
x86_64-linux and i686-mingw32 might be a good default setting for this.

>> Also note why I don't see this being possible to combine into one
>> bitbake command, much the same way you can't build 10 machines with one
>> bitbake command.
>
> Trying to get more than one machine into an SDK is something else I've
> pondered, but only lightly so far.

The same thought have hit me as well.
But let's get one thing done right first :-)

Esben Haabendal

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Tom Rini]

On Mon, Jan 05, 2009 at 03:31:53PM +0100, Esben Haabendal wrote:
> On Fri, Jan 2, 2009 at 5:37 AM, Tom Rini <trini@kernel.crashing.org> wrote:
> > On Fri, Jan 02, 2009 at 01:11:53AM +0000, Richard Purdie wrote:
> >
> > [snip]
> >> Ideally I want to be able to generate a Linux 32 bit toolchain, a Linux
> >> 64 bit toolchain and a windows one. If I can do that using canadian-sdk,
> >> the current -sdk stuff can be removed.
> >
> > It's a theory, and not one I've tested (and I know it won't work _today_
> > as there's some hard-coded C:\ paths right now.  Those can die if we
> > build it properly relocatible like CodeSourcery does).
> 
> Sorry, but where would that be?
> 
> In class/canadian-sdk.bbclass:
> 
> SDK_PATH_sdk-mingw32 = "/OpenEmbedded/${SDK_NAME}"
> SDK_REALPATH = "${SDK_PATH}"
> SDK_REALPATH_sdk-mingw32 = "C:/OpenEmbedded/${SDK_NAME}"
> 
> Which should only hard-code C:/ into mingw32 toolchains.  I don't
> remember seing C:\

Just mis-remembered.  But the big point is we should be able to compile
everyone for /OpenEmbedded/${SDK_NAME} and it should just work.

-- 
Tom Rini

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Koen Kooi]

On 05-01-09 15:31, Esben Haabendal wrote:
> On Fri, Jan 2, 2009 at 5:37 AM, Tom Rini<trini@kernel.crashing.org>  wrote:
>> On Fri, Jan 02, 2009 at 01:11:53AM +0000, Richard Purdie wrote:

>>> This does reverse the logic that the current sdk class we use however
>>> since MACHINE is now the machine we want to run the compiler on, not the
>>> machine we want to compile for. It should be simple enough to add some
>>> MACHINEs and version setups which correspond to a Linux 32 bit system, a
>>> Linux 64 bit system and a windows system though. Assuming the choice of
>>> TARGET for gcc-canadian is controlled by a variable like SDKTARGET, to
>>> run the builds I'd want, I'd run:
>>>
>>> MACHINE=i686-generic   SDKTARGET=armv5te-generic bitbake gcc-canadian
>>> MACHINE=x86-64-generic SDKTARGET=armv5te-generic bitbake gcc-canadian
>>> MACHINE=winxp-generic  SDKTARGET=armv5te-generic bitbake gcc-canadian
>> Or 'meta-toolchain-sbox' for an existing SDK type target.
>
> I'm sorry, but this seems like a dangerous way of starting confusion of terms.
> MACHINE in how I see OE is really the _TARGET_, ie. the small device this all
> is targeted at.

MACHINE is where the generated stuff will _run_ on, so MACHINE=x86 
SDKTARGET=armv5te would be more in line with what OE expects, but I 
agree it can be confusing if you are thinking in autotools terms.

regards,

Koen

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Tom Rini]

On Mon, Jan 05, 2009 at 06:29:24PM +0100, Koen Kooi wrote:
> On 05-01-09 15:31, Esben Haabendal wrote:
>> On Fri, Jan 2, 2009 at 5:37 AM, Tom Rini<trini@kernel.crashing.org>  wrote:
>>> On Fri, Jan 02, 2009 at 01:11:53AM +0000, Richard Purdie wrote:
>
>>>> This does reverse the logic that the current sdk class we use however
>>>> since MACHINE is now the machine we want to run the compiler on, not the
>>>> machine we want to compile for. It should be simple enough to add some
>>>> MACHINEs and version setups which correspond to a Linux 32 bit system, a
>>>> Linux 64 bit system and a windows system though. Assuming the choice of
>>>> TARGET for gcc-canadian is controlled by a variable like SDKTARGET, to
>>>> run the builds I'd want, I'd run:
>>>>
>>>> MACHINE=i686-generic   SDKTARGET=armv5te-generic bitbake gcc-canadian
>>>> MACHINE=x86-64-generic SDKTARGET=armv5te-generic bitbake gcc-canadian
>>>> MACHINE=winxp-generic  SDKTARGET=armv5te-generic bitbake gcc-canadian
>>> Or 'meta-toolchain-sbox' for an existing SDK type target.
>>
>> I'm sorry, but this seems like a dangerous way of starting confusion of terms.
>> MACHINE in how I see OE is really the _TARGET_, ie. the small device this all
>> is targeted at.
>
> MACHINE is where the generated stuff will _run_ on, so MACHINE=x86  
> SDKTARGET=armv5te would be more in line with what OE expects, but I  
> agree it can be confusing if you are thinking in autotools terms.

The real trick is the SDK is for a specific target machine.  Should
still be doable, just need to get some magic to deal with if SDKTARGET
being set, re-set MACHINE, except for some cases

-- 
Tom Rini

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Esben Haabendal]

On Mon, Jan 5, 2009 at 6:29 PM, Koen Kooi <k.kooi@student.utwente.nl> wrote:

>> I'm sorry, but this seems like a dangerous way of starting confusion of terms.
>> MACHINE in how I see OE is really the _TARGET_, ie. the small device this all
>> is targeted at.
>
> MACHINE is where the generated stuff will _run_ on, so MACHINE=x86
> SDKTARGET=armv5te would be more in line with what OE expects, but I agree it
> can be confusing if you are thinking in autotools terms.

Exactly. Let's stick with the solution that applys best to the golden
"Rule of Least Surprise",
which I believe in the case of the average OE user would be as Koen suggests.

/Esben

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Richard Purdie]

On Tue, 2009-01-06 at 21:51 +0100, Esben Haabendal wrote:
> On Mon, Jan 5, 2009 at 6:29 PM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> 
> >> I'm sorry, but this seems like a dangerous way of starting confusion of terms.
> >> MACHINE in how I see OE is really the _TARGET_, ie. the small device this all
> >> is targeted at.
> >
> > MACHINE is where the generated stuff will _run_ on, so MACHINE=x86
> > SDKTARGET=armv5te would be more in line with what OE expects, but I agree it
> > can be confusing if you are thinking in autotools terms.
> 
> Exactly. Let's stick with the solution that applys best to the golden
> "Rule of Least Surprise",
> which I believe in the case of the average OE user would be as Koen suggests.

Well, someone has to take a step back and check whether what we're doing
is sane, extensible and in keeping with OE's global architecture. Sadly
I'm concluding that the sdk class does things incorrectly, even
dangerously and that the canadian stuff just compounds the error. The
canadian stuff is actually over complicated due to this.

The reason OE is successful, keeps growing and is so powerful/extensible
is down to the core architecture being clean and kept to strong
principles.

Also, for the record I've played a large part in developing the sdk and
meta-toolchain stuff so I'm at least partially responsible for the
current mess. I do now think I got that wrong though.

Having said this, the canadian stuff is valuable and I don't want that
work to get lost or its developers to get demotivated. It does fill a
current need too.

So I propose we allow the canadian stuff to merge on the understanding
that its doing things the wrong way and that we will most likely change
to the "right" way in due course even if its less intuitive. That change
may well break things but we're saying now this will happen and that
people shouldn't be surprised.

Is that reasonable?

As for the mechanics of merging the branch, I'd like this to go in
stages and will follow up with further email/discussion on that subject.
I'm under time pressures at the moment so this won't be until later this
week though. Sorry for the delay :(.

Also, people keep talking about building sdks for more than one machine
with one bitbake command and other similar things. This is a really
tricky idea and needs a lot of careful thought so as not to pull down
the whole architecture. It is not a trivial issue.

Cheers,

Richard

-- 
Richard Purdie
Intel Open Source Technology Centre

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Tom Rini]

On Wed, Jan 07, 2009 at 12:14:26AM +0000, Richard Purdie wrote:
[snip]
> So I propose we allow the canadian stuff to merge on the understanding
> that its doing things the wrong way and that we will most likely change
> to the "right" way in due course even if its less intuitive. That change
> may well break things but we're saying now this will happen and that
> people shouldn't be surprised.
> 
> Is that reasonable?
> 
> As for the mechanics of merging the branch, I'd like this to go in
> stages and will follow up with further email/discussion on that subject.
> I'm under time pressures at the moment so this won't be until later this
> week though. Sorry for the delay :(.

Agreed and I didn't intend the current branch for merging, just review.

> Also, people keep talking about building sdks for more than one machine
> with one bitbake command and other similar things. This is a really
> tricky idea and needs a lot of careful thought so as not to pull down
> the whole architecture. It is not a trivial issue.

Yes, we need to have some sort of discussion about what's reasonable and
what's a best design for making SDKs come out of OE.

-- 
Tom Rini

Re: Native/Cross/SDK rethink (Was: Re: RFC: "Virtual" native and sdk recipes)

[Leon Woestenberg]

Richard,

On Wed, Jan 7, 2009 at 1:14 AM, Richard Purdie <rpurdie@rpsys.net> wrote:
> On Tue, 2009-01-06 at 21:51 +0100, Esben Haabendal wrote:
>> On Mon, Jan 5, 2009 at 6:29 PM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
>>
> Also, for the record I've played a large part in developing the sdk and
> meta-toolchain stuff so I'm at least partially responsible for the
> current mess. I do now think I got that wrong though.
>
The words of a true perfectionist, always learning from daring steps
in the past :-)

The core toolchain and SDK stuff bits you have contributed are very
much appreciated, so you are kindly invited to "mess it up" again.

Regards,
-- 
Leon

Re: RFC: "Virtual" native and sdk recipes

[Richard Purdie]

On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> I just had a look through the trini/canadian-sdk branch and there are
> bits I like and bits I dislike. I'll try and provide some feedback in
> due course with a view to getting the less controversial bits merged. I
> have some tweaks in poky to do with dynamic library extension handling
> for example (from playing with darwin targets) where it would pay us to
> find a common solution.

Sorry for the delayed feedback, what I've looked at so far follows
below. It was easiest to extract some patches from your tree and make
some commits of my own so these are in:

http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar

Basically I wanted to particularly review the changes to the existing OE
core classes/bitbake.conf. The changes in that branch are versions I'm
happy with. I had two concerns there:

1. I don't want to see ${HOST_EXEEXT} everywhere but I know why you 
   need it and this was something OE would inevitably face. As a 
   compromise I propose adding:

   EXEEXT = "${HOST_EXEEXT}"

   and using ${EXEEXT} which is fractionally less ugly for all common 
   uses. Could you update your patch to use ${EXEEXT} please assuming 
   we all agree on this. Its the same dilemma as the SOLIBS stuff I 
   have with Darwin :/

2. All the bb.data.inherits_class() stuff is ugly as sin and totally 
   unreadable. My series has a better patch.

Moving on to the rest of the code, I don't see a problem merging any of
the totally new files. How about the following merge process:

a) We push my tree into OE

b) You rebase onto my tree's changes and adjust the EXEEXT stuff.

c) You add changes which add the EXEEXT changes only to existing 
   recipes and commit that.

d) The checksums.ini changes are a no brainer.

e) You start adding the totally new files to OE directly in a logical 
   sequence of something like:

   i) Add canadian core classes (classes/canadian*)
   ii) Add mingw new recipes
   iii) Add misc support recipes (gmp/mprf-canadian)
   iv) Add new binutils recipes
   v) Add new gcc recipes

f) You make a diff of the remaining changes that are needed and we 
   review those.

Hopefully this reduces the overhead of the branch, lets you make
progress and then helps us review the remaining code.

Does that seam reasonable?

Things I've noticed in the rest of the patches are:

* The amount of code duplication in meta-toolchain/canadian-sdk - we 
  really need to think about refactoring that code. 
* I'm also not keen on the SDK_PREFIX -> SDK_PATH change. Why? It 
  breaks things for people. In fact please back this out before merging 
  anything above, I don't see a good reason for it.
* The choice of OVERRIDE you're using worries me as sdk- is far to much 
  of a generic expression. You don't use the overrides much and only 
  for DEFAULT_PREFERENCE. Can we not find a better way to do this and 
  avoid adding overrides? The fact these are there suggests something 
  with the PROVIDES and DEPENDS logic you're using is broken as it 
  should be fairly magic (and a lot of effort went into that for the 
  standard toolchains) :/

Also, this is all on the understanding I've previously mentioned that we
will need to rewrite and break this stuff at some point sooner or later.

Cheers,

Richard

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> > I just had a look through the trini/canadian-sdk branch and there are
> > bits I like and bits I dislike. I'll try and provide some feedback in
> > due course with a view to getting the less controversial bits merged. I
> > have some tweaks in poky to do with dynamic library extension handling
> > for example (from playing with darwin targets) where it would pay us to
> > find a common solution.
> 
> Sorry for the delayed feedback, what I've looked at so far follows
> below. It was easiest to extract some patches from your tree and make
> some commits of my own so these are in:
> 
> http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar
> 
> Basically I wanted to particularly review the changes to the existing OE
> core classes/bitbake.conf. The changes in that branch are versions I'm
> happy with. I had two concerns there:
> 
> 1. I don't want to see ${HOST_EXEEXT} everywhere but I know why you 
>    need it and this was something OE would inevitably face. As a 
>    compromise I propose adding:
> 
>    EXEEXT = "${HOST_EXEEXT}"
> 
>    and using ${EXEEXT} which is fractionally less ugly for all common 
>    uses. Could you update your patch to use ${EXEEXT} please assuming 
>    we all agree on this. Its the same dilemma as the SOLIBS stuff I 
>    have with Darwin :/
> 
> 2. All the bb.data.inherits_class() stuff is ugly as sin and totally 
>    unreadable. My series has a better patch.
> 
> Moving on to the rest of the code, I don't see a problem merging any of
> the totally new files. How about the following merge process:
> 
> a) We push my tree into OE
> 
> b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> 
> c) You add changes which add the EXEEXT changes only to existing 
>    recipes and commit that.
> 
> d) The checksums.ini changes are a no brainer.
> 
> e) You start adding the totally new files to OE directly in a logical 
>    sequence of something like:
> 
>    i) Add canadian core classes (classes/canadian*)
>    ii) Add mingw new recipes
>    iii) Add misc support recipes (gmp/mprf-canadian)
>    iv) Add new binutils recipes
>    v) Add new gcc recipes
> 
> f) You make a diff of the remaining changes that are needed and we 
>    review those.
> 
> Hopefully this reduces the overhead of the branch, lets you make
> progress and then helps us review the remaining code.
> 
> Does that seam reasonable?

Sounds good.  I'll see what I can do about finding somewhere to sit
while borrowing someones wifi at my new place and hopefully get
something going tomorrow.

> Things I've noticed in the rest of the patches are:
> 
> * The amount of code duplication in meta-toolchain/canadian-sdk - we 
>   really need to think about refactoring that code. 

Yes.  And for other purposes I'd like a few more hooks for the including
bb file.  The one OpenMoko pushed (or is trying to push? haven't looked)
is a good starting place.

> * I'm also not keen on the SDK_PREFIX -> SDK_PATH change. Why? It 
>   breaks things for people. In fact please back this out before merging 
>   anything above, I don't see a good reason for it.

Esben?

> * The choice of OVERRIDE you're using worries me as sdk- is far to much 
>   of a generic expression. You don't use the overrides much and only 
>   for DEFAULT_PREFERENCE. Can we not find a better way to do this and 
>   avoid adding overrides? The fact these are there suggests something 
>   with the PROVIDES and DEPENDS logic you're using is broken as it 
>   should be fairly magic (and a lot of effort went into that for the 
>   standard toolchains) :/

I'll see what I can do about removing that, then.

> Also, this is all on the understanding I've previously mentioned that we
> will need to rewrite and break this stuff at some point sooner or later.

Right.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
[snip]
> * I'm also not keen on the SDK_PREFIX -> SDK_PATH change. Why? It 
>   breaks things for people. In fact please back this out before merging 
>   anything above, I don't see a good reason for it.

I tried not doing that and got (for MACHINE=efika):
ERROR: Required build target 'canadian-sdk' has no buildable providers.
Missing or unbuildable dependency chain was: ['canadian-sdk',
'virtual//OpenEmbedded/angstrom/powerpcbinutils']

That said, all I did was a quick dropping of the change.  I'm going to
make sure that my current rebasing on top of your current branch builds
for at least one target then I'll see about dropping the PREFIX->PATH
change again.  But, I'm unsure if we can, in the current overall SDK
implementation.  We do need to say "I need
runs-on-mingw32-builds-for-powerpc binutils".  So the SDK_PREFIX ->
SDK_PATH (in most cases) change makes sense.  We need one variable for
the runs-on-builds-for and one for installs-into.  Or is there a trick
I'm missing (aside from relocatible SDK, which I want)?

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> > I just had a look through the trini/canadian-sdk branch and there are
> > bits I like and bits I dislike. I'll try and provide some feedback in
> > due course with a view to getting the less controversial bits merged. I
> > have some tweaks in poky to do with dynamic library extension handling
> > for example (from playing with darwin targets) where it would pay us to
> > find a common solution.
> 
> Sorry for the delayed feedback, what I've looked at so far follows
> below. It was easiest to extract some patches from your tree and make
> some commits of my own so these are in:
> 
> http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar

This appears to break, DISTRO=angstrom-2008.1 MACHINE=efika bitbake
meta-toolchain-sbox:
Parsing .bb files, please wait...done.
NOTE: Parsing finished. 6091 cached, 0 parsed, 288 skipped, 0 masked.
NOTE: Cache is clean, not saving.
NOTE: build 200901101526: started

OE Build Configuration:
BB_VERSION        = "1.8.11"
METADATA_BRANCH   = "trini/canadian-merge"
METADATA_REVISION = "5ebb709879ffac03ffffb98201f00811d42a79e6"
TARGET_ARCH       = "powerpc"
TARGET_OS         = "linux"
MACHINE           = "efika"
DISTRO            = "angstrom"
DISTRO_VERSION    = "2009.X-test-20090110"
TARGET_FPU        = "hard"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing runqueue
NOTE: Running task 1281 of 1655 (ID: 1597, /scratch/trini/canadian-oe/openembedded/packages/xorg-proto/xproto-native_7.0.13.bb, do_qa_staging)
NOTE: package xproto-native-7.0.13: started
NOTE: package xproto-native-1_7.0.13-r0: task do_qa_staging: started
NOTE: QA checking staging
ERROR: QA Issue: openssl.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: glib-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: xproto.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gobject-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gthread-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: libopkg.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: libcurl.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gmodule-no-export-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gmodule-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gio-unix-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: zlib.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gio-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA Issue: gmodule-export-2.0.pc failed sanity test (tmpdir) in path /scratch/trini/canadian-oe/build/tmp/staging/i686-linux/usr/lib/pkgconfig
ERROR: QA staging was broken by the package built above
ERROR: Error in executing: /scratch/trini/canadian-oe/openembedded/packages/xorg-proto/xproto-native_7.0.13.bb
ERROR: Exception:<type 'exceptions.SystemExit'> Message:1
ERROR: Printing the environment of the function
ERROR: Build of /scratch/trini/canadian-oe/openembedded/packages/xorg-proto/xproto-native_7.0.13.bb do_qa_staging failed
ERROR: Task 1597 (/scratch/trini/canadian-oe/openembedded/packages/xorg-proto/xproto-native_7.0.13.bb, do_qa_staging) failed
NOTE: Tasks Summary: Attempted 1280 tasks of which 1280 didn't need to be rerun and 1 failed.
ERROR: '/scratch/trini/canadian-oe/openembedded/packages/xorg-proto/xproto-native_7.0.13.bb' failed
NOTE: build 200901101526: completed

I'll try and poke around.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Sat, Jan 10, 2009 at 08:33:13AM -0700, Tom Rini wrote:
> On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> > On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> > > I just had a look through the trini/canadian-sdk branch and there are
> > > bits I like and bits I dislike. I'll try and provide some feedback in
> > > due course with a view to getting the less controversial bits merged. I
> > > have some tweaks in poky to do with dynamic library extension handling
> > > for example (from playing with darwin targets) where it would pay us to
> > > find a common solution.
> > 
> > Sorry for the delayed feedback, what I've looked at so far follows
> > below. It was easiest to extract some patches from your tree and make
> > some commits of my own so these are in:
> > 
> > http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar
> 
> This appears to break, DISTRO=angstrom-2008.1 MACHINE=efika bitbake
> meta-toolchain-sbox:

Found it!  The following is needed (I'll push to your branch shortly):

diff --git a/classes/insane.bbclass b/classes/insane.bbclass
index 3e32830..3a2cadc 100644
--- a/classes/insane.bbclass
+++ b/classes/insane.bbclass
@@ -383,12 +383,11 @@ def package_qa_check_staged(path,d):
 
     installed = "installed=yes"
     iscrossnative = False
+    pkgconfigcheck = tmpdir
     for s in ['cross', 'native', 'canadian-cross', 'canadian-native']:
         if bb.data.inherits_class(s, d):
             pkgconfigcheck = workdir
             iscrossnative = True
-    else:
-        pkgconfigcheck = tmpdir
 
     # find all .la and .pc files
     # read the content

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Thu, Jan 08, 2009 at 06:16:03PM -0700, Tom Rini wrote:
> On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
[snip]
> > * The choice of OVERRIDE you're using worries me as sdk- is far to much 
> >   of a generic expression. You don't use the overrides much and only 
> >   for DEFAULT_PREFERENCE. Can we not find a better way to do this and 
> >   avoid adding overrides? The fact these are there suggests something 
> >   with the PROVIDES and DEPENDS logic you're using is broken as it 
> >   should be fairly magic (and a lot of effort went into that for the 
> >   standard toolchains) :/
> 
> I'll see what I can do about removing that, then.

While this must have been needed at some point in development (perhaps
before the changes I made to get the stuff built as $(target)-$(runs-on)
not just $(runs-on)) it's no longer.  Dropped locally and building
everything fine still.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Esben Haabendal]

On Fri, Jan 9, 2009 at 2:16 AM, Tom Rini <trini@kernel.crashing.org> wrote:
> On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
>> * I'm also not keen on the SDK_PREFIX -> SDK_PATH change. Why? It
>>   breaks things for people. In fact please back this out before merging
>>   anything above, I don't see a good reason for it.
>
> Esben?

At best, I find it rather disturbing to have BUILD_PREFIX, HOST_PREFIX and
TARGET_PREFIX meaning one thing, and then having a SDK_PREFIX which
is totally different.  And given that we introduce a number of SDK_* variables
similar to the BUILD_*, HOST_* and TARGET_* variables, I think it will disturb
a reasonable number of other people to without this change.

As for it breaking for other people, I did a search of the org.openembedded.dev
before making this change, and found that SDK_PATH was used so sparingly,
that the change was so clean that I fail to see a problem with it.  Of-course,
I don't know how much non org.openembedded.dev code uses it, but I assume
that any such code should be fixed to rely on proper path variables and not
use the SDK_PREFIX (SDK_PATH) directly.

Best regards,
Esben
-- 
Esben Haabendal, Senior Software Consultant
DoréDevelopment ApS, Ved Stranden 1, 9560 Hadsund, DK-Denmark
Phone: +45 51 92 53 93, E-mail: eha@doredevelopment.dk
WWW: http://www.doredevelopment.dk

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 09, 2009 at 10:04:35AM -0700, Tom Rini wrote:
> On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> [snip]
> > * I'm also not keen on the SDK_PREFIX -> SDK_PATH change. Why? It 
> >   breaks things for people. In fact please back this out before merging 
> >   anything above, I don't see a good reason for it.
> 
> I tried not doing that and got (for MACHINE=efika):
> ERROR: Required build target 'canadian-sdk' has no buildable providers.
> Missing or unbuildable dependency chain was: ['canadian-sdk',
> 'virtual//OpenEmbedded/angstrom/powerpcbinutils']
> 
> That said, all I did was a quick dropping of the change.  I'm going to
> make sure that my current rebasing on top of your current branch builds
> for at least one target then I'll see about dropping the PREFIX->PATH
> change again.  But, I'm unsure if we can, in the current overall SDK
> implementation.  We do need to say "I need
> runs-on-mingw32-builds-for-powerpc binutils".  So the SDK_PREFIX ->
> SDK_PATH (in most cases) change makes sense.  We need one variable for
> the runs-on-builds-for and one for installs-into.  Or is there a trick
> I'm missing (aside from relocatible SDK, which I want)?

OK, I did some more thinking and looking into this.  The problem is that
(as Esben just noted) we have {TARGET,BUILD,HOST}_PREFIX that mean one
thing and SDK_PREFIX that means another.  Adding in the canadian stuff
also means that we need an SDK_PREFIX analogous to the others, along
with the "SDK is compiled for this path" variable.

We can either re-use SDK_PREFIX or in much of the new stuff,
s/${SDK_PREFIX}/${SDK_SYS}-/ but I really think the correct approach is
making SDK_PREFIX mean what {TARGET,BUILD,HOST}_PREFIX mean.  If you
really want, we can hold off on that until the big SDK rework.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Robert Schuster]

[-- Attachment #1: Type: text/plain, Size: 1944 bytes --]

Hi Richard,
I somehow missed this thread.

Richard Purdie schrieb:
> Proposal Step A:
> 
> Having to set:
> 
> FILESDIR = "${@os.path.dirname(bb.data.getVar('FILE',d,1))}/foo-${PV}"
> 
> and
> 
> S = "${WORKDIR}/foo-${PV}
> 
> is fairly pointless and it would be nice the the defaults just did the
> right thing. 
> 
> http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=b8f0510a5567f6d1b90934fe513fd40423f11086
> 
> is a patch introducing BASEPN which is PN with a range of specified
> suffixes removed (-native, -cross-sdk, -cross, -sdk as specified by
> SPECIAL_PKGSUFFIX).
> 
> FILESPATH and S can then be constructed with BASEPN instead of PN
> removing the need to need to hardcode them:
> 
> http://git.pokylinux.org/cgit.cgi/poky/commit/?h=experimental-virtualnative&id=ed829661d51b8f23628e97404ed507c2e56d8495
> 
> This reduces my hypothetical example to:
> 
> foo/foo.inc:
> 
> SRC_URI = "http://somwehere/${BASEPN}-${PV}.tar.bz2 \
>            file://some.patch;patch=1"
> inherit autotools
> 
> foo/foo-native_1.0.bb:
> require foo_${PV}.bb
> inherit native
> 
> The above proposal is worthwhile in itself alone.
Like koen said we have something like this in OE already but your
implementation looks better:

a) BASEPN, BASEP instead of BPN, BP:
This is less prone to be confused with "B". Furthermore I just tried to
put the BPN code into my local Poky and it turned out that somehow BPN
and BP are not set by the 'base_package_name' although its the same code
as in OE's org.oe.dev. There not many packages in OE which use BPN/BP so
we can quickly rename those that do.

I use BP/BPN more extensively in Jalimo but I already have a patch to
fix this.

b) Use of SPECIAL_PKGSUFFIX variable to define known suffixes
In OE's implementation this is hardcoded in the function code. Not so
nice but heck I am no python coder. :$

Regards
Robert



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 268 bytes --]

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> > I just had a look through the trini/canadian-sdk branch and there are
> > bits I like and bits I dislike. I'll try and provide some feedback in
> > due course with a view to getting the less controversial bits merged. I
> > have some tweaks in poky to do with dynamic library extension handling
> > for example (from playing with darwin targets) where it would pay us to
> > find a common solution.
> 
> Sorry for the delayed feedback, what I've looked at so far follows
> below. It was easiest to extract some patches from your tree and make
> some commits of my own so these are in:
> 
> http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar
> 
> Basically I wanted to particularly review the changes to the existing OE
> core classes/bitbake.conf. The changes in that branch are versions I'm
> happy with. I had two concerns there:
> 
> 1. I don't want to see ${HOST_EXEEXT} everywhere but I know why you 
>    need it and this was something OE would inevitably face. As a 
>    compromise I propose adding:
> 
>    EXEEXT = "${HOST_EXEEXT}"
> 
>    and using ${EXEEXT} which is fractionally less ugly for all common 
>    uses. Could you update your patch to use ${EXEEXT} please assuming 
>    we all agree on this. Its the same dilemma as the SOLIBS stuff I 
>    have with Darwin :/
> 
> 2. All the bb.data.inherits_class() stuff is ugly as sin and totally 
>    unreadable. My series has a better patch.
> 
> Moving on to the rest of the code, I don't see a problem merging any of
> the totally new files. How about the following merge process:
> 
> a) We push my tree into OE

My git-fu is weak and since we can't delete remote branches without
bugging someone, can we do this step?  I've got...

> b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> 
> c) You add changes which add the EXEEXT changes only to existing 
>    recipes and commit that.
> 
> d) The checksums.ini changes are a no brainer.
> 
> e) You start adding the totally new files to OE directly in a logical 
>    sequence of something like:
> 
>    i) Add canadian core classes (classes/canadian*)
>    ii) Add mingw new recipes
>    iii) Add misc support recipes (gmp/mprf-canadian)
>    iv) Add new binutils recipes
>    v) Add new gcc recipes

... this done locally, mostly, in quilt.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Tue, Jan 13, 2009 at 06:15:39PM -0700, Tom Rini wrote:
> On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> > On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> > > I just had a look through the trini/canadian-sdk branch and there are
> > > bits I like and bits I dislike. I'll try and provide some feedback in
> > > due course with a view to getting the less controversial bits merged. I
> > > have some tweaks in poky to do with dynamic library extension handling
> > > for example (from playing with darwin targets) where it would pay us to
> > > find a common solution.
> > 
> > Sorry for the delayed feedback, what I've looked at so far follows
> > below. It was easiest to extract some patches from your tree and make
> > some commits of my own so these are in:
> > 
> > http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar
> > 
> > Basically I wanted to particularly review the changes to the existing OE
> > core classes/bitbake.conf. The changes in that branch are versions I'm
> > happy with. I had two concerns there:
> > 
> > 1. I don't want to see ${HOST_EXEEXT} everywhere but I know why you 
> >    need it and this was something OE would inevitably face. As a 
> >    compromise I propose adding:
> > 
> >    EXEEXT = "${HOST_EXEEXT}"
> > 
> >    and using ${EXEEXT} which is fractionally less ugly for all common 
> >    uses. Could you update your patch to use ${EXEEXT} please assuming 
> >    we all agree on this. Its the same dilemma as the SOLIBS stuff I 
> >    have with Darwin :/
> > 
> > 2. All the bb.data.inherits_class() stuff is ugly as sin and totally 
> >    unreadable. My series has a better patch.
> > 
> > Moving on to the rest of the code, I don't see a problem merging any of
> > the totally new files. How about the following merge process:
> > 
> > a) We push my tree into OE
> 
> My git-fu is weak and since we can't delete remote branches without
> bugging someone, can we do this step?  I've got...
> 
> > b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> > 
> > c) You add changes which add the EXEEXT changes only to existing 
> >    recipes and commit that.
> > 
> > d) The checksums.ini changes are a no brainer.
> > 
> > e) You start adding the totally new files to OE directly in a logical 
> >    sequence of something like:
> > 
> >    i) Add canadian core classes (classes/canadian*)
> >    ii) Add mingw new recipes
> >    iii) Add misc support recipes (gmp/mprf-canadian)
> >    iv) Add new binutils recipes
> >    v) Add new gcc recipes
> 
> ... this done locally, mostly, in quilt.

Making more noise as I'm thinking on Friday I'll just merge RP's branch
to mainline.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Wed, Jan 14, 2009 at 04:17:05PM -0700, Tom Rini wrote:
> On Tue, Jan 13, 2009 at 06:15:39PM -0700, Tom Rini wrote:
> > On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> > > On Fri, 2009-01-02 at 01:11 +0000, Richard Purdie wrote:
> > > > I just had a look through the trini/canadian-sdk branch and there are
> > > > bits I like and bits I dislike. I'll try and provide some feedback in
> > > > due course with a view to getting the less controversial bits merged. I
> > > > have some tweaks in poky to do with dynamic library extension handling
> > > > for example (from playing with darwin targets) where it would pay us to
> > > > find a common solution.
> > > 
> > > Sorry for the delayed feedback, what I've looked at so far follows
> > > below. It was easiest to extract some patches from your tree and make
> > > some commits of my own so these are in:
> > > 
> > > http://git.openembedded.net/?p=openembedded.git;a=shortlog;h=refs/heads/rpurdie/canadian-sofar
> > > 
> > > Basically I wanted to particularly review the changes to the existing OE
> > > core classes/bitbake.conf. The changes in that branch are versions I'm
> > > happy with. I had two concerns there:
> > > 
> > > 1. I don't want to see ${HOST_EXEEXT} everywhere but I know why you 
> > >    need it and this was something OE would inevitably face. As a 
> > >    compromise I propose adding:
> > > 
> > >    EXEEXT = "${HOST_EXEEXT}"
> > > 
> > >    and using ${EXEEXT} which is fractionally less ugly for all common 
> > >    uses. Could you update your patch to use ${EXEEXT} please assuming 
> > >    we all agree on this. Its the same dilemma as the SOLIBS stuff I 
> > >    have with Darwin :/
> > > 
> > > 2. All the bb.data.inherits_class() stuff is ugly as sin and totally 
> > >    unreadable. My series has a better patch.
> > > 
> > > Moving on to the rest of the code, I don't see a problem merging any of
> > > the totally new files. How about the following merge process:
> > > 
> > > a) We push my tree into OE
> > 
> > My git-fu is weak and since we can't delete remote branches without
> > bugging someone, can we do this step?  I've got...
> > 
> > > b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> > > 
> > > c) You add changes which add the EXEEXT changes only to existing 
> > >    recipes and commit that.
> > > 
> > > d) The checksums.ini changes are a no brainer.
> > > 
> > > e) You start adding the totally new files to OE directly in a logical 
> > >    sequence of something like:
> > > 
> > >    i) Add canadian core classes (classes/canadian*)
> > >    ii) Add mingw new recipes
> > >    iii) Add misc support recipes (gmp/mprf-canadian)
> > >    iv) Add new binutils recipes
> > >    v) Add new gcc recipes
> > 
> > ... this done locally, mostly, in quilt.
> 
> Making more noise as I'm thinking on Friday I'll just merge RP's branch
> to mainline.

I'm fairly certian my git-fu did not fail and this has been done.  I'll
make branch of e-i -> e-v soon I hope (painting in the new house this
weekend).

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
[snip]
> a) We push my tree into OE

Done on your behalf now.

> b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> 
> c) You add changes which add the EXEEXT changes only to existing 
>    recipes and commit that.
> 
> d) The checksums.ini changes are a no brainer.
> 
> e) You start adding the totally new files to OE directly in a logical 
>    sequence of something like:
> 
>    i) Add canadian core classes (classes/canadian*)
>    ii) Add mingw new recipes
>    iii) Add misc support recipes (gmp/mprf-canadian)
>    iv) Add new binutils recipes
>    v) Add new gcc recipes

OK, this is the trini/canadian-merge branch.

> f) You make a diff of the remaining changes that are needed and we 
>    review those.

This is all that's left:
diff --git a/classes/package_ipk.bbclass b/classes/package_ipk.bbclass
index a20b6bb..3a97729 100644
--- a/classes/package_ipk.bbclass
+++ b/classes/package_ipk.bbclass
@@ -5,6 +5,7 @@ IMAGE_PKGTYPE ?= "ipk"
 
 IPKGCONF_TARGET = "${STAGING_ETCDIR_NATIVE}/opkg.conf"
 IPKGCONF_SDK =  "${STAGING_ETCDIR_NATIVE}/opkg-sdk.conf"
+IPKGCONF_CANSDK =  "${STAGING_ETCDIR_NATIVE}/opkg-canadian-sdk.conf"
 
 python package_ipk_fn () {
 	from bb import data
@@ -90,6 +91,10 @@ package_update_index_ipk () {
 			touch ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages
 			ipkg-make-index -r ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages -p ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages -l ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages.filelist -m ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/
 		fi
+		if [ -e ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/ ] ; then
+			touch ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages
+			ipkg-make-index -r ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages -p ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages -l ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages.filelist -m ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/
+		fi
 	done
 }
 
@@ -102,11 +107,13 @@ package_generate_ipkg_conf () {
 	mkdir -p ${STAGING_ETCDIR_NATIVE}/
 	echo "src oe file:${DEPLOY_DIR_IPK}" > ${IPKGCONF_TARGET}
 	echo "src oe file:${DEPLOY_DIR_IPK}" > ${IPKGCONF_SDK}
+	echo "src oe file:${DEPLOY_DIR_IPK}" > ${IPKGCONF_CANSDK}
 	ipkgarchs="${PACKAGE_ARCHS}"
 	priority=1
 	for arch in $ipkgarchs; do
 		echo "arch $arch $priority" >> ${IPKGCONF_TARGET}
 		echo "arch ${BUILD_ARCH}-$arch-sdk $priority" >> ${IPKGCONF_SDK}
+		echo "arch ${SDK_SYS}-sdk-$arch $priority" >> ${IPKGCONF_CANSDK}
 		priority=$(expr $priority + 5)
 		if [ -e ${DEPLOY_DIR_IPK}/$arch/Packages ] ; then
 		        echo "src oe-$arch file:${DEPLOY_DIR_IPK}/$arch" >> ${IPKGCONF_TARGET}
@@ -114,6 +121,9 @@ package_generate_ipkg_conf () {
 		if [ -e ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages ] ; then
 		        echo "src oe-${BUILD_ARCH}-$arch-sdk file:${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk" >> ${IPKGCONF_SDK}
 		fi
+		if [ -e ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages ] ; then
+		        echo "src oe-${SDK_SYS}-sdk-$arch file:${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch" >> ${IPKGCONF_CANSDK}
+		fi
 	done
 }
 
diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index 06fb893..a136548 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -89,6 +89,14 @@ TARGET_PREFIX = "${TARGET_SYS}-"
 TARGET_CC_ARCH = ""
 TARGET_EXEEXT = ""
 
+SDK_ARCH ?= "${BUILD_ARCH}"
+SDK_OS ?= "${BUILD_OS}"
+SDK_VENDOR ?= "${BUILD_VENDOR}"
+SDK_SYS = "${SDK_ARCH}${SDK_VENDOR}${@['-' + bb.data.getVar('SDK_OS', d, 1), ''][bb.data.getVar('SDK_OS', d, 1) == ('' or 'custom')]}"
+SDK_PREFIX = "${SDK_SYS}-"
+SDK_CC_ARCH ?= "${BUILD_CC_ARCH}"
+SDK_EXEEXT = ""
+
 # A shortcut for the commonly used value
 EXEEXT = "${HOST_EXEEXT}"
 
@@ -274,6 +282,8 @@ STAGING_PYDIR = "${STAGING_DIR}/lib/python2.4"
 # specific packages - hack around it for now.
 STAGING_DIR_TARGET = "${STAGING_DIR}/${BASEPKG_TARGET_SYS}"
 
+STAGING_DIR_SDK = "${STAGING_DIR}/${SDK_SYS}"
+
 DEPLOY_DIR = "${TMPDIR}/deploy"
 DEPLOY_DIR_TAR = "${DEPLOY_DIR}/tar"
 DEPLOY_DIR_IPK = "${DEPLOY_DIR}/ipk"
@@ -398,14 +408,17 @@ PATCHRESOLVE = 'noop'
 export BUILD_CPPFLAGS = "-isystem${STAGING_INCDIR_NATIVE}"
 export CPPFLAGS = "${TARGET_CPPFLAGS}"
 export TARGET_CPPFLAGS = "-isystem${STAGING_DIR_TARGET}${layout_includedir}"
+export SDK_CPPFLAGS = "-isystem${STAGING_DIR_SDK}${layout_includedir} -isystem${STAGING_DIR_HOST}${layout_includedir}"
 
 export BUILD_CFLAGS = "${BUILD_CPPFLAGS} ${BUILD_OPTIMIZATION}"
 export CFLAGS = "${TARGET_CFLAGS}"
 export TARGET_CFLAGS = "${TARGET_CPPFLAGS} ${SELECTED_OPTIMIZATION}"
+export SDK_CFLAGS = "${SDK_CPPFLAGS} ${SELECTED_OPTIMIZATION}"
 
 export BUILD_CXXFLAGS = "${BUILD_CFLAGS} -fpermissive"
 export CXXFLAGS = "${TARGET_CXXFLAGS}"
 export TARGET_CXXFLAGS = "${TARGET_CFLAGS} -fpermissive"
+export SDK_CXXFLAGS = "${SDK_CFLAGS} -fpermissive"
 
 export BUILD_LDFLAGS = "-L${STAGING_LIBDIR_NATIVE} \
                         -Wl,-rpath-link,${STAGING_LIBDIR_NATIVE} \
@@ -416,6 +429,9 @@ export TARGET_LDFLAGS = "-L${STAGING_DIR_TARGET}${layout_libdir} \
                          -Wl,-rpath-link,${STAGING_DIR_TARGET}${layout_libdir} \
                          -Wl,-O1 \
                          ${TARGET_LINK_HASH_STYLE}"
+export SDK_LDFLAGS = "-L${STAGING_DIR_SDK}${layout_libdir} \
+                         -Wl,-rpath-link,${STAGING_DIR_SDK}${layout_libdir} \
+                         -Wl,-O1"
 
 # Which flags to leave by strip-flags() in bin/build/oebuild.sh ?
 ALLOWED_FLAGS = "-O -mcpu -march -pipe"
diff --git a/packages/gcc/gcc-package-sdk.inc b/packages/gcc/gcc-package-sdk.inc
index 41f47ae..1882fbe 100644
--- a/packages/gcc/gcc-package-sdk.inc
+++ b/packages/gcc/gcc-package-sdk.inc
@@ -9,10 +9,10 @@ PACKAGES = "${PN} ${PN}-doc"
 
 FILES_${PN} = "\
     ${bindir}/* \
-    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/cc1 \
-    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/collect2 \
-    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/f771 \
-    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/cc1plus \
+    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/cc1${EXEEXT} \
+    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/collect2${EXEEXT} \
+    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/f771${EXEEXT} \
+    ${libexecdir}/gcc/${TARGET_SYS}/${BINV}/cc1plus${EXEEXT} \
     ${gcclibdir}/${TARGET_SYS}/${BINV}/*.o \
     ${gcclibdir}/${TARGET_SYS}/${BINV}/specs \
     ${gcclibdir}/${TARGET_SYS}/${BINV}/lib* \
@@ -29,17 +29,17 @@ FILES_${PN}-doc = "\
     "
 
 do_install () {
-	oe_runmake 'DESTDIR=${D}' install
+	oe_runmake 'DESTDIR=${D}' 'prefix=${SDK_PATH}' install
 
 	# Cleanup some of the ${libdir}{,exec}/gcc stuff ...
 	rm -r ${D}${libdir}/gcc/${TARGET_SYS}/${BINV}/install-tools
 	rm -r ${D}${libexecdir}/gcc/${TARGET_SYS}/${BINV}/install-tools
 
 	# We care about g++ not c++
-	rm -f ${D}${bindir}/*c++
+	rm -f ${D}${bindir}/*c++${EXEEXT}
 
 	# We don't care about the gcc-<version> copies
-	rm -f ${D}${bindir}/*gcc-?.?*
+	rm -f ${D}${bindir}/*gcc-?.?*${EXEEXT}
 
 	# We use libiberty from binutils
 	rm -f ${D}${prefix}/${TARGET_SYS}/lib/libiberty.a
@@ -50,7 +50,7 @@ do_install () {
 	for l in ${D}${bindir}/*; do
 		ln -sf "${bindir}/`basename $l`" "${D}${prefix}/${TARGET_SYS}/bin/`basename $l | sed -e 's,${TARGET_PREFIX},,'`"
 	done
-	ln -sf "${bindir}/${TARGET_PREFIX}gcc" "${D}${prefix}/${TARGET_SYS}/bin/cc"
+	ln -sf "${bindir}/${TARGET_PREFIX}gcc${EXEEXT}" "${D}${prefix}/${TARGET_SYS}/bin/cc${EXEEXT}"
 
 	# Manually run the target stripper since we won't get it run by
 	# the packaging.
@@ -60,4 +60,3 @@ do_install () {
 		${TARGET_PREFIX}strip ${D}${prefix}/${TARGET_SYS}/lib/libgcc_s.so.*
 	fi
 }
-
diff --git a/packages/meta/canadian-sdk.bb b/packages/meta/canadian-sdk.bb
new file mode 100644
index 0000000..150c752
--- /dev/null
+++ b/packages/meta/canadian-sdk.bb
@@ -0,0 +1,136 @@
+DESCRIPTION = "Meta package for building a installable toolchain"
+LICENSE = "MIT"
+DEPENDS = "ipkg-native ipkg-utils-native fakeroot-native sed-native zip-native"
+PR = "r2"
+
+inherit canadian-sdk meta
+
+SDK_DIR = "${WORKDIR}/sdk"
+SDK_OUTPUT = "${SDK_DIR}/image"
+SDK_DEPLOY = "${TMPDIR}/deploy/sdk"
+SDK_SUFFIX = "toolchain"
+
+FEED_ARCH ?= "${TARGET_ARCH}"
+
+IPKG_HOST = "ipkg-cl -f ${IPKGCONF_CANSDK} -o ${SDK_OUTPUT}"
+IPKG_TARGET = "ipkg-cl -f ${IPKGCONF_TARGET} -o ${SDK_OUTPUT}${prefix}"
+
+TOOLCHAIN_CANADIAN_HOST_TASK ?= "task-sdk-canadian-host"
+TOOLCHAIN_TARGET_TASK ?= "task-sdk-bare"
+
+RDEPENDS = "${TOOLCHAIN_TARGET_TASK} ${TOOLCHAIN_CANADIAN_HOST_TASK}"
+
+do_populate_sdk() {
+	rm -rf ${SDK_OUTPUT}
+	mkdir -p ${SDK_OUTPUT}
+
+	package_update_index_ipk
+	package_generate_ipkg_conf
+
+	for arch in ${PACKAGE_ARCHS}; do
+		revipkgarchs="$arch $revipkgarchs"
+	done
+
+	${IPKG_HOST} update
+	${IPKG_HOST} -force-depends install ${TOOLCHAIN_CANADIAN_HOST_TASK}
+
+	${IPKG_TARGET} update
+	${IPKG_TARGET} install ${TOOLCHAIN_TARGET_TASK}
+
+	mkdir -p ${SDK_OUTPUT}${prefix}/${TARGET_SYS}
+	cp -pPR ${SDK_OUTPUT}${prefix}/usr ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/
+	rm -rf ${SDK_OUTPUT}${prefix}/usr
+
+	cp -pPR ${SDK_OUTPUT}${prefix}/lib/* ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/
+	rm -rf ${SDK_OUTPUT}${prefix}/lib/*
+
+	cp -pPR ${SDK_OUTPUT}/usr/lib/ipkg ${SDK_OUTPUT}${prefix}/lib/
+	rm -rf ${SDK_OUTPUT}/usr/lib/ipkg/*
+	rmdir -p --ignore-fail-on-non-empty ${SDK_OUTPUT}/usr/lib/ipkg
+
+	for fn in `ls ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/`; do
+		if [ -h ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/$fn ]; then
+			link=`readlink ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/$fn`
+			bname=`basename $link`
+			if [ ! -e $link -a -e ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/$bame ]; then
+				rm ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/$fn
+				ln -s $bname ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/$fn
+			fi
+		fi
+	done
+
+	mv ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/gcc ${SDK_OUTPUT}${prefix}/lib
+
+	echo 'GROUP ( libpthread.so.0 libpthread_nonshared.a )' > ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/libpthread.so
+	echo 'GROUP ( libc.so.6 libc_nonshared.a )' > ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/lib/libc.so
+
+	# extract and store ipks, pkgdata and shlibs data
+	target_pkgs=`cat ${SDK_OUTPUT}${prefix}/${TARGET_SYS}/usr/lib/ipkg/status | grep Package: | cut -f 2 -d ' '`
+	mkdir -p ${SDK_OUTPUT}/${prefix}/ipk/
+	mkdir -p ${SDK_OUTPUT}/${prefix}/pkgdata/runtime/
+	mkdir -p ${SDK_OUTPUT}/${prefix}/${TARGET_SYS}/shlibs/
+	for pkg in $target_pkgs ; do
+		for arch in $revipkgarchs; do
+			pkgnames=${DEPLOY_DIR_IPK}/$arch/${pkg}_*_$arch.ipk
+			if [ -e $pkgnames ]; then
+				oenote "Found $pkgnames"
+				cp $pkgnames ${SDK_OUTPUT}/${prefix}/ipk/
+				orig_pkg=`ipkg-list-fields $pkgnames | grep OE: | cut -d ' ' -f2`
+				pkg_subdir=$arch${TARGET_VENDOR}${@['-' + bb.data.getVar('TARGET_OS', d, 1), ''][bb.data.getVar('TARGET_OS', d, 1) == ('' or 'custom')]}
+				mkdir -p ${SDK_OUTPUT}/${prefix}/pkgdata/$pkg_subdir/runtime
+				cp ${TMPDIR}/pkgdata/$pkg_subdir/$orig_pkg ${SDK_OUTPUT}/${prefix}/pkgdata/$pkg_subdir/
+				subpkgs=`cat ${TMPDIR}/pkgdata/$pkg_subdir/$orig_pkg | grep PACKAGES: | cut -b 10-`
+				for subpkg in $subpkgs; do
+					cp ${TMPDIR}/pkgdata/$pkg_subdir/runtime/$subpkg ${SDK_OUTPUT}/${prefix}/pkgdata/$pkg_subdir/runtime/
+					if [ -e ${TMPDIR}/pkgdata/$pkg_subdir/runtime/$subpkg.packaged ];then
+						cp ${TMPDIR}/pkgdata/$pkg_subdir/runtime/$subpkg.packaged ${SDK_OUTPUT}/${prefix}/pkgdata/$pkg_subdir/runtime/
+					fi
+					if [ -e ${STAGING_DIR_TARGET}/shlibs/$subpkg.list ]; then
+						cp ${STAGING_DIR_TARGET}/shlibs/$subpkg.* ${SDK_OUTPUT}/${prefix}/${TARGET_SYS}/shlibs/
+					fi
+				done
+				break
+			fi
+		done
+	done
+
+	# add missing link to libgcc_s.so.1
+	# libgcc-dev should be responsible for that, but it's not getting built
+	# RP: it gets smashed up depending on the order that gcc, gcc-cross and
+	# gcc-cross-sdk get built :( (30/11/07)
+	ln -sf libgcc_s.so.1 ${SDK_OUTPUT}/${prefix}/${TARGET_SYS}/lib/libgcc_s.so
+
+	# Fix or remove broken .la files
+	for i in `find ${SDK_OUTPUT}/${prefix}/${TARGET_SYS} -name \*.la`; do
+		sed -i 	-e "/^dependency_libs=/s,\([[:space:]']\)${layout_base_libdir},\1${prefix}/${TARGET_SYS}${layout_base_libdir},g" \
+			-e "/^dependency_libs=/s,\([[:space:]']\)${layout_libdir},\1${prefix}/${TARGET_SYS}${layout_libdir},g" \
+			-e "/^dependency_libs=/s,\-\([LR]\)${layout_base_libdir},-\1${prefix}/${TARGET_SYS}${layout_base_libdir},g" \
+			-e "/^dependency_libs=/s,\-\([LR]\)${layout_libdir},-\1${prefix}/${TARGET_SYS}${layout_libdir},g" \
+			-e 's/^installed=yes$/installed=no/' $i
+	done
+	rm -f ${SDK_OUTPUT}/${prefix}/lib/*.la
+
+	# Setup site file for external use
+	siteconfig=${SDK_OUTPUT}/${prefix}/site-config
+	touch $siteconfig
+	for sitefile in ${CONFIG_SITE} ; do
+		cat $sitefile >> $siteconfig
+	done
+}
+
+do_package_sdk() {
+	# package it up
+	mkdir -p ${SDK_DEPLOY}
+	cd ${SDK_OUTPUT}
+	fakeroot tar cfj ${SDK_DEPLOY}/${SDK_SYS}-sdk-${DISTRO}-${DISTRO_VERSION}-${FEED_ARCH}-${TARGET_OS}.tar.bz2 .${prefix}
+	rm -f ${SDK_DEPLOY}/${SDK_SYS}-sdk-${DISTRO}-${DISTRO_VERSION}-${FEED_ARCH}-${TARGET_OS}.zip
+	zip -r -D ${SDK_DEPLOY}/${SDK_SYS}-sdk-${DISTRO}-${DISTRO_VERSION}-${FEED_ARCH}-${TARGET_OS}.zip .${prefix}
+}
+
+do_populate_sdk[nostamp] = "1"
+do_populate_sdk[recrdeptask] = "do_package_write"
+addtask populate_sdk before do_build after do_install
+
+do_package_sdk[nostamp] = "1"
+do_package_sdk[recrdeptask] = "do_populate_sdk"
+addtask package_sdk before do_build after do_populate_sdk

So, we've got a few things in here:
- IPKGCONF_CANSDK stuff.  Analogous to the others we have now.
- The rest of the SDK_foo variables.
- ${EXEEXT} stuff in gcc-package.sdk.inc
- In gcc-package-sdk.inc's do_install, pass in 'prefix=${SDK_PATH}' to
  the install target.  This is needed in some path games that get played
  by gcc, iirc.
- canadian-sdk.bb

> * The amount of code duplication in meta-toolchain/canadian-sdk - we 
>   really need to think about refactoring that code. 

Totally agree, but can we bring this in THEN try and abstract things out
more (I'm thinking meta-toolchain.inc for a filename, and functions for
common stuff)?

> * I'm also not keen on the SDK_PREFIX -> SDK_PATH change. Why? It 
>   breaks things for people. In fact please back this out before merging 
>   anything above, I don't see a good reason for it.

See the other parts of this thread for a recap.  I've specifically put
this into the merge branch.

> * The choice of OVERRIDE you're using worries me as sdk- is far to much 
>   of a generic expression. You don't use the overrides much and only 
>   for DEFAULT_PREFERENCE. Can we not find a better way to do this and 
>   avoid adding overrides? The fact these are there suggests something 
>   with the PROVIDES and DEPENDS logic you're using is broken as it 
>   should be fairly magic (and a lot of effort went into that for the 
>   standard toolchains) :/

As I've said elsewhere, dropped.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Fri, Jan 16, 2009 at 09:47:28PM -0700, Tom Rini wrote:
> On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> [snip]
> > a) We push my tree into OE
> 
> Done on your behalf now.
> 
> > b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> > 
> > c) You add changes which add the EXEEXT changes only to existing 
> >    recipes and commit that.
> > 
> > d) The checksums.ini changes are a no brainer.
> > 
> > e) You start adding the totally new files to OE directly in a logical 
> >    sequence of something like:
> > 
> >    i) Add canadian core classes (classes/canadian*)
> >    ii) Add mingw new recipes
> >    iii) Add misc support recipes (gmp/mprf-canadian)
> >    iv) Add new binutils recipes
> >    v) Add new gcc recipes
> 
> OK, this is the trini/canadian-merge branch.

I would like to merge those new recipes (not the changes to existing
stuff) on the 26th, if no one objects.

-- 
Tom Rini

Re: RFC: "Virtual" native and sdk recipes

[Tom Rini]

On Thu, Jan 22, 2009 at 11:10:26AM -0700, Tom Rini wrote:
> On Fri, Jan 16, 2009 at 09:47:28PM -0700, Tom Rini wrote:
> > On Fri, Jan 09, 2009 at 12:54:50AM +0000, Richard Purdie wrote:
> > [snip]
> > > a) We push my tree into OE
> > 
> > Done on your behalf now.
> > 
> > > b) You rebase onto my tree's changes and adjust the EXEEXT stuff.
> > > 
> > > c) You add changes which add the EXEEXT changes only to existing 
> > >    recipes and commit that.
> > > 
> > > d) The checksums.ini changes are a no brainer.
> > > 
> > > e) You start adding the totally new files to OE directly in a logical 
> > >    sequence of something like:
> > > 
> > >    i) Add canadian core classes (classes/canadian*)
> > >    ii) Add mingw new recipes
> > >    iii) Add misc support recipes (gmp/mprf-canadian)
> > >    iv) Add new binutils recipes
> > >    v) Add new gcc recipes
> > 
> > OK, this is the trini/canadian-merge branch.
> 
> I would like to merge those new recipes (not the changes to existing
> stuff) on the 26th, if no one objects.

A few days later even, and done.  I'm going to start a new thread for my
next round of comment-getting.

-- 
Tom Rini