Packaged-staging and RPATH with native/cross/sdk

Packaged-staging and RPATH with native/cross/sdk

[Chris Conroy]

I'm trying to use packaged-staging to set up a sane prebuilt toolchain
environment to share with other developers. I view packaged-staging as a
promising route of creating a "Stage 3" version of OE (to use some
Gentoo parlance). While building the toolchain isn't all that painful,
even on a decent system it can add 1-2 hours to the build process, and
every individual developer needs to redo the toolchain and base
libraries for every feature branch. This is all needlessly wasted time
since they're all ultimately building the same things if on the same
arch.

Packaged-staging seems to give me everything we need in order to
accomplish this except for the way RPATH is handled. This is a known
issue which is worked around in packaged-staging with the following
snippet:

    # These classes encode staging paths into the binary data so can
only be
    # reused if the path doesn't change/
    if bb.data.inherits_class('native', d) or
bb.data.inherits_class('cross', d) or bb.data.inherits_class('sdk', d):
        path = bb.data.getVar('PSTAGE_PKGPATH', d, 1)
        path = path + bb.data.getVar('TMPDIR', d, 1).replace('/', '-')
        bb.data.setVar('PSTAGE_PKGPATH', path, d)


I'm currently toying with changing the references to rpath in
bitbake.conf to use $ORIGIN rather than an absolute path, but perhaps
some of you have better ideas. (I have a feeling this is going to run
into some nasty edge cases).

(The other, less elegant idea I am toying with is: modify the
packaged-staging install to hotfix the binary data to point at the new
install location, but this seems uglier and possibly harder to
implement)

If this special case is taken away, then distributions can much more
easily share a prebuilt toolchain and set of system libraries for
potential developers or with a team. It also gives us some other nice
properties (e.g. you want to relocate your OE directory because your
disk is filling up, but you don't want to have to do a full rebuild)


Thoughts, suggestions, reasons this won't ever possibly work?

--Chris Conroy

Re: Packaged-staging and RPATH with native/cross/sdk

[Chris Larson]

I've been using a $ORIGIN rpath in my builds, and it's definitely a
step in the right direction.  Iirc there are a few quirks with things
like perl modules and such, but you can work around those.

There are a lot of relocation issues with stuff we stage that need to
be fixed one by one to use those particular staging packages with tmp
in a different location, though.  Sadly, few upstream packages are
really relocatable.  Some of it can be worked around by exporting env
vars.. for example, you can point 'file' at its data with an env var,
and you can override nearly all the paths used in autoconf/automake
via env vars (the autotools branch i started work on does that).

On Thu, Jul 30, 2009 at 2:34 PM, Chris
Conroy<Chris.Conroy@hillcrestlabs.com> wrote:
> I'm trying to use packaged-staging to set up a sane prebuilt toolchain
> environment to share with other developers. I view packaged-staging as a
> promising route of creating a "Stage 3" version of OE (to use some
> Gentoo parlance). While building the toolchain isn't all that painful,
> even on a decent system it can add 1-2 hours to the build process, and
> every individual developer needs to redo the toolchain and base
> libraries for every feature branch. This is all needlessly wasted time
> since they're all ultimately building the same things if on the same
> arch.
>
> Packaged-staging seems to give me everything we need in order to
> accomplish this except for the way RPATH is handled. This is a known
> issue which is worked around in packaged-staging with the following
> snippet:
>
>    # These classes encode staging paths into the binary data so can
> only be
>    # reused if the path doesn't change/
>    if bb.data.inherits_class('native', d) or
> bb.data.inherits_class('cross', d) or bb.data.inherits_class('sdk', d):
>        path = bb.data.getVar('PSTAGE_PKGPATH', d, 1)
>        path = path + bb.data.getVar('TMPDIR', d, 1).replace('/', '-')
>        bb.data.setVar('PSTAGE_PKGPATH', path, d)
>
>
> I'm currently toying with changing the references to rpath in
> bitbake.conf to use $ORIGIN rather than an absolute path, but perhaps
> some of you have better ideas. (I have a feeling this is going to run
> into some nasty edge cases).
>
> (The other, less elegant idea I am toying with is: modify the
> packaged-staging install to hotfix the binary data to point at the new
> install location, but this seems uglier and possibly harder to
> implement)
>
> If this special case is taken away, then distributions can much more
> easily share a prebuilt toolchain and set of system libraries for
> potential developers or with a team. It also gives us some other nice
> properties (e.g. you want to relocate your OE directory because your
> disk is filling up, but you don't want to have to do a full rebuild)
>
>
> Thoughts, suggestions, reasons this won't ever possibly work?
>
> --Chris Conroy
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>



-- 
Chris Larson
clarson at kergoth dot com
clarson at mvista dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Software Engineer
MontaVista Software, Inc.

Re: Packaged-staging and RPATH with native/cross/sdk

[Richard Purdie]

On Thu, 2009-07-30 at 17:34 -0400, Chris Conroy wrote:
> I'm trying to use packaged-staging to set up a sane prebuilt toolchain
> environment to share with other developers. I view packaged-staging as a
> promising route of creating a "Stage 3" version of OE (to use some
> Gentoo parlance). While building the toolchain isn't all that painful,
> even on a decent system it can add 1-2 hours to the build process, and
> every individual developer needs to redo the toolchain and base
> libraries for every feature branch. This is all needlessly wasted time
> since they're all ultimately building the same things if on the same
> arch.
> 
> Packaged-staging seems to give me everything we need in order to
> accomplish this except for the way RPATH is handled. This is a known
> issue which is worked around in packaged-staging with the following
> snippet:
> 
>     # These classes encode staging paths into the binary data so can
> only be
>     # reused if the path doesn't change/
>     if bb.data.inherits_class('native', d) or
> bb.data.inherits_class('cross', d) or bb.data.inherits_class('sdk', d):
>         path = bb.data.getVar('PSTAGE_PKGPATH', d, 1)
>         path = path + bb.data.getVar('TMPDIR', d, 1).replace('/', '-')
>         bb.data.setVar('PSTAGE_PKGPATH', path, d)

Not meaning to sound negative but I think you'll hit some interesting
edge cases and this is why packaged staging currently works as above.

You do have one other option which is an external toolchain. The idea
here is that you have OE build an SDK to install into
say /usr/local/OE-toolchain-foobar, then ask your users to install this
and point OE at the toolchain in that folder. It will still build some
-native packages but you get a big speedup as the toolchain is the heavy
part.

Also, I think the -cross packages could be made relocatable quite
easily, particularly with sysroot and its the -native ones which are
hard. If the -cross packages were fixed, a large chunk of your build
time also goes away.

There are plenty of options but they all need a little bit of time
investment unfortunately.

Cheers,

Richard

Re: Packaged-staging and RPATH with native/cross/sdk

[Tom Rini]

On Thu, Jul 30, 2009 at 03:16:58PM -0700, Chris Larson wrote:

> I've been using a $ORIGIN rpath in my builds, and it's definitely a
> step in the right direction.  Iirc there are a few quirks with things
> like perl modules and such, but you can work around those.
> 
> There are a lot of relocation issues with stuff we stage that need to
> be fixed one by one to use those particular staging packages with tmp
> in a different location, though.  Sadly, few upstream packages are
> really relocatable.  Some of it can be worked around by exporting env
> vars.. for example, you can point 'file' at its data with an env var,
> and you can override nearly all the paths used in autoconf/automake
> via env vars (the autotools branch i started work on does that).

Perhaps we need to whitelist some cross/native/sdk packages?  What
people really notice is building gcc a few times, and that stuff is
relocatible (and if these aren't, it's a recipe bug).  Might be a few
others too, but just not rebuilding gcc will be noticed.

-- 
Tom Rini