sstate reuse for -native, -cross across different host glibc version, how to make it work?

sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Koen Kooi]

Hi,

I turned on sstate mirroring for angstrom recently and I'm getting reports of build failures due to missing GLIBC_2.14 symbols:

arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found (required by arm-angstrom-linux-gnueabi-gcc)

The sstate tarballs are built on a Fedora16 VM and the breakage occurs when it being used on systems with an older c library (e.g. debian). To get rid of this problem there are multiple options, but I think the 2 most obvious are:

1) inject host distroname and distroversion into the checksums
2) build everything against a native libc

Would it be appropriate to get 1) into oe-core before the branchpoint? Right now my workaround is 'rm *native* *cross* -f' from time to time. But that sucks for identical autobuilders.

regards,

Koen

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Richard Purdie]

On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
> I turned on sstate mirroring for angstrom recently and I'm getting
> reports of build failures due to missing GLIBC_2.14 symbols:
> 
> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
> version `GLIBC_2.14' not found (required by
> arm-angstrom-linux-gnueabi-gcc)
> 
> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
> when it being used on systems with an older c library (e.g. debian).
> To get rid of this problem there are multiple options, but I think the
> 2 most obvious are:
> 
> 1) inject host distroname and distroversion into the checksums
> 2) build everything against a native libc
> 
3) Use an older version on the VM which is the oldest distro you plan to
build against.

> Would it be appropriate to get 1) into oe-core before the branchpoint?

We've talked about this and its been on the "to fix" list but nobody has
got around to it. Its hard as we need to come up with something that
isn't going to kill performance of the checksum calculations. A cat
operation on a few files for each checksum for example isn't
appropriate. We may need to do something at the bitbake level as there
is also the issue of checksuming local files such as those in file://
urls and including that in the sstate checksums.

So whilst I'd love to see fixes for these and they are bugfixes, they're
going to have to be well written patches and its late in cycle for
invasive changes :(.

Cheers,

Richard

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Chris Larson]

On Fri, Mar 23, 2012 at 5:41 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
>> I turned on sstate mirroring for angstrom recently and I'm getting
>> reports of build failures due to missing GLIBC_2.14 symbols:
>>
>> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
>> version `GLIBC_2.14' not found (required by
>> arm-angstrom-linux-gnueabi-gcc)
>>
>> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
>> when it being used on systems with an older c library (e.g. debian).
>> To get rid of this problem there are multiple options, but I think the
>> 2 most obvious are:
>>
>> 1) inject host distroname and distroversion into the checksums
>> 2) build everything against a native libc
>>
> 3) Use an older version on the VM which is the oldest distro you plan to
> build against.
>
>> Would it be appropriate to get 1) into oe-core before the branchpoint?
>
> We've talked about this and its been on the "to fix" list but nobody has
> got around to it. Its hard as we need to come up with something that
> isn't going to kill performance of the checksum calculations. A cat
> operation on a few files for each checksum for example isn't
> appropriate. We may need to do something at the bitbake level as there
> is also the issue of checksuming local files such as those in file://
> urls and including that in the sstate checksums.
>
> So whilst I'd love to see fixes for these and they are bugfixes, they're
> going to have to be well written patches and its late in cycle for
> invasive changes :(.

At Mentor, right now, I'm using a ConfigParsed event handler to run
lsb_release and inject the DISTRO/DISTRO_VERSION into the metadata,
which I then use vardeps to ensure get into the appropriate
signatures. Not perfect, but gets the job done.
-- 
Christopher Larson

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Richard Purdie]

On Fri, 2012-03-23 at 07:12 -0700, Chris Larson wrote:
> On Fri, Mar 23, 2012 at 5:41 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
> >> I turned on sstate mirroring for angstrom recently and I'm getting
> >> reports of build failures due to missing GLIBC_2.14 symbols:
> >>
> >> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
> >> version `GLIBC_2.14' not found (required by
> >> arm-angstrom-linux-gnueabi-gcc)
> >>
> >> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
> >> when it being used on systems with an older c library (e.g. debian).
> >> To get rid of this problem there are multiple options, but I think the
> >> 2 most obvious are:
> >>
> >> 1) inject host distroname and distroversion into the checksums
> >> 2) build everything against a native libc
> >>
> > 3) Use an older version on the VM which is the oldest distro you plan to
> > build against.
> >
> >> Would it be appropriate to get 1) into oe-core before the branchpoint?
> >
> > We've talked about this and its been on the "to fix" list but nobody has
> > got around to it. Its hard as we need to come up with something that
> > isn't going to kill performance of the checksum calculations. A cat
> > operation on a few files for each checksum for example isn't
> > appropriate. We may need to do something at the bitbake level as there
> > is also the issue of checksuming local files such as those in file://
> > urls and including that in the sstate checksums.
> >
> > So whilst I'd love to see fixes for these and they are bugfixes, they're
> > going to have to be well written patches and its late in cycle for
> > invasive changes :(.
> 
> At Mentor, right now, I'm using a ConfigParsed event handler to run
> lsb_release and inject the DISTRO/DISTRO_VERSION into the metadata,
> which I then use vardeps to ensure get into the appropriate
> signatures. Not perfect, but gets the job done.

Are those patches available anywhere we can look at?

Cheers,

Richard

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Chris Larson]

On Fri, Mar 23, 2012 at 8:13 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2012-03-23 at 07:12 -0700, Chris Larson wrote:
>> On Fri, Mar 23, 2012 at 5:41 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> > On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
>> >> I turned on sstate mirroring for angstrom recently and I'm getting
>> >> reports of build failures due to missing GLIBC_2.14 symbols:
>> >>
>> >> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
>> >> version `GLIBC_2.14' not found (required by
>> >> arm-angstrom-linux-gnueabi-gcc)
>> >>
>> >> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
>> >> when it being used on systems with an older c library (e.g. debian).
>> >> To get rid of this problem there are multiple options, but I think the
>> >> 2 most obvious are:
>> >>
>> >> 1) inject host distroname and distroversion into the checksums
>> >> 2) build everything against a native libc
>> >>
>> > 3) Use an older version on the VM which is the oldest distro you plan to
>> > build against.
>> >
>> >> Would it be appropriate to get 1) into oe-core before the branchpoint?
>> >
>> > We've talked about this and its been on the "to fix" list but nobody has
>> > got around to it. Its hard as we need to come up with something that
>> > isn't going to kill performance of the checksum calculations. A cat
>> > operation on a few files for each checksum for example isn't
>> > appropriate. We may need to do something at the bitbake level as there
>> > is also the issue of checksuming local files such as those in file://
>> > urls and including that in the sstate checksums.
>> >
>> > So whilst I'd love to see fixes for these and they are bugfixes, they're
>> > going to have to be well written patches and its late in cycle for
>> > invasive changes :(.
>>
>> At Mentor, right now, I'm using a ConfigParsed event handler to run
>> lsb_release and inject the DISTRO/DISTRO_VERSION into the metadata,
>> which I then use vardeps to ensure get into the appropriate
>> signatures. Not perfect, but gets the job done.
>
> Are those patches available anywhere we can look at?

Not yet. I'll see about emailing them to the list for review this afternoon.
-- 
Christopher Larson

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[McClintock Matthew-B29882]

On Fri, Mar 23, 2012 at 7:41 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
>> I turned on sstate mirroring for angstrom recently and I'm getting
>> reports of build failures due to missing GLIBC_2.14 symbols:
>>
>> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
>> version `GLIBC_2.14' not found (required by
>> arm-angstrom-linux-gnueabi-gcc)
>>
>> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
>> when it being used on systems with an older c library (e.g. debian).
>> To get rid of this problem there are multiple options, but I think the
>> 2 most obvious are:
>>
>> 1) inject host distroname and distroversion into the checksums
>> 2) build everything against a native libc
>>
> 3) Use an older version on the VM which is the oldest distro you plan to
> build against.
>
>> Would it be appropriate to get 1) into oe-core before the branchpoint?
>
> We've talked about this and its been on the "to fix" list but nobody has
> got around to it. Its hard as we need to come up with something that
> isn't going to kill performance of the checksum calculations. A cat
> operation on a few files for each checksum for example isn't
> appropriate. We may need to do something at the bitbake level as there
> is also the issue of checksuming local files such as those in file://
> urls and including that in the sstate checksums.
>
> So whilst I'd love to see fixes for these and they are bugfixes, they're
> going to have to be well written patches and its late in cycle for
> invasive changes :(.

Can't you just inject a variable into native.bbclass:

LIBC_DEP = `ldd /bin/sh | grep libc`

-M

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2333 bytes --]

On Fri, Mar 23, 2012 at 04:35:12PM +0000, McClintock Matthew-B29882 wrote:
> On Fri, Mar 23, 2012 at 7:41 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
> >> I turned on sstate mirroring for angstrom recently and I'm getting
> >> reports of build failures due to missing GLIBC_2.14 symbols:
> >>
> >> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
> >> version `GLIBC_2.14' not found (required by
> >> arm-angstrom-linux-gnueabi-gcc)
> >>
> >> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
> >> when it being used on systems with an older c library (e.g. debian).
> >> To get rid of this problem there are multiple options, but I think the
> >> 2 most obvious are:
> >>
> >> 1) inject host distroname and distroversion into the checksums
> >> 2) build everything against a native libc
> >>
> > 3) Use an older version on the VM which is the oldest distro you plan to
> > build against.
> >
> >> Would it be appropriate to get 1) into oe-core before the branchpoint?
> >
> > We've talked about this and its been on the "to fix" list but nobody has
> > got around to it. Its hard as we need to come up with something that
> > isn't going to kill performance of the checksum calculations. A cat
> > operation on a few files for each checksum for example isn't
> > appropriate. We may need to do something at the bitbake level as there
> > is also the issue of checksuming local files such as those in file://
> > urls and including that in the sstate checksums.
> >
> > So whilst I'd love to see fixes for these and they are bugfixes, they're
> > going to have to be well written patches and its late in cycle for
> > invasive changes :(.
> 
> Can't you just inject a variable into native.bbclass:
> 
> LIBC_DEP = `ldd /bin/sh | grep libc`

That just catches one of the many variables.  If you want sstate reuse
of the "runs on host" side of things, you need to capture either all of
the libraries we may use, build the published feed linking vs these
libraries (VM or chroot or sysroot games) and the include some way to
say "yes, X is usbale here".  lsb_release and a little bit of mapping
(which can/should be done at the distro level) gets you this easier.

-- 
Tom

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[McClintock Matthew-B29882]

On Fri, Mar 23, 2012 at 12:13 PM, Tom Rini <tom.rini@gmail.com> wrote:
> On Fri, Mar 23, 2012 at 04:35:12PM +0000, McClintock Matthew-B29882 wrote:
>> On Fri, Mar 23, 2012 at 7:41 AM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> > On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
>> >> I turned on sstate mirroring for angstrom recently and I'm getting
>> >> reports of build failures due to missing GLIBC_2.14 symbols:
>> >>
>> >> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
>> >> version `GLIBC_2.14' not found (required by
>> >> arm-angstrom-linux-gnueabi-gcc)
>> >>
>> >> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
>> >> when it being used on systems with an older c library (e.g. debian).
>> >> To get rid of this problem there are multiple options, but I think the
>> >> 2 most obvious are:
>> >>
>> >> 1) inject host distroname and distroversion into the checksums
>> >> 2) build everything against a native libc
>> >>
>> > 3) Use an older version on the VM which is the oldest distro you plan to
>> > build against.
>> >
>> >> Would it be appropriate to get 1) into oe-core before the branchpoint?
>> >
>> > We've talked about this and its been on the "to fix" list but nobody has
>> > got around to it. Its hard as we need to come up with something that
>> > isn't going to kill performance of the checksum calculations. A cat
>> > operation on a few files for each checksum for example isn't
>> > appropriate. We may need to do something at the bitbake level as there
>> > is also the issue of checksuming local files such as those in file://
>> > urls and including that in the sstate checksums.
>> >
>> > So whilst I'd love to see fixes for these and they are bugfixes, they're
>> > going to have to be well written patches and its late in cycle for
>> > invasive changes :(.
>>
>> Can't you just inject a variable into native.bbclass:
>>
>> LIBC_DEP = `ldd /bin/sh | grep libc`
>
> That just catches one of the many variables.  If you want sstate reuse
> of the "runs on host" side of things, you need to capture either all of
> the libraries we may use, build the published feed linking vs these
> libraries (VM or chroot or sysroot games) and the include some way to
> say "yes, X is usbale here".  lsb_release and a little bit of mapping
> (which can/should be done at the distro level) gets you this easier.

Fair enough, but is the only solution to have native cache work per
distro? Maybe we can have compatible distros? These sstate-cache
starts to take up a lot of space if we want to share this via ISOs
etc. I've been testing quite successfully with native cache that is
used on several distros. (We are just building it on an old distro).
Maybe I'm getting completely lucky as well which would not surprise me
either.

Another though is improving the initial user experience by pointing
the SSTATE_MIRROR to the autobuilder so the first builds go so much
faster (experienced users could turn it off). We would need some
policy of keeping around sstate-cache for releases and maybe keeping
the last month of sstate cache from the current builds.

-M

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Chris Larson]

On Sat, Mar 24, 2012 at 11:23 AM, McClintock Matthew-B29882
<B29882@freescale.com> wrote:
> On Fri, Mar 23, 2012 at 12:13 PM, Tom Rini <tom.rini@gmail.com> wrote:
>> On Fri, Mar 23, 2012 at 04:35:12PM +0000, McClintock Matthew-B29882 wrote:
>>> On Fri, Mar 23, 2012 at 7:41 AM, Richard Purdie
>>> <richard.purdie@linuxfoundation.org> wrote:
>>> > On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
>>> >> I turned on sstate mirroring for angstrom recently and I'm getting
>>> >> reports of build failures due to missing GLIBC_2.14 symbols:
>>> >>
>>> >> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
>>> >> version `GLIBC_2.14' not found (required by
>>> >> arm-angstrom-linux-gnueabi-gcc)
>>> >>
>>> >> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
>>> >> when it being used on systems with an older c library (e.g. debian).
>>> >> To get rid of this problem there are multiple options, but I think the
>>> >> 2 most obvious are:
>>> >>
>>> >> 1) inject host distroname and distroversion into the checksums
>>> >> 2) build everything against a native libc
>>> >>
>>> > 3) Use an older version on the VM which is the oldest distro you plan to
>>> > build against.
>>> >
>>> >> Would it be appropriate to get 1) into oe-core before the branchpoint?
>>> >
>>> > We've talked about this and its been on the "to fix" list but nobody has
>>> > got around to it. Its hard as we need to come up with something that
>>> > isn't going to kill performance of the checksum calculations. A cat
>>> > operation on a few files for each checksum for example isn't
>>> > appropriate. We may need to do something at the bitbake level as there
>>> > is also the issue of checksuming local files such as those in file://
>>> > urls and including that in the sstate checksums.
>>> >
>>> > So whilst I'd love to see fixes for these and they are bugfixes, they're
>>> > going to have to be well written patches and its late in cycle for
>>> > invasive changes :(.
>>>
>>> Can't you just inject a variable into native.bbclass:
>>>
>>> LIBC_DEP = `ldd /bin/sh | grep libc`
>>
>> That just catches one of the many variables.  If you want sstate reuse
>> of the "runs on host" side of things, you need to capture either all of
>> the libraries we may use, build the published feed linking vs these
>> libraries (VM or chroot or sysroot games) and the include some way to
>> say "yes, X is usbale here".  lsb_release and a little bit of mapping
>> (which can/should be done at the distro level) gets you this easier.
>
> Fair enough, but is the only solution to have native cache work per
> distro? Maybe we can have compatible distros? These sstate-cache
> starts to take up a lot of space if we want to share this via ISOs
> etc. I've been testing quite successfully with native cache that is
> used on several distros. (We are just building it on an old distro).
> Maybe I'm getting completely lucky as well which would not surprise me
> either.

I really think in the long run maintaining solid chroots/vms for
builds will be more sustainable than trying to pull that off. Then
we'd know the native/cross shared states would be compatible with
others using the standard setup.
-- 
Christopher Larson

Re: sstate reuse for -native, -cross across different host glibc version, how to make it work?

[Khem Raj]

On Sat, Mar 24, 2012 at 11:23 AM, McClintock Matthew-B29882
<B29882@freescale.com> wrote:
> On Fri, Mar 23, 2012 at 12:13 PM, Tom Rini <tom.rini@gmail.com> wrote:
>> On Fri, Mar 23, 2012 at 04:35:12PM +0000, McClintock Matthew-B29882 wrote:
>>> On Fri, Mar 23, 2012 at 7:41 AM, Richard Purdie
>>> <richard.purdie@linuxfoundation.org> wrote:
>>> > On Fri, 2012-03-23 at 13:22 +0100, Koen Kooi wrote:
>>> >> I turned on sstate mirroring for angstrom recently and I'm getting
>>> >> reports of build failures due to missing GLIBC_2.14 symbols:
>>> >>
>>> >> arm-angstrom-linux-gnueabi-gcc: /lib/x86_64-linux-gnu/libc.so.6:
>>> >> version `GLIBC_2.14' not found (required by
>>> >> arm-angstrom-linux-gnueabi-gcc)
>>> >>
>>> >> The sstate tarballs are built on a Fedora16 VM and the breakage occurs
>>> >> when it being used on systems with an older c library (e.g. debian).
>>> >> To get rid of this problem there are multiple options, but I think the
>>> >> 2 most obvious are:
>>> >>
>>> >> 1) inject host distroname and distroversion into the checksums
>>> >> 2) build everything against a native libc
>>> >>
>>> > 3) Use an older version on the VM which is the oldest distro you plan to
>>> > build against.
>>> >
>>> >> Would it be appropriate to get 1) into oe-core before the branchpoint?
>>> >
>>> > We've talked about this and its been on the "to fix" list but nobody has
>>> > got around to it. Its hard as we need to come up with something that
>>> > isn't going to kill performance of the checksum calculations. A cat
>>> > operation on a few files for each checksum for example isn't
>>> > appropriate. We may need to do something at the bitbake level as there
>>> > is also the issue of checksuming local files such as those in file://
>>> > urls and including that in the sstate checksums.
>>> >
>>> > So whilst I'd love to see fixes for these and they are bugfixes, they're
>>> > going to have to be well written patches and its late in cycle for
>>> > invasive changes :(.
>>>
>>> Can't you just inject a variable into native.bbclass:
>>>
>>> LIBC_DEP = `ldd /bin/sh | grep libc`
>>
>> That just catches one of the many variables.  If you want sstate reuse
>> of the "runs on host" side of things, you need to capture either all of
>> the libraries we may use, build the published feed linking vs these
>> libraries (VM or chroot or sysroot games) and the include some way to
>> say "yes, X is usbale here".  lsb_release and a little bit of mapping
>> (which can/should be done at the distro level) gets you this easier.
>
> Fair enough, but is the only solution to have native cache work per
> distro? Maybe we can have compatible distros? These sstate-cache
> starts to take up a lot of space if we want to share this via ISOs
> etc. I've been testing quite successfully with native cache that is
> used on several distros. (We are just building it on an old distro).

thats the key. You can build it on older distros and chances of those
binaries running
on newer distros are far more than building on fairly new distro and
then trying to get the
sstate working on newer distro

> Maybe I'm getting completely lucky as well which would not surprise me
> either.
>
> Another though is improving the initial user experience by pointing
> the SSTATE_MIRROR to the autobuilder

and what if user has older distro than autobuilder. ?

so the first builds go so much
> faster (experienced users could turn it off). We would need some
> policy of keeping around sstate-cache for releases and maybe keeping
> the last month of sstate cache from the current builds.
>
> -M
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core