Shared state prebuild copy bug report (bug #602)

Shared state prebuild copy bug report (bug #602)

[Scott Garman]

Hello,

I did a build of poky-image-lsb from Poky master today and tried copying 
the shared state prebuilds (sstate-cache) to my laptop. Things looked 
good for a while as it populated my sysroot area from the prebuilds, but 
then it started building autoconf-native and ran into the following 
error during do_compile:

Can't locate Data/Dumper.pm in @INC (@INC contains: ../lib
/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
/toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl/5.8.8
/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl /toastssd/
poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
/toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl .) at
../lib/Autom4te/C4che.pm line 33.

"/toastssd" is a path specific to the original build machine, but does 
not exist on my laptop.

I have reported this as bug #602.

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: Shared state prebuild copy bug report (bug #602)

[Tian, Kevin]

>From: Scott Garman
>Sent: Friday, December 17, 2010 9:39 AM
>To: poky@pokylinux.org
>Subject: [poky] Shared state prebuild copy bug report (bug #602)
>
>Hello,
>
>I did a build of poky-image-lsb from Poky master today and tried copying
>the shared state prebuilds (sstate-cache) to my laptop. Things looked
>good for a while as it populated my sysroot area from the prebuilds, but
>then it started building autoconf-native and ran into the following
>error during do_compile:
>
>Can't locate Data/Dumper.pm in @INC (@INC contains: ../lib
>/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>/toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl/5.8.8
>/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl /toastssd/
>poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>/toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>/toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl .) at
>../lib/Autom4te/C4che.pm line 33.
>
>"/toastssd" is a path specific to the original build machine, but does
>not exist on my laptop.
>
>I have reported this as bug #602.
>
>Scott
>

That's interesting. Does similar error if you try it on same machine?

Thanks
Kevin

Re: Shared state prebuild copy bug report (bug #602)

[Scott Garman]

On 12/16/2010 06:06 PM, Tian, Kevin wrote:
>> From: Scott Garman
>> Sent: Friday, December 17, 2010 9:39 AM
>> To: poky@pokylinux.org
>> Subject: [poky] Shared state prebuild copy bug report (bug #602)
>>
>> Hello,
>>
>> I did a build of poky-image-lsb from Poky master today and tried copying
>> the shared state prebuilds (sstate-cache) to my laptop. Things looked
>> good for a while as it populated my sysroot area from the prebuilds, but
>> then it started building autoconf-native and ran into the following
>> error during do_compile:
>>
>> Can't locate Data/Dumper.pm in @INC (@INC contains: ../lib
>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>> /toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl/5.8.8
>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl /toastssd/
>> poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>> /toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl .) at
>> ../lib/Autom4te/C4che.pm line 33.
>>
>> "/toastssd" is a path specific to the original build machine, but does
>> not exist on my laptop.
>>
>> I have reported this as bug #602.
>>
>> Scott
>>
>
> That's interesting. Does similar error if you try it on same machine?

Do you mean rebuild poky-image-lsb from scratch on my laptop? I would 
imaging that error would not occur.

I'm guessing Perl creates a configuration file somewhere in the sysroot 
which contains full pathnames to it's Perl module directories. That 
config file gets rolled into the prebuild package and cannot be used on 
another build directory.

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: Shared state prebuild copy bug report (bug #602)

[Tian, Kevin]

>From: Garman, Scott A
>Sent: Friday, December 17, 2010 10:16 AM
>
>On 12/16/2010 06:06 PM, Tian, Kevin wrote:
>>> From: Scott Garman
>>> Sent: Friday, December 17, 2010 9:39 AM
>>> To: poky@pokylinux.org
>>> Subject: [poky] Shared state prebuild copy bug report (bug #602)
>>>
>>> Hello,
>>>
>>> I did a build of poky-image-lsb from Poky master today and tried copying
>>> the shared state prebuilds (sstate-cache) to my laptop. Things looked
>>> good for a while as it populated my sysroot area from the prebuilds, but
>>> then it started building autoconf-native and ran into the following
>>> error during do_compile:
>>>
>>> Can't locate Data/Dumper.pm in @INC (@INC contains: ../lib
>>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>>> /toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl/5.8.8
>>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl /toastssd/
>>> poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>>> /toastssd/poky/build/tmp/sysroots/x86_64-linux/usr/lib/perl/5.8.8
>>> /toastssd/poky/build/tmp/sysroots/x86_64-  linux/usr/lib/perl .) at
>>> ../lib/Autom4te/C4che.pm line 33.
>>>
>>> "/toastssd" is a path specific to the original build machine, but does
>>> not exist on my laptop.
>>>
>>> I have reported this as bug #602.
>>>
>>> Scott
>>>
>>
>> That's interesting. Does similar error if you try it on same machine?
>
>Do you mean rebuild poky-image-lsb from scratch on my laptop? I would
>imaging that error would not occur.

No, I mean you have to different builds pointing to same sstate_cache, with
the 1st build filling it first and then the 2nd build to reuse it. Ideally there
should be no hard dir encoded in the sstate package, or else it surely breaks.

>
>I'm guessing Perl creates a configuration file somewhere in the sysroot
>which contains full pathnames to it's Perl module directories. That
>config file gets rolled into the prebuild package and cannot be used on
>another build directory.
>

I think so, just curious how it may be reproduced in my side. I just have one
linux box in front now. :/

Or could you search which file @INC actually points to and its content?

Thanks
Kevin

Re: Shared state prebuild copy bug report (bug #602)

[Scott Garman]

On 12/16/2010 06:19 PM, Tian, Kevin wrote:
>> I'm guessing Perl creates a configuration file somewhere in the sysroot
>> which contains full pathnames to it's Perl module directories. That
>> config file gets rolled into the prebuild package and cannot be used on
>> another build directory.
>>
>
> I think so, just curious how it may be reproduced in my side. I just have one
> linux box in front now. :/
>
> Or could you search which file @INC actually points to and its content?

It appears that @INC is encoded into the perl interpreter itself. cd 
into your native sysroot directory and run ./usr/bin/perl -V

To reproduce this, bitbake an image in one directory. Then, create a 
separate directory somewhere else, and copy your sstate-cache over to 
the second build area. Finally (and this is important), rename your 
original build area.

I'd bet that will make this reproducable.

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: Shared state prebuild copy bug report (bug #602)

[Tian, Kevin]

>From: Garman, Scott A
>Sent: Friday, December 17, 2010 10:30 AM
>
>On 12/16/2010 06:19 PM, Tian, Kevin wrote:
>>> I'm guessing Perl creates a configuration file somewhere in the sysroot
>>> which contains full pathnames to it's Perl module directories. That
>>> config file gets rolled into the prebuild package and cannot be used on
>>> another build directory.
>>>
>>
>> I think so, just curious how it may be reproduced in my side. I just have one
>> linux box in front now. :/
>>
>> Or could you search which file @INC actually points to and its content?
>
>It appears that @INC is encoded into the perl interpreter itself. cd
>into your native sysroot directory and run ./usr/bin/perl -V
>
>To reproduce this, bitbake an image in one directory. Then, create a
>separate directory somewhere else, and copy your sstate-cache over to
>the second build area. Finally (and this is important), rename your
>original build area.
>
>I'd bet that will make this reproducable.
>

ah, I see your point. The rename is very point here as it may hide such issue
if original dir exists.

I'll try whether I can reproduce it.

Thanks
Kevin

Re: Shared state prebuild copy bug report (bug #602)

[Tian, Kevin]

>From: Garman, Scott A
>Sent: Friday, December 17, 2010 10:30 AM
>
>On 12/16/2010 06:19 PM, Tian, Kevin wrote:
>>> I'm guessing Perl creates a configuration file somewhere in the sysroot
>>> which contains full pathnames to it's Perl module directories. That
>>> config file gets rolled into the prebuild package and cannot be used on
>>> another build directory.
>>>
>>
>> I think so, just curious how it may be reproduced in my side. I just have one
>> linux box in front now. :/
>>
>> Or could you search which file @INC actually points to and its content?
>
>It appears that @INC is encoded into the perl interpreter itself. cd
>into your native sysroot directory and run ./usr/bin/perl -V
>
>To reproduce this, bitbake an image in one directory. Then, create a
>separate directory somewhere else, and copy your sstate-cache over to
>the second build area. Finally (and this is important), rename your
>original build area.
>
>I'd bet that will make this reproducable.
>

I assume that you're not using the latest master, correct? I'm asking here as I
encountered other errors earlier than this one due to a 'noexec' change from RP
yesterday, and thus your info would confirm my guess. :-)

Then after revert that commit, now I do reproduce this with same error as yours,
except that it happens on git-native. This is understood as I think all perl related
scripts will hit this issue.

However I haven't found which bit contains that bad link pointing to original build
directory. I checked tmp/sysroot/i686-linux/usr/lib/perl/5.8.8/Config.pm, where
all hard paths are changed to new build directory correctly... :/

Thanks
Kevin

perl binary hard-code path hurts sstate

[Tian, Kevin]

(change the title to better reflect the ongoing thread)

sstate.bbclass solves the absolute path issue by encoding a special pattern
(FIXMESTAGINGDIR) in all sysroot paths captured in plain text files (headers, 
config files, etc.), and then replace them with new path when installing into a 
different directory. This works well until below issue is reported.

The problem is that perl includes absolute path inside its binary:
	xxx/usr/lib/perl/5.8.8/CORE/libperl.so
(Nitin, please correct me whether this is the actual case)

I'm not sure how to work around it to allow a successful sstate reuse.

On the other hand, perhaps we could introduce some blacklist to mark some
packages not sstate-able? I think such case is rare. After rebuilding perl from
scratch in the new build dir, the error disappears and the build is continuing.
I'll see whether there're other similar issues cross the build.

Paul tried two machine before. I guess this issue doesn't expose earlier perhaps
because he is using same build hierarchy on two machines...

Thanks
Kevin

>From: Tian, Kevin
>Sent: Friday, December 17, 2010 12:47 PM
>
>>From: Garman, Scott A
>>Sent: Friday, December 17, 2010 10:30 AM
>>
>>On 12/16/2010 06:19 PM, Tian, Kevin wrote:
>>>> I'm guessing Perl creates a configuration file somewhere in the sysroot
>>>> which contains full pathnames to it's Perl module directories. That
>>>> config file gets rolled into the prebuild package and cannot be used on
>>>> another build directory.
>>>>
>>>
>>> I think so, just curious how it may be reproduced in my side. I just have one
>>> linux box in front now. :/
>>>
>>> Or could you search which file @INC actually points to and its content?
>>
>>It appears that @INC is encoded into the perl interpreter itself. cd
>>into your native sysroot directory and run ./usr/bin/perl -V
>>
>>To reproduce this, bitbake an image in one directory. Then, create a
>>separate directory somewhere else, and copy your sstate-cache over to
>>the second build area. Finally (and this is important), rename your
>>original build area.
>>
>>I'd bet that will make this reproducable.
>>
>
>I assume that you're not using the latest master, correct? I'm asking here as I
>encountered other errors earlier than this one due to a 'noexec' change from RP
>yesterday, and thus your info would confirm my guess. :-)
>
>Then after revert that commit, now I do reproduce this with same error as yours,
>except that it happens on git-native. This is understood as I think all perl related
>scripts will hit this issue.
>
>However I haven't found which bit contains that bad link pointing to original build
>directory. I checked tmp/sysroot/i686-linux/usr/lib/perl/5.8.8/Config.pm, where
>all hard paths are changed to new build directory correctly... :/
>
>Thanks
>Kevin
>_______________________________________________
>poky mailing list
>poky@yoctoproject.org
>https://lists.yoctoproject.org/listinfo/poky

Re: perl binary hard-code path hurts sstate

[Chris Larson]

On Thu, Dec 16, 2010 at 10:29 PM, Tian, Kevin <kevin.tian@intel.com> wrote:
> (change the title to better reflect the ongoing thread)
>
> sstate.bbclass solves the absolute path issue by encoding a special pattern
> (FIXMESTAGINGDIR) in all sysroot paths captured in plain text files (headers,
> config files, etc.), and then replace them with new path when installing into a
> different directory. This works well until below issue is reported.
>
> The problem is that perl includes absolute path inside its binary:
>        xxx/usr/lib/perl/5.8.8/CORE/libperl.so
> (Nitin, please correct me whether this is the actual case)
>
> I'm not sure how to work around it to allow a successful sstate reuse.
>
> On the other hand, perhaps we could introduce some blacklist to mark some
> packages not sstate-able? I think such case is rare. After rebuilding perl from
> scratch in the new build dir, the error disappears and the build is continuing.
> I'll see whether there're other similar issues cross the build.

These sorts of issues aren't new, or fun, sadly.  We've run into many
like this using packaged staging.  We had to use the variable to
disable packaged-staging for exactly this reason, so something similar
for sstate would indeed be useful for this.  Most of us lack the time
to patch upstreams with true package relocatability, unfortunately.
Note that there are cases where projects will obey environment
variables to override the hardcoded default paths, and in those cases
one can generate wrappers for native/cross/nativesdk binary
execution.. not ideal, but gets the job done, and there's a
convenience function for generating them in the upstream OE classes.

In the past, I've used a script that rebuilt every recipe, one by one,
with all of its dependencies prebuilt using a different build path.
Something like that would likely be useful for catching these sorts of
issues with sstate as well.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: perl binary hard-code path hurts sstate

[Richard Purdie]

On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
> These sorts of issues aren't new, or fun, sadly.  We've run into many
> like this using packaged staging.  We had to use the variable to
> disable packaged-staging for exactly this reason, so something similar
> for sstate would indeed be useful for this.

In cases like this I'd suggest adding the encoded path into the checksum
via vardeps. This means the sstate package will be reused when the path
matches but not otherwise.

We can then work on removing the dependencies which I agree is the goal
but we should get them identified first.

Cheers,

Richard

Re: perl binary hard-code path hurts sstate

[Gary Thomas]

On 12/17/2010 06:58 AM, Richard Purdie wrote:
> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>> These sorts of issues aren't new, or fun, sadly.  We've run into many
>> like this using packaged staging.  We had to use the variable to
>> disable packaged-staging for exactly this reason, so something similar
>> for sstate would indeed be useful for this.
>
> In cases like this I'd suggest adding the encoded path into the checksum
> via vardeps. This means the sstate package will be reused when the path
> matches but not otherwise.
>
> We can then work on removing the dependencies which I agree is the goal
> but we should get them identified first.

Sadly, this problem is not confined to perl.  I just tried this experiment:
   * Build Poky on MachineA to some directory $NEW
   * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
     the path for $POKYBASE is the same on both machines but $NEW differs.
     MachineA:$NEW does not exist on MachineB
   * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
     This build is in $NEW2
The process fails for me building the kernel:
   | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
Looking carefully, I can see that the compiler is looking for libgcc.a in
the directory path $NEW that is only on MachineA.

So this is the same problem Paul had, just for the cross compiler
instead of perl.

n.b. Is there a bug # I should use to be tracking this?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Gary Thomas
>Sent: Friday, December 17, 2010 11:09 PM
>
>On 12/17/2010 06:58 AM, Richard Purdie wrote:
>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>> These sorts of issues aren't new, or fun, sadly.  We've run into many
>>> like this using packaged staging.  We had to use the variable to
>>> disable packaged-staging for exactly this reason, so something similar
>>> for sstate would indeed be useful for this.
>>
>> In cases like this I'd suggest adding the encoded path into the checksum
>> via vardeps. This means the sstate package will be reused when the path
>> matches but not otherwise.
>>
>> We can then work on removing the dependencies which I agree is the goal
>> but we should get them identified first.
>
>Sadly, this problem is not confined to perl.  I just tried this experiment:
>   * Build Poky on MachineA to some directory $NEW
>   * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>     the path for $POKYBASE is the same on both machines but $NEW differs.
>     MachineA:$NEW does not exist on MachineB
>   * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>     This build is in $NEW2
>The process fails for me building the kernel:
>   | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>Looking carefully, I can see that the compiler is looking for libgcc.a in
>the directory path $NEW that is only on MachineA.
>
>So this is the same problem Paul had, just for the cross compiler
>instead of perl.
>
>n.b. Is there a bug # I should use to be tracking this?
>

I'll see whether I can reproduce this after RP's new fix about perl-native.

Thanks
Kevin

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Tian, Kevin
>Sent: Tuesday, December 21, 2010 9:10 AM
>
>>From: Gary Thomas
>>Sent: Friday, December 17, 2010 11:09 PM
>>
>>On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>> These sorts of issues aren't new, or fun, sadly.  We've run into many
>>>> like this using packaged staging.  We had to use the variable to
>>>> disable packaged-staging for exactly this reason, so something similar
>>>> for sstate would indeed be useful for this.
>>>
>>> In cases like this I'd suggest adding the encoded path into the checksum
>>> via vardeps. This means the sstate package will be reused when the path
>>> matches but not otherwise.
>>>
>>> We can then work on removing the dependencies which I agree is the goal
>>> but we should get them identified first.
>>
>>Sadly, this problem is not confined to perl.  I just tried this experiment:

Hi, Gary, I can't reproduce in my side, though I use only one machine.

>>   * Build Poky on MachineA to some directory $NEW
>>   * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>     the path for $POKYBASE is the same on both machines but $NEW differs.
>>     MachineA:$NEW does not exist on MachineB
>>   * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>     This build is in $NEW2

compare to your steps, mine is:

* build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
* then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
pointing to ${SSTATE-CACHE}
* move $DIR1 to $DIR1.bak

This way if there's reference to $DIR1, I should be able to observe failure

>>The process fails for me building the kernel:
>>   | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>Looking carefully, I can see that the compiler is looking for libgcc.a in
>>the directory path $NEW that is only on MachineA.

However I didn't see above failure and most sstate packages could be reused
successfully except several ones depending on perl-native...

>>
>>So this is the same problem Paul had, just for the cross compiler
>>instead of perl.
>>
>>n.b. Is there a bug # I should use to be tracking this?
>>
>
>I'll see whether I can reproduce this after RP's new fix about perl-native.

so Gary, could you try latest poky master or verify whether my single-machine steps
may expose same failure in your side? Or is there any step not equivalent between
our cases?

Thanks
Kevin

Re: perl binary hard-code path hurts sstate

[Gary Thomas]

On 12/21/2010 04:51 AM, Tian, Kevin wrote:
>> From: Tian, Kevin
>> Sent: Tuesday, December 21, 2010 9:10 AM
>>
>>> From: Gary Thomas
>>> Sent: Friday, December 17, 2010 11:09 PM
>>>
>>> On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>>> These sorts of issues aren't new, or fun, sadly.  We've run into many
>>>>> like this using packaged staging.  We had to use the variable to
>>>>> disable packaged-staging for exactly this reason, so something similar
>>>>> for sstate would indeed be useful for this.
>>>>
>>>> In cases like this I'd suggest adding the encoded path into the checksum
>>>> via vardeps. This means the sstate package will be reused when the path
>>>> matches but not otherwise.
>>>>
>>>> We can then work on removing the dependencies which I agree is the goal
>>>> but we should get them identified first.
>>>
>>> Sadly, this problem is not confined to perl.  I just tried this experiment:
>
> Hi, Gary, I can't reproduce in my side, though I use only one machine.
>
>>>    * Build Poky on MachineA to some directory $NEW
>>>    * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>>      the path for $POKYBASE is the same on both machines but $NEW differs.
>>>      MachineA:$NEW does not exist on MachineB
>>>    * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>>      This build is in $NEW2
>
> compare to your steps, mine is:
>
> * build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
> * then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
> pointing to ${SSTATE-CACHE}
> * move $DIR1 to $DIR1.bak
>
> This way if there's reference to $DIR1, I should be able to observe failure
>
>>> The process fails for me building the kernel:
>>>    | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>> Looking carefully, I can see that the compiler is looking for libgcc.a in
>>> the directory path $NEW that is only on MachineA.
>
> However I didn't see above failure and most sstate packages could be reused
> successfully except several ones depending on perl-native...
>
>>>
>>> So this is the same problem Paul had, just for the cross compiler
>>> instead of perl.
>>>
>>> n.b. Is there a bug # I should use to be tracking this?
>>>
>>
>> I'll see whether I can reproduce this after RP's new fix about perl-native.
>
> so Gary, could you try latest poky master or verify whether my single-machine steps
> may expose same failure in your side? Or is there any step not equivalent between
> our cases?

Yes, I'll try it now, thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl binary hard-code path hurts sstate

[Gary Thomas]

On 12/21/2010 05:30 AM, Gary Thomas wrote:
> On 12/21/2010 04:51 AM, Tian, Kevin wrote:
>>> From: Tian, Kevin
>>> Sent: Tuesday, December 21, 2010 9:10 AM
>>>
>>>> From: Gary Thomas
>>>> Sent: Friday, December 17, 2010 11:09 PM
>>>>
>>>> On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>>>> These sorts of issues aren't new, or fun, sadly. We've run into many
>>>>>> like this using packaged staging. We had to use the variable to
>>>>>> disable packaged-staging for exactly this reason, so something similar
>>>>>> for sstate would indeed be useful for this.
>>>>>
>>>>> In cases like this I'd suggest adding the encoded path into the checksum
>>>>> via vardeps. This means the sstate package will be reused when the path
>>>>> matches but not otherwise.
>>>>>
>>>>> We can then work on removing the dependencies which I agree is the goal
>>>>> but we should get them identified first.
>>>>
>>>> Sadly, this problem is not confined to perl. I just tried this experiment:
>>
>> Hi, Gary, I can't reproduce in my side, though I use only one machine.
>>
>>>> * Build Poky on MachineA to some directory $NEW
>>>> * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>>> the path for $POKYBASE is the same on both machines but $NEW differs.
>>>> MachineA:$NEW does not exist on MachineB
>>>> * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>>> This build is in $NEW2
>>
>> compare to your steps, mine is:
>>
>> * build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
>> * then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
>> pointing to ${SSTATE-CACHE}
>> * move $DIR1 to $DIR1.bak
>>
>> This way if there's reference to $DIR1, I should be able to observe failure
>>
>>>> The process fails for me building the kernel:
>>>> | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>>> Looking carefully, I can see that the compiler is looking for libgcc.a in
>>>> the directory path $NEW that is only on MachineA.
>>
>> However I didn't see above failure and most sstate packages could be reused
>> successfully except several ones depending on perl-native...
>>
>>>>
>>>> So this is the same problem Paul had, just for the cross compiler
>>>> instead of perl.
>>>>
>>>> n.b. Is there a bug # I should use to be tracking this?
>>>>
>>>
>>> I'll see whether I can reproduce this after RP's new fix about perl-native.
>>
>> so Gary, could you try latest poky master or verify whether my single-machine steps
>> may expose same failure in your side? Or is there any step not equivalent between
>> our cases?
>
> Yes, I'll try it now, thanks
>

Still fails.  There is one step I wasn't clear on - if you
just follow the steps above, you'll succeed and get a working
result, using only the staged (sstate cached) packages.  However,
if you try and build some other package, or in my case I forced
a build of the kernel, it fails.

Amended steps to reproduce this failure:
   * Build Poky on MachineA, results in MachineA:NEW1
   * Copy MachineA:NEW1/sstate-cache to MachineB:NEW1_CACHE
   * Build Poky on MachineB, results in MachineB:NEW2, using
     MachineB:NEW1_CACHE for SSTATE_MIRRORS
   * Disable SSTATE_MIRRORS in conf/local.conf on MachineB:NEW2
   * Build a package from scratch.  In my case, I rebuilt the kernel:
     - bitbake virtual/kernel -c clean
     - rm sstate-cache/sstate-linux-am*
     - bitbake virtual/kernel
     My kernel is based on a local recipe linux-am_X.Y.Z

     Note: I tried to simplify this process by using 'cleanall'
     but ran into a fetch problem (reported separately)

Notes:
   * For my testing, I used a simple package set, e.g. poky-image-minimal
   * My Poky tree is based on today's master 4cd40dbaa8aef7b6b9ff9ce892a576b802f4b1ec
   * I notice that this process now rebuilds some 20 packages from scratch
     on MachineB.  These include the perl-native, etc, that this thread has
     been discussing.  Is this expected?

Thanks for working on this; it's getting close and will be
a big help to my customers.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Gary Thomas [mailto:gary@mlbassoc.com]
>Sent: Tuesday, December 21, 2010 11:14 PM
>
>On 12/21/2010 05:30 AM, Gary Thomas wrote:
>> On 12/21/2010 04:51 AM, Tian, Kevin wrote:
>>>> From: Tian, Kevin
>>>> Sent: Tuesday, December 21, 2010 9:10 AM
>>>>
>>>>> From: Gary Thomas
>>>>> Sent: Friday, December 17, 2010 11:09 PM
>>>>>
>>>>> On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>>>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>>>>> These sorts of issues aren't new, or fun, sadly. We've run into many
>>>>>>> like this using packaged staging. We had to use the variable to
>>>>>>> disable packaged-staging for exactly this reason, so something similar
>>>>>>> for sstate would indeed be useful for this.
>>>>>>
>>>>>> In cases like this I'd suggest adding the encoded path into the checksum
>>>>>> via vardeps. This means the sstate package will be reused when the path
>>>>>> matches but not otherwise.
>>>>>>
>>>>>> We can then work on removing the dependencies which I agree is the goal
>>>>>> but we should get them identified first.
>>>>>
>>>>> Sadly, this problem is not confined to perl. I just tried this experiment:
>>>
>>> Hi, Gary, I can't reproduce in my side, though I use only one machine.
>>>
>>>>> * Build Poky on MachineA to some directory $NEW
>>>>> * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>>>> the path for $POKYBASE is the same on both machines but $NEW differs.
>>>>> MachineA:$NEW does not exist on MachineB
>>>>> * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>>>> This build is in $NEW2
>>>
>>> compare to your steps, mine is:
>>>
>>> * build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
>>> * then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
>>> pointing to ${SSTATE-CACHE}
>>> * move $DIR1 to $DIR1.bak
>>>
>>> This way if there's reference to $DIR1, I should be able to observe failure
>>>
>>>>> The process fails for me building the kernel:
>>>>> | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>>>> Looking carefully, I can see that the compiler is looking for libgcc.a in
>>>>> the directory path $NEW that is only on MachineA.
>>>
>>> However I didn't see above failure and most sstate packages could be reused
>>> successfully except several ones depending on perl-native...
>>>
>>>>>
>>>>> So this is the same problem Paul had, just for the cross compiler
>>>>> instead of perl.
>>>>>
>>>>> n.b. Is there a bug # I should use to be tracking this?
>>>>>
>>>>
>>>> I'll see whether I can reproduce this after RP's new fix about perl-native.
>>>
>>> so Gary, could you try latest poky master or verify whether my single-machine steps
>>> may expose same failure in your side? Or is there any step not equivalent between
>>> our cases?
>>
>> Yes, I'll try it now, thanks
>>
>
>Still fails.  There is one step I wasn't clear on - if you
>just follow the steps above, you'll succeed and get a working
>result, using only the staged (sstate cached) packages.  However,

That's good, which means so far we all reach the same place for the basic sstate usage. :-)

>if you try and build some other package, or in my case I forced
>a build of the kernel, it fails.

OK, this is indeed a different story. But I admit that this scenario is also important
as it's not unusual to have incremental build based on sstate packages.

>
>Amended steps to reproduce this failure:
>   * Build Poky on MachineA, results in MachineA:NEW1
>   * Copy MachineA:NEW1/sstate-cache to MachineB:NEW1_CACHE
>   * Build Poky on MachineB, results in MachineB:NEW2, using
>     MachineB:NEW1_CACHE for SSTATE_MIRRORS
>   * Disable SSTATE_MIRRORS in conf/local.conf on MachineB:NEW2

Any reason to disable SSTATE_MIRRORS? Is it a necessary step to trigger error?

>   * Build a package from scratch.  In my case, I rebuilt the kernel:
>     - bitbake virtual/kernel -c clean
>     - rm sstate-cache/sstate-linux-am*
>     - bitbake virtual/kernel
>     My kernel is based on a local recipe linux-am_X.Y.Z

It's a bit weird why kernel rebuild causes the error you observed. I'd suspect that
there're more recipes rebuilt caused by above action. If there happens to have
gcc bootstrap recipes included, this issue is then known since we know gcc bootstrap
has problem with sstate:

https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html
https://lists.yoctoproject.org/pipermail/poky/2010-November/000499.html

which requires a rework on gcc/libc bootstrap process which is currently in development.

>
>     Note: I tried to simplify this process by using 'cleanall'
>     but ran into a fetch problem (reported separately)

OK, I see that RP fixed this fetch issue in latest master, but I'll first try to use
same commit as yours to make sure mine is on the same page as you.

>
>Notes:
>   * For my testing, I used a simple package set, e.g. poky-image-minimal
>   * My Poky tree is based on today's master
>4cd40dbaa8aef7b6b9ff9ce892a576b802f4b1ec
>   * I notice that this process now rebuilds some 20 packages from scratch
>     on MachineB.  These include the perl-native, etc, that this thread has
>     been discussing.  Is this expected?

yes, there're 10 recipes rebuilt due to perl-native:
https://lists.yoctoproject.org/pipermail/poky/2010-December/001563.html

Your 20 is a bit higher than my previous case, which is worthy of a look. Could you
post them?

>
>Thanks for working on this; it's getting close and will be
>a big help to my customers.
>

Thanks for you detail info. I'll try it again to understand whether it's a new issue
or something we may expect. :-)

Thanks
Kevin

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

Hi, Gary,

I caught another bug when I did "bitbake virtual/kernel" following your instructions,
of course with a slightly change adapting to my single machine case (I ensure my
original build dir is renamed and also the PSTATE_MIRRORS is commented). It's 
about CFLAGS used to compile perf tool, which however is very 'perf' specific which
I'm still looking for a right fix.

I did try other recipes though, both native (gzip-native) and target recipes (db,
makedevs, ...) which all works as expected. This is quite different from your error
which I think is common blocking all target recipes in your side.

So when I'm tackling the 'perf' issue, could you post your logs in case we can
eye-catch some hints there? I think both the 1st build and 2nd failure log are useful
here.

In general as I replied in earlier mail, your error is very likely to be related to the
fact that one or some of gcc recipes got rebuild in your experiment, which need
verification from your log and I don't quite understand yet. :-)

Thanks
Kevin

>From: Tian, Kevin
>Sent: Wednesday, December 22, 2010 9:25 AM
>
>>From: Gary Thomas [mailto:gary@mlbassoc.com]
>>Sent: Tuesday, December 21, 2010 11:14 PM
>>
>>On 12/21/2010 05:30 AM, Gary Thomas wrote:
>>> On 12/21/2010 04:51 AM, Tian, Kevin wrote:
>>>>> From: Tian, Kevin
>>>>> Sent: Tuesday, December 21, 2010 9:10 AM
>>>>>
>>>>>> From: Gary Thomas
>>>>>> Sent: Friday, December 17, 2010 11:09 PM
>>>>>>
>>>>>> On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>>>>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>>>>>> These sorts of issues aren't new, or fun, sadly. We've run into many
>>>>>>>> like this using packaged staging. We had to use the variable to
>>>>>>>> disable packaged-staging for exactly this reason, so something similar
>>>>>>>> for sstate would indeed be useful for this.
>>>>>>>
>>>>>>> In cases like this I'd suggest adding the encoded path into the checksum
>>>>>>> via vardeps. This means the sstate package will be reused when the path
>>>>>>> matches but not otherwise.
>>>>>>>
>>>>>>> We can then work on removing the dependencies which I agree is the goal
>>>>>>> but we should get them identified first.
>>>>>>
>>>>>> Sadly, this problem is not confined to perl. I just tried this experiment:
>>>>
>>>> Hi, Gary, I can't reproduce in my side, though I use only one machine.
>>>>
>>>>>> * Build Poky on MachineA to some directory $NEW
>>>>>> * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>>>>> the path for $POKYBASE is the same on both machines but $NEW differs.
>>>>>> MachineA:$NEW does not exist on MachineB
>>>>>> * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>>>>> This build is in $NEW2
>>>>
>>>> compare to your steps, mine is:
>>>>
>>>> * build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
>>>> * then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
>>>> pointing to ${SSTATE-CACHE}
>>>> * move $DIR1 to $DIR1.bak
>>>>
>>>> This way if there's reference to $DIR1, I should be able to observe failure
>>>>
>>>>>> The process fails for me building the kernel:
>>>>>> | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>>>>> Looking carefully, I can see that the compiler is looking for libgcc.a in
>>>>>> the directory path $NEW that is only on MachineA.
>>>>
>>>> However I didn't see above failure and most sstate packages could be reused
>>>> successfully except several ones depending on perl-native...
>>>>
>>>>>>
>>>>>> So this is the same problem Paul had, just for the cross compiler
>>>>>> instead of perl.
>>>>>>
>>>>>> n.b. Is there a bug # I should use to be tracking this?
>>>>>>
>>>>>
>>>>> I'll see whether I can reproduce this after RP's new fix about perl-native.
>>>>
>>>> so Gary, could you try latest poky master or verify whether my single-machine steps
>>>> may expose same failure in your side? Or is there any step not equivalent between
>>>> our cases?
>>>
>>> Yes, I'll try it now, thanks
>>>
>>
>>Still fails.  There is one step I wasn't clear on - if you
>>just follow the steps above, you'll succeed and get a working
>>result, using only the staged (sstate cached) packages.  However,
>
>That's good, which means so far we all reach the same place for the basic sstate usage. :-)
>
>>if you try and build some other package, or in my case I forced
>>a build of the kernel, it fails.
>
>OK, this is indeed a different story. But I admit that this scenario is also important
>as it's not unusual to have incremental build based on sstate packages.
>
>>
>>Amended steps to reproduce this failure:
>>   * Build Poky on MachineA, results in MachineA:NEW1
>>   * Copy MachineA:NEW1/sstate-cache to MachineB:NEW1_CACHE
>>   * Build Poky on MachineB, results in MachineB:NEW2, using
>>     MachineB:NEW1_CACHE for SSTATE_MIRRORS
>>   * Disable SSTATE_MIRRORS in conf/local.conf on MachineB:NEW2
>
>Any reason to disable SSTATE_MIRRORS? Is it a necessary step to trigger error?
>
>>   * Build a package from scratch.  In my case, I rebuilt the kernel:
>>     - bitbake virtual/kernel -c clean
>>     - rm sstate-cache/sstate-linux-am*
>>     - bitbake virtual/kernel
>>     My kernel is based on a local recipe linux-am_X.Y.Z
>
>It's a bit weird why kernel rebuild causes the error you observed. I'd suspect that
>there're more recipes rebuilt caused by above action. If there happens to have
>gcc bootstrap recipes included, this issue is then known since we know gcc bootstrap
>has problem with sstate:
>
>https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html
>https://lists.yoctoproject.org/pipermail/poky/2010-November/000499.html
>
>which requires a rework on gcc/libc bootstrap process which is currently in development.
>
>>
>>     Note: I tried to simplify this process by using 'cleanall'
>>     but ran into a fetch problem (reported separately)
>
>OK, I see that RP fixed this fetch issue in latest master, but I'll first try to use
>same commit as yours to make sure mine is on the same page as you.
>
>>
>>Notes:
>>   * For my testing, I used a simple package set, e.g. poky-image-minimal
>>   * My Poky tree is based on today's master
>>4cd40dbaa8aef7b6b9ff9ce892a576b802f4b1ec
>>   * I notice that this process now rebuilds some 20 packages from scratch
>>     on MachineB.  These include the perl-native, etc, that this thread has
>>     been discussing.  Is this expected?
>
>yes, there're 10 recipes rebuilt due to perl-native:
>https://lists.yoctoproject.org/pipermail/poky/2010-December/001563.html
>
>Your 20 is a bit higher than my previous case, which is worthy of a look. Could you
>post them?
>
>>
>>Thanks for working on this; it's getting close and will be
>>a big help to my customers.
>>
>
>Thanks for you detail info. I'll try it again to understand whether it's a new issue
>or something we may expect. :-)
>

Re: perl binary hard-code path hurts sstate

[Gary Thomas]

On 12/22/2010 05:07 AM, Tian, Kevin wrote:
> Hi, Gary,
>
> I caught another bug when I did "bitbake virtual/kernel" following your instructions,
> of course with a slightly change adapting to my single machine case (I ensure my
> original build dir is renamed and also the PSTATE_MIRRORS is commented). It's
> about CFLAGS used to compile perf tool, which however is very 'perf' specific which
> I'm still looking for a right fix.
>
> I did try other recipes though, both native (gzip-native) and target recipes (db,
> makedevs, ...) which all works as expected. This is quite different from your error
> which I think is common blocking all target recipes in your side.
>
> So when I'm tackling the 'perf' issue, could you post your logs in case we can
> eye-catch some hints there? I think both the 1st build and 2nd failure log are useful
> here.
>
> In general as I replied in earlier mail, your error is very likely to be related to the
> fact that one or some of gcc recipes got rebuild in your experiment, which need
> verification from your log and I don't quite understand yet. :-)

No GCC packages were rebuilt as part of my experiment.  Note that
I could have just as easily reproduced it by bumping PR in my kernel
recipe on the second machine, just to force it to be rebuilt.

The problem here is that the GCC built on the first machine (used
as basis of the sstate-cached packages) has some hard-coded paths
in it which don't exist on the second machine.  Any package built
using that cross compiler could suffer this error.

It turns out my list of packages rebuilt by perl-native _was_ 10.
My test count was just too stupid:
   % ls -l tmp/work/*/*/temp/log.do_compile* | wc
which sadly included the symlinks.  Leaving the symlinks out, I
got this package set:
    armv7a-poky-linux-gnueabi/libtool-cross-2.4-r0
    my_board-poky-linux-gnueabi/my-console-image-1.0-r0
    i686-linux/git-native-1.7.3.2-r0
    i686-linux/libxml2-native-2.7.7-r2
    i686-linux/libxslt-native-1.1.26-r0
    i686-linux/openssl-native-0.9.8p-r2
    i686-linux/perl-native-5.8.8-r14
    i686-linux/pseudo-native-0.0+git0+c9792c7acdb1cac90549ff08db12a8bf0c53cdcf-r16
    i686-linux/python-native-2.6.6-nk0.0
    i686-linux/rpm-native-5.1.10-r7

>> From: Tian, Kevin
>> Sent: Wednesday, December 22, 2010 9:25 AM
>>
>>> From: Gary Thomas [mailto:gary@mlbassoc.com]
>>> Sent: Tuesday, December 21, 2010 11:14 PM
>>>
>>> On 12/21/2010 05:30 AM, Gary Thomas wrote:
>>>> On 12/21/2010 04:51 AM, Tian, Kevin wrote:
>>>>>> From: Tian, Kevin
>>>>>> Sent: Tuesday, December 21, 2010 9:10 AM
>>>>>>
>>>>>>> From: Gary Thomas
>>>>>>> Sent: Friday, December 17, 2010 11:09 PM
>>>>>>>
>>>>>>> On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>>>>>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>>>>>>> These sorts of issues aren't new, or fun, sadly. We've run into many
>>>>>>>>> like this using packaged staging. We had to use the variable to
>>>>>>>>> disable packaged-staging for exactly this reason, so something similar
>>>>>>>>> for sstate would indeed be useful for this.
>>>>>>>>
>>>>>>>> In cases like this I'd suggest adding the encoded path into the checksum
>>>>>>>> via vardeps. This means the sstate package will be reused when the path
>>>>>>>> matches but not otherwise.
>>>>>>>>
>>>>>>>> We can then work on removing the dependencies which I agree is the goal
>>>>>>>> but we should get them identified first.
>>>>>>>
>>>>>>> Sadly, this problem is not confined to perl. I just tried this experiment:
>>>>>
>>>>> Hi, Gary, I can't reproduce in my side, though I use only one machine.
>>>>>
>>>>>>> * Build Poky on MachineA to some directory $NEW
>>>>>>> * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>>>>>> the path for $POKYBASE is the same on both machines but $NEW differs.
>>>>>>> MachineA:$NEW does not exist on MachineB
>>>>>>> * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>>>>>> This build is in $NEW2
>>>>>
>>>>> compare to your steps, mine is:
>>>>>
>>>>> * build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
>>>>> * then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
>>>>> pointing to ${SSTATE-CACHE}
>>>>> * move $DIR1 to $DIR1.bak
>>>>>
>>>>> This way if there's reference to $DIR1, I should be able to observe failure
>>>>>
>>>>>>> The process fails for me building the kernel:
>>>>>>> | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>>>>>> Looking carefully, I can see that the compiler is looking for libgcc.a in
>>>>>>> the directory path $NEW that is only on MachineA.
>>>>>
>>>>> However I didn't see above failure and most sstate packages could be reused
>>>>> successfully except several ones depending on perl-native...
>>>>>
>>>>>>>
>>>>>>> So this is the same problem Paul had, just for the cross compiler
>>>>>>> instead of perl.
>>>>>>>
>>>>>>> n.b. Is there a bug # I should use to be tracking this?
>>>>>>>
>>>>>>
>>>>>> I'll see whether I can reproduce this after RP's new fix about perl-native.
>>>>>
>>>>> so Gary, could you try latest poky master or verify whether my single-machine steps
>>>>> may expose same failure in your side? Or is there any step not equivalent between
>>>>> our cases?
>>>>
>>>> Yes, I'll try it now, thanks
>>>>
>>>
>>> Still fails.  There is one step I wasn't clear on - if you
>>> just follow the steps above, you'll succeed and get a working
>>> result, using only the staged (sstate cached) packages.  However,
>>
>> That's good, which means so far we all reach the same place for the basic sstate usage. :-)
>>
>>> if you try and build some other package, or in my case I forced
>>> a build of the kernel, it fails.
>>
>> OK, this is indeed a different story. But I admit that this scenario is also important
>> as it's not unusual to have incremental build based on sstate packages.
>>
>>>
>>> Amended steps to reproduce this failure:
>>>    * Build Poky on MachineA, results in MachineA:NEW1
>>>    * Copy MachineA:NEW1/sstate-cache to MachineB:NEW1_CACHE
>>>    * Build Poky on MachineB, results in MachineB:NEW2, using
>>>      MachineB:NEW1_CACHE for SSTATE_MIRRORS
>>>    * Disable SSTATE_MIRRORS in conf/local.conf on MachineB:NEW2
>>
>> Any reason to disable SSTATE_MIRRORS? Is it a necessary step to trigger error?
>>
>>>    * Build a package from scratch.  In my case, I rebuilt the kernel:
>>>      - bitbake virtual/kernel -c clean
>>>      - rm sstate-cache/sstate-linux-am*
>>>      - bitbake virtual/kernel
>>>      My kernel is based on a local recipe linux-am_X.Y.Z
>>
>> It's a bit weird why kernel rebuild causes the error you observed. I'd suspect that
>> there're more recipes rebuilt caused by above action. If there happens to have
>> gcc bootstrap recipes included, this issue is then known since we know gcc bootstrap
>> has problem with sstate:
>>
>> https://lists.yoctoproject.org/pipermail/poky/2010-November/000270.html
>> https://lists.yoctoproject.org/pipermail/poky/2010-November/000499.html
>>
>> which requires a rework on gcc/libc bootstrap process which is currently in development.
>>
>>>
>>>      Note: I tried to simplify this process by using 'cleanall'
>>>      but ran into a fetch problem (reported separately)
>>
>> OK, I see that RP fixed this fetch issue in latest master, but I'll first try to use
>> same commit as yours to make sure mine is on the same page as you.
>>
>>>
>>> Notes:
>>>    * For my testing, I used a simple package set, e.g. poky-image-minimal
>>>    * My Poky tree is based on today's master
>>> 4cd40dbaa8aef7b6b9ff9ce892a576b802f4b1ec
>>>    * I notice that this process now rebuilds some 20 packages from scratch
>>>      on MachineB.  These include the perl-native, etc, that this thread has
>>>      been discussing.  Is this expected?
>>
>> yes, there're 10 recipes rebuilt due to perl-native:
>> https://lists.yoctoproject.org/pipermail/poky/2010-December/001563.html
>>
>> Your 20 is a bit higher than my previous case, which is worthy of a look. Could you
>> post them?
>>
>>>
>>> Thanks for working on this; it's getting close and will be
>>> a big help to my customers.
>>>
>>
>> Thanks for you detail info. I'll try it again to understand whether it's a new issue
>> or something we may expect. :-)
>>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Gary Thomas [mailto:gary@mlbassoc.com]
>Sent: Wednesday, December 22, 2010 8:35 PM
>
>On 12/22/2010 05:07 AM, Tian, Kevin wrote:
>> Hi, Gary,
>>
>> I caught another bug when I did "bitbake virtual/kernel" following your instructions,
>> of course with a slightly change adapting to my single machine case (I ensure my
>> original build dir is renamed and also the PSTATE_MIRRORS is commented). It's
>> about CFLAGS used to compile perf tool, which however is very 'perf' specific which
>> I'm still looking for a right fix.
>>
>> I did try other recipes though, both native (gzip-native) and target recipes (db,
>> makedevs, ...) which all works as expected. This is quite different from your error
>> which I think is common blocking all target recipes in your side.
>>
>> So when I'm tackling the 'perf' issue, could you post your logs in case we can
>> eye-catch some hints there? I think both the 1st build and 2nd failure log are useful
>> here.
>>
>> In general as I replied in earlier mail, your error is very likely to be related to the
>> fact that one or some of gcc recipes got rebuild in your experiment, which need
>> verification from your log and I don't quite understand yet. :-)
>
>No GCC packages were rebuilt as part of my experiment.  Note that
>I could have just as easily reproduced it by bumping PR in my kernel
>recipe on the second machine, just to force it to be rebuilt.
>
>The problem here is that the GCC built on the first machine (used
>as basis of the sstate-cached packages) has some hard-coded paths
>in it which don't exist on the second machine.  Any package built
>using that cross compiler could suffer this error.

GCC always contains some hard links to original build dir, through --with-sysroot
or --with-build-sysroot as the default config. Poky encapsulates ${CC} with 
"--sysroot=${STAGING_DIR_TARGET} to override build config to actual target
sysroot, which should also work in 2nd build dir if it's set correctly. In my failure,
the Makefile of kernel perf tool overrides ${CC} directly which makes --sysroot
abandoned and then failed to search some header files.

so your case looks weird to me. Just double confirm, any recipe in your
2nd build now will fail following same steps, right?

>
>It turns out my list of packages rebuilt by perl-native _was_ 10.

that's same list as what I observed so far.

>My test count was just too stupid:
>   % ls -l tmp/work/*/*/temp/log.do_compile* | wc
>which sadly included the symlinks.  Leaving the symlinks out, I
>got this package set:
>    armv7a-poky-linux-gnueabi/libtool-cross-2.4-r0
>    my_board-poky-linux-gnueabi/my-console-image-1.0-r0
>    i686-linux/git-native-1.7.3.2-r0
>    i686-linux/libxml2-native-2.7.7-r2
>    i686-linux/libxslt-native-1.1.26-r0
>    i686-linux/openssl-native-0.9.8p-r2
>    i686-linux/perl-native-5.8.8-r14
>    i686-linux/pseudo-native-0.0+git0+c9792c7acdb1cac90549ff08db12a8bf0c53cdcf-r16
>    i686-linux/python-native-2.6.6-nk0.0
>    i686-linux/rpm-native-5.1.10-r7

In the build where you see above kernel failure, is the kernel the only one
being rebuilt, or are there any other packages rebuild triggered implicitly?

I'll try to find a 2nd machine tomorrow...

Thanks
Kevin

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Tian, Kevin
>Sent: Wednesday, December 22, 2010 9:39 PM
>
>>From: Gary Thomas [mailto:gary@mlbassoc.com]
>>Sent: Wednesday, December 22, 2010 8:35 PM
>>
>>On 12/22/2010 05:07 AM, Tian, Kevin wrote:
>>> Hi, Gary,
>>>
>>> I caught another bug when I did "bitbake virtual/kernel" following your instructions,
>>> of course with a slightly change adapting to my single machine case (I ensure my
>>> original build dir is renamed and also the PSTATE_MIRRORS is commented). It's
>>> about CFLAGS used to compile perf tool, which however is very 'perf' specific which
>>> I'm still looking for a right fix.
>>>
>>> I did try other recipes though, both native (gzip-native) and target recipes (db,
>>> makedevs, ...) which all works as expected. This is quite different from your error
>>> which I think is common blocking all target recipes in your side.
>>>
>>> So when I'm tackling the 'perf' issue, could you post your logs in case we can
>>> eye-catch some hints there? I think both the 1st build and 2nd failure log are useful
>>> here.
>>>
>>> In general as I replied in earlier mail, your error is very likely to be related to the
>>> fact that one or some of gcc recipes got rebuild in your experiment, which need
>>> verification from your log and I don't quite understand yet. :-)
>>
>>No GCC packages were rebuilt as part of my experiment.  Note that
>>I could have just as easily reproduced it by bumping PR in my kernel
>>recipe on the second machine, just to force it to be rebuilt.
>>
>>The problem here is that the GCC built on the first machine (used
>>as basis of the sstate-cached packages) has some hard-coded paths
>>in it which don't exist on the second machine.  Any package built
>>using that cross compiler could suffer this error.
>
>GCC always contains some hard links to original build dir, through --with-sysroot
>or --with-build-sysroot as the default config. Poky encapsulates ${CC} with
>"--sysroot=${STAGING_DIR_TARGET} to override build config to actual target
>sysroot, which should also work in 2nd build dir if it's set correctly. In my failure,
>the Makefile of kernel perf tool overrides ${CC} directly which makes --sysroot
>abandoned and then failed to search some header files.
>
>so your case looks weird to me. Just double confirm, any recipe in your
>2nd build now will fail following same steps, right?
>
>>
>>It turns out my list of packages rebuilt by perl-native _was_ 10.
>
>that's same list as what I observed so far.
>
>>My test count was just too stupid:
>>   % ls -l tmp/work/*/*/temp/log.do_compile* | wc
>>which sadly included the symlinks.  Leaving the symlinks out, I
>>got this package set:
>>    armv7a-poky-linux-gnueabi/libtool-cross-2.4-r0
>>    my_board-poky-linux-gnueabi/my-console-image-1.0-r0
>>    i686-linux/git-native-1.7.3.2-r0
>>    i686-linux/libxml2-native-2.7.7-r2
>>    i686-linux/libxslt-native-1.1.26-r0
>>    i686-linux/openssl-native-0.9.8p-r2
>>    i686-linux/perl-native-5.8.8-r14
>>
>i686-linux/pseudo-native-0.0+git0+c9792c7acdb1cac90549ff08db12a8bf0c53cdcf-r16
>>    i686-linux/python-native-2.6.6-nk0.0
>>    i686-linux/rpm-native-5.1.10-r7
>
>In the build where you see above kernel failure, is the kernel the only one
>being rebuilt, or are there any other packages rebuild triggered implicitly?
>
>I'll try to find a 2nd machine tomorrow...
>

In the meantime, could you also help do an experiment on a single machine? You
just need to make sure that the 1st build dir is renamed before the 2nd build
starts. Then the rest steps could be same as your original instructions. This should
help narrow this issue down further. :-)

Thanks
Kevin

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Gary Thomas [mailto:gary@mlbassoc.com]
>Sent: Tuesday, December 21, 2010 11:14 PM
>
>On 12/21/2010 05:30 AM, Gary Thomas wrote:
>> On 12/21/2010 04:51 AM, Tian, Kevin wrote:
>>>> From: Tian, Kevin
>>>> Sent: Tuesday, December 21, 2010 9:10 AM
>>>>
>>>>> From: Gary Thomas
>>>>> Sent: Friday, December 17, 2010 11:09 PM
>>>>>
>>>>> On 12/17/2010 06:58 AM, Richard Purdie wrote:
>>>>>> On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>>>>>>> These sorts of issues aren't new, or fun, sadly. We've run into many
>>>>>>> like this using packaged staging. We had to use the variable to
>>>>>>> disable packaged-staging for exactly this reason, so something similar
>>>>>>> for sstate would indeed be useful for this.
>>>>>>
>>>>>> In cases like this I'd suggest adding the encoded path into the checksum
>>>>>> via vardeps. This means the sstate package will be reused when the path
>>>>>> matches but not otherwise.
>>>>>>
>>>>>> We can then work on removing the dependencies which I agree is the goal
>>>>>> but we should get them identified first.
>>>>>
>>>>> Sadly, this problem is not confined to perl. I just tried this experiment:
>>>
>>> Hi, Gary, I can't reproduce in my side, though I use only one machine.
>>>
>>>>> * Build Poky on MachineA to some directory $NEW
>>>>> * Copy Poky ($POKYBASE) and $NEW/sstate-cache to MachineB, in this case
>>>>> the path for $POKYBASE is the same on both machines but $NEW differs.
>>>>> MachineA:$NEW does not exist on MachineB
>>>>> * Try to build on Machine B, pointing SSTATE_MIRRORS to $NEW/sstate-cache
>>>>> This build is in $NEW2
>>>
>>> compare to your steps, mine is:
>>>
>>> * build Poky in $DIR1, with sstate generated in ${SSTATE-CACHE} (a global one)
>>> * then with same $POKYBASE, create a new build $DIR2, with SSTATE_MIRRORS
>>> pointing to ${SSTATE-CACHE}
>>> * move $DIR1 to $DIR1.bak
>>>
>>> This way if there's reference to $DIR1, I should be able to observe failure
>>>
>>>>> The process fails for me building the kernel:
>>>>> | arm-poky-linux-gnueabi-ld: libgcc.a: No such file: No such file or directory
>>>>> Looking carefully, I can see that the compiler is looking for libgcc.a in
>>>>> the directory path $NEW that is only on MachineA.
>>>
>>> However I didn't see above failure and most sstate packages could be reused
>>> successfully except several ones depending on perl-native...
>>>
>>>>>
>>>>> So this is the same problem Paul had, just for the cross compiler
>>>>> instead of perl.
>>>>>
>>>>> n.b. Is there a bug # I should use to be tracking this?
>>>>>
>>>>
>>>> I'll see whether I can reproduce this after RP's new fix about perl-native.
>>>
>>> so Gary, could you try latest poky master or verify whether my single-machine steps
>>> may expose same failure in your side? Or is there any step not equivalent between
>>> our cases?
>>
>> Yes, I'll try it now, thanks
>>
>
>Still fails.  There is one step I wasn't clear on - if you
>just follow the steps above, you'll succeed and get a working
>result, using only the staged (sstate cached) packages.  However,
>if you try and build some other package, or in my case I forced
>a build of the kernel, it fails.
>
>Amended steps to reproduce this failure:
>   * Build Poky on MachineA, results in MachineA:NEW1
>   * Copy MachineA:NEW1/sstate-cache to MachineB:NEW1_CACHE
>   * Build Poky on MachineB, results in MachineB:NEW2, using
>     MachineB:NEW1_CACHE for SSTATE_MIRRORS
>   * Disable SSTATE_MIRRORS in conf/local.conf on MachineB:NEW2
>   * Build a package from scratch.  In my case, I rebuilt the kernel:
>     - bitbake virtual/kernel -c clean
>     - rm sstate-cache/sstate-linux-am*
>     - bitbake virtual/kernel
>     My kernel is based on a local recipe linux-am_X.Y.Z
>
>     Note: I tried to simplify this process by using 'cleanall'
>     but ran into a fetch problem (reported separately)

I finally found the 2nd machine and followed the exact steps as above based on
same commit you provided. Again, I can't reproduce your problem, and everything
works fine by trying fresh build for several recipes including kernel.

I'm not sure how much you have added in your own layer (like your own kernel
here), and it'd be helpful if you could try a pure poky master (and if possible a
single machine test to ease future replication).

BTW, my MachineA is Ubuntu9.04 and MachineB is FC13. Both are 32bit.

In your specific case, could you double-confirm whether libgcc.a exists in your
MachineB, and provide some info how you identified MachineA is referenced
for that library?

Thanks
Kevin

>
>Notes:
>   * For my testing, I used a simple package set, e.g. poky-image-minimal
>   * My Poky tree is based on today's master
>4cd40dbaa8aef7b6b9ff9ce892a576b802f4b1ec
>   * I notice that this process now rebuilds some 20 packages from scratch
>     on MachineB.  These include the perl-native, etc, that this thread has
>     been discussing.  Is this expected?
>
>Thanks for working on this; it's getting close and will be
>a big help to my customers.
>

Re: perl binary hard-code path hurts sstate

[Tian, Kevin]

>From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
>Sent: Friday, December 17, 2010 9:59 PM
>
>On Fri, 2010-12-17 at 06:49 -0700, Chris Larson wrote:
>> These sorts of issues aren't new, or fun, sadly.  We've run into many
>> like this using packaged staging.  We had to use the variable to
>> disable packaged-staging for exactly this reason, so something similar
>> for sstate would indeed be useful for this.
>
>In cases like this I'd suggest adding the encoded path into the checksum
>via vardeps. This means the sstate package will be reused when the path
>matches but not otherwise.
>
>We can then work on removing the dependencies which I agree is the goal
>but we should get them identified first.
>

with RP's new fix, I can verify this issue at least solved for now. I'll think about
how to improve it later.

BTW, with this change, now there're 10 native recipes requiring rebuild consequently:

perl-native -> openssl-native -> git-native -> pseudo-native -> libtool-cross
                                 |              |> prelink-native
                                 |              |> kern-tools-native
                                 |> python-native -> libxml2-native ->libxslt-native

Thanks
Kevin

Re: Shared state prebuild copy bug report (bug #602)

[Scott Garman]

On 12/16/2010 08:47 PM, Tian, Kevin wrote:
>> It appears that @INC is encoded into the perl interpreter itself. cd
>> into your native sysroot directory and run ./usr/bin/perl -V
>>
>> To reproduce this, bitbake an image in one directory. Then, create a
>> separate directory somewhere else, and copy your sstate-cache over to
>> the second build area. Finally (and this is important), rename your
>> original build area.
>>
>> I'd bet that will make this reproducable.
>>
>
> I assume that you're not using the latest master, correct? I'm asking here as I
> encountered other errors earlier than this one due to a 'noexec' change from RP
> yesterday, and thus your info would confirm my guess. :-)

No, I managed to do this test with today's commit 
(ac4364698618a541624d1a00f27b638815e5a3f5).

> Then after revert that commit, now I do reproduce this with same error as yours,
> except that it happens on git-native. This is understood as I think all perl related
> scripts will hit this issue.

This will impact a ton of recipes as perl is used heavily by autotools.

> However I haven't found which bit contains that bad link pointing to original build
> directory. I checked tmp/sysroot/i686-linux/usr/lib/perl/5.8.8/Config.pm, where
> all hard paths are changed to new build directory correctly... :/

Thanks for looking into this.

Scott

-- 
Scott Garman
Embedded Linux Distro Engineer - Yocto Project

Re: Shared state prebuild copy bug report (bug #602)

[Richard Purdie]

On Thu, 2010-12-16 at 21:43 -0800, Scott Garman wrote:
> On 12/16/2010 08:47 PM, Tian, Kevin wrote:
> >
> > I assume that you're not using the latest master, correct? I'm asking here as I
> > encountered other errors earlier than this one due to a 'noexec' change from RP
> > yesterday, and thus your info would confirm my guess. :-)
> 
> No, I managed to do this test with today's commit 
> (ac4364698618a541624d1a00f27b638815e5a3f5).
> 
> > Then after revert that commit, now I do reproduce this with same error as yours,
> > except that it happens on git-native. This is understood as I think all perl related
> > scripts will hit this issue.
> 
> This will impact a ton of recipes as perl is used heavily by autotools.
> 
> > However I haven't found which bit contains that bad link pointing to original build
> > directory. I checked tmp/sysroot/i686-linux/usr/lib/perl/5.8.8/Config.pm, where
> > all hard paths are changed to new build directory correctly... :/
> 
> Thanks for looking into this.

I've committed a workaround for this in the form of:

http://git.pokylinux.org/cgit.cgi/poky/commit/?h=master&id=9975c00c1a6a9ae1c39678dec0986f4c62418c96

which means perl-native will always be rebuilt but at least builds
should then work.

I've left bug #602 open as a hint to investigate a proper fix for this,
maybe by setting an environment variable and/or a wrapper script.

At present the aim is to get the sstate packages working, then we can
look at details like this.

Cheers,

Richard