Update gone terribly wrong

Update gone terribly wrong

[Gary Thomas]

I updated my Poky tree to current master this morning (4c695dbcd12b295a4215422622006f1f58469297)

Now, when I try to build in an extant tree, I get errors like these:
   /home/local/poky-amltd/scripts/bitbake: line 51: 28850 Segmentation fault      (core dumped) PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib 
PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@

So, I removed pseudodone and got
   /home/local/poky-amltd/scripts/bitbake: line 43: 28891 Segmentation fault      (core dumped) bitbake pseudo-native tar-replacement-native -c populate_sysroot

So I figured I'd just restart, so I tried this:
   % mv tmp tmp.BAD
   % mv sstate-cache sstate-mirror
   ... set SSTATE_MIRRORS to use the old cache
   %  tail -n3 conf/local.conf
   SSTATE_MIRRORS ?= "\
   file://.* file:///local/p60_poky/sstate-mirror/"

Sadly, it's rebuilding everything, reusing nothing :-(

Any ideas what went wrong and why the sstate mirror is not working (again)?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Update gone terribly wrong

[Gary Thomas]

On 03/15/2011 07:51 AM, Gary Thomas wrote:
> I updated my Poky tree to current master this morning (4c695dbcd12b295a4215422622006f1f58469297)
>
> Now, when I try to build in an extant tree, I get errors like these:
> /home/local/poky-amltd/scripts/bitbake: line 51: 28850 Segmentation fault (core dumped) PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib
> PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
>
> So, I removed pseudodone and got
> /home/local/poky-amltd/scripts/bitbake: line 43: 28891 Segmentation fault (core dumped) bitbake pseudo-native tar-replacement-native -c populate_sysroot
>
> So I figured I'd just restart, so I tried this:
> % mv tmp tmp.BAD
> % mv sstate-cache sstate-mirror
> ... set SSTATE_MIRRORS to use the old cache
> % tail -n3 conf/local.conf
> SSTATE_MIRRORS ?= "\
> file://.* file:///local/p60_poky/sstate-mirror/"
>
> Sadly, it's rebuilding everything, reusing nothing :-(
>
> Any ideas what went wrong and why the sstate mirror is not working (again)?
>

1/2 an answer to the sstate question; it looks like something has changed the
[md5] hash on the state files.  I ran bitbake with strace to see what files it
was looking at, for example, it tried to look at
   sstate-libxau-native-i686-linux-1.0.6-r0-i686-2-c96b705204df4a0b4b85a9629d15fe53_populate-sysroot.tgz
when the old sstate held
   sstate-libxau-native-i686-linux-1.0.6-r0-i686-2-cb04fd60d897b3222c3c046e571117ff_populate-sysroot.tgz

What controls this hash and why did it change for every package?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Update gone terribly wrong

[Richard Purdie]

On Tue, 2011-03-15 at 07:51 -0600, Gary Thomas wrote:
> I updated my Poky tree to current master this morning (4c695dbcd12b295a4215422622006f1f58469297)
> 
> Now, when I try to build in an extant tree, I get errors like these:
>    /home/local/poky-amltd/scripts/bitbake: line 51: 28850 Segmentation fault      (core dumped) PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib 
> PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
> 
> So, I removed pseudodone and got
>    /home/local/poky-amltd/scripts/bitbake: line 43: 28891 Segmentation fault      (core dumped) bitbake pseudo-native tar-replacement-native -c populate_sysroot

This looks like python itself is segfaulting, even without pseudo loaded
so it could well be something outside our control :/. Which python
version is that?

If you still have the build around it would be interesting to see if
just removing the cache directory fixed that or trying to narrow down
which files are triggering the segfault.

> So I figured I'd just restart, so I tried this:
>    % mv tmp tmp.BAD
>    % mv sstate-cache sstate-mirror
>    ... set SSTATE_MIRRORS to use the old cache
>    %  tail -n3 conf/local.conf
>    SSTATE_MIRRORS ?= "\
>    file://.* file:///local/p60_poky/sstate-mirror/"
> 
> Sadly, it's rebuilding everything, reusing nothing :-(
> 
> Any ideas what went wrong and why the sstate mirror is not working (again)?

Likely, some of the changes in master meant the sstate checksums were
invalidated and it therefore required a rebuild. The sstate files are
only used if the checksums are found to be valid.

Cheers,

Richard

Re: Update gone terribly wrong

[Gary Thomas]

On 03/15/2011 08:03 AM, Richard Purdie wrote:
> On Tue, 2011-03-15 at 07:51 -0600, Gary Thomas wrote:
>> I updated my Poky tree to current master this morning (4c695dbcd12b295a4215422622006f1f58469297)
>>
>> Now, when I try to build in an extant tree, I get errors like these:
>>     /home/local/poky-amltd/scripts/bitbake: line 51: 28850 Segmentation fault      (core dumped) PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib
>> PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
>>
>> So, I removed pseudodone and got
>>     /home/local/poky-amltd/scripts/bitbake: line 43: 28891 Segmentation fault      (core dumped) bitbake pseudo-native tar-replacement-native -c populate_sysroot
>
> This looks like python itself is segfaulting, even without pseudo loaded
> so it could well be something outside our control :/. Which python
> version is that?

Python 2.6.4 on Fedora 13

>
> If you still have the build around it would be interesting to see if
> just removing the cache directory fixed that or trying to narrow down
> which files are triggering the segfault.

When I moved sstate-cache & tmp, the problem went away.  I kept the old trees
and will try and figure out what's crashing.

>
>> So I figured I'd just restart, so I tried this:
>>     % mv tmp tmp.BAD
>>     % mv sstate-cache sstate-mirror
>>     ... set SSTATE_MIRRORS to use the old cache
>>     %  tail -n3 conf/local.conf
>>     SSTATE_MIRRORS ?= "\
>>     file://.* file:///local/p60_poky/sstate-mirror/"
>>
>> Sadly, it's rebuilding everything, reusing nothing :-(
>>
>> Any ideas what went wrong and why the sstate mirror is not working (again)?
>
> Likely, some of the changes in master meant the sstate checksums were
> invalidated and it therefore required a rebuild. The sstate files are
> only used if the checksums are found to be valid.
>
> Cheers,
>
> Richard

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------