Issue with (gcc-cross-|glibc-)initial and packaged staging

Issue with (gcc-cross-|glibc-)initial and packaged staging

[Thomas Horsten]

I've spotted a rather annoying interim build failure but I'm not sure
how to best tackle the situation.

For example, gcc-cross-initial and gcc-cross install some of the same
files in staging. Normally this doesn't matter, since gcc-cross
overwrites the earlier ones installed by -initial, but when using
packaged staging and many threads (BBTHREADS=10 for example), it
sometimes happens that the pstaging packages for gcc-cross-initial and
gcc-cross get installed into staging in the wrong order (or at the same
time in two different threads), resulting in all builds after that
failing due to undefined references in the lib files from -initial.

Whether this happens or not is a timing dependent issue, so it's
sometimes hard to reproduce.

Any ideas how to solve this issue? I'm thinking that after gcc-cross
(and glibc-cross) have been built there is no need to depend on the
-initial packages any more. Another option would be to install them with
different paths.

Thomas

Re: Issue with (gcc-cross-|glibc-)initial and packaged staging

[Khem Raj]

On Tue, Mar 9, 2010 at 3:40 AM, Thomas Horsten
<thomas.horsten@citrix.com> wrote:
> I've spotted a rather annoying interim build failure but I'm not sure
> how to best tackle the situation.
>
> For example, gcc-cross-initial and gcc-cross install some of the same
> files in staging. Normally this doesn't matter, since gcc-cross
> overwrites the earlier ones installed by -initial, but when using
> packaged staging and many threads (BBTHREADS=10 for example), it
> sometimes happens that the pstaging packages for gcc-cross-initial and
> gcc-cross get installed into staging in the wrong order (or at the same
> time in two different threads), resulting in all builds after that
> failing due to undefined references in the lib files from -initial.

toolchain build happens in a sequence due to catch 22 it has to get
itself bootstrapped.
In a better world when staging is always packaged then gcc-initial
would create packages
which would be obsoleted by gcc-cross.

>
> Whether this happens or not is a timing dependent issue, so it's
> sometimes hard to reproduce.
>
> Any ideas how to solve this issue? I'm thinking that after gcc-cross
> (and glibc-cross) have been built there is no need to depend on the
> -initial packages any more.

yes but for that you need to have the notion of toolchain is not being
build from scratch and in
other case when you have already bootstrapped it. This state
information needs to be saved somehow
obsoleting package feature in dpkg or bitbake would help in this
regard when it sees that gcc-cross is
already installed it would replace all gcc-cross-initial and
gcc-cross-intermediate references with gcc-cross


 Another option would be to install them with
> different paths.

I think if there

>
> Thomas
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: Issue with (gcc-cross-|glibc-)initial and packaged staging

[Tom Rini]

On Tue, 2010-03-09 at 11:40 +0000, Thomas Horsten wrote:
> I've spotted a rather annoying interim build failure but I'm not sure
> how to best tackle the situation.
> 
> For example, gcc-cross-initial and gcc-cross install some of the same
> files in staging. Normally this doesn't matter, since gcc-cross
> overwrites the earlier ones installed by -initial, but when using
> packaged staging and many threads (BBTHREADS=10 for example), it
> sometimes happens that the pstaging packages for gcc-cross-initial and
> gcc-cross get installed into staging in the wrong order (or at the same
> time in two different threads), resulting in all builds after that
> failing due to undefined references in the lib files from -initial.

So, my guess is that what happens is that when we copy the stamps from
the pstage package over to the live stamps directory to check them,
another thread comes along and sees they're in and we get out of order.
Can you try something like (pastebin'd to avoid patchwork doing
something silly):
http://pastebin.com/rCijESQ8

This is possibly not ideal, but as I comment there, grab the lock for
staging before we copy stamps in (and release it if there's problems).
But this should fix the problem, yes?  If gcc-cross-initial thread is
going and the gcc-cross thread fires, it'll be stuck waiting for
gcc-initial to finish up fully.

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Issue with (gcc-cross-|glibc-)initial and packaged staging

[Denys Dmytriyenko]

On Tue, Mar 09, 2010 at 01:07:29PM -0700, Tom Rini wrote:
> On Tue, 2010-03-09 at 11:40 +0000, Thomas Horsten wrote:
> > I've spotted a rather annoying interim build failure but I'm not sure
> > how to best tackle the situation.
> > 
> > For example, gcc-cross-initial and gcc-cross install some of the same
> > files in staging. Normally this doesn't matter, since gcc-cross
> > overwrites the earlier ones installed by -initial, but when using
> > packaged staging and many threads (BBTHREADS=10 for example), it
> > sometimes happens that the pstaging packages for gcc-cross-initial and
> > gcc-cross get installed into staging in the wrong order (or at the same
> > time in two different threads), resulting in all builds after that
> > failing due to undefined references in the lib files from -initial.
> 
> So, my guess is that what happens is that when we copy the stamps from
> the pstage package over to the live stamps directory to check them,
> another thread comes along and sees they're in and we get out of order.
> Can you try something like (pastebin'd to avoid patchwork doing
> something silly):
> http://pastebin.com/rCijESQ8
> 
> This is possibly not ideal, but as I comment there, grab the lock for
> staging before we copy stamps in (and release it if there's problems).
> But this should fix the problem, yes?  If gcc-cross-initial thread is
> going and the gcc-cross thread fires, it'll be stuck waiting for
> gcc-initial to finish up fully.

As I've previously seen and complained on IRC about a similar issue with many 
BB threads failing in packaged staging, but for different packages, I would 
like to try this patch as well. BTW, looking into the patch it is funny that I 
also tried to fix it by using SYSROOT_LOCK, but I don't think I was doing it 
properly. I'll see if Tom's patch fixes it...

-- 
Denys

Re: Issue with (gcc-cross-|glibc-)initial and packaged staging

[Chris Larson]

On Tue, Mar 9, 2010 at 2:05 PM, Denys Dmytriyenko <denis@denix.org> wrote:

> On Tue, Mar 09, 2010 at 01:07:29PM -0700, Tom Rini wrote:
> > On Tue, 2010-03-09 at 11:40 +0000, Thomas Horsten wrote:
> > > I've spotted a rather annoying interim build failure but I'm not sure
> > > how to best tackle the situation.
> > >
> > > For example, gcc-cross-initial and gcc-cross install some of the same
> > > files in staging. Normally this doesn't matter, since gcc-cross
> > > overwrites the earlier ones installed by -initial, but when using
> > > packaged staging and many threads (BBTHREADS=10 for example), it
> > > sometimes happens that the pstaging packages for gcc-cross-initial and
> > > gcc-cross get installed into staging in the wrong order (or at the same
> > > time in two different threads), resulting in all builds after that
> > > failing due to undefined references in the lib files from -initial.
> >
> > So, my guess is that what happens is that when we copy the stamps from
> > the pstage package over to the live stamps directory to check them,
> > another thread comes along and sees they're in and we get out of order.
> > Can you try something like (pastebin'd to avoid patchwork doing
> > something silly):
> > http://pastebin.com/rCijESQ8
> >
> > This is possibly not ideal, but as I comment there, grab the lock for
> > staging before we copy stamps in (and release it if there's problems).
> > But this should fix the problem, yes?  If gcc-cross-initial thread is
> > going and the gcc-cross thread fires, it'll be stuck waiting for
> > gcc-initial to finish up fully.
>
> As I've previously seen and complained on IRC about a similar issue with
> many
> BB threads failing in packaged staging, but for different packages, I would
> like to try this patch as well. BTW, looking into the patch it is funny
> that I
> also tried to fix it by using SYSROOT_LOCK, but I don't think I was doing
> it
> properly. I'll see if Tom's patch fixes it...


Tom's patch looks like a good idea to me, to ensure the area of code where
the stamps don't match the staging contents is protected.  Note that when
the stamp policy is full or whitelist, this issue does not occur, since the
setscene's of dependent recipes are run prior to the setscene of the recipe
which depends upon them.  I could see an argument for that occurring
regardless of the stamp policy, but this'll do.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics