Issues with parallel build? (Was: gtk+native missing dep ?)

Issues with parallel build? (Was: gtk+native missing dep ?)

[Andrea Adami]

On Sat, May 8, 2010 at 1:12 PM, Andrea Adami <andrea.adami@gmail.com> wrote:
> PARALLEL_MAKE = "-j5"
> #BB_NUMBER_THREADS = "4"
> BB_NUMBER_THREADS = "2"
>
> fixed the things. This time the dependencies were (correctly) built before...
>
> Andrea
>

I repeated the build of console-image and gtk+-native always fails
using 4 threads.
Anyone can confirm?

Regards

Andrea

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Tom Rini]

On Fri, 2010-05-14 at 19:47 +0200, Andrea Adami wrote:
> On Sat, May 8, 2010 at 1:12 PM, Andrea Adami <andrea.adami@gmail.com> wrote:
> > PARALLEL_MAKE = "-j5"
> > #BB_NUMBER_THREADS = "4"
> > BB_NUMBER_THREADS = "2"
> >
> > fixed the things. This time the dependencies were (correctly) built before...
> >
> > Andrea
> >
> 
> I repeated the build of console-image and gtk+-native always fails
> using 4 threads.
> Anyone can confirm?

With DISTRO= ?  I do most of my builds with 4 and haven't seen this.
But with a failure log on gtk+-native, figuring out the missing dep
shouldn't be too bad.

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Andrea Adami]

> With DISTRO= ?  I do most of my builds with 4 and haven't seen this.
> But with a failure log on gtk+-native, figuring out the missing dep
> shouldn't be too bad.

FWIW distro is Angstrom, devel branch.

DEPENDS_virtclass-native = "libpng-native atk-native pango-native
cairo-native libxrender-native libxext-native"

It's not a problem of missing deps, I suppose, because

PARALLEL_MAKE = "-j5"
#BB_NUMBER_THREADS = "4"
BB_NUMBER_THREADS = "2"
fixed the things. This time the dependencies were (correctly) built before.

Short resume:

i repeatdly see (Angstrom) console-image failing *from scratch*
because gtk+native do_compile errors out with:

 Cannot load module
/oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la:
/oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/mo
dules/input/.libs/im-xim.so: undefined symbol: g_realloc_n
| /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la
does not export GTK+ IM module API:
/oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0

g_realloc_n is defined in glib-2.0

I've inspected the workdir and I've noticed  "atk-native"  and
"pango-native" builds were not finished when gt+-native broke.
Both atk-native and pango-native (and cairo-native, which was built)
have glib-2 as explicit dependency, so something seems wrong...

Regards
Andrea

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Tom Rini]

On Fri, 2010-05-14 at 23:21 +0200, Andrea Adami wrote: 
> > With DISTRO= ?  I do most of my builds with 4 and haven't seen this.
> > But with a failure log on gtk+-native, figuring out the missing dep
> > shouldn't be too bad.
> 
> FWIW distro is Angstrom, devel branch.
> 
> DEPENDS_virtclass-native = "libpng-native atk-native pango-native
> cairo-native libxrender-native libxext-native"
> 
> It's not a problem of missing deps, I suppose, because
> 
> PARALLEL_MAKE = "-j5"
> #BB_NUMBER_THREADS = "4"
> BB_NUMBER_THREADS = "2"
> fixed the things. This time the dependencies were (correctly) built before.

Right.  Lower number of threads means less stuff going on at once.  I've
seen a lot of dep issues that only show up once you have a lot of
threads, or have packaged staging (which then marks off a lot of things
in the dep graph).

> Short resume:
> 
> i repeatdly see (Angstrom) console-image failing *from scratch*
> because gtk+native do_compile errors out with:
> 
>  Cannot load module
> /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la:
> /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/mo
> dules/input/.libs/im-xim.so: undefined symbol: g_realloc_n
> | /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la
> does not export GTK+ IM module API:
> /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0
> 
> g_realloc_n is defined in glib-2.0
> 
> I've inspected the workdir and I've noticed  "atk-native"  and
> "pango-native" builds were not finished when gtk+-native broke.
> Both atk-native and pango-native (and cairo-native, which was built)
> have glib-2 as explicit dependency, so something seems wrong...

So, it really sounds like a bitbake bug.  Doing a bitbake -e on gtk
+-native gives:
DEPENDS_virtclass-native="autoconf-native automake-native libtool-native
gnu-config-native coreutils-native libpng-native atk-native pango-native
cairo-native libxrender-native libxext-native"
DEPENDS="autoconf-native automake-native libtool-native
gnu-config-native coreutils-native libpng-native atk-native pango-native
cairo-native libxrender-native libxext-native"

So it's supposed to be waiting on all of that stuff, but it's not.  And
I'm sure RP will need to know what version of bitbake you're using.

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Richard Purdie]

On Fri, 2010-05-14 at 17:04 -0700, Tom Rini wrote:
> On Fri, 2010-05-14 at 23:21 +0200, Andrea Adami wrote: 
> > Short resume:
> > 
> > i repeatdly see (Angstrom) console-image failing *from scratch*
> > because gtk+native do_compile errors out with:
> > 
> >  Cannot load module
> > /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la:
> > /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/mo
> > dules/input/.libs/im-xim.so: undefined symbol: g_realloc_n
> > | /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la
> > does not export GTK+ IM module API:
> > /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0
> > 
> > g_realloc_n is defined in glib-2.0
> > 
> > I've inspected the workdir and I've noticed  "atk-native"  and
> > "pango-native" builds were not finished when gtk+-native broke.
> > Both atk-native and pango-native (and cairo-native, which was built)
> > have glib-2 as explicit dependency, so something seems wrong...
> 
> So, it really sounds like a bitbake bug.  Doing a bitbake -e on gtk
> +-native gives:
> DEPENDS_virtclass-native="autoconf-native automake-native libtool-native
> gnu-config-native coreutils-native libpng-native atk-native pango-native
> cairo-native libxrender-native libxext-native"
> DEPENDS="autoconf-native automake-native libtool-native
> gnu-config-native coreutils-native libpng-native atk-native pango-native
> cairo-native libxrender-native libxext-native"
> 
> So it's supposed to be waiting on all of that stuff, but it's not.  And
> I'm sure RP will need to know what version of bitbake you're using.

There is insufficient evidence here to point at any one thing. I will
say that the threading dependency code has been around for a while and
most bugs like this turn out to be issues in the metadata, not bitbake.

When you say they were "not finished", what do you mean exactly?

Its normal they would still be packaging for example whilst gtk+ builds
since any dependency is on the populate_sysroot task of
pango-native/atk-native.

Cheers,

Richard

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Andrea Adami]

Same issue happens during rebuild from packaged staging


Cannot load module
/oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la:
/oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/.libs/im-xim.so:
undefined symbol: g_realloc_n
| /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la
does not export GTK+ IM module API:
/oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/.libs/im-xim.so:
undefined symbol: g_realloc_n

...

NOTE: package gtk+-native-2.20.0-r8.0: task do_compile: Failed
ERROR: TaskFailed event exception, aborting
ERROR: Build of
virtual:native:/oe/openembedded/recipes/gtk+/gtk+_2.20.0.bb do_compile
failed
...

Andrea

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Andrea Adami]

Bump: issue seems still present :/

Don't know how related/unrelated but on @beagle irc I spotted this bits:

[17:06]	<gst-kaps1>	On Angstrom, gst-inspect doesnt work
[17:07]	<gst-kaps1>	it says g_malloc_n symbol not found ?

[17:10]	<ant_work>	gst-kaps1: interesting. Have you had any build-issue?
[17:11]	<gst-kaps1>	ant_work: no


Andrea

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Denys Dmytriyenko]

On Thu, May 20, 2010 at 12:57:23AM +0200, Andrea Adami wrote:
> Same issue happens during rebuild from packaged staging
> 
> 
> Cannot load module
> /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la:
> /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/.libs/im-xim.so:
> undefined symbol: g_realloc_n
> | /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/im-xim.la
> does not export GTK+ IM module API:
> /oe/build/tmp/work/i686-linux/gtk+-native-2.20.0-r8.0/gtk+-2.20.0/modules/input/.libs/im-xim.so:
> undefined symbol: g_realloc_n

If anybody cares, this issue is still present with the latest glib/gtk+ on 
Angstrom with 4+ threads...

-- 
Denys

> 
> ...
> 
> NOTE: package gtk+-native-2.20.0-r8.0: task do_compile: Failed
> ERROR: TaskFailed event exception, aborting
> ERROR: Build of
> virtual:native:/oe/openembedded/recipes/gtk+/gtk+_2.20.0.bb do_compile
> failed
> ...
> 
> Andrea

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Michael Lippautz]

> On Fri, 2010-05-14 at 19:47 +0200, Andrea Adami wrote:
> > On Sat, May 8, 2010 at 1:12 PM, Andrea Adami <andrea.adami@gmail.com> wrote:
> > I repeated the build of console-image and gtk+-native always fails
> > using 4 threads.
> > Anyone can confirm?

Have seen such issues using 4 threads in other recipes too (can't
remember the names though)

Seems to be some race condition. (Using Angstrom, dunno if relevant)

Regards,
Michael

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Andrea Adami]

I'm using bitbake 1.10

commit bef07c6dc7e3b7b8c995d70ea429d141278b6eb6
Author: Chris Larson <chris_larson@mentor.com>
Date:   Thu Apr 15 08:04:35 2010 -0700

    Bump version from 1.9.0 to 1.10.0

I'd like to add that it seems a bit strange to see all that stuff in
the deps of console-image...but this was already discussed and in this
thread is OT ;)


Regards

Andrea

Re: Issues with parallel build? (Was: gtk+native missing dep ?)

[Leon Woestenberg]

Hello,

On Fri, May 14, 2010 at 7:47 PM, Andrea Adami <andrea.adami@gmail.com> wrote:
> I repeated the build of console-image and gtk+-native always fails
> using 4 threads.
> Anyone can confirm?
>
I can confirm this happens, but not always for me.

Tthus probably racy.

Regards,
-- 
Leon