bitbake freeze :(

bitbake freeze :(

[Steffen Sledz]

Since a few days i see this crazy behavior. Sometimes bitbake freezes (everytime at the same stages if it freezes) when building a target (e.g. with "bitbake drapptempl"). top tells me that a "make all" process is running and consuming all resources of one cpu. The last lines of an strace show always something like this:

...
5909  stat64(".deps/drapptempl-main.Po", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
5909  lstat64(".deps/drapptempl-main.Tpo", {st_mode=S_IFREG|0644, st_size=32496, ...}) = 0
5909  lstat64(".deps/drapptempl-main.Po", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
5909  rename(".deps/drapptempl-main.Tpo", ".deps/drapptempl-main.Po") = 0
5909  close(0)                          = 0
5909  close(1)                          = 0
5909  close(2)                          = 0
5909  exit_group(0)                     = ?
5898  <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 5909
5898  --- SIGCHLD (Child exited) @ 0 (0) ---
5898  sigreturn()                       = ? (mask now [])
5898  rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU], NULL, 8) = 0
5898  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
5898  stat64("drapptempl-main.o", {st_mode=S_IFREG|0644, st_size=107848, ...}) = 0

The recipe for the freezing task looks like this:

 DESCRIPTION = "HydraIP Linux Application Template"
 SECTION = "console/utils"
 PRIORITY = "optional"
 DEPENDS = "libdrhip"

 #SRCREV = "1390"
 SRCREV = "${AUTOREV}" 
 PR = "r1"

 SRC_URI="svn://something.else/repos/HydraIP/trunk/LINUX;module=drapptempl;proto=https"

 PV = "svnr${SRCREV}"
 S = "${WORKDIR}/drapptempl"

 EXTRA_OECONF = "--with-libdrhip-path=${STAGING_DIR_TARGET}"

 inherit autotools

Any idea what's going wrong here? Or how to isolate the problem?

Steffen

Re: bitbake freeze :(

[Khem Raj]

On Tue, Sep 22, 2009 at 6:22 AM, Steffen Sledz <sledz@dresearch.de> wrote:
> Since a few days i see this crazy behavior. Sometimes bitbake freezes (everytime at the same stages if it freezes) when building a target (e.g. with "bitbake drapptempl"). top tells me that a "make all" process is running and consuming all resources of one cpu. The last lines of an strace show always something like this:
>
> ...
> 5909  stat64(".deps/drapptempl-main.Po", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
> 5909  lstat64(".deps/drapptempl-main.Tpo", {st_mode=S_IFREG|0644, st_size=32496, ...}) = 0
> 5909  lstat64(".deps/drapptempl-main.Po", {st_mode=S_IFREG|0644, st_size=8, ...}) = 0
> 5909  rename(".deps/drapptempl-main.Tpo", ".deps/drapptempl-main.Po") = 0
> 5909  close(0)                          = 0
> 5909  close(1)                          = 0
> 5909  close(2)                          = 0
> 5909  exit_group(0)                     = ?
> 5898  <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 5909
> 5898  --- SIGCHLD (Child exited) @ 0 (0) ---
> 5898  sigreturn()                       = ? (mask now [])
> 5898  rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT TERM XCPU], NULL, 8) = 0
> 5898  rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
> 5898  stat64("drapptempl-main.o", {st_mode=S_IFREG|0644, st_size=107848, ...}) = 0
>
> The recipe for the freezing task looks like this:
>
>  DESCRIPTION = "HydraIP Linux Application Template"
>  SECTION = "console/utils"
>  PRIORITY = "optional"
>  DEPENDS = "libdrhip"
>
>  #SRCREV = "1390"
>  SRCREV = "${AUTOREV}"
>  PR = "r1"
>
>  SRC_URI="svn://something.else/repos/HydraIP/trunk/LINUX;module=drapptempl;proto=https"
>
>  PV = "svnr${SRCREV}"
>  S = "${WORKDIR}/drapptempl"
>
>  EXTRA_OECONF = "--with-libdrhip-path=${STAGING_DIR_TARGET}"
>
>  inherit autotools
>
> Any idea what's going wrong here? Or how to isolate the problem?

you could try couple of things. Firstly try to isolate if its a
bitbake problem or not
likely its not but still make sure. You can go into the workdir and
run the task script
run.do_<task> whatever it is stuck at. If it succeeeds then you know
its a problem with
bitbake if not then the build mechanism for this particular package
has problem. You might
try to disable parallel build i.e. use -j 1 to make sure that its not
some parallel build
issue.

>
> Steffen
>
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: bitbake freeze :(

[Steffen Sledz]

Khem Raj wrote:
> you could try couple of things. Firstly try to isolate if its a
> bitbake problem or not
> likely its not but still make sure. You can go into the workdir and
> run the task script
> run.do_<task> whatever it is stuck at. If it succeeeds then you know
> its a problem with
> bitbake if not then the build mechanism for this particular package
> has problem. You might
> try to disable parallel build i.e. use -j 1 to make sure that its not
> some parallel build
> issue.

I made some tests on this.

First result was that if calling

 > bitbake drapptempl -c clean
 > bitbake drapptempl
 Ctrl-C
 > bitbake drapptempl

the first build request freezes in do_compile stage, the second one 
succeeds (seen in 5 of 5 test runs). Also calling run.do_compile
succeeds everytime if tried after the first freeze.

After that i checked the parallel build options in my local.conf.
The tests above were made with

 PARALLEL_MAKE = "-j 4"
 BB_NUMBER_THREADS = "4"

I tried to set the BB_NUMBER_THREADS to 1 but the problem was still
existing.

After that i tried to set the PARALLEL_MAKE to "-j 1" and no the
recipe was build fine.  :)

So it seems that there is a problem with parallel make. But it only
occurs in a few of our recipes. Therefor i believe that there must be
something special in there makefiles/recipes.

After a detailed inspection i noted that they all depend on a library
which uses glib (incl. some tools like gob2). The library itself does
not have the build problems. But could the problem result from this?

Steffen

Re: bitbake freeze :(

[Khem Raj]

On Wed, Sep 23, 2009 at 12:32 AM, Steffen Sledz <sledz@dresearch.de> wrote:
> Khem Raj wrote:
>> you could try couple of things. Firstly try to isolate if its a
>> bitbake problem or not
>> likely its not but still make sure. You can go into the workdir and
>> run the task script
>> run.do_<task> whatever it is stuck at. If it succeeeds then you know
>> its a problem with
>> bitbake if not then the build mechanism for this particular package
>> has problem. You might
>> try to disable parallel build i.e. use -j 1 to make sure that its not
>> some parallel build
>> issue.
>
> I made some tests on this.
>
> First result was that if calling
>
>  > bitbake drapptempl -c clean
>  > bitbake drapptempl
>  Ctrl-C
>  > bitbake drapptempl
>
> the first build request freezes in do_compile stage, the second one
> succeeds (seen in 5 of 5 test runs). Also calling run.do_compile
> succeeds everytime if tried after the first freeze.
>
> After that i checked the parallel build options in my local.conf.
> The tests above were made with
>
>  PARALLEL_MAKE = "-j 4"
>  BB_NUMBER_THREADS = "4"
>
> I tried to set the BB_NUMBER_THREADS to 1 but the problem was still
> existing.
>
> After that i tried to set the PARALLEL_MAKE to "-j 1" and no the
> recipe was build fine.  :)
>
> So it seems that there is a problem with parallel make. But it only
> occurs in a few of our recipes. Therefor i believe that there must be
> something special in there makefiles/recipes.
>
> After a detailed inspection i noted that they all depend on a library
> which uses glib (incl. some tools like gob2). The library itself does
> not have the build problems. But could the problem result from this?
>

disable parallel build for this recipe.

> Steffen
>
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: bitbake freeze :(

[Cliff Brake]

On Tue, Sep 22, 2009 at 9:22 AM, Steffen Sledz <sledz@dresearch.de> wrote:
> Since a few days i see this crazy behavior. Sometimes bitbake freezes (everytime at the same stages if it freezes) when building a target (e.g. with "bitbake drapptempl"). top tells me that a "make all" process is running and consuming all resources of one cpu.

I had an issue recently where opkg was prompting the user to overwrite
a file in the rootfs phase.  I would ctrl-C bitbake when it appeared
to be hung, but soon learned there were additional bitbake processes
still running.  When I ran killall bitbake, the error message showed
up.  You might find something by running bitbake with -DDDD.

Cliff

Re: bitbake freeze :(

[Denys Dmytriyenko]

On Tue, Sep 22, 2009 at 02:28:22PM -0400, Cliff Brake wrote:
> On Tue, Sep 22, 2009 at 9:22 AM, Steffen Sledz <sledz@dresearch.de> wrote:
> > Since a few days i see this crazy behavior. Sometimes bitbake freezes (everytime at the same stages if it freezes) when building a target (e.g. with "bitbake drapptempl"). top tells me that a "make all" process is running and consuming all resources of one cpu.
> 
> I had an issue recently where opkg was prompting the user to overwrite
> a file in the rootfs phase.  I would ctrl-C bitbake when it appeared
> to be hung, but soon learned there were additional bitbake processes
> still running.  When I ran killall bitbake, the error message showed
> up.  You might find something by running bitbake with -DDDD.

Now, that you mentioned it, I have also seen it before! There was a file clash 
between 2 recipes providing the same file, so opkg was waiting for the user 
input asking to overwrite the file during the rootfs step. But opkg I/O is 
being redirected, so it appears that bitbake is hung. I found out about the 
problem from the rootfs log file in the tmp/work directory, where the 
output is being redirected to... :)

-- 
Denys

Re: bitbake freeze :(

[Steffen Sledz]

Cliff Brake schrieb:
>> Since a few days i see this crazy behavior. Sometimes bitbake
>> freezes (everytime at the same stages if it freezes) when building
>> a target (e.g. with "bitbake drapptempl"). top tells me that a
>> "make all" process is running and consuming all resources of one cpu.
> 
> I had an issue recently where opkg was prompting the user to overwrite
> a file in the rootfs phase.  I would ctrl-C bitbake when it appeared
> to be hung, but soon learned there were additional bitbake processes
> still running.  When I ran killall bitbake, the error message showed
> up.  You might find something by running bitbake with -DDDD.

I'm baking a single recipe. The freeze occurs in the do_compile stage.
So this seems to be another problem.

Steffen