Multiple issues building from dizzy or master

Multiple issues building from dizzy or master

[Lauren Post]

We've been trying for a few weeks to move all our builds to master (yesterday dizzy for poky which is the same) and been having random strange problems on multiple jenkins machines we do our daily builds on.  I talked to Otavio about it but wonder if we are missing something.    I'd appreciate any ideas.   We can build our kernel and uboot fine but when we try to build a more complex image with qt5 we have not succeeded any build since yesterday.

As of today we are now getting errors in db compilation
On both locally and on Jenkins we keep getting a db compile error as of today.
./db_cxx.h:59:22: fatal error: iostream.h: No such file or directory
 #include <iostream.h>
                      ^
---------------------------------------------------------------------
For last week we've been battling random gcc_runtime configure errors only on Jenkins machines.  

This is the configure error.
checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.

-------------------------------------------------------------------------------------------------------
As of yesterday morning, we are also having many unpack errors on many files that are disappearing from our downloads before unpack.  The fetch is working but then when it unpacks the package has disappeared from downloads.  I've seen this on multiple components (libx11, xproto, inputproto, util-linux).  Seems to happen more often with rm_work set but I've seen it without it set.

I'd appreciate any ideas.  The only one I can reproduce locally is the db error above but the others seem to only happen on Jenkins machines.  We have 4 different Jenkins machines all of which can't build on the master branch but have built fine for months on daisy and dora branches.  

By the way you can't set your gcc to 4.8.2 - liburcu breaks if you try to use the older compiler.  So not sure if any of the gcc runtime issues are related to the new gcc version.   Not sure if we should keep gcc 4.8.2 in dizzy if it can't be used.

Thanks for any ideas.  I've tried everything I can think of.

Lauren Post
i.MX Freescale 

Re: Multiple issues building from dizzy or master

[Mark Hatle]

On 10/7/14, 10:24 AM, Lauren Post wrote:
> We've been trying for a few weeks to move all our builds to master (yesterday dizzy for poky which is the same) and been having random strange problems on multiple jenkins machines we do our daily builds on.  I talked to Otavio about it but wonder if we are missing something.    I'd appreciate any ideas.   We can build our kernel and uboot fine but when we try to build a more complex image with qt5 we have not succeeded any build since yesterday.
>
> As of today we are now getting errors in db compilation
> On both locally and on Jenkins we keep getting a db compile error as of today.
> ./db_cxx.h:59:22: fatal error: iostream.h: No such file or directory
>   #include <iostream.h>
>                        ^
> ---------------------------------------------------------------------
> For last week we've been battling random gcc_runtime configure errors only on Jenkins machines.
>
> This is the configure error.
> checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.

I hit this issue w/ Aarch64 builds, and the issue was that gcc linker definition 
did not match the installed library path definition.

In the aarch64 case, the ld.so lives in '/lib', and everything else in the 'oe 
libdir' (which if set to 'lib64') was causing issue.  I finally fixed it in my 
local build by symlinking ld.so into both places.

(I realize this isn't likely on the same architecture, but I suspect the same 
issue is possible here.  Basically the compiler and glibc are not synchronized 
and are looking in different locations for their components.)

> -------------------------------------------------------------------------------------------------------
> As of yesterday morning, we are also having many unpack errors on many files
> that are disappearing from our downloads before unpack. The fetch is working but
> then when it unpacks the package has disappeared from downloads. I've seen this
> on multiple components (libx11, xproto, inputproto, util-linux). Seems to happen
> more often with rm_work set but I've seen it without it set.

I don't use rm_work myself.  But a few of my coworkers do.  The only place 
they've hit an issue is on a specific CentOS based machine.  We've not been able 
to track it down, but rm_work appeared to run BEFORE the final staging...

> I'd appreciate any ideas. The only one I can reproduce locally is the db
> error  above but the others seem to only happen on Jenkins machines. We have 4
> different Jenkins machines all of which can't build on the master branch but
> have built fine for months on daisy and dora branches.

There was a recent bug found that affect systems with 64-bit inodes.  Check if 
you have that, you may need a 'pseudo' update... back porting from master should 
be fine for older versions.  (Master was updated yesterday?)

> By the way you can't set your gcc to 4.8.2 - liburcu breaks if you try to
> use  the older compiler. So not sure if any of the gcc runtime issues are related to
> the new gcc version. Not sure if we should keep gcc 4.8.2 in dizzy if it can't
> be used.
>
> Thanks for any ideas.  I've tried everything I can think of.
>
> Lauren Post
> i.MX Freescale
>

Re: Multiple issues building from dizzy or master

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2717 bytes --]

On Tue, Oct 07, 2014 at 03:24:35PM +0000, Lauren Post wrote:
> We've been trying for a few weeks to move all our builds to master (yesterday dizzy for poky which is the same) and been having random strange problems on multiple jenkins machines we do our daily builds on.  I talked to Otavio about it but wonder if we are missing something.    I'd appreciate any ideas.   We can build our kernel and uboot fine but when we try to build a more complex image with qt5 we have not succeeded any build since yesterday.
> 
> As of today we are now getting errors in db compilation
> On both locally and on Jenkins we keep getting a db compile error as of today.
> ./db_cxx.h:59:22: fatal error: iostream.h: No such file or directory
>  #include <iostream.h>
>                       ^

Check this thread:
https://www.mail-archive.com/openembedded-core@lists.openembedded.org/msg55742.html

you need to detect why your db build doesn't define HAVE_CXX_STDHEADERS

> ---------------------------------------------------------------------
> For last week we've been battling random gcc_runtime configure errors only on Jenkins machines.  
> 
> This is the configure error.
> checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.
> 
> -------------------------------------------------------------------------------------------------------
> As of yesterday morning, we are also having many unpack errors on many files that are disappearing from our downloads before unpack.  The fetch is working but then when it unpacks the package has disappeared from downloads.  I've seen this on multiple components (libx11, xproto, inputproto, util-linux).  Seems to happen more often with rm_work set but I've seen it without it set.
> 
> I'd appreciate any ideas.  The only one I can reproduce locally is the db error above but the others seem to only happen on Jenkins machines.  We have 4 different Jenkins machines all of which can't build on the master branch but have built fine for months on daisy and dora branches.  
> 
> By the way you can't set your gcc to 4.8.2 - liburcu breaks if you try to use the older compiler.  So not sure if any of the gcc runtime issues are related to the new gcc version.   Not sure if we should keep gcc 4.8.2 in dizzy if it can't be used.
> 
> Thanks for any ideas.  I've tried everything I can think of.
> 
> Lauren Post
> i.MX Freescale 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Multiple issues building from dizzy or master

[Lauren Post]

Last night I moved poky back to 9/16  "0db15bb0626b16201d0767d87e91c61828a449ef" before the sstate commit and my builds are working now.  Before that I tried reverting some of the earlier patches such as (533a852eddb9a5e77a15001756f03b929a1d0a0b ) brought in on 10/6 but that did not work so something that came in last few weeks is also  causing problems.

My biggest problems is the fetch/unpack.  With tip of poky the task fetch is working but before unpack the package disappears and task unpack fails.  It happens on many random components and always in downloads I see a lock on the component but no package.  I see this with dirty and clean builds.

It will take me time to figure out which poky commit is causing the problems but my team has been waiting on these builds to proceed with testing so I can only try on one machine.  It is much harder to hit locally - I see it more readily on Jenkins build machines.

I just wanted to verify first it was a poky problem  On all 4 jenkins machines building last night none with the reverted poky back to 9/16 had the fetch/unpack issues I was seeing with the tip of dizzy branch in poky yesterday.

Lauren Post
i.MX Freescale

-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Lauren Post
Sent: Tuesday, October 07, 2014 10:25 AM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] Multiple issues building from dizzy or master

We've been trying for a few weeks to move all our builds to master (yesterday dizzy for poky which is the same) and been having random strange problems on multiple jenkins machines we do our daily builds on.  I talked to Otavio about it but wonder if we are missing something.    I'd appreciate any ideas.   We can build our kernel and uboot fine but when we try to build a more complex image with qt5 we have not succeeded any build since yesterday.

As of today we are now getting errors in db compilation On both locally and on Jenkins we keep getting a db compile error as of today.
./db_cxx.h:59:22: fatal error: iostream.h: No such file or directory  #include <iostream.h>
                      ^
---------------------------------------------------------------------
For last week we've been battling random gcc_runtime configure errors only on Jenkins machines.  

This is the configure error.
checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.

-------------------------------------------------------------------------------------------------------
As of yesterday morning, we are also having many unpack errors on many files that are disappearing from our downloads before unpack.  The fetch is working but then when it unpacks the package has disappeared from downloads.  I've seen this on multiple components (libx11, xproto, inputproto, util-linux).  Seems to happen more often with rm_work set but I've seen it without it set.

I'd appreciate any ideas.  The only one I can reproduce locally is the db error above but the others seem to only happen on Jenkins machines.  We have 4 different Jenkins machines all of which can't build on the master branch but have built fine for months on daisy and dora branches.  

By the way you can't set your gcc to 4.8.2 - liburcu breaks if you try to use the older compiler.  So not sure if any of the gcc runtime issues are related to the new gcc version.   Not sure if we should keep gcc 4.8.2 in dizzy if it can't be used.

Thanks for any ideas.  I've tried everything I can think of.

Lauren Post
i.MX Freescale
--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Multiple issues building from dizzy or master

[Lauren Post]

I've narrowed down that 1 or 2 updates on poky from 9/16 are causing my problems.     If I revert only these 2 commits my build problems are resolved.

In our build script on Jenkins we always build kernel, then uboot with a force deploy then our image then we build the image.  I see the errors when we build the  image.  I believe the force deploy might be part of the problem.  We do that so in case of dirty builds we always guarantee the kernel and uboot in the images directory (although I hit this with clean builds every time).  On local machine if I only build image I never see the problem, only when I build locally with our build script.

This gcc revision I believe is causing the db and gcc runtime issues.
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dizzy&id=14ace86d508a5bdbb8139ce6af09de4d89b44595 

I also think this poky revision might be causing the fetch/unpack issues.  In this issue we fetch works based on log output, unpack fails and when I look at downloads I see the <package>.lock in the downloads directory.  Happens randomly on many components.  I'm doing a build with only gcc patch above to confirm if the unpack failures come back without this commit below reverted.
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dizzy&id=7d80f8e9468253496a7097685aac8f468940a9c5

Unfortunately to verify this I've had to do several clean builds all over poky to narrow this down.  I'll further narrow down the 2 patches mentioned above.

All our builds set to the 9/16 commit before these patches work fine with no problems.  I've been using one Jenkins machine to narrow this down.

Lauren

-----Original Message-----
From: Post Lauren-RAA013 
Sent: Wednesday, October 08, 2014 7:52 AM
To: openembedded-core@lists.openembedded.org
Subject: RE: Multiple issues building from dizzy or master

Last night I moved poky back to 9/16  "0db15bb0626b16201d0767d87e91c61828a449ef" before the sstate commit and my builds are working now.  Before that I tried reverting some of the earlier patches such as (533a852eddb9a5e77a15001756f03b929a1d0a0b ) brought in on 10/6 but that did not work so something that came in last few weeks is also  causing problems.

My biggest problems is the fetch/unpack.  With tip of poky the task fetch is working but before unpack the package disappears and task unpack fails.  It happens on many random components and always in downloads I see a lock on the component but no package.  I see this with dirty and clean builds.

It will take me time to figure out which poky commit is causing the problems but my team has been waiting on these builds to proceed with testing so I can only try on one machine.  It is much harder to hit locally - I see it more readily on Jenkins build machines.

I just wanted to verify first it was a poky problem  On all 4 jenkins machines building last night none with the reverted poky back to 9/16 had the fetch/unpack issues I was seeing with the tip of dizzy branch in poky yesterday.

Lauren Post
i.MX Freescale

-----Original Message-----
From: openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of Lauren Post
Sent: Tuesday, October 07, 2014 10:25 AM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] Multiple issues building from dizzy or master

We've been trying for a few weeks to move all our builds to master (yesterday dizzy for poky which is the same) and been having random strange problems on multiple jenkins machines we do our daily builds on.  I talked to Otavio about it but wonder if we are missing something.    I'd appreciate any ideas.   We can build our kernel and uboot fine but when we try to build a more complex image with qt5 we have not succeeded any build since yesterday.

As of today we are now getting errors in db compilation On both locally and on Jenkins we keep getting a db compile error as of today.
./db_cxx.h:59:22: fatal error: iostream.h: No such file or directory  #include <iostream.h>
                      ^
---------------------------------------------------------------------
For last week we've been battling random gcc_runtime configure errors only on Jenkins machines.  

This is the configure error.
checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.

-------------------------------------------------------------------------------------------------------
As of yesterday morning, we are also having many unpack errors on many files that are disappearing from our downloads before unpack.  The fetch is working but then when it unpacks the package has disappeared from downloads.  I've seen this on multiple components (libx11, xproto, inputproto, util-linux).  Seems to happen more often with rm_work set but I've seen it without it set.

I'd appreciate any ideas.  The only one I can reproduce locally is the db error above but the others seem to only happen on Jenkins machines.  We have 4 different Jenkins machines all of which can't build on the master branch but have built fine for months on daisy and dora branches.  

By the way you can't set your gcc to 4.8.2 - liburcu breaks if you try to use the older compiler.  So not sure if any of the gcc runtime issues are related to the new gcc version.   Not sure if we should keep gcc 4.8.2 in dizzy if it can't be used.

Thanks for any ideas.  I've tried everything I can think of.

Lauren Post
i.MX Freescale
--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: Multiple issues building from dizzy or master

[Burton, Ross]

On 9 October 2014 16:50, Lauren Post <Lauren.Post@freescale.com> wrote:
> I also think this poky revision might be causing the fetch/unpack issues.  In this issue we fetch works based on log output, unpack fails and when I look at downloads I see the <package>.lock in the downloads directory.  Happens randomly on many components.  I'm doing a build with only gcc patch above to confirm if the unpack failures come back without this commit below reverted.
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dizzy&id=7d80f8e9468253496a7097685aac8f468940a9c5

That commit changes the format of a comment in the "bitbake -e"
output.  If you've scripts that invoke that and parse the output then
they'll need to be updated, but otherwise it won't have caused
problems with fetching.

Ross

Re: Multiple issues building from dizzy or master

[Richard Purdie]

On Thu, 2014-10-09 at 15:50 +0000, Lauren Post wrote:
> I've narrowed down that 1 or 2 updates on poky from 9/16 are causing my problems.     If I revert only these 2 commits my build problems are resolved.
> 
> In our build script on Jenkins we always build kernel, then uboot with a force deploy then our image then we build the image.  I see the errors when we build the  image.  I believe the force deploy might be part of the problem.  We do that so in case of dirty builds we always guarantee the kernel and uboot in the images directory (although I hit this with clean builds every time).  On local machine if I only build image I never see the problem, only when I build locally with our build script.
> 
> This gcc revision I believe is causing the db and gcc runtime issues.
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dizzy&id=14ace86d508a5bdbb8139ce6af09de4d89b44595 
> 
> I also think this poky revision might be causing the fetch/unpack
> issues.  In this issue we fetch works based on log output, unpack
> fails and when I look at downloads I see the <package>.lock in the
> downloads directory.  Happens randomly on many components.  I'm doing
> a build with only gcc patch above to confirm if the unpack failures
> come back without this commit below reverted.
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dizzy&id=7d80f8e9468253496a7097685aac8f468940a9c5

Do you have a link to the full error logs of the issue you're seeing? I
doubt this second commit would cause issues. It only affects the output
of bitbake -e when variable history tracking is enabled.

The first patch fixed several different problems but is is possible you
have a different race issue in your toolchain bootstrap process. Its
hard for me to comment without seeing the errors or knowning which
layers are involved.

Cheers,

Richard