glibc broken when linked with gold Was: [oe] State of OE World, 2019-03-16

glibc broken when linked with gold Was: [oe] State of OE World, 2019-03-16

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 9854 bytes --]

On Wed, Mar 27, 2019 at 09:39:07PM +0100, Martin Jansa wrote:
> On Wed, Mar 27, 2019 at 05:35:07PM +0100, Martin Jansa wrote:
> > On Wed, Mar 27, 2019 at 05:17:54PM +0100, Martin Jansa wrote:
> > > On Sun, Mar 17, 2019 at 08:26:37AM -0700, Khem Raj wrote:
> > > > http://www.openembedded.org/wiki/Bitbake_World_Status
> > > > 
> > > > == Failed tasks 2019-03-16 ==
> > > > 
> > > > INFO: jenkins-job.sh-1.8.45 Complete log available at
> > > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.report.20190317_082308.log
> > > > 
> > > >     * sources/openembedded-core/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.58.3.bb:do_compile
> > > 
> > > I was hit by this one as well on some arm targets.
> > > 
> > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.world.qemuarm.20190321_215508.log/bitbake.log
> > > 
> > > shows that it's actually from qemu-arm segfault inside g-ir-scanner-qemuwrapper
> > > 
> > > | qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > | /home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/build/g-ir-scanner-qemuwrapper: line 6:  1959 Segmentation fault      (core dumped) PSEUDO_UNLOAD=1 qemu-arm -r 3.2.0 -L /home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot -E LD_LIBRARY_PATH=$GIR_EXTRA_LIBS_PATH:.libs:/home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot//usr/lib:/home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot//lib "$@"
> > > 
> > > The interesting part is that this happens only when glibc is built with gold enabled.
> > > 
> > > I've put a bit more details in temporary work-around here:
> > > https://github.com/shr-distribution/meta-smartphone/commit/8f06eb355ead85464b1a1bbaa82584504df15743
> > > 
> > > I'm comparing qemuarm glibc build with bfd and gold to see if there is some
> > > significant difference which might cause this and I plan to flash some small
> > > image on my targets to see if libc is broken in runtime as well or only when
> > > running inside qemu-arm.
> > > 
> > > Is anyone else seeing this as well?
> > 
> > Checking older world status on the wiki shows that this issue was
> > introduced somewhere around christmas
> > 
> > This is last world build before the issue:
> > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.report.20181219_112425.log
> > ...
> > == Tested changes (not included in master yet) - openembedded-core ==
> > latest upstream commit: 
> > 14c291e1fb gcc-runtime: Add missing libc dependency
> > not included in master yet: 
> > 7e2ab991fa python/python3: use cc_basename to replace CC for checking compiler
> > ced915026d python-native: fix one do_populate_sysroot warning
> > aee47f3e82 netbase: add entry to /etc/hosts according to /etc/hostname
> > a09e2db43a sstate: add support for caching shared workdir tasks
> > c104a34166 grub2: Fix passing null to printf formats
> > 6f364ff8c4 gnupg: Upgrade to 2.2.12 release
> > 0224fec86b glibc: Upgrade towards 2.29 release
> > 2486349782 gcc-9.0: Add recipes for upcoming gcc 9.0 release in mid-2019
> > b3ab29bdbb gcc-runtime: Drop building libmpx
> > 
> > and this is the first which reported qemu-arm segfault:
> > == Tested changes (not included in master yet) - openembedded-core ==
> > latest upstream commit: 
> > 95659bed3f populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > not included in master yet: 
> > 0c9db0ae7d python/python3: use cc_basename to replace CC for checking compiler
> > bb3eb6bc41 python-native: fix one do_populate_sysroot warning
> > 08a205f872 netbase: add entry to /etc/hosts according to /etc/hostname
> > b81c8650d0 sstate: add support for caching shared workdir tasks
> > 702be42dc9 glibc: Remove site_config and glibc-initial
> > fc230822d8 gcc: Drop gcc-cross-initial and use gcc-cross instead
> > 8652df3a0d gcc: Drop the -initial versions of the compiler
> > 0c2f6dfa2f recipes: Drop virtual/libc-for-gcc
> > c64531600d newlib: Move away from gcc-initial dependency
> > 0d5fb4428f libssp: Remove dependency on gcc-initial
> > 7dd8829d3e musl: Move away from gcc-initial dependency
> > 9abdb4d6ca tcmode-default: Drop pinnings for gcc-initial based recipes
> > e02724e8f8 base.bbclass, classextend.py: Drop catering to gcc-initial
> > 7756cb7d17 oeqa/concurrencytest: fix for locating meta-selftest
> > f4c68c39e7 crosssdk/cross-canadian: Set LIBCOVERRIDE correctly
> > 6fc9bb5698 glibc: Enable --with-default-link
> > d4cb282bc6 gcc-9.0: Add recipes for upcoming gcc 9.0 release in mid-2019
> > 6efe313b46 glibc: Upgrade towards 2.29 release
> > dac5dc5067 grub2: Fix passing null to printf formats
> > bb36d9dc8c gnupg: Upgrade to 2.2.12 release
> > 81059437a5 gnupg: Do not apply -Woverride-init guard for gcc >= 9
> > 6fa05e5ace nfs-utils: Disable format-overflow as Werror
> > 8fab7b23ef musl: Update to latest master
> > a5641cac42 libxcrypt: Upgrade to 4.4.2
> > c90ec03d07 gdb: Backport a fix to build with gcc9
> > 1adae92573 valgrind: Fix x86_64 build failure with gcc9
> > 38e9e09104 ffmpeg: Backport patch to fix build with gcc9
> > 4e1fa57743 binutils: Upgrade to latest on 2.31 release branch
> > 5cce276d4a binutils: Upgrade to upcoming 2.32 release
> > 
> > and the changes introduced in oe-core between these are:
> > docker-shr @ ~/openembedded-core $ git log --oneline 14c291e1fb..95659bed3f
> > 95659bed3f populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > 085530e8ff i2c-tools: upgrade 4.0 -> 4.1
> > 673c2dcf10 common-licenses: update Libpng license text
> > 099aecfaa3 libpng: upgrade 1.6.35 -> 1.6.36
> > 6574bb7523 harfbuzz: upgrade 2.1.3 -> 2.2.0
> > 8286e9d79d sqlite3: upgrade 3.25.3 -> 3.26.0
> > a33170ff9a libatomic-ops: upgrade 7.6.6 -> 7.6.8
> > 6a10ff0bcd gsettings-desktop-schemas: upgrade 3.28.0 -> 3.28.1
> > dd2c4d9d85 eudev: upgrade 3.2.5 -> 3.2.7
> > 0860216a81 icecc: patchelf is needed by icecc-create-env
> > d2fcaeb153 icecc: Don't generate recipe-sysroot symlinks at recipe-parsing time
> > 4455c46bf7 icecc: Syntax error meant that we weren't waiting for tarball generation
> > fa6bae80bc icecc: Trivial simplification
> > 6d6788a3ea icecc: readlink -f on the recipe-sysroot gcc/g++
> > 1ee90ec89a libxcrypt: Fix nativesdk build with gcc9
> > 547174fc83 gcc-runtime: Drop building libmpx
> > 
> > It's possible that it's not caused by one of these oe-core changes, but one of the pending
> > yoe changes (like those 2 binutils upgrades) which were merged to oe-core in some form a bit
> > later.
> > 
> > I'll verify if the issue is reproducible with oe-core 95659bed3f and continue the bisect from there.
> 
> git bisect narrowed it down to these oe-core commits:
> 
> da527d73f5 oeqa/utils/qemurunner: Print output when failed to login
> 7bbc453ac2 (refs/bisect/bad) glibc: Enable --with-default-link
> 5d0307fbef (HEAD) base.bbclass, classextend.py: Drop catering to gcc-initial
> 5b2fd1c724 tcmode-default: Drop pinnings for gcc-initial based recipes
> fa47e2e654 musl: Move away from gcc-initial dependency
> 9dc6fcd492 libssp: Remove dependency on gcc-initial
> 44539118ce newlib: Move away from gcc-initial dependency
> 122217b421 recipes: Drop virtual/libc-for-gcc
> d84971928b gcc: Drop the -initial versions of the compiler
> 62b7308b8c gcc: Drop gcc-cross-initial and use gcc-cross instead
> c5df105e75 glibc: Remove site_config and glibc-initial
> 94bece16c2 crosssdk/cross-canadian: Set LIBCOVERRIDE correctly
> 56d2493a9a oeqa/concurrencytest: fix for locating meta-selftest
> 95659bed3f (glibc-95659bed3f, refs/bisect/good-95659bed3f6f3216b346f70cfc9ffae9788c0fc1) populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> 085530e8ff i2c-tools: upgrade 4.0 -> 4.1
> 
> my guess was the very last commit "glibc: Enable --with-default-link",
> but unfortunately without this fix I cannot build glibc with gold enabled.
> 
> Will compare the glibc build to see is there is something obviously wrong
> in the output.

I've confirmed that forcing bfd linker in glibc do_compile resolves the
issue with gobject-introspection (image build still running will try to
flash it tomorrow).

The simples work-around in oe-core:
http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=054c72716f0648ff2f0770c1010857596442223e
or as bbappend:
https://github.com/webOS-ports/meta-webos-ports/commit/020d31892780b69fa64a3a6a146283e831ccd2e9
this is what I'm testing with now, notice that setting just LDFLAGS
isn't enough, because LDFLAGS are unset in glibc.do_compile

I don't know if it's worth spending more time trying to get the glibc
working with gold, it looks like:
http://git.openembedded.org/openembedded-core/commit/?id=7bbc453ac2b8d63680855789948a145fc448017f
resolved the linker script issue (I was seeing the same error with
oe-core 5d0307fbef just this fix) reported here:
https://sourceware.org/bugzilla/show_bug.cgi?id=24148

but there are other bugs like:
https://sourceware.org/bugzilla/show_bug.cgi?id=10937
https://sourceware.org/bugzilla/show_bug.cgi?id=18103
e.g. gentoo is also explicitly disabling gold for glibc builds:
https://github.com/gentoo/gentoo/blob/16e1c37bdc84b2ac5df10511f63e478d8e49435e/sys-libs/glibc/glibc-2.29-r1.ebuild#L767

Can someone confirm that glibc built with gold works for him on arm
(I haven't seen this on x86, aarch64)? Or should we just force bfd in
oe-core recipe?

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: glibc broken when linked with gold Was: [oe] State of OE World, 2019-03-16

[Khem Raj]

On Wed, Mar 27, 2019 at 3:32 PM Martin Jansa <martin.jansa@gmail.com> wrote:
>
> On Wed, Mar 27, 2019 at 09:39:07PM +0100, Martin Jansa wrote:
> > On Wed, Mar 27, 2019 at 05:35:07PM +0100, Martin Jansa wrote:
> > > On Wed, Mar 27, 2019 at 05:17:54PM +0100, Martin Jansa wrote:
> > > > On Sun, Mar 17, 2019 at 08:26:37AM -0700, Khem Raj wrote:
> > > > > http://www.openembedded.org/wiki/Bitbake_World_Status
> > > > >
> > > > > == Failed tasks 2019-03-16 ==
> > > > >
> > > > > INFO: jenkins-job.sh-1.8.45 Complete log available at
> > > > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.report.20190317_082308.log
> > > > >
> > > > >     * sources/openembedded-core/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.58.3.bb:do_compile
> > > >
> > > > I was hit by this one as well on some arm targets.
> > > >
> > > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.world.qemuarm.20190321_215508.log/bitbake.log
> > > >
> > > > shows that it's actually from qemu-arm segfault inside g-ir-scanner-qemuwrapper
> > > >
> > > > | qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > > | /home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/build/g-ir-scanner-qemuwrapper: line 6:  1959 Segmentation fault      (core dumped) PSEUDO_UNLOAD=1 qemu-arm -r 3.2.0 -L /home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot -E LD_LIBRARY_PATH=$GIR_EXTRA_LIBS_PATH:.libs:/home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot//usr/lib:/home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot//lib "$@"
> > > >
> > > > The interesting part is that this happens only when glibc is built with gold enabled.
> > > >
> > > > I've put a bit more details in temporary work-around here:
> > > > https://github.com/shr-distribution/meta-smartphone/commit/8f06eb355ead85464b1a1bbaa82584504df15743
> > > >
> > > > I'm comparing qemuarm glibc build with bfd and gold to see if there is some
> > > > significant difference which might cause this and I plan to flash some small
> > > > image on my targets to see if libc is broken in runtime as well or only when
> > > > running inside qemu-arm.
> > > >
> > > > Is anyone else seeing this as well?
> > >
> > > Checking older world status on the wiki shows that this issue was
> > > introduced somewhere around christmas
> > >
> > > This is last world build before the issue:
> > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.report.20181219_112425.log
> > > ...
> > > == Tested changes (not included in master yet) - openembedded-core ==
> > > latest upstream commit:
> > > 14c291e1fb gcc-runtime: Add missing libc dependency
> > > not included in master yet:
> > > 7e2ab991fa python/python3: use cc_basename to replace CC for checking compiler
> > > ced915026d python-native: fix one do_populate_sysroot warning
> > > aee47f3e82 netbase: add entry to /etc/hosts according to /etc/hostname
> > > a09e2db43a sstate: add support for caching shared workdir tasks
> > > c104a34166 grub2: Fix passing null to printf formats
> > > 6f364ff8c4 gnupg: Upgrade to 2.2.12 release
> > > 0224fec86b glibc: Upgrade towards 2.29 release
> > > 2486349782 gcc-9.0: Add recipes for upcoming gcc 9.0 release in mid-2019
> > > b3ab29bdbb gcc-runtime: Drop building libmpx
> > >
> > > and this is the first which reported qemu-arm segfault:
> > > == Tested changes (not included in master yet) - openembedded-core ==
> > > latest upstream commit:
> > > 95659bed3f populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > > not included in master yet:
> > > 0c9db0ae7d python/python3: use cc_basename to replace CC for checking compiler
> > > bb3eb6bc41 python-native: fix one do_populate_sysroot warning
> > > 08a205f872 netbase: add entry to /etc/hosts according to /etc/hostname
> > > b81c8650d0 sstate: add support for caching shared workdir tasks
> > > 702be42dc9 glibc: Remove site_config and glibc-initial
> > > fc230822d8 gcc: Drop gcc-cross-initial and use gcc-cross instead
> > > 8652df3a0d gcc: Drop the -initial versions of the compiler
> > > 0c2f6dfa2f recipes: Drop virtual/libc-for-gcc
> > > c64531600d newlib: Move away from gcc-initial dependency
> > > 0d5fb4428f libssp: Remove dependency on gcc-initial
> > > 7dd8829d3e musl: Move away from gcc-initial dependency
> > > 9abdb4d6ca tcmode-default: Drop pinnings for gcc-initial based recipes
> > > e02724e8f8 base.bbclass, classextend.py: Drop catering to gcc-initial
> > > 7756cb7d17 oeqa/concurrencytest: fix for locating meta-selftest
> > > f4c68c39e7 crosssdk/cross-canadian: Set LIBCOVERRIDE correctly
> > > 6fc9bb5698 glibc: Enable --with-default-link
> > > d4cb282bc6 gcc-9.0: Add recipes for upcoming gcc 9.0 release in mid-2019
> > > 6efe313b46 glibc: Upgrade towards 2.29 release
> > > dac5dc5067 grub2: Fix passing null to printf formats
> > > bb36d9dc8c gnupg: Upgrade to 2.2.12 release
> > > 81059437a5 gnupg: Do not apply -Woverride-init guard for gcc >= 9
> > > 6fa05e5ace nfs-utils: Disable format-overflow as Werror
> > > 8fab7b23ef musl: Update to latest master
> > > a5641cac42 libxcrypt: Upgrade to 4.4.2
> > > c90ec03d07 gdb: Backport a fix to build with gcc9
> > > 1adae92573 valgrind: Fix x86_64 build failure with gcc9
> > > 38e9e09104 ffmpeg: Backport patch to fix build with gcc9
> > > 4e1fa57743 binutils: Upgrade to latest on 2.31 release branch
> > > 5cce276d4a binutils: Upgrade to upcoming 2.32 release
> > >
> > > and the changes introduced in oe-core between these are:
> > > docker-shr @ ~/openembedded-core $ git log --oneline 14c291e1fb..95659bed3f
> > > 95659bed3f populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > > 085530e8ff i2c-tools: upgrade 4.0 -> 4.1
> > > 673c2dcf10 common-licenses: update Libpng license text
> > > 099aecfaa3 libpng: upgrade 1.6.35 -> 1.6.36
> > > 6574bb7523 harfbuzz: upgrade 2.1.3 -> 2.2.0
> > > 8286e9d79d sqlite3: upgrade 3.25.3 -> 3.26.0
> > > a33170ff9a libatomic-ops: upgrade 7.6.6 -> 7.6.8
> > > 6a10ff0bcd gsettings-desktop-schemas: upgrade 3.28.0 -> 3.28.1
> > > dd2c4d9d85 eudev: upgrade 3.2.5 -> 3.2.7
> > > 0860216a81 icecc: patchelf is needed by icecc-create-env
> > > d2fcaeb153 icecc: Don't generate recipe-sysroot symlinks at recipe-parsing time
> > > 4455c46bf7 icecc: Syntax error meant that we weren't waiting for tarball generation
> > > fa6bae80bc icecc: Trivial simplification
> > > 6d6788a3ea icecc: readlink -f on the recipe-sysroot gcc/g++
> > > 1ee90ec89a libxcrypt: Fix nativesdk build with gcc9
> > > 547174fc83 gcc-runtime: Drop building libmpx
> > >
> > > It's possible that it's not caused by one of these oe-core changes, but one of the pending
> > > yoe changes (like those 2 binutils upgrades) which were merged to oe-core in some form a bit
> > > later.
> > >
> > > I'll verify if the issue is reproducible with oe-core 95659bed3f and continue the bisect from there.
> >
> > git bisect narrowed it down to these oe-core commits:
> >
> > da527d73f5 oeqa/utils/qemurunner: Print output when failed to login
> > 7bbc453ac2 (refs/bisect/bad) glibc: Enable --with-default-link
> > 5d0307fbef (HEAD) base.bbclass, classextend.py: Drop catering to gcc-initial
> > 5b2fd1c724 tcmode-default: Drop pinnings for gcc-initial based recipes
> > fa47e2e654 musl: Move away from gcc-initial dependency
> > 9dc6fcd492 libssp: Remove dependency on gcc-initial
> > 44539118ce newlib: Move away from gcc-initial dependency
> > 122217b421 recipes: Drop virtual/libc-for-gcc
> > d84971928b gcc: Drop the -initial versions of the compiler
> > 62b7308b8c gcc: Drop gcc-cross-initial and use gcc-cross instead
> > c5df105e75 glibc: Remove site_config and glibc-initial
> > 94bece16c2 crosssdk/cross-canadian: Set LIBCOVERRIDE correctly
> > 56d2493a9a oeqa/concurrencytest: fix for locating meta-selftest
> > 95659bed3f (glibc-95659bed3f, refs/bisect/good-95659bed3f6f3216b346f70cfc9ffae9788c0fc1) populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > 085530e8ff i2c-tools: upgrade 4.0 -> 4.1
> >
> > my guess was the very last commit "glibc: Enable --with-default-link",
> > but unfortunately without this fix I cannot build glibc with gold enabled.
> >
> > Will compare the glibc build to see is there is something obviously wrong
> > in the output.
>
> I've confirmed that forcing bfd linker in glibc do_compile resolves the
> issue with gobject-introspection (image build still running will try to
> flash it tomorrow).
>
> The simples work-around in oe-core:
> http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=054c72716f0648ff2f0770c1010857596442223e
> or as bbappend:
> https://github.com/webOS-ports/meta-webos-ports/commit/020d31892780b69fa64a3a6a146283e831ccd2e9
> this is what I'm testing with now, notice that setting just LDFLAGS
> isn't enough, because LDFLAGS are unset in glibc.do_compile
>
> I don't know if it's worth spending more time trying to get the glibc
> working with gold, it looks like:

i agree, your change to force bfd linker is acceptable approach
please propose it to OE-Core directly.


> http://git.openembedded.org/openembedded-core/commit/?id=7bbc453ac2b8d63680855789948a145fc448017f
> resolved the linker script issue (I was seeing the same error with
> oe-core 5d0307fbef just this fix) reported here:
> https://sourceware.org/bugzilla/show_bug.cgi?id=24148
>
> but there are other bugs like:
> https://sourceware.org/bugzilla/show_bug.cgi?id=10937
> https://sourceware.org/bugzilla/show_bug.cgi?id=18103
> e.g. gentoo is also explicitly disabling gold for glibc builds:
> https://github.com/gentoo/gentoo/blob/16e1c37bdc84b2ac5df10511f63e478d8e49435e/sys-libs/glibc/glibc-2.29-r1.ebuild#L767
>
> Can someone confirm that glibc built with gold works for him on arm
> (I haven't seen this on x86, aarch64)? Or should we just force bfd in
> oe-core recipe?
>
> --
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: glibc broken when linked with gold Was: [oe] State of OE World, 2019-03-16

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 11445 bytes --]

On Wed, Mar 27, 2019 at 10:08:32PM -0700, Khem Raj wrote:
> On Wed, Mar 27, 2019 at 3:32 PM Martin Jansa <martin.jansa@gmail.com> wrote:
> >
> > On Wed, Mar 27, 2019 at 09:39:07PM +0100, Martin Jansa wrote:
> > > On Wed, Mar 27, 2019 at 05:35:07PM +0100, Martin Jansa wrote:
> > > > On Wed, Mar 27, 2019 at 05:17:54PM +0100, Martin Jansa wrote:
> > > > > On Sun, Mar 17, 2019 at 08:26:37AM -0700, Khem Raj wrote:
> > > > > > http://www.openembedded.org/wiki/Bitbake_World_Status
> > > > > >
> > > > > > == Failed tasks 2019-03-16 ==
> > > > > >
> > > > > > INFO: jenkins-job.sh-1.8.45 Complete log available at
> > > > > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.report.20190317_082308.log
> > > > > >
> > > > > >     * sources/openembedded-core/meta/recipes-gnome/gobject-introspection/gobject-introspection_1.58.3.bb:do_compile
> > > > >
> > > > > I was hit by this one as well on some arm targets.
> > > > >
> > > > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.world.qemuarm.20190321_215508.log/bitbake.log
> > > > >
> > > > > shows that it's actually from qemu-arm segfault inside g-ir-scanner-qemuwrapper
> > > > >
> > > > > | qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > > > | /home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/build/g-ir-scanner-qemuwrapper: line 6:  1959 Segmentation fault      (core dumped) PSEUDO_UNLOAD=1 qemu-arm -r 3.2.0 -L /home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot -E LD_LIBRARY_PATH=$GIR_EXTRA_LIBS_PATH:.libs:/home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot//usr/lib:/home/jenkins/oe/world/yoe/build/tmpfs/work/armv7vet2hf-neon-yoe-linux-gnueabi/gobject-introspection/1.58.3-r0/recipe-sysroot//lib "$@"
> > > > >
> > > > > The interesting part is that this happens only when glibc is built with gold enabled.
> > > > >
> > > > > I've put a bit more details in temporary work-around here:
> > > > > https://github.com/shr-distribution/meta-smartphone/commit/8f06eb355ead85464b1a1bbaa82584504df15743
> > > > >
> > > > > I'm comparing qemuarm glibc build with bfd and gold to see if there is some
> > > > > significant difference which might cause this and I plan to flash some small
> > > > > image on my targets to see if libc is broken in runtime as well or only when
> > > > > running inside qemu-arm.
> > > > >
> > > > > Is anyone else seeing this as well?
> > > >
> > > > Checking older world status on the wiki shows that this issue was
> > > > introduced somewhere around christmas
> > > >
> > > > This is last world build before the issue:
> > > > http://logs.nslu2-linux.org/buildlogs/oe/world/warrior/log.report.20181219_112425.log
> > > > ...
> > > > == Tested changes (not included in master yet) - openembedded-core ==
> > > > latest upstream commit:
> > > > 14c291e1fb gcc-runtime: Add missing libc dependency
> > > > not included in master yet:
> > > > 7e2ab991fa python/python3: use cc_basename to replace CC for checking compiler
> > > > ced915026d python-native: fix one do_populate_sysroot warning
> > > > aee47f3e82 netbase: add entry to /etc/hosts according to /etc/hostname
> > > > a09e2db43a sstate: add support for caching shared workdir tasks
> > > > c104a34166 grub2: Fix passing null to printf formats
> > > > 6f364ff8c4 gnupg: Upgrade to 2.2.12 release
> > > > 0224fec86b glibc: Upgrade towards 2.29 release
> > > > 2486349782 gcc-9.0: Add recipes for upcoming gcc 9.0 release in mid-2019
> > > > b3ab29bdbb gcc-runtime: Drop building libmpx
> > > >
> > > > and this is the first which reported qemu-arm segfault:
> > > > == Tested changes (not included in master yet) - openembedded-core ==
> > > > latest upstream commit:
> > > > 95659bed3f populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > > > not included in master yet:
> > > > 0c9db0ae7d python/python3: use cc_basename to replace CC for checking compiler
> > > > bb3eb6bc41 python-native: fix one do_populate_sysroot warning
> > > > 08a205f872 netbase: add entry to /etc/hosts according to /etc/hostname
> > > > b81c8650d0 sstate: add support for caching shared workdir tasks
> > > > 702be42dc9 glibc: Remove site_config and glibc-initial
> > > > fc230822d8 gcc: Drop gcc-cross-initial and use gcc-cross instead
> > > > 8652df3a0d gcc: Drop the -initial versions of the compiler
> > > > 0c2f6dfa2f recipes: Drop virtual/libc-for-gcc
> > > > c64531600d newlib: Move away from gcc-initial dependency
> > > > 0d5fb4428f libssp: Remove dependency on gcc-initial
> > > > 7dd8829d3e musl: Move away from gcc-initial dependency
> > > > 9abdb4d6ca tcmode-default: Drop pinnings for gcc-initial based recipes
> > > > e02724e8f8 base.bbclass, classextend.py: Drop catering to gcc-initial
> > > > 7756cb7d17 oeqa/concurrencytest: fix for locating meta-selftest
> > > > f4c68c39e7 crosssdk/cross-canadian: Set LIBCOVERRIDE correctly
> > > > 6fc9bb5698 glibc: Enable --with-default-link
> > > > d4cb282bc6 gcc-9.0: Add recipes for upcoming gcc 9.0 release in mid-2019
> > > > 6efe313b46 glibc: Upgrade towards 2.29 release
> > > > dac5dc5067 grub2: Fix passing null to printf formats
> > > > bb36d9dc8c gnupg: Upgrade to 2.2.12 release
> > > > 81059437a5 gnupg: Do not apply -Woverride-init guard for gcc >= 9
> > > > 6fa05e5ace nfs-utils: Disable format-overflow as Werror
> > > > 8fab7b23ef musl: Update to latest master
> > > > a5641cac42 libxcrypt: Upgrade to 4.4.2
> > > > c90ec03d07 gdb: Backport a fix to build with gcc9
> > > > 1adae92573 valgrind: Fix x86_64 build failure with gcc9
> > > > 38e9e09104 ffmpeg: Backport patch to fix build with gcc9
> > > > 4e1fa57743 binutils: Upgrade to latest on 2.31 release branch
> > > > 5cce276d4a binutils: Upgrade to upcoming 2.32 release
> > > >
> > > > and the changes introduced in oe-core between these are:
> > > > docker-shr @ ~/openembedded-core $ git log --oneline 14c291e1fb..95659bed3f
> > > > 95659bed3f populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > > > 085530e8ff i2c-tools: upgrade 4.0 -> 4.1
> > > > 673c2dcf10 common-licenses: update Libpng license text
> > > > 099aecfaa3 libpng: upgrade 1.6.35 -> 1.6.36
> > > > 6574bb7523 harfbuzz: upgrade 2.1.3 -> 2.2.0
> > > > 8286e9d79d sqlite3: upgrade 3.25.3 -> 3.26.0
> > > > a33170ff9a libatomic-ops: upgrade 7.6.6 -> 7.6.8
> > > > 6a10ff0bcd gsettings-desktop-schemas: upgrade 3.28.0 -> 3.28.1
> > > > dd2c4d9d85 eudev: upgrade 3.2.5 -> 3.2.7
> > > > 0860216a81 icecc: patchelf is needed by icecc-create-env
> > > > d2fcaeb153 icecc: Don't generate recipe-sysroot symlinks at recipe-parsing time
> > > > 4455c46bf7 icecc: Syntax error meant that we weren't waiting for tarball generation
> > > > fa6bae80bc icecc: Trivial simplification
> > > > 6d6788a3ea icecc: readlink -f on the recipe-sysroot gcc/g++
> > > > 1ee90ec89a libxcrypt: Fix nativesdk build with gcc9
> > > > 547174fc83 gcc-runtime: Drop building libmpx
> > > >
> > > > It's possible that it's not caused by one of these oe-core changes, but one of the pending
> > > > yoe changes (like those 2 binutils upgrades) which were merged to oe-core in some form a bit
> > > > later.
> > > >
> > > > I'll verify if the issue is reproducible with oe-core 95659bed3f and continue the bisect from there.
> > >
> > > git bisect narrowed it down to these oe-core commits:
> > >
> > > da527d73f5 oeqa/utils/qemurunner: Print output when failed to login
> > > 7bbc453ac2 (refs/bisect/bad) glibc: Enable --with-default-link
> > > 5d0307fbef (HEAD) base.bbclass, classextend.py: Drop catering to gcc-initial
> > > 5b2fd1c724 tcmode-default: Drop pinnings for gcc-initial based recipes
> > > fa47e2e654 musl: Move away from gcc-initial dependency
> > > 9dc6fcd492 libssp: Remove dependency on gcc-initial
> > > 44539118ce newlib: Move away from gcc-initial dependency
> > > 122217b421 recipes: Drop virtual/libc-for-gcc
> > > d84971928b gcc: Drop the -initial versions of the compiler
> > > 62b7308b8c gcc: Drop gcc-cross-initial and use gcc-cross instead
> > > c5df105e75 glibc: Remove site_config and glibc-initial
> > > 94bece16c2 crosssdk/cross-canadian: Set LIBCOVERRIDE correctly
> > > 56d2493a9a oeqa/concurrencytest: fix for locating meta-selftest
> > > 95659bed3f (glibc-95659bed3f, refs/bisect/good-95659bed3f6f3216b346f70cfc9ffae9788c0fc1) populate_sdk_ext.bbclass: Include site.conf in parsing for contents for local.conf
> > > 085530e8ff i2c-tools: upgrade 4.0 -> 4.1
> > >
> > > my guess was the very last commit "glibc: Enable --with-default-link",
> > > but unfortunately without this fix I cannot build glibc with gold enabled.
> > >
> > > Will compare the glibc build to see is there is something obviously wrong
> > > in the output.
> >
> > I've confirmed that forcing bfd linker in glibc do_compile resolves the
> > issue with gobject-introspection (image build still running will try to
> > flash it tomorrow).
> >
> > The simples work-around in oe-core:
> > http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=054c72716f0648ff2f0770c1010857596442223e
> > or as bbappend:
> > https://github.com/webOS-ports/meta-webos-ports/commit/020d31892780b69fa64a3a6a146283e831ccd2e9

If you need to use this, then add "export" as well which is needed
because of the "unset" in the main recipe (wasn't needed when I had both
these changes included in the build at first).
https://github.com/webOS-ports/meta-webos-ports/commit/0e737ad167a14591e73ca3fa47d31108038ec3e2

> > this is what I'm testing with now, notice that setting just LDFLAGS
> > isn't enough, because LDFLAGS are unset in glibc.do_compile
> >
> > I don't know if it's worth spending more time trying to get the glibc
> > working with gold, it looks like:
> 
> i agree, your change to force bfd linker is acceptable approach
> please propose it to OE-Core directly.

Send for oe-core:
https://patchwork.openembedded.org/patch/159874/
but more information might be needed before it gets applied to oe-core
as discussed with RP. I'll try to get serial cable working on my Nexus 5
to test it there in runtime, but maybe someone could beat me to it with
some more easily hackable 32bit arm board.

> 
> > http://git.openembedded.org/openembedded-core/commit/?id=7bbc453ac2b8d63680855789948a145fc448017f
> > resolved the linker script issue (I was seeing the same error with
> > oe-core 5d0307fbef just this fix) reported here:
> > https://sourceware.org/bugzilla/show_bug.cgi?id=24148
> >
> > but there are other bugs like:
> > https://sourceware.org/bugzilla/show_bug.cgi?id=10937
> > https://sourceware.org/bugzilla/show_bug.cgi?id=18103
> > e.g. gentoo is also explicitly disabling gold for glibc builds:
> > https://github.com/gentoo/gentoo/blob/16e1c37bdc84b2ac5df10511f63e478d8e49435e/sys-libs/glibc/glibc-2.29-r1.ebuild#L767
> >
> > Can someone confirm that glibc built with gold works for him on arm
> > (I haven't seen this on x86, aarch64)? Or should we just force bfd in
> > oe-core recipe?
> >
> > --
> > Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: glibc broken when linked with gold Was: [oe] State of OE World, 2019-03-16

[Trevor Woerner]

On Thu 2019-03-28 @ 03:40:33 PM, Martin Jansa wrote:
> Send for oe-core:
> https://patchwork.openembedded.org/patch/159874/
> but more information might be needed before it gets applied to oe-core
> as discussed with RP. I'll try to get serial cable working on my Nexus 5
> to test it there in runtime, but maybe someone could beat me to it with
> some more easily hackable 32bit arm board.
> 
> > > Can someone confirm that glibc built with gold works for him on arm
> > > (I haven't seen this on x86, aarch64)? Or should we just force bfd in
> > > oe-core recipe?

I have an Olimex imx233-olinuxino-maxi that I build for and test regularly
(arm926ej-s, ARMv5).

Yesterday I performed two builds from master with the following configuration:

	Build Configuration:
	BB_VERSION           = "1.40.0"
	BUILD_SYS            = "x86_64-linux"
	NATIVELSBSTRING      = "opensuseleap-15.0"
	TARGET_SYS           = "arm-oe-linux-gnueabi"
	MACHINE              = "imx233-olinuxino-maxi"
	DISTRO               = "nodistro"
	DISTRO_VERSION       = "nodistro.0"
	TUNE_FEATURES        = "arm armv5 thumb dsp"
	TARGET_FPU           = "soft"
	meta-freescale       = "master:08173f472244817b85fe4af8ab2add7a0090b122"
	meta-fsl-arm-extra   = "master:b4d2c8462220e7038adffc0a151767ef8da07c5d"
	meta                 = "master:aeaca9bb1b1c8bf44818945dc4b2cbd6d4b5cef2"
	meta-oe
	meta-python
	meta-networking      = "master:24dd9328f7ec964bbad25b7061366f9f2ff6194d"

Both builds explicitly included gobject-introspection.

The first build was done as-is (i.e. using the bfd linker). The second build
had the following in conf/local.conf:

	DISTRO_FEATURES_append = " ld-is-gold"

The default build succeeded. The gold build failed as expected while running
qemu trying to build gobject-introspection.

The good build boots on the hardware and runs fine (as far as I can tell):

	OpenEmbedded nodistro.0 imx233-olinuxino-maxi ttyAMA0

	imx233-olinuxino-maxi login: root
	root@imx233-olinuxino-maxi:~# uname -a
	Linux imx233-olinuxino-maxi 5.0.3-fslc+g20d8efe36350 #1 Thu Mar 28 17:09:20 UTC 2019 armv5tejl armv5tejl armv5tejl GNU/Linux
	root@imx233-olinuxino-maxi:~# g-ir-compiler --help
	Usage:
	  g-ir-compiler [OPTION?]

	Help Options:
	  -h, --help                    Show help options

	Application Options:
	  --includedir                  include directories in GIR search path
	  -o, --output=FILE             output file
	  -m, --module=NAME             module to compile
	  -l, --shared-library=FILE     shared library
	  --debug                       show debug messages
	  --verbose                     show verbose messages

Re: glibc broken when linked with gold Was: [oe] State of OE World, 2019-03-16

[Trevor Woerner]

An image built with gold fails on the hardware.

I was able to build core-image-minimal with gold enabled:

configuration:
	Build Configuration:
	BB_VERSION           = "1.40.0"
	BUILD_SYS            = "x86_64-linux"
	NATIVELSBSTRING      = "opensuseleap-15.0"
	TARGET_SYS           = "arm-oe-linux-gnueabi"
	MACHINE              = "imx233-olinuxino-maxi"
	DISTRO               = "nodistro"
	DISTRO_VERSION       = "nodistro.0"
	TUNE_FEATURES        = "arm armv5 thumb dsp"
	TARGET_FPU           = "soft"
	meta-fsl-arm-extra   = "master:b4d2c8462220e7038adffc0a151767ef8da07c5d"
	meta-freescale       = "master:08173f472244817b85fe4af8ab2add7a0090b122"
	meta                 = "master:aeaca9bb1b1c8bf44818945dc4b2cbd6d4b5cef2"

conf/local.conf addition:
	DISTRO_FEATURES_append = " ld-is-gold"

When run, the kernel is unable to run init, and panics:

	[    4.050000] Freeing unused kernel memory: 308K
	[    4.050000] This architecture does not have kernel memory protection.
	[    4.060000] Run /sbin/init as init process
	[    4.120000] hub 1-1:1.0: USB hub found
	[    4.130000] hub 1-1:1.0: 3 ports detected
	[    4.140000] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
	[    4.140000] CPU: 0 PID: 1 Comm: init Not tainted 5.0.3-fslc+g20d8efe36350 #1
	[    4.140000] Hardware name: Freescale MXS (Device Tree)
	[    4.140000] [<c0010df8>] (unwind_backtrace) from [<c000e730>] (show_stack+0x10/0x14)
	[    4.140000] [<c000e730>] (show_stack) from [<c001a9bc>] (panic+0xec/0x2d8)
	[    4.140000] [<c001a9bc>] (panic) from [<c001d3b0>] (do_exit+0xa44/0xaa4)
	[    4.140000] [<c001d3b0>] (do_exit) from [<c001e65c>] (do_group_exit+0x38/0xbc)
	[    4.140000] [<c001e65c>] (do_group_exit) from [<c002a1c0>] (get_signal+0x110/0x98c)
	[    4.140000] [<c002a1c0>] (get_signal) from [<c000dd6c>] (do_work_pending+0x120/0x5fc)
	[    4.140000] [<c000dd6c>] (do_work_pending) from [<c0009034>] (slow_work_pending+0xc/0x20)
	[    4.140000] Exception stack(0xc343dfb0 to 0xc343dff8)
	[    4.140000] dfa0:                                     bebd7f10 484113c4 00000000 48431fd0
	[    4.140000] dfc0: 00000934 00000000 00000000 00000000 00000000 00000000 00021504 bebd7f0c
	[    4.140000] dfe0: 48431fd0 bebd7ed0 48410a40 484113f4 00000010 ffffffff
	[    4.140000] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---