[Buildroot] Shared objects, load order?

[Buildroot] Shared objects, load order?

[Joachim Pihl]

I only have one hurdle left before the upgrade from 0.10.0 to 2009.11 is  
in place. It may be related to .so's referring other .so's.

In the beginning, I thought that _none_ of our C++ applications would run  
on the updated platform, they would all die with a segmentation fault  
before reaching main(). Then I discovered something interesting, namely  
that _one_ application actually runs. All the apps are production code,  
they run on the same hardware with the old buildroot system, so the fact  
that they do not all behave the same is intriguing. I think it may be  
caused by some .so referring something in another .so that has not been  
loaded yet. ldd shows that the working app and one of the crashing ones  
reference (in essence) the same .so's, but lists them in different order.

Works:

# ldd /multicon/gyda
	libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
	libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
	libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
	libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
	libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x401c4000)
	libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
	libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
	libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
	libm.so.0 => /lib/libm.so.0 (0x40376000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
	libc.so.0 => /lib/libc.so.0 (0x403a3000)
	libdl.so.0 => /lib/libdl.so.0 (0x40401000)
	librt.so.0 => /lib/librt.so.0 (0x4040c000)
	libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
	ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)


Crashes:

# ldd /multicon/phy
	libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x4000e000)
	libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
	libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
	libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
	libm.so.0 => /lib/libm.so.0 (0x401c0000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
	libc.so.0 => /lib/libc.so.0 (0x401ed000)
	libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
	librt.so.0 => /lib/librt.so.0 (0x40265000)
	libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
	libdl.so.0 => /lib/libdl.so.0 (0x40277000)
	ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)


Any changes in uclibc regarding this over the last two years? I have seen  
that linkers are sometimes fuzzy about the ordering of static libraries, I  
have never had a problem with shared ones.


-- 
Joachim Pihl
Senior R&D Engineer
Nevion Europe

Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com

------------------------------------------------------
The Global Video Transport Leader
------------------------------------------------------

[Buildroot] Shared objects, load order?

[Joachim Pihl]

On Wed, 02 Dec 2009 09:59:07 +0100, Lionel Landwerlin  
<llandwerlin@gmail.com> wrote:

> Hi Joachim,
>
> Did you try to run gdb on one of your application to see where it  
> crashes ?

No useful info found when running gdbserver on target, I believe it hasn't  
even reached main() yet when it crashes.

> I recently had a similar problem
> (http://sourceware.org/ml/crossgcc/2009-11/msg00120.html).
> In my case, my applications where crashing in a weak symbol in
> libdl.so which should had been
> replaced by the strong symbol of libpthread.so. Though, my
> applications were linked using -ldl -lpthread,
> libdl was loaded first and its weak symbol for the pthread_once
> function was still valid even after the
> libpthread load's.
> From my first investigations, it seems to be a problem with the way
> libraries are built (I had this problem
> with directfb-1.0, not with directfb-1.4).

I suspect libboost_thread-mt.so is referencing libpthread.so.0 without  
actually loading it, but right now I don't have the time to investigate.  
Critical bug fix time...

>
>>
>> Works:
>>
>> # ldd /multicon/gyda
>>        libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
>>        libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
>>        libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
>>        libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
>>        libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
>>        libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so  
>> (0x401c4000)
>>        libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
>>        libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
>>        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
>>        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
>>        libm.so.0 => /lib/libm.so.0 (0x40376000)
>>        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
>>        libc.so.0 => /lib/libc.so.0 (0x403a3000)
>>        libdl.so.0 => /lib/libdl.so.0 (0x40401000)
>>        librt.so.0 => /lib/librt.so.0 (0x4040c000)
>>        libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
>>        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>
>>
>> Crashes:
>>
>> # ldd /multicon/phy
>>        libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so  
>> (0x4000e000)
>>        libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
>>        libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
>>        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
>>        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
>>        libm.so.0 => /lib/libm.so.0 (0x401c0000)
>>        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
>>        libc.so.0 => /lib/libc.so.0 (0x401ed000)
>>        libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
>>        librt.so.0 => /lib/librt.so.0 (0x40265000)
>>        libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
>>        libdl.so.0 => /lib/libdl.so.0 (0x40277000)
>>        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>
>>


-- 
Joachim Pihl
Senior R&D Engineer
Nevion

Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com

-----------------------------------
The Global Video Transport Leader
-----------------------------------

[Buildroot] Shared objects, load order?

[Michael S. Zick]

On Wed December 2 2009, Joachim Pihl wrote:
> 
> I only have one hurdle left before the upgrade from 0.10.0 to 2009.11 is  
> in place. It may be related to .so's referring other .so's.
> 
> In the beginning, I thought that _none_ of our C++ applications would run  
> on the updated platform, they would all die with a segmentation fault  
> before reaching main(). Then I discovered something interesting, namely  
> that _one_ application actually runs. All the apps are production code,  
> they run on the same hardware with the old buildroot system, so the fact  
> that they do not all behave the same is intriguing. I think it may be  
> caused by some .so referring something in another .so that has not been  
> loaded yet. ldd shows that the working app and one of the crashing ones  
> reference (in essence) the same .so's, but lists them in different order.
> 
> Works:
> 
> # ldd /multicon/gyda
> 	libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
> 	libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
> 	libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
> 	libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
> 	libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
> 	libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x401c4000)
> 	libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
> 	libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
> 	libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
> 	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
> 	libm.so.0 => /lib/libm.so.0 (0x40376000)
> 	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
> 	libc.so.0 => /lib/libc.so.0 (0x403a3000)
> 	libdl.so.0 => /lib/libdl.so.0 (0x40401000)
> 	librt.so.0 => /lib/librt.so.0 (0x4040c000)
> 	libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
> 	ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
> 
> 
> Crashes:
> 
> # ldd /multicon/phy
> 	libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x4000e000)
> 	libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
> 	libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
> 	libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
> 	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
> 	libm.so.0 => /lib/libm.so.0 (0x401c0000)
> 	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
> 	libc.so.0 => /lib/libc.so.0 (0x401ed000)
> 	libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
> 	librt.so.0 => /lib/librt.so.0 (0x40265000)
> 	libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
> 	libdl.so.0 => /lib/libdl.so.0 (0x40277000)
> 	ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
> 
> 
> Any changes in uclibc regarding this over the last two years? I have seen  
> that linkers are sometimes fuzzy about the ordering of static libraries, I  
> have never had a problem with shared ones.
> 
> 

Did you enable sstrip?
The sstrip in the 2009.8 (and probably in 2009.11, since I never saw it fixed)
creates object (and statically linked) objects with corrupted headers.

Turn it off, rebuild, report back.

Mike

[Buildroot] Shared objects, load order?

[Joachim Pihl]

On Wed, 02 Dec 2009 10:04:31 +0100, Joachim Pihl <jpihl@nevion.com> wrote:
>
> I suspect libboost_thread-mt.so is referencing libpthread.so.0 without
> actually loading it, but right now I don't have the time to investigate.

I have now confirmed that the problem is related to the specified order of  
the libraries when linking. I changed the order for the failing  
applications to match the one app that worked and voil?! - they came back  
to life. I don't think this is good, cross development is hard enough as  
it is without these kinds of gotchas.

>
>>
>>>
>>> Works:
>>>
>>> # ldd /multicon/gyda
>>>        libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
>>>        libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
>>>        libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
>>>        libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
>>>        libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
>>>        libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>> (0x401c4000)
>>>        libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
>>>        libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
>>>        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
>>>        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
>>>        libm.so.0 => /lib/libm.so.0 (0x40376000)
>>>        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
>>>        libc.so.0 => /lib/libc.so.0 (0x403a3000)
>>>        libdl.so.0 => /lib/libdl.so.0 (0x40401000)
>>>        librt.so.0 => /lib/librt.so.0 (0x4040c000)
>>>        libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
>>>        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>
>>>
>>> Crashes:
>>>
>>> # ldd /multicon/phy
>>>        libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>> (0x4000e000)
>>>        libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
>>>        libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
>>>        libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
>>>        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
>>>        libm.so.0 => /lib/libm.so.0 (0x401c0000)
>>>        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
>>>        libc.so.0 => /lib/libc.so.0 (0x401ed000)
>>>        libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
>>>        librt.so.0 => /lib/librt.so.0 (0x40265000)
>>>        libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
>>>        libdl.so.0 => /lib/libdl.so.0 (0x40277000)
>>>        ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>
>>>
>
>


-- 
Joachim Pihl
Senior R&D Engineer
Nevion Europe

Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com

------------------------------------------------------
The Global Video Transport Leader
------------------------------------------------------

[Buildroot] Shared objects, load order?

[Michael S. Zick]

On Wed December 2 2009, Joachim Pihl wrote:
> On Wed, 02 Dec 2009 13:44:40 +0100, Michael S. Zick <minimod@morethan.org>  
> wrote:
> >
> > Did you enable sstrip?
> > The sstrip in the 2009.8 (and probably in 2009.11, since I never saw it  
> > fixed)
> > creates object (and statically linked) objects with corrupted headers.
> >
> > Turn it off, rebuild, report back.
> 
> BR2_PACKAGE_SSTRIP_HOST enabled, BR2_PACKAGE_SSTRIP_TARGET disabled, but  
> BR2_STRIP_strip is the one used.
> 
> Will rebuild anyway with BR2_PACKAGE_SSTRIP_HOST disabled, but I doubt it  
> should make any difference.
> 

The "file" command will report the corrupted headers if present.
Just "file" some old/new objects.

Mike

[Buildroot] Shared objects, load order?

[Lionel Landwerlin]

Yeah using LD_PRELOAD=/lib/libpthread.so solved my problems too...


On Wed, Dec 2, 2009 at 1:54 PM, Joachim Pihl <jpihl@nevion.com> wrote:
> On Wed, 02 Dec 2009 10:04:31 +0100, Joachim Pihl <jpihl@nevion.com> wrote:
>>
>> I suspect libboost_thread-mt.so is referencing libpthread.so.0 without
>> actually loading it, but right now I don't have the time to investigate.
>
> I have now confirmed that the problem is related to the specified order of
> the libraries when linking. I changed the order for the failing applications
> to match the one app that worked and voil?! - they came back to life. I
> don't think this is good, cross development is hard enough as it is without
> these kinds of gotchas.
>
>>
>>>
>>>>
>>>> Works:
>>>>
>>>> # ldd /multicon/gyda
>>>> ? ? ? libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
>>>> ? ? ? libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
>>>> ? ? ? libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
>>>> ? ? ? libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
>>>> ? ? ? libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
>>>> ? ? ? libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>>> (0x401c4000)
>>>> ? ? ? libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
>>>> ? ? ? libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
>>>> ? ? ? libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
>>>> ? ? ? libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
>>>> ? ? ? libm.so.0 => /lib/libm.so.0 (0x40376000)
>>>> ? ? ? libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
>>>> ? ? ? libc.so.0 => /lib/libc.so.0 (0x403a3000)
>>>> ? ? ? libdl.so.0 => /lib/libdl.so.0 (0x40401000)
>>>> ? ? ? librt.so.0 => /lib/librt.so.0 (0x4040c000)
>>>> ? ? ? libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
>>>> ? ? ? ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>>
>>>>
>>>> Crashes:
>>>>
>>>> # ldd /multicon/phy
>>>> ? ? ? libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>>> (0x4000e000)
>>>> ? ? ? libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
>>>> ? ? ? libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
>>>> ? ? ? libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
>>>> ? ? ? libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
>>>> ? ? ? libm.so.0 => /lib/libm.so.0 (0x401c0000)
>>>> ? ? ? libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
>>>> ? ? ? libc.so.0 => /lib/libc.so.0 (0x401ed000)
>>>> ? ? ? libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
>>>> ? ? ? librt.so.0 => /lib/librt.so.0 (0x40265000)
>>>> ? ? ? libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
>>>> ? ? ? libdl.so.0 => /lib/libdl.so.0 (0x40277000)
>>>> ? ? ? ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>>
>>>>
>>
>>
>
>
> --
> Joachim Pihl
> Senior R&D Engineer
> Nevion Europe
>
> Tel: +47 33 48 94 66
> Fax: +47 33 48 99 98
> Mobile: +47 91 33 98 91
> jpihl at nevion.com
> www.nevion.com
>
> ------------------------------------------------------
> The Global Video Transport Leader
> ------------------------------------------------------
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] Shared objects, load order?

[Joachim Pihl]

On Wed, 02 Dec 2009 13:44:40 +0100, Michael S. Zick <minimod@morethan.org>  
wrote:
>
> Did you enable sstrip?
> The sstrip in the 2009.8 (and probably in 2009.11, since I never saw it  
> fixed)
> creates object (and statically linked) objects with corrupted headers.
>
> Turn it off, rebuild, report back.

BR2_PACKAGE_SSTRIP_HOST enabled, BR2_PACKAGE_SSTRIP_TARGET disabled, but  
BR2_STRIP_strip is the one used.

Will rebuild anyway with BR2_PACKAGE_SSTRIP_HOST disabled, but I doubt it  
should make any difference.

-- 
Joachim Pihl
Senior R&D Engineer
Nevion Europe

Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com

------------------------------------------------------
The Global Video Transport Leader
------------------------------------------------------