* [Buildroot] Shared objects, load order?
@ 2009-12-02 7:16 Joachim Pihl
[not found] ` <4d28d4b50912020059t6ae52e8bx7b0f76db774e4cdd@mail.gmail.com>
2009-12-02 12:44 ` Michael S. Zick
0 siblings, 2 replies; 7+ messages in thread
From: Joachim Pihl @ 2009-12-02 7:16 UTC (permalink / raw)
To: buildroot
I only have one hurdle left before the upgrade from 0.10.0 to 2009.11 is
in place. It may be related to .so's referring other .so's.
In the beginning, I thought that _none_ of our C++ applications would run
on the updated platform, they would all die with a segmentation fault
before reaching main(). Then I discovered something interesting, namely
that _one_ application actually runs. All the apps are production code,
they run on the same hardware with the old buildroot system, so the fact
that they do not all behave the same is intriguing. I think it may be
caused by some .so referring something in another .so that has not been
loaded yet. ldd shows that the working app and one of the crashing ones
reference (in essence) the same .so's, but lists them in different order.
Works:
# ldd /multicon/gyda
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x401c4000)
libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
libm.so.0 => /lib/libm.so.0 (0x40376000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
libc.so.0 => /lib/libc.so.0 (0x403a3000)
libdl.so.0 => /lib/libdl.so.0 (0x40401000)
librt.so.0 => /lib/librt.so.0 (0x4040c000)
libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
Crashes:
# ldd /multicon/phy
libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x4000e000)
libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
libm.so.0 => /lib/libm.so.0 (0x401c0000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
libc.so.0 => /lib/libc.so.0 (0x401ed000)
libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
librt.so.0 => /lib/librt.so.0 (0x40265000)
libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
libdl.so.0 => /lib/libdl.so.0 (0x40277000)
ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
Any changes in uclibc regarding this over the last two years? I have seen
that linkers are sometimes fuzzy about the ordering of static libraries, I
have never had a problem with shared ones.
--
Joachim Pihl
Senior R&D Engineer
Nevion Europe
Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com
------------------------------------------------------
The Global Video Transport Leader
------------------------------------------------------
^ permalink raw reply [flat|nested] 7+ messages in thread[parent not found: <4d28d4b50912020059t6ae52e8bx7b0f76db774e4cdd@mail.gmail.com>]
* [Buildroot] Shared objects, load order?
[not found] ` <4d28d4b50912020059t6ae52e8bx7b0f76db774e4cdd@mail.gmail.com>
@ 2009-12-02 9:04 ` Joachim Pihl
2009-12-02 12:54 ` Joachim Pihl
0 siblings, 1 reply; 7+ messages in thread
From: Joachim Pihl @ 2009-12-02 9:04 UTC (permalink / raw)
To: buildroot
On Wed, 02 Dec 2009 09:59:07 +0100, Lionel Landwerlin
<llandwerlin@gmail.com> wrote:
> Hi Joachim,
>
> Did you try to run gdb on one of your application to see where it
> crashes ?
No useful info found when running gdbserver on target, I believe it hasn't
even reached main() yet when it crashes.
> I recently had a similar problem
> (http://sourceware.org/ml/crossgcc/2009-11/msg00120.html).
> In my case, my applications where crashing in a weak symbol in
> libdl.so which should had been
> replaced by the strong symbol of libpthread.so. Though, my
> applications were linked using -ldl -lpthread,
> libdl was loaded first and its weak symbol for the pthread_once
> function was still valid even after the
> libpthread load's.
> From my first investigations, it seems to be a problem with the way
> libraries are built (I had this problem
> with directfb-1.0, not with directfb-1.4).
I suspect libboost_thread-mt.so is referencing libpthread.so.0 without
actually loading it, but right now I don't have the time to investigate.
Critical bug fix time...
>
>>
>> Works:
>>
>> # ldd /multicon/gyda
>> libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
>> libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
>> libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
>> libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
>> libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
>> libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>> (0x401c4000)
>> libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
>> libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
>> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
>> libm.so.0 => /lib/libm.so.0 (0x40376000)
>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
>> libc.so.0 => /lib/libc.so.0 (0x403a3000)
>> libdl.so.0 => /lib/libdl.so.0 (0x40401000)
>> librt.so.0 => /lib/librt.so.0 (0x4040c000)
>> libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
>> ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>
>>
>> Crashes:
>>
>> # ldd /multicon/phy
>> libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>> (0x4000e000)
>> libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
>> libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
>> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
>> libm.so.0 => /lib/libm.so.0 (0x401c0000)
>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
>> libc.so.0 => /lib/libc.so.0 (0x401ed000)
>> libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
>> librt.so.0 => /lib/librt.so.0 (0x40265000)
>> libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
>> libdl.so.0 => /lib/libdl.so.0 (0x40277000)
>> ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>
>>
--
Joachim Pihl
Senior R&D Engineer
Nevion
Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com
-----------------------------------
The Global Video Transport Leader
-----------------------------------
^ permalink raw reply [flat|nested] 7+ messages in thread* [Buildroot] Shared objects, load order?
2009-12-02 9:04 ` Joachim Pihl
@ 2009-12-02 12:54 ` Joachim Pihl
2009-12-02 13:04 ` Lionel Landwerlin
0 siblings, 1 reply; 7+ messages in thread
From: Joachim Pihl @ 2009-12-02 12:54 UTC (permalink / raw)
To: buildroot
On Wed, 02 Dec 2009 10:04:31 +0100, Joachim Pihl <jpihl@nevion.com> wrote:
>
> I suspect libboost_thread-mt.so is referencing libpthread.so.0 without
> actually loading it, but right now I don't have the time to investigate.
I have now confirmed that the problem is related to the specified order of
the libraries when linking. I changed the order for the failing
applications to match the one app that worked and voil?! - they came back
to life. I don't think this is good, cross development is hard enough as
it is without these kinds of gotchas.
>
>>
>>>
>>> Works:
>>>
>>> # ldd /multicon/gyda
>>> libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
>>> libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
>>> libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
>>> libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
>>> libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
>>> libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>> (0x401c4000)
>>> libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
>>> libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
>>> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
>>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
>>> libm.so.0 => /lib/libm.so.0 (0x40376000)
>>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
>>> libc.so.0 => /lib/libc.so.0 (0x403a3000)
>>> libdl.so.0 => /lib/libdl.so.0 (0x40401000)
>>> librt.so.0 => /lib/librt.so.0 (0x4040c000)
>>> libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
>>> ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>
>>>
>>> Crashes:
>>>
>>> # ldd /multicon/phy
>>> libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>> (0x4000e000)
>>> libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
>>> libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
>>> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
>>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
>>> libm.so.0 => /lib/libm.so.0 (0x401c0000)
>>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
>>> libc.so.0 => /lib/libc.so.0 (0x401ed000)
>>> libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
>>> librt.so.0 => /lib/librt.so.0 (0x40265000)
>>> libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
>>> libdl.so.0 => /lib/libdl.so.0 (0x40277000)
>>> ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>
>>>
>
>
--
Joachim Pihl
Senior R&D Engineer
Nevion Europe
Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com
------------------------------------------------------
The Global Video Transport Leader
------------------------------------------------------
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] Shared objects, load order?
2009-12-02 12:54 ` Joachim Pihl
@ 2009-12-02 13:04 ` Lionel Landwerlin
0 siblings, 0 replies; 7+ messages in thread
From: Lionel Landwerlin @ 2009-12-02 13:04 UTC (permalink / raw)
To: buildroot
Yeah using LD_PRELOAD=/lib/libpthread.so solved my problems too...
On Wed, Dec 2, 2009 at 1:54 PM, Joachim Pihl <jpihl@nevion.com> wrote:
> On Wed, 02 Dec 2009 10:04:31 +0100, Joachim Pihl <jpihl@nevion.com> wrote:
>>
>> I suspect libboost_thread-mt.so is referencing libpthread.so.0 without
>> actually loading it, but right now I don't have the time to investigate.
>
> I have now confirmed that the problem is related to the specified order of
> the libraries when linking. I changed the order for the failing applications
> to match the one app that worked and voil?! - they came back to life. I
> don't think this is good, cross development is hard enough as it is without
> these kinds of gotchas.
>
>>
>>>
>>>>
>>>> Works:
>>>>
>>>> # ldd /multicon/gyda
>>>> ? ? ? libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
>>>> ? ? ? libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
>>>> ? ? ? libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
>>>> ? ? ? libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
>>>> ? ? ? libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
>>>> ? ? ? libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>>> (0x401c4000)
>>>> ? ? ? libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
>>>> ? ? ? libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
>>>> ? ? ? libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
>>>> ? ? ? libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
>>>> ? ? ? libm.so.0 => /lib/libm.so.0 (0x40376000)
>>>> ? ? ? libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
>>>> ? ? ? libc.so.0 => /lib/libc.so.0 (0x403a3000)
>>>> ? ? ? libdl.so.0 => /lib/libdl.so.0 (0x40401000)
>>>> ? ? ? librt.so.0 => /lib/librt.so.0 (0x4040c000)
>>>> ? ? ? libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
>>>> ? ? ? ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>>
>>>>
>>>> Crashes:
>>>>
>>>> # ldd /multicon/phy
>>>> ? ? ? libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so
>>>> (0x4000e000)
>>>> ? ? ? libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
>>>> ? ? ? libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
>>>> ? ? ? libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
>>>> ? ? ? libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
>>>> ? ? ? libm.so.0 => /lib/libm.so.0 (0x401c0000)
>>>> ? ? ? libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
>>>> ? ? ? libc.so.0 => /lib/libc.so.0 (0x401ed000)
>>>> ? ? ? libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
>>>> ? ? ? librt.so.0 => /lib/librt.so.0 (0x40265000)
>>>> ? ? ? libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
>>>> ? ? ? libdl.so.0 => /lib/libdl.so.0 (0x40277000)
>>>> ? ? ? ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>>>>
>>>>
>>
>>
>
>
> --
> Joachim Pihl
> Senior R&D Engineer
> Nevion Europe
>
> Tel: +47 33 48 94 66
> Fax: +47 33 48 99 98
> Mobile: +47 91 33 98 91
> jpihl at nevion.com
> www.nevion.com
>
> ------------------------------------------------------
> The Global Video Transport Leader
> ------------------------------------------------------
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] Shared objects, load order?
2009-12-02 7:16 [Buildroot] Shared objects, load order? Joachim Pihl
[not found] ` <4d28d4b50912020059t6ae52e8bx7b0f76db774e4cdd@mail.gmail.com>
@ 2009-12-02 12:44 ` Michael S. Zick
2009-12-02 13:04 ` Joachim Pihl
1 sibling, 1 reply; 7+ messages in thread
From: Michael S. Zick @ 2009-12-02 12:44 UTC (permalink / raw)
To: buildroot
On Wed December 2 2009, Joachim Pihl wrote:
>
> I only have one hurdle left before the upgrade from 0.10.0 to 2009.11 is
> in place. It may be related to .so's referring other .so's.
>
> In the beginning, I thought that _none_ of our C++ applications would run
> on the updated platform, they would all die with a segmentation fault
> before reaching main(). Then I discovered something interesting, namely
> that _one_ application actually runs. All the apps are production code,
> they run on the same hardware with the old buildroot system, so the fact
> that they do not all behave the same is intriguing. I think it may be
> caused by some .so referring something in another .so that has not been
> loaded yet. ldd shows that the working app and one of the crashing ones
> reference (in essence) the same .so's, but lists them in different order.
>
> Works:
>
> # ldd /multicon/gyda
> libxml2.so.2 => /usr/lib/libxml2.so.2 (0x4000e000)
> libgd.so.2 => /usr/lib/libgd.so.2 (0x4012b000)
> libpng12.so.0 => /usr/lib/libpng12.so.0 (0x4016b000)
> libpthread.so.0 => /lib/libpthread.so.0 (0x40191000)
> libz.so.1 => /usr/lib/libz.so.1 (0x401ab000)
> libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x401c4000)
> libspread.so.2 => /usr/lib/libspread.so.2 (0x401d8000)
> libcrypt.so.0 => /lib/libcrypt.so.0 (0x4023f000)
> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x4025c000)
> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x402c5000)
> libm.so.0 => /lib/libm.so.0 (0x40376000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40390000)
> libc.so.0 => /lib/libc.so.0 (0x403a3000)
> libdl.so.0 => /lib/libdl.so.0 (0x40401000)
> librt.so.0 => /lib/librt.so.0 (0x4040c000)
> libnsl.so.0 => /lib/libnsl.so.0 (0x40415000)
> ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>
>
> Crashes:
>
> # ldd /multicon/phy
> libboost_thread-mt.so => /usr/lib/libboost_thread-mt.so (0x4000e000)
> libspread.so.2 => /usr/lib/libspread.so.2 (0x40022000)
> libcrypt.so.0 => /lib/libcrypt.so.0 (0x40089000)
> libsqlite3.so.0 => /usr/lib/libsqlite3.so.0 (0x400a6000)
> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x4010f000)
> libm.so.0 => /lib/libm.so.0 (0x401c0000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x401da000)
> libc.so.0 => /lib/libc.so.0 (0x401ed000)
> libpthread.so.0 => /lib/libpthread.so.0 (0x4024b000)
> librt.so.0 => /lib/librt.so.0 (0x40265000)
> libnsl.so.0 => /lib/libnsl.so.0 (0x4026e000)
> libdl.so.0 => /lib/libdl.so.0 (0x40277000)
> ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x40000000)
>
>
> Any changes in uclibc regarding this over the last two years? I have seen
> that linkers are sometimes fuzzy about the ordering of static libraries, I
> have never had a problem with shared ones.
>
>
Did you enable sstrip?
The sstrip in the 2009.8 (and probably in 2009.11, since I never saw it fixed)
creates object (and statically linked) objects with corrupted headers.
Turn it off, rebuild, report back.
Mike
^ permalink raw reply [flat|nested] 7+ messages in thread* [Buildroot] Shared objects, load order?
2009-12-02 12:44 ` Michael S. Zick
@ 2009-12-02 13:04 ` Joachim Pihl
2009-12-02 13:03 ` Michael S. Zick
0 siblings, 1 reply; 7+ messages in thread
From: Joachim Pihl @ 2009-12-02 13:04 UTC (permalink / raw)
To: buildroot
On Wed, 02 Dec 2009 13:44:40 +0100, Michael S. Zick <minimod@morethan.org>
wrote:
>
> Did you enable sstrip?
> The sstrip in the 2009.8 (and probably in 2009.11, since I never saw it
> fixed)
> creates object (and statically linked) objects with corrupted headers.
>
> Turn it off, rebuild, report back.
BR2_PACKAGE_SSTRIP_HOST enabled, BR2_PACKAGE_SSTRIP_TARGET disabled, but
BR2_STRIP_strip is the one used.
Will rebuild anyway with BR2_PACKAGE_SSTRIP_HOST disabled, but I doubt it
should make any difference.
--
Joachim Pihl
Senior R&D Engineer
Nevion Europe
Tel: +47 33 48 94 66
Fax: +47 33 48 99 98
Mobile: +47 91 33 98 91
jpihl at nevion.com
www.nevion.com
------------------------------------------------------
The Global Video Transport Leader
------------------------------------------------------
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Buildroot] Shared objects, load order?
2009-12-02 13:04 ` Joachim Pihl
@ 2009-12-02 13:03 ` Michael S. Zick
0 siblings, 0 replies; 7+ messages in thread
From: Michael S. Zick @ 2009-12-02 13:03 UTC (permalink / raw)
To: buildroot
On Wed December 2 2009, Joachim Pihl wrote:
> On Wed, 02 Dec 2009 13:44:40 +0100, Michael S. Zick <minimod@morethan.org>
> wrote:
> >
> > Did you enable sstrip?
> > The sstrip in the 2009.8 (and probably in 2009.11, since I never saw it
> > fixed)
> > creates object (and statically linked) objects with corrupted headers.
> >
> > Turn it off, rebuild, report back.
>
> BR2_PACKAGE_SSTRIP_HOST enabled, BR2_PACKAGE_SSTRIP_TARGET disabled, but
> BR2_STRIP_strip is the one used.
>
> Will rebuild anyway with BR2_PACKAGE_SSTRIP_HOST disabled, but I doubt it
> should make any difference.
>
The "file" command will report the corrupted headers if present.
Just "file" some old/new objects.
Mike
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2009-12-02 13:04 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-12-02 7:16 [Buildroot] Shared objects, load order? Joachim Pihl
[not found] ` <4d28d4b50912020059t6ae52e8bx7b0f76db774e4cdd@mail.gmail.com>
2009-12-02 9:04 ` Joachim Pihl
2009-12-02 12:54 ` Joachim Pihl
2009-12-02 13:04 ` Lionel Landwerlin
2009-12-02 12:44 ` Michael S. Zick
2009-12-02 13:04 ` Joachim Pihl
2009-12-02 13:03 ` Michael S. Zick
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox