[Buildroot] [PATCH] autossh: honour LDFLAGS

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Waldemar Brodkorb]

Fixes:
 http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973

Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
---
It is indeed a change from uClibc-ng 1.0.18 and upward, but I think
it is the fault of the package not respecting LDFLAGS and so not
passing -static. I am unsure why it just worked in the past.
When it fails linker is doing following:
-lnsl -lgcc -lc -lgcc
When -static is used, following is passed:
-lnsl --start-group -lgcc -lc --end-group 

The gcc -dumpspecs from both gcc's are the same. I have no idea why
gcc behaves in another way when toolchain is compiled with 1.0.17.
---
 package/autossh/0001-honour-LDFLAGS.patch | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 package/autossh/0001-honour-LDFLAGS.patch

diff --git a/package/autossh/0001-honour-LDFLAGS.patch b/package/autossh/0001-honour-LDFLAGS.patch
new file mode 100644
index 0000000..2e6160b
--- /dev/null
+++ b/package/autossh/0001-honour-LDFLAGS.patch
@@ -0,0 +1,16 @@
+Honour LDFLAGS, especially required when static linking
+
+Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
+
+diff -Nur autossh-1.4e.orig/Makefile.in autossh-1.4e/Makefile.in
+--- autossh-1.4e.orig/Makefile.in	2015-02-10 05:41:58.000000000 +0100
++++ autossh-1.4e/Makefile.in	2016-11-25 12:49:06.052122855 +0100
+@@ -31,7 +31,7 @@
+ 
+ 
+ $(TARGET):	$(OFILES)
+-		$(CC) $(CPPFLAGS) -o $(TARGET) $(OFILES) $(LIBS)
++		$(CC) $(CPPFLAGS) $(LDFLAGS) -o $(TARGET) $(OFILES) $(LIBS)
+ 
+ clean:
+ 		- /bin/rm -f *.o *.a *.core *~
-- 
2.1.4

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Thomas Petazzoni]

Hello,

On Fri, 25 Nov 2016 19:00:16 +0100, Waldemar Brodkorb wrote:
> Fixes:
>  http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973
> 
> Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
> ---
> It is indeed a change from uClibc-ng 1.0.18 and upward, but I think
> it is the fault of the package not respecting LDFLAGS and so not
> passing -static. I am unsure why it just worked in the past.
> When it fails linker is doing following:
> -lnsl -lgcc -lc -lgcc
> When -static is used, following is passed:
> -lnsl --start-group -lgcc -lc --end-group 

Applied to master.

The difference in the gcc behavior is due to gcc/config/gnu-user.h:

#define LINK_GCC_C_SEQUENCE_SPEC \
  "%{static:--start-group} %G %L %{static:--end-group}%{!static:%G}"

So, when you're linking dynamic, you have:

	-lgcc -lc -lgcc

And when you're linking statically, you have:

	--start-group -lgcc -lc --end-group

The --start-group/--end-group ask the linker to loop between -lgcc and
-lc until all unresolved symbols have been resolved. So
dl_iterate_phdr() being defined in the C library, but used in libgcc, I
guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

However, this interestingly doesn't happen for other architectures such
as ARM. Maybe dl_iterate_phdr is not used?

In any case, could you try to send your patch upstream to the autossh
developers?

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Max Filippov]

On Mon, Nov 28, 2016 at 1:28 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> On Fri, 25 Nov 2016 19:00:16 +0100, Waldemar Brodkorb wrote:
>> Fixes:
>>  http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973
>>
>> Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
>> ---
>> It is indeed a change from uClibc-ng 1.0.18 and upward, but I think
>> it is the fault of the package not respecting LDFLAGS and so not
>> passing -static. I am unsure why it just worked in the past.
>> When it fails linker is doing following:
>> -lnsl -lgcc -lc -lgcc
>> When -static is used, following is passed:
>> -lnsl --start-group -lgcc -lc --end-group
>
> Applied to master.
>
> The difference in the gcc behavior is due to gcc/config/gnu-user.h:
>
> #define LINK_GCC_C_SEQUENCE_SPEC \
>   "%{static:--start-group} %G %L %{static:--end-group}%{!static:%G}"
>
> So, when you're linking dynamic, you have:
>
>         -lgcc -lc -lgcc
>
> And when you're linking statically, you have:
>
>         --start-group -lgcc -lc --end-group
>
> The --start-group/--end-group ask the linker to loop between -lgcc and
> -lc until all unresolved symbols have been resolved. So
> dl_iterate_phdr() being defined in the C library, but used in libgcc, I
> guess the dynamic way (-lgcc -lc -lgcc) doesn't work.
>
> However, this interestingly doesn't happen for other architectures such
> as ARM. Maybe dl_iterate_phdr is not used?

ARM libgcc has its own custom unwinder: unwind-arm.c, which doesn't
reference dl_iteratre_phdr, and it doesn't use unwind-dw2-fde-dip.c, which does.
But it is reproducible with MIPS.

-- 
Thanks.
-- Max

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Waldemar Brodkorb]

Hi Thomas,
Thomas Petazzoni wrote,

> Hello,
> 
> On Fri, 25 Nov 2016 19:00:16 +0100, Waldemar Brodkorb wrote:
> > Fixes:
> >  http://autobuild.buildroot.net/results/08a458572a2e9c599dd32b837b1e5c02a6721973
> > 
> > Signed-off-by: Waldemar Brodkorb <wbx@openadk.org>
> > ---
> > It is indeed a change from uClibc-ng 1.0.18 and upward, but I think
> > it is the fault of the package not respecting LDFLAGS and so not
> > passing -static. I am unsure why it just worked in the past.
> > When it fails linker is doing following:
> > -lnsl -lgcc -lc -lgcc
> > When -static is used, following is passed:
> > -lnsl --start-group -lgcc -lc --end-group 
> 
> Applied to master.
> 
> The difference in the gcc behavior is due to gcc/config/gnu-user.h:
> 
> #define LINK_GCC_C_SEQUENCE_SPEC \
>   "%{static:--start-group} %G %L %{static:--end-group}%{!static:%G}"
> 
> So, when you're linking dynamic, you have:
> 
> 	-lgcc -lc -lgcc
> 
> And when you're linking statically, you have:
> 
> 	--start-group -lgcc -lc --end-group
> 
> The --start-group/--end-group ask the linker to loop between -lgcc and
> -lc until all unresolved symbols have been resolved. So
> dl_iterate_phdr() being defined in the C library, but used in libgcc, I
> guess the dynamic way (-lgcc -lc -lgcc) doesn't work.

But why gcc behaves different?
BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required
BR+gcc5+musl+static - no extra -static in LDFLAGS required
BR+gcc5+uClibc-ng-1.0.19 - extra -static required!
 
> In any case, could you try to send your patch upstream to the autossh
> developers?

Already done.

best regards
 Waldemar

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Thomas Petazzoni]

Hello,

On Tue, 29 Nov 2016 05:16:55 +0100, Waldemar Brodkorb wrote:

> > The --start-group/--end-group ask the linker to loop between -lgcc and
> > -lc until all unresolved symbols have been resolved. So
> > dl_iterate_phdr() being defined in the C library, but used in libgcc, I
> > guess the dynamic way (-lgcc -lc -lgcc) doesn't work.  
> 
> But why gcc behaves different?
> BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required
> BR+gcc5+musl+static - no extra -static in LDFLAGS required
> BR+gcc5+uClibc-ng-1.0.19 - extra -static required!

I would suspect it's a fallout of the "merge everything in libc"
change, but I really can't figure out why that would make a difference.

Could it be that with older uClibc versions, gcc was not finding the
dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind
code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr
symbol, enables unwinding support, which cause this extra reference
from libgcc to libc?

You would have to compare the config.log of libgcc for
BR+gcc5+uClibc-ng-1.0.17 and BR+gcc5+uClibc-ng-1.0.19 to see what's the
difference.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Max Filippov]

On Tue, Nov 29, 2016 at 12:43 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> On Tue, 29 Nov 2016 05:16:55 +0100, Waldemar Brodkorb wrote:
>> > The --start-group/--end-group ask the linker to loop between -lgcc and
>> > -lc until all unresolved symbols have been resolved. So
>> > dl_iterate_phdr() being defined in the C library, but used in libgcc, I
>> > guess the dynamic way (-lgcc -lc -lgcc) doesn't work.
>>
>> But why gcc behaves different?
>> BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required
>> BR+gcc5+musl+static - no extra -static in LDFLAGS required
>> BR+gcc5+uClibc-ng-1.0.19 - extra -static required!
>
> I would suspect it's a fallout of the "merge everything in libc"
> change, but I really can't figure out why that would make a difference.
>
> Could it be that with older uClibc versions, gcc was not finding the
> dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind
> code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr
> symbol, enables unwinding support, which cause this extra reference
> from libgcc to libc?

No. The difference is in the __pthread_initialize_minimal function.
Prior to uclibc-1.0.18 release it used to work because a call to
__pthread_initialize_minimal from __uClibc_init was resolved to
a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c
which didn't reference pthread_unwind.
Now it is resolved to a function from
libpthread/linuxthreads/pthread.c, and that file pulls in other
unused functions, which reference __GI___pthread_unwind,
and all the way down to the dl_iterate_phdr.

More details:
 http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html

-- 
Thanks.
-- Max

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Waldemar Brodkorb]

Hi Max,
Max Filippov wrote,

> On Tue, Nov 29, 2016 at 12:43 AM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
> > On Tue, 29 Nov 2016 05:16:55 +0100, Waldemar Brodkorb wrote:
> >> > The --start-group/--end-group ask the linker to loop between -lgcc and
> >> > -lc until all unresolved symbols have been resolved. So
> >> > dl_iterate_phdr() being defined in the C library, but used in libgcc, I
> >> > guess the dynamic way (-lgcc -lc -lgcc) doesn't work.
> >>
> >> But why gcc behaves different?
> >> BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required
> >> BR+gcc5+musl+static - no extra -static in LDFLAGS required
> >> BR+gcc5+uClibc-ng-1.0.19 - extra -static required!
> >
> > I would suspect it's a fallout of the "merge everything in libc"
> > change, but I really can't figure out why that would make a difference.
> >
> > Could it be that with older uClibc versions, gcc was not finding the
> > dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind
> > code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr
> > symbol, enables unwinding support, which cause this extra reference
> > from libgcc to libc?
> 
> No. The difference is in the __pthread_initialize_minimal function.
> Prior to uclibc-1.0.18 release it used to work because a call to
> __pthread_initialize_minimal from __uClibc_init was resolved to
> a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c
> which didn't reference pthread_unwind.
> Now it is resolved to a function from
> libpthread/linuxthreads/pthread.c, and that file pulls in other
> unused functions, which reference __GI___pthread_unwind,
> and all the way down to the dl_iterate_phdr.
> 
> More details:
>  http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html

Sorry that I didn't looked into your report in more detail last time.
May be I was too confused ;)

I really can't imagine that any code from libpthread/linuxthreads
can be involved when we compile Xtensa with NPTL support.

I believe there is a bug or a change in behavior regarding static
linking prior to 1.0.18.

See:
wbx at helium:~/buildroot/output/build/uclibc-1.0.19 $ grep -r __pthread_initialize_minimal *                                                                          
Binary file lib/libc.a matches
Binary file libc/misc/internals/__uClibc_main.os matches
libc/misc/internals/__uClibc_main.c:extern void weak_function __pthread_initialize_minimal(void);
libc/misc/internals/__uClibc_main.c:extern void __pthread_initialize_minimal(void);
libc/misc/internals/__uClibc_main.c:     * __pthread_initialize_minimal so we can use pthread_locks
libc/misc/internals/__uClibc_main.c:    if (likely(__pthread_initialize_minimal!=NULL))
libc/misc/internals/__uClibc_main.c: __pthread_initialize_minimal();
libpthread/nptl_db/td_ta_thr_iter.c:      /* __pthread_initialize_minimal has not run.  There is just the main
libpthread/nptl_db/td_ta_map_lwp2thr.c:     before __pthread_initialize_minimal has gotten far enough.  They
libpthread/nptl_db/td_thr_validate.c:   /* __pthread_initialize_minimal has not run yet.
libpthread/linuxthreads/internals.h:void __pthread_initialize_minimal (void);
libpthread/linuxthreads/pthread.c:void __pthread_initialize_minimal(void)
libpthread/nptl/init.c:void __pthread_initialize_minimal_internal (void) attribute_hidden;
libpthread/nptl/init.c:__pthread_initialize_minimal_internal (void) 
libpthread/nptl/init.c:strong_alias (__pthread_initialize_minimal_internal, 
libpthread/nptl/init.c:       __pthread_initialize_minimal)
libpthread/nptl/allocatestack.c:   initialize this, since it's done in __pthread_initialize_minimal.  */
libpthread/nptl/sysdeps/generic/libc-tls.c:extern void __pthread_initialize_minimal(void) __attribute__((weak));
libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void)
libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void)
libpthread/nptl/sysdeps/alpha/elf/pt-initfini.c:        bsr     $26, __pthread_initialize_minimal_internal !samegp \n\
Binary file libpthread/nptl/sysdeps/xtensa/libc-tls.os matches
Binary file libpthread/nptl/init.os matches
Binary file utils/getconf matches

libpthread/nptl/sysdeps/xtensa/libc-tls.c includes
libpthread/nptl/sysdeps/generic/libc-tls.c.

Prior to 1.0.18 libpthread.so was linked with special LDFLAGS:
LDFLAGS-libpthread.so +=
$(top_builddir)lib/$(UCLIBC_LDSO_NAME)-$(VERSION).so
$(top_builddir)lib/libdl-$(VERSION).so \
       -Wl,-z,nodelete,-z,initfirst,-init=$(SYMBOL_PREFIX)__pthread_initialize_minimal_internal

May be this might be a reason we see different behavior.

best regards
 Waldemar

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Waldemar Brodkorb]

Hi,
Waldemar Brodkorb wrote,

> Hi Max,
> Max Filippov wrote,
> 
> > On Tue, Nov 29, 2016 at 12:43 AM, Thomas Petazzoni
> > <thomas.petazzoni@free-electrons.com> wrote:
> > > On Tue, 29 Nov 2016 05:16:55 +0100, Waldemar Brodkorb wrote:
> > >> > The --start-group/--end-group ask the linker to loop between -lgcc and
> > >> > -lc until all unresolved symbols have been resolved. So
> > >> > dl_iterate_phdr() being defined in the C library, but used in libgcc, I
> > >> > guess the dynamic way (-lgcc -lc -lgcc) doesn't work.
> > >>
> > >> But why gcc behaves different?
> > >> BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required
> > >> BR+gcc5+musl+static - no extra -static in LDFLAGS required
> > >> BR+gcc5+uClibc-ng-1.0.19 - extra -static required!
> > >
> > > I would suspect it's a fallout of the "merge everything in libc"
> > > change, but I really can't figure out why that would make a difference.
> > >
> > > Could it be that with older uClibc versions, gcc was not finding the
> > > dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind
> > > code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr
> > > symbol, enables unwinding support, which cause this extra reference
> > > from libgcc to libc?
> > 
> > No. The difference is in the __pthread_initialize_minimal function.
> > Prior to uclibc-1.0.18 release it used to work because a call to
> > __pthread_initialize_minimal from __uClibc_init was resolved to
> > a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c
> > which didn't reference pthread_unwind.
> > Now it is resolved to a function from
> > libpthread/linuxthreads/pthread.c, and that file pulls in other
> > unused functions, which reference __GI___pthread_unwind,
> > and all the way down to the dl_iterate_phdr.
> > 
> > More details:
> >  http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html
> 
> Sorry that I didn't looked into your report in more detail last time.
> May be I was too confused ;)
> 
> I really can't imagine that any code from libpthread/linuxthreads
> can be involved when we compile Xtensa with NPTL support.
> 
> I believe there is a bug or a change in behavior regarding static
> linking prior to 1.0.18.
> 
> See:
> wbx at helium:~/buildroot/output/build/uclibc-1.0.19 $ grep -r __pthread_initialize_minimal *                                                                          
> Binary file lib/libc.a matches
> Binary file libc/misc/internals/__uClibc_main.os matches
> libc/misc/internals/__uClibc_main.c:extern void weak_function __pthread_initialize_minimal(void);
> libc/misc/internals/__uClibc_main.c:extern void __pthread_initialize_minimal(void);
> libc/misc/internals/__uClibc_main.c:     * __pthread_initialize_minimal so we can use pthread_locks
> libc/misc/internals/__uClibc_main.c:    if (likely(__pthread_initialize_minimal!=NULL))
> libc/misc/internals/__uClibc_main.c: __pthread_initialize_minimal();
> libpthread/nptl_db/td_ta_thr_iter.c:      /* __pthread_initialize_minimal has not run.  There is just the main
> libpthread/nptl_db/td_ta_map_lwp2thr.c:     before __pthread_initialize_minimal has gotten far enough.  They
> libpthread/nptl_db/td_thr_validate.c:   /* __pthread_initialize_minimal has not run yet.
> libpthread/linuxthreads/internals.h:void __pthread_initialize_minimal (void);
> libpthread/linuxthreads/pthread.c:void __pthread_initialize_minimal(void)
> libpthread/nptl/init.c:void __pthread_initialize_minimal_internal (void) attribute_hidden;
> libpthread/nptl/init.c:__pthread_initialize_minimal_internal (void) 
> libpthread/nptl/init.c:strong_alias (__pthread_initialize_minimal_internal, 
> libpthread/nptl/init.c:       __pthread_initialize_minimal)
> libpthread/nptl/allocatestack.c:   initialize this, since it's done in __pthread_initialize_minimal.  */
> libpthread/nptl/sysdeps/generic/libc-tls.c:extern void __pthread_initialize_minimal(void) __attribute__((weak));
> libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void)
> libpthread/nptl/sysdeps/generic/libc-tls.c:__pthread_initialize_minimal (void)
> libpthread/nptl/sysdeps/alpha/elf/pt-initfini.c:        bsr     $26, __pthread_initialize_minimal_internal !samegp \n\
> Binary file libpthread/nptl/sysdeps/xtensa/libc-tls.os matches
> Binary file libpthread/nptl/init.os matches
> Binary file utils/getconf matches
> 
> libpthread/nptl/sysdeps/xtensa/libc-tls.c includes
> libpthread/nptl/sysdeps/generic/libc-tls.c.
> 
> Prior to 1.0.18 libpthread.so was linked with special LDFLAGS:
> LDFLAGS-libpthread.so +=
> $(top_builddir)lib/$(UCLIBC_LDSO_NAME)-$(VERSION).so
> $(top_builddir)lib/libdl-$(VERSION).so \
>        -Wl,-z,nodelete,-z,initfirst,-init=$(SYMBOL_PREFIX)__pthread_initialize_minimal_internal
> 
> May be this might be a reason we see different behavior.

I think it is not the reason. I think it is just how pthread_cancel
and some other code in NPTL is implemented. There is some circular
dependency we can't change after we have integrated all the code in
libc.a. Musl does not use any functionality from libgcc, so it has
not the problem with the static linking issues we are seeing.

So as long as we have the pthread_cancel code which use libgcc
internally, we can't avoid any breakage from packages not correctly
passing -static while linking.

If anyone thinks the analysis is wrong, please speak up.

best regards
 Waldemar

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Max Filippov]

Hello,

TL;DR: -static is good, order of objects in libc.a is important.

On Tue, Nov 29, 2016 at 9:00 PM, Waldemar Brodkorb <wbx@openadk.org> wrote:
>> On Tue, Nov 29, 2016 at 12:43 AM, Thomas Petazzoni
>> <thomas.petazzoni@free-electrons.com> wrote:
>> > On Tue, 29 Nov 2016 05:16:55 +0100, Waldemar Brodkorb wrote:
>> >> > The --start-group/--end-group ask the linker to loop between -lgcc and
>> >> > -lc until all unresolved symbols have been resolved. So
>> >> > dl_iterate_phdr() being defined in the C library, but used in libgcc, I
>> >> > guess the dynamic way (-lgcc -lc -lgcc) doesn't work.
>> >>
>> >> But why gcc behaves different?
>> >> BR+gcc5+uClibc-ng-1.0.17 - no extra -static in LDFLAGS required
>> >> BR+gcc5+musl+static - no extra -static in LDFLAGS required
>> >> BR+gcc5+uClibc-ng-1.0.19 - extra -static required!
>> >
>> > I would suspect it's a fallout of the "merge everything in libc"
>> > change, but I really can't figure out why that would make a difference.
>> >
>> > Could it be that with older uClibc versions, gcc was not finding the
>> > dl_iterate_phdr symbol in uClibc, and therefore disabling the unwind
>> > code, and now, due to a change in uClibc, gcc sees the dl_iterate_phdr
>> > symbol, enables unwinding support, which cause this extra reference
>> > from libgcc to libc?
>>
>> No. The difference is in the __pthread_initialize_minimal function.
>> Prior to uclibc-1.0.18 release it used to work because a call to
>> __pthread_initialize_minimal from __uClibc_init was resolved to
>> a function defined in libpthread/nptl/sysdeps/generic/libc-tls.c
>> which didn't reference pthread_unwind.
>> Now it is resolved to a function from
>> libpthread/linuxthreads/pthread.c, and that file pulls in other
>> unused functions, which reference __GI___pthread_unwind,
>> and all the way down to the dl_iterate_phdr.
>>
>> More details:
>>  http://lists.busybox.net/pipermail/buildroot/2016-November/177477.html
>
> Sorry that I didn't looked into your report in more detail last time.
> May be I was too confused ;)
>
> I really can't imagine that any code from libpthread/linuxthreads
> can be involved when we compile Xtensa with NPTL support.

Oops, wrong path, wrong searching tool. But the idea:
in the failing case __pthread_initialize_minimal is taken from
libpthread/nptl/init.c and it's way bigger than
__pthread_initialize_minimal used for linking with the uClibc-ng-1.0.17.

Now, I believe that adding the missing -static is the right
solution, because even with uClibc-ng-1.0.17 the linking fails
for a simple application that uses pthread:

$ cat hello.c
#include <pthread.h>

int main()
{
        pthread_create(NULL, NULL, NULL, NULL);
        return 0;
}

$ xtensa-buildroot-linux-uclibc-gcc -pthread hello.c -o hello
.../libgcc.a(unwind-dw2-fde-dip.o): In function `__gthread_mutex_lock':
.../gthr-default.h:748: undefined reference to `dl_iterate_phdr'
collect2: error: ld returned 1 exit status

The only thing that worries me at this point is that now we have
a lot of pthread internals linked into the program image even when
the program doesn't use them.

E.g. autossh built with uClibc-ng-1.0.17:

$ size -A autossh
autossh:
section            size      addr
.text             63771   4194484
.rodata           28112   4258256
.eh_frame           792   4286368
.tbss                 8   4291256
.ctors                8   4291256
.dtors                8   4291264
.jcr                  4   4291272
.data.rel.ro        160   4291276
.data               692   4291440
.bss              15524   4292136
.comment             31         0
.xtensa.info         56         0
.debug_aranges      288         0
.debug_info        2195         0
.debug_abbrev       398         0
.debug_line        6327         0
.debug_frame         40         0
.debug_str          468         0
.debug_loc           40         0
.xt.lit            3752         0
.xt.prop          58596         0
Total            181270

$ nm autossh  | grep pthread | wc -l
10

vs. autossh built with uClibc-ng-1.0.19:

$ size -A autossh
autossh:
section            size      addr
.text             83955   4194484
.rodata           28564   4278440
.eh_frame          1628   4307004
.tdata                4   4312728
.tbss                 8   4312732
.ctors                8   4312732
.dtors                8   4312740
.jcr                  4   4312748
.data.rel.ro        160   4312752
.data               736   4312912
.bss              24516   4313648
.comment             35         0
.xtensa.info         56         0
.debug_aranges      384         0
.debug_info       20096         0
.debug_abbrev      2925         0
.debug_line       14492         0
.debug_frame       1432         0
.debug_str         4789         0
.debug_loc        11212         0
.debug_ranges       608         0
.xt.lit            4384         0
.xt.prop          77364         0
Total            277368

$ nm autossh | grep pthread | wc -l
77

That makes me wonder, how pthread initialization is done
in uClibc-ng-1.0.17 for applications that use pthread and that
don't. So what I see is the following:

- an application that uses pthread_create references
  libpthread/nptl/init.c (because pthread_create.c includes
  allocatestack.c, which references __xidcmd defined in init.c
  from its __nptl_setxid) and thus gets __pthread_initialize_minimal
  from it.

- an application that doesn't use pthread_create gets its weakly
  defined __pthread_initialize_minimal from the
  libpthread/nptl/sysdeps/generic/libc-tls.c, which provides
  __libc_setup_tls referenced from the __uClibc_init.

So we're interested in the following three files:
- init.os (built from libpthread/nptl/init.c, with strong
   __pthread_initialize_minimal)
- libc-tls.os (built from libpthread/nptl/sysdeps/xtensa/libc-tls.c,
   with weak __pthread_initialize_minimal)
- __uClibc_main.os (built from libc/misc/internal/__uClibc_main.c
   with undefined reference to __pthread_initialize_minimal).

Comparing uClibc-ng-1.0.17 with uClibc-ng-1.0.19 I see the following:
1.0.17 had libc.a with libc-tls.os and __uClibc_main.os and
libpthread.a with init.os.
1.0.19 has libc.a with libc-tls.os, __uClibc_main.os and init.os in it
in that order.

I haven't found any related documentation, but apparently with 1.0.19
when a definition of __pthread_initialize_minimal is searched for the
undefined symbol in __uClibc_main.os, the init.os is considered
first. Manually removing and re-adding libc-tls.os and init.os so
that the final order is __uClibc_main.os, libc-tls.os and init.os
restores the 1.0.17 behavior.

> Prior to 1.0.18 libpthread.so was linked with special LDFLAGS:
> LDFLAGS-libpthread.so +=
> $(top_builddir)lib/$(UCLIBC_LDSO_NAME)-$(VERSION).so
> $(top_builddir)lib/libdl-$(VERSION).so \
>        -Wl,-z,nodelete,-z,initfirst,-init=$(SYMBOL_PREFIX)__pthread_initialize_minimal_internal
>
> May be this might be a reason we see different behavior.

AFAICS it only affects shared library, i.e. not related to static linking case.

-- 
Thanks.
-- Max

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Waldemar Brodkorb]

Hi Max,
Max Filippov wrote,

> Hello,
> 
> TL;DR: -static is good, order of objects in libc.a is important.

Thanks for the very detailed analysis.
Do you have any idea how we could restore the old order and keep
the binaries smaller?

best regards
 Waldemar

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Max Filippov]

On Thu, Dec 1, 2016 at 9:25 PM, Waldemar Brodkorb <wbx@openadk.org> wrote:
>> Hello,
>>
>> TL;DR: -static is good, order of objects in libc.a is important.
>
> Thanks for the very detailed analysis.
> Do you have any idea how we could restore the old order and keep
> the binaries smaller?

One ugly solution could be putting __uClibc-main.os before both libc-tls.os
and init.os, something like the following:

diff --git a/libc/misc/internals/Makefile.in b/libc/misc/internals/Makefile.in
index ae094ee..d2400d2 100644
--- a/libc/misc/internals/Makefile.in
+++ b/libc/misc/internals/Makefile.in
@@ -25,7 +25,7 @@ libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS
 else
 libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.os
 endif
-libc-static-y += $(MISC_INTERNALS_OUT)/__uClibc_main.o
+libc-static-y := $(MISC_INTERNALS_OUT)/__uClibc_main.o $(libc-static-y)
 libc-static-$(UCLIBC_FORMAT_FLAT_SEP_DATA) += \
   $(MISC_INTERNALS_OUT)/shared_flat_initfini.o \
   $(MISC_INTERNALS_OUT)/shared_flat_add_library.o

-- 
Thanks.
-- Max

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Waldemar Brodkorb]

Hi Max,
Max Filippov wrote,

> On Thu, Dec 1, 2016 at 9:25 PM, Waldemar Brodkorb <wbx@openadk.org> wrote:
> >> Hello,
> >>
> >> TL;DR: -static is good, order of objects in libc.a is important.
> >
> > Thanks for the very detailed analysis.
> > Do you have any idea how we could restore the old order and keep
> > the binaries smaller?
> 
> One ugly solution could be putting __uClibc-main.os before both libc-tls.os
> and init.os, something like the following:
> 
> diff --git a/libc/misc/internals/Makefile.in b/libc/misc/internals/Makefile.in
> index ae094ee..d2400d2 100644
> --- a/libc/misc/internals/Makefile.in
> +++ b/libc/misc/internals/Makefile.in
> @@ -25,7 +25,7 @@ libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS
>  else
>  libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.os
>  endif
> -libc-static-y += $(MISC_INTERNALS_OUT)/__uClibc_main.o
> +libc-static-y := $(MISC_INTERNALS_OUT)/__uClibc_main.o $(libc-static-y)
>  libc-static-$(UCLIBC_FORMAT_FLAT_SEP_DATA) += \
>    $(MISC_INTERNALS_OUT)/shared_flat_initfini.o \
>    $(MISC_INTERNALS_OUT)/shared_flat_add_library.o
> 

Thanks for the patch, I tried it.
But unforunately it breaks other applications as busybox init.
Not startup possible in Qemu to test the smaller autossh.

best regards
 Waldemar

[Buildroot] [PATCH] autossh: honour LDFLAGS

[Max Filippov]

On Sun, Dec 4, 2016 at 4:06 AM, Waldemar Brodkorb <wbx@openadk.org> wrote:
>> On Thu, Dec 1, 2016 at 9:25 PM, Waldemar Brodkorb <wbx@openadk.org> wrote:
>> >> Hello,
>> >>
>> >> TL;DR: -static is good, order of objects in libc.a is important.
>> >
>> > Thanks for the very detailed analysis.
>> > Do you have any idea how we could restore the old order and keep
>> > the binaries smaller?
>>
>> One ugly solution could be putting __uClibc-main.os before both libc-tls.os
>> and init.os, something like the following:
>>
>> diff --git a/libc/misc/internals/Makefile.in b/libc/misc/internals/Makefile.in
>> index ae094ee..d2400d2 100644
>> --- a/libc/misc/internals/Makefile.in
>> +++ b/libc/misc/internals/Makefile.in
>> @@ -25,7 +25,7 @@ libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.oS
>>  else
>>  libc-shared-y += $(MISC_INTERNALS_OUT)/__uClibc_main.os
>>  endif
>> -libc-static-y += $(MISC_INTERNALS_OUT)/__uClibc_main.o
>> +libc-static-y := $(MISC_INTERNALS_OUT)/__uClibc_main.o $(libc-static-y)
>>  libc-static-$(UCLIBC_FORMAT_FLAT_SEP_DATA) += \
>>    $(MISC_INTERNALS_OUT)/shared_flat_initfini.o \
>>    $(MISC_INTERNALS_OUT)/shared_flat_add_library.o
>
> Thanks for the patch, I tried it.
> But unforunately it breaks other applications as busybox init.

Well, it shows that the current uClibc-ng is broken without pthread
code. The root cause of the init breakage is a call to
_pthread_cleanup_push_defer from mallopt, which currently points
to NULL, because _pthread_cleanup_push_defer is weak and its
definition was removed from __uClibc_main.c in the unification
patch. Restoring it fixes init for me.
I tried to follow the story of these definitions, but it seems to be a
chain of failures and attempted fixes. So I'm not sure this fixes
everything and doesn't break something else ): It sure needs testing.

diff --git a/libc/misc/internals/__uClibc_main.c
b/libc/misc/internals/__uClibc_main.c
index 46e24d8..0446b08 100644
--- a/libc/misc/internals/__uClibc_main.c
+++ b/libc/misc/internals/__uClibc_main.c
@@ -68,6 +68,42 @@ uintptr_t __stack_chk_guard attribute_relro;

 void internal_function _dl_aux_init (ElfW(auxv_t) *av);

+#ifdef __UCLIBC_HAS_THREADS__
+/*
+ * uClibc internal locking requires that we have weak aliases
+ * for dummy functions in case libpthread.a is not linked in.
+ * This needs to be in compilation unit that is pulled always
+ * in or linker will disregard these weaks.
+ */
+
+static int __pthread_return_0 (pthread_mutex_t *unused) { return 0; }
+weak_alias (__pthread_return_0, __pthread_mutex_lock)
+weak_alias (__pthread_return_0, __pthread_mutex_trylock)
+weak_alias (__pthread_return_0, __pthread_mutex_unlock)
+
+int weak_function
+__pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *attr)
+{
+        return 0;
+}
+
+void weak_function
+_pthread_cleanup_push_defer(struct _pthread_cleanup_buffer *__buffer,
+                            void (*__routine) (void *), void *__arg)
+{
+        __buffer->__routine = __routine;
+        __buffer->__arg = __arg;
+}
+
+void weak_function
+_pthread_cleanup_pop_restore(struct _pthread_cleanup_buffer *__buffer,
+                             int __execute)
+{
+        if (__execute)
+                __buffer->__routine(__buffer->__arg);
+}
+#endif /* __UCLIBC_HAS_THREADS__ */
+
 #endif /* !SHARED */

 /* Defeat compiler optimization which assumes function addresses are
never NULL */


-- 
Thanks.
-- Max