[Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Mike McTernan]

Hi,

Checking xeno-config.in from 2.4.1, it has the following:

XENO_POSIX_LDFLAGS="-L${staging}${libdir} -lpthread_rt -lpthread -lrt
@XENO_USER_APP_LDFLAGS@"

On Blackfin uClinux 2008R1.5 linking fails with -lpthread_rt and
-lpthread in that order:

bfin-uclinux-gcc -I/usr/xenomai/include -I/usr/xenomai/include/posix
-D_GNU_SOURCE -D_REENTRANT -D__XENO__ -o hello hello.o
-Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib -lpthread_rt
-lpthread -lrt
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
x.os): In function `__pthread_once_fork_child':
libpthread/linuxthreads.old/mutex.c:357: undefined reference to
`___wrap_pthread_cond_init'
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
x.os): In function `__pthread_once':
libpthread/linuxthreads.old/mutex.c:308: undefined reference to
`___wrap_pthread_cond_wait'
libpthread/linuxthreads.old/mutex.c:325: undefined reference to
`___wrap_pthread_cond_broadcast'
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
x.os): In function `pthread_once_cancelhandler':
libpthread/linuxthreads.old/mutex.c:281: undefined reference to
`___wrap_pthread_cond_broadcast'
collect2: ld returned 1 exit status
make: *** [hello] Error 1

It looks like the uClinux pthread implementation makes use of its own
functions and hence needs the library containing the wrappers specified
after the pthread lib to enable resolving of references, as per the ld
manual:

"Normally, an archive is searched only once in the order that it is
specified on the command line. If a symbol in that archive is needed to
resolve an undefined symbol referred to by an object in an archive that
appears later on the command line, the linker would not be able to
resolve that reference."

I'd have thought there are potentially some circular dependencies
between the pthread and pthread_rt libs.  Would it help portability to
surround these options with -Wl,--start-group <libs> -Wl,--end-group?
i.e.

XENO_POSIX_LDFLAGS="-L${staging}${libdir} -Wl,--start-group -lpthread_rt
-lpthread -Wl,--end-group -lrt @XENO_USER_APP_LDFLAGS@"

I've done this locally and I can build the apps without re-ordering the
libraries.

Regards,

Mike

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Philippe Gerum]

Mike McTernan wrote:
> Hi,
> 
> Checking xeno-config.in from 2.4.1, it has the following:
> 

Please upgrade. The Xenomai/blackfin port is actively maintained, and bugs have
been fixed since then.

> XENO_POSIX_LDFLAGS="-L${staging}${libdir} -lpthread_rt -lpthread -lrt
> @XENO_USER_APP_LDFLAGS@"
> 
> On Blackfin uClinux 2008R1.5 linking fails with -lpthread_rt and
> -lpthread in that order:
> 
> bfin-uclinux-gcc -I/usr/xenomai/include -I/usr/xenomai/include/posix

This should rather be bfin-linux-uclibc-gcc to produce fdpic binaries;
bfin-uclinux-gcc should be used to build the ELF kernel or FLAT format apps.
Xenomai user-space support should be built using bfin-linux-uclibc-gcc.

> -D_GNU_SOURCE -D_REENTRANT -D__XENO__ -o hello hello.o
> -Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib -lpthread_rt
> -lpthread -lrt
> /opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
> x.os): In function `__pthread_once_fork_child':
> libpthread/linuxthreads.old/mutex.c:357: undefined reference to
> `___wrap_pthread_cond_init'
> /opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
> x.os): In function `__pthread_once':
> libpthread/linuxthreads.old/mutex.c:308: undefined reference to
> `___wrap_pthread_cond_wait'
> libpthread/linuxthreads.old/mutex.c:325: undefined reference to
> `___wrap_pthread_cond_broadcast'
> /opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
> x.os): In function `pthread_once_cancelhandler':
> libpthread/linuxthreads.old/mutex.c:281: undefined reference to
> `___wrap_pthread_cond_broadcast'
> collect2: ld returned 1 exit status
> make: *** [hello] Error 1
> 
> It looks like the uClinux pthread implementation makes use of its own
> functions and hence needs the library containing the wrappers specified
> after the pthread lib to enable resolving of references, as per the ld
> manual:
> 
> "Normally, an archive is searched only once in the order that it is
> specified on the command line. If a symbol in that archive is needed to
> resolve an undefined symbol referred to by an object in an archive that
> appears later on the command line, the linker would not be able to
> resolve that reference."
> 
> I'd have thought there are potentially some circular dependencies
> between the pthread and pthread_rt libs.  Would it help portability to
> surround these options with -Wl,--start-group <libs> -Wl,--end-group?
> i.e.
> 
> XENO_POSIX_LDFLAGS="-L${staging}${libdir} -Wl,--start-group -lpthread_rt
> -lpthread -Wl,--end-group -lrt @XENO_USER_APP_LDFLAGS@"
> 
> I've done this locally and I can build the apps without re-ordering the
> libraries.
> 

Which is likely not going to run properly. uClibc does not want to call the
Xenomai wrappers from its internal code, but its own code, because it should
apply to regular linuxthreads, not Xenomai-enabled RT threads. The only sane way
to solve this kind of dependencies properly would be to go for a two-stage
linking phase, i.e. first you do a partial link with your app and the Xenomai
libraries applying the wrappers (-Wl,@/usr/xenomai/lib/posix.wrappers), then you
complete the link without any wrapping against libpthread.so.

Here is a trivial Makefile I have just used to produce an executable against
2008-r1.5-rc3 and 2.4.1, without any problem. Obviously this also depends on the
references dragged in by your example code; mine was pulling condvar, mutex and
thread support through the common pthread_* calls. If this simple Makefile does
not work with a properly built Xenomai userland support, I'd be interested to
see your example code. TIA,

X := /usr/xenomai/blackfin
CFLAGS := $(shell $(X)/bin/xeno-config --posix-cflags)
LDFLAGS := $(shell $(X)/bin/xeno-config --posix-ldflags)
CC :=  $(shell $(X)/bin/xeno-config --cc)

all: hello

hello: hello.c
	$(CC) -o $@ $< $(CFLAGS) $(LDFLAGS)

clean:
	$(RM) hello

> Regards,
> 
> Mike
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
> 


-- 
Philippe.

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Mike McTernan]

Philippe Gerum wrote:
> 
> Mike McTernan wrote:
> > Hi,
> >
> > Checking xeno-config.in from 2.4.1, it has the following:
> >
> 
> Please upgrade. The Xenomai/blackfin port is actively maintained, and
bugs
> have been fixed since then.

Apologies - I mistyped.  I checked 2.4.6.1.

> > XENO_POSIX_LDFLAGS="-L${staging}${libdir} -lpthread_rt -lpthread
-lrt
> > @XENO_USER_APP_LDFLAGS@"
> >
> > On Blackfin uClinux 2008R1.5 linking fails with -lpthread_rt and
> > -lpthread in that order:
> >
> > bfin-uclinux-gcc -I/usr/xenomai/include -I/usr/xenomai/include/posix
> 
> This should rather be bfin-linux-uclibc-gcc to produce fdpic binaries;
> bfin-uclinux-gcc should be used to build the ELF kernel or FLAT format
> apps.
> Xenomai user-space support should be built using
bfin-linux-uclibc-gcc.

For various reasons we build all our apps as FLAT or ZFLAT, hence use
bfin-uclinux-gcc -Wl,-elf2flt to link, which I think is common for
embedded.  

I presumed the binary format doesn't matter to Xenomai - is FLAT
supported?

> > -D_GNU_SOURCE -D_REENTRANT -D__XENO__ -o hello hello.o
> > -Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib -lpthread_rt
> > -lpthread -lrt
> >
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
> > x.os): In function `__pthread_once_fork_child':
> > libpthread/linuxthreads.old/mutex.c:357: undefined reference to
> > `___wrap_pthread_cond_init'
> >
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
> > x.os): In function `__pthread_once':
> > libpthread/linuxthreads.old/mutex.c:308: undefined reference to
> > `___wrap_pthread_cond_wait'
<snip>
> >
> > It looks like the uClinux pthread implementation makes use of its
own
> > functions and hence needs the library containing the wrappers
specified
> > after the pthread lib to enable resolving of references, as per the
ld
> > manual:
> >
> > "Normally, an archive is searched only once in the order that it is
> > specified on the command line. If a symbol in that archive is needed
to
> > resolve an undefined symbol referred to by an object in an archive
that
> > appears later on the command line, the linker would not be able to
> > resolve that reference."
> >
> > I'd have thought there are potentially some circular dependencies
> > between the pthread and pthread_rt libs.  Would it help portability
to
> > surround these options with -Wl,--start-group <libs>
-Wl,--end-group?
> > i.e.
> >
> > XENO_POSIX_LDFLAGS="-L${staging}${libdir} -Wl,--start-group
-lpthread_rt
> > -lpthread -Wl,--end-group -lrt @XENO_USER_APP_LDFLAGS@"
> >
> > I've done this locally and I can build the apps without re-ordering
the
> > libraries.
> >
> 
> Which is likely not going to run properly. uClibc does not want to
call
> the
> Xenomai wrappers from its internal code, but its own code, because it
> should
> apply to regular linuxthreads, not Xenomai-enabled RT threads. The
only
> sane way
> to solve this kind of dependencies properly would be to go for a
two-stage
> linking phase, i.e. first you do a partial link with your app and the
> Xenomai
> libraries applying the wrappers
(-Wl,@/usr/xenomai/lib/posix.wrappers),
> then you
> complete the link without any wrapping against libpthread.so.

I think you are using ELF, so maybe you don't see the problem - the
dynamic linker maybe patching up the internal pthread references?  I'm
using FLAT, which is essentially a static link, so everything has to be
fully resolved.

While I'm not familiar with the internals of Xenomai, I'd have thought
that intercepting the internal call would be inefficient, but not
harmful.  Presumably we end up in the normal pthread lib because the
calling thread has no Xenomai context.  When the wrapper is called from
pthread internally, it should always find there is no Xenomai-thread
context for the caller and hence go back into the normal pthread lib.
FWIW the compiled test app runs and produces the expected behaviour,
although is too trivial to be a thorough test.  

> Here is a trivial Makefile I have just used to produce an executable
> against 2008-r1.5-rc3 and 2.4.1, without any problem. Obviously this 
> also depends on the references dragged in by your example code; mine
> was pulling condvar, mutex and thread support through the common 
> pthread_* calls. If this simple Makefile does not work with a properly

> built Xenomai userland support, I'd be interested to see your example 
> code. TIA,
> 
> X := /usr/xenomai/blackfin
> CFLAGS := $(shell $(X)/bin/xeno-config --posix-cflags)
> LDFLAGS := $(shell $(X)/bin/xeno-config --posix-ldflags)
> CC :=  $(shell $(X)/bin/xeno-config --cc)
> 
> all: hello
> 
> hello: hello.c
> 	$(CC) -o $@ $< $(CFLAGS) $(LDFLAGS)
> 
> clean:
> 	$(RM) hello
> 

Unfortunately this does not link either (note I added -Wl,-elf2flt).
Here's the program I am trying to compile, the compile output, and the
xeno-config output:

#include <stdio.h>
#include <sys/mman.h>
#include <pthread.h>

void *demo(void *arg)
{
  printf("Hello world\n");
}

int main(int argc, char* argv[])
{
    pthread_t tid;

    mlockall(MCL_CURRENT | MCL_FUTURE);
    pthread_create(&tid, NULL, demo, NULL);
    sleep(1);
    return 0;
}

$ make
bfin-uclinux-gcc -o hello hello.c -I/usr/xenomai/include
-I/usr/xenomai/include/posix -D_GNU_SOURCE -D_REENTRANT -D__XENO__
-Wl,-elf2flt -Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib
-lpthread_rt -lpthread -lrt
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
x.os): In function `__pthread_once_fork_child':
libpthread/linuxthreads.old/mutex.c:357: undefined reference to
`___wrap_pthread_cond_init'
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
x.os): In function `__pthread_once':
libpthread/linuxthreads.old/mutex.c:308: undefined reference to
`___wrap_pthread_cond_wait'
libpthread/linuxthreads.old/mutex.c:325: undefined reference to
`___wrap_pthread_cond_broadcast'
/opt/uClinux/bfin-uclinux/bfin-uclinux/runtime/usr/lib/libpthread.a(mute
x.os): In function `pthread_once_cancelhandler':
libpthread/linuxthreads.old/mutex.c:281: undefined reference to
`___wrap_pthread_cond_broadcast'
collect2: ld returned 1 exit status
make: *** [hello] Error 1 

$ /usr/xenomai/bin/xeno-config
xeno-config --verbose
    --version="2.4.0"
    --cc="bfin-uclinux-gcc"
    --arch="blackfin"
    --prefix="/usr/xenomai"
    --xeno-cflags="-I/usr/xenomai/include -D_GNU_SOURCE -D_REENTRANT
-D__XENO__"
    --xeno-ldflags="-L/usr/xenomai/lib -lpthread "
    --posix-cflags="-I/usr/xenomai/include -I/usr/xenomai/include/posix
-D_GNU_SOURCE -D_REENTRANT -D__XENO__"
    --posix-ldflags="-L/usr/xenomai/lib -lpthread_rt -lpthread -lrt"
    --library-dir="/usr/xenomai/lib"
Usage xeno-config OPTIONS
Options :
        --help
        --v,--verbose
        --version
        --cc
        --arch
        --prefix
        --xeno-cflags
        --xeno-ldflags
        --posix-cflags
        --posix-ldflags
        --lib*-dir,--libdir,--user-libdir

Regards,

Mike

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Philippe Gerum]

Mike McTernan wrote:
> Philippe Gerum wrote:
>> Mike McTernan wrote:
>>> Hi,
>>>


> 
> I think you are using ELF, so maybe you don't see the problem - the
> dynamic linker maybe patching up the internal pthread references?  I'm
> using FLAT, which is essentially a static link, so everything has to be
> fully resolved.

Yes, this is where FLAT is biting you. All refs have to be resolved, so the
linker tries to pick whatever it needs at link time.

> 
> While I'm not familiar with the internals of Xenomai, I'd have thought
> that intercepting the internal call would be inefficient, but not
> harmful.  Presumably we end up in the normal pthread lib because the
> calling thread has no Xenomai context.  When the wrapper is called from
> pthread internally, it should always find there is no Xenomai-thread
> context for the caller and hence go back into the normal pthread lib.
> FWIW the compiled test app runs and produces the expected behaviour,
> although is too trivial to be a thorough test.  
> 

An internal uclibc call to grab e.g. a regular mutex from a Xenomai thread would
likely be an issue though.

>> Here is a trivial Makefile I have just used to produce an executable
>> against 2008-r1.5-rc3 and 2.4.1, without any problem. Obviously this 
>> also depends on the references dragged in by your example code; mine
>> was pulling condvar, mutex and thread support through the common 
>> pthread_* calls. If this simple Makefile does not work with a properly
> 
>> built Xenomai userland support, I'd be interested to see your example 
>> code. TIA,
>>
>> X := /usr/xenomai/blackfin
>> CFLAGS := $(shell $(X)/bin/xeno-config --posix-cflags)
>> LDFLAGS := $(shell $(X)/bin/xeno-config --posix-ldflags)
>> CC :=  $(shell $(X)/bin/xeno-config --cc)
>>
>> all: hello
>>
>> hello: hello.c
>> 	$(CC) -o $@ $< $(CFLAGS) $(LDFLAGS)
>>
>> clean:
>> 	$(RM) hello
>>
> 
> Unfortunately this does not link either (note I added -Wl,-elf2flt).
> Here's the program I am trying to compile, the compile output, and the
> xeno-config output:
> 

This is all due to the FLAT mode. So I guess your next option is two-stage
linking in you don't want to rely on shared fdpic libs.

> #include <stdio.h>
> #include <sys/mman.h>
> #include <pthread.h>
> 

-- 
Philippe.

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Mike McTernan]

Philippe Gerum wrote:
> Mike McTernan wrote:
> > Philippe Gerum wrote:
> >> Mike McTernan wrote:
> >>> Hi,
> >>>
> > I think you are using ELF, so maybe you don't see the problem - the
> > dynamic linker maybe patching up the internal pthread references?
I'm
> > using FLAT, which is essentially a static link, so everything has to
be
> > fully resolved.
> 
> Yes, this is where FLAT is biting you. All refs have to be resolved,
so
> the linker tries to pick whatever it needs at link time.
<snip> 
> This is all due to the FLAT mode. So I guess your next option is
two-stage
> linking in you don't want to rely on shared fdpic libs.

Unfortunately moving away from ZFLAT isn't trivial for us and is an
unlikely step.

I think Xenomai should support FLAT though, it's just this link problem,
right?

Unless you have some hints as to how the two-stage link is done, I'm
going to go and try to get this working and report back later.

Regards,

Mike

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Philippe Gerum]

Mike McTernan wrote:
> Philippe Gerum wrote:
>> Mike McTernan wrote:
>>> Philippe Gerum wrote:
>>>> Mike McTernan wrote:
>>>>> Hi,
>>>>>
>>> I think you are using ELF, so maybe you don't see the problem - the
>>> dynamic linker maybe patching up the internal pthread references?
> I'm
>>> using FLAT, which is essentially a static link, so everything has to
> be
>>> fully resolved.
>> Yes, this is where FLAT is biting you. All refs have to be resolved,
> so
>> the linker tries to pick whatever it needs at link time.
> <snip> 
>> This is all due to the FLAT mode. So I guess your next option is
> two-stage
>> linking in you don't want to rely on shared fdpic libs.
> 
> Unfortunately moving away from ZFLAT isn't trivial for us and is an
> unlikely step.
> 
> I think Xenomai should support FLAT though, it's just this link problem,
> right?
>

Yes, Xenomai does not care about the underlying binary format. It is just a
convenience for us to follow the default one uClinux uses on the Blackfin.

> Unless you have some hints as to how the two-stage link is done, I'm
> going to go and try to get this working and report back later.
> 

I was thinking about something along these lines (you probably need the flat
directive to be passed to the linker as well); totally untested, but the idea is
illustrated:

X=/usr/xenomai
CFLAGS= $(shell $(X)/bin/xeno-config --posix-cflags) -g
WRAPPERS := $(X)/lib/posix.wrappers
LDFLAGS_1 = -Bstatic -L$(X)/lib @$(WRAPPERS)
LDFLAGS_2 = -static -L$(X)/lib -lpthread -lrt
CC = $(shell $(X)/bin/xeno-config --cc)
LD = $(shell $(CC) --print-prog-name=ld)

all: hello

hello: hello-stage1.o
	$(CC) -o $@ $< $(LDFLAGS_2)

hello-stage1.o: hello.o $(X)/lib/libpthread_rt.a
	$(LD) -r -o $@ $^ $(LDFLAGS_1)

hello.o: hello.c
	$(CC) -c $< $(CFLAGS)

clean:
	rm -f *.o hello

> Regards,
> 
> Mike
> 


-- 
Philippe.

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Mike McTernan]

Philippe Gerum wrote:
> Mike McTernan wrote:
> > I think Xenomai should support FLAT though, it's just this link
problem,
> > right?
> 
> Yes, Xenomai does not care about the underlying binary format. It is
just
> a convenience for us to follow the default one uClinux uses on the
> Blackfin.

I think FLAT is possibly more popular for embedded apps which Blackfin
goes into.  From our experience ZFLAT are much smaller and load quickly.

> I was thinking about something along these lines

You were thinking very well.  That does indeed produce a linked binary
which runs.

Any chance this can be added to future Xenomai releases?  Would it be
okay for me to add a page to your wiki describing how to static link? 

Regards,

Mike

Re: [Xenomai-help] lib order fails for xeno-config --posix-ldflags on Blackfin

[Philippe Gerum]

Mike McTernan wrote:
> Philippe Gerum wrote:
>> Mike McTernan wrote:
>>> I think Xenomai should support FLAT though, it's just this link
> problem,
>>> right?
>> Yes, Xenomai does not care about the underlying binary format. It is
> just
>> a convenience for us to follow the default one uClinux uses on the
>> Blackfin.
> 
> I think FLAT is possibly more popular for embedded apps which Blackfin
> goes into.  From our experience ZFLAT are much smaller and load quickly.
> 
>> I was thinking about something along these lines
> 
> You were thinking very well.  That does indeed produce a linked binary
> which runs.
> 
> Any chance this can be added to future Xenomai releases?  Would it be
> okay for me to add a page to your wiki describing how to static link? 
>

Sure.

> Regards,
> 
> Mike
> 


-- 
Philippe.