[Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

Hi,

thanks for the quick response regarding the irritating warning.

My next problem appears while debugging an application on a Blackfin target
with gdbserver + gdb. Debugging hangs when the first rt_task_create is reached.

Starting in my application:

153     int prio = T_HIPRIO/2;
154     int r = rt_task_create(&thread, thread_name.c_str(), 128*1024, prio, T_FPU|T_JOINABLE);

...

Single-stepi'ing the call, gdb loses contact shortly after clone() in
http://git.uclibc.org/uClibc/tree/libpthread/linuxthreads.old/pthread.c
It looks like this could be related to a manager thread?

These are the last lines from gdb (using stepi)

45  in libc/sysdeps/linux/bfin/clone.c
0x00e4cb00 in clone (fn=@0xec235c: 0xed5a64 <__pthread_manager_event>, child_stack=0xefb5cc, flags=3840, arg=0x0)
     at libc/sysdeps/linux/bfin/clone.c:45
45  in libc/sysdeps/linux/bfin/clone.c
0x00ed8734 in __pthread_initialize_manager () at libpthread/linuxthreads.old/pthread.c:600
600 libpthread/linuxthreads.old/pthread.c: No such file or directory.
     in libpthread/linuxthreads.old/pthread.c
606 in libpthread/linuxthreads.old/pthread.c
0x00ed8738  606 in libpthread/linuxthreads.old/pthread.c
615 in libpthread/linuxthreads.old/pthread.c
614 in libpthread/linuxthreads.old/pthread.c
0x00ed8740  614 in libpthread/linuxthreads.old/pthread.c
616 in libpthread/linuxthreads.old/pthread.c
612 in libpthread/linuxthreads.old/pthread.c
615 in libpthread/linuxthreads.old/pthread.c
616 in libpthread/linuxthreads.old/pthread.c
617 in libpthread/linuxthreads.old/pthread.c
620 in libpthread/linuxthreads.old/pthread.c
0x00ed8754  620 in libpthread/linuxthreads.old/pthread.c
0x00ed4066 in ?? ()
    from /opt/uClinux-2010R1-RC5_tools-RC4/bfin-linux-uclibc/bfin-linux-uclibc/runtime/lib/libpthread.so.0
0x00ed406a in ?? ()
    from /opt/uClinux-2010R1-RC5_tools-RC4/bfin-linux-uclibc/bfin-linux-uclibc/runtime/lib/libpthread.so.0
0x00ed406e in ?? ()
    from /opt/uClinux-2010R1-RC5_tools-RC4/bfin-linux-uclibc/bfin-linux-uclibc/runtime/lib/libpthread.so.0



I need to find out how to prepare the uClibc source so that gdb can find it for further details. But maybe someone yet has an idea about a possible cause (and remedy)?

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Gilles Chanteperdrix]

Kolja Waschk wrote:
> Hi,
> 
> thanks for the quick response regarding the irritating warning.
> 
> My next problem appears while debugging an application on a Blackfin target
> with gdbserver + gdb. Debugging hangs when the first rt_task_create is reached.
> 
> Starting in my application:
> 
> 153     int prio = T_HIPRIO/2;
> 154     int r = rt_task_create(&thread, thread_name.c_str(), 128*1024, prio, T_FPU|T_JOINABLE);

Is there any trace on the serial console?


-- 
                                                                Gilles.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

>> 154     int r = rt_task_create(&thread, thread_name.c_str(), 128*1024, prio, T_FPU|T_JOINABLE);
> Is there any trace on the serial console?

Hasn't been, but I now after I enabled Xenomai debugging and watchdog, at least there's some indication what's going wrong. The "last words" of the application are

...
> Xenomai: registered exported object  (mutexes)
> Xenomai: registered exported object  (mutexes)
> Xenomai: watchdog triggered -- signaling runaway thread 'pfda'
> Backtrace: 0xbcdcc6, 0, 0, 0

And in gdb it shows up as a SIGXCPU:

Program received signal SIGXCPU, CPU time limit exceeded.
...
(gdb) bt
#0  pthread_setcanceltype (type=0, oldtype=0x0) at libpthread/linuxthreads.old/forward.c:161
#1  0x00f35450 in __wrap_pthread_cond_wait (cond=<value optimized out>, mutex=<value optimized out>)
     at ../../../../xenomai-2.5.5.2/src/skins/posix/cond.c:123
#2  0x00be72c6 in consume_socket (ctx=0x837788, sp=0x1778240) at mongoose.c:3871
#3  0x00be753a in worker_thread (ctx=0x837788) at mongoose.c:3914
...


Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

>> Xenomai: watchdog triggered -- signaling runaway thread 'pfda'

The loop causing this is

3870 while (ctx->sq_head == ctx->sq_tail && ctx->stop_flag == 0) {
3871   pthread_cond_wait(&ctx->sq_full, &ctx->mutex);
3872 }

Further investigation showed that (wrapped) pthread_cond_wait actually returns
EPERM instead of waiting, but only when running with gdb(?)

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

> Further investigation showed that (wrapped) pthread_cond_wait actually
> returns EPERM instead of waiting, but only when running with gdb(?)

I verified that all used mutexes and condition variables have been created
successfully and the ctx->mutex was just locked successfully by the thread
calling pthread_cond_wait. For some reason, GDB doesn't let me step further
into pthread_cond_wait so I cannot determine a good reason for EPERM.

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Philippe Gerum]

On Wed, 2010-12-22 at 13:11 +0100, Kolja Waschk wrote:
> >> Xenomai: watchdog triggered -- signaling runaway thread 'pfda'
> 
> The loop causing this is
> 
> 3870 while (ctx->sq_head == ctx->sq_tail && ctx->stop_flag == 0) {
> 3871   pthread_cond_wait(&ctx->sq_full, &ctx->mutex);
> 3872 }
> 
> Further investigation showed that (wrapped) pthread_cond_wait actually returns
> EPERM instead of waiting, but only when running with gdb(?)

Your Makefile may have an issue. Do you happen to link with
libpthread_rt.so in addition to libnative.so?

> 
> Kolja
> 
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

> Your Makefile may have an issue. Do you happen to link with
> libpthread_rt.so in addition to libnative.so?

Yes, because most of the code uses the native API, but one part uses pthread_* calls.. ?

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Philippe Gerum]

On Wed, 2010-12-22 at 15:14 +0100, Kolja Waschk wrote:
> > Your Makefile may have an issue. Do you happen to link with
> > libpthread_rt.so in addition to libnative.so?
> 
> Yes, because most of the code uses the native API, but one part uses pthread_* calls.. ?
> 

Ok, you should have a look at this commit then, and introduce
wrap-link.sh in the link rules for your app: 
http://git.xenomai.org/?p=xenomai-gch.git;a=commit;h=8f5330c9da75f5fdedee5b7cf60e525ee5845300


> Kolja
> 

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Philippe Gerum]

On Wed, 2010-12-22 at 15:17 +0100, Philippe Gerum wrote:
> On Wed, 2010-12-22 at 15:14 +0100, Kolja Waschk wrote:
> > > Your Makefile may have an issue. Do you happen to link with
> > > libpthread_rt.so in addition to libnative.so?
> > 
> > Yes, because most of the code uses the native API, but one part uses pthread_* calls.. ?
> > 
> 
> Ok, you should have a look at this commit then, and introduce
> wrap-link.sh in the link rules for your app: 
> http://git.xenomai.org/?p=xenomai-gch.git;a=commit;h=8f5330c9da75f5fdedee5b7cf60e525ee5845300
> 

In short, it is Xenomai's pthread_cond_wait() version which returns
EPERM to the linuxthread manager, because it does not want to be called
from a non-Xenomai thread. Obviously, the manager thread should not
invoke any Xenomai service in the first place, and this is where the
link magic bites.

> 
> > Kolja
> > 
> 

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

> wrap-link.sh in the link rules for your app:

It seems to target a problem when the application is linked statically, but in my case most of the code is put into a dynamic library. I can use the ..posix.wrappers when building the library and
omit the option when building the executable, but that doesn't seem to make a difference

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Philippe Gerum]

On Wed, 2010-12-22 at 16:25 +0100, Kolja Waschk wrote:
> > wrap-link.sh in the link rules for your app:
> 
> It seems to target a problem when the application is linked statically, but in my case most of the code is put into a dynamic library. I can use the ..posix.wrappers when building the library and
> omit the option when building the executable, but that doesn't seem to make a difference

Check whether the thread manager does attempt to call into the Xenomai
replacement for the pthread_cond_wait call.

> 
> Kolja
> 
> 

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

Hi,

> Check whether the thread manager does attempt to call into the Xenomai
> replacement for the pthread_cond_wait call.

I have to apologize; the pthread_cond_wait -> EPERM appeared after enabling
debug output to trace the actual problem I had when I started this mail thread,
but is seems unrelated. I have completely disabled that part of code, and with
it also disabled the use of all pthread* functions. Only native skin functions
are used by my remaining code and I'm back to the original problem (gdb hangs
at some point when stepi'ing through rt_task_start).

Even though debugging is enabled, nothing is printed on the console. The target
cannot be reached on the console (Ctrl-C doesn't work) and GDB also doesn't 
react on Ctrl-C. telnet to the target works, however. cat /proc/xenomai/sched
shows the half started task as "X Console_Output_Thread":

/ # cat /proc/xenomai/sched
CPU  PID    CLASS  PRI      TIMEOUT   TIMEBASE   STAT       NAME
   0  0      rt      49      -         master     R          ROOT
   0  303    rt      49      -         master     RT         main
   0  305    rt      49      -         master     X          Console_Output_Thread

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

> cannot be reached on the console (Ctrl-C doesn't work) and GDB also doesn't 
> react on Ctrl-C. telnet to the target works, however. cat /proc/xenomai/sched

gdb doesn't seem to follow the task that called rt_task_create, but, maybe, the
manager task? I was able to reproduce the behaviour with a small test program;
gdb also became unresponsive in the same way when I tried to step into or over
rt_task_create, but came back again when the created task finishes or a
breakpoint in the main task was reached. For me, this behavious was unexpected..

In my actual application however I never get past the rt_task_create, even if 
I set a breakpoint at some line somewhen later.

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Philippe Gerum]

On Wed, 2010-12-22 at 19:13 +0100, Kolja Waschk wrote:
> > cannot be reached on the console (Ctrl-C doesn't work) and GDB also doesn't 
> > react on Ctrl-C. telnet to the target works, however. cat /proc/xenomai/sched
> 
> gdb doesn't seem to follow the task that called rt_task_create, but, maybe, the
> manager task? I was able to reproduce the behaviour with a small test program;
> gdb also became unresponsive in the same way when I tried to step into or over
> rt_task_create, but came back again when the created task finishes or a
> breakpoint in the main task was reached. For me, this behavious was unexpected..
> 
> In my actual application however I never get past the rt_task_create, even if 
> I set a breakpoint at some line somewhen later.

What if you don't single step but (c)ontinue?

> 
> Kolja
> 
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

>> In my actual application however I never get past the rt_task_create, even if
>> I set a breakpoint at some line somewhen later.

> What if you don't single step but (c)ontinue?

It "hangs".

But wait, just in this moment I discovered that I can get further if I
set all priorities, i.e. priority of the main task (with rt_task_shadow) and the
tasks where I couldn't get past rt_task_create, to 0 instead of 49 as before. Then at
least "cont" works to the next breakpoint (stepping over with "next" still doesn't).

Debugging with gdb at the moment feels unnecessarily complicated. Very often
the "next" doesn't bring me to the next line in my current source, but to some unexpected
place (e.g. to a rt_mutex_acquire_inner() which was called in a subroutine four
levels deeper..). I don't remember me becoming that confused with my previous
setup (Blackfin 2.6.28 kernel with Xenomai 2.4.7) or in other C++ apps ;)

Kolja

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Philippe Gerum]

On Wed, 2010-12-22 at 20:00 +0100, Kolja Waschk wrote:
> >> In my actual application however I never get past the rt_task_create, even if
> >> I set a breakpoint at some line somewhen later.
> 
> > What if you don't single step but (c)ontinue?
> 
> It "hangs".
> 
> But wait, just in this moment I discovered that I can get further if I
> set all priorities, i.e. priority of the main task (with rt_task_shadow) and the
> tasks where I couldn't get past rt_task_create, to 0 instead of 49 as before. Then at
> least "cont" works to the next breakpoint (stepping over with "next" still doesn't).
> 

This is typically a linuxthreads issue. Because there is a hidden
manager in the loop which wants to get involved into the process of
starting other threads, you can end up with priority issues breaking the
internal protocol. The manager thread is not known from Xenomai, so in
your case, it looks like starved from CPU albeit other threads expect it
to kick in at some point. This can only happen if some of the protocol
is based on busy waiting, which seems to be the case there.

> Debugging with gdb at the moment feels unnecessarily complicated. Very often
> the "next" doesn't bring me to the next line in my current source, but to some unexpected
> place (e.g. to a rt_mutex_acquire_inner() which was called in a subroutine four
> levels deeper..). I don't remember me becoming that confused with my previous
> setup (Blackfin 2.6.28 kernel with Xenomai 2.4.7) or in other C++ apps ;)

This is likely not related to the runtime environment at all. GDB has
not a great track record when it comes to multi-thread debugging even if
the situation improved over time, and things tend to get worse when
linuxthreads are in the loop because of the number of magic interactions
with the manager thread, and use of signals internally. What does have
an impact is your toolchain and how aggressive it is when optimizing the
code. What you see is not what you get in that case.

> 
> Kolja
> 

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> On Wed, 2010-12-22 at 20:00 +0100, Kolja Waschk wrote:
>>>> In my actual application however I never get past the rt_task_create, even if
>>>> I set a breakpoint at some line somewhen later.
>>> What if you don't single step but (c)ontinue?
>> It "hangs".
>>
>> But wait, just in this moment I discovered that I can get further if I
>> set all priorities, i.e. priority of the main task (with rt_task_shadow) and the
>> tasks where I couldn't get past rt_task_create, to 0 instead of 49 as before. Then at
>> least "cont" works to the next breakpoint (stepping over with "next" still doesn't).
>>
> 
> This is typically a linuxthreads issue. Because there is a hidden
> manager in the loop which wants to get involved into the process of
> starting other threads, you can end up with priority issues breaking the
> internal protocol. The manager thread is not known from Xenomai, so in
> your case, it looks like starved from CPU albeit other threads expect it
> to kick in at some point. This can only happen if some of the protocol
> is based on busy waiting, which seems to be the case there.
> 
>> Debugging with gdb at the moment feels unnecessarily complicated. Very often
>> the "next" doesn't bring me to the next line in my current source, but to some unexpected
>> place (e.g. to a rt_mutex_acquire_inner() which was called in a subroutine four
>> levels deeper..). I don't remember me becoming that confused with my previous
>> setup (Blackfin 2.6.28 kernel with Xenomai 2.4.7) or in other C++ apps ;)
> 
> This is likely not related to the runtime environment at all. GDB has
> not a great track record when it comes to multi-thread debugging even if
> the situation improved over time, and things tend to get worse when
> linuxthreads are in the loop because of the number of magic interactions
> with the manager thread, and use of signals internally. What does have
> an impact is your toolchain and how aggressive it is when optimizing the
> code. What you see is not what you get in that case.

And gdb 6.5 over linuxthreads is pretty much outdated... Also, some bugs
have been fixed in Xenomai 2.5.3. So, I think it is useless pursuing
investigating this issue until you can confirm that you can reproduce it
with current versions of libc, gdb and Xenomai.

-- 
                                                                Gilles.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Waschk,Kolja]

Hi,

> And gdb 6.5 over linuxthreads is pretty much outdated... Also, some bugs

That's what comes in the most current Blackfin toolchain release 2010R1

> have been fixed in Xenomai 2.5.3. So, I think it is useless pursuing

I'm using 2.5.5.2. Just referred to 2.4.7 being my base for comparison

Kolja


-- 
mr. kolja waschk - haubach-39 - 22765 hh - germany
fon +49 40 889130-34 - fax -35 - http://www.ixo.de

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Gilles Chanteperdrix]

Waschk,Kolja wrote:
> Hi,
> 
>> And gdb 6.5 over linuxthreads is pretty much outdated... Also, some bugs
> 
> That's what comes in the most current Blackfin toolchain release 2010R1

Really? gdb 6.5 was released in 2006, this is seriously outdated. Also,
please check that you are using the exact same version of cross-gdb and
gdbserver

> 
>> have been fixed in Xenomai 2.5.3. So, I think it is useless pursuing
> 
> I'm using 2.5.5.2. Just referred to 2.4.7 being my base for comparison

Ok. I asked, you did not answer, so looked back at the posts you made,
and only one post mentioned a version, and it was 2.5.3.

-- 
                                                                Gilles.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Gilles Chanteperdrix]

Kolja Waschk wrote:
>>> Xenomai: watchdog triggered -- signaling runaway thread 'pfda'
> 
> The loop causing this is
> 
> 3870 while (ctx->sq_head == ctx->sq_tail && ctx->stop_flag == 0) {
> 3871   pthread_cond_wait(&ctx->sq_full, &ctx->mutex);
> 3872 }
> 
> Further investigation showed that (wrapped) pthread_cond_wait actually returns
> EPERM instead of waiting, but only when running with gdb(?)

This looks like an old bug. What version of Xenomai are you using?

-- 
                                                                Gilles.

[Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Kolja Waschk]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1944 bytes --]

Hi,

I've made up a small test program for my issue with pthread_cond_wait and gdb.
Sorry for posting code again, I've made it as compact as possible.

The behaviour of the attached program depends on whether it is run via
gdbserver or without.


When run via gdbserver, the pthread_cond_wait sometimes returns EPERM:

> Remote debugging from host 10.0.10.10
> T0: pthread_cond_wait: 1
> ... repeats for T0 and T1 ...
> T1: pthread_cond_wait: 1
> T1: pthread_cond_wait: 1
> T0: pthread_cond_wait: 1
> T2: pthread_cond_wait: 0
> Child exited with retcode = 0


Started directly, it prints what I would expect:

> T0: pthread_cond_wait: 0
> T1: pthread_cond_wait: 0
> T2: pthread_cond_wait: 0


Am I missing something when configuring the threads, cond variable etc.?

About my environment:

  - Xenomai 2.5.5.2, unmodified
  - blackfin FDPIC toolchain 2010R1-RC4 (gcc 4.3.5, gdb 6.6, gdbserver 6.5)
  - blackfin-linux-dist 2010R1-RC5 (kernel 2.6.34 with I-pipe upgraded to 1.15-01)


I compile "try.c" with the following command line:

bfin-linux-uclibc-gcc \
   -I/opt/uClinux/blackfin-linux-dist/staging/usr/include \
   -L/opt/uClinux/blackfin-linux-dist/staging/usr/lib \
   -Wl,@/opt/uClinux/blackfin-linux-dist/staging/usr/lib/posix.wrappers \
   -g -fPIC -fmessage-length=0 -mfast-fp -D_GNU_SOURCE -D_REENTRANT -D__XENO__ \
   -o try try.c -lxenomai -lpthread_rt


The gdb script contains only a few commands:

set verbose on
set solib-absolute-prefix notexistent
set solib-search-path /opt/uClinux/blackfin-linux-dist/staging/usr/lib:/opt/uClinux/bfin-linux-uclibc/bfin-linux-uclibc/runtime/lib
file try 
target remote 10.0.10.9:2222
break main
cont
cont


The pthread_cond_wait, returning EPERM immediately, causes a lockup in a
program that I'm trying to debug (see old thread about gdb/gdbserver hanging).
I currently know no measure against this behaviour other than to sleep for a
while in that case...

Thanks for any hints!
Kolja

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: TEXT/x-csrc; name=try.c, Size: 1307 bytes --]

/* 2010-12-27 - test program for pthread_cond_wait */

#include <stdio.h>
#include <string.h>
#include <pthread.h>

#define PRIORITY 1
#define NUM_THREADS 3

volatile int run;
pthread_cond_t cond;
pthread_mutex_t mutex;

void* test_thread(void *cookie)
{
    int tnum = *(int*)cookie;

    pthread_mutex_lock(&mutex);
    while (run)
    {
        int r = pthread_cond_wait(&cond, &mutex);
        printf("T%d: pthread_cond_wait: %d\n", tnum, r);
    }
    pthread_mutex_unlock(&mutex);

    return NULL;
}

int main (void)
{
    int i, r;
    int tnum[NUM_THREADS];
    pthread_t tid[NUM_THREADS];

    run = 1;
    r = pthread_cond_init(&cond, NULL); if (r!=0) printf("cond_init: %d\n", r);
    r = pthread_mutex_init(&mutex, NULL); if (r!=0) printf("mutex_init: %d\n", r);

    for (i=0; i<NUM_THREADS; i++)
    {
        tnum[i] = i;
        r = pthread_create(&(tid[i]), NULL, test_thread, &(tnum[i]));
        if (r!=0) printf("create(%d): %d\n", i, r);
    }

    sleep(1);
    run = 0;
    r = pthread_cond_broadcast(&cond); if (r!=0) printf("cond_broadcast: %d\n", r);

    for (i=0; i<NUM_THREADS; i++)
    {
        void *rp;
        r = pthread_join(tid[i], &rp);
        if (r!=0) printf("join(%d): %d\n", i, r);
    }

    return r;
}

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Philippe Gerum]

On Mon, 2010-12-27 at 15:44 +0100, Kolja Waschk wrote:
> Hi,
> 
> I've made up a small test program for my issue with pthread_cond_wait and gdb.
> Sorry for posting code again, I've made it as compact as possible.
> 
> The behaviour of the attached program depends on whether it is run via
> gdbserver or without.

The test code creates SCHED_OTHER threads. This is likely not what you
wanted to do.

> 
> 
> When run via gdbserver, the pthread_cond_wait sometimes returns EPERM:
> 
> > Remote debugging from host 10.0.10.10
> > T0: pthread_cond_wait: 1
> > ... repeats for T0 and T1 ...
> > T1: pthread_cond_wait: 1
> > T1: pthread_cond_wait: 1
> > T0: pthread_cond_wait: 1
> > T2: pthread_cond_wait: 0
> > Child exited with retcode = 0
> 
> 
> Started directly, it prints what I would expect:
> 
> > T0: pthread_cond_wait: 0
> > T1: pthread_cond_wait: 0
> > T2: pthread_cond_wait: 0
> 
> 
> Am I missing something when configuring the threads, cond variable etc.?
> 
> About my environment:
> 
>   - Xenomai 2.5.5.2, unmodified
>   - blackfin FDPIC toolchain 2010R1-RC4 (gcc 4.3.5, gdb 6.6, gdbserver 6.5)
>   - blackfin-linux-dist 2010R1-RC5 (kernel 2.6.34 with I-pipe upgraded to 1.15-01)
> 
> 
> I compile "try.c" with the following command line:
> 
> bfin-linux-uclibc-gcc \
>    -I/opt/uClinux/blackfin-linux-dist/staging/usr/include \
>    -L/opt/uClinux/blackfin-linux-dist/staging/usr/lib \
>    -Wl,@/opt/uClinux/blackfin-linux-dist/staging/usr/lib/posix.wrappers \
>    -g -fPIC -fmessage-length=0 -mfast-fp -D_GNU_SOURCE -D_REENTRANT -D__XENO__ \
>    -o try try.c -lxenomai -lpthread_rt
> 
> 
> The gdb script contains only a few commands:
> 
> set verbose on
> set solib-absolute-prefix notexistent
> set solib-search-path /opt/uClinux/blackfin-linux-dist/staging/usr/lib:/opt/uClinux/bfin-linux-uclibc/bfin-linux-uclibc/runtime/lib
> file try 
> target remote 10.0.10.9:2222
> break main
> cont
> cont
> 
> 
> The pthread_cond_wait, returning EPERM immediately, causes a lockup in a
> program that I'm trying to debug (see old thread about gdb/gdbserver hanging).
> I currently know no measure against this behaviour other than to sleep for a
> while in that case...
> 
> Thanks for any hints!
> Kolja
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Kolja Waschk]

Hi,

> The test code creates SCHED_OTHER threads. This is likely not what you
> wanted to do.

It indeed doesn't describe my original problem, but while compacting the test code, I found that it doesn't make a difference. The behaviour is the same (or at least the symptoms are the same) ..

   1. ... with the threads created with prio>0, EXPLICIT_SCHED and SCHED_FIFO
   2. ... with additionally the main thread being set up for SCHED_FIFO too

Kolja

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Philippe Gerum]

On Mon, 2010-12-27 at 17:50 +0100, Kolja Waschk wrote:
> Hi,
> 
> > The test code creates SCHED_OTHER threads. This is likely not what you
> > wanted to do.
> 
> It indeed doesn't describe my original problem, but while compacting the test code, I found that it doesn't make a difference. The behaviour is the same (or at least the symptoms are the same) ..
> 
>   1. ... with the threads created with prio>0, EXPLICIT_SCHED and SCHED_FIFO
>   2. ... with additionally the main thread being set up for SCHED_FIFO too

A possible explanation is that we know ptrace() is sending your app
linux signals, particularly with legacy linuxthreads under the hood.
Since your thread relaxes due to printf() as well, the combination of
both events normally bumps the current thread out of real-time mode AND
stops the RPI boost too, which may cause the next thread to preempt.

What happens if you only log the debug info in some global array, and
dump its contents after all threads are joined, instead of issuing
printf() in the thread body?

> 
> Kolja
> 

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Kolja Waschk]

> Since your thread relaxes due to printf() as well, the combination of
> What happens if you only log the debug info in some global array, and

As a quick test I just removed the printf() in the thread body. When run under
gdbserver, the target seems to locks up. Only hard reset can bring it back.
Without gdbserver it behaves well.

Kolja

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Philippe Gerum]

On Mon, 2010-12-27 at 18:27 +0100, Kolja Waschk wrote:
> > Since your thread relaxes due to printf() as well, the combination of
> > What happens if you only log the debug info in some global array, and
> 
> As a quick test I just removed the printf() in the thread body. When run under
> gdbserver, the target seems to locks up. Only hard reset can bring it back.
> Without gdbserver it behaves well.

Which means that pthread_cond_wait() likely returns with an error.

> 
> Kolja
> 
> 
> 

-- 
Philippe.

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait (was: gdb/gdbserver hangs...)

[Kolja Waschk]

Hi,

>> gdbserver, the target seems to locks up. Only hard reset can bring it back.
> Which means that pthread_cond_wait() likely returns with an error.

I later replaced the printf with a pthread_yield() to work around the "lockup" and made the threads record the errors, if any, to variables instead of complex output; as you suggested.

The pthread_cond_wait always(!) return EPERM in the first two threads when run with gdb,  only the pthread_cond_wait in the third thread succeeds. That appears to be the same result as with the original test code. Even the very first pthread_cond_wait() already returned EPERM, even before the first printf().

Kolja

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait

[Gilles Chanteperdrix]

Kolja Waschk wrote:
> Hi,
> 
> I've made up a small test program for my issue with pthread_cond_wait and gdb.
> Sorry for posting code again, I've made it as compact as possible.

It would have been nice if you had posted the code in the mail body,
instead of attaching it, this would have allowed us to comment it.

So, here is the comment without the code. When pthread_cond_wait fails,
if you enter pthread_cond_wait again, it will fail again, forever, and
you get a lockup. sched_yield will likely not change anything. What you
should do is remove the printf when there is no error. And when there is
an error, print the error, then exit.

Now, about pthread_cond_wait, the error looks a lot like something which
was supposed to be fixed in the 2.5 branch. Could you run the
posix-cond-torture test case to verify that what was supposed to be
fixed is indeed fixed with the libc/gdb combination you are using?

-- 
                                                                Gilles.

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait

[Kolja Waschk]

Hi,

> posix-cond-torture test case to verify that what was supposed to be

When run standalone, it seems to pass; the output is

mple_condwait
relative_condwait
absolute_condwait
sig_norestart_condwait
sig_restart_condwait
sig_norestart_condwait_mutex
sig_restart_condwait_mutex
sig_norestart_double
sig_restart_double
cond_destroy_whilewait
Test OK

When started via gdbserver, I get a NULL pointer access immediately after gdb
attaches to the target. I cannot say if that's related or if it just proves
some kind of inconsistency in my setup. Maybe the trace can shed some light on
it. I again tried to keep my posting compact by not quoting the full exception
context but if you think I missed some important info, I have the full log
still available

...
  RETS: <0x016f6eda> [ /lib/libpthread.so.0 + 0x6eda ]
  PC  : <0x016f6f10> [ /lib/libpthread.so.0 + 0x6f10 ]
DCPLB_FAULT_ADDR: <0x00000000> /* Maybe null pointer? */
ICPLB_FAULT_ADDR: <0x016f6f10> [ /lib/libpthread.so.0 + 0x6f10 ]
> libpthread/linuxthreads.old/signals.c:127
...
       FAULT : <0x016f6f10> [ /lib/libpthread.so.0 + 0x6f10 ] P1 = [P3]
      Source : <0x016f6f0e> [ /lib/libpthread.so.0 + 0x6f0e ] P3 = [P2]
    5 Target : <0x016f6eee> [ /lib/libpthread.so.0 + 0x6eee ]
      Source : <0x016f6ee2> [ /lib/libpthread.so.0 + 0x6ee2 ] IF CC JUMP pcrel (BP)
    6 Target : <0x016f6eda> [ /lib/libpthread.so.0 + 0x6eda ]
      Source : <0x016f6e64> [ /lib/libpthread.so.0 + 0x6e64 ] RTS
    7 Target : <0x016f6e60> [ /lib/libpthread.so.0 + 0x6e60 ]
      Source : <0x016f6e3c> [ /lib/libpthread.so.0 + 0x6e3c ] JUMP.S
    8 Target : <0x016f6e18> [ /lib/libpthread.so.0 + 0x6e18 ]
      Source : <0x016f6ed6> [ /lib/libpthread.so.0 + 0x6ed6 ] CALL pcrel
    9 Target : <0x016f6ec8> [ /lib/libpthread.so.0 + 0x6ec8 ]
      Source : <0xffa00d12> { __common_int_entry + 0xce } RTI
   10 Target : <0xffa00cb0> { __common_int_entry + 0x6c }
      Source : <0xffa00f80> { _evt_system_call + 0x64 } JUMP.S
   11 Target : <0xffa00f80> { _evt_system_call + 0x64 }
      Source : <0xffa00982> { _system_call + 0xee } RTS
   12 Target : <0xffa0097c> { _system_call + 0xe8 }
      Source : <0x000031fe> { _do_notify_resume + 0x52 } RTS
   13 Target : <0x000031fa> { _do_notify_resume + 0x4e }
      Source : <0x000031dc> { _do_notify_resume + 0x30 } IF CC JUMP pcrel (BP)
   14 Target : <0x000031ce> { _do_notify_resume + 0x22 }
      Source : <0x000031aa> { _do_signal + 0x126 } RTS
   15 Target : <0x000031a0> { _do_signal + 0x11c }
      Source : <0x00003118> { _do_signal + 0x94 } IF CC JUMP pcrel (BP)
...
Return addresses in stack:
     address : <0x016f5f66> [ /lib/libpthread.so.0 + 0x5f66 ]
> libpthread/linuxthreads.old/mutex.c:316

For reference again the OS info, it's blackfin-linux-dist 2010R1-RC5 which, as
far as the lib names tell, uses linuxthreads.old of uClibc-0.9.29. I-Pipe is
updated to 1.15-01 in dist's kernel 2.6.34 and Xenomai was updated to 2.5.5.2.

Originally, I-Pipe 1.14-02 comes with the dist and Xenomai 2.5.3. I updated them because the
latter always showed an "kernel is anterior to 2.5.2" warning at startup and
2.5.5.2 looked like it would change something in that regard on NOMMU, and in
order to have at least Xenomai at a version that might be worth examining.

Thanks for looking at this issue
Kolja

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait

[Kolja Waschk]

>> posix-cond-torture test case to verify that what was supposed to be

As a side note, cond-torture-native (not -posix) locks up with gdbserver/gdb after
displaying just "simple_condwait". My problems seem to be not just related to the POSIX skin.

Kolja

Re: [Xenomai-help] gdb/gdbserver and pthread_cond_wait

[Gilles Chanteperdrix]

Kolja Waschk wrote:
>>> posix-cond-torture test case to verify that what was supposed to be
> 
> As a side note, cond-torture-native (not -posix) locks up with gdbserver/gdb after
> displaying just "simple_condwait". My problems seem to be not just related to the POSIX skin.

Ok. I can confirm I am unable to reproduce this issue on the NPTL based
setups I have access to, the LD_ASSUME_KERNEL environment variable no
longer seems to do the trick to force linuxthreads. So, I could not test
a linuxthreads based system.

Your issue seems to be that the condvar waiting syscall is restarted
whereas it should not, and indeed, native and posix condition variables
work the same way. To move forward, we should follow the syscall path
and see at what point it is restarted whereas it should not when running
under gdb.

-- 
                                                                Gilles.

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Andreas Glatz]

Hi,

>
>> Check whether the thread manager does attempt to call into the  
>> Xenomai
>> replacement for the pthread_cond_wait call.
>
> I have to apologize; the pthread_cond_wait -> EPERM appeared after  
> enabling
> debug output to trace the actual problem I had when I started this  
> mail thread,
> but is seems unrelated. I have completely disabled that part of  
> code, and with
> it also disabled the use of all pthread* functions. Only native  
> skin functions
> are used by my remaining code and I'm back to the original problem  
> (gdb hangs
> at some point when stepi'ing through rt_task_start).
>
> Even though debugging is enabled, nothing is printed on the  
> console. The target
> cannot be reached on the console (Ctrl-C doesn't work) and GDB also  
> doesn't
> react on Ctrl-C. telnet to the target works, however. cat /proc/ 
> xenomai/sched
> shows the half started task as "X Console_Output_Thread":
>
> / # cat /proc/xenomai/sched
> CPU  PID    CLASS  PRI      TIMEOUT   TIMEBASE   STAT       NAME
>    0  0      rt      49      -         master     R          ROOT
>    0  303    rt      49      -         master     RT         main
>    0  305    rt      49      -         master     X           
> Console_Output_Thread
>
> Kolja
>

Which version of gdb/gdbserver are you using? We had a similar  
problem before upgrading to gdb/gdbserver 7.0.
Unfortunately, I didn't hv enough time to investigate the issue. So i  
just accepted it being some kind of magic...

Andreas

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Andreas Glatz]

Hi again,

>
>> Check whether the thread manager does attempt to call into the  
>> Xenomai
>> replacement for the pthread_cond_wait call.
>
> I have to apologize; the pthread_cond_wait -> EPERM appeared after  
> enabling
> debug output to trace the actual problem I had when I started this  
> mail thread,
> but is seems unrelated. I have completely disabled that part of  
> code, and with
> it also disabled the use of all pthread* functions. Only native  
> skin functions
> are used by my remaining code and I'm back to the original problem  
> (gdb hangs
> at some point when stepi'ing through rt_task_start).
>
> Even though debugging is enabled, nothing is printed on the  
> console. The target
> cannot be reached on the console (Ctrl-C doesn't work) and GDB also  
> doesn't
> react on Ctrl-C. telnet to the target works, however. cat /proc/ 
> xenomai/sched
> shows the half started task as "X Console_Output_Thread":
>
> / # cat /proc/xenomai/sched
> CPU  PID    CLASS  PRI      TIMEOUT   TIMEBASE   STAT       NAME
>    0  0      rt      49      -         master     R          ROOT
>    0  303    rt      49      -         master     RT         main
>    0  305    rt      49      -         master     X           
> Console_Output_Thread
>

I forgot to describe our problem: Gdb hung at rt_task_start() when  
trying to step through it. But debugging worked fine if I set a  
breakpoint in the task body, way after any rt_task_start(). Btw, do  
you call rt_task_start() from a RT task body?

Andreas

Re: [Xenomai-help] gdb/gdbserver hangs during rt_task_create

[Kolja Waschk]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1474 bytes --]

Hi,

> I forgot to describe our problem: Gdb hung at rt_task_start() when trying to 
> step through it. But debugging worked fine if I set a breakpoint in the task

Looks similar, but I observed the problem already with first rt_task_create()
(which, however, causes a call to rt_task_start() for the manager thread(?)).

> body, way after any rt_task_start(). Btw, do you call rt_task_start() from a 
> RT task body?

Yes. To be exact, I call rt_task_create and rt_task_start from my main task,
which is RT after calling rt_task_shadow.

I've attached a small example that allows to examine the behaviour. The following
happens in my setup, i.e. when I start "try" on the Blackfin target with
gdbserver and connect with gdb from the host:

1. gdb set to break at "main" runs to the line with mlockall(...)
2. I can step line by line using "next" to the one with rt_task_create

Regardless now whether I type "next" or set a breakpoint in one of the following
lines up to the rt_task_join line and "cont", gdb just prints the following
lines and seems to freeze. Ctrl-C doesn't do anything. Finally after 8 seconds
the subtask finishes and gdb reports "Program exited normally".


24 if (r == 0) r = rt_task_create(&subtask, "subtask", 0, PRIO, T_FPU|T_JOINABLE);
(gdb)  next
[New Thread 304]
Reading in symbols for ../../../../xenomai-2.5.5.2/src/skins/native/task.c...done.
<freeze>


Kolja


P.S. It's gdbserver 6.5 and gdb 6.6; unmodified Blackfin tools 2010R1-RC4.





[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: TEXT/X-CSRC; NAME=try.c, Size: 838 bytes --]

#include <stdio.h>
#include <sys/mman.h>
#include <native/task.h>

#define PRIO 49

void subtask_entry(void *cookie)
{
  sleep(8);
}

int main ()
{
  int r;
  int i;
  RT_TASK maintask;
  RT_TASK subtask;

  mlockall(MCL_CURRENT|MCL_FUTURE);

  r = rt_task_shadow(&maintask, "maintask", PRIO, 0);
  if (r != 0) printf("Couldn't rt_task_shadow: ");

  if (r == 0) r = rt_task_create(&subtask, "subtask", 0, PRIO, T_FPU|T_JOINABLE);
  if (r != 0) printf("Couldn't rt_task_create: ");

  if (r == 0) r = rt_task_start(&subtask, subtask_entry, NULL);
  if (r != 0) printf("Couldn't rt_task_start: ");

  if (r == 0) for (i=0; i<5; i++) { printf("sleep %d\n", i); sleep(1); }

  if (r == 0) r = rt_task_join(&subtask);
  if (r != 0) printf("Couldn't rt_task_join: ");

  printf("r=%d\n", r);
  return r;
}