[Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Mauro Salvini]

[-- Attachment #1: Type: text/plain, Size: 3899 bytes --]

Hi, 

as from mail subject, I have an issue with rt_task_join() when called into shared object destructor.

I run xenomai 2.5.5.2 IPipe patch 2.7-4 on x86 2.6.35.7 kernel, Ubuntu Lucid 10.04.1. 
I have a simple code attached to mail, where main program opens a shared object with dlopen(). Shared object constructor launches a joinable real-time task. Main program sleeps 5 seconds and then calls dlclose(). Shared object destructor breaks real time task cycle and joins task, but rt_join_call() hangs application indefinitely. 

Initially it seems me to be due to this libc bug: 
https://bugzilla.redhat.com/show_bug.cgi?id=549813 

In facts my system originally has libc6 version 2.11.1 that contains this bug (attached example to bugtrace hangs on pthread_join() call). 
So I patched libc with suggested patch (that was applied on libc6 2.12, but unfortunately I cannot install it from .deb package because it was built for Ubuntu 10.10 only). Then I rebuild deb package with debuild command and updated my libc6 library: pthread_join() issue disappears, but rt_task_join() issue stills remain. 

I tried to run same xenomai-patched kernel on an Ubuntu 10.10 system (that comes with libc6 version 2.12), same result obtained (rt_task_join() hangs). 

I run test application under gdb, this is the backtrace for each task when it hangs: 

(gdb) info thread
  3 Thread 0xb7e34b70 (LWP 1684)  0xb7fe2424 in __kernel_vsyscall ()
  2 Thread 0xb7e42b70 (LWP 1683)  0xb7fe2424 in __kernel_vsyscall ()
* 1 Thread 0xb7e436d0 (LWP 1680)  0xb7fe2424 in __kernel_vsyscall ()


thread 1: 
(gdb) bt
#0  0xb7fe2424 in __kernel_vsyscall ()
#1  0xb7fb7b5d in pthread_join () from /lib/tls/i686/cmov/libpthread.so.0
#2  0xb7fd7181 in rt_task_join () from /usr/xenomai/lib/libnative.so.3
#3  0xb7e357ad in TestModExit () at TestMod.c:35
#4  0xb7e35668 in __do_global_dtors_aux () from ./libTestMod.so
#5  0xb7e35820 in _fini () from ./libTestMod.so
#6  0xb7ff578e in ?? () from /lib/ld-linux.so.2
#7  0xb7ff6247 in ?? () from /lib/ld-linux.so.2
#8  0xb7fa8ca4 in ?? () from /lib/tls/i686/cmov/libdl.so.2
#9  0xb7ff0836 in ?? () from /lib/ld-linux.so.2
#10 0xb7fa909c in ?? () from /lib/tls/i686/cmov/libdl.so.2
#11 0xb7fa8cda in dlclose () from /lib/tls/i686/cmov/libdl.so.2
#12 0x080486b1 in main (argc=1, argv=0xbffff864) at main.c:18


thread 2: 
(gdb) bt
#0  0xb7fe2424 in __kernel_vsyscall ()
#1  0xb7fbe736 in nanosleep () from /lib/tls/i686/cmov/libpthread.so.0
#2  0xb7fad92e in printer_loop () from /usr/xenomai/lib/librtdk.so.0
#3  0xb7fb696e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#4  0xb7f1ba4e in clone () from /lib/tls/i686/cmov/libc.so.6


thread 3: 
(gdb) bt
#0  0xb7fe2424 in __kernel_vsyscall ()
#1  0xb7fbdaf9 in __lll_lock_wait () from /lib/tls/i686/cmov/libpthread.so.0
#2  0xb7fb9149 in _L_lock_839 () from /lib/tls/i686/cmov/libpthread.so.0
#3  0xb7fb8fdb in pthread_mutex_lock () from /lib/tls/i686/cmov/libpthread.so.0
#4  0xb7ff45cd in ?? () from /lib/ld-linux.so.2
#5  0xb7f524a2 in ?? () from /lib/tls/i686/cmov/libc.so.6
#6  0xb7ff0836 in ?? () from /lib/ld-linux.so.2
#7  0xb7f525a1 in ?? () from /lib/tls/i686/cmov/libc.so.6
#8  0xb7f526bb in __libc_dlopen_mode () from /lib/tls/i686/cmov/libc.so.6
#9  0xb7fbfb47 in pthread_cancel_init () from /lib/tls/i686/cmov/libpthread.so.0
#10 0xb7fbfcbd in _Unwind_ForcedUnwind () from /lib/tls/i686/cmov/libpthread.so.0
#11 0xb7fbd788 in __pthread_unwind () from /lib/tls/i686/cmov/libpthread.so.0
#12 0xb7fb79e0 in pthread_exit () from /lib/tls/i686/cmov/libpthread.so.0
#13 0xb7fd8665 in rt_task_trampoline () from /usr/xenomai/lib/libnative.so.3
#14 0xb7fb696e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
#15 0xb7f1ba4e in clone () from /lib/tls/i686/cmov/libc.so.6


It seems to be another issue into libc6. Or my Xenomai system could be corrupted/misconfigured elsewhere?

Thanks in advance, regards.


[-- Attachment #2: sample.zip --]
[-- Type: application/x-zip-compressed, Size: 1494 bytes --]

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Gilles Chanteperdrix]

Mauro Salvini wrote:
> Hi, 
> 
> as from mail subject, I have an issue with rt_task_join() when called into shared object destructor.
> 
> I run xenomai 2.5.5.2 IPipe patch 2.7-4 on x86 2.6.35.7 kernel, Ubuntu Lucid 10.04.1. 

Do you have the same issue with the latest version of the branch you
use? That is 2.5.6?

Has not pthread_join alone the same behaviour?

-- 
                                                                Gilles.

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Gilles Chanteperdrix]

Mauro Salvini wrote:
> Hi, 
> 
> as from mail subject, I have an issue with rt_task_join() when called into shared object destructor.
> 
> I run xenomai 2.5.5.2 IPipe patch 2.7-4 on x86 2.6.35.7 kernel, Ubuntu Lucid 10.04.1. 
> I have a simple code attached to mail, where main program opens a shared object with dlopen(). Shared object constructor launches a joinable real-time task. Main program sleeps 5 seconds and then calls dlclose(). Shared object destructor breaks real time task cycle and joins task, but rt_join_call() hangs application indefinitely. 
> 
> Initially it seems me to be due to this libc bug: 
> https://bugzilla.redhat.com/show_bug.cgi?id=549813 
> 
> In facts my system originally has libc6 version 2.11.1 that contains this bug (attached example to bugtrace hangs on pthread_join() call). 
> So I patched libc with suggested patch (that was applied on libc6 2.12, but unfortunately I cannot install it from .deb package because it was built for Ubuntu 10.10 only). Then I rebuild deb package with debuild command and updated my libc6 library: pthread_join() issue disappears, but rt_task_join() issue stills remain. 
> 
> I tried to run same xenomai-patched kernel on an Ubuntu 10.10 system (that comes with libc6 version 2.12), same result obtained (rt_task_join() hangs). 
> 
> I run test application under gdb, this is the backtrace for each task when it hangs: 
> 
> (gdb) info thread
>   3 Thread 0xb7e34b70 (LWP 1684)  0xb7fe2424 in __kernel_vsyscall ()
>   2 Thread 0xb7e42b70 (LWP 1683)  0xb7fe2424 in __kernel_vsyscall ()
> * 1 Thread 0xb7e436d0 (LWP 1680)  0xb7fe2424 in __kernel_vsyscall ()
> 
> 
> thread 1: 
> (gdb) bt
> #0  0xb7fe2424 in __kernel_vsyscall ()
> #1  0xb7fb7b5d in pthread_join () from /lib/tls/i686/cmov/libpthread.so.0
> #2  0xb7fd7181 in rt_task_join () from /usr/xenomai/lib/libnative.so.3
> #3  0xb7e357ad in TestModExit () at TestMod.c:35
> #4  0xb7e35668 in __do_global_dtors_aux () from ./libTestMod.so
> #5  0xb7e35820 in _fini () from ./libTestMod.so
> #6  0xb7ff578e in ?? () from /lib/ld-linux.so.2
> #7  0xb7ff6247 in ?? () from /lib/ld-linux.so.2
> #8  0xb7fa8ca4 in ?? () from /lib/tls/i686/cmov/libdl.so.2
> #9  0xb7ff0836 in ?? () from /lib/ld-linux.so.2
> #10 0xb7fa909c in ?? () from /lib/tls/i686/cmov/libdl.so.2
> #11 0xb7fa8cda in dlclose () from /lib/tls/i686/cmov/libdl.so.2
> #12 0x080486b1 in main (argc=1, argv=0xbffff864) at main.c:18
> 
> 
> thread 2: 
> (gdb) bt
> #0  0xb7fe2424 in __kernel_vsyscall ()
> #1  0xb7fbe736 in nanosleep () from /lib/tls/i686/cmov/libpthread.so.0
> #2  0xb7fad92e in printer_loop () from /usr/xenomai/lib/librtdk.so.0
> #3  0xb7fb696e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #4  0xb7f1ba4e in clone () from /lib/tls/i686/cmov/libc.so.6
> 
> 
> thread 3: 
> (gdb) bt
> #0  0xb7fe2424 in __kernel_vsyscall ()
> #1  0xb7fbdaf9 in __lll_lock_wait () from /lib/tls/i686/cmov/libpthread.so.0
> #2  0xb7fb9149 in _L_lock_839 () from /lib/tls/i686/cmov/libpthread.so.0
> #3  0xb7fb8fdb in pthread_mutex_lock () from /lib/tls/i686/cmov/libpthread.so.0
> #4  0xb7ff45cd in ?? () from /lib/ld-linux.so.2
> #5  0xb7f524a2 in ?? () from /lib/tls/i686/cmov/libc.so.6
> #6  0xb7ff0836 in ?? () from /lib/ld-linux.so.2
> #7  0xb7f525a1 in ?? () from /lib/tls/i686/cmov/libc.so.6
> #8  0xb7f526bb in __libc_dlopen_mode () from /lib/tls/i686/cmov/libc.so.6
> #9  0xb7fbfb47 in pthread_cancel_init () from /lib/tls/i686/cmov/libpthread.so.0
> #10 0xb7fbfcbd in _Unwind_ForcedUnwind () from /lib/tls/i686/cmov/libpthread.so.0
> #11 0xb7fbd788 in __pthread_unwind () from /lib/tls/i686/cmov/libpthread.so.0
> #12 0xb7fb79e0 in pthread_exit () from /lib/tls/i686/cmov/libpthread.so.0
> #13 0xb7fd8665 in rt_task_trampoline () from /usr/xenomai/lib/libnative.so.3
> #14 0xb7fb696e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
> #15 0xb7f1ba4e in clone () from /lib/tls/i686/cmov/libc.so.6
> 
> 
> It seems to be another issue into libc6. Or my Xenomai system could be corrupted/misconfigured elsewhere?

It looks like a typical pthread_join deadlock. The thread you are
joining is locked on a pthread mutex, that some other thread (I would
say, the one calling pthread_join) has. It can not work. You should not
call pthread_join while holding a mutex.

If this is not the issue, would you please take the time to post a
self-contained test case which I can run to reproduce the issue?

Thanks.


-- 
                                                                Gilles.

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Mauro Salvini]

[-- Attachment #1: Type: text/plain, Size: 436 bytes --]

> Do you have the same issue with the latest version of the branch you 
> use? That is 2.5.6? 

Hi, 

Xenomai 2.5.6 on 2.6.35.9 kernel, IPipe patch 2.8-04 has same behavior. 

Now I'm compiling 2.6.37 kernel with 2.9-00 IPipe patch, I'll try it and give you results. 


> Has not pthread_join alone the same behaviour? 

Yes, in past I got this behavior on pthread_join()/rt-task_join(), work-arounded without using T_JOINABLE flag. 



[-- Attachment #2: Type: text/html, Size: 650 bytes --]

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Mauro Salvini]

[-- Attachment #1.1: Type: text/plain, Size: 1489 bytes --]



> It looks like a typical pthread_join deadlock. The thread you are 
> joining is locked on a pthread mutex, that some other thread (I would 
> say, the one calling pthread_join) has. It can not work. You should not 
> call pthread_join while holding a mutex. 

Yes, it looks like this, but I don't use any mutex in my task. It looks like dlclose() locks a mutex used also by pthread_join() (that was called into shared object destructor), and here is deadlock. But executing realtime task as normal thread does not raise this issue. 

Another strange behaviour: if task was not created with rt_task_create()-rt_task_start(), but becomes real time from a thread with rt_task_shadow(), it works properly. 

Summarizing: 
- create task with rt_task_create()-rt_task_start() and join it in shared object destructor with rt_task_join() -> deadlock 
- create a thread with pthread_create(), make it real-time with rt_task_shadow() and join it in destructor with rt_task_join() -> works fine 
- create a thread with pthread_create(), join it in destructor with pthread_join() -> works fine 

Same behaviors on kernel 2.6.37 with Xenomai 2.5.6 Ipipe 2.9-00 . 

> If this is not the issue, would you please take the time to post a 
> self-contained test case which I can run to reproduce the issue? 

Was attached on my first mail, I re-attach it to this mail. I also added 2nd and 3rd case explained above: you can select each case with three #defines into TestMod.c. 

Many thanks, regards. 


[-- Attachment #1.2: Type: text/html, Size: 1971 bytes --]

[-- Attachment #2: sample.zip --]
[-- Type: application/x-zip-compressed, Size: 1751 bytes --]

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Gilles Chanteperdrix]

Mauro Salvini wrote:
> 
>> It looks like a typical pthread_join deadlock. The thread you are 
>> joining is locked on a pthread mutex, that some other thread (I
>> would say, the one calling pthread_join) has. It can not work. You
>> should not call pthread_join while holding a mutex.
> 
> Yes, it looks like this, but I don't use any mutex in my task. It
> looks like dlclose() locks a mutex used also by pthread_join() (that
> was called into shared object destructor), and here is deadlock. But
> executing realtime task as normal thread does not raise this issue.

The trace you sent clearly shows that the problem happens because
rt_task_trampoline calls pthread_exit. In your tests with
rt_task_shadow/pthread_create, did you try to use pthread_exit? Or could
you try replacing the call to pthread_exit in rt_task_trampoline with a
return?


-- 
                                                                Gilles.

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Gilles Chanteperdrix]

Gilles Chanteperdrix wrote:
> Mauro Salvini wrote:
>>> It looks like a typical pthread_join deadlock. The thread you are 
>>> joining is locked on a pthread mutex, that some other thread (I
>>> would say, the one calling pthread_join) has. It can not work. You
>>> should not call pthread_join while holding a mutex.
>> Yes, it looks like this, but I don't use any mutex in my task. It
>> looks like dlclose() locks a mutex used also by pthread_join() (that
>> was called into shared object destructor), and here is deadlock. But
>> executing realtime task as normal thread does not raise this issue.
> 
> The trace you sent clearly shows that the problem happens because
> rt_task_trampoline calls pthread_exit. In your tests with
> rt_task_shadow/pthread_create, did you try to use pthread_exit? Or could
> you try replacing the call to pthread_exit in rt_task_trampoline with a
> return?

Answering to myself: no, in your examples you did not use pthread_exit,
that is the difference with the rt_task_create case.

So, could you try the following patch?

diff --git a/src/skins/native/task.c b/src/skins/native/task.c
index be4ea2c..70ba6f7 100644
--- a/src/skins/native/task.c
+++ b/src/skins/native/task.c
@@ -113,7 +113,7 @@ static void *rt_task_trampoline(void *cookie)

       fail:

-	pthread_exit((void *)err);
+	return (void *)err;
 }

 int rt_task_create(RT_TASK *task,


-- 
                                                                Gilles.

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Mauro Salvini]

> Gilles Chanteperdrix wrote: 
>> Mauro Salvini wrote: 
>>>> It looks like a typical pthread_join deadlock. The thread you are 
>>>> joining is locked on a pthread mutex, that some other thread (I 
>>>> would say, the one calling pthread_join) has. It can not work. You 
>>>> should not call pthread_join while holding a mutex. 
>>> Yes, it looks like this, but I don't use any mutex in my task. It 
>>> looks like dlclose() locks a mutex used also by pthread_join() (that 
>>> was called into shared object destructor), and here is deadlock. But 
>>> executing realtime task as normal thread does not raise this issue. 
>> 
>> The trace you sent clearly shows that the problem happens because 
>> rt_task_trampoline calls pthread_exit. In your tests with 
>> rt_task_shadow/pthread_create, did you try to use pthread_exit? Or could 
>> you try replacing the call to pthread_exit in rt_task_trampoline with a 
>> return? 
> 
> Answering to myself: no, in your examples you did not use pthread_exit, 
> that is the difference with the rt_task_create case. 
> 
> So, could you try the following patch? 
> 
> diff --git a/src/skins/native/task.c b/src/skins/native/task.c 
> index be4ea2c..70ba6f7 100644 
> --- a/src/skins/native/task.c 
> +++ b/src/skins/native/task.c 
> @@ -113,7 +113,7 @@ static void *rt_task_trampoline(void *cookie) 
> 
> fail: 
> 
> - pthread_exit((void *)err); 
> + return (void *)err; 
> } 
> 
> int rt_task_create(RT_TASK *task, 
> 
> 

Yes, patch works: rt_task_join works correctly now. 

By the way, before apply patch, as you suggest in your penultimate mail, I tried to use pthread_exit in tests with rt_task_shadow/pthread_create: application hangs as with rt_task_create().

So, I deduce that pthread_exit must not be called into shared object threads/tasks, to avoid joining hangs when object destructor was called. Is this a libc bug or is the normal way of coding?

Thank you very much Gilles for your precious help! 
Regards 

Re: [Xenomai-help] rt_task_join() call hangs in shared lib destructor

[Gilles Chanteperdrix]

Mauro Salvini wrote:
>> Gilles Chanteperdrix wrote:
>>> Mauro Salvini wrote:
>>>>> It looks like a typical pthread_join deadlock. The thread you
>>>>> are joining is locked on a pthread mutex, that some other
>>>>> thread (I would say, the one calling pthread_join) has. It
>>>>> can not work. You should not call pthread_join while holding
>>>>> a mutex.
>>>> Yes, it looks like this, but I don't use any mutex in my task.
>>>> It looks like dlclose() locks a mutex used also by
>>>> pthread_join() (that was called into shared object destructor),
>>>> and here is deadlock. But executing realtime task as normal
>>>> thread does not raise this issue.
>>> The trace you sent clearly shows that the problem happens because
>>>  rt_task_trampoline calls pthread_exit. In your tests with 
>>> rt_task_shadow/pthread_create, did you try to use pthread_exit?
>>> Or could you try replacing the call to pthread_exit in
>>> rt_task_trampoline with a return?
>> Answering to myself: no, in your examples you did not use
>> pthread_exit, that is the difference with the rt_task_create case.
>> 
>> 
>> So, could you try the following patch?
>> 
>> diff --git a/src/skins/native/task.c b/src/skins/native/task.c 
>> index be4ea2c..70ba6f7 100644 --- a/src/skins/native/task.c +++
>> b/src/skins/native/task.c @@ -113,7 +113,7 @@ static void
>> *rt_task_trampoline(void *cookie)
>> 
>> fail:
>> 
>> - pthread_exit((void *)err); + return (void *)err; }
>> 
>> int rt_task_create(RT_TASK *task,
>> 
>> 
> 
> Yes, patch works: rt_task_join works correctly now.
> 
> By the way, before apply patch, as you suggest in your penultimate
> mail, I tried to use pthread_exit in tests with
> rt_task_shadow/pthread_create: application hangs as with
> rt_task_create().
> 
> So, I deduce that pthread_exit must not be called into shared object
> threads/tasks, to avoid joining hangs when object destructor was
> called. Is this a libc bug or is the normal way of coding?
> 
> Thank you very much Gilles for your precious help! Regards

It definitely has nothing to do with Xenomai, whether this is a glibc
bug or not, only glibc maintainers can answer.

But the workaround is easy, just create a function to shutdown the
thread in the .so object. And call this function before dlclose.

-- 
					    Gilles.